HyperDynamics: Meta-Learning Object and Agent Dynamics with Hypernetworks
Authors: Zhou Xian
Shamit Lal
Hsiao-Yu Tung
Emmanouil Antonios Platanios
Katerina Fragkiadaki
Venue: ICLR 2021 Poster (2021)

Abstract: We propose HyperDynamics, a dynamics meta-learning framework that conditions on an agent’s interactions with the environment and optionally its visual observations, and generates the parameters of neural dynamics models based on inferred properties of the dynamical system. Physical and visual properties of the environment that are not part of the low-dimensional state yet affect its temporal dynamics are inferred from the interaction history and visual observations, and are implicitly captured in the generated parameters. We test HyperDynamics on a set of object pushing and locomotion tasks. It outperforms existing dynamics models in the literature that adapt to environment variations by learning dynamics over high dimensional visual observations, capturing the interactions of the agent in recurrent state representations, or using gradient-based meta-optimization. We also show our method matches the performance of an ensemble of separately trained experts, while also being able to generalize well to unseen environment variations at test time. We attribute its good performance to the multiplicative interactions between the inferred system properties—captured in the generated parameters—and the low-dimensional state representation of the dynamical system.

Published as a conference paper at ICLR 2021
HYPERDYNAMICS: META-LEARNING OBJECT AND AGENT DYNAMICS WITH HYPERNETWORKS
Zhou Xian, Shamit Lal, Hsiao-Yu Yu Fish Tung, Emmanouil Antonios Platanios & Katerina Fragkiadaki School of Computer Science, Carnegie Mellon University, Pittsburgh, PA 15213, USA {xi (xianzl, shamith t.1, htung, e. w.w.pppllaatoo............ e cs cmu. edu
ABSTRACT
We propose HyperDynamicss a dynamics meta-learning framework that condi- tions on agent's interactions with the environment and optionally its visual observations, and generates the parameters of neural dynamics models based on inferred properties of the dynamical system. Physical and visual properties of the environment that are not part of the low-dimensional state yet affect its temporal dynamics are inferred from the interaction history and visual observations, and are set
implicitly captured in the generated parameters. We test HyperIDynmices of object pushing and locomotion tasks. It outperforms existing dynamics models in the literature that adapt to environment variations by learning dynamics over high dimensional visual observations, capturing the interactions of the agent in re- current state representations, or using gradient-based meta-optimization. We also show our method matches the performance of an ensemble of separately trained experts, while also being able to generalize well to unseen environment variations at test time. We attribute its good performance to the multiplicative interactions between the inferred system and the low-dimensional state representation of the dynamical system.
in the generated parameters--
INTRODUCTION
Humans learn dynamics models that predict results of their interactions with the environment, and use such predictions for selecting actions to achieve intended goals (Miall & Wolpert 1996 Haruno et al. 1999). These models capture intuitive physics and mechanics of the world and are remarkably versatile: they are expressive and can be applied all kinds of environments that we encounter in our daily lives, with varying dynamics and diverse visual and physical properties. In addition, humans do not consider these models fixed over the course of interaction; we observe how the environment behaves in response to our actions and quickly adapt model for the situation hand based on new observations. Let us consider the scenario of moving an object on the ground. We can infer how heavy the object is by simply looking at it, and not move as much as expected, we might realize it is heavier than we thought and increase the force we apply (Hamrick al. 2011).
then decide how hard to push. If it does
Motivated by this, we propose HyperDDnamicss, a dynamics meta-learning framework for that gen- erates parameters for dynamies models (experts) dedicated the situation hand, based on obser- vations of how the environment behaves. HyperDynamics has three main modules: i) an encoding module that encodes a few agent-enviromment interactions and the agent's visual observations into a latent feature code, which captures the properties of the dynamical system, ii) hypernttork (Ha al, (016) that conditions on the latent feature code and generates parameters of a dynamics model dedicated to the observed system, and iii) target dynamics model constructed using the generated parameters that takes as input the current fow-dimensional system state and the agent action, and predicts the next system state, shown in Figure [ model as an expert, as it specializes on encoding the dynamics of a particular scene at a certain point in time. HyperDynamics conditions els on the fly. It can be trained in an end-to-end differentiable manner to minimize state prediction error of the generated experts in each task.
We will be referring to this target dynamics
real-time observations and generates dedicated expert mod-

---

Published as a conference paper at ICLR 2021
Visual Encoder E,a
Dynamics Model F
Visual Input
A6,
7
Interactions
Wights Gammndaonn
Interaction Encoder F.g
Hypernetwork H
Figure 1: HyperDynamies encodes the visual observations and set of agent-envirroment interactions and generates the parameters a dynamics model dedicated to current environment and timestep using a hy- pernetwork. HyperrDnamicss for pushing follows the general formulation, with a visual encoder for detecting the object in 3D and encoding its shape, and an interaction encoder for encoding a small history of interactions of the agent with the environment.
Many contemporary dynamics learning approaches assume a fixed system without considering po- tential change in the underlying dynamics, and train fixed dynamics model (Watter et al.) 2015 Banijamali et al., 2018; Fragkiadaki et al., 2015) Zhang al. (019).. Such expert models tend to fail when the system behavior changes. A natural solution to address this is to train separate expert for each dynamical system. However, this can hardly scale and doesn't 't transfer to systems with novel properties. Inspired by this need, HyperDynamics aims to infer system's 's properties by sim- ply observing how behaves, and automatically generate an expert model that is dedicated to the observed system. In order address system variations and obtain generalizable dynamics models, many other prior works propose to condition on visual inputs and encode history of interactions to capture the physical properties of the system (Finn & Levine 2016 Ebert al. 2018 Xu et al.) 2019b Pathak et al. 2017 Xu et al. 2019a Li et al. 2018 Sanchez- et al. [20 Hafner al. 2019). These methods attempt to infer system properties; yet, they optimize single and fixed global dynamics model, which takes input both static system properties and fast-changing states, with the hope that such model can handle variations of systems and generalize. We argue that representation which encodes system information varies across different system variations, and each instance of this presentation ideally should correspond to a different dynamics model that needs to be used in the corresponding setting. These models different, but also share a lot of information as similar systems will have similar dynamics functions. HyperDynamics makes explicit assump- tions about the relationships between these systems, and attempts to exploit this regularity and learn such commonalities across different system variations. There also exsists a family of approaches that attempt online model adaptation via meta-learning (Finn al. 2017 Nagabandi et al.) [019]] Clavera et al. 2018: 8 Nagabandi et al., (0018)). These methods perform online adaptation on the pa- rameters of the dynamics models through gradient descent. In this work, empirically show that our approach adapts better than such methods to unseen system variations.
We evaluate HyperDynamics on single- and multi-step state predictions, well as downstream model-based control tasks. Specifically, we apply it in a series of object pushing and locomotion tasks. Our experiments show that HyperDynamics able to generate performant dynamics models that match the performance of separately and directly trained experts, while also enabling effective generalization to systems with novel properties in a few-shot manner. We attribute its good per- formance to the multiplicative way of combining explicit factorization of encoded system features with the low-dimensional state representation of the system. In summary, our contributions are as follows:
We propose HyperDynamics, a general dynamics learning framework that is able to gener- ate expert dynamics models on the fly, conditioned on system properties.
We apply our method to the contexts of object pushing and locomotion, and demonstrate that matches performance of separately trained gystem-specifee experts.
We show that method generalizes well to systems with novel properties, outperforming contemporary methods that either optimize a single global model, or attempt online model adaptation via meta-learning
2

---

Published as a conference paper at ICLR 2021
2
RELATED WORK
2.1 MODEL LEARNING AND MODEL ADAPTATION
To place HyperDynamics in the context of existing literature on model learning and adaptation, we distinguish four groups of methods, depending on whether they explicitly distinguish between dynamic- suchiicsschhaa as joint configurations of an agent poses of objects being manipulated and static -scch as mass, inertia, 3D object shape- properties of the system, and whether they update the dynamics model parameters or the static property values of the system.
(i) No adaptation. These methods concern dynamics model over either low-dimensional states or high-dimensional visual inputs of specific system, without considering potential changes in the underlying dynamics. These models tend to fail when the system behavior changes. Such expert dynamics models are popular formulations in the literature (Watter et al. 2015 Banijamali et al. 2018 Fragkiadaki et al. 22013] Zhang et al. 2019). They can be adapted through gradient descent at any test system, yet that would require large number of interactions.
(ii) Visual dynamics and recurrent state representations. These methods operate over high- dimensional visual observations of systems with variety of properties (Finn & Levine [016 Ebert
2018 Li et al.] [2118] Xu et al., 2019b] Mathieu et al.] 2015] Oh et al.) 2015 Pathak et al.] 2017 hoping the visual inputs could capture both the state and properties of the system. Some also
ct al.
attempt to encode a history of interactions with a recurrent hidden state (Xu et al. 2019a Li et al.)
2018
al. 2018b Hafner et al. 2019). in order to implicitly capture information regarding the physical properties of the system. These methods use a single and fixed global dynamics model that takes system properties as input directly, together with its state
[beer al. 2018, Manchez-Gonzalee
and action.
(iii) System identification (Bhat al. 2002). These methods model explicitly the properties of the dynamical system using small set of semantically meaningful physics parameters, such as mass, inertia, damping, etc. (Ajay et al.) 2019; Mrowca et al. 2018), and adapt by learning their values
from data on-the-fly (Chen et al. 2018 Li et al. 2019 Sanchez-Gonzalez et al. 2018b Battaglia
ct al. 2016, Sanchez-Gonzalez Gonzalez et al. 2018a).
(iv) Online model adaptation via meta- leaanng (Finn et al., 2017; Nagabandi et al. 2019 Clavera (ct al. 2018 Nagabandi et al. (011a)). These methods perform online adaptation on the parameters of the dynamics models through gradient descent. They are optimized to produce good parameter initialization that can be adapted to the local context with only a few gradient steps at test time.
The expert models in (i) work well for their corresponding systems but fail handle system varia- tions. Methods in (ii) and (ii), though attempting to infer system properties, optimize a single and fixed global dynamics model, which takes input both statie system properties and fast-changing states. Methods in (iv) attempt online update for a learned dynamics model. HyperDynamics dif- fers from these methods in that generates condii-cond dynamics models on the fly. In our experimental Section, we compare with all these methods and empirically show that our method adapts better to unseen systems.
2.2 BACKGROUND ON HYPERNETWORKS
Central to HyperDynamics is the notion of using one network (the hypernetwork) to generate the weights of another network (the target network), conditioned on some forms of embeddings. The weights of the target network are not model parameters; during training, their gradients are back- propagated to the weights of the hypernetwork (Chang et al. 2019). This idea initially proposed in (Ha et al. 2016),, where the authors demonstrate leveraging the structural features of the original network to generate network layers, and achieve model compression while maintaining competitive performance for both image recognition and language modelling tasks. Since then, it has been ap- plied to various domains. In Ratzlaff & Fuxin| (2019) the authors use hypernetworss to generate an ensemble of networks for the same task to improve robustness to adversarial data. Platanios et al. (2018) attempts to generate language encoders conditioned on the language context for better weight sharing across languages in machine translation setting. Other applications include but not limited to: weight pruning (Liu et al., 2019), multi-task learning (Klocek et al. 2019 Serrà et al. 2019 Meyerson & Miikkulainen 2019). neural architecture search (Zhang et al.
2018 Brock al.
3

---

Published as a conference paper at ICLR 2021
2017). and continual learning (von Oswald et al. 2019). To the best of our knowledge, this is the first work that applies the idea of generating model parameters to the domain of model-based RL, and proposes to generate expert models conditioned system properties.
HYPERDYNAMICS
HyperDynamics generates expert dynamics models on the fly, conditioned observed system fea- tures. We study applying HyperDynamics to a series of object pushing and locomotion tasks, but in principle it is agnostic to the specific context of dynamics learning task. In this section, first present the general form of HyperIYyamicss and then describe the task-specific modules we used.
Given dynamical system (which is typically composed of an agent and its external environment) with a certain set of properties, HyperDynamics observes both how it behaves under a few actions (interactions), well as its visual appearance when needed. Then, visual encoder Eiiis encodes the visual observation I into a latent visual feature code 2 and an interaction encoder Eint encodes k-step interaction trajectory T (so. Go ...Sk, ax) into latent physies feature code z We then combine these two features codes by concatenating them, =" = [Zant; Zuish, forming a single vector that captures the underlying dynamics properties of the system. hypernetwork H maps 2" set of neural network weights for forward dynamics model F", which is a mapping from the state of the system and the action of the agent, to the state change at the subsequent time step I + 1: 6s, = FF (st,, dz). The hypernetwork is fully-connected network and generates all parameters in single-shot as the output of it's final layer. The detailed architecture of HyperDynamics for an object pushing example is shown in Figure
We meta-train HyperDynamics by backpropagating error gradients of the generated dynamics model's prediction, all the way our visual and interaction encoders and the hypernetwork. Let O denote the pool of all systems used for training. Let T denote the length of the collected trajecto- ries. Let N denote the number of trajectories collected per system. The final prediction loss for our model is defined as:
N ΣΣΣ ON
O
1
Eyres
1ōst F(si, az; ([Fois(TE (I; Eine(T; (T)+))
=-i=l
where Φ, vi and denote the learned parameters of Evis. Enn and H. respectively. We consider
set of locomotion tasks for which only the interaction history is used as input, in accordance with
prior work (e.g.. Finn et al. 2017 Nagabandi et al. 2019 Clavera al., 2018 Nagabandi et al. 2018a), and a pushing task on diverse set of objects, where the visual encoder encodes the 3D shape of the object.
3.1 HYPERDYNAMICS FOR OBJECT PUSHING
For our object pushing setting, consider robotic agent equipped with RGB-D camera to ob- serve the world and an end-effector to interact with the objects. The agent's goal is to push objects to
desired locations on table surface. At time t, the state of our pushing system is st =< obj st ob)
where the object state st is a 13-vector containing the object's position p p orientation q (represented as quaternion). as well as its translational and rotational velocities v and the robot state 87 is a 3-vector representing the position of its end-effector. For controlling the robot, we use position-contro in our experiments, so the action movement of the robot's end-effector.
(ōz, 8y, 52) is a 3-vector representing the
Objects behave differently under pushing due to varying properties such shape, mass, and friction relative to the table surface. For this task, HyperDDnamicss tries to infer object properties from its observations and generate dynamics model specific the object being manipulated. While the 3D object shape can be inferred by visual observations, physics propertics such as mass and friction hard to discern visually. Instead, they can be inferred by observing the motion of object result to a few interactions with the robot (Xu et al.) 2019b). To produce the code z", we use both Enio for encoding shape information and Einr for encoding physics properties.
Given an object to be pushed, we assume its properties stay unchanged during the course of manipu- lation. To produce the interaction trajectory т, the robot pushes the object by l actions sampled from
4

---

Published as a conference paper at ICLR 2021
random directions. Eist then maps T to latent code 2int cient to yield informative zint for pushing, and using a bigger value does not help further improve performance. We use simple fully-comnected network for Eint-
RR In practice we found &: = 5 suffi-
Given a single RGB-D image I of the scene, Evis produces object-centric viis E R to encode the shape information of the objects in the scene. We build our Evis using the recently proposed Geometry-Aware Recurrent Networks (GRNNs) (Tung et al. 2019), which learn to map single 2.5D images to complete 3D feature grids M E Ruxhxaxe that represent the appearance and the shape of the scene and its objects. GRNNS learn such mappings using differentiable unprojection (2.5D to 3D) and projection (3D to 2.5D) neural modules and can be optimized end-to-end for both 3D object detection and view prediction, and complete missing or occluded shape information while optimized for such objectives. helps us implicitly handle possible minor occlusions caused by the robot end- effector during data collection. GRNNs detect objects using 3D object detector that operates over its latent 3D feature map M of the scene. The object detector maps the scene feature map M to a variable number of axis-aligned 3D bounding boxes of the objects (and their segmentation masks if needed). Its architecture is similar to Mask R-CNN (He et al. (017)) but it uses RGB-D inputs and produces 3D bounding boxes. We refer readers to Tung et al. (0199] for more details.
Evss first detects all the objects in the scene, and then crops the scene map around the object we want to push to produce an object-centric 3D feature map M crop(M), which is then used for shape encoding. We opt for such 3D representations since it allows more realistic pushing setting: we do not require the camera viewpoint to be fixed throughout training and testing, and we can handle robot-object occlusions better since encoder learns to complete missing (occluded) shape information in the map M We train Evis for object detection jointly with the prediction loss in Equation D We also add additional regularization losses by reconstructing the object's 's shape, where the visual code Zeis is passed into decoder Deis to reconstruct the object shape Volj, Note that the reconstruction loss can be applied from pure visual data, by observing objects from multiple viewpoints, without necessarily interacting with them. This important because moving camera around objects is cheaper than interacting with them through the robot's end-effectors. In addition, since our tasks concern planar pushing, we found reconstructing a top-down 2D shape instead of the the full 3D one suffices for regularization, essentially predicting a top-down view of the object from any perspective viewpoint.
Object 3D orientation: context or state? The shape of rigid objects does not change over time; rather, its orientation changes. The framework discussed so far intuitively divides the input informa- tion among different modules: the hypemnttwrk generates a dynamics model conditioned on object properties, and the dynamics model takes care of low-dimensional object states. Then, one natural choice is that the object state includes both its position and orientation information, and zvis encodes the object shape information oriented at its canonical orientation, independent of its actual orienta- tion. However, when there's 's only few objects used for training, the hypemeework will only strive to optimize for a discrete set of dynamics models, as it only sees a discrete set of object-specific Zwis. thus potentially leading severe overfiting on the training data. Instead of producing zvis encode object shape information at its canonical pose, we propose to encode the object shape oriented at its current orientation each time step, and discard the orientation information from the states fed into the generated dynamics model, as their generated parameters would already encode the orien- tation information. This way, we move the orientation information from the state input (fed into the generated dynamics model) to the shape input (encoded into 2,ia and fed into the hypernetwork H). Using oriented as opposed to canonical shape as input permits to have way more training examples, even from small number of objects with distinct shapes. Intuitively, H now treats the same object with different orientations as distinct objects, and generates a different expert for cach. This helps smoothen and widen the data distribution seen by H.
3.2 HYPERDYNAMICS FOR LOCOMOTION
For locomotion tasks, we consider a legged robot walking in a dynamic environment with changing terrains, where their properties are only locally consistent. At timestep t, the state contains the joint velocities and joint positions of the robot, as well the position of its center of mass. The dimension of the state depends on the actual morphology of the robot. Torque-control is used to send low-level torque commands to the robot.
5

---

Published as a conference paper at ICLR 2021
Unlike the object pushing setting where the properties of the objects can be safely assumed to main constant, here the system dynamics changes rapidly due to varying terrain properties. Hence, adopt an online adaptation setting where timestep the interaction data is taken to be the most recent trajectory 7 = (st-k. 0t-kk at-1)- Similarly, we use a simple fully-connected network to encode the interaction trajectory. In practice, we found k 16 (equal to trajectory seg- ment of around 0.3 seconds) to be informative enough for Eine to infer accurate terrain properties, and using bigger value does not provide further performance gain. The encoded Zint IR is then used by H to generate dynamics model for the local segment of the terrain in real-time.
3.3 MODEL UNROLLING AND ACTION PLANNING
Action-conditioned dynamies models can be unrolled forward in time for long-term planning and control tasks. We apply our approach with model predictive control (MPC) for action planning and control. Specifically, we use random shooting (RS) (Nagabandi et al. 2018b) for action planning, unroll our model forward with randomly sampled action sequences, pick the best performing one, and replan at every timestep using updated state information to avoid compounding errors.
The unrolling mechanism of HyperDynamics for pushing is straightforward: at each timestep t, generated dynamics model F2 predicts 5x for the system. Afterwards, we compute &++1 = B6 where L represents simple summation for positions and velocities, and quaternion composition for orientations. We then re-orient the object-centrie feature map M using the updated object orientation obj and generate a new dynamics model Fi+1- Finally, the updated full state st+1. excluding 4:+1, is used by F[41 for motion prediction. For locomotion, each timestep we obtain one Zints generate a dynamics model F3 for the local context, and use it for all unrolling steps into the future, since future terrain properties are not predictable.
EXPERIMENTS
Our experiments aim to answer these questions: (1) Is HyperDynamics able to generate dynamics models across environment variations that perform as well as expert dynamics models, which are trained specifically each environment variation? (2) Does HyperDDyamics generalize to systems with novel properties? (3) How does HyperDynamics compare with methods that either use fixed global dynamics models or adapt their parameters during the course of interaction through meta- optimization? We test our proposed framework in the tasks of both object pushing and locomotion, and describe each of them in details below.
4.1 OBJECT PUSHING
Many prior works that learn object dynamics consider only quasi-static pushing poking, where an object always starts to move or stops together with the robot's end-effector (Finn & Levine 2016 Agrawal et al. 2016 Li et al. 2018 Pinto & Guptal 2017). We go beyond simple quasi-static pushing by varying the physics parameters of the object and the scene, and allow an object to slide by itself if pushed with a high speed. A PID controller controls the robot's torque in its joint space under the hood, based on position-control commands, so varying the magnitude of the action could lead to different pushing speed of the end-effector. We test HyperDynamics on its motion prediction accuracy for single- and multi-step object pushing, as well as its performance when used for pushing objects desired locations with MPC.
Experimental Setup. We train our model with data generated using the PyBullet simulator (Coumans & Bail 2016-2019). Our setup uses a Kuka robotic arm equipped with a single rod- shaped end-effector, as shown in Figure [ Our dataset consists of only 31 different object meshes with distinct shapes, including objects from the MIT Push dataset YYu et al. 2016) and 20 objects selected from four categories (camera, mug, bowl and bed) in the ShapeNet Dataset (Chang et al.) 2015). We split our dataset so that 24 objects are used for training and 7 are used for testing. The objects can move freely on a planar table surface of size 0.6m x 0.6m. For data collection, we randomly sample one object with randomized mass and friction coefficient, and place it on the table with random starting position. The mass is uniformly sampled from [300, 1000], the friction coef- ficient of the object material is uniformly sampled from [Se 12e 4) and the friction coefficient of the table surface is set to be 10, all using the default units in PyBuletr. Then, instantiate the

---

Published as a conference paper at ICLR 2021
Table 2: Pushing success rate.
Table I: Motion prediction error (in centimeters).
Model
Seen Objects
Nevel Objects
Model
Seen Objects
Novel Objects
wid abs
wi abs
obs
wio
obs
Ens
0.32
4.22=2211
1.66:4077
5.83.2 49
Expert- Ens
0.92
0.72
0.80
0.44
1.45=+0662
5.89=2.46
1.72:40.66
8.48-+2.82
XYZ
0.80
0.42
0.74
0.44
Direct
1.01:10.41
5.14::2.00
1.24:10.54
5.60-12.59
Direct
0.88
0.62
0.84
0.58
MB-MAML
1.68:10.61
8.9123.79
1.76.10.73
9.05.15.86
MB MAML
0.70
0.42
0.68
0.38
HyporDynamics
0.83=:0.35
4.27=2.24
0.93:+0.53
4.77:22.57
VF
0.62
0.52
DensePlysNet
0.70
0.58
HyperDynamics
0.92
end-effector nearby the object and collect pushing sequences with length of 5 timesteps, where each action a random horizontal displacement of the robot's end-effector ranging from 3am to 6cm, and each timestep is defined to be 800ms. Our method relies on a single 2.51) image as input for object detection and motion prediction. Note that although collecting actual interactions with ob- jects is expensive and time-consuming, data consisting of only varying shapes are cheap and easily accessible. To ensure žuis produced by the visual encoder Evis captures sufficient shape information for pushing, we train it on the whole ShapeNet (Chang et al., (015) for shape autoencoding as an auxiliary task, jointly with the dynamics prediction task. See Appendix |A.1 for more details.
Baselines. We compare our model against these baselines: (1) XYZ: a model that only take as input the action and states of the object and the agent, without having access to its visual feature or physics properties, similar to Andrychowicz et al. (2020) and [Wu et al. (2017). (2) Direct: this baseline uses the same visual encoder and interaction encoder our model does, and also uses the same input, while it passes the encoded latent code & directly into a dynamics model in addition to the system state. Compared to ours, the only difference is that it directly optimizes single global dynamics model instead of generating experts. This approach resembles the most common way of conditioning on object features used in the current literature (Finn & Levine 2016 Agrawal iet al. al. 2019) where global dynamics model takes as input both the state and properties of the
2016 Li et al. 20188] Xu et al., [019a]] Fragkiadaki et al. 2015 Zhang et al.. 2019: Hafner
dynamical system. (3) Visual Foresight (VF) (Ebert et al., 2018) and
(Xu et al.
2019ap: these two models rely on 2D visual inputs of the current frame, and predict 2D optical flows that transform the current frame to the future frame. These models also use global dynamics model for varying system properties, and feed system features directly into the model, following the same philosophy of Direct. These two baselines are implemented using the architectures described in their corresponding papers. (4) MB-MAML (Nagabandi et al. 2019): a meta-trained model which applics model-agnostic meta-learning to model-based RL, and trains dynamics model that can be rapidly adapted to the local context when given newly observed data. This baseline also uses the same input as model does; the only difference is that it uses the interaction data for online model update. (5) Experi-Ens: we train a separate expert model for each distinct object, forming an ensemble of experts. This baseline assumes access to the ground-truth orientation, mass, and friction, and serves as an oracle when tested on seen objects. For unseen objects, we select a corresponding expert from the ensemble using shape-based nearest-neighhour retrieval, essentially performing system identification.
Implementation Details. In order to ensure a fair comparison, all the dynamics models used in Direct, MB-MAMI. and Expert-Ens use the exactly same architecture as the one generated by Hy- perDynammccs The interaction encoder is a fully- cally-connected network with one hidden layer of size 8. The hypemetwork is a fully-connected network with one hidden layer of size 16. The visual encoder follows the architecture of (Tung et al. 2019) and its encoder uses 3 convolutional layers, each followed by a max pooling layer, with fully-connected layer the end. It uses kernels of size 5, 3 and 3, with 2, 4, and 8 channels respectively. Its decoder uses 3 transposed convolutional layers, with 16, and 1 channels and the same kernel size of 4. The generated dynamics model is a fully-connected network with 3 hidden layers, each of size 32; the same applies to the dynamics models used XYZ, Direct and Expert-Ens. We use leaky ReLUU as activations for all the modules. We use batch size of 8 and a learning rate of 1e 3. We collected 50, 000 pushing trajectories for training, and 1, 000 trajectories for testing. All models are trained till convergence for 500K steps.
We evaluate all the methods on prediction error for both single-step (t = 1) and multi-step unrolling (t = 5) (Table []. We also test their performance for downstream manipulation tasks, where the robot needs to push the objects to desired target locations with MPC, in scenes both with and without obstacles. When obstacles are present, the agent needs to plan a collision-fre path. We report their success rates in 50 trials in Table 2 Red numbers in bold denote the best performance across all
7

---

Published as a conference paper at ICLR 2021
models, while black numbers in bold represent the oracle performance (provided by Expert-Ens) VF and DensePhysNer are not tested for motion prediction since they work in 2D image space and do not predict explicit object motions, and are not applicable for collision checking and collision-fre free pushing for the same reason.
Our model outperforms all baselines by margin on both prediction and control tasks. When tested on objects seen during training, while our model needs to infer orientation information and physics properties via encoding visual input and interactions, it performs on par with the Expert-Ens oracle, which assumes access to ground-truth orientation, mass and friction coefficient. This shows that the dynamics model generated by HyperDynamics on the fly is as performant as the separately and di- rectly trained dynamics experts. The XYZ baseline depicts the best performance possible when shape and physics properties of the objects are unknown. When tested with novel objects unseen during training, our model shows a performance similar to seen objects, outperforming XYZ. by great mar- gin, demonstrating that our model transfers acquired knowledge to novel objects well. Expert-Ent on novel objects performs similarly to XYZ, suggesting that with only handful of distinct objects in the training set, applying these dedicated models to novel objects, though after nearest- neighbuu searches, yields poor generalization. Performance gain of model over Direct suggests our hi- erarchical way of conditioning on system features and generating experts outperforms the common framework in the literature, which uses global model with fixed parameters for varying system dynamics. Our model also shows a clear performance gain over MB-MAMI, when only 5 data sam- ples of interaction are available for online adaptation. This indicates that our framework, which directly generates dynamics model given the system properties, is more sample-efficient than the meta-trained dynamics model prior that needs adapt to the context via extra tuning.
4.2 LOCOMOTION
Experimental Setup. We set up environment using the Mu- JoCo simulator (Todorvv et al. 2012). In particular, we consider two types of robots: a planar half-chectah and quadrupedal "ant" robot. For each robot, we consider two environments where the terrains changing: (1) Slope: the robot needs to walk over a terrain consisting of multiple slopes with changing steepness. (2) Pier: the robot needs walk over a series of blocks floating on the water surface, where damping and friction properties vary between the blocks. As result, we consider these tasks in total: Cheetah- Slope, Ant-Slope, Cheetah-Pier and Ant-Pier. Figure 2 shows two of them. Unlike the setting of object pushing, randomly and uni- formly sampled states in the robot's state space do not match the actual state distribution encountered when the robot moves on the terrain. As a result, on-policy data collection is performed: the lat- est model is used for action selection to collect best-performing trajectories, which are then used to update the dataset for model training. For Slope, the environment is comprised of a series of con- secutive terrain segments, each of which has a randomly sampled steepness. During test time, we also evaluate unseen terrains with either higher or lower steepness. For Pier, each of the blocks floating on the water has randomly sampled friction and damping properties, and moves vertically when the robot runs over resulting in rapidly changing terrain. Similarly, during test time we also evaluate using blocks with properties outside of the training distribution. See Appendix A.2 for more details.
Figure 2: We consider types of rohots and two environments for locomotion. Top: Ant-Pier. Down: Cheetah-Slope.
Baselines. We compare our model against these baselines: (1) Recurrent Sanchez-Gonzaled et al., 2018b): recurrent model that also uses GRUs (Cho et al., | 2014) to encode a history of interactions to infer system dynamics and perform system identification implicitly. (2) MB- MB-MAMI. (Nagabandi et al. 2019): a model meta-trained using MAMI., same as the one used for object pushing. uses the same 16-step interaction trajectory for online adaptation. (3) Expert-Ens: An ensemble of separately trained experts, similar to the one used for object pushing. The only difference is that in pushing, train one expert for each distinct object, while now have continuous distribution of terrain properties used for training. Therefore, for each task we pick several environments with properties evenly sampled from the continuous range, and train an expert for each of them.
Implementation Details. Again, in order to have a fair comparison, all the forward dynamics models used in Recurrent and MB-MAML. use the same architecture the one generated by Hy-
8

---

Published as a conference paper at ICLR 2021
Table 3: Comparison of average total return for locomotion tasks.
Cheetah- Slape
Chestah-Pre Pier
Ant- Ant-Slape
Ant Pier
Model
Soon
Novel
Scen
Novel
Seon
Navel
Experi-Ene
59.2
76.9:142.4
354.61182.99
321,11177.33
142.1
1242:71.9
213.0
179.311722
Recurrntt
64.4221.0 00
76.64338.
282.52
278.711324
133.9::88.4
142.1=66.1
176.4=87.7
170.5288..1
MB MAML
55.3+24.5
62.7+37.7 7
291.3# 3+
297.1+151.9
104.5+445.
110.4 =53.2
180.2=91.2
171.0+87.11
Hyperbynamics
607.9=63.1
124.5:24.7
349.4
337.211784
138.8::6..3
140.1=72.00
216.61:2122
108.42103.8
perDynamics. Both Recurrent and MB-MAML use the same input as our model does. Our encoder is a fully-connected network with 2 hidden layers, both of size 128, followed by ReLU activations and a final layer which outputs the encoded 2oxt- The hypernetwork is a fully- culle-connected network with one hidden layer of size 16. The generated dynamics model is fully-connected network with two hidden layers, each of size 128; the same applies to the dynamics models used in Recurrent and MB-MAML. During planning and control, 500 random action sequences are sampled with a planning horizon of 20 steps. Data collection is performed an on-policy manner. We start the first iteration of data collection with randomly sampled actions and collect 10 rollouts, each with 500 steps. We iterate over the data for 100 epochs before proceeding to next iteration of data collection. At each iteration, the size of training data is capped at 50000 steps, randomly sampled from all available trajectories collected. Here we use batch size of 128 and a learning rate of le 3. All models are trained for 150 iterations till convergence. In addition, early stopping is applied if an rolling average of total return decreases.
We apply all the methods with MPC for action selection and control, and report in Table the average return computed over 500 episodes. Again, red numbers denote the best performance and the black ones represent the oracle performance. In all tasks, HyperDynamiess is able infer accurate system properties and generate corresponding dynamics models that match the oracle Expert- Ens on seen terrains, and shows a great advantage over it when tested on unseen terrains. Recurrent also performs reasonably well on Anl-Slope, but our model outperforms both MB-MAML. and Recurrent on most of the tasks. Note that here Recurrent can be viewed as serving the same role of the Direct baseline in pushing, since it feeds system features together with the states to fixed global dynamics model. The results suggest that our explicit factorization of system features and the multiplicative way of aggregating it with -iw-eemeniional states provide clear advantage over methods that either train a fixed global model or perform online parameter adaptation on a meta- mneta-trained dynamics prior.
5 CONCLUSION
We presented HyperDDnamicc,, a dynamics meta-learning framework that conditions on system fea- tures inferred from observations and interactions with the environment to generate parameters for dynamics models dedicated to the observed system on the fly. We evaluate our framework in the context of object pushing and locomotion. Our experimental evaluations show that dynamics mod- els generated by HyperDynamics perform on par with an ensemble of directly trained experts in the training environments, while other baselines fail to do so. In addition, to transfer knowledge acquired from seen systems to novel systems with unseen properties, even when only 24 distinct objects are used for training in the object pushing task. Our method presents explicit factorizations of system properties and state representations, and provides a multiplicative way for them to interact, resulting a performance gain over both methods that employ global yet fixed dynamics models, and methods using gradient-ber based meta-optimization. The proposed framework is agnostic to the specific form of dynamics learning tasks, and we hope this work could stimulate future research into weight generation for adaptation of dynamics models in more versa- tile tasks and environments. Handling more varied visual tasks, predicting both the architecture and the parameters of the target dynamics model, and applying our method in real-world scenarios are interesting avenues for future work. For pushing, we believe our method has the potential to transfer to real-world without heavy fine-tuning, since it uses geometry- aware representation, suggested in Tung et al. (2020). For locomotion, our model should also be casily trainable with data collected in real-world, following the pipeline described in Nagabandi et al. (019).
framework is able

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
This paper is based upon work supported by Sony AI, ARL 1911NF1820218, and DARPA Com- mon Sense program. Fish Tung is supported by Yahoo InMind Fellowship and Siemens Future- Maker Fellowship.
REFERENCES
Pulkit Agrawal, Ashvin V Nair. Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. In Advances in neural information processing systems, pp. 5074-5082, 2016.
Anurag Ajay, Maria Bauza, Jiajun Wu, Nima Fazeli, Joshua Tenenbaum, Alberto Rodriguez, and
Leslie Kaelbling. Combining physical simulators and object-based networks for control, 04 2019.
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3 20, 2020.
Ershad Banijamali, Rui Shu, Hung Bui, Ali Ghodsi, et al. Robust locally-linear controllable em- bedding. In International Conference on Artificial Intelligence and Statistics, pp. 1751-1759, 2018.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. In Advances in neural information processing systems, pp. 4502-4510, 2016.
Kiran S. Bhat, Steven M. Seitz, and Jovan Popovic. Computing the physical parameters of rigid- body motion from video. In Anders Heyden, Gunnar Sparr, Mads Nielsen, and Peter Johansen (eds.), ECCV, volume 2350 of Lecture Notes in Computer Science, pp. 551-565. Springer, 2002. ISBN 33745--37
URL
de wttpc//oo/ccc//////////o//o
eccv2002-1 html #BhatsP02
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Smash: one-shot model archi- tecture search through hypernetworks. arXiv preprint arXiv:1708.05344 2017.
Angel X. Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi. and Fisher Yu. ShapeNet: An Information-Rich 3D Model Repository. Technical Report #1Xiv:1512.0301 [cs.GR], Stanford University - Princeton University - Toyota Technological Institute at Chicago, 2015.
Oscar Chang, Lampros Flokas, and Hod Lipson. Principled weight initialization for hypernetworks. In International Conference on Learning Representations, 2019.
Tao Chen, Adithyavairavan Murali, and Abhinav Gupta. Hardware conditioned policies for multi- robot transfer learning. In Proceedings of the 32nd International Conference on Neural Informa- tion Processing Systems, pp. 9355-9366. Curran Associates Inc., 2018.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.122 1259, 2014.
Ignasi Clavera, Anusha Nagabandi, Ronald Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt: Meta-learning for model-based control. 03 2018.
Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, 2016-2019.
robotics and machine learning.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual fore- sight: Model-based deep reinforcement learning for Assion-based robotic control. arXiv preprint 2018.
10

---

Published as a conference paper at ICLR 2021
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. CoRR, abs/1610.00096, 2016. URL
wttp://rrrrgabb//...... 00696
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-keaming for fast adaptation of deep networks. CoRR, abs/ 2017. URL http:://arxivv.or//aa//////
03400
Katerina Fragkiadaki, Pulkit Agrawal, Sergey Levine, and Jitendra Malik. Learning visual predictive models of physics for playing billiards. CoRR, 11111414,07444 2015.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint 1eXiv:1009.0900 2016.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pp. 2555-2565, 2019.
Jessica Hamrick, Peter Battaglia, and Joshua B Tenenbaum. Internal physics models guide proba- bilistic judgments about object dynamics. In Proceedings of the 33rd annual conference of the cognitive science society. pp. 1545-1550. Cognitive Science Society Austin, TX, 2011.
Masahiko Haruno, Daniel M Wolpert, and Mitsuo Kawato. Multiple paired forward- inwerr--nveers models for human motor learning and control. In M. J. Kearns, S. A. Solla, and D. A. Cohn (eds.), Advances in Neural Information Processing Systems 11, pp. 31-37. MIT Press, 1999.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. CORR, 06870
abs/1703.06870, 2017. URL
Sylwester Klocek, Lukasz Maziarka, Maciej Wolczyk, Jacek Tabor, Jakub Nowak, and Marek Smieja. Hypernewoor functional image representation. In International Conference on Arti- ficial Neural Networks, pp. 496-510. Springer, 2019.
Jue Kun Li, Wee Sun Lee, and David Hsu. Push-net: Deep planar pushing for objects with unknown physical properties. In Robotics: Science and Systems, 2018.
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B. Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In International Confer- ence on Learning Representations, 2019. URL
net forett/foummiiommm
FJgbSn09Ym
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE International Conference Computer Vision, pp. 0006-33965 2019.
Michaël Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. CoRR, abs/1511.05440,, 2015. URL http:://axxvv.or// abs//111
05440
Elliot Meyerson and Risto Miikkulainen. Modular universal reparameterization: Deep multi-task learning across diverse domains. In Advances Neural Information Processing Systems, pp- 7903-7914, 2019.
R. C. Miall and D. M. Wolpert. Forward models for physiological motor control. Neural Netw. 9 URL
1877, November 1996. ISSN 0893-6080. doi: 10. (00116/08899600009))0000 http://dx. doi. www.oogg100... 1 0100010 6 50893-888880 6080 (96) 00035-4]
Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Li Fei-Fei, Joshua B Tenenbaum, and Daniel LK Yamins. Flexible neural representation for physies prediction. In Advances in Neural Information Processing Systems, 2018.
Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via meta-learning: Continual adaptation for model-based 12 2018a.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE Interna- fional Conference on Robotics and Automation (ICRA), pp. 7559-7566. IEEE, 2018b.
11

---

Published as a conference paper at ICLR 2021
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S. Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta- reinforcement learning. In International Conference Learning Representations, 2019.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, and Satinder Singh. Action video prediction using deep networks in atari games. arXiv preprint arXivv1507.0087500 2015.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curriosiyy-driveneexplorri by self-supervised prediction. CoRR, abs/1705.05530 2017. URL http://arxiv. 17755.0355
Lerrel Pinto and Abhinav Gupta. Learning push by grasping: Using multiple tasks for effective learning. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 2161- 2168. IEEE, 2017.
Emmanouil Antonios Platanios, Mrinmaya Sachan, Graham Neubig, and Tom Mitchell. Contex- tual parameter generation for universal neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 425 435,435,25 2018.
Neale Ratzlaff and Li Fuxin. Hypergan: A generative model for diverse, performant neural networks. In International Conference on Machine Learning, pp. 5361-5369, 2019.
Alvaro Sanchez- Nicolas Heess, Jost Tobias Springenberg. Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp- 1470-4479) Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018a. PMLR.
Alvaro Sanchez-Gonzaler Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks learnable physics engines for inference and control. In International Conference on Machine Learning, pp. 447004479, 2018b.
Joan Serrà, Santiago Pascual, and Carlos Segura Perales. Blow: single-scale hyperconditioned flow for non-parallel raw-audio voice conversion. Advances in Neural Information Processing Systems, 32:6793-6803, 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Hsiao- Yu Fish Tung, Ricson Cheng, and Katerina Fragkiadaki. Learning spatial common sense with geometry-awee aware recurrent networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2595-2603, 2019.
Hsiao- Yu Fish Tung, Zhou Xian, Mihir Prabhudesai, Shamit Lal, and Katerina Fragkiadaki. 3d-oes: Viewpoint-invane invariant Abject-factoriag environment simulators. arXiv preprint arXiv:2011.06464 2020.
Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F Grewe. Continual learning with hypernetworks. arXiv preprint (:XXvv99900006955 2019.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances in neural information processing systems, pp. 2746-2754, 2015.
Jajjun Wu, Erika Lu, Pushmeet Kohli, Bill Freeman, and Josh Tenenbaum. Learning to see physics via visual de-animation In Advances in Neural Information Processing Systems, pp. 153 164, 2017.
Zhenjia Xu, Jiajun Wu, Andy Zeng, Joshua B. Tenenbaum, and Shuran Song. Densephysnet: Learning dense physical object representations via multi-step dynamic interactions. CoRR, abs/1906.03853, 2019a. htt.:tttp//axxv.cooo 1906. 03853
12

---

Published as a conference paper at ICLR 2021
Zhenjia Xu, Jiajun Wu, Andy Zeng, Joshua B Tenenbaum, and Shuran Song. Densephysnet: Learn- ing dense physical object representations via multi-step dynamic interactions. arXiv preprint arXiv:1906.03853, 2019b.
Kuan-Ting Yu, Maria Bauzá, Nima Fazeli, and Alberto Rodriguez. More than million ways be pushed: A high-fidelity experimental data set of planar pushing. CoRR, abs/1604.04038, 2016.
URL http:::/www....og/////p/ htagg/abs/166/ 04038
Chris Zhang. Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. arXiv preprint urXiv:/810.05749, 2018.
Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew Johnson, and Sergey Levine. Solar: Deep structured representations for model-based reinforcement learning. In International Conference Machine Learning, pp. 7444-7455 2019.
13

---

Published as a conference paper at ICLR 2021
Figure Top: objects used for training. Bottom: 7 objects used testing. Objects selected from
ShapeNet Dataset (Chang et al. 2015) are realistic objects seen during daily life, and objects from the MIT Push dataset (Yu et al. 2016) consist of basic geometries such as rectangle, triangle. and ellipse. Note that the shape of the testing objects differ from those used for training. (The third mug object used for testing handleless.)
A EXPERIMENTAL DETAILS
A.1 OBJECT PUSHING
Our dataset consists of 31 object meshes with distinct shapes, including objects from the MIT Push dataset (Yu et al. 2016) and 20 objects selected from four categories (camera, mug, bowl and bed) the ShapeNet Dataset (Chang et al.] 2015). 24 objects are used for generating the training data and objects are used for testing (see Figure 3). The size of the objects ranges from 7em to 12cm. For each pushing trajectory, we randomly sample an object with randomized mass and friction coefficient, and place it on a 0.6m x 0.6m table surface at a uniformly sampled random position. At the begining of each trajectory, we instantiate the robot's 's end-effector nearby the object, with random distance from the object's center ranging from 6cm 10cm. When sampling pushing actions, with probability of 0.2 the agent pushes towards a completely random direction, and with a probability of 0.8 it pushes towards a point randomly sampled the object's bounding box. Each action is displacement with magnitude sampled from [3cm, 6cm].
Our method is viewpoint-invan 9 different views uniformly with different azimuth angles ranging from the left side to the right side of the robot, and the same elevation angles of 60 degrees. All cameras are looking at 0.1m above the center of the table, and 1m away from the look-at point. At test time, model is tested with single and randomly selected view. All images are 128 x 128.
and relies on a single 2.5D image as input. We place cameras at
For object pushing with MPC, place randomly selected object in the scene with random initial position, and sample a random target location for it. At each step, we evaluate 30 random action sequences with the model before taking an action. For pushing without obstacle, the maximum distance of the target to the initial position for each object is capped at 0.25m, and a planning horizon of step is used since greedy action selection suffices for it. For pushing with obstacles, randomly place 2 additional obstacles in the scene, selected from the same pool of available objects. We place the first obstacle between the starting and goal positions with a small perturbation sampled from [-2em, 2cm]. so that there exists no straight collision-fres path from the starting position to the goal. The second obstacle placed randomly with distance from the first one ranging in [15cm, 25cm). The distance from the target to the initial position is uniformly sampled from the range [0.24m, 0.40mm]. The model needs to plan collision free path and thus planning horizon of 10 steps is used. For both tasks, successful if the object ends up within 4cm from the target position, without colliding with any obstacles along the way.
50 trials and evaluate the success rate, where it is considered
Additional model details. HyperDynamics for pushing follows the general formulation discussed in Section It's composed of a visual encoder Emis. interaction encoder Fint and hypernet-
14

---

Published as a conference paper at ICLR 2021
work H. These components trained together to minimize the dynamics prediction loss Lpreds as defined in Equation | Meanwhile, Evis is also trained jointly for object detection and shape reconstruction. Our E follows the architecture of GRNNs (Tung et al. 2019), which uses sim- ilar detection architecture as Mask R-CNN (He et al. 2017). except that it takes 2.5D input and produces 3D bounding boxes. The detection loss Leet is identical as the one defined in He et al. 2017). supervised using ground-truth object positions provided by the simulator. Evis then uses the bounding boxes to crop out fixed-size feature map for the object under pushing, and encodes it into visual code Zvis. In order to help Evis produce an informative Evis, we added an auxiliary shape reconstruction loss, where Zuis is passed into a decoder Dess to reconstruct the object shape voll, In practice, since our tasks concern planar pushing, we found reconstructing top-down 2D shape instead of the full 3D one suffices to produce informative Zvis- This essentially performing view prediction for the top-down viewpoint. The network is trained to regress to the ground-truth top-down view of the object, which is provided by the simualtor and does not have any occlusion. The view prediction loss Cup of GRNNs uses standard cross-entropy pixel matching loss, where the pixel intensity is normalized. We refer readers to (Tung et al. 2019) for more details. The final loss for HyperDynamics for pushing reads:
L = Lepred + Liiet + Cop
Note that while it's expensive to collect actual pushing data with objects, data consisting of various shapes are easily accessible. In order to ensure Zuis produced by the shape encoder Eshape captures sufficient shape information, we train it on the whole ShapeNet (Chang et al., 2015) for shape autoencoding (essentially top-down view prediction) to jointly optimize for Lup. In practice, we load random objects in the simulation and collect rendered RGB-D images from randomized viewpoints. During training. data fed into Eais alternates between such rendered data and images of actual objects being pushed, while the other components are only trained with the objects being pushed. Note that such auxiliary data is only used to optimize for Lup. (It's also possible to use such data to optimize for Laer. but in our experiments training object detection with only the objects being pushed suffices.) The reason for doing this is that we can collect such visual data very fast using easily accessible object meshes, while we only need to interact with a few objects with actual pushing- This helps our model generalize to novel objects unseen during actual pushing, since Evis is now able to produce a meaningful Zuis for these novel objects.
A.2 LOCOMOTION
The Slope environment consists of a series of consecutive upward slopes with length of 15m. For training, the height of each slope is randomly sampled from [0.5m, 3.5m), and for testing, the height is randomly sampled from [0m, 0.25m] U [3.75m, 4m]. The Pier environment consists of series of consecutive blocks with length of 4m. For training, the damping coefficient of each block is randomly sampled from [0.2, 0.8). and for testing, the damping coefficient is randomly sampled from [0, 0.1] J [0.9. 1.0]. 5 experts are trained to form the expert ensemble for each task. The half- cheetah robot has 6 controllable joints while the ant has 10. The reward function for half-cheetah is defined to be #+t1-z;
0.01 0.05]lae]] 2. and for ant it is F441 + 0.05, where Ie refers to the x-coordinate of the agent at time l.
0.02
B ADDITIONAL EXPERIMENTAL RESULTS
B.1 MODEL ABLATIONS
In order evaluate the effect of each network component in our model and motivate our design choices, we present an ablation study here, and report the performance of each ablated model in Table and For pushing, our model contains a visual decoder for top-down shape reconstruction (top-down view prediction). uses a cropping step to produce object-centric feature maps, and uses Zvis E 8 and Zint E IR?.. For locomotion, our model uses Zint E R.a We vary the sizes of these latent codes and evaluate how they affect the model performance. In addition, in order to help understand the performance of the 3D detector in our visual encoder, we also report the prediction error using ground-true truth object positions input. The results in the table indicates that for pushing, it's essential to include the visual decoder to regularize the training, otherwise the model overfits badly on the training data. The cropping step is also important and the object-centric feature map
15

---

Published as a conference paper at ICLR 2021
Table 4: Prediction of HyperDynamies for pushing with different model ablations.
Model
Ours
object posilions
wo vieual docader
wio cropping
R
R
E R
Error (seen)
0.83.10.00 0.3.5
0.76..0.32
0.87 0.44
1.45
0.82:10.43
0.84:10.37
0.83:10.38
0.81:10..34
(novel)
0.93:10.53
0.86::0.44
4.32
1.76=0.30
0.94:0.51
0.93:10.52
0.94:10.55
Table 5: Average total return of HyperDynamics for locomotion with different model ablations.
Cheetah-SSape Slape
Cheetah- Pier
Ant-Slope
Anti Pier
Model
Novel
Ours
607.9=63.1
124.5=64.7
149.4+189.0
837.24179.4
138.8465.3
140.1::72.0
216.6:1212
208 4:4 103.8
Fint
Pint
25
14.
122.9
363.71173.8
829.81173.1
134.5:18888
134,91.7.88
214221224.1
206.71197.3
Table 6: Comparison of prediction error (x 1x10 for locomotion tasks.
Model
Cheetsh-Slope
Cheetan-Pler
Ant-Slape
Ant-Pler
Novel
Experi-Ene
3.4
26.11:6.4
5.4
12.0
8.9
25.4:992
1..2=4.7
41.7113.2
5.2+1.7
39.1+18.00
MB-MAML
5.74:2.2
21.0:19.2
6,1
12.4
12.84:5.2
29.1 #:1.2
18.9
42.3 4:19.7
HyperDynamics
3.51:0.6
17.9:1..3
5.6:1..0
11.3::4.7
9.3.13.4
22.8:27.9
14.7=253
33.2:114.2
helps the model to make more accurate predictions. As for the object detector, when compared to the model using ground -rutt object positions, our model using detected object positions only shows a minor performance drop, suggesting that the 3D detector in the visual encoder is able to accurately detect the object locations. The table also motivates our choice for the dimension the latent code Zois (pushing) and Zint (both pushing and locomotion). as further increasing their sizes does not result any clear performance gain. (In Table none of the numbers marked in red as the results are similar across all ablated models.)
B.2 ADDITIONAL RESULTS ON PREDICTION ERROR FOR LOCOMOTION
In the locomotion tasks, the data collected is not i.i.d., since depends on the model that's being optimized online. In order show clear comparison between our method and baselines regard- ing their prediction accuracy, we train HyperDynamics till convergence for each task, and use this converged model with MPC to collected a fixed dataset. Then, we compare the prediction error of all models on this single dataset and report the results in Table 6 The eπror is computed as the mean squared error between the predicted state and the actual state the robot's state space. Again, our model is able to match the performance of Expert-Ens on the seen terrains and still performs reasonably well on unseen terrains, outperforming the baselines consistently.
16


Reviews:
[{"review_id": "jarth-Nxn6", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=== Summary\n\nThis paper proposes a framework, HyperDynamics, that takes in observations of how the environment changes when applying rounds of interactions, and then, generates parameters to help a learning-based dynamics model quickly adapt to new environments.\n\nThe framework consists of three modules:\n- an encoding module that maps the observation of a few agent-environment interactions into a latent feature vector,\n- a hypernetwork that conditions on the latent vector and generates all parameters of a dynamics model dedicated to the observed system, and\n- a target dynamics model constructed using the generated parameters that predicts the future state by taking the current system state and the input action as input.\n\nThe authors evaluate the framework in a series of object pushing and locomotion tasks. They have shown that a single HyerDynamics model allows few-shot adaptation to new environments, outperforming several baselines while maintaining a performance that is on par with a set of models trained separately for each environment.\n\n\n=== Strengths\n\nThis paper targets an important question of building a more generalizable dynamics model that can perform online adaptation to environments with different physical properties and scenarios that are not seen during training.\n\nThe authors have evaluated the method in several object pushing and robot locomotion tasks and shown superior performance over baselines that uses recurrent state representations or gradient-based meta-optimization.\n\nMany practical treatments used in the pipeline can be good references for the community to learn from, e.g., how to encode object information in 3d, specific representation of the object orientation, and the use of Geometry-Aware Recurrent Networks (GRNNs) to learn 3D feature grids, etc.\n\n\n=== Weaknesses\n\nAlthough I like the idea of this paper, I believe the authors should provide more clarification and illustration of the experimental results to solidify the claims in the paper:\n\n(1) What are the objects used in the pushing task? The authors claim that their \"dataset consists of only 31 different object meshes with distinct shapes.\" It is important to include images of the objects to give the readers a better understanding of how diverse the dataset is and how different the geometry of the \"seen\" and \"novel\" objects are. This can help the readers better appreciate the generalization ability of the proposed method.\n\n(2) It would be great if the authors can include some qualitative examples, e.g., video, to show the performance of the method. Purely from the numbers in the tables, it is hard for the readers to imagine how well the proposed approach solves the tasks.\n\n(3) It would make the paper more illustrative if the authors can include some analysis and visualization of the learned representations in the middle of the network. For example:\n- How are the latent embeddings different for different objects?\n- Are there any correlations between the embeddings and the actual physical properties?\n- How do the interactions affect the embedding? Will different interaction sequences result in the same embedding?\n- How do the learned representations from Geometry-Aware Recurrent Networks (GRNNs) look like? The authors claim that it can \"complete missing or occluded shape information.\" Can the authors provide some concrete evidence supporting this claim in the specific scenarios used in this paper? How do different numbers of interactions affect the quality of the representation?\n\n(4) How does E_vis detect the objects in the scene? Are these detections in 2d or 3d? How accurate is the detection algorithm?\n\n(5) The beginning of Section 3.1 describes that an object's orientation is represented as a quaternion. However, at the end of Section 3.1, the authors suggest that they \"discard the orientation information from states fed into the generated dynamics model.\" This seems to me makes the \"state\" an incomplete representation of the environment, where the authors only predict the position of the object, which makes me wonder:\n\nHow does the model encode the geometry of the object? Will the missing of the orientation information introduce any ambiguities or uncertainties? What if the object is re-oriented? It may be better to include comparisons of different state representations.\n\nAlso, in Section 3.3, the authors suggest that they update the orientation using quaternion composition, which seems to be inconsistent with what has been described before. Haven't the model already discarded the orientation information?\n\n\n=== Other comments\n\nThis paper only shows experiments in the simulation. I'm curious, are there any gaps before applying the method to the real world, and what are these gaps? For example, how long does the model take to optimize the action trajectories when performing MPC? Can it support real-time feedback control in real physical scenarios, especially when the environment is dynamic?\n\n\nModel-predictive control relies on the environment's feedback to correct the action sequences, which can achieve a good control performance while tolerating a larger long-term prediction error. In your experiments, how important is the accuracy of the dynamics model? In other words, even if some baselines have a poorer forward prediction performance, will MPC be able to bridge some of the performance gaps?\n\n\nIn table 3, why are there multiple red numbers in the Ant-Slope columns?\n\n\nTypo?\nPage 4, Section 3.1:\n\"E_int then maps z_int to a 1-dimensional code z_int \\in R^2.\"\nThis sentence seems weird. How does E_int map z_int to its own? Why does a \"1-dimensional code\" lies in a 2d space?\n\n\n=== Post rebuttal\n\nThe authors' response and the revisions to the manuscript have greatly improved the quality and clarity of the paper. Most of my major concerns regarding the implementation and evaluation details have been sufficiently addressed; hence, I decide to increase the score from 5 (Marginally below acceptance threshold) to 6 (Marginally above acceptance threshold).", "extra_scores": {}, "extra_text": {}}, {"review_id": "QEUdyjtJajy", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe authors present \"HyperDynamics\", a novel method for system-identification and learning of flexible forward models that can be used in planning tasks. The presented methods is generic and is shown on both locomotion and pushing tasks with different simulated robots. \n\nI enjoyed reading this work a lot and I hope it gets accepted. It's a clever idea and most flaws that I'm about to point out are easily addressable by the authors.\n\n## Strengths & Weaknesses\n\n#### Strengths\n\n1) The method is generic (shown to work across tasks and environments).\n2) The baselines are strong. When I started reading your paper, I thought that DensePhysNet and some form of MAML would be good candidates for this to compare again and it turns out these were indeed included.\n3) Figure 1 + caption as well as the introduction to section 3 do a great job at introducing the architecture in a way that would allow the reader to create a basic implementation.\n4) Code was included. I didn't run it but it's clean and seems functional from what I can tell.\n\n#### Weaknesses\n\n1) You really, really need to be more clear in the main paper on the implementation details. You can't move the amount of training data to the appendix and it's not good practice to only include the network architecture by name in the main paper. And what are all your losses? You make the method looks super simple but then you train on shapenet, some 2D reconstruction, something about cropping, and there's a GRU in there too (full backprop vs truncated backprop?). The appendix shines a _little_ light on this but you need to be way more specific in the main paper. That has to stand on its own.\n2) You don't motivate all the nitty-gritty implementation choices. Why did you add the decoder? What's the performance if you remove it? What about the cropping? What if you don't do an object-centric feature map, but instead a few CNN layers? What are the individual contributions of all these details?\n3) DensePhysNet, Visual Foresight, and many other works in this domain use (simple) real-robot experiments to demonstrate that their method can handle realistic robot noise. Obviously there's a global pandemic happening at the moment, so I won't require you adding this for the rebuttal, but I think in order to really establish this method (maybe before putting it on Arxiv), you'd have to add some real-robot experiments. This can be as simple as a 180USD RealSense and a 500USD robot arm plus a few objects and a playfield. It's become a standard for system-identification-style works and it's justified in my opinion since your method isn't inherently useful in simulation where the user has access to all the information and can arbitrarily reset/reposition the model. And since you don't have ShapeNet data for many real-world objects (which you seem to need for pretraining), could you at least add a sentence or two detailing how this would transfer to real-world problems?\n\n**TL;DR my main requests:** (2) Motivate implementation details (add ablations if you have any) and (1) be more explicit about them in the main part of the paper.\n\n## Impact & Recommendation\n\nDespite that there seem to be a lot of hacks that make this method in the specific settings, I think the general idea behind it is sound. And I think the authors show that it performs better than the sota, at least in simulation. Therefore I'd recommend acceptance given that the authors add the requested information. In its current shape, it's a 6 for me but if my main concerns are addressed, I'm happy to up this to a 7 or if major improvements are made and my questions below are answered, to an 8.\n\n## Questions, Nitpicks, Comments\n\n- Kudos for not making another acronym method\n- There are a lot of typos and orthographic errors, would recommend a spell-checked or getting this proofed. Examples: section 2 \"poinclouds\", section 2 \"properties in hand\" -> \"properties at hand\"\n- Maybe start the introduction with an example, e.g. how children are able to chew on a block of wood to assess its hardness and then build towers with it. \n- **important** Introduction: when you go over (i-iv), that feels a bit too long and lit-reviewy and misplaced in the introduction. I would recommend the following changes: (a) trim this severely, only mention that there are model-based methods that usually do only one environment and there's meta-learning and how your method is more adaptable than either, (b) move this into the literature section, where you have to come back to it anyway, (c) move the Hypernetworks section from the literature into a separate \"Background\" section and develop it a bit further, since it's less \"competing method\" and more \"you should know about this to understand out method\".\n- Also in the introduction, you present (i-iv), and you mention how your method is better/different than (i-iii) but you never address (iv).\n- It's become a standard to summarize the contributions again at the end of the introduction, ideally as bullet points. Please add these.\n- In equation (1), why is the ordering O-T-N for the sums? I feel like ONT would be more natural, no?\n- When reading the method, my main question was if the method would work on \"dense\" trajectories or on before-and-after photos like DensePhysNet. This is only answered a few pages later but I think this belongs in 3-Overview or 3.1. Just to be clear, you're gathering trajectories of length 4s, i.e. 5 frames of 800ms where you do NOT retract the robot arm when pushing, right? (Compared to DensePhysNet, where the arm is never visible because they take photos before and after complete standstill). If that's the case, how do you deal with occlusion from the arm?\n- Do you encourage object-object interactions in any way or do they just occur randomly? Or do you only ever experiment with single objects?\n- The object orientation vs state section isn't super clear? You're subtracting an object's absolute starting position+orientation from it's future trajectory points?\n- In 3.2: Why a GRU, why not LSTM? Why k=16 (and similarly why k=5)... This ties into the main criticism from above. Please motivate your choices.\n- In 4.1: I think this is a typo, but it says you added beds to your experiment table. I think they'd be a bit too large, no? :D\n- 4.1: specify the random mass+friction range, please!\n- 4.1: same with the total amount of training data/frames\n- And since you won't have ShapeNet \n- 4.2: I think it's a half-cheetah, not a cheetah.\n- 4.2: I don't understand why it's unrealistic to assume arbitrary resetting in simulation. That's one of the benefits of running simulations and common practice. \n- 5: What do you mean \"predicting both the structure and parameters of the target dynamics model\"? Parameters is clear (mass, friction, etc.) but what's the structure here?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "79sHHkBwOLB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "== Update ==\n\nThank you for your detailed response. The newly added clarifications and sanity checks have greatly improved the quality of the paper, and I am therefore increasing my score from 4 to 6. I believe the model capacity comparison (Table 6) is especially important for demonstrating the value of the new architecture, and would recommend mentioning that result in the main paper.\n\n== Original Review ==\n\nThe paper proposes a model for predicting the dynamics of a physical system based on hypernetworks:\ngiven some observed interactions and some visual input, the hypernetwork outputs the parameters of a\ndynamics model, which then predicts the evolution of the system's state over time. Experiments are\nconducted on an object pushing and a locomotion task.\n\n\nStrengths:\n 1. The paper addresses an important question, namely, how a dynamics model may adapt to\n    environments that don't fully match its training distribution.\n 2. The proposed use of a hypernetwork is plausible and novel to my knowledge.\n 3. The related work section appears comprehensive, and, to my knowledge,  does not miss any major\n    prior work.\n\nWeaknesses:\n 1. The main claim of the paper is that hyperdynamics network offers better prediction accuracy and\n    generalization than a standard dynamics model. I feel like the evaluation of this question is\n    confounded by the choice of tasks and baselines. On the pushing benchmark, the XYZ, VF, and\n    DensePhysNet operate on different modalities than HyperDynamics (either no state information or\n    no visual information), and are therefore difficult to compare. For the MB-MAML baseline, this\n    is not specified. The Expert-Ens model cannot be expected to generalize, since it is designed to\n    overfit on individual objects. As a result, only the 'Direct' baseline clearly operates in the\n    same experimental regime as HyperDynamics. However, nothing is reported on the model\n    architecture or the training method for that baseline, raising the question if its model\n    capacity was competitive. My impression is that this experimental design blurs the effects of\n    (a) using side-information to infer system properties, and (b) utilizing such information\n    through a hypernetwork as opposed to a standard dynamics predictor. If the goal is to evaluate\n    the new architecture, these should be disentangled.\n 2. On the locomotion benchmark, the Recurrent baseline is similarly unclear.  Sanchez-Gonzalez et\n    al. is cited, but that paper focusses on comparing recurrent models based on graph networks to\n    those based on MLPs, and it is unclear which model was used.\n 3. No results are reported for the prediction accuracy on the locomotion task, which would have\n    helped evaluate the performance of the dynamics models more directly than the task scores.\n 4. Many of these issues could have been avoided by testing on established benchmarks from the\n    literature, for which results are available. If there is a simulator available, generalization\n    ability could still have been tested by varying the physical constants of the dataset.\n 5. The paper contains a decent amount of typos and grammatical errors.\n\n\nOverall, while the paper presents an interesting idea, the experimental evaluation is not convincing\nin its current state: Baseline architectures are not fully specified, many of them did not receive\nthe same input, and no benchmark task with previously reported results has been used. As a result, I\nrecommend rejection at this time.\n\n\nQuestions:\n 1. In eq. 1, should omega be a parameter of H(.) instead of F(.)?\n 2. Section 3.1 introduces the \"1-dimensional code $z_{int} \\in \\mathbb{R}^2$\". So is it one or\n    two-dimensional?\n 3. Overall, the dimensionality of the latent codes and hidden layers seems incredibly small, e.g.,\n    only 1/2 numbers to encode prior interactions, and 8 to encode shape. Is there really no benefit\n    to using higher capacity models?", "extra_scores": {}, "extra_text": {}}, {"review_id": "2eHzqqsA0n", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary:\nThis paper proposes an adaptive dynamics model based on the idea of hypernetworks. It is demonstrated that this approach compares favorably to other ways of adapting dynamics models such as conditioning on a separate feature input and meta learning by gradient-based model updates. The proposed approach is evaluated on Pushing and Locomotion tasks.\n\n\n#### Pros:\n- The proposed approach for conditioning dynamics models on rollouts to model system-specific properties using the hypernetworks idea seems novel and is interesting.\n- Paper is clearly written, the provided figures help understanding\n- Outperforms state-of-the-art adaptive dynamics modeling approaches [Nagabandi et al., 2019], [Sanchez-Gonzalez et al., 2018b]\n- Reasonable baselines are used for comparison, such as fixed model (XYZ), input feature conditioning (Direct), expert ensemble, and state-of-the-art adaptive dynamics models\n\n#### Cons:\n- The paper does not explain training details for the architecture sufficiently well. How are the network components trained, especially the visual recognition part for object pushing? What kind of supervision with ground truth is required to train the components, for instance for object detection and shape representation? Are components pretrained and how? Which losses/data are used for training? \n- Its unclear why moving from a canonical to an oriented shape representation in Sec. 3.1 should improve results. Shouldnt this limit generalization and require more training data?\n- Giving standard deviations in addition to the average values in table 1-3 would complete the numerical results\n- Sec. 1) Why is PlaNet [Hafner et al., 2019] listed as \"no adaptation\", although it contains a recurrent state representation? \n- It appears magical that the approach performs better on novel than on seen objects during training for Cheetah-Slope or Ant-Slope in Table 3. Please discuss.\n\n#### Recommendation: \nThe paper reads well and proposes an interesting novel approach which could deserve acceptance.\nThe paper should address the points raised in paper weaknesses.\n\n#### Questions for rebuttal:\nPlease address points raised above in \"weaknesses\".\n\n#### Typos:\n- p2: \"They are are\"\n- p4: \"E_int then maps z_int to a 1-dimensional code z_int ∈ R2\" - shouldn't this be \"E_int maps interactions to 2-dimensional code z_int ∈ R2\"?\n- p4: \"which is typically comprised of an agent and its external environment\" -> \"which typically comprises / which is typically composed of\"\n- Table 1: Motion rediction error -> Motion prediction error\n- Advice: In Figure 1, the concatenation symbol is slightly misleading, as it could\nbe interpreted as elementwise multiplication. Maybe replace it by $[\\cdot,\\cdot]$.\n\n\n#### Post-rebuttal comments:\nThe authors' comments addressed my concerns on method and experimental details mostly well. I keep with my rating \"6: Marginally above acceptance threshold\".\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes "HyperDynamics" a framework that takes into account the history of an agents recent interactions with the environment to predict physical parameters such as mass and friction. These parameters are fed into a forward dynamics model, represented as a neural network, that is used for control.

Pros:
- addresses an important problem (adapting dynamics models to "new" environments) and provides strong baselines
- well written and authors have improved clarity even further based on reviewers comments

Cons:
- I agree with the reviewer that it is currently unclear how well this will transfer to the real world
- The idea of predicting physical parameters from a history of environment interactions is not not novel in itself (although the proposed framework is, as far as I know). The authors should include related work along the lines of (1) (this is just one paper that comes to mind, others exist)

(1) Preparing for the Unknown: Learning a Universal Policy with Online System Identification
Decision: Accept (Poster)