Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: === Summary

This paper proposes a framework, HyperDynamics, that takes in observations of how the environment changes when applying rounds of interactions, and then, generates parameters to help a learning-based dynamics model quickly adapt to new environments.

The framework consists of three modules:
- an encoding module that maps the observation of a few agent-environment interactions into a latent feature vector,
- a hypernetwork that conditions on the latent vector and generates all parameters of a dynamics model dedicated to the observed system, and
- a target dynamics model constructed using the generated parameters that predicts the future state by taking the current system state and the input action as input.

The authors evaluate the framework in a series of object pushing and locomotion tasks. They have shown that a single HyerDynamics model allows few-shot adaptation to new environments, outperforming several baselines while maintaining a performance that is on par with a set of models trained separately for each environment.


=== Strengths

This paper targets an important question of building a more generalizable dynamics model that can perform online adaptation to environments with different physical properties and scenarios that are not seen during training.

The authors have evaluated the method in several object pushing and robot locomotion tasks and shown superior performance over baselines that uses recurrent state representations or gradient-based meta-optimization.

Many practical treatments used in the pipeline can be good references for the community to learn from, e.g., how to encode object information in 3d, specific representation of the object orientation, and the use of Geometry-Aware Recurrent Networks (GRNNs) to learn 3D feature grids, etc.


=== Weaknesses

Although I like the idea of this paper, I believe the authors should provide more clarification and illustration of the experimental results to solidify the claims in the paper:

(1) What are the objects used in the pushing task? The authors claim that their "dataset consists of only 31 different object meshes with distinct shapes." It is important to include images of the objects to give the readers a better understanding of how diverse the dataset is and how different the geometry of the "seen" and "novel" objects are. This can help the readers better appreciate the generalization ability of the proposed method.

(2) It would be great if the authors can include some qualitative examples, e.g., video, to show the performance of the method. Purely from the numbers in the tables, it is hard for the readers to imagine how well the proposed approach solves the tasks.

(3) It would make the paper more illustrative if the authors can include some analysis and visualization of the learned representations in the middle of the network. For example:
- How are the latent embeddings different for different objects?
- Are there any correlations between the embeddings and the actual physical properties?
- How do the interactions affect the embedding? Will different interaction sequences result in the same embedding?
- How do the learned representations from Geometry-Aware Recurrent Networks (GRNNs) look like? The authors claim that it can "complete missing or occluded shape information." Can the authors provide some concrete evidence supporting this claim in the specific scenarios used in this paper? How do different numbers of interactions affect the quality of the representation?

(4) How does E_vis detect the objects in the scene? Are these detections in 2d or 3d? How accurate is the detection algorithm?

(5) The beginning of Section 3.1 describes that an object's orientation is represented as a quaternion. However, at the end of Section 3.1, the authors suggest that they "discard the orientation information from states fed into the generated dynamics model." This seems to me makes the "state" an incomplete representation of the environment, where the authors only predict the position of the object, which makes me wonder:

How does the model encode the geometry of the object? Will the missing of the orientation information introduce any ambiguities or uncertainties? What if the object is re-oriented? It may be better to include comparisons of different state representations.

Also, in Section 3.3, the authors suggest that they update the orientation using quaternion composition, which seems to be inconsistent with what has been described before. Haven't the model already discarded the orientation information?


=== Other comments

This paper only shows experiments in the simulation. I'm curious, are there any gaps before applying the method to the real world, and what are these gaps? For example, how long does the model take to optimize the action trajectories when performing MPC? Can it support real-time feedback control in real physical scenarios, especially when the environment is dynamic?


Model-predictive control relies on the environment's feedback to correct the action sequences, which can achieve a good control performance while tolerating a larger long-term prediction error. In your experiments, how important is the accuracy of the dynamics model? In other words, even if some baselines have a poorer forward prediction performance, will MPC be able to bridge some of the performance gaps?


In table 3, why are there multiple red numbers in the Ant-Slope columns?


Typo?
Page 4, Section 3.1:
"E_int then maps z_int to a 1-dimensional code z_int \in R^2."
This sentence seems weird. How does E_int map z_int to its own? Why does a "1-dimensional code" lies in a 2d space?


=== Post rebuttal

The authors' response and the revisions to the manuscript have greatly improved the quality and clarity of the paper. Most of my major concerns regarding the implementation and evaluation details have been sufficiently addressed; hence, I decide to increase the score from 5 (Marginally below acceptance threshold) to 6 (Marginally above acceptance threshold).
review_title: Interesting work, but lacking many evaluation details

final_score: rating: 6: Marginally above acceptance threshold
review_content: #### Summary:
This paper proposes an adaptive dynamics model based on the idea of hypernetworks. It is demonstrated that this approach compares favorably to other ways of adapting dynamics models such as conditioning on a separate feature input and meta learning by gradient-based model updates. The proposed approach is evaluated on Pushing and Locomotion tasks.


#### Pros:
- The proposed approach for conditioning dynamics models on rollouts to model system-specific properties using the hypernetworks idea seems novel and is interesting.
- Paper is clearly written, the provided figures help understanding
- Outperforms state-of-the-art adaptive dynamics modeling approaches [Nagabandi et al., 2019], [Sanchez-Gonzalez et al., 2018b]
- Reasonable baselines are used for comparison, such as fixed model (XYZ), input feature conditioning (Direct), expert ensemble, and state-of-the-art adaptive dynamics models

#### Cons:
- The paper does not explain training details for the architecture sufficiently well. How are the network components trained, especially the visual recognition part for object pushing? What kind of supervision with ground truth is required to train the components, for instance for object detection and shape representation? Are components pretrained and how? Which losses/data are used for training? 
- Its unclear why moving from a canonical to an oriented shape representation in Sec. 3.1 should improve results. Shouldnt this limit generalization and require more training data?
- Giving standard deviations in addition to the average values in table 1-3 would complete the numerical results
- Sec. 1) Why is PlaNet [Hafner et al., 2019] listed as "no adaptation", although it contains a recurrent state representation? 
- It appears magical that the approach performs better on novel than on seen objects during training for Cheetah-Slope or Ant-Slope in Table 3. Please discuss.

#### Recommendation: 
The paper reads well and proposes an interesting novel approach which could deserve acceptance.
The paper should address the points raised in paper weaknesses.

#### Questions for rebuttal:
Please address points raised above in "weaknesses".

#### Typos:
- p2: "They are are"
- p4: "E_int then maps z_int to a 1-dimensional code z_int ∈ R2" - shouldn't this be "E_int maps interactions to 2-dimensional code z_int ∈ R2"?
- p4: "which is typically comprised of an agent and its external environment" -> "which typically comprises / which is typically composed of"
- Table 1: Motion rediction error -> Motion prediction error
- Advice: In Figure 1, the concatenation symbol is slightly misleading, as it could
be interpreted as elementwise multiplication. Maybe replace it by $[\cdot,\cdot]$.


#### Post-rebuttal comments:
The authors' comments addressed my concerns on method and experimental details mostly well. I keep with my rating "6: Marginally above acceptance threshold".

review_title: Review for HyperDynamics: Generating Expert Dynamics Models by Observation

final_score: rating: 6: Marginally above acceptance threshold
review_content: == Update ==

Thank you for your detailed response. The newly added clarifications and sanity checks have greatly improved the quality of the paper, and I am therefore increasing my score from 4 to 6. I believe the model capacity comparison (Table 6) is especially important for demonstrating the value of the new architecture, and would recommend mentioning that result in the main paper.

== Original Review ==

The paper proposes a model for predicting the dynamics of a physical system based on hypernetworks:
given some observed interactions and some visual input, the hypernetwork outputs the parameters of a
dynamics model, which then predicts the evolution of the system's state over time. Experiments are
conducted on an object pushing and a locomotion task.


Strengths:
 1. The paper addresses an important question, namely, how a dynamics model may adapt to
    environments that don't fully match its training distribution.
 2. The proposed use of a hypernetwork is plausible and novel to my knowledge.
 3. The related work section appears comprehensive, and, to my knowledge,  does not miss any major
    prior work.

Weaknesses:
 1. The main claim of the paper is that hyperdynamics network offers better prediction accuracy and
    generalization than a standard dynamics model. I feel like the evaluation of this question is
    confounded by the choice of tasks and baselines. On the pushing benchmark, the XYZ, VF, and
    DensePhysNet operate on different modalities than HyperDynamics (either no state information or
    no visual information), and are therefore difficult to compare. For the MB-MAML baseline, this
    is not specified. The Expert-Ens model cannot be expected to generalize, since it is designed to
    overfit on individual objects. As a result, only the 'Direct' baseline clearly operates in the
    same experimental regime as HyperDynamics. However, nothing is reported on the model
    architecture or the training method for that baseline, raising the question if its model
    capacity was competitive. My impression is that this experimental design blurs the effects of
    (a) using side-information to infer system properties, and (b) utilizing such information
    through a hypernetwork as opposed to a standard dynamics predictor. If the goal is to evaluate
    the new architecture, these should be disentangled.
 2. On the locomotion benchmark, the Recurrent baseline is similarly unclear.  Sanchez-Gonzalez et
    al. is cited, but that paper focusses on comparing recurrent models based on graph networks to
    those based on MLPs, and it is unclear which model was used.
 3. No results are reported for the prediction accuracy on the locomotion task, which would have
    helped evaluate the performance of the dynamics models more directly than the task scores.
 4. Many of these issues could have been avoided by testing on established benchmarks from the
    literature, for which results are available. If there is a simulator available, generalization
    ability could still have been tested by varying the physical constants of the dataset.
 5. The paper contains a decent amount of typos and grammatical errors.


Overall, while the paper presents an interesting idea, the experimental evaluation is not convincing
in its current state: Baseline architectures are not fully specified, many of them did not receive
the same input, and no benchmark task with previously reported results has been used. As a result, I
recommend rejection at this time.


Questions:
 1. In eq. 1, should omega be a parameter of H(.) instead of F(.)?
 2. Section 3.1 introduces the "1-dimensional code $z_{int} \in \mathbb{R}^2$". So is it one or
    two-dimensional?
 3. Overall, the dimensionality of the latent codes and hidden layers seems incredibly small, e.g.,
    only 1/2 numbers to encode prior interactions, and 8 to encode shape. Is there really no benefit
    to using higher capacity models?
review_title: An interesting idea held back by unconvincing experimental design

final_score: rating: 7: Good paper, accept
review_content: ## Summary

The authors present "HyperDynamics", a novel method for system-identification and learning of flexible forward models that can be used in planning tasks. The presented methods is generic and is shown on both locomotion and pushing tasks with different simulated robots. 

I enjoyed reading this work a lot and I hope it gets accepted. It's a clever idea and most flaws that I'm about to point out are easily addressable by the authors.

## Strengths & Weaknesses

#### Strengths

1) The method is generic (shown to work across tasks and environments).
2) The baselines are strong. When I started reading your paper, I thought that DensePhysNet and some form of MAML would be good candidates for this to compare again and it turns out these were indeed included.
3) Figure 1 + caption as well as the introduction to section 3 do a great job at introducing the architecture in a way that would allow the reader to create a basic implementation.
4) Code was included. I didn't run it but it's clean and seems functional from what I can tell.

#### Weaknesses

1) You really, really need to be more clear in the main paper on the implementation details. You can't move the amount of training data to the appendix and it's not good practice to only include the network architecture by name in the main paper. And what are all your losses? You make the method looks super simple but then you train on shapenet, some 2D reconstruction, something about cropping, and there's a GRU in there too (full backprop vs truncated backprop?). The appendix shines a _little_ light on this but you need to be way more specific in the main paper. That has to stand on its own.
2) You don't motivate all the nitty-gritty implementation choices. Why did you add the decoder? What's the performance if you remove it? What about the cropping? What if you don't do an object-centric feature map, but instead a few CNN layers? What are the individual contributions of all these details?
3) DensePhysNet, Visual Foresight, and many other works in this domain use (simple) real-robot experiments to demonstrate that their method can handle realistic robot noise. Obviously there's a global pandemic happening at the moment, so I won't require you adding this for the rebuttal, but I think in order to really establish this method (maybe before putting it on Arxiv), you'd have to add some real-robot experiments. This can be as simple as a 180USD RealSense and a 500USD robot arm plus a few objects and a playfield. It's become a standard for system-identification-style works and it's justified in my opinion since your method isn't inherently useful in simulation where the user has access to all the information and can arbitrarily reset/reposition the model. And since you don't have ShapeNet data for many real-world objects (which you seem to need for pretraining), could you at least add a sentence or two detailing how this would transfer to real-world problems?

**TL;DR my main requests:** (2) Motivate implementation details (add ablations if you have any) and (1) be more explicit about them in the main part of the paper.

## Impact & Recommendation

Despite that there seem to be a lot of hacks that make this method in the specific settings, I think the general idea behind it is sound. And I think the authors show that it performs better than the sota, at least in simulation. Therefore I'd recommend acceptance given that the authors add the requested information. In its current shape, it's a 6 for me but if my main concerns are addressed, I'm happy to up this to a 7 or if major improvements are made and my questions below are answered, to an 8.

## Questions, Nitpicks, Comments

- Kudos for not making another acronym method
- There are a lot of typos and orthographic errors, would recommend a spell-checked or getting this proofed. Examples: section 2 "poinclouds", section 2 "properties in hand" -> "properties at hand"
- Maybe start the introduction with an example, e.g. how children are able to chew on a block of wood to assess its hardness and then build towers with it. 
- **important** Introduction: when you go over (i-iv), that feels a bit too long and lit-reviewy and misplaced in the introduction. I would recommend the following changes: (a) trim this severely, only mention that there are model-based methods that usually do only one environment and there's meta-learning and how your method is more adaptable than either, (b) move this into the literature section, where you have to come back to it anyway, (c) move the Hypernetworks section from the literature into a separate "Background" section and develop it a bit further, since it's less "competing method" and more "you should know about this to understand out method".
- Also in the introduction, you present (i-iv), and you mention how your method is better/different than (i-iii) but you never address (iv).
- It's become a standard to summarize the contributions again at the end of the introduction, ideally as bullet points. Please add these.
- In equation (1), why is the ordering O-T-N for the sums? I feel like ONT would be more natural, no?
- When reading the method, my main question was if the method would work on "dense" trajectories or on before-and-after photos like DensePhysNet. This is only answered a few pages later but I think this belongs in 3-Overview or 3.1. Just to be clear, you're gathering trajectories of length 4s, i.e. 5 frames of 800ms where you do NOT retract the robot arm when pushing, right? (Compared to DensePhysNet, where the arm is never visible because they take photos before and after complete standstill). If that's the case, how do you deal with occlusion from the arm?
- Do you encourage object-object interactions in any way or do they just occur randomly? Or do you only ever experiment with single objects?
- The object orientation vs state section isn't super clear? You're subtracting an object's absolute starting position+orientation from it's future trajectory points?
- In 3.2: Why a GRU, why not LSTM? Why k=16 (and similarly why k=5)... This ties into the main criticism from above. Please motivate your choices.
- In 4.1: I think this is a typo, but it says you added beds to your experiment table. I think they'd be a bit too large, no? :D
- 4.1: specify the random mass+friction range, please!
- 4.1: same with the total amount of training data/frames
- And since you won't have ShapeNet 
- 4.2: I think it's a half-cheetah, not a cheetah.
- 4.2: I don't understand why it's unrealistic to assume arbitrary resetting in simulation. That's one of the benefits of running simulations and common practice. 
- 5: What do you mean "predicting both the structure and parameters of the target dynamics model"? Parameters is clear (mass, friction, etc.) but what's the structure here?


review_title: Great idea, slightly flawed paper but probably possible to improve in rebuttal period.

Meta-review: This paper proposes "HyperDynamics" a framework that takes into account the history of an agents recent interactions with the environment to predict physical parameters such as mass and friction. These parameters are fed into a forward dynamics model, represented as a neural network, that is used for control.

Pros:
- addresses an important problem (adapting dynamics models to "new" environments) and provides strong baselines
- well written and authors have improved clarity even further based on reviewers comments

Cons:
- I agree with the reviewer that it is currently unclear how well this will transfer to the real world
- The idea of predicting physical parameters from a history of environment interactions is not not novel in itself (although the proposed framework is, as far as I know). The authors should include related work along the lines of (1) (this is just one paper that comes to mind, others exist)

(1) Preparing for the Unknown: Learning a Universal Policy with Online System Identification
Decision: Accept (Poster)