Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Meta-RL is an important direction to apply RL to real-world applications which require an ability to adapt to new tasks/environments. This paper tackles meta-RL using hypernetworks that produces the parameters of main agent (i.e., another neural networks). First, the authors show that performance can be very sensitive to initialization of hypernetworks and propose two initialization schemes. The first one is not sharing parameters between tasks (i.e., no bias) and independently initializing parameter per each task and the second one is only utilizing the shared one. The authors evaluate the proposed methods on several meta-rl benchmarks.
strengths_and_weaknesses: * Strengths

1. Interesting research direction: the idea of utilizing hypernetwork for meta-RL sounds interesting (but I also expect that it also has some limitations; see `Weaknesses`)

* Weaknesses

1. writing: I may miss something but the learning components for hyper-network are not clear. At the beginning, I thought that this framework is training NNs (hypernetworks parameterized by $\theta$) which generates parameters ($\phi$) of main agents. However, from Section 4.2, $\theta$ is gone and parameters of hypernetwork ($W,b$) become the parameters of main agent ($\phi$). Overall, current description is bit confusing and hard to understand.

2. soundness: several components are not convincing. First, the authors mentioned that task id is not given in meta-rl but they assume that task id (one-hot vector) is given as inputs to hypernetworks. Also, the proposed initializations just look like independent initialization or one shared parameters. Because of that, it is bit hard to say that the proposed method is convincing and novel.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I'd like to recommend "weak reject" due to the weaknesses mentioned above.
issues: * As mentioned in Weaknesses, it would be nice if the authors could clarify the confusing parts (e.g., structure of hypernetworks, problem formulation and so on).
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper has two contributions: 1. it demonstrates the use of hypernetworks for meta reinforcement learning and shows that such approach outperforms VariBAD; 2. it proposes a novel initialization scheme for hypernetworks which makes training possible.
strengths_and_weaknesses: The paper is clearly written and easy to follow. Its main downside is that it is unclear whether the authors want to make a contribution to meta-RL or to research on hypernetworks. In its current form, it doesn't present enough experimental evidence to be considered a significant contribution to either of the two fields.

Specifically, given that the main novelty is the proposed initialization scheme for hypernetworks, I would expect to see supervised learning experiments since those are the benchmarks that the baselines were evaluated on (see e.g. the HFI paper).

Regarding meta-RL, I would expect to see comparisons to more baselines than just VariBAD. In particular, you report 19% meta-test success rate for ML10. Looking at the meta-world paper, they report 35.8% meta-test success rate for ML10. You report 5.6% for VariBAD so why is it considered the state-of-the-art?
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Paper needs more thorough experiments evaluating the proposed methods.
issues: See the strengths&weaknesses section.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposed using hypernetwork in Meta Reinforcement Learning problem. The authors stated that initialization is crucial for the hypernetwork performance. Thus, they proposed and tested two initialization methods: Weight-HyperInit and Bias-HyperInit. In this work, the author run experiment on 2D navigation, MuJoCo, ML1 and ML10 benchmarks in Meta RL setting. Using VariBAD algorithm as baselines, they showed the effectiveness of their method on those baselines. They also compared the performance of using hypernetwork and VariBAD baseline with the equivalent base network size. Also, they pointed out that this method still has some limitations and needs to be tested on real robots.
strengths_and_weaknesses: Strength
1. Well stated motivation in using hypernetwork for meta reinforcement learning.
2. Good performance in control problems.
3. Clear analysis in the scalability of the hyper-network.

Weakness
1. The meta-RL problems are variants between different RL task. Evaluating in meta-RL where each tasks differ in the goal context would make this paper stronger.
2. Specifying how many tasks are used for meta-training stage and meta-test stage will make this paper stronger.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I recommend weak-accept of this paper. The proposed method is efficient and the paper is well written. The experiment showed the effectiveness of the proposed method although some of the details might be missed.
issues: 1. In section 5.2, the variance of Bias-HyperInit is big. Does that mean using hypernetwork is still sensitive to initialization in Pick-Place tasks?
2. Section 5.3, the author mentioned comparing performance for equivalent number of parameters in the entire model. But I could not find the related results.
3. The proposed hypernetwork architecture is quite straightforward. Does that mean a near-optimal policy for each meta-RL problems in the experiment can be considered of having a linear transform in the policy parameter space?
review_title: null

Meta-review: metareview: Strengths:
- Novel research direction
- Good empirical results in simulation for some tasks

Weaknesses:
- Some important parts of the method are not clear (e.g., how is task id inferred by the network)
- Only baseline is VariBAD, which is not SOTA on some of the tasks. 
- No comparison of the proposed initialization in supervised learning

Post rebuttal/discussion:

The authors clarified the unclear points raised in the reviews, and added a comparison with RL^2. 
While supervised learning experiments will strengthen this paper, there is sufficient interest here to CoRL audience even without them. 
Finally, all reviewers agreed that this paper can be accepted.

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)