Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The ability of a model to adapt to online data can allow robots to achieve better and more robust performance when deployed in dynamic environments. Current methods, such as gradient-based meta-learning can be difficult to train and implement. In this work, the authors propose a novel adaptation framework for neural networks that is based on Monte Carlo Dropout. At train time, the network is trained using dropout (random masks applied at each forward pass). However, at test time, the authors do inference over dropout masks to find the best performing masks based on observed online data. This is done by particle filtering over masks using the proposed SMCD method. The authors show in two domains that their method leads to better prediction performance than baselines and further analyze the information content of the masks.
strengths_and_weaknesses: This paper addresses an important and long-standing question in the robotics community: how can we adapt models efficiently based on online, observed data? One strength is the generality and capacity of the proposed approach that has potential to scale and model complex phenomena. Additionally, the approach is well-presented and straightforward to implement.

The main weaknesses of the paper in its current state relate to the “Experimental Results” section. These weaknesses can be divided into clarity and additional comparisons. 

In terms of clarity, I believe addressing the following concerns would help the reader better understand the intuition/strengths behind the proposed approach:
- For the proposed SMCD algorithm, there are burn-in interactions to seed the adaptation process. Does the non-SMCD baselines observe the same total number of interactions before prediction error is reported?
- It would be beneficial to have more details about the Reptile baseline algorithms. Specifically, how does the R+S baseline work, where Reptile is combined with SMCD work?
- In Figures 3 and 5, why does M-N outperform R-N? Without adaptation, what are the differences between these models that would explain this performance gap?
- In Figure 3, why does R-N outperform R-G? More details on the Reptile baseline might help answer this.

The authors provide many insightful ablation studies to their proposed algorithm (for example, the no-adaptation baseline, or the oracle model). As the authors develop a novel framework for adaptive model learning, these ablations are important to understand where performance gains come from and I believe the following ablations would provide further insight:
- As I understand, in the two-link arm domain, the networks do not have access to the link length and are adapting to this unknown task context. A baseline model that has no adaptation, but has access to all relevant state as input would be a useful benchmark as I would expect it to be an upper bound on performance.
- There are methods in the literature that explicitly capture variability using a latent variable, as opposed to the proposed method which captures task variability using all the weights in the network combined with dropout masks. A comparison with this class of models would also add insight (does the proposed approach lead to better performance by using the entire model to capture task variability instead of a low-dimensional latent-variable?). One approach in this category is that of Saemundsson et al., 2018 (“Meta reinforcement learning with latent variable gaussian processes”).

The following are minor comments:
- Is there any intuition why a mean-mask estimator in Equation 6 works well? Would you expect this distribution to be unimodal?
- I would be interested in reading any additional insights into how the network specializes for different contexts (as shown by the dropout mask analysis) even though random masks are used at training time
- What are the units in Figure 3?
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: In the current state, I recommend a “Weak Reject” for this paper. The authors tackle an interesting problem and propose a novel framework for learning adaptive models. However, additional clarity is required to fully interpret the strengths and weaknesses of the proposed method. I look forward to the authors’ responses as I believe many of the identified concerns are addressable.

Post-Discussion Update:
I would like to thank the reviewers for taking the time to respond to all the points in my review. I believe that the additional clarity greatly improves the paper and have updated my score to a weak accept. In particular, the explanation provided for why R-N outperforms M-N is very intuitive.The additional experiments also provide a stronger set of baseline methods.
issues: Please see the weaknesses section.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a meta learning algorithm for adaptive robotic control in dynamic and unstructured environments. 

The key idea is to first create a pre-trained neural net with dropout regularization (that was used during training) on multi-task data representing the entire task space. During the online phase, a posterior (or belief) over the dropout mask is being updated and propagated via a MCMC particle filtering scheme.

At each time step, the average of the propagated particles (representing the mask posterior) could be used to condition the pre-trained net (e.g. retaining most relevant neurons while dropping irrelevant neurons) to fit the current dynamic of a changing environment.

The experiment is conducted on two settings: (1) a toy setting where the task is to predict next state given current state & input control; and (2) a more realistic setting where the goal is to predict the (latent) landing skill level of human operators given their demonstrated landing sequences. 
strengths_and_weaknesses: Strength:

The idea of repurposing the dropout regularization as a mechanism for online adaptation of a pre-trained model is interesting.
The paper is well-motivated and has a comprehensive review of literature on model pre-training & adaptation techniques. 
The mathematical exposition is neatly presented, communicating concisely the key idea via detailed pseudocode.

Weakness:

Although the idea is interesting, it has several restrictions: 

1. First, given the formulation, it can only be applied to situations where all tasks in the space of interest share the same output space. This is much less flexible than other meta learning approach, such as MAML, which allows training & test tasks to focus on different output sub-spaces.  

2. Second, generating pre-trained net on a master space of output & multi-task data that represents the entire task space is probably much more expensive in both compute and memory footprint. 

For more sophisticated learning scenarios, the joint output space of the tasks could easily have tens of thousands of dimensions even though each task only focus on a subspace of output with less than a handful of dimensions. For such cases, it is arguable that a MAML model (whose output dimension is no more than that of any single task) is more cost-efficient & more suitable as an adaptation base.

3. Next, although the motivation of this paper highlight use cases where model adaptation is necessary to enable adaptive interaction to changes in environment or behavior of human collaborator (line 33), the actual reported experiments do not reflect such cases. 

In fact, both experiments are not about learning and adapting control policies. Instead, one experiment is about predicting next state given acting policies while the other is about predicting intrinsic attributes of human operators given their demonstrated landing sequences. The authors should consider including dedicated experiments towards learning adaptive control policies, as these experiments are more relevant to what is being motivated.

4. Last, the comparison baseline is a bit limited to Reptile and does not include any of the more recent work in meta learning. The authors also mentioned that there was comparison with first-order MAML (line 205) but that does not seem to be included in the reported experiment. Moreover, there should also be discussion & comparison of processing complexity of the baselines. For example, for the proposed method to be effective, what would be the necessary sample size?
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper proposes a seemingly interesting idea for meta learning via learning and propagating a posterior on dropout mask for a pre-trained net on a collective set of multi-task data. However, the authors had not considered practical aspects of such idea, e.g. scalability, memory & compute footprint etc. From the technical description, it is likely that the proposed method is significantly more costly than existing methods, at least in term of representation complexity.

In terms of experiments, there is a lack of comparison with more recent work in meta learning. The experiments also focus largely on toy dataset & in settings that are only remotely relevant to what was used to motivate this paper. Given this, I believe this paper is still a work in progress that unfortunately does not meet the bar for publication yet.
issues: I have pointed out several issues with this paper above. To summarize:

1. Lack of consideration regarding scalability, memory & compute complexity (in both technical narrative & experiment)
2. Experiment focuses on toy settings that do not match well with the motivation
3. Comparison with more recent baselines in meta learning is missing

While I do think the idea presented here is interesting, I am not quite convinced of its impact because of the above. I encourage the authors to expand the paper according to the above to complete & strengthen this work.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper is considering the problem of adapting neural models to changes of task settings. The proposed method is to train a neural model with using dropout technique and then compute the most suitable dropout mask for the particular task at each step. The authors compare their approach with an oracle and another meta-learning algorithm called Reptile. The proposed method does not require any additional training of existing neural model and outperform more complex meta learning (Reptile) algorithm.
strengths_and_weaknesses: Strengths:
* The proposed method doesn’t require additional training of underlying neural model.
* Authors consider two different experiments.
* Outperforms the existing solution.

Weaknesses:
* It does not clear from the article how the number of mask N affect the method performance.
* Authors consider only simulated environments
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This is an interesting paper with potential impact for robot online adaptation, however there is no evidence that this, method would be good enough in complex real world robotic problem.
issues: * Describe the effect of the parameter N that stand for number of mask to use.
* Discuss how the method may / may not be used in real-world applications.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a method to perform system identification with with a world model by leveraging dropout masks.  The general idea is to train a neural network on a randomized version of the system dynamics, sampled from a domain encompassing that which is likely to be seen in the final system.  This network is trained with dropout, using no particular training regime, therefore re-sampling a dropout mask at each forward pass.  Then, during use, there is a 'Mask Adaptation' phase. This is similar in spirit to CEM, and samples a set of random masks, evaluates which ones provide the best next-state estimation, and then re-samples masks around the space of well-performing masks from the previous iteration to fine-tune the mask selection.  This process continues permanently, allowing for not only initial sys-id, but also on-line adaptation to non-stationarity in the system dynamics.
strengths_and_weaknesses: I find the idea very clever, simple, and easy to implement.  It makes intuitive sense that this could work well, and it provides an appealing approach to dealing not only with sys-id but also adaptivity to non-stationarity in the system dynamics.  These are both very real prbolems in physical systems and having a simple, stable, easy-to-implement solution is very appealing.

The main downside of the paper for me is the very limited experimental results.  I would have really liked to see this approach in use in more controls-oriented tasks, on higher-dimensional systems.  There exist very easy to use environments such as DM Control, Gym or Brax that can be easily perturbed (or even more specific manipulation environments such as Robosuite), and existing model-based controllers such as PDDM, or various MPPI implementations which could easily be extended with this adaptive world-model.  Experiments in these domains would be much more convincing in demonstrating this approach's practical abilities in higher-dimensional control tasks.

I'm also not convinced the formal description of the approach in Eq. 1-3 was particularly heplful, I found it rather more confusing and only understood the mechanism by reading Alg. 1.  I would encourage the authors to perhaps provide a closer comparison to CEM, perhaps the algorithm could be cast as a CEM variant which would also make it easier to contextualize and describe.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I think the core idea of the paper is interesting and applicable to a large number of domains.  It feels so intuitive that I wonder if it has been published elsewhere in a different form, but I wasn't able to find a paper suggesting an explicit adaptation phase for dropout-induced variability in the network outputs.  The paper correctly (to the extent of my knowledge) references existing work in dropout approaches to bayesian inference.
issues: The main issues are described above: the mathematical description of the algorithm is a bit awkward, and could perhaps be better contextualized relative to existing approaches (EM, CEM).  The other main issue is the relatively simple and low-impact experimental section which really misses the opportunity to make a compelling case for robotics control in higher-dimensional systems.  I nevertheless argue for a weak accept because I find the suggested approach very appealing, but will not defend it if other reviewers find the experimental section lacking.
review_title: null

Meta-review: metareview: Tackling the important problem of online adaptation with the proposed dropout mask adaptation during runtime is a simple but clever idea, as it requires no additional training, is easy to use, and could potentially be transferred to many domains and models. The approach is well motivated and presented.

However, there are some open questions and concerns raised by the reviewers. Please see the following proposed action items based on these (more details in the corresponding reviews), and the consensus is that most concerns are addressable and hence the authors are encouraged to think about investigating them.

Proposal of action items:
- Clarification of the burn-in interactions with respect to the baseline models and reported prediction errors
- Clarification of some of the reported results and models (R+S, M-N vs. R-N, R-N vs. R-G, see review)
- Consider additional ablations and comparisons for more insights
- Consider more experiments with respect to learning adaptive control strategies, higher-dimensional systems, different underlying models, etc.. , to create a strong message for the proposed approach (see review for proposed environments)
- Clarify practical aspects of the approach, e.g. scalability, memory & compute footprint, etc.
- Clarifying the effect of the number of masks N affect performance

-----

I’d to thank all reviewers and authors for constructive feedback, clarifications, and additional material. There is consensus that the presented method tackles an important problem and that the method is well motivated and presented. While there is no clear consensus in the recommendations, the updated and significantly enriched paper (in addition to the rebuttal discussions), tackled most of the raised concerns. My recommendation is to accept the paper due to its strengths that outweigh the flaws. The proposed idea is clever and simple, easy to use on a variety of settings and models, and has the potential to spark many new ideas in this direction.

confidence: 3: The area chair is somewhat confident
Decision: Accept (Poster)