Adapting Neural Models with Sequential Monte Carlo Dropout
Authors: Pamela Carreno
Dana Kulic
Michael Burke
Venue: CoRL 2022 Poster (2022)
Keywords: Model Adaptation
Meta-Learning
Online Robot Control and Prediction

Abstract: The ability to adapt to changing environments and settings is essential for robots acting in dynamic and unstructured environments or working alongside humans with varied abilities or preferences. This work introduces an extremely simple and effective approach to adapting neural models in response to changing settings, without requiring any specialised meta-learning strategies. We first train a standard network using dropout, which is analogous to learning an ensemble of predictive models or distribution over predictions. At run-time, we use a particle filter to maintain a distribution over dropout masks to adapt the neural model to changing settings in an online manner. Experimental results show improved performance in control problems requiring both online and look-ahead prediction, and showcase the interpretability of the inferred masks in a human behaviour modelling task for drone tele-operation. 

Adapting Neural Models with Sequential Monte Carlo Dropout
Pamela Carreno-Medrano
Dana Kulić
Michael Burke
Department of Electrical and Computer Systems Engineering Monash University
Australia
(pamela. pamela.carreno, dana. kulic, michael.g g. burke) @monash edu
The ability to adapt to changing environments and settings is essential for robots acting in dynamic and unstructured environments or working alongside extremely humans with varied abilities preferences. This work introduces simple and effective approach to adapting neural models in response to changing settings. We first train a standard network using dropout, which is analogous learning an ensemble of predictive models or distribution over predictions. run-time, use particle filter to maintain distribution over dropout masks adapt the neural model to changing settings in an online manner. Experimental results show improved performance in control problems requiring both online and look-ahead prediction, and showcase the interpretability of the inferred masks human behaviour modelling task for drone opeeatinnt
Abstract:
Keywords: Model Adaptation, Online Robot Control and Prediction
Introduction
Neural models and policies are now ubiquitous in modern robotics. The prevailing approach to training these follows two stage process large, comprehensive collection of data (often state and action pairs) is used to train suitable model or policy, which is then frozen and deployed. Unfortunately, this results in models that are unable to adapt to changes in the environment, which is a particular concern in robotics. For example, it would be preferable for a robot dynamics model handle context dependent kinematic or dynamic properties, or collaborative robot relying on predictions of human behaviour to adapt to different human abilities or preferences. Many existing adaptive control techniques [1] attempting to tackle this problem rely on carefully considered para- metric models, but these may lack the requisite capacity for prediction that is typically associated with neural models. In contrast, meta-learning and adaptive neural control approaches addressing this problem are often quite cumbersome train and implement.
This paper introduces simple and effective approach achieve adaptation for neural network models. First we train model, fe(-) using dropout [2]. on data gathered from different settings or tasks. Dropout can be considered analogous randomly masking neurons in the network, producing
an ensemble of predictive models, or distribution over possible outputs [3]. At run-time, a particle filter used to update the dropout mask, depending on the predictive quality of the model, so as to condition the network to operate in given context. This allows for neural models that can easily and rapidly adapt to changes in the environment behaviour of human collaborator in online settings.
Results show that this simple-o-implemmnt approach improves upon a more complex first-order meta-learning training strategy, and alleviate potential failures due to catastrophic forgetting. Moreover, we show that the resulting masks can be interrogated to obtain additional information about the adaptations required for particular contexts. We also show how this information can be leveraged in human modelling application to answer questions about given user. In summary, the core contributions of this work are to show that: (1) sequential Monte Carlo dropout an effective technique for online model adaptation and control in rapidly changing settings, and (2) the masks
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

inferred by this process provide a context-depeneen 'signature' that is of particular value for human behaviour modelling in support of personalised robot decision making.
2 Background and Related Work
Model adaptation: Model adaptation in an online fashion is a core focus of adaptive control [1], and typically involves the design of suitable observer or estimation scheme [4] infer parameters of interest. Many of these system identification approaches [5, 6, 7, 8, 9] rely on models conditioned on known physical properties like friction, mass and gravity, and infer these from data. These approaches require knowledge of the parametric form of uncertainty and are only feasible for very specific model structures [10]. Our approach introduces an adaptation mechanism applicable to more general predictive neural models.
For deep learning, model adaptation can be studied generalisation problem in meta-learning context. Popular approaches like model agnostic meta-leaming (MaML) [11] seek to learn network weights that can be easily adapted for different tasks by gradient descent via specialised training regimes that alternate between task specific training and maintaining a representation that is well suited to multiple tasks. Although time-consuming, faster, first-order meta-learning approxima- tions such as Reptile [12] have proven relatively effective. The idea of enhancing generalisation by learning well-structured internal representations underpins a number of works such as ProtoNet [13] and Relation networks [14]. alongside more recent approaches that rely on multi-task training with an explicit learned auxiliary input [15, 16] that can be adapted online. Our approach can also be viewed through representation learning lens, in that masks 'prime" the network to produce con- text dependent outputs. This representation-prinn prrmmin perspective also aligns with recently developed transformer- based architectures tackling meta- meta-learning (e.g,, [17, 18, 19]. Here, key embeddings are used to unlock network components required for context specific prediction. The dropout mask used in our work is analogous to a transformer key that is inferred at run time.
Several works targeting online model adaptation have argued that we should ideally update our models and beliefs in response to evidence in a Bayesian fashion. This approach is taken in clastic weight consolidation, which aims to reduce the plasticity of network weights that are important to a task [20]. There is also some evidence that MaML can be reinterpreted as a Bayesian hicrarchical model [21, 22]. Dropout [2] popular regularisation technique for neural network training, and has also been used to represent prediction uncertainty in deep learning [3]. We leverage dropout within a sequential importance sampling framework to provide an explicit Bayesian update rule for mask inference and model adaptation. Like [20], this preserves connections important to a task.
As alternative to back-propagation, evolutionary, sampling-bases approaches to training neural models [23, 24, 25] have been explored and shown to be effective in a reinforcement learning context [26]. Similarly, the cross entropy method is a popular technique for global sampling-basee based optimi- sation and planning [27, 28]. Pathnet [29] uses evolution to find the neural network pathways that should be trained for specific tasks. Particle filtering has also been used prune neural networks for compression [30]. Our work is similar, but considers the online adaptation case and samples dropout masks to infer a distribution over masks, rather than considering all weights and connections.
The dropout mask inferred at run-time using our approach can be considered as embedding summary of the context-specific adaptation required by the network at given time step. This is valuable for adaptive human modelling in shared-autonomy and similar collabo- rative applications, where context-specific masks could help guide the type of assistance offered to an operator. Human behaviour models help robots to understand human preferences, predict future human actions, and adapt to changes in human behaviour. Two general approaches to human be- haviour modelling are the most common in robotic applications. The first postulates that humans are rational agents whose actions are in accordance with some (un)known intentions and rewards [31]. To account for human variability and diversity in behaviour in particular deviations from opti- mal action choices, these models are augmented with computational interpretations of high-level decision factors such as trust [32], expertise [33], fatigue [34], and adaptability [35], among others. These factors can be inferred in an online manner during interaction, thus allowing robots to adapt their responses to different human users (e.g., [36)). Often preferred because of their simplicity and interpretability, these models suffer two main drawbacks. First, they strongly rely on simplistic parametric model structures, which can limit model expressivity and predictive power. In particular,
Human modelling:

---

most models assume that a single high-level decision factor is sufficient to capture and explain most of the observed human behaviour (e.g., [36]). Second, to make inference tractable in real- these approaches are often limited to simplified state and action space representations, and further assume that the model parameters remain stationary for the duration of the whole interaction (e.g.. [35])- Our approach aims to relax these constraints by introducing an adaptation mechanism that can be used with more general predictive and expressive neural models and that also captures changing behaviour.
In the second family of approaches, collected human behaviour data is used train machine learn- ing models that predict future behaviour (i.e.. imitation learning) [37, 38, 39]. Because of their data-driven nature and assuming good quality data is provided during training, these models can im- plicitly capture most of the sources of variability in the observed human behaviour without making explicit assumptions about what those sources are. Thus, these models of human behaviour are no longer limited to single decision factors. However, this increase in model expressivity and predictive power comes at the cost of limited interpretability [40]. Furthermore, these models can fail to gener- alize and adapt to humans whose behaviour significantly differ from the training data set [16]. Our approach addresses the interpretability and adaptation limitations of imitation- -baed human models through the online estimation of dropout masks. As shown in Sec. 5, this mask inference process allows for successful local adaptation in extrapolation (to human users never seen during training). Similarly, we show that contextual information pertaining to high-level factors such human operator's 's skill level and preferences can be queried through a simple k-nearest mask approach.
3 Method
We consider a standard Markov decision process, with x, denoting a state at time i, and RG denoting the reward received time after transitioning from state x,-1 state x,, due to control action u,- Let fo(:) be a neural network with parameters 0. Networks such as these are frequently trained to approximate dynamics models for future state prediction x; fo (x; Uz), for reward prediction RG ~ ge (X uz), or as policies for selecting an action given a state u, ~ (Xe).
Our goal is to find way of adapting a pre-trained network to changing settings in response some deviation between predicted states, actions or rewards and those actually observed. Let fo( denote neural network trained using dropout regularisation. Here M, represents binary mask which suppresses or drops neurons in the network. When training with dropout, neurons are ran- domly masked during the forward pass. We do not consider any specialised training regime, and simply use multi-task training samples drawn at random from a large training dataset.
We now show how this can be exploited to enable online adaptation. For convenience, we restrict the discussion below to a dynamics model setting where a neural network is trained to predict a state x,, given a previous state X,-1 and action u, However, it should be noted that the proposed approach can be used to adapt policies, reward predictors and in many other settings.
3.1 Online Mask Adaptation
We use the deviation between predicted x, and actual observations z, to form a Gaussian likelihood,
p( M.) -2(2: u, MM), E),
(1)
where Σ denotes measurement noise. This likelihood measures the chances of making an observa- tion given a dropout mask M.. Taking an ensemble view of model training with dropout, we aim to find the mask and corresponding model, that results in the most accurate prediction in given setting. Formally, our goal is to determine a posterior distribution over dropout masks p(M, (1:)) conditioned on series of observations z i:r- To do so, formulate sequential update rule
p(M./zaa) x p(z,
(2)) where
M. p PMM (M = M.
using appropriate transition model p(M M,-1) for the dropout mask. In this work, we propose to randomly bit-flip some small fraction (d ()) of the mask for the transition model, p( (M, M,- This allows for some exploration over potential new masks and adaptation to new settings while
3

---

keeping existing masks close to those associated with networks best suited to current settings. Im- portantly, bit flipping can be efficiently computed using an xor operation.
Since the integral in (3) is intractable, we approximate it using importance sampling and hence reduce the computation of the posterior over masks
bootstrap particle filter [41]. That is, we approximate the distribution over masks using a set of N weighted particles, p(M, Z1:e) R E wy)(M)) with weights wi; p(z, (M)), particle M; drawn from the transition distribution P(M )), and Dirac measure 8(-) at point (-). A resampling step is used after each iteration to prevent particle degeneracy. This produces distribution over masks and a corresponding posterior over the predicted state
N N
1
21:t) R
8(M);
(4)
t=1
pxx1|xx-11 U1 21zt) = p(xx x1 ue 2ia. MM))(M
N
8 Ux, Σ N
(5)
For downstream tasks like control, where only interested in the adapted prediction model, compute a minimum- memm squared error mask estimator
N M,
Σ N
1
M
from the mask particles and use this condition the network fe(.). We note that since the likelihoods are bounded and we use a standard resampling scheme, conver- gence of the proposed approach is inde- pendent of the state dimension, with the rate of convergence in & [42]. The fil- ter also allows for variable particle sizes, which can be increased or decreased on- line depending on the effective particle size as defined by Nejs = EN E (w [43]. Algorithm 1 illustrates the adap- tation process, which we term sequential Monte Carlo dropout (SMCD). We first sample N possible masks from the prior mask distribution by bit flipping, and then make a batch prediction of possible states for each mask. We then compute the like- lihood for each mask using (1), normalise
Algorithm 1 Sequential Monte Carlo Dropout (SMCD)
Model Adaptation
Input: Trained net fe. Observation sequence Z1:T, Transition probability d, Measurement uncertainty Σ Output: Mask estimate sequence M Initialise N random masks {M) | for t T do
for 0... .N do
Sample M; ~ pMM M by bit-flipping
Sample x; fo (x1-1x111 un M;)
Evaluate w A(zz x 2)
end for
> m;)(MM) Σ
Draw N masks: [M)
Compute best mask estimate M. N Ei M
end for
and re-sample, obtain an approximate distribution over masks that gives predictions close the observation. Finally, we compute a minimum-meet mean squared eтror mask, which can be used to con- dition the network in downstream tasks. The process repeats recursively and online.
key benefit of this approach is that it does not change the underlying models learned. The original prediction can be recovered any time, by simply restoring resampling the mask. This is in contrast to approaches that continue to train or take gradient steps in response to errors, which may be suitable for continual learning, but are vulnerable to catastrophic forgetting [44, 20].
4 Experiments
We investigate model adaptation using SMCD both simulation and with real-world human data. In simulation, we use two-link arm with changing link lengths and a 7 Degree of Freedom (DOF)

---

Panda arm reaching task with varying end-effector length. We evaluate the performance of the pro- posed method in prediction and control using the the two-link arm task and 7-DOF Panda reaching task respectively. The real-world data corresponds to drone tele-operation scenario where operators of varying skill levels completed several landing tasks. This scenario illustrates mask interpretability by showing how different high-level characteristics (an operator's landing strategy and skill level) can be determined by a simple mask comparison process. We refer the reader to the Appendix for additional experiments including applications to Reinforcement Learning (RL) settings.
4.1 Two-Link Arm Task (Forward Kinematics)
The 2-link arm follows the dynamics and forward kinematics:
Φi W1 42 u2
(7)
x = 1 (q)) l2 cos( Q1 + 92). y = l sin( + l2 sin(q1 + 92).
(8)
We train neural models fạ) to perform forward kinematics (predicting end-effector position x [cc y/" given joint angles q = [4), q2]²) for variable limb lengths 4, 12. Training data is gathered by randomly initialising y; U( ( T). and episodically motor babbling uq (O, 1) for 10 steps. We generate 150 episodes for 1000 tasks, with fixed link lengths 4, ~ (11, 0.3) in each task.
The two-link forward kinematics task is used to investigate the adaptation properties of SMCD in N- step look-ahead prediction after I' adaptation steps'. Specifically, we evaluate prediction error for an unknown manipulator with link lengths (4: (1,, fixed over motor babbling episode (u, (0, 1)), starting from a randomly sampled angle d: ~ U(- π². This evaluates the "interpolation" ability and suitability for receding- seceding-horizon horizon model predictive control tasks. We benchmark against an oracle model, no adaptation, a latent variable model, and a model trained using Reptile [12], a first order meta-learning approach, and investigate adaptation using both gradient descent and SMCD.
4.2 7-DOF Panda Arm Reaching Task (Manipulator Control)
The 7-DOF Panda manipulator simulated using the python robotics toolbox [45] with a tool that introduces arbitrary unknown end-effector extension (1 ~ U'-0.5, 0.5)). As with the two-link arm task, we train neural models fo(q) to perform forward kinematics (predicting end-effector posi- tion [xr y, 2| given joint angles q 1g1, ....... 9rl " Our goal is to drive the Panda from fixed reference pose to within 5 cm of a randomly selected and kinematically feasible end-effector goal Xg in less than 200 time steps. To reach the goal, use the proportional-derivan law -KJJ (gg)= K24, where Jr corresponds to the Jacobian of the end-effector position (RR)) joint with respect to the angles (RT). We consider the case where measurement of the true end effector position is only available every n-th timestep, which requires model-reference control when measurements to global context changes under a realistic control setting in which observations are captured at a lower rate than the required controller frequency.
(PD) control
not available. This measures the ability of the adapted model to generalise
4.3 Drone Landing Task (Mask Interpretability)
We use the human data previously introduced in [46] for this task?, This dataset comprises 528 tele- operation trajectories collected for 33 human operators (16 trajectories per operator) with different skill levels in a simulated environment. Here, operators were asked to pilot and land a drone on one of five platforms by providing linear velocity commands through keyboard inputs. As shown in Fig la, the operator's ' view was fixed to a single-point perspective, which resulted an increase in task complexity the further away the target platform was located. The simulated environment and landing task sequence were the same for all operators.
We train neural models variable tasks (i.e., different target platforms), user skill levels, and landing strategies (see Fig. 1b for examples of tele-operation trajectories with these characteristics). A tele-operation trajectory
fo (xx) ) to predict the operator's s action given the current state æ; for
Results showcasing SMCD's performance in online, on-policy adaptation using proportionallderivaiyy (PD) control law for this task are shown Appendix. Secondary analysis of this data was reviewed and approved by Monash University Research Ethics Com-
mittee.
5

---

Expert Orthogenal
Expert Biagunal
Novice
Dlagoral
platforms
Trgeet platform
(a) Drone landing simulation environment.
(b) Examples of tele- operation trajectories (YZ view).
Figure I:
Drone landing data consists of tele-operation trajectories collected in simulated envi- ronment (a). Data includes trajectories executed by operators of varying skill (one novice and one expert) and captures different landing strategies (orthogonal diagonal) as shown in (b).
(æ1, U1........ TT, Ur) consists of sequence of states and actions, where each state 2y (Pr. Ve,de) denotes the drone's ' 3D position p: = (Iz, yt. 2t). linear velocity V; = (dt, Ut, & and distance d, gg' pilz to target platform g' at time t; and u, (#t. ÿt, 24) corresponds to the linear velocity commands provided by the human operator. We randomly split the dataset into training (83%), known-operatorr testing (10%), and unknownssperrtors testing (7%) sets.
To investigate the interpretability of the masks obtained using SMCD we evaluate whether one high- level characteristic of trajectory (landing strategy) and one characteristic about the operator (skill level) are captured by the inferred masks (see Fig. lb for trajectories showcasing these characteris- measure how the prediction confidence of each target characteristic changes as the number of observed state-action pairs seen during mask inference increases. We use the distance-based confidence score proposed in [47] evaluate this. Given the mask estimate, M for trajectory obtained after observations, with
tics). Specifically,
< T, let (( {M train 1 denote the set k-nearest masks M in the training set, and let {er } denote the corresponding characteristic labels associated with masks in A 1(M Following [47], the confidence score D(T) for the assignment of trajectory characteristic based the mask estimate M1. is defined
Σx Ej=Lo= [M
D(T)
ΣØ= M
M Mi
1) (the higher the better) is monotonically related to the local density of tra- jectories in the training set that share similar characteristics and lie in the neighborhood of trajectory T, as defined by the distance between inferred masks.
The score D(T) E (0.1)
Experimental Results
Two-link arm: look-ahead prediction results: Fig. 2 shows the root mean square prediction error (RMSE) over 1000 trials for varied burn in steps and prediction horizons (identical for all methods compared). We compare best performing models* trained using multi-task pre-training (M) and Rep- tile meta-learning (R), and then adapted using either SMCD (S), gradient descent (G), or no adap- tation (N). We also include an oracle model baseline (O) where link-length parameters in a known forward kinematics model are inferred directly using a particle filter and a latent variable model (LV) where task variability and contextual information hidden embedding, E R², of a variational recurrent auto-encoder (VRNN) [48]. At test time, burn-in steps are used update the VRNN's hidden embedding later used predict each step of the look-ahead horizon".
both captured using the low-dimensional
Multi-task training with SMCD is most effective (M+S), with the exception of the oracle model. Interestingly, adaptation using gradient descent seems to exhibit more local adaptation, as prediction
3 CCharacteristic definitions and details are provided in the Appendix
'Best performing models were selected following hyperparameter search see Appendix 'Since neither the multi-task nor the Reptile model networks are recurrent models, the VRNN model was constrained to train using sequences of length 2 make the comparison between models possible.

---

Burn stops: 50 Lookahood stops 0.13
0.31
0:46
0:43
0:46
0.68
0.80
1.69
Burn steps: 50 Lookaheed steps 25
0.19
0.38
0:47
0.56
0.67
0.68
0.92
1.67
1.5
steps: 50
steps
1.2
Bumn steps: 100 Lookahead steps 0.20
0.35
0.48
0:45
0.56
0.71
0.94
1.73
RMSE
Burn steps: 100 Lookaheee steps 25
0.9
Bum
Lookahnod steps 0.29
0:45
0.50
0.67
0.78
0.71
1.37
1.73
Burn in steps: 150 Lookaheed steps 10
0.22
0.39
0.51
0.45
0.62
0.71
0.95
1.65
0.6
Burn steps:
steps
0.3
Bumn steps: 150 Lookahead steps
0.31
0.48
0.50
0. 66
0.81
0.71
1.36
1.68
O+p
M45
M+N
M+G
R+5
R+N
R4G
WV4N
Figure 2: Look-ahead prediction errors. (O: Oracle, M: multi-task, LV: Latent Variable, R: Reptile;
P: Particle Filter, S: SMCD, G: Gradient descent, N: No adaptation.)
error grows larger with look-ahead distance than when SMCD is applied. Local adaptation seems to provide an accurate model in specific region of state space, rather than globally adapting to particular link length. The Reptile model performs surprisingly poorly, particularly given that was trained for substantially longer (1000 epochs) than the multi-task model (200 epochs). A similar performance observed for the latent variable model.
Measurement rate (1/1)
Measurement rate (1/5)
Measurement rate (1/10)
Measurement rate (1/20
100
100
100
100
Task successes (%))
Task successes 1%6)
Task successes 1%6)
Task successes 1%6
60
60
60
60
40
40
40
40
20
20
20
20
M+G dr-11-33
5 (N-10)
M+ (N=50)
M+5 (N- 1001
M4 5001
M+5 =10000
M (N-2000)
M+5 (N-5000)
M+5 (N= 1001
11 5001
M+5 -10001
M+S (N- N--0000
M+5 (N-5000)
M (0=50)
M+5 100)
5000
M+5 16000
M+S (N=2000)
MrS (N-5000)
MHG (ir1ee3)
Me (0)
M+5 (N= 100)
500
M+5 (N- 10000
M+S 200000
MrS (N-5000)
MGG dr-
M+G (r-
M+
M+
M+5
#1
M+5
M4s
Figure 3: 7-DOF Panda manipulator control over increasing horizons. (M: multi-task, FK: An-
alytical, G: Gradient descent, S: SMCD, N: No adaptation) The orange line indicates best case
performance when true measurements are available every time step.
7-DOF Panda manipulator: online control results: Fig. 3 shows the success rates in the adaptive Panda control task for decreasing measurement rates. We used a 3-layer, 1024 dimensional fully connected network with ReLU activations to learn forward kinematics models. The model was trained with 100000 sampled joint angles and end-effector position combinations, with an arbitrary tool offset. A total of 100 reaching tasks were completed during testing and results averaged across seeds.
We compare an analytical model (FK) that fails to account for the tool offset and a multi-task model with no adaptation (M+N), adapted using gradient descent (M+G), and adapted using SMCD (M+S). When true measurements are available at every time step. FK represents an oracle model that only fails due to singularity effects in the control law. M+G performance degrades significantly as the rate of measurement decreases and the model is required to be used for longer control periods. When selecting the M+G learning rate, we found that this performance degradation increases with learning rate, which appears to indicate that gradient descent is only suitable to adapt representations to very local contexts, and the model no longer useful in other regions of the state space. In contrast, M+S appears to be much more effective at capturing global context, although performance still degrades when the model used over longer periods with no feedback. Increasing the number of particles results in moderately improved performance.
7

---

Drone landing task: mask interpretability results: Fig. 4 shows the average confidence score obtained for the prediction of the true skill level of trajectory's 's operator (i.e., novice, intermediate, or expert) and true landing strategy (i.e., orthogonal or diagonal) for all trajectories in our testing sets as the number of time steps used during mask inference increases.
Overall, we observe relatively good av- erage confidence scores (~~ 0.7) for both testing sets. This suggests that the masks inferred using SMCD con- tain enough global information to allow for the identification of operator skill level and the landing strategy. Both characteristics can be identified early on in the trajectory. This is particu- larly important for the landing strategy, which appears to change more towards the end a trajectory. Better con- fidence scores are obtained for trajec- tories executed by unknown operators than for the known operator testing in- stances. This might be due to imbal- ances in the number of instances that belong to each possible characteristic, visual inspection of trajectories indicated unknown operators exhibited trajectory characteristics more common in the dataset. Additional results on mask interpretability and look-ahead prediction for this task provide in the Appendix.
Landing Strategy
1.0
0.8
Confidencc score
0.4
0.2
Known operators
Uninown
Time steps
Figure 4: Average confidence score for the correct assign- ment of the true skill level and landing strategy for all testing trajectories (I-sigma shaded traces). Results are presented for the set of 5-nearest masks.
Limitations
Although neural dynamics modelling holds the potential to scale to high dimensional inputs of varied modalities, this work has mostly considered adaptation in proprioceptive state spaces. Future work should explore the potential of the proposed approach to operate in models using higher dimensional image inputs. It should noted that control with adaptive neural models may lack the stability and safety guarantees associated with conventional dynamics modelling. Moreover, although SMCD is fast, and converges rapidly, doing so does require an energy expensive GPU. While all the models considered in this work can be fit small size embedded GPU, users of the neural adaptation schemes proposed here should carefully consider their model size to determine whether sampling or pro-ppropagttion is more computationally efficient, given the relative efficacy in the multi-task training setting. Given the nature of representations learned by neural networks, this is likely to be task dependent. However, shown above, model adaptation using SMCD can identify and adjust to contexts and unlock neural representations highly suited to model-based control in settings where gradient based adaptation schemes can result in catastrophic forgetting.
7 Conclusion
This work proposed a simple neural network adaption strategy, relying on sequential Monte Carlo adaption of the masks in a dropout regularisation network layer. These masks offer level of inter- pretability, capturing both local and some global contextual information, as illustrated in a real world human behavioural modelling application. Results show that when paired with multi-task training, this simple adaptation strategy outperforms more computationally complex meta-learning schemes and overcomes some of the catastrophic forgetting problems associated with gradient descent in online adaptation settings.
8

---

Acknowledgments
We thank the reviewers for their useful comments and valuable recommendations.
References
[1] A. Ioannou and J. Sun, Robust adaptive control, volume 1. PTR Prentice-Hall Upper Saddle River, NJ, 1996.
[2] N. Srivastava, G. Hinton, A. Krizhevsky, et al. Dropout: simple way to prevent neural networks from overfitting, JMLR, 15(1):1929-1958, 2014.
[3] Y. Gal and Z. Ghahramani. Dropout as a Bayesian approximation: Representing model uncer- tainty in deep learning. ICML, volume 48, pages 1050-1059, 2016.
[4] E. Sontag. Observers and Dynamic Feedback, pages 315-345, Springer New York, New York, NY, 1998.
[5] M. Burke. Path-following control of a velocity constrained tracked vehicle incorporating adap- tive slip estimation. In ICRA, pages 97-102, 2012.
[6] Wu, I. Yildirim, J. J. Lim, et al. Galileo: Perceiving physical object properties by integrating physics engine with deep learning. In NIPS, volume 28, page 127-135, 2015.
[7] M. Asenov, M. Burke. D. Angelov, al. Vid2param: Modeling of dynamics parameters from
video. IEEE R-AL, 5(2):414-44 4214 2019.
[8] M. Jaques, M. Burke, and T. M. Hospedales. thysiis-as-invvese--grapiii Unsupervised phys- ical parameter estimation from video. In ICLR, 2020.
[9] Ramos, R. Possas, and D. Fox. Bayessim: Adaptive domain randomization via probabilistic inference for robotics simulators. In RSS, FreiburgimBreisgau, Germany, June 2019.
[10] L. Brunke, M. Greeff, A. W. Hall, et al. Safe learning in robotics: From learning-basee control safe reinforcement learning. Annu. rev. control robot. auton, 5, 2021.
[11] C. Finn, P. Abbeel, and S. Levine. agdel-agnostic meta-leaming for fast adaptation of deep networks. In ICML, pages 1 1126-1135. PMLR, 2017.
[12] A. Nichol, J. Achiam, and J. Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv: 08031 2018
[13] Snell, Swersky, and R. Zemel. Prototypical networks for few-shot learning. NIPS, 30, 2017.
[14] F. Sung, Y. Yang. L. Zhang, et al. Learning to compare: Relation network for few-shot learning. In CVPR, pages 1199-1208, 2018
[15] S. Belkhale, R. Li, G. Kahn, et al. Model-based meta-reinforcemen learning for flight with suspended payloads. JEEE R-AL, 6(2):1471-1477) 2021.
[16] C. Laidlaw and A. Dragan. The boltzmann policy distribution: Accounting for systematic suboptimality in human models. ICRL., 2022.
[17] S. Reed, K. Zolna, E. Parisotto, et al. A generalist agent. arXiv preprint arXiv:220.006175. 2022.
[18] A. Zhmoginov, M. Sandler, and M. Vladymyrov. Hypertransformer: Model generation for supervised and semi-supervised few-shot learning, arXiv preprint arXiv:2001.04182, 2022.
[19] A. Gupta, L. Fan, S. Ganguli, and L. Fei-Fei. Metamorph: Learning universal controllers with transformers. arXiv preprint arXiv:2203. 1191,, 2022.
[20] Kirkpatrick, R. Pascanu, N. Rabinowitz, et Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, (14113)335113524 2017.

---

[21] E. Grant, C. Finn, S. Levine, et al. Recasting crudient-basedd meta-learning as hierarchical Bayes, arXiv preprint 1.1.:018010009300 2018.
[22] J. Yoon, T. Kim, O. Dia, et al. Bayesian model-agnostic meta-learning. NIPS, 31, 2018.
[23] O. Miglino, H. H. Lund, and S. Nolfi. Evolving mobile robots in simulated and real environ- ments. Artificial life, 2(4):417-434, 1995.
[24] K. O, Stanley and R. Mikkulainen. Evolving Neural Networks through Augmenting Topolo- gies. Evolutionary Computation, 10(2):99-227, 06 2002.
[25] J. Schmidhuber, D. Wierstra, M. Gagliolo, and F. Gomez. Training Recurrent Networks by Evolino. Neural Computation, 19(3):75--77,, 03 2007.
[26] Salimans, Ho, Chen, et al. Evolution strategies as a scalable alternative reinforcement learning. arXiv preprint arXiv: 7703 2017.
[27] P.-T. De Boer, D. P. Kroese, S. Mannor, and R. Y. Rubinstein. A tutorial the cross-entropy method. Annals of operations research, 134(1):19-05 2005.
[28] M. Kobilarov. Cross-entropy motion planning. I/RR,
2012.
[29] Imai, Kawai, and H. Nobuhara. Stepwise pathnet: a layer-by-layer knowledge-ssllccion based transfer learning algorithm. Scientific Reports, 10(1):1-14, 2020.
[30] S. Anwar, K. Hwang, and W. Sung. Structured pruning of deep convolutional neural networks. ACM Journal on Emerging Technologies in Computing Systems (JETC), 13(3):1-88, 2017.
[31] C. L. Baker, R. Saxe, and J. B. Tenenbaum. Action understanding as inverse planning. Cogni- tion, 113(3):22--39, 2009.
[32] M. Chen, S. Nikolaidis, H. Soh, et al. Trust-aware decision making for human-robot collabo- ration: Model learning and planning. ACM THRI, 9(2):1-23, 2020.
[33] Carreno-Medrano, S. L. Smith, and D. Kulix0107:. Joint estimation of expertise and reward preferences from human demonstrations. IEEE TRO, pages 1-18, 2022.
[34] L. Peternel, N. Tsagarakis, D. Caldwell, and Ajoudani. Adaptation of robot physical be- haviour to human fatigue in human-robot co-manipulation. In Humanoids, pages 489-494. IEEE, 2016.
[35] S. Nikolaidis, D. Hsu, and S. Srinivasa. Human-robot mutual adaptation in collaborative tasks: Models and experiments. IJRR, 36(5-7):618-633 2017.
[36] D. Fridovich- Keil, A. Bajesy, J. F. Fisac, et al. Confidence-awee aware motion prediction for real- time collision avoidance. IJRR, 39(2-3):250-26) 2020.
[37] D. Vogt, S. Stepputtis, Grehl, et al. system for learning continuous human robot interac- tions from human-human demonstrations. In ICRA, pages 2882-2889, 2017.
[38] B. Wang, E. Adeli, H.-k. Chiu, al. Imitation learning for human pose prediction. In ICCV, pages 7124-7133, 2019.
[39] M. Carroll, R. Shah, M. K. Ho, et al. On the utility of learning about humans for human-AI coordination. Advances in neural information processing systems, 32, 2019.
[40] C. Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-213. 2019.
[41] A. Doucet, S. Godsill, and C. Andrieu. On sequential Monte Carlo sampling methods for Bayesian filtering. Statistics and computing, 1973):177-208 2000.
[42] D. Crisan and A. Doucet. survey of convergence results on particle filtering methods for practitioners. IEEE TSP, 50(3);736-746, 2002. doi: 10. 10977799877773.
10

---

[43] A. Kong, J. S. Liu, and W. H. Wong. Sequential imputations and bayesian missing data prob- lems. JASA, 89(425):278-255 288, 1994.
[44] R. M. French, Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128-133) 1999.
[45] Corke and J. Haviland. Not your grandmother's toolbox-the robotics toolbox reinvented for python. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 11357-11363. IEEE, 2021.
[46] K. Backman, D. Kulić, and H. Chung. Learning assist drone landings. IEEE R-AL, 6(2): 3192-3199, 2021.
[47] A. Mandelbaum and D. Weinshall. Distance- confidence score for neural network clas- sifiers. arXiv preprint (70.:0:779.09444 2017.
[48] O. Fabius, J. R. van Amersfoort, and P. Kingma, Variational recurrent auto-encoders. In ICLR (Workshop), 2015.


Reviews:
[{"review_id": "ZSgWJmukPoB", "reviewer": "Reviewer_YMaq", "summary": "This paper presents a method to perform system identification with with a world model by leveraging dropout masks.  The general idea is to train a neural network on a randomized version of the system dynamics, sampled from a domain encompassing that which is likely to be seen in the final system.  This network is trained with dropout, using no particular training regime, therefore re-sampling a dropout mask at each forward pass.  Then, during use, there is a 'Mask Adaptation' phase. This is similar in spirit to CEM, and samples a set of random masks, evaluates which ones provide the best next-state estimation, and then re-samples masks around the space of well-performing masks from the previous iteration to fine-tune the mask selection.  This process continues permanently, allowing for not only initial sys-id, but also on-line adaptation to non-stationarity in the system dynamics.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I find the idea very clever, simple, and easy to implement.  It makes intuitive sense that this could work well, and it provides an appealing approach to dealing not only with sys-id but also adaptivity to non-stationarity in the system dynamics.  These are both very real prbolems in physical systems and having a simple, stable, easy-to-implement solution is very appealing.\n\nThe main downside of the paper for me is the very limited experimental results.  I would have really liked to see this approach in use in more controls-oriented tasks, on higher-dimensional systems.  There exist very easy to use environments such as DM Control, Gym or Brax that can be easily perturbed (or even more specific manipulation environments such as Robosuite), and existing model-based controllers such as PDDM, or various MPPI implementations which could easily be extended with this adaptive world-model.  Experiments in these domains would be much more convincing in demonstrating this approach's practical abilities in higher-dimensional control tasks.\n\nI'm also not convinced the formal description of the approach in Eq. 1-3 was particularly heplful, I found it rather more confusing and only understood the mechanism by reading Alg. 1.  I would encourage the authors to perhaps provide a closer comparison to CEM, perhaps the algorithm could be cast as a CEM variant which would also make it easier to contextualize and describe.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think the core idea of the paper is interesting and applicable to a large number of domains.  It feels so intuitive that I wonder if it has been published elsewhere in a different form, but I wasn't able to find a paper suggesting an explicit adaptation phase for dropout-induced variability in the network outputs.  The paper correctly (to the extent of my knowledge) references existing work in dropout approaches to bayesian inference.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "The main issues are described above: the mathematical description of the algorithm is a bit awkward, and could perhaps be better contextualized relative to existing approaches (EM, CEM).  The other main issue is the relatively simple and low-impact experimental section which really misses the opportunity to make a compelling case for robotics control in higher-dimensional systems.  I nevertheless argue for a weak accept because I find the suggested approach very appealing, but will not defend it if other reviewers find the experimental section lacking."}}, {"review_id": "ZObzAIcv0Bo", "reviewer": "Reviewer_PBaL", "summary": "This paper proposes a meta learning algorithm for adaptive robotic control in dynamic and unstructured environments. \n\nThe key idea is to first create a pre-trained neural net with dropout regularization (that was used during training) on multi-task data representing the entire task space. During the online phase, a posterior (or belief) over the dropout mask is being updated and propagated via a MCMC particle filtering scheme.\n\nAt each time step, the average of the propagated particles (representing the mask posterior) could be used to condition the pre-trained net (e.g. retaining most relevant neurons while dropping irrelevant neurons) to fit the current dynamic of a changing environment.\n\nThe experiment is conducted on two settings: (1) a toy setting where the task is to predict next state given current state & input control; and (2) a more realistic setting where the goal is to predict the (latent) landing skill level of human operators given their demonstrated landing sequences. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nThe idea of repurposing the dropout regularization as a mechanism for online adaptation of a pre-trained model is interesting.\nThe paper is well-motivated and has a comprehensive review of literature on model pre-training & adaptation techniques. \nThe mathematical exposition is neatly presented, communicating concisely the key idea via detailed pseudocode.\n\nWeakness:\n\nAlthough the idea is interesting, it has several restrictions: \n\n1. First, given the formulation, it can only be applied to situations where all tasks in the space of interest share the same output space. This is much less flexible than other meta learning approach, such as MAML, which allows training & test tasks to focus on different output sub-spaces.  \n\n2. Second, generating pre-trained net on a master space of output & multi-task data that represents the entire task space is probably much more expensive in both compute and memory footprint. \n\nFor more sophisticated learning scenarios, the joint output space of the tasks could easily have tens of thousands of dimensions even though each task only focus on a subspace of output with less than a handful of dimensions. For such cases, it is arguable that a MAML model (whose output dimension is no more than that of any single task) is more cost-efficient & more suitable as an adaptation base.\n\n3. Next, although the motivation of this paper highlight use cases where model adaptation is necessary to enable adaptive interaction to changes in environment or behavior of human collaborator (line 33), the actual reported experiments do not reflect such cases. \n\nIn fact, both experiments are not about learning and adapting control policies. Instead, one experiment is about predicting next state given acting policies while the other is about predicting intrinsic attributes of human operators given their demonstrated landing sequences. The authors should consider including dedicated experiments towards learning adaptive control policies, as these experiments are more relevant to what is being motivated.\n\n4. Last, the comparison baseline is a bit limited to Reptile and does not include any of the more recent work in meta learning. The authors also mentioned that there was comparison with first-order MAML (line 205) but that does not seem to be included in the reported experiment. Moreover, there should also be discussion & comparison of processing complexity of the baselines. For example, for the proposed method to be effective, what would be the necessary sample size?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes a seemingly interesting idea for meta learning via learning and propagating a posterior on dropout mask for a pre-trained net on a collective set of multi-task data. However, the authors had not considered practical aspects of such idea, e.g. scalability, memory & compute footprint etc. From the technical description, it is likely that the proposed method is significantly more costly than existing methods, at least in term of representation complexity.\n\nIn terms of experiments, there is a lack of comparison with more recent work in meta learning. The experiments also focus largely on toy dataset & in settings that are only remotely relevant to what was used to motivate this paper. Given this, I believe this paper is still a work in progress that unfortunately does not meet the bar for publication yet.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "I have pointed out several issues with this paper above. To summarize:\n\n1. Lack of consideration regarding scalability, memory & compute complexity (in both technical narrative & experiment)\n2. Experiment focuses on toy settings that do not match well with the motivation\n3. Comparison with more recent baselines in meta learning is missing\n\nWhile I do think the idea presented here is interesting, I am not quite convinced of its impact because of the above. I encourage the authors to expand the paper according to the above to complete & strengthen this work."}}, {"review_id": "NExh5mXrd1W", "reviewer": "Reviewer_J9eA", "summary": "The ability of a model to adapt to online data can allow robots to achieve better and more robust performance when deployed in dynamic environments. Current methods, such as gradient-based meta-learning can be difficult to train and implement. In this work, the authors propose a novel adaptation framework for neural networks that is based on Monte Carlo Dropout. At train time, the network is trained using dropout (random masks applied at each forward pass). However, at test time, the authors do inference over dropout masks to find the best performing masks based on observed online data. This is done by particle filtering over masks using the proposed SMCD method. The authors show in two domains that their method leads to better prediction performance than baselines and further analyze the information content of the masks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper addresses an important and long-standing question in the robotics community: how can we adapt models efficiently based on online, observed data? One strength is the generality and capacity of the proposed approach that has potential to scale and model complex phenomena. Additionally, the approach is well-presented and straightforward to implement.\n\nThe main weaknesses of the paper in its current state relate to the “Experimental Results” section. These weaknesses can be divided into clarity and additional comparisons. \n\nIn terms of clarity, I believe addressing the following concerns would help the reader better understand the intuition/strengths behind the proposed approach:\n- For the proposed SMCD algorithm, there are burn-in interactions to seed the adaptation process. Does the non-SMCD baselines observe the same total number of interactions before prediction error is reported?\n- It would be beneficial to have more details about the Reptile baseline algorithms. Specifically, how does the R+S baseline work, where Reptile is combined with SMCD work?\n- In Figures 3 and 5, why does M-N outperform R-N? Without adaptation, what are the differences between these models that would explain this performance gap?\n- In Figure 3, why does R-N outperform R-G? More details on the Reptile baseline might help answer this.\n\nThe authors provide many insightful ablation studies to their proposed algorithm (for example, the no-adaptation baseline, or the oracle model). As the authors develop a novel framework for adaptive model learning, these ablations are important to understand where performance gains come from and I believe the following ablations would provide further insight:\n- As I understand, in the two-link arm domain, the networks do not have access to the link length and are adapting to this unknown task context. A baseline model that has no adaptation, but has access to all relevant state as input would be a useful benchmark as I would expect it to be an upper bound on performance.\n- There are methods in the literature that explicitly capture variability using a latent variable, as opposed to the proposed method which captures task variability using all the weights in the network combined with dropout masks. A comparison with this class of models would also add insight (does the proposed approach lead to better performance by using the entire model to capture task variability instead of a low-dimensional latent-variable?). One approach in this category is that of Saemundsson et al., 2018 (“Meta reinforcement learning with latent variable gaussian processes”).\n\nThe following are minor comments:\n- Is there any intuition why a mean-mask estimator in Equation 6 works well? Would you expect this distribution to be unimodal?\n- I would be interested in reading any additional insights into how the network specializes for different contexts (as shown by the dropout mask analysis) even though random masks are used at training time\n- What are the units in Figure 3?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "In the current state, I recommend a “Weak Reject” for this paper. The authors tackle an interesting problem and propose a novel framework for learning adaptive models. However, additional clarity is required to fully interpret the strengths and weaknesses of the proposed method. I look forward to the authors’ responses as I believe many of the identified concerns are addressable.\n\nPost-Discussion Update:\nI would like to thank the reviewers for taking the time to respond to all the points in my review. I believe that the additional clarity greatly improves the paper and have updated my score to a weak accept. In particular, the explanation provided for why R-N outperforms M-N is very intuitive.The additional experiments also provide a stronger set of baseline methods.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please see the weaknesses section.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "5VgYBC8qXVZ", "reviewer": "Reviewer_iKY7", "summary": "This paper is considering the problem of adapting neural models to changes of task settings. The proposed method is to train a neural model with using dropout technique and then compute the most suitable dropout mask for the particular task at each step. The authors compare their approach with an oracle and another meta-learning algorithm called Reptile. The proposed method does not require any additional training of existing neural model and outperform more complex meta learning (Reptile) algorithm.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The proposed method doesn’t require additional training of underlying neural model.\n* Authors consider two different experiments.\n* Outperforms the existing solution.\n\nWeaknesses:\n* It does not clear from the article how the number of mask N affect the method performance.\n* Authors consider only simulated environments", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This is an interesting paper with potential impact for robot online adaptation, however there is no evidence that this, method would be good enough in complex real world robotic problem.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "* Describe the effect of the parameter N that stand for number of mask to use.\n* Discuss how the method may / may not be used in real-world applications.", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)