Planning for Sample Efficient Imitation Learning
Authors: Zhao-Heng Yin
Weirui Ye
Qifeng Chen
Yang Gao
Venue: NeurIPS 2022 Accept (2022)

Abstract: Imitation learning is a class of promising policy learning algorithms that is free from many practical issues with reinforcement learning, such as the reward design issue and the exploration hardness. However, the current imitation algorithm struggles to achieve both high performance and high in-environment sample efficiency simultaneously. Behavioral Cloning (BC) does not need in-environment interactions, but it suffers from the covariate shift problem which harms its performance. Adversarial Imitation Learning (AIL) turns imitation learning into a distribution matching problem. It can achieve better performance on some tasks but it requires a large number of in-environment interactions. Inspired by the recent success of EfficientZero in RL, we propose EfficientImitate (EI), a planning-based imitation learning method that can achieve high in-environment sample efficiency and performance simultaneously. Our algorithmic contribution in this paper is two-fold. First, we extend AIL into the MCTS-based RL. Second, we show the seemingly incompatible two classes of imitation algorithms (BC and AIL) can be naturally unified under our framework, enjoying the benefits of both. We benchmark our method not only on the state-based DeepMind Control Suite but also on the image version which many previous works find highly challenging. Experimental results show that EI achieves state-of-the-art results in performance and sample efficiency. EI shows over 4x gain in performance in the limited sample setting on state-based and image-based tasks and can solve challenging problems like Humanoid, where previous methods fail with a small amount of interactions. Our code is available at https://github.com/zhaohengyin/EfficientImitate.

Planning for Sample Efficient Imitation Learning
Zhao-Heng Yin'
Weirui Yet
Qifeng Chen
Yang Gao 118
"HKUST TTinnghua University
Shanghaii Qi Zhi Institute
Abstract
Imitation learning is class of promising policy learning algorithms that is free from many practical issues with reinforcement learning, such as the reward design issue and the exploration hardness. However, the current imitation algorithm strug- gles to achieve both high performance and high in-environment sample efficiency simultaneously. Behavioral Cloning (BC) does need in-environment interac- tions, but it suffers from the covariate shift problem which harms its performance. Adversarial Imitation Learning (AIL)) turns imitation learning into distribution matching problem. It can achieve better performance on some tasks but it requires large number of in-environment interactions. Inspired by the recent success of EfficientZero in RL, we propose Efficientlmitate (EI), a planning- basdd imitation learning method that can achieve high in-environment sample efficiency and per- formance simultaneously. Our algorithmic contribution in this paper is two-fold. First, we extend AlL into the MCTS based RL. Second, we show the seemingly incompatible two classes of imitation algorithms (BC and AIL) can be naturally unified under our framework, enjoying the benefits of both. We benchmark our method not only on the state-based DeepMind Control Suite, but also on the image version which many previous works find highly challenging. Experimental results show that El achieves state-of-ther thete----- results in performance and sample efficiency. shows over 4x gain in performance in the limited sample setting on state-based and image-based tasks and can solve challenging problems like Humanoid, where previous methods fail with a small amount of interactions. Our code available at https://gii t/tbbbb.gttttub../. com /ffa:a/ccccaaggi//ffifcfcaaiiitt
Introduction
The real-world sequential decision process in robotics is highly challenging. Robots have to handle high dimensional input such as images, need to solve long horizon problems, some critical timesteps need highly accurate maneuver, and the learning process the real robot has to be sample efficient. Imitation learning is a promising approach to solving those problems, given a small dataset of expert demonstrations. However, current imitation algorithms struggle achieve these goals simultaneously, There are two kinds of popular imitation learning algorithms, Behavior Cloning (BC) and Adversarial Imitation Learning (AIL). BC formulates imitation learning as a supervised learning problem.
needs no in-environment samples, but it suffers from the covariate shift issue [37 often leading
to test time performance degradation. Adversarial Imitation Learning (AIL) [6. casts imitation
learning as distribution matching problem. Though AIL suffers less from the covariate shift problem and can perform better than BC on some domains, it requires an impractical number of online interactions [9 22 and can perform badly image inputs (34). These drawbacks heavily limit its application in fields like robotics, where physical robot time matters. In summary, current imitation
"zhaoheng. baasoheng-yindccee wwwyccinnntnt...mmmmmmm ust. hk, cqffust. .k fyrr200malls tainghua.co edu. en, gaoyangiiis@tssnngua..cc edu. Corresponding author.
en,
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Act
Input: State
Output: Planning Result
Value Estimate
Policy Estimate
Optimize
Encode
P(s)
* (8)
Policy/ Value
Planning
Root
BC Action
Model /AL Reward
AIL Reward
Agent
Expert
Buffer
Data
Replay
Buffer
The system-level overview of Efficientlmitate. The agent (yellow area) takes actions in the environment and stores the data in the replay buffer. The data in the replay buffer and expert buffer are then used to train a model and AlL reward. The planning module then searches for improved policy and value in each state in the replay buffer, based which the policy and value networks are optimized. Right: The planning procedure. We use a continuous version EfficientZero as our planner. We find that MCTS uniquely benefits from BC and can unify BC and AIL. For the expansion of each node, we sample actions from both the current policy (black arrow) and a BC policy (blue arrow). We use AIL reward (yellow cube) to encourage long-term distribution matching. MCTS searches (pink area) for best actions to maximize the cumulative AlL reward and update the estimated value and policy of the root node. The output of the planning procedure the value estimate and the policy estimate of s, and the value and policy networks are optimized to fit them.
Figure 1: Left:
learning algorithms fail achieve high online testing performance and high in-environment sample efficiency at the same time. Further, the two types of imitation algorithms seem to be incompatible since they are based on two completely different training objectives. Previous work finds that is hard to unify them naively 32).
propose planning-base imitation algorithm named Efficientlmitate (EI) that achieves high test performance and high in-environment sample efficiency the same time. Our method extends AIL model- based setting with multi-step losses under the MCTS-based RL framework. Our algorithm also unifies the two types of the previous imitation algorithms (BC and AIL.) naturally, thanks to the planning component of our algorithm. Intuitively, BC gives coarse solution that is correct most of the time but fails to match the expert's 's behavior the long term. On the other hand, AIL knows the goal of the learning, i.e., matching state action distribution, but doesn't it give the solution directly. Our method's 's planning component unifies those two methods and shows a significant performance boost, especially in the harder tasks such as Humanoid. We illustrate the detailed procedure in Figure !
Inspired by the recent success in sample efficient RL, such EfficientZero 48
We validate our idea not only state-based tasks but also on image- based tasks, which relatively few previous algorithms can handle them (34]. El achieves state-of-the-art results in sample efficiency and performance. It shows over 4x gain in performance in the limited sample setting and image-based tasks. On harder tasks such as Humanoid, the gain is even larger. A by-product of this paper is that we extend the recent EfficientZero algorithm to the continuous action space, We open-source the code at /https://giii //git/b...btttt../ com/ http://wwnnngyii...fiiiiiimmmmma future research.
based
to facilitate
Our contributions in this paper are summarized as follows.
We present Efficientlmitate, sample efficient imitation learning algorithm based on MCTS.
We identify that MCTS can benefit from BC by using BC actions during the search, which is crucial for challenging tasks. Efficientllmitate suggests a natural way to unify BC and AIL.
We conduct experiments in both the state and the image domain to evaluate EI. Experimental results show that EI can achieve state-of ohe-the-rt sample efficiency and performance.

---

2 Related Work
2.1 Imitation Learning
Imitation learning (IL) aims to solve sequential decision-making problems with expert demonstration data. It has wide applications in games and robotics 4 Compared with RL, one major benefit of IL that it can avoid the notorious reward design problem. IL can be divided into two branches: BC [ and IRL [T. To solve the covariate shift problem of BC, researchers propose methods like dataset aggregation 37 and noise injectioo [4]]. But these methods either require extra expert queries or exert constraints over the learning process. A recent variant branch of IRI. is the Adversarial Imitation Learning (AIL) [16 O AIL models IL as a state-action distribution matching problem. Many works extend AIL by using better distribution metrics such f-divergence (50) and Wasserstein distance [5] Though these methods can learn better reward functions and somewhat speed up the training, they do not directly focus AIL's sample efficiency problem. Some works have drawn attention to sample efficiency, for example, [22
propose to use off-policy training in AlL reformulates AlL objective in an offline min-max optimization process, but recent work points out that it is an improved form of BC (28). VMAIL (34 uses model-based approach to improve sample efficiency. It collects latent rollout with variational model and reduces online interactions. MoBILE [21] also shows a provably sample efficient model-based imitation approach. Compared with these methods, our method introduces MCTS planning to the off-policy imitation learning and uses it to unify BC and AlL to take advantage of both.
38 51
training to reduce sample complexity. ValueDICE
23
The idea of combining BC and AIL helps to improve the sample efficiency can be traced back to [6 GAIL suggests using BC to initialize the policy network, but [32 finds that this does not work well because the initialized BC knowledge will be corrupted in AIL training. Then, [T9] proposes use an annealed (decaying) BC loss to regularize policy training to solve this problem. But in this work, we find that such a BC regularizer can be harmful to exploration when BC is incorrect. One concurrent work mitigates this issue by using adaptive BC loss and replacing AIL reward with -primal-transport based imitation reward [3] We also notice that AlphaGoo 433 involves BC in their method, but they focus on RL rather than II., and BC is only used to initialize the policy. Different
from these methods, El uses BC actions as candidates in MCTS.
2.2 Sample Efficiency in RL
The sample efficiency problem of imitation learning is closely related that in RL. One line of work finds that the reward signal is not a good data source for representation learning in RL and one reason for sample inefficiency. Then they utilize self-supervised representation learning to accelerate representation learning and improve the sample efficiency. Researchers propose to use contrastive learning 266.. consistency-bant learning
41 49 48), or pretrained representation [42
for this purpose. Some works also explore the possibility of applying self-supervised representation learning to imitation learning
Another line of work focuses on RL with learned model, which is promising for sample efficient learning [8] 10-12 [8 27 29 48 [4] These approaches usually imagine additional rollouts with the learned model or use it as a more compact environment representation for RL. Besides, some
works also find that data augmentation can effectively improve sample efficiency 25
45
EI also
benefits from these approaches. It includes a model and applies representation learning to boost sample
efficiency. application, people also consider to augment RL with the demonstration 15 35
30
20
to improve the sample efficiency. This can be viewed as combination of RI. and imitation learning and favor RL. on real robots. We believe that our method can also be extended to this setting.
3 Background
3.1 Setting
We formalize the sequential decising making problem as Markov Decision Process M (S, A, Here, S is the state space, A is the action space, R. is the reward function, and T is the transition dynamics. The agent's 's state at timestep t is E S. The agent takes action and
3

---

receives reward r, = R(,,, ae). Its state at timestep t + 1 is then S(+1 T(ss, The objective of the agent is to maximize the return Ei=o 7"T,, where 7 is discount factor.
In the imitation learning problem studied here, the agent has no access to the reward function and transition dynamics T. It is provided with fixed expert demonstration dataset D = {Tr). Here, each T; = (of ao of, ff ...57, af) is an expert trajectory that can achieve high performance in M. The agent can not solicit extra expert demonstrations but can interact with the MDP, observing new states and actions, but not rewards. this work, we define (in-environment) sample efficiency as the number of online interactions during training. We expect the agent to achieve high performance within a fixed online sample budget.
3.2
BC and AIL
BC considers imitation learning as a supervised learning problem. It trains a policy network π to minimize the following loss function:
L Le ff .a log π (a, ls,"
(I)
AIL treats imitation learning as distribution matching problem. One typical AIL algorithm is GAIL. It trains a discriminator D to distinguish the agent generated state-action tuple (8i, 0;) from those (sf a.) in the demonstration dataset by minimizing
L = -E
[o ,a)) [log( ((s," aa + log(1 - D(s;, a:))]
where is the state-action distribution induced by the agent. Meanwhile, it trains the agent to maximize the return with respect to the adversarial reward r; log(1 D(s,, at)) using any on-policy RL algorithm.
3.3 MuZero and its Extensions
Our planning method is based MuZero [39] and its extensions. MuZero learns environment model for MCTS. The model consists of an encoder network f. a dynamics network g, and a reward network R. It operates on abstract states [48.. Concretely, it gets the abstract state h2 of the current state 8t by h4 = (. It can then predicts the future abstract states recursively by h4+1 g(e,, az), and the rewards by R(h,,a,) az). Besides the model, MuZero also contains policy network and value network. The policy network provides a prior over the actions at each node, and the value network calculates the expected return of the node. MuZero uses the model, the policy network, and the value network to search for improved policy and value for each state with MCTS. We refer the readers to the original MuZero paper for details.
Sampled MuZero Sampled MuZero [17] extends MuZero from the discrete action domain to the continuous action domain, which is of our interest in this paper. At each node expand, Sampled MuZero samples actions K { from current policy (a]ss). During the search, selects action a" from the sampled actions that maximize the probabilistic upper confidence bound
22, 1 1 N(s,a)
a* arg max Q(+, a) + (a)) aE(a;)
where [[ass) = Σ, I a.) Q(s, is the current Q-estimation of the pair (s, a). (Ns,, a) denotes the times that this pair is visited in MCTS. c(s) is a weighting coeffcient. During policy optimization, MuZero minimizes the Kullback- the MCTS statistics TMCTS at the root node D (1MCST) IT).
divergence between the current policy π and
EfficientZero We also apply EfficientZero 48 in this paper. Efficient/ero improves the sample efficiency of MuZero by using self-supervised representation learning method to regularize the hidden representation. It uses a SimSiam-style structure 3 to enforce the similarity between the predicted future representation and the real future representation.
4 Efficientlmitate
In this section, we present our Efficientlmitate algorithm. We first present an MCTS-based approach to solving the AIL problem in Section 4.1 Then we show simple yet effective method unify BC

---

Mltnn
SE
hg
hi-1
HEin
TC
Mgc
Pigc
Braay
Multi-step Discriminator Lass
Martrting Icg-iaattood
Multi-step BC Loss
Figure 2: Computation flow loss functions. Left:
Multi-step Discriminator Loss. We not distinguish between the calculation for expert and agent here, and use superscript (F) to indicate that the computation applies to both. Right: Multi-step BC Loss. applies to the expert sequences.
and AIL with MCTS in Section 4.2 We briefly discuss the implementation Scction 4.3 and the full details can be found in the Appendix.
4.1 Extending AIL to MCTS- RL
Traditionally, the adversarial imitation learning (AIL) algorithm trains discriminator D between the policy samples and the expert samples and uses some form of D, such as - -log(1 DD), as the reward function. Then some model-free RL algorithms are used to maximize the cumulative reward. In MCTS-based RI. algorithms, such as MuZero (39 and EfficientZero [8], the reward function is used in the value target computation and the MCTS search. The use in value target computation is similar to prior model-free RL algorithms, where the value target is computed with n-step value bootstrapping on the actual observations. However, during the MCTS search, the rewards are computed on the abstract state obtained by running the forward dynamics function hx+1 = ghe, az) multiple times. If we were training the discriminator only on actual observations of the expert and the policy rollouts, the discriminator might not generalize well to abstract states outputted by the forward dynamics functions. Therefore, we train the discriminator with the model- boseel-bsed rollout. Specifically, we sample P. F. sequence (st, Qc-1, ..., (s+n) ) in replay buffer B and expert sequence (s az a.' an+n) in demonstration dataset D and minimizes following multi-step discriminator loss function:
CD -PEs,, Bo D D.
log ( ( + log(1 D(h D(hu+e ap+)))
Here, he4s (and hy=+) terms are produced by the forward dynamics in EfficientZero (Figure 2 We use the GAIL transition reward R(h, a) - llog(1 D(h, a)), and then the MCTS planner searches for action that can maximize cumulative GAIL reward to guarantee long-term distribution matching. Note that V-MAIL also propose a similar discriminator training technique, but under the Dreamer model.
Besides, since the discriminator's input is based on the representation rather than raw input, the discriminator should be trained with the encoder jointly. However, this can lead to a highly non- stationary reward during the bootstrap value calculation. To mitigate this issue, we also propose to use target discriminator for bootstrap value calculation. This can make the training more stable.
Though we use the GAIL reward here, one can also use other kinds of AIL and IRL. reward functions proposed in recent research. Using the GAIL reward can already achieve satisfactory performance in our experiments. When the real reward presents, one may also combine this into planning [20]. This may favor application scenarios where handcrafting a reward function is not hard. We do not study this case here and leave it to future work.
4.2 Unifying BC and AIL in MCTS
As discussed in related work, researchers realize that using BC can improve AIL's sample efficiency by providing a good initial BC policy or constraining the policy to BC. However, these existing good enough in practice. The main pitfall in these methods is that BC knowledge in the policy network is destined to be forgotten if the policy network trained with the AIL objective, and then BC will longer be helpful [32].
solutions
We observe that MCTS can naturally unify the BC and AIL. methods, enjoying the benefit of both and being free from this pitfall. We propose to plug BC actions into MCTS as candidates each
5

---

node and use a planning process to search for an improved solution. This time, the BC actions are consistently considered throughout the entire training procedure without being forgotten. Concretely, we train BC policy BBC and use a mixture policy म for the sampling at each node in MCTS:
= BC + (1 a.)..
is mixture factor, which is fixed during training and is the current policy. We use 0.25 this paper. This ensures that small fraction action samples are from the BC policy. During planning, the BC actions are evaluated and will be frequently visited and selected output if they can lead to long-term distribution matching. This can then reduce the effort of finding good expert-like actions from scratch desired. Moreover, another unique advantage of this procedure is that it does not fully trust BC like [19), which forces the output the policy network to be close to BC. When BC is wrong due to covariate shifts insufficient demonstrations, it can neglect these BC actions and allow the policy to search for better actions. This ensures that BC does not hurt training. However, arising question is whether this approach can be applied to other
due to the conceptual simplicity, model- based methods. Here, we take Dreamer [ as an example. Though Dreamer builds a model of the environment, it only words, the model is not used to evaluate whether a specific BC action is good or not in the long term, so idea can not be applied directly Dreamer. From this example, we that the core of idea is to leverage the power of planning, only with which the long-term outcomes of certain (BC) actions can be calculated.
the model roll-out the policy for policy optimization. In other
For the training of TBC, we minimize the following multi-step BC objective (Figure 2):
Iog(TTo (b++)))
BB (
(6)
This is to avoid distributional shifts during multi-step prediction in MCTS. For the training of the policy, we still minimize D L (TMCTS |T).
Note that the BC design proposed here is not coupled with AIL. It can go beyond imitation learning and be applied in other robot learning settings, such RL with demonstration 35
4.3 Implementation
We first implement a continuous version EfficientZero for planning, and the details can be found in the Appendix. The BC policy network is simply a duplicate of the policy network. The discriminator and BC policy networks share the same encoder network with the policy network and value network. The overall loss function for optimization is
LEZ + A&ED + AscLBc.
LEZ is EfficientZer'ss loss function (excluding reward loss). All the networks are trained jointly to minimize this loss function 7 We use the Reanalyze algorithm 40 48 for offline training, and we require that all the samples should be reanalyzed.
Figure 3: Part of the tasks used in our experiments. From left to right: Reacher, Finger Spin, Cheetah Run, Walker Walk, Hopper Hop, Humanoid Walk.
Experiments
In this section, we evaluate the sample efficiency of the proposed method. We measure the sample efficiency by evaluating the performance of an algorithm a small number of online samples. We also analyze the effect of the BC actions and planning.

---

Table 1: Evaluation result on the state-based DeepMind Control Suite. We use the average score on three random seeds. Our method can achieve the Mate-of-theear result compared with the baselines.
Task
Cartpole
Ball
Reacher
Finger
Cheetah
Walker
Hopper
Humanoid
Budget
10k
10k
50k
50k
50k
50k
50k
500k
BC
0.59
0.44
0.83
0.76
0.58
0.16
0.03
0.11
DAC
0.13
0.18
0.22
0.53
0.33
0.26
0.00
0.01
+0.12
=0.01
+0.02
+0.05
#0.04
=0.04
=0.00
=0.00
ValueDICE
0.21
0,23
0.15
0.04
0.50
0.54
0.03
0.00
+0.01
+0.01
+0.01
+0.01
20.08
20.09
=0.00
=0.00
SQIL
0.23
0.27
0.21
0.02
0.05
0.11
0.24
0.06
20002
+0.00
20.03
Ours
0.98
0.99
0.90
0.99
0.96
1.03
0.92
0.74
#0.01
#0.01
#0.04
10.00
:0.02
10.01
10.02
10.04
Table 2: Evaluation result on the image-based DeepMind Control Suite. We use the average score on three random seeds. Our method can achieve oatte-of-the-are results compared with the baselines.
Task
Cartpole
Ball
Finger
Cheetah
Reacher
Walker
Hopper
Budget
50k
50k
50k
50k
100k
100k
200k
BC
0.30
0.32
0.14
0.37
0.26
0.15
0.02
DAC
0,08
0.26
0.00
0.04
0.25
0.10
0.01
#0.01
+0.02
+0.00
#0.01
20.05
#0.02
#0.00
ValueDICE
0,18
0.27
0.01
0.06
0.15
0.08
0.00
10.02
10.02
10.00
10.01
10.02
10.00
10.00
SQIL
0.26
0.77
0.00
0.06
0.36
0.32
0.04
VMAIL
0.57
0.61
0.06
0.13
0.34
0.24
0.07
+0.03
#0.11
+0.03
+0.04
+0.02
20.07
20.04
Ours
0.94
0.93
1.00
0.92
0.86
0.98
0.70
+0.02
#0.01
#0.01
10.01
10.06
#0.01
#0.01
5.1 Setup
We use the DeepMind Control Suite 44 for evaluation. We use the following tasks: Cartpole Swingup, Reacher Easy, Ball-in-cup Catch, Finger Spin, Cheetah Run, Walker Walk, Hopper Hop. and Humanoid Walk. We conduct both state-based and image-based experiments. Note that many previous imitation learning works use the OpenAl Gym 2 version of these tasks for evaluation. We find that the DMControl version used here brings extra challenges by using more challenging initial states. Take the Walker task as an example; the initial state in OpenAl Gym standing. However, in DMControl, the agent's initial state is lying the ground, and the agent should also learn stand up first from very limited data. For the state-based experiments, we allow 10k-50k online steps in the environment based on the difficulty of each task. Since learning robust and meaningful visual representation requires more data for image-based experiments, we allow 50k- 10kk online steps. Detailed setup will be shown in the result. We train SAC [9] policies to collect expert demonstrations for imitation learning. The expert demonstrations are not subsampled. We use demonstrations in the base- experiment, except for Reacher and Humanoid, where use 20 demonstrations. We use 20 demonstrations in the image-basse experiments.
5.2 Baselines
We present several baselines of sample efficient imitation learning. (1) DAC DAC [22] is an adversarial off-policy imitation learning method. matches the distribution of the replay buffer and that of the expert demonstration dataset using the TD3 [7] algorithm. (2) SQIL SQIL [36] is non-adversarial off- off-co imitation learning method. It labels all the expert transitions with reward 1 and non-ex non-expert transitions with reward O. Then it trains SAC policy over these relabeled data.
SQIL a regularized form BC. (3) Valu ValueDICE 23 considers imitation learning as distribution matching problem and solves with min-max optimization process.
(4) VMAIL VMAII. 34 a model-based visual imitation learning method. learns a variational model for simulating on-policy rollouts. We only evaluate VMAIL the image-based domain, they did in

---

Finger
Hopper
0.3
0.0
0.4
0.2
SOL
Cuas
505
1006
2558
2003
500c
1000
150K
2004
500
100R
2508
200R
500
10oc
LS0C
200C
Sigps
Staza
Stpps
Staza
Figure 4: The performance the state-based tasks. The results averaged three seeds. The shaded area displays the range one standard deviation.
Hepper
1.0
0.3
0.4
0.3
0.2
BiCC
0.2
0.2
SOL
0.0
OvD
50x
100%
B00K
200K
100C
150K
200c
50x
100%
200K
200k
100K
150K
200C
2rips
Stace
Srips
Stace
Figure The performance curve on the image-based tasks. The results are averaged over three seeds. The shaded area displays the range of one standard deviation.
the original paper. Besides the online imitation learning baselines, we also include BC as an offline baseline.
5.3 Results
State-based experiments Table I shows the state-based experiments' evaluation results within the given budget. We also plot the performance curve of four challenging tasks in Figure [ The performance is normalized to 0.0 and 1.0 with respect to the performance of the random agent and the expert. We find that our proposed method can outperform all the baselines by large margin. Except for BC, these baseline methods could hardly learn meaningful behaviors using a limited online budget. We find that DAC strong baseline. performance can grow the expert in 200k samples in most tasks except Hopper, where it will eventually get stuck. Our method is much more sample efficient than the best of these baseline methods. our method only requires about 20k steps to reach near-expert performance, equivalent to 80 online trajectories (around 0.5 hours real). This result notable for the robotics community. shows that online imitation learning is possible with only a handful of trials, and applying it directly a real locomotion robot possible.
some tasks like Walker Walk and Cheetah Run,
Image-based experiments So far, image-based tasks are still challenging for adversarial imitation learning algorithms, and the evaluation of most of the prior AIL works carried out in the state-based tasks. Table 2 shows the evaluation result within the given budget in the magge-based experiments (see Figure for curves). Our method can also learn expert behavior given a slightly larger budget. Still, most of the baselines fail to match experts' behavior using the given budget. We notice an inherent difficulty learning robust visual representation for adversarial training the limited data set in image-based tasks. Discriminators can judge whether behavior is expert-like using various possible features in this case. Solving this open problem is out the scope of this paper. the presence of such difficulty, El can still achieve good sample efficiency in most of the tasks.

---

Humanood
Hopppi
Walker
Finger
1.-
3.0
3.2
6.4
0.9
0.5
6.1
6.2
6.2
3.2
5.0
00
0.3
3.0
Oum
50eo5
Figure 6: The performance curves of our method with and without BC actions at each expansion. The plots are sorted according the difficulty of the corresponding task. The leftmost one the most difficult task, Humanoid. The results are averaged over three seeds. BC actions have great impact on the sample efficiency and performance.
5.4 Analysis
Effect of BC We earry out ablation analysis on the BC to see whether it helps in our method. We set a = 0 to remove all the BC actions and see how our method's performance and sample efficiency will change. The result is shown Figure We find that the performance of our method degrades after we remove all the BC actions in MCTS. The effect is task-specific, and we find BC is more helpful in those challenging tasks. For example, in tasks that are high-dimensional or have a hard exploration process like Humanoid and Hopper, removing BC actions will make the learning process stuck in local minima. At that local minima, the agent struggles to find the correct action that matches the expert's behavior. Although removing BC actions does not trap the agent in local minima in some relatively simpler tasks like Cheetah and Walker, it slows down the training. It doubles the number of online interactions reach expert performance. This result confirms that using BC actions can indeed provide good solution to the distribution matching problem, which can help to speed up learning and improve performance. We also notice that even when remove the BC actions, the method is still able to outperform the previous baselines; this suggests that planning with AIL alone is also powerful.
Humanoid
Other Ways to use BC We then study another two variants of using BC: (1). BC-Ann. This variant does not use BC actions in MCTS but exerts an annealing BC loss to the policy network like [19. (2). BC-Rep. This variant does not BC actions in MCTS but still uses BC loss to regularize the representation. We test these variants the Humanoid Walk (Figure 77). We find that these variants do not lead to an essential improvement. For BC-Ann, harms the performance in the early stage (be- fore 100k) since the BC regularization will constrain the agent's policy near the BC policy, which contains an error and hinders learning. The agent only starts to acquire meaningful behav- ior after the regularization decays. but at that time, BC does not help much and can not lead to improvement. Compared with BC-Ann, BC-Reg is more helpful here. This is possibly because BC-Reg makes the encoder focus on more useful features. However, BC-Reg still gets stuck in a local minimum. This result suggests that using BC actions directly for exploration can be essential for improving AIL.. Using BC simply as a regularizer may not be the ideal approach though can be useful sometimes.
0.5
Dura fivis SCI
Cart frn
Cura Tep.)
Figure 7: Results of different ways of using BC.
Ablation of Planning In this part, study to what extent planning can help learn and whether insufficient search in MCTS leads to inferior performance. We study K, the number of sampled actions at each node, and N, the number of simulations. The default value of K and N in the previous experiments are 16 and 50. We sweep K E [4, 8, 16, 24} and N E (5. 10, 25, 50} to evaluate their effects. We collect the result on the state-based Cheetah, Walker, and Hopper task and report the averaged relative performance change at the given budget used in previous experiments (see Table 3). The general trend that larger K and N lead to better imitation results. We find that varying K only affects the performance a little, and K 4 can also work well. Compared with K, N has larger

---

impact. When the number of simulations becomes small, the performance drops significantly. This result also explains why we can achieve large improvement over DAC even without BC.
Table 3: Ablation of planning. We report the relative change of performance the given budget.
Param K
K
16
24
5
N
10
25
50
Result -8.4%
-3.2%
0.0%
1.1%
-27.3%
-15.5%
-4.6%
0.0%
6
Discussion
In this paper, we presented Efficientlmitate, an MCTS-based imitation learning algorithm. We extended AIL to model-based setting and solved it with MCTS in sample-efficient way. Morcover, we proposed method to unify BC and AlL MCTS, enjoying the benefits of both. Experimental results in state-based and image-based tasks showed that Efficientlmitate can achieve state-of-the-art sample efficiency and performance.
Limitations One limitation of this work is that the computation process of MCTS is more expensive compared with that the model-free methods, though this is common issue of model-based methods. One possible approach mitigate this issue be using better MCTS acceleration methods [47). Besides, this paper, we did not study the long horizon problem with multiple objects, which is common
robotic manipulation. However, this requires the model predict the interaction with multiple objects, which is still a challenging open problem in the learning community (33) and orthogonal our contribution. We believe that our framework can be combined with the works in this field to handle this challenge.
Future Work There are many problems study along our direction. First, since we only use the vanilla AlL algorithm here, it is interesting to see if using more advanced algorithms such as pppimal-ranssorrtboee -baeed learning (5] will make our algorithm more powerful. Second, due to the modularity of our method, one can try to extend Efficientlmitate to more general settings like RL with demonstration, which will also favor the application scenarios. Third, in this work we consider an online learning setting, one possible future direction is to study the use of EfficientImitate on the existing offline interaction dataset to further reduce the dependence on in-environment samples.
In conclusion, we believe that this work shows promising direction and opens up new possibilities for model-based methods in robot learning.
Acknowledgments and Disclosure of Funding
This work is supported by the Ministry of Science and Technology of the People's Republic of China, the 2030 Innovation Megaprojects "Program on New Generation Artificial Intelligence" (Grant No. 2021AAA0150000; This work is also supported by grant from the Guoqiang Institute, Tsinghua University.
References
[1] M. Bain and C. Sammut. framework for behavioural cloning. In Machine Intelligence 15, pages 103-129, 1995.
[2] G. Brockman, V. Cheung, L. Pettersson, Schneider, J. Schulman, Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv: 1606.01540, 2016.
[3] X. Chen and K. He. Exploring simple siamese representation learning. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[4] X. Chen, S. Toyer, C. Wild, S. Emmons, I. Fischer, K.-H. Lee, N. Alex, S. H. Wang, P. Luo, Russell, et al. empirical investigation of representation learning for imitation. In Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
10

---

[5] R. Dadashi, L. Hussenot, M. Geist, and O. Pietquin. Primal wasserstein imitation learning, In International Conference on Learning Representations, 2021.
[6] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning. In International Conference Learning Representations. 2018.
[7] Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, 2018.
[8] C. Gelada, S. Kumar, J. Buckman, O. Nachum, and M. G. Bellemare. Deepmdp: Learning continuous latent space models for representation learning. In International Conference on Machine Learning, 2019.
[9] Haarnoja, A. Zhou, P. Abbeel, and Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning. 2018
[10] D. Hafner, T. Lillicrap, Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International Conference Machine Learning, 2019.
[11] D. Hafner, T. P. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.
[12] D. Hafner, T. P. Lillicrap, M. Norouzi, and J. Ba. Mastering atari with discrete world models.
In International Conference on Learning Representations, 2021.
[13] S. Haldar, V. Mathur, D. Yarats, and L. Pinto. Watch and match: Supercharging imitation with regularized optimal transport. In Conference on Robot Learning. 2022.
[14] N. Hansen, X. Wang, and H. Su. Temporal difference learning for model predictive control. In International Conference Machine Learning, 2022.
[15] T. Hester, M. Vecerik, O. Pictquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, Quan, A. Sendonaris, I. Osband, et al. Deep q-learning from demonstrations. In AAAI Conference on Artificial Intelligence, 2018.
[16] J. Ho and S. Ermon. Generative adversarial imitation learning. In Neural Information Processing Systems, 2016.
[17] Hubert, Schrittwieser, Antonoglou, M. Barekatain, S. Schmitt, and D. Silver. Learning and planning in complex action spaces. In International Conference on Machine Learning. 2021.
[18] M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization. In Neural Information Processing Systems, 2019.
[19] R. Jena, C. Liu, and K. Sycara. Augmenting GAIL with BC for sample efficient imitation learning. In Conference on Robot Learning, 2020.
[20] B. Kang, Z. Jie, and J. Feng. Policy optimization with demonstrations. In International Conference on Muchine Learning, 2018
[21] R. Kidambi, J. Chang, and W. Sun. Mobile: Model-based imitation learning from observation alone. In Neural Information Processing Systems, 2021.
[22] Kostrikov, K. Agrawal, D. Dwibedi, S. Levine, and J. Tompson. Discriminator-actoon critic: Addressing sample inefficiency and reward bias in adversarial imitation learning, In International Conference on Learning Representations, 2019.
[23] I. Kostrikov, O. Nachum, and J.Tompson. Imitation learning via off-policy distribution matching. In International Conference on Learning Representations, 2020.
[24] M. Laskey, J. Lee, R. Fox, A. Dragan, and K. Goldberg. Dart: Noise injection for robust imitation learning. In Conference on Robot Learning, 2017.
11

---

[25] M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data. In Neural Information Processing Systems, 2020.
[26] M. Laskin, A. Srinivas, and P. Abbeel. CURL: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, 2020.
[27] A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine. Stochastic latent actor-critic: Deep
reinforcement learning with latent variable model. Neural Information Processing Systems, 2020.
[28] Z. Li, T. Xu, Y. Yu, and Z.-Q. Luo. Rethinking valuedice: Does it really improve performance? In International Conference on Learning Representations, 2022.
[29] Y. Mu, Y. Zhuang, B. Wang, G. Zhu, W. Liu, Chen, P. Luo, S. Li, C. Zhanng and J. Hao. Model- based reinforcement learning via imagination with derived memory. In Neural Information Processing Systems, 2021.
[30] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In IEEE International Conference on Robotics and Automation, 2018.
[31] A. Y. Ng, S. J. Russell, et al. Algorithms for inverse reinforcement learning. In International Conference Machine Learning, 2000.
[32] M. Orsini, A. Raichuk, L. Hussenot, D. Vincent, R. Dadashi, S. Girgin, M. Geist, O. Bachem, O. Pietquin, and M. Andrychowicz. What matters for adversarial imitation learning? In Neural Information Processing Systems, 2021.
[33] H. Qi, X. Wang. D. Pathak, Y. Ma, and J. Malik. Learning long-term visual dynamics with region proposal interaction networks. In International Conference on Learning Representations, 2021.
[34] R. Rafailov, T. Yu, A. Rajeswaran, and C. Finn. Visual adversarial imitation learning using variational models. In Neural Information Processing Systems, 2021.
[35] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Robotics: Science and Systems, 2018.
[36] S. Reddy, A. D. Dragan, and S. Levine. SQIL: imitation learning via reinforcement learning with sparse rewards. In International Conference on Learning Representations, 2020.
[37] S. Ross and D. Bagnell. Efficient reductions for imitation learning. In International Conference on Artificial Intelligence and Statisties, 2010.
[38] F. Sasaki, T. Yohira, and A. Kawaguchi. Sample efficient imitation learning for continuous control. In International Conference on Learning Representations, 2018
[39] J. Schritwiessrr, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis. T. Graepel, al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(783)):604-60) 609, 2020.
[40] J. Schritwieser, T. Hubert, A. Mandhane, M. Barekatain, I. Antonoglou, and D. Silver. Online and offline reinforcement learning by planning with a learned model. In Neural Information Processing Systems, 2021.
[41] M. Schwarzer, A. Anand, R. Goel, R. D. Hjelm, A. C. Courville, and P. Bachman. Data-efficient reinforcement learning with self predictvd representations. In International Conference Learning Representations, 2021.
[42] M. Schwarzer, N. Rajkumar, M. Noukhovitch, A. Anand, L. Charlin, R. D. Hjelm, P. Bachman, and A. C. Courville, Pretraining representations for data-efficient reinforcement learning. In Neural Information Processing Systems, 2021.
12

---

[43] D. Silver. A. Huang, J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam. M. Lanetot, al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016.
[44] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 0066000660.... 2018.
[45] D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Mastering visual continuous control: Im- proved daga-au reinforcement learning. In International Conference Learning Representations, 2022.
[46] D. Yarats, I. Kostrikov, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. International Conference on Learning Representations. 2020.
[47] W. Ye, P. Abbeel, and Y. Gao. Spending thinking time wisely: Accelerating mcts with virtual expansions. In Neural Information Processing Systems, 2022.
[48] W. Ye, S. Liu, T. Kurutach, P. Abbeel, and Y. Gao. Mastering atari games with limited data. In Neural Information Processing Systems, 2021.
[49] T. Yu, C. Lan, W. Zeng, M. Feng, Z. Zhang, and Z. Chen. Playvirtual: Augmenting cycle- consistent virtual trajectories for reinforcement learning. In Neural Information Processing Systems, 2021.
[50] X. Zhang, Y. Li, Z. Zhang, and Z.-L. Zhang. f-gail: Learning f-divergence for generative adversarial imitation learning. In Neural Information Processing Systems, 2020.
[51] Z. Zhu, K. Lin, B. Dai, and J. Zhou. Off- Dff-policy imitation learning from observations. In Neural Information Processing Systems, 2020.
13


Reviews:
[{"review_id": "kOqy2Fzo9Ds", "reviewer": "Reviewer_bqbS", "summary": "The paper proposes a method to improve sample efficiency of the model-based imitation learning. The method consists in combining three components:\n* Efficient-Zero algorithm\n* GAIL-style imitation learning approach which will learn a discriminator and provide it to Efficient-Zero algorithm to specify the reward\n* BC algorithm to learn a policy which will regularize the acting distribution of Efficient-Zero\n\nThe method demonstrates improved data efficiency (in terms of online rollouts) compared to the baseline, and is conceptually simple.", "questions": "* Why is it important to use EfficientZero? Can we use any other model-based method?\n* Can it be that the main benefit comes from using EfficientZero as the algorithm? What if we use this algorithm in other model-based imitation learning baselines?\n* Why did the author choose a specific number of trajectories? How does the method & baselines behave when different number of trajectories are considered?\n* Can you provide more intuition / evidence on why each component of the algorithm is important ? (I.e., GAIL, EfficientZero, BC)\n", "limitations": "I think the authors adequately presented the limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenghts:\n* A method which combines different tricks and achieves a competitive performance in a model-based imitation learning setting\n* Conceptual simplicity of the method\n* The paper studies an important problem - model-based imitation learning\n\nWeaknesses:\n* The paper made multiple somehow arbitrary choices which led to a quite an efficient algorithm, but lacks an intuition on why each of the choices was important. For example, the proposed technique (GAIL & BC component) can be plugged-and-played to any other model-based method as well as non model-based method. Is EfficientZero algorithm really crucial for this technique to work? The paper would benefit from more evidence on why each of the proposed components are important. What would happen if instead of model-based algorithm, we use a model-free algorithm with similar acting distribution (mixture between acting and BC policies), having an additional term to optimize for the GAIL-learned reward function?\n* Can it be that baseline methods can be significantly improved if EfficientZero algorithm used in these? Can it be that the main benefit in this work comes from using EfficientZero as the main RL algorithm?\n* The authors somewhat arbitrary choose the number of expert trajectories. Why exactly these numbers? How do baseline methods perform when more or less expert demonstrations are available?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "VbPH5LO8aVC", "reviewer": "Reviewer_q56x", "summary": "This paper presents EfficientImitate (EI), which combines Adversarial Imitation Learning (AIL) with model-based planning with MCTS, for sample-efficient imitation learning. During training, an expert buffer is maintained in addition to the standard replay buffer used in EfficientZero. A discriminator is used to predict rewards by contrasting expert demonstrations and agent replays. Next, MCTS is performed each time step using the predicted reward from AIL. In addition, EI unifies BC with AIL by directly including BC actions in the MCTS, which significantly improves the performance. Experiments suggest that EI has achieved significantly better performance than the SOTA methods in various continuous control tasks. \n", "questions": "1/ What are the failure modes of the proposed method?\n\n2/ Can the learned reward signal and state embeddings be interpreted in some way? \n\n3/ How are the reward, value function, and action distribution represented in the MCTS? There are different implementations for Sample Muzero. Some of them are using distributional rewards and values, with discretized action spaces, while others are using continuous actions, e.g., Gaussians, and searching with sampled actions. In addition, how do these design choices affect the final performance of EI?", "limitations": "I think one limitation of work is that for lots of continuous control tasks, inference speed is critical. However, MCTS is a slow process. Although it achieves a very good performance during evaluation, it might not be appropriate for real-world situations, where inference speed is critical. \n\nOverall, I still think this is a valuable paper and would like to vote for acceptance.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \nOverall, the paper is well-written and easy to follow. Solid experiments have been conducted to support the claims made by the paper. The proposed idea has achieved a very strong performance in experiments, outperforming the state-of-the-art (SOTA) methods by a large margin, with much fewer environment interactions. \n\nThere are some minor weaknesses of the paper. \n\n1/ The work mainly focuses on the continuous control domains, which has a relatively dense reward and limited planning horizon. However, MCTS is naturally designed for tasks with discrete action spaces, e.g., the game of Go or Atari games, and they are not considered during the experiments. My feeling is that with a sparse reward and long-planning horizon, learning reward from AIL might not be sufficient and may cause extra issues to MCTS. In that case, balancing BC and AIL will be a tricky issue.\n\n2/ The proposed idea is relatively straightforward: it is a simple combination of AIL and MCTS. I think it would be better to dig deeper into the proposed framework. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Mj13WOQOia", "reviewer": "Reviewer_t1UE", "summary": "This paper proposes a new IL called EfficientImitate (EI), a planning-based imitation learning method that can achieve high sample efficiency and performance. EI extends AIL to a model-based setting and solve it with MCTS with high sample-efficiency. EI can benefit from BC by using the BC actions in the search. EI provides a natural way to unify the two classes of existing IL methods, BC and AIL. EI uses BC actions as candidates in MCTS. The performance of EI in both state-based and image-based control tasks are amazingly good.", "questions": "1. The base method, MuZero and EfficientZero, also works well in discrete action space. How is the performance of EI in discrete action space?\n\n2. EI uses the Reanalyzed algorithm for offline training, and requires all the samples should be reanalyzed. What is the motivation for using the Reanalyzed algorithm here, and how significant does it contribute to the sample-efficiency of EI? Have you used the Reanalyzed algorithm for other baselines?\n\n3. How to choose the balancing weights $\\lambda_d$ and $\\lambda_{bc}$ in Eq. (7)? ", "limitations": "1. The performance of EI with discrete action space is unknown.\n\n2. There is no discussion on limitations and potential social impacts.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "==Originality==\n\nThe proposed EI is novel and interesting. The main idea is to extend AIL to a model-based setting with multi-step losses under the MCTS-based RL framework. Thanks to the planning component of the algorithm, EI naturally unifies two types of previous imitation learning methods (BC and AIL), and shows a significant performance boost. The connections and differences of this work and previous work are well discussed and the related work are cited in the paper.\n\n==Quality==\n\nThe proposed method is technically sound. The experiments are conducted in diverse settings with ablations, and the results well supports the claims. Although there are some limitations, this work is a complete piece of work in planning-based IL.\n\n==Clarity==\n\nThis paper is clearly written and well organized. The problem is well formulated. The method is clearly motivated and introduced. The limitations and potential impacts are also discussed.\n\n==Significance==\n\nThis paper provides an novel sample-efficient IL method that extends AIL into the MCTS-RL.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept