CUP: Critic-Guided Policy Reuse
Authors: Jin Zhang
Siyuan Li
Chongjie Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: reinforcement learning
policy reuse
transfer learning

Abstract: The ability to reuse previous policies is an important aspect of human intelligence. To achieve efficient policy reuse, a Deep Reinforcement Learning (DRL) agent needs to decide when to reuse and which source policies to reuse. Previous methods solve this problem by introducing extra components to the underlying algorithm, such as hierarchical high-level policies over source policies, or estimations of source policies' value functions on the target task. However, training these components induces either optimization non-stationarity or heavy sampling cost, significantly impairing the effectiveness of transfer. To tackle this problem, we propose a novel policy reuse algorithm called Critic-gUided Policy reuse (CUP), which avoids training any extra components and efficiently reuses source policies. CUP utilizes the critic, a common component in actor-critic methods, to evaluate and choose source policies. At each state, CUP chooses the source policy that has the largest one-step improvement over the current target policy, and forms a guidance policy. The guidance policy is theoretically guaranteed to be a monotonic improvement over the current target policy. Then the target policy is regularized to imitate the guidance policy to perform efficient policy search. Empirical results demonstrate that CUP achieves efficient transfer and significantly outperforms baseline algorithms.

CUP: Critic-Guided Policy Reuse
Jin Zhang', Siyuan LiP, Chongjie Zhang'
Institute for Interdisciplinary Information Sciences, Tsinghua University, China School of Computer Science and Technology. Harbin Institute of Technology, China j atn=zhan20mmalss tsinghua. edu.
A1siyuan199511@gaan com
chongjie@tsinghua. www.gghuuuuuuu......... cn
Abstract
The ability to reuse previous policies an important aspect of human intelli- gence. To achieve efficient policy reuse, a Deep Reinforcement Learning (DRL) agent needs to decide when to reuse and which source policies to reuse. Previous methods solve this problem by introducing extra components to the underlying algorithm, such as hierarchical high-level policies over source policies, or estima- tions of source policies' value functions the target task. However, training these components induces either optimization not-sttiionarity or heavy sampling cost, significantly impairing the effecttivnness of transfer. To tackle this problem, propose a novel policy reuse algorithm called Critic-gUided Policy reuse (CUP), which avoids training any extra components and efficiently reuses source policies. CUP utilizes the critic, a common component in actor-critic methods, to evaluate and choose source policies. At each state, CUP chooses the source policy that has the largest one-step improvement over the current target policy, and forms a guidance policy. The guidance policy is theoretically guaranteed be monotonic improvement over the current target policy. Then the target policy is regularized to imitate the guidance policy to perform efficient policy search. Empirical results demonstrate that CUP achieves efficient transfer and significantly outperforms baseline algorithms.
Introduction
Human intelligence can solve new tasks quickly by reusing previous policies (Guberman & Greenfield
199T). Despite remarkable success, current Deep Reinforcement Learning (DRL) agents lack this
knowledge transfer ability (Silver et al. 2017 Vinyals et al. 2019 Ceron & Castro 2021). leading to enormous computation and sampling cost, As consequence, a large number of works have been studying the problem of policy reuse in DRL, i.e., how to efficiently reuse source policies to speed up
target policy learning (Fernández & Veloso 2006
Barreto et al., 2018 Li et al. 2019 Yang et al..
2020b)
fundamental challenge towards policy reuse is: how does an agent with access multiple source policies decide when and where to use them (Fernández & Veloso 2006 Kurenkov et al. 2020 Cheng et al. 2020)? Previous methods solve this problem by introducing additional components to the underlying DRL algorithm, such as hierarchical high-level policies over source policies (Li et al.
2018, 2019: Yang et al. 2020b), or estimations of source policies' value functions on the target task (Brreto et al. 2017 2018: Cheng et al.) (020). However, training these components significantly staaioonaar (Pateria et al. 202T). and estimating the value functions for every source policy is computationally expensive and with high sampling cost. Thus, the objective of this study is to address the question:
impairs the effectiveness of transfer, hierarchical structures induce optimization
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Can we achieve efficient transfer without training additional components?
Notice that actor-critic methods (Lillicrap al. 2016 Fujimoto al., 2018 Haarnoja et al. 2018 learn a critic that approximates the actor's s Q lunction and serves as a natural way to evaluate policies. Based on this observation, we propose novel policy reuse algorithm that utilizes the critic to choose source policies. The proposed algorithm, called Critic-g training any additional components and achieves efficient transfer. At each state, CUP chooses the source policy that has the largest one-step improvement over the current target policy, thus forming a guidance policy. Then CUP guides learning by regularizing the target policy to imitate the guidance policy. This approach has the following advantages. First, the one-step improvement can be estimated simply by querying the critic, and no additional components are needed be trained. Secondly, the guidance policy is theoretically guaranteed to be monotonic improvement over the current target policy, which ensures that CUP can reuse the source policies to improve the current target policy. Finally, CUP is conceptually simple and easy to implement, introducing very few hyper-parameters to the underlying algorithm.
Policy reuse (CUP), avoids
We evaluate CUP on Meta- World (Yu et al. 2020), a popular reinforcement learning benchmark composed of multiple robot arm manipulation tasks. Empirical results demonstrate that CUP achieves efficient transfer and significantly outperforms baseline algorithms.
2 Preliminaries
Reinforcement learning (RL) deals with Markov Decision Processes (MDPs). A MDP can be modelled by a tuple (S, A. p. 7), with state space S. action space A. reward function r(s, a). transition function p(s'|s, a), and discount factor (Sutton & Barto 2018). In this study, we focus on MDPs with continuous action spaces. RL's objective is find a policy क (als) that maximizes the cumulative discounted return R(T)= = E EE= ("r(st; 4t)]
While CUP is generally applicable
wide range of actor-critic algorithms, in this work SAC (Haarnoja et al. 2018) as the underlying algorithm. The soft Q function and soft V function (Harrojj et al.) 2017) of policy ı are defined as:
((s,,a)-((s,k) + 7 [s)
V*(() E- [Q, a) log (a]=)),
(2)
where > is the entropy weight. SAC's loss functions are defined as:
Leritic (Qo) E((anna))) (Qo(s, - (V+
Lactor (T&) = EanD (anxs(())) (a (olss) - Qs(s
(3)
Lentropy (a) EAND [BaNx [- log
(als)) log aH]]
where is the replay buffer, H. is hyper- representing the target entropy, and network parameters, 0 is target network's 's parameters, and Vi((s) (Qp
log 7 [a]ss] is the target soft value function.
We define the soft expected advantage of action probability distribution (-1s) over policy πy at state as:
E (s, nT) = (a (-1+) [Qr, (8, a) a lo n. (a|s) - Vz,, (+)]
EA,, (s, nT)) measures the one-step performance improvement brought by following n. instead of Ty at state s, and following v, afterwards.
The field of policy reuse focuses on solving target MDP M efficiently by transferring knowledge from set of source policies ( T.,..., ..., J. We denote the target policy learned on M at iteration t as faar and corresponding soft Q function as Q In this work, we assume that the source policies and the target policy share the same state and action spaces.

---

3 Critic-Guided Policy Reuse
This section presents CUP, efficient policy algorithm that does not require training any additional components. CUP is built upon actor-critic methods. In each iteration, CUP uses the critic form guidance policy from the source policies and the current target policy. Then CUP guides policy search by regularizing the target policy to imitate the guidance policy. Section presents how to form guidance policy by aggregating source policies through the critic, and proves that the guidance policy is guaranteed to be monotonic improvement over the current target policy. We also prove that the target policy is theoretically guaranteed improve by imitating the guidance policy. presents the overall framework of CUP.
Section 3.2
3.1 Critic-Guided Source Policy Aggregation
CUP utilizes action probabilities proposed by source policies to improve the current target policy, and forms a guidance policy. At iteration of target policy learning, for each state s, the agent has access of candidate action probability distributions proposed by the n source policies and the current target policy: IT; {mi (-1s), 72( (-(s),,--- Ta (-18), T (-1s)) The guidance policy n can be formed by combining the action probability distributions that have the largest soft expected advantage Tear at each state
Ty (+|s) arg max EAr (s, T) arg max Bast()ls)) (Q as a log n(-la)el1y
[a]s)) for all s S.
m(-bs)ell
(5) The second equation holds as adding V (s) to all soft expected advantages does not affect the result of the arg max operator. Eq. immplies that at each state, we can choose which source policy to follow simply by querying its expected soft Q value under v/ar- Noticing that with function approximation, the exact soft Q value cannot be acquired. The following theorem enables us to form the guidance policy with an approximated soft Q function, and guarantees that the guidance policy is a monotonic improvement over the current target policy.
Theorem
Let Qnian be an approximation of Qui.. such that @ (s. Qutar (s, cfor all S,
(6)
Define
(-((s)) = arg max (a Qxt. (s,a) a log T(als) for all E S. x(-1a)E11;
(7)
Then,
De for all S.
(8) 2 Vasion (8)
(8)
1
Theorem provides way to choose source policies using an approximation of the current target policy's s soft Q value. As SAC learns such an approximation, the guidance policy can be formed without training any additional components.
The next question is, how to incorporate the guidance policy my into target policy learning? The following theorem demonstrates that policy improvement can be guaranteed if the target policy is optimized to stay close to the guidance policy.
Theorem If
DKL
Tiar 1
(-15)][=(((15)) S 8 for all S
(9)
then
V ((+1(()) 2 V (8) 2 Iu 25 Rmaa + a 11 Sear
2e + af for all E S,
mar
(10)
7)2
where Rmar max (r(s.)) a) is the largest possible absolute value of the reward, He+1
max H( is the largest possible absolute difference of the policy entropy.
++1 (-1s)) is the largest entropy of 1+1 and Hmaz max (((orr(()s))) H( (-1s))
3

---

According to Theorem 2 the target policy can be improved by minimizing the KL divergence between the target policy and the guidance policy. Thus we can use the KL divergence as an auxiliary loss to guide target policy learning. Proofs of this section are deferred to Appendix [B and Appendix B.2 Theorem T and Theorem can be extended to common "hard" value functions (deferred to Appendix B.3), so CUP is also applicable to actor-critic algorithms that uses "hard" Bellman updates, such as A3C (Mnih et al. 2016).
3.2 CUP Framework
Source Policies
T1 N2, ... nn
Guidance Policies
g
g
Target Policy Learning
tar
T
T
T
T+ Tar
Guidance Policy Formation
KL Regularization
Original RL Update
Figure 1: CUP framework. In each iteration, CUP first forms guidance policy by querying the critic, then guides policy learning by adding KL. regularization to policy search.
In this subsection propose the overall framework of CUP. As shown in Fig. 1 each iteration f,
CUP first forms a guidance policy Ty according to Eq. then provides additional guidance to policy search by regularizing the target policy 4++1 to imitate Th (Wu et al. 2019 Fujimoto & Gu 2021
Specifically, CUP minimizes the following loss to optimize Ttar
LeU Acup(Ttar (1tt)) Lactor ( + EAND (.DKL ( (-1s) | (1s))]
(11)
where Leactor is the original actor loss defined Eq. 3 and B. > 0) is hyper-paramette controlling the weight of regularization. In practice, find that using fixed weight for regularization has two problems. First, it is difficult to balance the scale between Lactor and the regularization term, because Lactor grows as the Q value gets larger. Secondly, fixed weight cannot reflect the agent's confidence on TT. For example, when no source policies have positive soft expected advantages, TT n
Then the agent should not imitate т. anymore, as TH cannot provide any guidance to further improve performance. Noticing that the soft expected advantage serves a natural confidence measure, weight the KL divergence with corresponding soft expected advantage at that state:
B, = 31 min EEAi (s, (B) 32 si... (s)1)
(12)
where EAr; (s. E (am Qx. (s, agg (/(ss) V. (s) is the approxi- mated soft expected advantage, 31 B2 > O are two hyper-parameters, and V= (s) Ea-nia-((1s)) [Qr... (s.a) a log ((ar) is the approximated soft value function. This adaptive regularization weight automatically balances between the two losses, and ignores the regularization term at states where Ti cannot improve over t(ar anymore. We further upper clip the expected advantage with the absolute value of B2 Vnc to avoid the agent being overly confident about T due function approximation error e.
CUP's pseudo-code is presented in Alg. [ The modifications CUP made SAC are marked in red.
Additional implementation details are deferred to Appendix D.I

---

Algorithm CUP
Source policies { .2,, Tah hyper-parameters' λo, AB2, Xx, Ao, T, H. 31, Initialize replay buffer D
Require:
Initialize actor To, entropy weight a, critic Q target networks Qo, < Qu Q8 + Q02 while not done do for each environment step do
Og ~ Tо
St+1 p(s++1 [st, a4) {st, ae (8,, az); $141
D - U
end for
for each gradient step do
Sample minibatch b from D
Query source policies' action probabilities {T1((1s), (2(1)))) [.(s)))...(.(sss))) for states in b
Compute expected advantages according Eq. (4). form my according Eq.
0g - 8, Ao Vo, (Qo,) for i (1,2)
Φ 6 x V ILCuu((T (T
a - a Aa Vo Lentropp (x))
0, + 70, + (1 - T)0, for {1.2}
end for end while
4 Experiments
We evaluate Meta-World (Yu al. 2020). popular reinforcement learning benchmark composed of multiple robot manipulation tasks. These tasks are both correlated (performed by the same Sawyer robot arm) and distinct (interacting with different objects and having different reward functions), and serve as a proper evaluation benchmark for policy reuse. The source policies are achieved by training three representative tasks: Reach, Push, and Pick-Place. We choose several complex tasks as target tasks, including Hammer, Peg-Insert-Side, Push-Wall, Pick-Place- Wall, Push-Back, and Shelf-Place. Among these target tasks, Hammer and Peg-Insert-Side require interacting with objects unseen in the source tasks. In Push-Wall and Pick-Place-Waz Wall, there is a wall between the object and the goal. In Push-Back, the goal distribution is different from Push. In Shelf-Place, the robot is required to put a block on shelf, and the shelf is unseen in the source tasks. Video demonstrations of these tasks are available litpss: /tte:://mmaaaaaee world gi thub io/] Similar to the settings in Yang et al. (2020a), in our experiments the goal position is randomly reset at the start of every episode. Codes are available httttttt:/// /etttps:/ggithae www www.aao..ooo//Naaaaaaaaaaa
4.1 Transfer Performance on Meta-Wort World
We compare against several representative baseline algorithms, including HAAR ( et al. 2019). PTF (Yang et al.) 20200)) MULTIPOLAR (Barekatain al., 2021), and MAMBA (Cheng al.) 2020 Among these algorithms, HAAR and PTF learn hierarchical high-level policies over source policies. MAMBA aggregates source policies' V functions to form baseline function, and performs policy improvement over the baseline function. MULTIPOLAR learns a weighted sum of source policies' action probabilities, and learss additional network to predict residuals. We also compare against the original SAC algorithm. All the results are averaged over six random seeds. As shown Figure 2 CUP the only algorithm that achieves efficient transfer on all six tasks, significantly outperforming the original SAC algorithm. HAAR has a jump-start performance on Push- Wall and Pick-Pick-Wall but fails further improve due optimization non-stationarity induced by jointly training high-level and low-level policies. MULTIPOLAR achieves comparable performance on Push-Wall and Peg-Insert Insert-Side, because the Push source policy useful on Push- Wall (implied by HAAR's good jump-start performance), and learning residuals on Peg- Peg-Insert-SS Insert-Side is easier (implied by SAC's fast learning). In Pick-Place- Wall, the Pick-Place source policy is useful, but the residual difficult to learn, so MULTIPOLAY does not work. For the remaining three tasks, the source policies are less useful, and MULTIPOLAR fails on these tasks. PTF fails as its hierarchical policy only gets updated when the agent chooses similar actions to one of the source policies, which is quite rare when
5

---

Push-Back-V2
Hammer V2
Shelf-Place-V
4000
40:0
2000
2000
1390
1000
590
Millon Samples
Milion Samples
Vilion Samples
Pick Place Woce-WWll-V2
Push-Wall-V2
Pag Insert Side
1..0
30.0
300
550
1290
30
10
Millon Samples
Millin
Semples
CUP (Ours)
SAC
HAAR
MULTIPOLAR
PTF
MAMBA
Figure Evaluation of CUP and several baselines on various World tasks. Dashed areas represent 95% bootstrapped confidence intervals. CUP achieves substantially better performance than baseline algorithms.
the source and target tasks are distinct. MAMBA fails as estimating all source policies' V functions accurately is sampling inefficient. Algorithm performance evaluated by success rate is deferred to Appendix E.1
4.2 Analyzing the Guidance Policy
This subsection provides visualizations CUP's source policy selection. Fig. [3shhws the percentages of each source policy being selected throughout training Push-Wall. At early stages of training, the source policies are selected more frequently as they have positive expected advantages, which means that they can be used to improve the current target policy. As training proceeds and the target policy becomes better, the source policies are selected less frequently. Among these three source policies, Push is chosen more frequently than the other two source policies, it is more related the target task. Figure presents the source policies' expected advantages over an episode at convergence in Pick-Place- Wall. The Push source policy and Reach source policy almost always have negative expected advantages, which implies that these two source policies can hardly improve the current target policy anymore. Meanwhile, the Pick-Place source policy has expected advantages close to zero after 100 environment steps, which implies that the Pick-Place source policy is close to the target policy at these steps. Analyses all six tasks well as analyses on HAAR's source policy selection are deferred to Appendix E.2 and Appendix E.6 respectively.
4.3 Ablation Study
This subsection evaluates CUP's sensitivity typer-parameter settings and the number of source policies. We also evaluate CUP's robustness against random source policies, which do not provide meaningful candidate actions for solving target tasks.
4.3.1
Sensitivity
For all the experiments in Section 4.1 we use the same set of hyper- saper-parameters, which indicates that CUP generally applicable a wide range of tasks without particular fine-tuning. CUP introduces only two additional hypee-paraaeten to the underlying SAC algorithm, and we further test CUP's sensitivity to these additional hyper-pane param-parameters. As shown in Fig. 5 CUP generally robust to the choice of tlyper-parameee and achieves stable performance.

---

Push-Wali-V2
1.2
Target Palicy
Push
Pick-Place
Success
0.
Millonn Samples
Figure 3: Percentages of source policies being selected by CUP during training on Push-Wal Wall. The green dashed line represents the target policy's 's success rate on the task.
Pick-Place-Wall-v2
20
-30
Target Policy
440
Reach
Push
-50
PlaccP
100
200
Steps
Figure 4: Expected advantages of source policies at convergence on Pick-Place-Wei Wall. The horizontal axis represents the environment steps of an episode.
4.3.2 Number of Source Policies
We evaluate CUP as well as baseline algorithms larger source policy We add three policies to the original source policy set, which solve three simple tasks including Drawer-Close, Push-Wall, Wall, and Coffee-Button, This forms a source policy set composed of six policies. As shown in Fig. 6 CUP still the only algorithm that solves all the six target tasks efficiently. MULTIPOLAR suffers from decrease in performance, which indicates that learning the weighted actions becomes more difficult as the number of source policies grows. The rest of the baseline algorithms have similar performance to those using three source policies. Fig. provides a more direct comparison of CUP's performance with different number of source policies. CUP is able to utilize the additional souree policies further improve its performance, especially on Pick-Place- Wall and Ins--nner--S,de Further detailed analysis is deferred to Appendix E.3
of source policies'
4.3.3 Interference of Random Source Policies
In order to evaluate the efficiency of CUP's critic-guided source policy aggregation, we add random policies to the set source policies. As shown Fig. ((a) adding up to random source policies does not affect CUP's performance. This indicates that CUP can efficiently choose which source policy to follow even if there exist many source policies that are not meaningful. Adding and random source policies leads to slight drop in performance. This drop is because that as the number
7

---

Push-Wall-V2
Pick-Place-Wall-
= 30.9; 3
30 3e-
P.-20.0,-2c-
--20,0,-2e--
4000
- 40,00 46
40,8, 40
9. 25-50,9,55--
3000
50,,,,
Retur
3000
A. 100, -1-
Return
2500
A. 1000,8; =
2000
2000
1500
1000
1000
500
Million Samples
Million Samples
Figure 5: Ablation studies on a wide range of hyper-parameters. CUP performs well on a wide range of hyper-parameters.
Push-Back-v2
Hammer
600
200
550
1030
1200
0.3
2.
255
1.0
3.5
2.0
3.0
35
53
2.5
300
Million
Pick- Place Woll-V2
Push-Wall-V2
Pog Insert Side V2
4000
3000
3230
Ratum
Ax0
1090
300
0.30 037
2335
1.55
Millon Samples
Millon Samples
Millonn Semples
CUP (Ours)
SAC
HAAR
MULTIPOLAR
PTF
MAMBA
Figure 6: Performance of CUP and baseline algorithms on various Meta-World tasks, with set of six source policies.
of random policies grows, random actions sampled, and taking argmax these actions' expected advantages is more likely to be affected by errors in value estimation.
To further investigate CUP's ability to ignore unsuitable source policies, we design another transfer setting that consists of another two source policy sets. The first set consists three random policies that useless for the target task, and the second set adds the Reach policy to the first set. As demonstrated in Fig. 8(b) when none of the source policies are useful, CUP performs similarly to the original SAC, and its sample efficiency is almost unaffected by the uscless source policies. When there exists a useful source policy, CUP can efficiently utilize it to improve performance, even there many useless source policies.
Related Work
Policy reuse. series of works on policy reuse utilize source policies for exploration value-based
algorithms (Fernández & Veloso 2006 Li & Zhang 2018 Gimelfarb et al.) 2021). but they are not applicable to policy gradient methods due to the oft-policyness problem (Fujimoto al.) (201).. AC- Teach (Kurenkov et al. 2020) mitigates this problem by improving the actor over behavior policy's 's

---

Sheif-Place-v2
Hammer V2
Push-Back-V2
4530
0030
3000
2500
2120
2290
150
1000
1230
1230
Milloon Sampless
Milion Samples
Millon Somples
Pick Place-Wall-v>
Push-Wall-V2
Pag Insert Side V2
4230
3500
20.0
3030
2000
2150
2230
100
1530
Vilion Samples
Stmples
Stmples
CUP (Three Sources)
CUP (Six Sources)
SAC
Figure 7: Comparison of CUP's performance with different number of source policies.
Push-Wall-V2
Push-Wall-V2
CUP
3 Random Sources
4000
Random cielcces
Random
Random
Add 4 Random colicies
3000
Add 5 Random Pillcies
3000
Return
2000
2000
1000
0
0.0
1.5
2.5
3.0
0.0
1.5 2.0 2.5
3.0
4.0
Million Samples
Million Samples
(a)
(h)
Figure 8: Ablation studies on CUP's sensitivity to useless source policies. (a) Adding up to 3 random
policies to the source policy set does not affect CUP's performance. (b) Ablation study in setting where most source policies are useless. If none of the source policies are useful (3 Random Sources), CUP performs similarly to the original SAC. Even if only one of the four source policies is useful (3 Random Sources+Reach), CUP is still able to efficiently utilize the useful source policy to improve learning performance.
value estimation, but still fails in complex tasks. One branch of methods train hierarchical high-level policies over source policies. CAPS (Li et al., 2018) guarantees the optimality of the hierarchical policies by adding primitive skills to the low-level policy set, but inapplicable to MDPs with continuous action spaces, HAAR (Li et al. 2019) fine-tunes low-level policies to ensure optimality, but joint training of high-level and low-level policies induce optimization non-stationarity (Pateria et al. 202T). PTF (Yang et al., 2020b) trains a hierarchical policy, which is imitated by the target policy. However, the hierarchical policy only gets updated when the target policy chooses similar actions to one of the source policies, so PTF fails in complex tasks with large action spaces. Another branch works aggregate source policies via their Q functions V functions on target task. Barreto et al. (2017) and Barreto et al. (2018) focus on the situation where source tasks and target tasks share same dynamics, and aggregate source policies by choosing the policy that has the largest Q at each state. They successor features to mitigate the heavy computation cost

---

brought by estimating Q functions for all source policies. MAMBA (Cheng et al. 2020) forms baseline function by aggregating source policies' V functions, and guides policy search by improving the policy over the baseline function. Finally, MULTIPOLAR (Barekatain et al.) 2021 learns weighted sum over source policies' actions, and learns an auxiliary network to predict residuals around the aggregated actions. MULTIPOLAR is computationally expensive, as it requires querying all the source policies at every sampling step. Our proposed method, CUP, focuses on the setting of learning continuous-action MDPs with actor-critic methods. CUP is both computationally and sampling efficient, as does not require training any additional components.
Policy regularization. Adding regularization to policy optimization is common approach to induce prior knowledge into policy learning. Distral (Teh al. (017) achieves inter-task transfer by imitating an average policy distilled from policies of related tasks. In offline RI., policy regularization serves as a common technique to keep the policy close the behavior policy used collect the dataset (Wu et al. [019 Nair et al. Fujimoto & Gu 2021). CUP uses policy regularization as a means to provide additional guidance to policy search with the guidance policy.
Conclusion
In this study, we address the problem of reusing source policies without training any additional components. By utilizing the critic as natural evaluation of source policies, we propose CUP, an efficient policy reuse algorithm without training any additional components. CUP is conceptually simple, easy implement, and has theoretical guarantees. Empirical results demonstrate that CUP achieves efficient transfer wide range of tasks. As for future work, CUP assumes that all source policies and the target policy share the same state and action spaces, which limits CUP's s application to more general scenarios. One possible future direction is to take inspiration from previous works that map the state and action spaces of an MDP another MDP with similar high-level structure (Wan) et al. 2020 ZZhang et al. 2020; Heng et al., 2022, van der Pol et al. (020bba) a). Another interesting direction is to incorporate CUP into the continual learning setting (Rolnick et al., 22019) Khetarpal et al. 2020), in which an agent gradually enriches its source policy set in an online manner.
Acknowledgements
This work is supported in part by Science and Technology Innovation 2030 "New Generation Artificial Intelligence" Major Project (No. 2018AAA0100904), National Natural Science Foundation of China (62176135), and China Academy of Launch Vehicle Technology (CALT2022-18).
References
Barekatain, M., Yonetani, R., and Hamaya, M. Multipolar: multi-source policy aggregation for transfer reinforeement learning between diverse environmental dynamics. In Proceedings of the Twenty-inthh International Conference on International Joint Conferences on Artificial Intelligence, pp. 3108-3116, 2021.
Barreto, A., Dabney, W., Munos, R., Hunt, J., Schaul, T., van Hasselt, H. P., and Silver, D. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 30, 2017.
Barreto, A., Borsa, D., Quan, J., Schaul, T., Silver, D., Hessel, M., Mankowitz, D., Zidek, A., and Munos, R. Transfer deep reinforcement learning using successor features and generalised policy improvement. In International Conference on Machine Learning, pp. 501-510. PMLR, 2018.
Ceron, S. O. and Castro, S. Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research. In International Conference Machine Learning, pp. 1373-1383. PMLR, 2021.
Cheng, C.-A., Kolobov, A., and Agarwal, A. Policy improvement via imitation of multiple oracles. Advances in Neural Information Processing Systems, 33:5587-5598,, 2020.
Fedotov, A. A., Harremoës, P., and Topsoe, Refinements pinsker's 's inequality. IEEE Tronsactions on Information Theory, 49(6):149111499 2003.
10

---

Fernández, F. and Veloso, M. Probabilistic policy reuse in a reinforcement learning agent. In Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems, pp. 720-727, 2006.
Fujimoto, S. and Gu, S. S. A minimalist approach to offline reinforcement learning, Advances in Neural Information Processing Systems, 34, 2021.
Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Fujimoto, S., Meger, D.. and Precup. D. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-2062. PMLR, 2019.
Gimelfarb, M., Sanner, S., and Lee, C.-G. Contextual policy transfer in reinforcement learning domains via deep mixttrres-ffexpperr In Uncertainty in Artificial Intelligence, pp. 1787-1797. PMIR, 2021.
Guberman, S. R. and Greenfield, M. Learning and transfer in everyday cognition. Cognitive Development, 6(3):233-260, 1991.
Haarnoja, T., Tang. H., Abbeel, P., and Levine, S. Reinforcement learning with deep energy-based policies, In International Conference on Machine Learning, pp. 1352-1361. PMLR, 2017.
Haarnoja, T., Zhou, A.. Hartikainen, K., Tucker, G., Ha, S., Tan, J.. Kumar. V., Zhu. H., Gupta, A.,
Abbeel, P., al. Soft actor-critic algorithms and applications. arXiv preprint 112.::881. 2018.
Heng. Y., Yang, T., ZHENG, Y., Jianye, H., and Taylor, M. E. Cross-domain adaptive transfer rein- forcement learning based on state-action correspondence. The 38th Conference on Uncertainty in Artificial Intelligence, 2022.
Kakade, S. and Langford, J. Approximately optimal approximate reinforcement learning. In In Proc. 19th International Conference on Machine Learning. Citeseer, 2002.
Khetarpal, K., Riemer, M., Rish, I,, and Precup, D. Towards continual reinforcement learning: A review and perspectives. arXiv preprint arXiv:2012.13490. 2020.
Kurenkov, A., Mandlekar, A., Martin Martin, R., Savarese, S., and Garg, A. Ac-teach: A bayesian actor-critic method for policy learning with an ensemble of suboptimal teachers. In Conference on Robot Learning, pp. 7 717-734. PMLR, 2020.
Li, S. and Zhang, C. An optimal online method of selecting source policies for reinforeement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Li, S., Gu, F., Zhu, G., and Zhang, C. Context-aware policy reuse. arXiv preprint arXiv: 1806.033,,, 2018.
Li, S., Wang, R., Tang, M., and Zhang, C. Hierarchical reinforcement learning with advantage-based auxiliary rewards. Advances in Neural Information Processing Systems, 32, 2019.
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Frez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. In ICLR (Poster), 2016.
Mnih, V., Badia, A. P., Mirza, M.. Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937 PMLR, 2016.
Nair, A., Gupta, A., Dalal, M., and Levine, Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09330 09359, 2020.
Ostrovski, G., Castro, P. S., and Dabney, W. The difficulty of passive learning in deep reinforcement learning. Advances Neural Information Processing Systems, 34:23283-23295, 2021.
Pateria, S., Subagdja, B., Tan, A.-h., and Quek, C. Hierarchical reinforcement learning: A compre- hensive survey, ACM Computing Surveys (CSUR), 54(5):1-33 202..

---

Rolnick, D., Ahuja. A., Schwarz, J., Lillicrap. T., and Wayne, G. Experience replay for continual learning, Advances Neural Information Processing Systems, 32, 2019.
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. Mastering the game of go without human knowledge. nature, 550 (7676):354-33) 359, 2017.
Sodhani, S., Zhang, A., and Pineau, Multi-task reinforcement learning with context- based representations. In International Conference Machine Learning, pp. 9767-9779. PMLR, 2021.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018.
Teh, Y., Bapst, V. Czarnecki, M.. Quan, J., Kirkpatrick, J., Hadsell, R., Heess, N., and Pascanu, R. Distral: Robust multitask reinforcement learning. Advances in neural information processing systems, 30, 2017.
der Pol, E., Kipf, T., Oliehoek, F. A., and Welling, M. Plannable approximations to mdp homomorphisms: Equivariance under actions. arXiv preprint arXiv:2002.11963 2020a.
van der Pol, E.,, Worrall, D., van Hoof, H., Oliehoek, F., and Welling, M. Mdp homomorphic networks:
Group symmetries in reinforcement learning. Advances in Neural Information Processing Systems, 33:4119-4210, 2020b.
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350- -540003-5,2220 2019.
Wan, M., Gangwani, T., and Peng, J. Mutual information based knowledge transfer under state-action dimension mismatch. arXiv preprint arXiv:2006.0700) 2020.
Wu, Y., Tucker, G., and Nachum, O. Behavior regularized offline reinforcement learning. arXiv preprint ar
2019.
Yang, R., Xu, H., Wu, Y., and Wang. X. Multi-task reinforcement learning with soft modularization. Advances in Neural Information Processing Systems, 33:4767-4777, 2020a.
Yang, T., Hao, J., Meng, Z., Zhang, Z., Hu, Y., Chen, Y., Fan, C., Wang, W., Wang. Z., and Peng. J. Efficient deep reinforcement learning through policy transfer. In AAMAS, pp. 2053-2055, 2020b.
Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, Meta-world: benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning, pp. 1094-1100. PMLR, 2020.
Zhang. Q. Xiao, T., Efros, A. A., Pinto, L., and Wang. X. Learning cross- domain correspondence for control with dynamics cycle-consistenge arXiv preprint arXiv:20/2.098/ 2020.
12


Reviews:
[{"review_id": "gfo-IuBEJzb", "reviewer": "Reviewer_nMQw", "summary": "This paper considers the problem of policy reusing in reinforcement learning. It assumes that there are a bunch of source policies pre-trained on related tasks. The agent is interacting with the environment to learn a target policy for the target task and hopes to make use of the available source policies. The problem is to determine when and how to use which source policy.\n\nThis paper proposes CUP to employ the critic learned on the target task to select the proper source policy in each state. To be specific, there is a set of source policies and the agent's current target policy is also considered as one possible choice in the source policy set. CUP chooses the source policy with the largest one-step improvement over the current target policy. The chosen source policy in each state together forms the guidance policy. It is theoretically proved that the value of the guidance policy can be higher than the value of the current target policy if the learned critic is accurate enough. Then the target policy is trained to imitate the guidance policy by minimizing their KL divergence. The weight of this KL divergence term in policy learning is adaptively changed during training, according to the estimated advantages of the guidance policy.\n\nThe authors conduct experiments on Meta-World and compare CUP with basic SAC, recent works HAAR, PTF, MULTIPOLAR, and MAMBA. The experimental results show the advantages of CUP. The ablative study shows that CUP is relatively robust to the choice of hyper-parameter value. Adding more source policies can be beneficial if the source policy is related to the target task.  In the set of source policies, adding up to 3 random policies does not hurt the performance of CUP, but adding 4 random policies is problematic.", "questions": "Could the proposed method perform robustly on settings with different sets of source policies? For example, what will happen if there are some policies really unsuitable or even harmful for the target task? Could CUP properly ignore these source policies?\n\nAny intuitive explanation about why 4 random policies hurt the performance much in Figure 5(b)?\n\nTo get the guidance policy according to equation (7), what will happen when $\\pi$ and $\\pi^t_{tar}$ are very different and $\\tilde{Q}_{\\pi^t_{tar}}$ suffers from the over-estimation issue? For example, the target policy $\\pi^t_{tar}$ may rarely select an action $a_0$ at the state $s$, so the value estimate $\\tilde{Q}_{\\pi^t_{tar}}(s,a_0)$ is much higher than the true value $Q_{\\pi^t_{tar}}(s,a_0)$. Then the source policy selecting action $a_0$ often at the state $a$ will be chosen at this state. Yet, it may not be really beneficial for the target policy learning. This choice of source policy will hurt the sample efficiency of CUP. Do you observe this issue? Any comments about preventing it?", "limitations": "It seems that the authors just briefly mention one limitation in section 2. \"We assume that the source policies and the target policy share the same state and action space\". This assumption is widely used in prior works about policy transfer, and the authors did not propose to solve this issue. One possible choice is to learn state and action correspondence to transfer the source policy to the target state and action space, e.g., 'Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency'.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-organized and generally written clearly. The proposed method CUP is novel and interesting with theoretical and empirical support.\n\nPros:\n\nThe proposed method is technically reasonable and supported by the theoretical ground.\nThe evaluation is solid and analysis of CUP in ablation study help understand CUP better.\n\nCons:\nThe proposed method theoretically relies on a well-trained critic. However, the choice of source policy might be problematic if the value estimate by the critic is not accurate enough, especially when the source policy and target policy are quite different.\n\nOne critical detail is not clearly explained in the paper. How to calculate the soft estimated advantage for each source policy according to equation (4)? Getting the expectation seems not very simple given continuous action space. Then it is hard to tell whether CUP is really much more convenient than prior works using hierarchical reinforcement learning or source policy value estimation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aLhzSxG0PV9", "reviewer": "Reviewer_9Jza", "summary": "CUP is an algorithm for re-using previously learned policies to guide the training of a new policy on a different-but-related task. CUP does this by selecting a single guide policy from among the library of pretrained policies at each step, which the student policy is then trained to behave similar to using a KL divergence regularization term.", "questions": "The conclusion section of the paper is pretty limited. I appreciate how space is limited, but some discussion of broader limitations and possible avenues for future improvement would be nice if space can be found.\n\nIn Figure 3, I'm surprised how little CUP seems to use any guide policy throughout training. I'm not sure offhand how to tap into it, but this seems like it might be a sign of leaving performance on the table? The least trained target policy is only getting updated with the KL term about half the time on a task where the push guide policy should be highly informative.\n\nRelated to that, I wonder what the percentages would be if training on one of the source tasks? For example, would the push policy get used more if training on the push task? It could provide a useful indicator for whether there's more to be gained from the source policies.", "limitations": "I included discussion and suggestions for limitations in the previous sections, and while I'd like to see more \"hard\" test cases the existing experiments do provide some idea of the limitations of CUP. The potential for negative social impact from this work is limited but is addressed in the appendix. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I liked this paper. The algorithm is novel, clearly presented, and conceptually simple (a plus). The experiments provide reasonable evidence that CUP improves performance compared to both baseline and alternative teacher-student transfer algorithms.\n\nI do have a few concerns, though I don't think any of this invalidates the results presented:\n\n-If I understand correctly, taking an argmax among policies using a partially-trained value function seems prone to bias/error magnification. Given relatively poor Q estimates of equal magnitude for each policy's sampled actions (as can happen early in training), the guide policy selected will tend to be the one that samples actions which Q is most (unrealistically) optimistic about. Further, the estimated advantage KL term weighting makes these updates larger. \nI think the authors appreciate this, hence their value function upper bound on the weighting term and not using the KL term for the first 0.5M environment steps (as per A.4), and the result is an algorithm that works in practice (as shown by the experiments). That said, it does make me wonder how well performance will hold up as the difference between source and target tasks increases (where many actions sampled by source policies will be bad for any given state). The random-source-policy ablation sort-of tests this, but still assumes a subset of source policies are relatively high-performing. Basically, can CUP be used to gain a training benefit from weak teacher policies?\n\n-Relatedly, the bound in theorem 2 is dependent on the difference between source and target policies( as well as reward magnitude), and could be a very large bound given adversarial values. I'm willing to accept that this isn't an issue in practice (at least for Metaworld), but I'm curious to see how those factors impact empirical performance.\n\n-Connected to the above two points, while it may be something of a stereotype for reviewers to ask for more experiments, additional experiments on other Metaworld tasks (either the full suite of 50 or select \"hard\" tasks that are less similar to the source tasks) would improve the paper in a worthwhile way. Ideally I'd like to have some qualitative evidence for how different aspects of source versus target tasks affect performance for CUP. In the current results it looks like CUP improves over SAC less on Hammer and Peg-Insert-Side, the two \"more novel\" tasks, but without more tasks or deeper analysis it's hard to say anything conclusive. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NVTR0evMDc", "reviewer": "Reviewer_HzuR", "summary": "This paper is meant to achieve efficient policy reuse for resolving complex tasks. Specifically, they introduce Critic-gUided Policy reuse (CUP), evaluating and choosing appropriate source policies to regularize the training on the target tasks. Experiments shows convincing improvements over kinds of baselines\n", "questions": "1. You mentioned that “MULTIPOLAR fails in more complex tasks” but the algorithm works in the last two figures. So what is the order of difficulty of these six tasks?\n\n2. Why not provide the percentages and expected advantages figure for all tasks (in the Appendix)? Better include all figures in the Appendix for completeness.\n\n3. It seems the proposed method reuses the source policies by distilling them into a single one. Can the author provide more discussion about the advantage (e.g., stability, intuition, efficiency) to do so compared to those HRL works who learned to choose different source policies? Why distill all knowledge into a single policy a better idea?\n", "limitations": "The authors have addressed their limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \nThe topic of reusing simple source policies for resolving complex tasks is important. The paper is clearly stated, well written, and easy to follow. The proposed method is intuitive, simple and straightforward. The experiment comparison is strong and convincing.\n\nWeakness: \n- It seems that the proposed method tends to be affected by the chosen of hyper-parameters. Although the authors show in Figure 5 that “CUP performs well on a wide range of hyper-parameters”, it is not quite a large range. From the formulation, I find it will be a little tricky to tune the two $\\beta$s to reach a balanced imitation. \n- List all hyper-parameters used in your experiments and provide both default and more guidance to the hyper-parameter settings will help relieve this concern.\n\nI am willing to vote for an accept to this paper, but I would like to do so after the authors can relieve my concern about the hyper parameters and completeness (also see Questions below).\n\n===\n\nAfter the first round of rebuttal, the author addressed most of my concerns, and I am increasing my score to 5.\n\n===\n\nAfter the second round of rebuttal, the author further addressed my concerns, and I am increasing my score to 6.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EUDOBQDIv-f", "reviewer": "Reviewer_biXP", "summary": "This paper proposes a novel policy reuse algorithm Critic-gUided Policy reuse (CUP), which avoids training any extra components and efficiently reuses source policies. CUP chooses the source policy at each state that has the largest one-step improvement over the current target policy and forms a guidance policy. The target policy can be regularized to imitate the guidance policy to perform an efficient policy search.", "questions": "1. In the experimental part, I don't see a noticeable improvement in the comparison of CUP’s performance with different numbers of source policies, which cannot be concluded that CUP is able to utilize the additional source policies to further improve its performance. \n\n2. From how CUP works, it should not be affected by random policies because it chooses the largest soft expected advantage at each state $s$. As the number of random policies grows, the performance should not be affected even though the computational complexity will increase. But why CUP can only adapt to 3 random policies?", "limitations": "The author should discuss the limitations of CUP. For example, CUP obeys a very strong assumption that the source policies and the target policy share the same state and action spaces, which is not common in the real world. For example, how can CUP be applied to different robot transfer settings, e.g., different types or different numbers of joints? The strong assumption limits the extension of CUP to more general scenarios. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This paper not only proves that the guidance policy is guaranteed to be a monotonic improvement over the current target policy but also proves that the target policy is theoretically guaranteed to improve by imitating the guidance policy.\n\n2. The experimental part of this paper is adequate in addition to transfer performance, including analyzing the guidance policy, the sensitivity to hyper-parameter settings and the number of source policies, and the interference with random source policies. The writing ideas of this part are also very clear.\n\nWeaknesses:\n\n1. Some descriptions are not clear and should be clarified. In the experimental part, I have questions about the analysis of some experimental results. For example, I don't see a noticeable improvement in the comparison of CUP’s performance with different numbers of source policies, which cannot be concluded that CUP is able to utilize the additional source policies to further improve its performance. \n\n2. The conclusion merely repeats what was said in the introduction, lacking the limitations of CUP. For example, CUP obeys a very strong assumption that the source policies and the target policy share the same state and action spaces, which is not common in the real world. The strong assumption limits the extension of CUP to more general scenarios[1-3]. The author should add some reflections on directions for future improvements.\n\n[1] Mutual Information Based Knowledge Transfer Under State-Action Dimension Mismatch. UAI 2020.\n\n[2] Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency. ICLR 2021.\n\n[3] Cross-domain Adaptive Transfer Reinforcement Learning Based on State-Action Correspondence. UAI 2022.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept