Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper considers the problem of policy reusing in reinforcement learning. It assumes that there are a bunch of source policies pre-trained on related tasks. The agent is interacting with the environment to learn a target policy for the target task and hopes to make use of the available source policies. The problem is to determine when and how to use which source policy.

This paper proposes CUP to employ the critic learned on the target task to select the proper source policy in each state. To be specific, there is a set of source policies and the agent's current target policy is also considered as one possible choice in the source policy set. CUP chooses the source policy with the largest one-step improvement over the current target policy. The chosen source policy in each state together forms the guidance policy. It is theoretically proved that the value of the guidance policy can be higher than the value of the current target policy if the learned critic is accurate enough. Then the target policy is trained to imitate the guidance policy by minimizing their KL divergence. The weight of this KL divergence term in policy learning is adaptively changed during training, according to the estimated advantages of the guidance policy.

The authors conduct experiments on Meta-World and compare CUP with basic SAC, recent works HAAR, PTF, MULTIPOLAR, and MAMBA. The experimental results show the advantages of CUP. The ablative study shows that CUP is relatively robust to the choice of hyper-parameter value. Adding more source policies can be beneficial if the source policy is related to the target task.  In the set of source policies, adding up to 3 random policies does not hurt the performance of CUP, but adding 4 random policies is problematic.
strengths_and_weaknesses: The paper is well-organized and generally written clearly. The proposed method CUP is novel and interesting with theoretical and empirical support.

Pros:

The proposed method is technically reasonable and supported by the theoretical ground.
The evaluation is solid and analysis of CUP in ablation study help understand CUP better.

Cons:
The proposed method theoretically relies on a well-trained critic. However, the choice of source policy might be problematic if the value estimate by the critic is not accurate enough, especially when the source policy and target policy are quite different.

One critical detail is not clearly explained in the paper. How to calculate the soft estimated advantage for each source policy according to equation (4)? Getting the expectation seems not very simple given continuous action space. Then it is hard to tell whether CUP is really much more convenient than prior works using hierarchical reinforcement learning or source policy value estimation.
questions: Could the proposed method perform robustly on settings with different sets of source policies? For example, what will happen if there are some policies really unsuitable or even harmful for the target task? Could CUP properly ignore these source policies?

Any intuitive explanation about why 4 random policies hurt the performance much in Figure 5(b)?

To get the guidance policy according to equation (7), what will happen when $\pi$ and $\pi^t_{tar}$ are very different and $\tilde{Q}_{\pi^t_{tar}}$ suffers from the over-estimation issue? For example, the target policy $\pi^t_{tar}$ may rarely select an action $a_0$ at the state $s$, so the value estimate $\tilde{Q}_{\pi^t_{tar}}(s,a_0)$ is much higher than the true value $Q_{\pi^t_{tar}}(s,a_0)$. Then the source policy selecting action $a_0$ often at the state $a$ will be chosen at this state. Yet, it may not be really beneficial for the target policy learning. This choice of source policy will hurt the sample efficiency of CUP. Do you observe this issue? Any comments about preventing it?
limitations: It seems that the authors just briefly mention one limitation in section 2. "We assume that the source policies and the target policy share the same state and action space". This assumption is widely used in prior works about policy transfer, and the authors did not propose to solve this issue. One possible choice is to learn state and action correspondence to transfer the source policy to the target state and action space, e.g., 'Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency'.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a novel policy reuse algorithm Critic-gUided Policy reuse (CUP), which avoids training any extra components and efficiently reuses source policies. CUP chooses the source policy at each state that has the largest one-step improvement over the current target policy and forms a guidance policy. The target policy can be regularized to imitate the guidance policy to perform an efficient policy search.
strengths_and_weaknesses: Strengths:

1. This paper not only proves that the guidance policy is guaranteed to be a monotonic improvement over the current target policy but also proves that the target policy is theoretically guaranteed to improve by imitating the guidance policy.

2. The experimental part of this paper is adequate in addition to transfer performance, including analyzing the guidance policy, the sensitivity to hyper-parameter settings and the number of source policies, and the interference with random source policies. The writing ideas of this part are also very clear.

Weaknesses:

1. Some descriptions are not clear and should be clarified. In the experimental part, I have questions about the analysis of some experimental results. For example, I don't see a noticeable improvement in the comparison of CUP’s performance with different numbers of source policies, which cannot be concluded that CUP is able to utilize the additional source policies to further improve its performance. 

2. The conclusion merely repeats what was said in the introduction, lacking the limitations of CUP. For example, CUP obeys a very strong assumption that the source policies and the target policy share the same state and action spaces, which is not common in the real world. The strong assumption limits the extension of CUP to more general scenarios[1-3]. The author should add some reflections on directions for future improvements.

[1] Mutual Information Based Knowledge Transfer Under State-Action Dimension Mismatch. UAI 2020.

[2] Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency. ICLR 2021.

[3] Cross-domain Adaptive Transfer Reinforcement Learning Based on State-Action Correspondence. UAI 2022.


questions: 1. In the experimental part, I don't see a noticeable improvement in the comparison of CUP’s performance with different numbers of source policies, which cannot be concluded that CUP is able to utilize the additional source policies to further improve its performance. 

2. From how CUP works, it should not be affected by random policies because it chooses the largest soft expected advantage at each state $s$. As the number of random policies grows, the performance should not be affected even though the computational complexity will increase. But why CUP can only adapt to 3 random policies?
limitations: The author should discuss the limitations of CUP. For example, CUP obeys a very strong assumption that the source policies and the target policy share the same state and action spaces, which is not common in the real world. For example, how can CUP be applied to different robot transfer settings, e.g., different types or different numbers of joints? The strong assumption limits the extension of CUP to more general scenarios.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: CUP is an algorithm for re-using previously learned policies to guide the training of a new policy on a different-but-related task. CUP does this by selecting a single guide policy from among the library of pretrained policies at each step, which the student policy is then trained to behave similar to using a KL divergence regularization term.
strengths_and_weaknesses: Overall, I liked this paper. The algorithm is novel, clearly presented, and conceptually simple (a plus). The experiments provide reasonable evidence that CUP improves performance compared to both baseline and alternative teacher-student transfer algorithms.

I do have a few concerns, though I don't think any of this invalidates the results presented:

-If I understand correctly, taking an argmax among policies using a partially-trained value function seems prone to bias/error magnification. Given relatively poor Q estimates of equal magnitude for each policy's sampled actions (as can happen early in training), the guide policy selected will tend to be the one that samples actions which Q is most (unrealistically) optimistic about. Further, the estimated advantage KL term weighting makes these updates larger. 
I think the authors appreciate this, hence their value function upper bound on the weighting term and not using the KL term for the first 0.5M environment steps (as per A.4), and the result is an algorithm that works in practice (as shown by the experiments). That said, it does make me wonder how well performance will hold up as the difference between source and target tasks increases (where many actions sampled by source policies will be bad for any given state). The random-source-policy ablation sort-of tests this, but still assumes a subset of source policies are relatively high-performing. Basically, can CUP be used to gain a training benefit from weak teacher policies?

-Relatedly, the bound in theorem 2 is dependent on the difference between source and target policies( as well as reward magnitude), and could be a very large bound given adversarial values. I'm willing to accept that this isn't an issue in practice (at least for Metaworld), but I'm curious to see how those factors impact empirical performance.

-Connected to the above two points, while it may be something of a stereotype for reviewers to ask for more experiments, additional experiments on other Metaworld tasks (either the full suite of 50 or select "hard" tasks that are less similar to the source tasks) would improve the paper in a worthwhile way. Ideally I'd like to have some qualitative evidence for how different aspects of source versus target tasks affect performance for CUP. In the current results it looks like CUP improves over SAC less on Hammer and Peg-Insert-Side, the two "more novel" tasks, but without more tasks or deeper analysis it's hard to say anything conclusive. 

questions: The conclusion section of the paper is pretty limited. I appreciate how space is limited, but some discussion of broader limitations and possible avenues for future improvement would be nice if space can be found.

In Figure 3, I'm surprised how little CUP seems to use any guide policy throughout training. I'm not sure offhand how to tap into it, but this seems like it might be a sign of leaving performance on the table? The least trained target policy is only getting updated with the KL term about half the time on a task where the push guide policy should be highly informative.

Related to that, I wonder what the percentages would be if training on one of the source tasks? For example, would the push policy get used more if training on the push task? It could provide a useful indicator for whether there's more to be gained from the source policies.
limitations: I included discussion and suggestions for limitations in the previous sections, and while I'd like to see more "hard" test cases the existing experiments do provide some idea of the limitations of CUP. The potential for negative social impact from this work is limited but is addressed in the appendix.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper is meant to achieve efficient policy reuse for resolving complex tasks. Specifically, they introduce Critic-gUided Policy reuse (CUP), evaluating and choosing appropriate source policies to regularize the training on the target tasks. Experiments shows convincing improvements over kinds of baselines

strengths_and_weaknesses: Strength: 
The topic of reusing simple source policies for resolving complex tasks is important. The paper is clearly stated, well written, and easy to follow. The proposed method is intuitive, simple and straightforward. The experiment comparison is strong and convincing.

Weakness: 
- It seems that the proposed method tends to be affected by the chosen of hyper-parameters. Although the authors show in Figure 5 that “CUP performs well on a wide range of hyper-parameters”, it is not quite a large range. From the formulation, I find it will be a little tricky to tune the two $\beta$s to reach a balanced imitation. 
- List all hyper-parameters used in your experiments and provide both default and more guidance to the hyper-parameter settings will help relieve this concern.

I am willing to vote for an accept to this paper, but I would like to do so after the authors can relieve my concern about the hyper parameters and completeness (also see Questions below).

===

After the first round of rebuttal, the author addressed most of my concerns, and I am increasing my score to 5.

===

After the second round of rebuttal, the author further addressed my concerns, and I am increasing my score to 6.
questions: 1. You mentioned that “MULTIPOLAR fails in more complex tasks” but the algorithm works in the last two figures. So what is the order of difficulty of these six tasks?

2. Why not provide the percentages and expected advantages figure for all tasks (in the Appendix)? Better include all figures in the Appendix for completeness.

3. It seems the proposed method reuses the source policies by distilling them into a single one. Can the author provide more discussion about the advantage (e.g., stability, intuition, efficiency) to do so compared to those HRL works who learned to choose different source policies? Why distill all knowledge into a single policy a better idea?

limitations: The authors have addressed their limitations.
review_title: null

Meta-review: metareview: The paper proposes a method for how to leverage a list of pretrained policies for learning a new task, by picking the guidance policy through maximal one-step policy improvement evaluated with the learned critic.

Contribution is simple, but writing, theories, and experiments/ablation studies are clean and easy to follow. There is a consensus among the reviewers for the acceptance of the paper. 

Minor comments:
- adding a mechanism for automatically growing and pruning source policies could be nice extension, especially on life-long continual learning environment, where once you learned a novel-enough high-reward policy you may want to add it to the source, so when environment changes and changes back, it can reuse that learn optimal behavior. [1]
- a fun experiment to include is to ignore reward and only do imitation during policy improvement (just KL term), while still using reward critic for policy selection. If we know the source policies sufficiently cover the full optimal policy, then this could be a good debugging test.

[1] Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., ... & Hadsell, R. (2016). Progressive neural networks. arXiv preprint arXiv:1606.04671.
confidence: Certain
Decision: Accept