Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a Reward-Switching Policy Optimization (RSPO) that discovers diverse policies that are locally optimal and sufficiently different from the existing ones. This is done by adaptive switching among extrinsic and intrinsic rewards that are used for policy learning. RSPO makes use of both the trajectory filtering objective and the exploration objective.
main_review: The problem is well-formulated and experiments are validating the claims authors made in the paper. The overall idea of the proposed method makes sense to me. 

I have the following concerns and questions regarding this paper. I can adjust my score accordingly after the authors reply to these questions:

(1) My first concern is regarding $\delta$. I appreciate the author’s effort to provide an empirical way of adjusting it, but I didn’t find it intuitive. Such an important parameter requires a better Ablation study. The results of Table 1 shows the occurrence of different strategies by stopping the reward switching. But does it say much? Or could it be just randomly happening? I suggest showing a percentage of each strategy that is discovered, or any better way of investigating this parameter.

(2) My second concern is regarding using the reward prediction error as your reward-driven intrinsic reward. To my understanding, this value may never be accurate, specially at the start of learning $f(.,.,\psi)$. Also does it somehow relate to this paper? https://proceedings.neurips.cc/paper/2018/file/51de85ddd068f0bc787691d356176df9-Paper.pdf

(3) The ordering of the Figures in the experiments could be much better. I believe authors can fix this issue as it affects the readability of paper. E.g. Figure 4 in the main text has been referred to only in the Appendix. Page 8 contains 3 figures, start from 7, then 6, and then 5. The location of Table 1 doesn’t match with the corresponding text. Figure 7 (a) does not clearly show other approaches (PG,DIPG,RND) due to the overlay. Maybe authors can have a small zoom-in box in there?

summary_of_the_review: Overall, the paper seems to contribute to the field, but small changes are required before publication.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In the paper "Continuously Discovering Novel Strategies via Reward-Switching Policy Optimization", the authors introduce a new reinforcement learning algorithm to iteratively form a collection of high-performing and diverse policies. The main idea is that at each iteration of the algorithm, a new policy is optimised so that it maximises the extrinsic reward and satisfy the condition of being different enough from the policy already in the collection. In order to solve this constrained reinforcement learning problem, the authors propose a new loss that effectively rejects the strategies that are similar to already discovered ones and provide an intrinsic reward for pushing these strategies as far as possible before considering the extrinsic reward. They evaluate the benefits of the proposed algorithm on a large set of experiments, including continuous control tasks, and complex multi-agent domains. 
main_review: Overall the paper is of high quality. It is well structured and describes concisely and clearly the work done. The experimental evaluation is appropriate and demonstrates the benefits of the proposed method in a large variety of domains. 

While the novelty of this paper is clear from the domain of deep reinforcement learning, the paper does not consider recent approaches that combine QD algorithms and policy gradient algorithms, such as QD-RL[1] or PGA-MAP-Elites[2]. These approaches can generate collections of hundreds of NN policies and would appear as suitable baselines for this paper. One might say that these approaches require a well-defined behaviour descriptor (like MAP-Elites approaches). Yet, it is possible to use distance-based archives (like the unstructured archive in [3]) which only require the definition of a distance function and a threshold value, like in eq.(1). 

An aspect of the algorithm that remains unclear to me after looking at the experimental results is if every iteration necessarily leads to a policy added to the collection. Figures 1.a, and 7.b show that it is possible to have fewer strategies than the number of iterations. Is it because the authors listed the possible strategies and they counted how many of these were discovered by the algorithms, or is it that in certain cases, the algorithms would fail to produce a policy that satisfies the hard constraint and thus fail?

Minor comments: 
- MAP-Elites belongs to the family of QD algorithms, and there is a typo in its name in the paper. 
- "Justin K. Pugh, L. B. Soros, and K. Stanley. Quality diversity: A new frontier for evolutionary computation." appears twice in the reference list.

[1] Cideron, G., Pierrot, T., Perrin, N., Beguir, K., & Sigaud, O. (2020). QD-RL: Efficient Mixing of Quality and Diversity in Reinforcement Learning. arXiv preprint arXiv:2006.08505.
[2] Nilsson, O., & Cully, A. (2021, June). Policy gradient assisted MAP-Elites. In Proceedings of the Genetic and Evolutionary Computation Conference (pp. 866-875).
[3] Cully, A., & Demiris, Y. (2017). Quality and diversity optimization: A unifying modular framework. IEEE Transactions on Evolutionary Computation, 22(2), 245-259.
summary_of_the_review: Overall the paper is of high quality. It is well structured and describes concisely and clearly the work done. The experimental evaluation is appropriate and demonstrates the benefits of the proposed method in a large variety of domains. I believe the paper could be improved by also considering recent work from the QD literature which could be applied in similar domains while scaling to larger collections.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an algorithm for discovering diverse strategies in reinforcement learning. In the proposed method, policies are sequentially trained with the constraint that encourages to find different behaviors. During the training process, the extrinsic and intrinsic rewards are switched based on a metric for quantifying the novelty of trajectories. The proposed method is evaluated on 4 domains, including OpenAI Gym Mujoco tasks and StarCraftII Multi-Agent Challenge. The experimental results show that the proposed method discovers diverse strategies in these domains.
main_review: ### Strong points
- Proposed method finds the diverse strategies in the experiments

### Weak points
- It is not clear what is really optimized by the proposed algorithm
- design of the experiment is unclear and not convincing in some parts
- proposed algorithm seem to have many hyperparameters to be tuned

I describe the detailed comment to each point below.

### Meaning of the objective function
 I do not see a clear connection between the constrained optimization in Eq. (2) and (5). Although it is stated that the rejection-sampling-based approach enables to directly optimize Eq. (2) by strictly enforcing all the hard constraints, I don't understand why optimization problems in Eq. (2) and (5) are equivalent. It is necessary to show the equivalence mathematically. 

In my understanding, the policy update is simply PPO, and the only difference from PPO is the use of the specific form of the reward function. In my opinion, the reward with the indicator function may encourage the policy not to overlap with other policies, but the policy update does NOT have any hard constraint.
In addition, the reward function in Eq. (9) also has the intrinsic reward which does not have the indicator function. The intrinsic reward seems to encourage the diverse behavior, but this term does not have any hard constraint. It is not clear what is really constrained in the  policy update using the proposed algorithm. 

In addition, I'm not sure what is meant by "we also remark that trajectory filtering is closely related to the derivation of the
policy ratio clipping term in Proximal Policy Optimization (Schulman et al., 2017)." I encourage authors to mathematically  show the relation using equations.

If the authors cannot mathematically support the claims I pointed out above, these claims should be removed or modified.
In the current form, these claims are not properly supported, and the contribution is limited to just a reward engineering without theoretical justification.

### Experiment design
In the experiment section, the term "iteration" appears several times, but the definition of the iteration is not clear to me.
In my understanding, policies are trained sequentially in the proposed method, and I guess that a policy is optimized in each iteration. Then, I'm not sure how many environment steps are used in each iteration. If it requires significantly many steps, it is not fair to compare the results of keep running baseline methods. Rather, the baseline methods should be evaluated by repeating initializing and training the policy multiple times. For example, what happens if we run DIPG multiple times?  Another possible easy baseline is running SAC multiple times. (I guess that SAC is more sample efficient than simple policy gradient, and we can run SAC more times than PG with the same number of environment steps.)

### Hyperparameters
Judging from the appendix, hyperparameters such as $\lambda^{int}_B$, $\lambda^{int}_R$, $\alpha$, entropy-coefficient, PPO epochs, batch size are carefully tuned in the experiment. For fair comparison, hyperparameters of baseline methods should also be carefully tuned. In addition, the effect of some importance hyperparameter such as $\lambda^{int}_B$, $\lambda^{int}_R$, $\alpha$ should be analyzed.

Minor comments

- The proposed method is evaluated based on diversity metric, but the results of the average return based on the extrinsic reward should also be reported. As the goal of the proposed algorithm is to obtain the diverse policies that solves given tasks, the average return based on the extrinsic reward is also an important metric.

- I'm not sure whether the results in Tables 2 and 3 are the mean values over multiple trials or a single trial. If the results in Tables 2 and 3 are the mean values over multiple trials, the standard deviations should also be reported. Similar modifications should be made for Tables 6, 7, and 8 in Appendix as well.

 
summary_of_the_review: Although the paper reports some interesting results, some of statements regarding the proposed algorithm do not seem well-supported.
In addition, the experimental section requires some improvement to justify the claims. 

======== Comments after the author response ===========

I appreciate the authors' effort to address my concerns. During the conversation, the authors modified the several parts, which are crucial to support the problem formulation.  However, even after the in-depth discussion, I am not fully convinced that the claims are correctly proved. 
Therefore, I keep the initial score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a method to train diverse policies. New policies are trained to maximize the original reward while also optimized to be distinctive compared to previous policies. This is achieved by switching between optimizing for original reward and diversity reward. Multiple environments are used to demonstrate the proposed method is able to generate policies with distinctive behaviors.
main_review: Pro:
1. Simple approach that can generate policies with distinctive behaviors. Furthermore, a wide range of environments and many baseline methods are used to verify the effectiveness of the proposed approach.
2. The method is clearly described. The idea of trajectory filtering based on the sum of cross entropy and how to make use of rejected trajectories with the intrinsic reward to promote diversity is new to me.

Issue:
1. Some result indicates that the additional intrinsic reward can lead to suboptimal behaviors in terms of the extrinsic result. e.g., in the mujco environments, the 1st policy always demonstrates the best performance, and the subsequent policies can either perform worse or better than previous ones . It will be nice to provide a discussion about this. 
summary_of_the_review: This paper presents a simple and effective approach to generating policies with diverse behaviors. I believe this will be an important contribution to the field.
review_title: null

Meta-review: In this paper, a new method is proposed to discover diverse policies solving a given task. The key ideas are to (1) learn one policy at a time, with each new policy trying to be different enough from the previous ones, and (2) switch between two rewards on a per-trajectory basis: the "normal" reward on trajectories that are unlikely enough under previoiusly discovered policies, and a "diversity-inducing" reward on trajectories that are too likely (so as to push the policy being learned away from the previous ones). The main benefit of this switching mechanism is to ensure that the new policy will be optimal, because the reward signal isn't "diluted" by the diversity-inducing signal as long as the policy stays far away from the previous ones.

After the discussion period, most reviewers clearly recommended acceptance of the paper. One reviewer remained on the "reject" side though, especially due to an unconvincing theoretical analysis of the method, in spite of several back and forth with authors. I also had my own concerns regarding that part after reading the paper, and further discussions with authors eventualy led to a significant rewrite of the corresponding theorems and proofs. I believe the final version (shared in comments by authors after the dealine for paper revisions) to at least be technically correct, though the relevance of the theory w.r.t. practical usage of the method is still not entirely convincing (e.g., assumptions regarding the number of distinct global optima, and the need for positive rewards).

That being said, in spite of these concerns regarding the practical significance of the theoretical analysis, I believe the paper has a strong enough empirical validation, and the method is (1) simple, (2) intuitively reasonable, (3) original due to the trajectory-switching mechanism, which makes me recommend acceptance.
Decision: Accept (Poster)