Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: An important problem in RL is to compute a diverse set of strategies that optimize the (total) reward. One strategy is to simultaneously optimize over all strategies, as with population-based training (PBT), but this can be computationally challenging. An alternative is to use a greedy algorithm, as with iterative learning (IL), which iteratively learns a new policy that is sufficiently diverse from all policies computed so far. The paper sheds light on the performance guarantees and the convergence properties of IL methods. Furthermore, it argues that, at least in the scenarios considered in the paper, a diversity measure based on state distances is more meaningful than action-based diversity. The authors combine IL and state-based diversity measures, and propose a new framework, State-based Intrinsic-reward Policy Optimization (SIPO), for discovering diverse RL strategies in an iterative fashion. For optimization, they use two-timescale gradient descent ascent (GDA), which comes with convergence guarantees. SIPO is empirically evaluated on two challenging multi-agent environments, namely, Star-Craft Multi-Agent Challenge and Google Research Football, where it is able to discover diverse and human-interpretable strategies.

strength_and_weaknesses: Strengths
- The work provides several interesting insights: (i) the fact that IL can in fact achieve rewards similar to PBT by just trading-off half the diversity; (ii) the fact that IL can converge easily to diverse solutions by imposing a large diversity constraint, as opposed to PBT which suffers from slow convergence (or even divergence); (iii) the fact that in certain scenarios action-based diversity may produce similar policies, which motivates the authors to introduce a state-based diversity measure.
- The connection of SIP with intrinsic rewards is interesting. Intrinsic rewards and curiosity are often used to signal whether the agent has encountered a known or surprising state, so they fit well with the state-based diversity. 
- SIPO comes with theoretical guarantees (when coupled with GDA).
- The two multi-agent games considered in the experimental evaluation are quite complex, and SIPO is able to achieve good results.

Weaknesses
- I understand that in the scenarios considered in this work, a state-based diversity measure is more appropriate. However, is this always the case? For example, one case where a state-based diversity measure would not make sense is when there are many irrelevant states. In that case, even if a new policy is quite diverse in terms of the irrelevant states, it may not be much more interesting than the existing policy. Do the authors make the claim that state-based diversity should always be preferred to action-based diversity? That would be a strong statement, and I am not sure enough evidence is provided in the test for such a claim.
- The authors claim that the strategies discovered by SIPO are not only diverse but human-interpretable as well. I do not dispute that this is the case in the studied scenarios. But I do not see how the proposed framework can generally guarantee interpretability. Is it sufficient to just enforce state-based diversity for this purpose? I do agree that SIPO would be expected to result in diverse policies, but I personally fail to see how human-interpretability can arise out of SIPO.
- In Table 4, the authors mention the number of visually distinct strategies discovered by different methods. Maybe I missed that in the text, but how did the authors come up with the exact number? This seems a very subjective metric, since different observers may report a different number. I was wondering whether some other methodology was followed instead.
- When it comes to comparing SIPO with other competitors (including RSPO), I am not sure that the current empirical evaluation does a very convincing job. Tables 3 and 4 indeed show improved numbers for SIPO, but RSPO also performs quite well (e.g., in Table 4). The empirical analysis shows to some extent that SIPO produces diverse policies, but I am not so sure that it shows very strongly that SIPO is consistently better than RSPO.
clarity,_quality,_novelty_and_reproducibility: - The paper is not hard to follow and presentation is quite good.
- Even though I did not check all proofs in detail, the theory seems to be sound. One issue is that the papers make some strong statements that are not necessarily backed by significant evidence (see particularly first and second weaknesses above). I think more precise statements would strengthen the paper.
- The empirical analysis shows that SIPO can indeed learn diverse policies; however, I am not convinced that it provides strong evidence for the consistent superiority of SIPO compared to RSPO.
- The work is quite novel, in the sense that it provides several interesting insights about IL (vs. PBT), and additionally it is the first work (to the best of my knowledge) to rigorously analyze the convergence properties of IL with GDA. 
- The authors provide details on experimental settings and hyperparameters to facilitate reproducibility.
summary_of_the_review: Overall, I tend to be positive about the work because it provides various a number of interesting insights together with a rigorous analysis of an IL algorithm (SIPO with GDA). The results seem to confirm that SIPO can compute diverse policies in the multi-agent games of the evaluation. That said, the paper makes various strong claims, which may not be fully substantiated in the text. It would make sense to revisit some of these statements to increase precision. Furthermore, even though SIPO can get slightly better numbers than RSPO in the evaluation, I have the feeling that the empirical analysis does not provide very strong evidence for the superior performance of SIPO compared to RSPO.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Combining iterative learning (IL) and the state-based diversity measure, this paper proposed a new framework called a powerful diversity-driven RL algorithm, State-based Intrinsic-reward Policy Optimization (SIPO), with provable convergence properties. IL repeatedly learns a single novel policy that is sufficiently different from previous ones. The paper proved that, for any policy pool derived by PBT, they can always use IL to obtain another policy pool of the same rewards and competitive diversity scores. In addition, they also present a novel state-based diversity measure with two tractable realizations, which can impose a stronger and much smoother diversity constraint than existing action-based metrics. It has been verified experimentally that SIPO is able to consistently derive strategically diverse and human-interpretable policies that cannot be discovered by existing baselines.
strength_and_weaknesses: a)	Strength:
This paper addresses a very important problem in reinforcement learning: policies with similar rewards may have fundamentally different behaviors. Therefore, the authors propose SIPO to discover as many different policies as possible. This paper objectively expounds on the disadvantages of the PBT method and proposes a very novel method that mathematically gives a state-based diversity measure and proves the convergence. Through experimental results, they find that SIPO is able to consistently derive strategically diverse and human-interpretable policies that cannot be discovered by existing baselines. The graphs are very clear. Figure 2 compares the training process of SIPO and PBT from a geometric perspective, which proves the advantages of SIPO and gives a lot of inspiration. The RL environments are also complex, which can fully prove the ability of the SIPO, and provides a well-made demo.

b)	Weaknesses: 
When measuring Action-Based diversity, the author claims that A can be any distance metric. For different distance metrics, theoretical analysis is no problem, but will it affect the final experimental results? Maybe the author can consider adding these Experiments and Demonstrations. There are also many improvements to the PBT method in the future, perhaps the author can consider making further comparisons to reflect the advantages of SIPO.

clarity,_quality,_novelty_and_reproducibility: Clarity: The author has a very clear display of the structure of SIPO,  and the graph is given from a geometrical point of view, which is very easy to understand, and the description of the experimental settings and details is also relatively clear.
Quality: The charts are well made, and the visual effects of the experimental environment constructed are also very good.
Novelty: SIPO solves the problem of discovering diverse high-reward policies in complex RL scenarios of PBT and gives an explanation from a novel geometrical perspective.
Reproducibility: The author provided the original data, graphs, and code, the reproducibility is worthy of recognition.

summary_of_the_review: This paper addresses the problem of discovering diverse high-reward policies in complex RL scenarios. An in-depth comparison of PBT and IL is carried out, and it is found that IL is easier to optimize and can obtain comparable solutions to PBT. Furthermore, the authors illustrate the necessity of combining IL with a diversity measure defined on the state distance through specific failure cases of action-based diversity measures, and finally propose a state-based intrinsic reward policy optimization algorithm (SIPO) and prove that The proposed framework and the analysis of the diversity measure will bring a lot of inspiration to future work on policy diversity and PBT.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper explores methods for discovering diverse policies for RL problems, specifically focusing on MARL. This is well-motivated, as these types of methods have become increasingly prominent in recent years and played a part of large scale successes such as AlphaStar. The paper then disappointingly focuses on a new algorithm "SIP" which is rather similar to many other similar algorithms in the space. My suggestion would be to focus on the analysis of design choices and produce a useful paper for the community, rather than trying to win on some benchmarks which makes the results hard to trust and subsequently transfer. 
strength_and_weaknesses: ### Strengths

- This paper covers many very interesting areas, for example comparing state vs. action representations and parallel vs. sequential population learning. If these were executed in a more rigorous way it would make a great contribution.
- The results in Figure 6 are interesting. 
- The experiments are in relevant and reasonably high dimensional tasks.
- Related work seems to be correctly cited.

### Weaknesses

- The biggest weakness here is that the paper is framed as a new algorithm to rule them all when it really doesn't need to be. The actual algorithm isn't particularly novel, it is very similar to approaches like DOMiNO (Zahavy et al), which train policies using state occupancy metrics. Instead, if the authors focused on analyzing 1. Parallel vs. Sequential population training and 2. State vs. action representations then this would be a good paper I would accept. 
- I think it is wrong to say prior work "typically" use action based representations for diversity. This is really just DvD and TrajeDi. Many previous works, including the original Novelty search, DIAYN etc have used diversity in the state space/trajectories. It is not super novel to switch back to states.
- The failure mode of action based metrics presented in Figure 3 is not necessarily true. In DvD, the actions are compared for *the same states*, while here it is for different states. The only state where the policies take different actions here is s0, otherwise it is possible that both policies act the same way everywhere else. So concretely, I disagree with the L2 norm being 2\sqrt{2} for D(\pi_1, \pi_2) in the case where the states are sampled from a joint distribution of the state visitation from both policies. 
- How does bolding work in the tables, e.g. Table 4 & 5? It seems like in both cases there are multiple cells with the same value but the one with your method is bolded and the others aren't. Seems totally unnecessary to try and look like you "win". 

Minor issues (did not impact score):
- Section 3 title, "Preliminary" -> "Preliminaries"
- Sec 4.1 "motivation example" -> "motivating example"
- Sec 4.1 "worst case in 1-D setting" -> "in a 1-D setting"
clarity,_quality,_novelty_and_reproducibility: The ideas here are not novel, it is all taken from other recent papers and recombined. The analysis could be novel, but it is currently only a secondary contribution. 
summary_of_the_review: Nothing in this paper is new and that is totally fine. But, what I dislike is the approach of claiming to invent a new algorithm that "wins" on a specific benchmark chosen by the authors and making the paper all about this algorithm. Instead, if the authors reframe this paper as an analysis of joint vs. sequential population training algorithms, and a comparison of state vs. action representations, with theory and experiments showing this trade-off then it would be a useful contribution and I'd likely vote to accept. The paper I just described would be interesting to many, more broadly useful, and might also get cited a fair amount.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a simple Iterative Learning algorithm and distance metric, and applies it to two MARL domains. The algorithm and metric are not particularly novel, but the analysis is a nice touch. However, the method might be limited to very particular environments and state representations. Two reviewers mentioned the limitation with state-based similarity measures, and did not (IMHO) get satisfying responses from the authors. 

Reviewer GEJD has some valid points, particularly when it comes to how the paper is framed (a thorough analysis of existing algorithm components would have been preferably to claiming novelty and superior performance at all costs). Disappointingly, the authors seem to not really understand this reviewer's criticisms, and their answers are somewhat beside the point (for example, GEJD is not arguing that the problem with iterative learning is that big companies dont use it, which the authors seem to believe). At the same, reviewer GEJD's comments sound a little bit they wanted the authors to have written a different paper than they did, which is not a valid critique.

Still, the argument from GEJD that the algorithm would not handle procedurally generated environments is important. It's not only about the visual input, but the fact that states can be very different even for identical action sequences.

I am a little bit surprised that the authors do not mention quality-diversity in their related work, given that what their algorithm is arguably a QD algorithm. A few QD papers are cited, but erroneously referred to as "multiobjective", suggesting that the authors have not digested that literature. 

On balance, I think this paper has some value despite limited novelty, but the author's somewhat disingenuous engagement with the reviewers together with the strange non-discussion of QD leads me to believe that the authors need to be engage a bit more honestly with both reviewers and related work.
justification_for_why_not_higher_score: I am not entirely happy with the responses to reviewers nor with the treatment of related work, and the work has limited novelty.
Decision: Reject