Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the question of policy diversity in RL, with the specific goal of optimizing the diversity of obtained policies _within_ different quality levels (as one would, for example, want different AI difficulty levels within video games). 

To do so, the authors first propose a framework that describes this specific task, which they refer to as QSD (quality-similar diversity), with an associated QSD score. This score is meant to apply to diversity metrics built upon user-specified behavior descriptors (BDs), as opposed to diversity metrics that are learned as part of the optimization process. 

Second, the authors show that one can obtain unbiased estimates of the gradient of such a user-specified BD with respect to the policy (and thus, an estimate of the diversity metric's gradient). 

Finally, the authors use population-based algorithms to optimize the QSD score, showing across a variety of continuous and discrete tasks that their proposed algorithm (QSD-PBT) outperforms a variety of RL algorithms with different approaches to diversity optimization.
strength_and_weaknesses: # Strengths
- Clarity: this paper is very clearly written; the overall phrasing of the problem, literature review, and description of the methodology are all very clear.
- The experiments verify the usefulness of the proposed approach, and compare to a wide range of baselines.
- The main theoretical results on estimating the diversity gradient may be of independent interest.
- The details in the final QSD-PBT algorithm are investigated with a corresponding ablation study which confirms the discussion in the main paper.

# Weaknesses
- The one section I found could benefit from some clarification is $\S$ 3.3; I found the description of the mechanism by which the different $\lambda_i$ are obtained somewhat unclear. Am I correct in understanding that for each stratification level, all $\lambda_0$ are equal (set to a high value which presumably is not particularly influential)? How does this tie into the reference to $\lambda_{\infty, 1}$ and $\lambda_{\infty, 2}$?
- One potential weakness is that all evaluations of QSD-PBT use the introduced QSD-score metric. I'd be interested in whether the authors considered other ways of combining the stratified diversity terms of (1) (for example, one could consider the worst diversity score across all intervals, although the resulting optimization difficulties may be sufficient to discard this out of hand). 

# Questions
- Maybe I missed this result: do you report the highest quality discovered by each algorithm (regardless of diversity)? On a related note, how are the quality intervals defined for Figure 1: are they predetermined by the range of possible rewards, or are they chosen based on the highest reward over all approaches $R_{max}$?
- Figure 1 shows that there is a trade-off between quality and diversity; however, this trade-off disappears in most of Figure 3. Do you know what this could be due to?
clarity,_quality,_novelty_and_reproducibility: As discussed above, I found the paper to be overall quite clear and well written. The quality of the theoretical and experimental sections seems on par with ICLR expectations. Finally, the paper addresses an interesting and - to my knowledge - new sub-problem in RL. Although the QSD problem could benefit from a stronger motivation, the overall contribution is sufficiently novel. 

Based on my familiarity with the relevant literature, however, I am not certain of the novelty of the proposed algorithm.
summary_of_the_review: This paper addresses the problem of learning diverse policies of differing quality. This paper is well-written, with theoretical results that may be of independent interest, and a novel population-based algorithm that is experimentally verified across a variety of tasks. Although I have a few remaining questions about the experimental evaluation, this is overall an interesting work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies the topic of diversity in reinforcement learning. The authors propose the quality similar diversity problem, whose goal is finding diverse policies within multiple quality levels. In particular, the paper considers task-specific diversity where the user defines certain behavior descriptors (BDs) that are a function of trajectories and capture the type of diversity the user prefers. Given this problem statement, the authors present methods for estimating BDs via the policy gradient theorem. The gradient is then combined with a population-based training via an adaptive diversity loss. This leads to an RL algorithm that optimizes the quality-similar population diversity of a set of policies. Empirical evaluations show improved performance over prior methods w.r.t. diversity of policies across multiple quality levels. 
strength_and_weaknesses: **Strengths**
- The problem statement is interesting and distinct from diversity problems studied in prior work. In particular, this paper aims at finding a set of diverse policies in a hierarchy of quality levels. This is useful in certain AI applications such as games and appears related to curriculum learning tasks. The problem formulation in this paper can be of independent interest to other topics in RL as well.
- Besides the quality-similar diversity problem statement, another technical contribution of this paper is noting that an exact gradient of the user-specified BD based on trajectories can be computed directly using the policy gradient theorem. Thus, one can obtain an unbiased sample-based approximation to the gradient and utilize techniques developed for policy gradient methods.
- Guidelines for adaptive diversity loss and handling multiple quality levels are useful.
- The paper presents a thorough review and categorization of prior diversity methods.
- The paper is well-written and well-organized. The authors provide a solid motivation behind their work and discuss applications and intuition behind their design choices throughout the paper.
- The empirical evaluations show strong performance compared to prior methods in both continuous and discrete tasks. 

**Weaknesses** 

- The setting considered in the paper requires multiple design choices, which brings the practicality of the method into question. Apart from designing rewards, one needs to design a set of diversity design functions that depend on trajectories (which include a large number of states and actions) and may be difficult to elicit desired behavior. Another design element is the diversity measure function (an example is given which is the determinant of a kernel matrix.)
- In Section 3.2, the authors mention a relationship between large $\lambda$, which determines the strength of diversity, and optimistic initial values used for exploration. This connection is unclear and the two seem rather distinct from my perspective.
- Despite the authors discussing this in Section 4 and Appendix, It is still unclear to me whether particular choices of BD would favor certain algorithms. Since BDs can be technically anything, it would be good to also empirically compare the performance of algorithms given random BDs.

clarity,_quality,_novelty_and_reproducibility: The paper is written clearly and the quality and novelty of technical contributions are good. Code is not included in the supplementary material. 
summary_of_the_review: The presented method appears to be useful, practical, simple, and well-motivated. The authors do a good job explaining their method, justifying their design choices, and providing intuition. There are no major weaknesses that would justify rejection.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses task-specific diversity problem, which exhibits in policies with similar expected return (what the authors called quality). The task specificity is framed by a set of scalar functions of a trajectory defined by the user, which are called Behavior Descriptors (BDs). A BD captures some property of a given trajectory, which is preferred by the user, e.g., number of times that a self-driving agent shifts gear (say the less the better). The authors propose an estimator for each BD, based on which they compute the gradient of BD w.r.t the policy's parameters. This gradient is then used to augment standard policy grad in a tunable manner, the result of which is a class of algorithms, called population-based RL algs.
strength_and_weaknesses: Strengths:

1. The main objective (to produce a set of diverse policies at multiple quality levels) seems useful and interesting to investigate. 
2. Formulation of the Quality-Similar Diversity (QSD) is novel and seems to achieve superior results. 

Weaknesses:

1. The paper lacks some analysis of sensitivity as well as computational overhead. In particular, it is not clear how the rate with which $\lambda_i$'s are changed will impact convergence and stability of the algorithm as a whole. Additionally, the choice of $f$ can have significant impact on wall-clock time and computational requirements. 
2. The exposition of concepts can be improved as it lacks accuracy at points, which causes confusion (see the comments below). 
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is *relatively* clear and easy to follow. However, there are points, which pauses/baffles the reader (see the comments below). 

Quality: The submission provides both theoretical exposition of the problem and experimental evaluation of the proposed method. Broadly speaking, it meets the level of an accepted ICLR paper. 

Reproducibility: The authors provide several hyper parameters and detailed general implementations. However, I do not see a reproducibility statement in the paper about code release. 
summary_of_the_review: - Definition of BD needs more explanation. Do you need some restrictions such as being bounded (or finite)? 

- First paragraph of page 3: what do you mean by a measure function? Do you mean a function satisfying to be non-negative, zero for the empty set, and countable additive? If yes, I would probably make it explicit and simply discuss what these properties mean in the case of $f$. If no, then the definition of $f$ does not require it to be non-negative. I am wondering if $f$ can also be negative, then the QSD score can be zero for some given M-partition even if all $\mathcal{B}_i$ are strictly non-zero. This may make the definition counterintuitive. 

- Page 3, first paragraph: depending on the definition of $f$, its evaluation might become a computational bottleneck. Some explanation on computational complexity would be apropos. 

- Section 3, Lines 5-6: “Note that both the user-specified BD and the quality of a policy are defined on a trajectory” -> not correct according to your definition of quality. The return (not the quality) is defined for a trajectory. 

- Page 4, first paragraph: suggestion: for the BD advantage function, you can observe that its expectation over $\pi_{\theta}$ is zero, which simply follows from your definition of value functions. Hence, in practice, it might be quite beneficial to subtract the mean (taken over mini-batch) from BD advantage to make the implementation more robust. 

- Moreover, for computation of advantage functions, I strongly recommend the following rather than GAE:
“Direct Advantage Estimation”, https://arxiv.org/abs/2109.06093

- Section 3.2: For maximizing diversity, section 4 of the following reference also seems relevant:
https://arxiv.org/abs/2110.01548 
(They have an ensemble of Q networks, while you have a set of policy networks.)


Minors:

- Last paragraph of Introduction: the acronym PBT used before it is defined. 

- First paragraph of section 2: your reward function is indeed an expected reward as it only defines over state-action and not the full transition ($r(s,a) = \mathbb{E}_{s’} ~r(s,a,s’)$).

- First paragraph of section 2: $T$ is not defined. More importantly, it looks like that you are considering episodic MDPs (also evident from allowing $\gamma=1$). If so, make it explicit. Moreover, it is not clear whether you have termination at $t=T$ or not. If yes, you need to also define terminal states (whose value is zero by definition). 

- Section 3, 3rd line -> I would say “conventional” rater than “traditional”, since the reference is too recent to be a tradition.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper formulates a problem that the authors term Quality-Similar Diversity, which focuses on achieving diversity among similar-quality policies. The authors propose a population based reinforcement learning approach to address this problem, and evaluate their approach on MuJoCo and Atari domains.

Reviewers were generally positive about the paper, noting the interesting and novel problem formulation with potential for wider impact, the systematic empirical validation and strong empirical results. The theoretical insights that guided the development of the approach were found valuable, as was the thorough overview and categorization of existing diversity measures. The paper was generally found clear and readable.

Initial reviews noted a number of areas for improvement. In particular, specific sections of the paper could benefit from increased clarity (subsections of 3, 4). Reviewers asked questions about the practicality of the method (because multiple design choices need to be made), about the evaluation setup (in terms of the newly introduced QSD metric), and about sensitivity analysis and computational requirements). The authors provided detailed answers during the rebuttal phase. They updated the manuscript to increase clarity, and added additional experiments). Reviewer questions were well addressed, and the consensus is to recommend accepting the paper.
justification_for_why_not_higher_score: This is a solid accept for a paper with a relatively narrower audience. An increase of the score would be on the basis of an argument for more general interest in the theoretical or empirical results, but I don't see a strong case for this.
justification_for_why_not_lower_score: The paper makes a novel contribution in its formulation of quality-similar diversity. It is well executed and systematic.
Decision: Accept: poster