Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a sample-efficient QD environment generation algorithm that maintains a deep surrogate model for predicting agent behaviors in new environments. Results in two benchmark domains show that this method outperforms existing QD environment generation algorithms.
strengths_and_weaknesses: Strengths:
Clean method with experimental results and ablations to demonstrate the effectiveness of their method compared with QD baselines

Weakness:
1. Unclear presentation: 
    a. Figure 1 is confusing with many arrows and no starting point, while the illustration is not intuitive too
    b. Undefined symbols and operations in pseudo-code, e.g. $y$, qd.ask, qd.tell
2. Lack of efficiency analysis: the experimental results only show the evaluation of solutions, but not how many iterations to take in order to get these solutions.



questions: See weakness 2
limitations: 1. Not scalable to some tasks where the measure functions are inconvenient/impossible to define
2. Need a pre-trained agent
3. Need to compare with more baselines, e.g. PAIRED, ACCEL
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper addresses the problem of auto-generating RL environments. Classic environment generation methods could be slow due to expensive and time-consuming simulations. The authors proposed DSAGE, which leverages a surrogate model to predict the behaviors of an agent in new environments. The surrogate model is used to accelerate the generation of environments. The proposed DSAGE is evaluated on two benchmark domains and achieves better QD scores and coverage than the baselines. 

strengths_and_weaknesses: Originality  /  Significance:   
Existing works that rely on QD require lots of simulations which are expensive and slow.  The ideal of using a surrogate model to predict the behavior of an agent and thus reduce simulation time is intriguing and reasonable. This is an interesting combination of behavior prediction and QD approaches.   

Quality:  
The reviewer has some concern regarding the experimental results.   

1.The author claims that the proposed DSAGE could accelerate the environment generation process. However, the claim is not supported by experimental results. In the experimental section, no speed information is reported. It would be helpful to compare the speed of the proposed DSAGE with baselines.   
2.The author claims that diverse auto-generated environments could improve agent’s capability of generalizing to new environments. The claim is also not supported by experiments.   
3. The proposed method is evaluated on two simple domains: grid-based maze and 2D Mario. It is unclear if the proposed approach could salce to more complex domains.   

Clarity:   
The reviewer is not an expert in environment generation but has expertise in RL. However, the reviewer found the paper somewhat difficult to follow. Improvements may be required. Please see the Question section for details.   

questions: - Could the proposed approach scale to more complex environments beyond 2D grid-based maze and 2D Mario?   



- How is the solution quality of the proposed method compared with the classic QD method that needs lots of simulation and long time? The reviewer expected there is a trade-off between solution quality and time. 
- Reporting the time needed for each method would be very helpful. 
- The symbol $\hat{y}$, $\hat{f}$, $\hat{m}$ are undefined. 
- In figure 1, “CMA-ME” and the “Sample latent code” are not explained in the text, which is confusing. 
- The term “ancillary agent” is used throughout the paper without formal definition. It confuses the reviewer while reading the paper. 
- L 139 reads “The tell method adds the solution to the surrogate archive based on the predicted objective and measure values”. How exactly does the tell method add  solutions? Is there a threshold that is used to determine whether to add a solution or not?
- The selection of the objective f and measures m seem critical. How do you make the selections? In addition, in more complex environments, the number of different measure functions may increase. How does the increased number of measure functions affect the performance of the proposed approach? 
limitations: The authors adequately addressed the limitations and potential negative impact in the paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper accelerates the quality diversity procedure for environment generation by using a learned predictor rather than agent evaluations. They add auxiliary losses on occupancy measures to improve the prediction process and a downsampling procedure to decrease the number of candidates.  
strengths_and_weaknesses: ### Strengths:
- This paper appears to be a real accelerator for QD algorithms and the authors carefully demonstrate that each component of the algorithm contributes to the improvement. The algorithm seems very useful.
- The paper is very clearly written.

### Weaknesses:
- The paper is quite similar to a related work and the addition of an auxiliary prediction loss is not a huge novelty.

### A few writing comments:
- In line 138 I’m not sure that “ancillary agent behavior” is a phrase whose meaning is universally known and I’m not sure it was defined before this point? It’s explained in 152 but when I read it on 138 I didn’t understand at the moment and was just confused instead.
- Perhaps I missed it but I didn’t see anywhere where $y$ was defined. I understand that it’s the ancillary but I don’t think that fact is in the paper except in algorithm 1.
- The CMA-ES step seems to be missing from algorithm 1?
- This is a personal preference but given that the method performing better than baselines is the point of the paper, the inclusion of the “hypothesis” in line 222 seems very unnecessary.
- I think either Figure 7 or Table 1 from the supplement should be in the main paper, since improved prediction is one of the main contributions of the paper
- The method seems similar to “Deep surrogate assisted MAP-Elites for automated hearthstone deckbuilding” which is acknowledged in the related work. There the differentiating claim is that that work applies prediction to the decks rather than the agent performance. In that case, it seems the differentiating factor of this work is a series of tools that enable these methods to work in the agent-based case (namely the ancillary information and the downsampling helping with prediction) rather than a new method. This is not super clear from the paper as it seems DSAGE-BASIC is essentially the method from the hearthstone paper? If these are the main differentiating factors, then more analysis of the impact of these tools should probably be in the main paper rather than in the supplement and claiming that a new algorithm has been constructed seems unnecessary. For example, that downsampling improved run-time by 4x is pretty neat and the speedup of the surrogate loop is worth mentioning!
questions: - The agents are stochastic, why are the surrogate predictors deterministic? Would some of the challenges in prediction go away with a stochastic predictor?
- Why would downsampling help with prediction accuracy as it appears to?
limitations: The authors adequately address the limitations.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents DSAGE, a new approach for generating environments that elicit diverse behaviors for a given agent. The authors consider two settings: 1) a maze environment using an RL agent and 2) Mario with an A* planner. In both cases, DSAGE does exactly as intended. In a set of comprehensive experiments the authors show that each individual component in DSAGE plays an important role, which will be helpful for future work. This paper could have a high impact as an approach for evaluating the robustness for RL agents, and thus I believe it is a useful contribution and should be presented at NeurIPS. 
strengths_and_weaknesses: \+ The environments generated by DSAGE are qualitatively diverse, and watching the videos there are clearly some interesting settings that break both of the agents. This provides us with a greater understanding of their respective robustness. 

\+ The experimental design is rigorous with many baselines and ablations.

\+ The work is on an important topic that is currently receiving significant interest. This work could have an elevated impact as a result.

\- I was hoping to see some sort of "summary" of the coverage/success of each agent with different regions of failure. For example, some way to analyze how much of the total environment space can be solved by an agent and what types of environments it typically does well/poorly on. It would be super interesting to then use this as a means to compare agents in the same environment. For example, how does ACCEL compare to other RL agents/planners such as PAIRED or A*? If DSAGE is capable of producing a summary like this, then that would increase its impact and I would move to a stronger accept.

Minor:
- I think the citation for MiniGrid is missing, which is the environment for the Maze experiments.
questions: 1. The environments considered are both 2D navigation tasks where DSAGE is essentially “landscaping”. Does this approach work for other settings such as continuous control of robotic environments?
2. Hate to ask for experiments in the rebuttal, but, is it possible to include a random environment generation baseline (i.e. just Domain Randomization)? I would be curious to see how much variation in behavior occurs just using random samples. 
limitations: Limitations are discussed in Section 8. This meets the requirement.
review_title: null

Meta-review: metareview: The authors describe a method of scaling the benefits of quality diversity (QD) optimization for automatic generation of RL environments, by replacing costly agent evaluation with a learnt surrogate model. They then demonstrate that this method improves agent performance in two settings: a maze environment using RL and a Mario environment using A* planning.

The reviewers agree that the proposed DSAGE algorithm is both novel and technically sound. Where they disagree is on whether QD optimization as a field of research has inherent value for RL research or the broader NeurIPS readership. To summarize each of their respective stances:

R1: "[...] I still agree this method is solid and improves previous QD optimization methods. But I didn't directly see its contribution to a broader community." 

R2: "[...] I have some skepticisms about QD methods in general and prefer unsupervised methods that do not require human specifications of what constitutes an interesting environment. However, it seems unfair to hold this paper responsible for general disagreements with QD style methods and instead seems like the criteria should be whether this is an interesting or useful contribution to researchers who either work on environment design or on researchers who work on QD + RL."

R3: "[...] this is a novel problem setting for the NeurIPS community, combining level generation (from the games community) and RL. It is highly relevant in combination with new work on generalization in RL in particular, and evaluating the robustness of our agents."

I have no great insight on whether QD methods will prove to be valuable for improving agent generalization or robustness in future. But I tend to agree with R3; in my opinion NeurIPS is the right community to be exploring these questions, and this work has made meaningful and rigorous contributions to this particular line of research. I am recommending this paper for acceptance but note that this is borderline and may require reevaluation in the context of other submissions.
confidence: Less certain
Decision: Accept