Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors Proposed A Web IDE  to

i) decrease the complexity of environment generation using PCG
ii) simplify the build process
iii) reproducibility

which allows: 
i) design and debug arbitrarily, complex PCG grid-world environments 
ii)  visualize, evaluate, and record the performance of trained agent models in the Web 


The work is a useful one. However, I think the justifications can be heavily improved to put this work in a better context with the current related works. Also, It is unclear whether it is possible to incorporate other environments and if possible how much engineering effort would be needed. Even in the case, that further incorporation of non-grid-world environments is not possible, the manuscript should outline what would be the implications of that. I am looking forward to the rebuttal and am open to changing my rating. 



strengths: 
i) An IDE towards generating an environment and evaluating agents, which is of practical use 
ii) Ability to collect human trajectories
weaknesses: i) Only supports Grid Environments. In section 2.3 they make a good case for why the grid world is important. However, I think to improve the manuscript it should be discussed that, what RL aspects the paper/IDE currently cannot cover due to this focus on the grid environments. Also, see the related following point.

ii) I understand the "portability" a web IDE gives, or how it can work as a central point, however, does the choice of Web IDE limits the supported environments? a) How much effort would be needed to support environments like strategy games like StarCraft [1], Minecraft, RObotics, or multi-agent sports environments like Soccer [2], which might require much more "graphical" capabilities than grid world environments? b) Why a Web interface is preferable over A traditional Python package/ software? I think this should be discussed in the paper and also included in the introduction at least briefly. 

[1] Vinyals, Oriol, et al. "Starcraft ii: A new challenge for reinforcement learning." arXiv preprint arXiv:1708.04782 (2017)

[2] Kurach, Karol, et al. "Google research football: A novel reinforcement learning environment." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020.

iii) A human study on hinting how it can help different aspects of an RL researchers productivity would be a very strong addition to the paper. 
correctness: N/A
clarity: 

Mostly the paper is well written. 
I have one point: 

i) Does it Only allow agents that are written in Tensorflow and/or converted to TensorflowJS ?. I am not entirely sure about this point. Can PyTorch models be used too? 


A few other clarity points are discussed in the other sections of this feedback. 
relation to prior work: 
1) Previous works have also focused on using Probabilistic programming languages/DSLs with similar motivations such as generalization, ease of environment generation, and compilation of the environments abstracting away lower level details. I think the manuscript can be improved by adding a small section making a comparison to other similar works e.g., PCG/DSL for Soccer[1], NetHack[2] (which is mentioned in the paper) on i) The choice of the DSL and  How it differs, ii) on the level of abstraction, iii) mode of use iv) user friendliness/capabilities of the DSLs.

[1] Azad, Abdus Salam, et al. "Programmatic Modeling and Generation of Real-Time Strategic Soccer Environments for Reinforcement Learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 6. 2022. 

[2] Samvelyan, Mikayel, et al. "Minihack the planet: A sandbox for open-ended reinforcement learning research." arXiv preprint arXiv:2109.13202 (2021).

additional feedback: 1) 
Line 32-35 
" developing new environments that provide the necessary challenges for RL methods
 is a costly process, requiring deep expertise in software engineering, high performance computing
(for efficient distributed training), and game design.  ...This skillset is closer to that of a videogame
35 developer than a typical machine learning researcher. " 

--- the way it is written, it seems to be claiming that PCG algorithms can already generate game levels that are close to that are generated by Video game developers and better than what SOftware developers can generate. I would recommend to Either substantiate  this with references or rephrase to avoid accidentally making claims that are not really in scope of the paper. 

2) Line 38-39 " i.e. creating probabilistic programs
39 that specify distributions over environment configurations, adds considerable engineering overhead to". Please add references, recently efforts have been made to make environment generation easier using Probabilistic Languages for gridworld or real time games providing compilers and helper libraries.
review_title: A useful and practical IDE  for RL, but requires better justifications to put it in better context  and/or broader scope 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work introduces the GriddlyJS platform, a web-based IDE for RL based on the previously introduced Griddly engine. The web-tool provides functionality to design, test, and analyse RL environments based on configuration files for the Griddly engine. The tool also provides functionality to visualise, evaluate, and record human and agent episodes which can be used to analyse trained agents, better understand tasks, and publish interactive policy demonstrations. As an example, an online version of the Crafter environment is implemented and released in GriddlyJS, and an experiment demonstrates that manually defined tasks in the new environment are significantly more challenging than procedurally-generated, random environments.
strengths: I believe a tool like GriddlyJS could greatly impact the research field of RL by (1) simplifying the process of designing new tasks, (2) understanding tasks by interacting with them, (3) simplifying analysis of trained policies with its step-by-step rendering tool in modifiable tasks, and (4) allowing to publish informative demonstrations of agents. This makes for great potential value and significance of GriddlyJS and is also novel to the best of my knowledge. I believe the design, manipulation and analysis of existing environments is likely the most impactful and important aspect of GriddlyJS.

The UI of GriddlyJS is intuitive and simple with most functionality easily accessible.

weaknesses: 1. In line with its potential and rich feature set, it appears essential to provide detailed documentation and a simple process for its features. However, I find the current documentation despite its simple UI lacking with several features mentioned in this work missing descriptions. See two main unclear components below with less pressing issues listed under the documentation section of this review.
    
    1.1. TF.js models: The publication and visualization of agent policies for reproducibility, debugging, and analysis is presented as a core feature of GriddlyJS. However, it is not explained how TF.js models can be integrated within GriddlyJS.
    
    1.2. Multi-agent support: Griddly supports multi-agent environment and this work also mentions support for multiple agents. How can these multiple agents be controlled in the human “play” interface? It appears only the first agent is controlled with the assigned keybindings and other agents stay static while episodes go on as long as any, even static, agents are still alive (tested in Crafter and Spider environment with manually multiple agent entities being placed in the environment).

2. The evaluation presented in Figure 6 appears confusing to me.

    2.1. It appears that the randomly generated DR levels are already solved with ~50% before any training and no progress is apparent throughout the entire training. This suggests that 50% of the DR generated tasks are trivial (highlighting the relevance of human-generated tasks) but also indicates that no significant training is exhibited throughout the entirety of training rendering the learning curve invaluable. If I correctly interpret these results, the claim might also be that a random policy solves 50% of DR-generated levels while human levels are only solved with a significantly smaller probability.

    2.2. The PPO agent only seems to be trained for 300 updates which appears like very little training. Is it correct that each update is computed with batches of 512 steps (PPO rollout length) x 64 (64 PPO workers) for a total of ~10M timesteps?
correctness: To the best of my knowledge, the claims in the paper appear correct. I have some concerns and questions regarding the evaluation presented in Section 4.2 - see weakness 2. for more details.
clarity: The paper is clearly written. I found a single typo in l. 182 (“... added to the it.” --> “... added to the grid.”).
relation to prior work: The work does a good job of explaining the underlying Griddly engine and positioning the work within related systems.
additional feedback: I believe the usefulness and impact of GriddlyJS largely depends on its ease of use to lower the barrier of entry. In line with this thinking, I would like to propose two features which I believe could greatly simplify the use of GriddlyJS and improve its adoption:
1. It would be useful to provide an option to directly download created environments as gym environment (e.g. as .py file which registers a gym environment) and collection of environments within a project. It appears that GDY files could be copied into a file and loaded with Griddly but automating this process could simplify this process significantly and improve GriddlyJS’s value as an IDE for RL environments.
2. Designing new environments rather than new tasks within pre-existing projects within GriddlyJS still requires some expertise with GDY. Its interface is fairly intuitive and I commend its authors for its design, but it might be possible to simplify the generation of new environments by providing blueprints/ prototypes for objects with common features. Example could be collectible objects and locations which give reward (positive or negative) and potentially terminate episodes, movable or static enemies and similar generic concepts. These could be loaded and easily modified but serve as a starting point for environment designers to use. The Crafter environment is a complicated example with large amounts of logic, but its GDY files shown in GriddlyJS has about 2400 lines to specify all objects, logic, their appearance etc. Designing such environments currently still requires significant expertise with Griddly and would likely take large amounts of efforts and time.
review_title: An intuitive and useful tool for RL research which needs some more documentation

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces a new Web intelligent development environment (IDE) for creating gridworld environments, GriddlyJS. GriddlyJS is an easy-to-use IDE for creating procedurally generated gridworlds, where the environment designer can edit and modify these gridworlds using code or a GUI. Users can also create a set of hand-designed gridworlds for reinforcement learning algorithms to learn on. GriddlyJS is different from Griddly in that it is a frontend for accessing Griddly in an easier way.

strengths: The paper highlights the importance of procedurally generated content (PCG) in reinforcement learning research. In particular, the paper outlines why PCG is important for generalizable and robust RL algorithms. The paper also provides strong evidence for why gridworlds are an important tool for testing the generalizability of RL algorithms, which is useful to support the need for GriddlyJS.

GriddlyJS turns the Griddly into a somewhat no-code framework, which can increase the accessibility of creating RL environments.  Users who have little coding experience can more easily create content for RL research.

weaknesses: The level to which GriddlyJS is a contribution is unclear.

The main argument the paper presents is that environment construction for RL research is difficult since it requires the skills akin to video-game development. To that end, the paper presents GriddlyJS, which can help mitigate the need for these skills.  Although I am not familiar with Griddly, it seems like Griddly is a framework that already mitigates the needs for video-game developer skills.  How does GriddlyJS make working with PCG content easier than Griddly does, besides using a GUI? It seems as if GriddlyJS is simply a graphical extension/update to Griddly.

Ideally, a new benchmark would provide researchers with a new way to characterize or test algorithms for which existing benchmarks cannot be used. This does not seem to be the case with GriddlyJS.

The experiment on PPO was not described in detail. How were hyperparameters chosen? How many runs were used? Furthermore, I am unsure of how the experiment differentiates how GriddlyJS can show the out of distribution generalization of PPO better than Griddly can since, from my understanding, environments can also be hand-designed with Griddly (just not using a GUI). 

Overall, GriddlyJS seems more like a software engineering project than a research contribution. GriddlyJS may be better suited to the JMLR open-source ML software track than to the Neurips benchmark track.

correctness: The claims made in the paper are mostly correct. As I previously stated, I am not very familiar with Griddly, but it seems as if some of the characteristics attributed to GriddlyJS should actually be attributed to the Griddly backend.

clarity: The paper is well-written and easy to follow.
relation to prior work: The paper thoroughly discusses the relation of GriddlyJS to other PCG environments for RL research. The paper highlights how GriddlyJS differs from these other benchmarks.

additional feedback: The following issues did not affect the scoring of the paper:

Line 143: "GDY" has already been defined above.

Line 294: Missing period at sentence end
review_title: Useful Software, but is it a Benchmark?

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: - The paper introduces a WebAssembly-based wrapper around the `Griddly` framework for grid-world environments.
- The tool enables the definition, play and recording of trajectories of grid-based RL environments directly in the browser.
strengths: - Lowers the barrier to create novel grid-world environments.
- Grid-world environments are a good testbed for RL as they present a moderate complexity that can easily be tuned and become more complex.
- Procedural content generation in `Griddly` enables the definition of interesting RL environments with test and training distributions which can easily be tried and shared. Though this is rather a property of `Griddly` and not its web wrapper.
weaknesses: - Grid-world environments are only relevant to certain kinds of RL researchers, which exclude continuous control or environments with more complicated dynamics. However, this limitation does not speak against its acceptance.
- Debugging of a policy online does only work for the `Grafter Escape Rooms`  environment. Locally this works only with ONNX exportable models. This is the major weakness for a tool to test RL agents. 
- Automatic performance evaluation of the policy on a set of levels would also be nice to have and aid reproducibility
correctness: - The tool is presented in a sound way, though a major drawback is that evaluating an RL agent does not work in a simple way. This is not apparent from the paper.
clarity: - The paper is well written and has a good structure.
relation to prior work: - The paper gives a nice overview over the different approaches to procedurally generated grid wolds.
additional feedback: - Link in the help to the model directory (https://github.com/Bam4d/Griddly/tree/level_editor/js/griddlyjs-app/public/model) is broken.
- In Firefox on MacOS, the user has to scroll to see the whole UI. Its elements should be responsive to prevent this.
review_title: The paper presents a useful wrapper for the web around the `Griddly` framework to develop grid-based environments, but lacks the possibility to properly evaluate RL agents.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper presents a web-based integrated development environment (IDE), GriddlyJS, especially for Gridworld reinforcement learning (RL) environments. It abstracts the details of implementing RL environment that requires skillsets of game developers and make RL researchers quickly design new environments by themselves even without such skillsets. By experimental comparison between procedurally generated levels and human-designed levels, it is demonstrated that human-in-the-loop level design is helpful to make RL agents robust. Furthermore, GriddlyJS streamlines the RL research procedure consisting of environment development, agent evaluation, and publication so that empowers research productivity and reproducibility in RL society.
strengths: 1. The introduced web-based IDE, that can be run in any web browsers with visual interface, is high-quality and easy to use, so it would be helpful for from beginners to experts in RL society.
2. Both the documentation website and Github repository are well-organized with good examples.

weaknesses: 1. The experiment is conducted only on a single environment (Figure 6). Further analysis on various environments would be helpful to highlight the difference between DR and human-designed levels.
2. The authors expect that human-in-the-loop environment design would mitigate out-of-distribution problems and make RL agents robust, but no experimental result has provided. If some experimental results are provided at least on a toy domain (as following the manner described in line 280), it would emphasize the usefulness of human-in-the-loop level design.
3. As described in line 342, GriddlyJS does not currently persist user-generated data on a dedicated server, but I think it is very important to guarantee that GriddlyJS supports active sharing among RL researchers.

correctness: Most of claims sound correct, but more experiments on the usefulness of human-in-the-loop level design would be helpful as in second paragraph of weakness.
clarity: The paper is well-written and easy to follow.
relation to prior work: The authors clearly discussed how this work differs from previous works in section 5.
additional feedback: - Question
    * Is it available RL agent with codebase written in PyTorch framework on GriddlyJS? Since both of TensorFlow and PyTorch framework are widely used by RL researchers, support (maybe in the future) for both frameworks would be very helpful.

- Minor typo
    - In section 4.3, line 294, “wide range of downstream use cases.” (missing “.”)
review_title: Handy and useful web-based IDE for Gridworld RL environment development

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents a web IDE, GriddlyJS, developed using React.js for reinforcement learning researchers to easily design and debug grid-world environments. It is based on the Griddly grid-world game engine. The tool also provides a user interface to visualize agent’s behavior, record trajectories and show the recorded replays. It allows users to design environments with procedural content generation, by designing environments of different levels, to evaluate agents’ robustness and generalization. The paper also includes a walkthrough of building such an environment as an example.
strengths: - Clear motivation, fully working system prototype and example.
- The web IDE provides a nice user interface, where I can easily create or edit a level of the provided environment with the visual editor.
- Backed by Griddly, the created environment can be easily transferred into Python for RL algorithm training.

weaknesses: - The system could be further optimized. E.g. I tried to add new level or delete the level, it takes quite a long time to process.
- To evaluate the agent's performance, we have to run the IDE locally and provide the model via ONNX format, which is not convenient.

Minor
- Typo around 748-749
correctness: Most are correct to my knowledge.
clarity: The paper is clearly written.
relation to prior work: Related works are clearly discussed.
additional feedback: 1. I notice that the example environment has a very long GDY file (over 5k lines). Maybe some of them are auto-generated (like the maps of different levels), but some need our efforts to code, right? 
I can imagine this would be a very challenging and time-consuming task for environment designers.
Would you compare how much efforts needed using your IDE with other tools? E.g., I could also use the MiniHack Level Editor to graphically put objects on the map and the relevant codes are generated automatically.

2. Do you plan to provide some baseline results?
review_title: A useful grid-world environment design tool via Web

Meta-review: metareview: The paper describes a web interface to design RL grid worlds based on the griddly engine.  This is a useful tool that lowers the barrier for experimenting with various RL grid worlds.  The authors addressed the main concerns of the reviewers who unanimously recommend acceptance.
confidence: 5: The area chair is absolutely certain
Decision: Accept