Title: Playing SNES in the Retro Learning Environment
Authors: Nadav Bhonker
Shai Rozenberg
Itay Hubara
Venue: ICLR 2017
Keywords: Reinforcement Learning
Deep learning
Games

Abstract: Mastering a video game requires skill, tactics and strategy. While these attributes may be acquired naturally by human players, teaching them to a computer program is a far more challenging task. In recent years, extensive research was carriedout in the field of reinforcement learning and numerous algorithms were introduced, aiming to learn how to perform human tasks such as playing video games. As a result, the Arcade Learning Environment (ALE) (Bellemare et al., 2013) has become a commonly used benchmark environment allowing algorithms to train on various Atari 2600 games. In many games the state-of-the-art algorithms outperform humans. In this paper we introduce a new learning environment, the Retro Learning Environment — RLE, that can run games on the Super Nintendo Entertainment System (SNES), Sega Genesis and several other gaming consoles. The environment is expandable, allowing for more video games and consoles to be easily added to the environment, while maintaining the same interface as ALE. Moreover, RLE is compatible with Python and Torch. SNES games pose a significant challenge to current algorithms due to their higher level of complexity and versatility.

TL;DR: Investigating Deep Reinforcement Learning algorithms in a new framework based on the SNES gaming console

Review discussion:
I'd appreciate a bit more context on the motivation for why SNES is an appropriate next benchmark collection, apart from "more games". Do you foresee novel AI challenges that could not be studied on any of the existing benchmark domains? In the same vein, maybe cast a somewhat wider net on related work: there are many other existing and studied game-based benchmarks (off the top of my head: Starcraft, Project Malmo, GVGAI, TORCS, ...) -- try to justify the niche you're filling with SNES.
Thank you for your comment, we will revise the paper with a more thorough explanation of the SNES advantages and featured challenges in comparison to other game-based environments (including the new Universe).
In comparison to Atari, SNES games presents numerous new challenges.
For instance, the need of strategy and avoiding greedy policies, the combination of several tasks  at once (e.g., detecting objects and solving a maze and others).
In comparison with Universe, we present two major advantages:
The first is the speed which you can ran the simulation (hundreds of fps vs 60 fps).
The second is our ability to modify and view the game's internals. This allows to train agents with RAM observation and to modify the game's internals (for example, in the game Mortal Kombat, we randomly initialize the positions of both agents).
In comparison with other single-game environments (Project Malmo, Infinite Mario, etc.) we provide a single environment, encapsulating many games.
Since games complexity increased through time, we chose the SNES for the preliminary version of RLE so that the challenges will be gradual, and for its wide variety of over 700 games, each will take about an hour to add to the environment.
Support of more advanced game consoles will be added in the future with the hope of leading to algorithmic breakthroughs much like ALE did.
The authors list"A new benchmarking technique, allowing algorithms to compete against each other, rather than playing against the in-game AI" as a contribution, but there is a severe lack of detail on how this works.
I am referring to Section 4.3.
Firstly, from what I can tell, in both cases of the rivalry training uses the in-game AI, is this true? If so, the statement in the intro is misleading.
From 4.3: "The second use was to initially train two agents against the in-game AI, and then resuming training while rivaling one another, and evaluating against the in-game AI separately." Can you elaborate on "resuming training while rivaling one another": won't this only work in multiplayer games where a second agent is controllable by DQN?
For example, what is the "in-game AI" for Super Mario?
In 4.3.1, all that's listed is win ratios but not the score each obtained against the other: what are the scores? In this second rivalry training: has the objective changed (i.e. maximize probability of winning).. or are the agents still trying to maximize reward?
Thank you for your comment, we will revise the paper and emphasize the different experiments done in the "Rivalry Training" section.
The rivalry training refers to multiplayer games only (currently we have been experimenting on Mortal Kombat, but we intend to investigate other games as well).
In the second rivalry use you referred to, we trained an agent against the in-game AI initially and measured its performance.
Then, we resumed it's training against another pre-trained agent (also trained against the in-game AI) with the hope of improving its performances.
We observed that the agent tends to learn the best policy against its current opponent.
Therefore, the agent's performances against the in-game AI, after being training against an agent of a different algorithm, were inferior to those of an agent who was trained solely against the in-game AI.
Following this observation we decided conduct another experiment in which the agent was trained against both the in-game AI and the additional agent iteratively, switching the opponent after each episode (rather than a large number of epochs against one following the same against the other) with the hope of learning a better generalized policy, the results of this experiment will be incorporated in our next revision.
The first rivalry use is intended to be a new form of benchmarking inspired by real world competitions in which players are measured by the outcome of the match (win or lose) rather than the actual score.
This method is intended to compliment the standard benchmarking method (measuring their scores against the in-game AI), by measuring whether the algorithm overfit the given problem, similarly to the results that appeared in the second method.
In both cases we use the in-game AI but the evaluation of the agent's performances is done either against the in-game AI or against another agent. We will remove this ambiguity from the paper.
Hi,
A couple questions:
1. Aren't there legal issues related to ROM usage?
2. How would you relate your environment to DeepMind's Lab and OpenAI's Universe that just came out last week? (I guess you'll need to update the paper to account for them).
In particular do you plan to integrate your environment within Universe?
Thanks!
1. As far as we're aware, using ROMs in RLE is no different than using ROMs in ALE.
2.
Regarding other environments, which were published after our submission, such as universe (OpenAI) and LAB (DeepMind). We argue that our environment incorporates a large number of games and has the ability to integrate advanced consoles.
Nevertheless the amount of work invested in creating new environments, by other leading research groups, highlights the importance of this work.
We indeed intend to integrate our work within OpenAI's framework (already sent a request).
However, we find Gym more suitable as our environment can run on any computer and doesn't require to connect to a server via a vnc port.
This paper presents a valuable new collection of video game benchmarks, in an extendable framework, and establishes initial baselines on a few of them.
Reward structures: for how many of the possible games have you implemented the means to extract scores and incremental reward structures? From the github repo it looks like about 10 -- do you plan to add more, and when?
“rivalry” training: this is one of the weaker components of the paper, and it should probably be emphasised less. On this topic, there is a vast body of (uncited) multi-agent literature, it is a well-studied problem setup (more so than RL itself). To avoid controversy, I would recommend not claiming any novel contribution on the topic (I don’t think that you really invented “a new method to train an agent by enabling it to train against several opponents” nor “a new benchmarking technique for agents evaluation, by enabling them to compete against each other, rather than playing against the in-game AI”).
Instead, just explain that you have established single-agent and multi-agent baselines for your new benchmark suite.
Your definition of Q-function (“predicts the score at the end of the game given the current state and selected action”) is incorrect. It should read something like: it estimates the cumulative discounted reward that can be obtained from state s, starting with action a (and then following a certain policy).
Minor:
* Eq (1): the Q-net inside the max() is the target network, with different parameters theta’
* the Du et al. reference is missing the year
* some of the other references should point at the corresponding published papers instead of the arxiv versions
Thank you for your review and comments.
Q: Reward structures: for how many of the possible games have you implemented the means to extract scores
A: All the games we've implemented include the reward function and can be learnt.
Implementing a game is a process very similar to that in Universe. Adding a game consists of four steps:
1. Create a new file for the game and adding it to the game list.
2. Defining the minimal meaningful possible actions for a game.
3. Defining the starting actions to be performed from initialization until reaching the game screen.
4. Defining a reward and terminal function by extracting the score from the game's RAM.
In the following weeks we intent to add several games. A guide describing the process is also available in our wiki: https://github.com/nadavbh12/Retro-Learning-Environment/wiki/How-to-add-a-new-game
Additionally, we've added support for new consoles: Sega Genesis/Mega Drive, Master System, and several others.
Q:“rivalry” training...To avoid controversy, I would recommend not claiming any novel contribution on the topic...
Instead, just explain that you have established single-agent and multi-agent baselines for your new benchmark suite.
A: Thank you for pointing that out. We revised the paper and added cited related works. The simplicity of multi-agent training using RLE enabled us to run some interesting experiments. We hope that this will encourage more research in that direction.
We fixed all your minor comments as well - thank you for pointing them out.
This paper introduces a new reinforcement learning environment called « The Retro Learning Environment”, that interfaces with the open-source LibRetro API to offer access to various emulators and associated games (i.e. similar to the Atari 2600 Arcade Learning Environment, but more generic). The first supported platform is the SNES, with 5 games (more consoles and games may be added later). Authors argue that SNES games pose more challenges than Atari’s (due to more complex graphics, AI and game mechanics). Several DQN variants are evaluated in experiments, and it is also proposed to compare learning algorihms by letting them compete against each other in multiplayer games.
I like the idea of going toward more complex games than those found on Atari 2600, and having an environment where new consoles and games can easily be added sounds promising. With OpenAI Universe and DeepMind Lab that just came out, though, I am not sure we really need another one right now. Especially since using ROMs of emulated games we do not own is technically illegal: it looks like this did not cause too much trouble for Atari but it might start raising eyebrows if the community moves to more advanced and recent games, especially some Nintendo still makes money from.
Besides the introduction of the environment, it is good to have DQN benchmarks on five games, but this does not add a lot of value. The authors also mention as contribution "A new benchmarking technique, allowing algorithms to compete against each other, rather than playing against the in-game AI", but this seems a bit exaggerated to me: the idea of pitting AIs against each other has been at the core of many AI competitions for decades, so it is hardly something new. The finding that reinforcement learning algorithms tend to specialize to their opponent is also not particular surprising.
Overall I believe this is an ok paper but I do not feel it brings enough to the table for a major conference. This does not mean, however, that this new environment won't find a spot in the (now somewhat crowded) space of game-playing frameworks.
Other small comments:
- There are lots of typos (way too many to mention them all)
- It is said that Infinite Mario "still serves as a benchmark platform", however as far as I know it had to be shutdown due to Nintendo not being too happy about it
- "RLE requires an emulator and a computer version of the console game (ROM file) upon initialization rather than a ROM file only. The emulators are provided with RLE" => how is that different from ALE that requires the emulator Stella which is also provided with ALE?
- Why is there no DQN / DDDQN result on Super Mario?
- It is not clear if Figure 2 displays the F-Zero results using reward shaping or not
- The Du et al reference seems incomplete
Thank you for your review.
Q: With OpenAI Universe and DeepMind Lab that just came out, though, I am not sure we really need another one right now.
A: After we published our first draft of the paper, Universe and Lab were announced, which in some sense appear to deem RLE relevancy.
However, we believe that RLE provides value that may compliment other environments rather than replace.
While Universe has the game variety and Lab has speed and access to in-game elements, RLE has all three advantages. In our latest revision, we've included a more through comparison (see table 2 in the revised edition).
Moreover we believe that the latest achievements in the growing gaming community including very accurate psychical engines and incredibly realistic scenes which can help create a better AI algorithm. The RLE take the first steps towards combining the two worlds. By using LibRetro environment new consoles can be added easily (we have recently added supported to several additional consoles see section 3.2 in the revised paper).
Q: "RLE requires an emulator and a computer version of the console game (ROM file)... how is that different from ALE that requires the emulator Stella which is also provided with ALE?
A: ALE is built on top of the Stella emulator, therefore it requires a ROM file only. To make RLE a more general environment we separated the gaming console emulator from the RL framework. This is a more technical detail regarding the interface itself which requires another argument.
Q: Why is there no DQN / DDDQN result on Super Mario?
A: DQN and D-DDQN results on Mario were absent due to our limited computational resources and were added in a later revision (see figure 2).
Thanks you for the other comments. We've polished the paper in its later revisions.
The paper presents a new environment, called Retro Learning Environment (RLE), for reinforcement learning. The authors focus on Super Nintendo but claim that the interface supports many others (including ALE). Benchmark results are given for standard algorithms in 5 new Super Nintendo games, and some results using a new "rivalry metric".
These environments (or, more generally, standardized evaluation methods like public data sets, competitions, etc.) have a long history of improving the quality of AI and machine learning research. One example in the past few years was the Atari Learning Environment (ALE) which has now turned into a standard benchmark for comparison of algorithms and results. In this sense, the RLE could be a worthy contribution to the field by encouraging new challenging domains for research.
That said, the main focus of this paper is presenting this new framework and showcasing the importance of new challenging domains. The results of experiments themselves are for existing algorithms. There are some new results that show reward shaping and policy shaping (having a bias toward going right in Super Mario) help during learning. And, yes, domain knowledge helps, but this is obvious. The rivalry training is an interesting idea, when training against a different opponent, the learner overfits to that opponent and forgets to play against the in-game AI;
but then oddly, it gets evaluated on how well it does against the in-game AI!
Also the part of the paper that describes the scientific results (especially the rivalry training) is less polished, so this is disappointing.
In the end, I'm not very excited about this paper.
I was hoping for a more significant scientific contribution to accompany in this new environment. It's not clear if this is necessary for publication, but also it's not clear that ICLR is the right venue for this work due to the contribution being mainly about the new code (for example, mloss.org could be a better 'venue', JMLR has an associated journal track for accompanying papers: http://www.jmlr.org/mloss/)
--- Post response:
Thank you for the clarifications.
Ultimately I have not changed my opinion on the paper.
Though I do think RLE could have a nice impact long-term, there is little new science in this paper, ad it's either too straight-forward (reward shaping, policy-shaping) or not quite developed enough (rivalry training).
Thank you for your review and comments.
Q: The authors focus on Super Nintendo but claim that the interface supports many others (including ALE).
A: In an effort to further expand RLE, we added support for several new gaming consoles: Sega Genesis/Mega Drive, Master System, Game Gear and several others, all implemented by the same LibRetro core. Please see revised paper section 3.2 .
Q: The rivalry training is an interesting idea ... the learner overfits to that opponent and forgets to play against the in-game AI;
but then oddly, it gets evaluated on how well it does against the in-game AI!.
Also the part of the paper that describes the scientific results (especially the rivalry training) is less polished..
A: In our second experiment, the goal was to train an agent to achieve a better score vs the in-game AI. Since the trained agent (D-DDQN) beats the in-game AI (as seen in section 4.1.1), we assume that by adding several training episodes against the trained D-DDQN agent results will improve.
The fact that the agent overfits that badly to its opponent policy was somewhat surprising, especially since we used the exact same setting for the additional training episodes. The only difference was the rivals policy - we maintained the same characters, same levels and random initialization.
An additional experiment we conducted, and incorporated to a later revision, takes this one step further and combines training by alternating the rival's policy in each episode, switching between the in-game AI, a vanilla DQN agent and D-DDQN agent.
This indeed lead to a more generalized policy which achieves much better results when evaluate against a new policy (i.e., same game higher difficulty setting. See Section 4.1.2).
Thank you for your honest evaluation and reference to other 'venue'.

Meta-review:
role: PC
data: comment: The authors present a new set of environments, similar to ALE but based on Super Nintendo rather than Atari. This is a great asset and could be important for RL research, but it doesn't merit ICLR publication because of the lack of novel research ideas. Hopefully the authors will consider another venue to publish this paper, such as perhaps a journal or workshop.

Decision: Reject