Repository: openai/gym

Issue:
AtariEnv is doubly nondeterminstic due to ALEInterface's repeat_action_probability setting
In the current implementation, it looks like there are two sources of nondeterminism in `AtariEnv`:
1. Repeating actions between 2-4 times, as implemented by OpenAI in the `_step` method of the `AtariEnv` class.
2. The `repeat_action_probability` setting of the underlying `ALEInterface`, which is not explicitly set by gym code and _defaults to 0.25_. Section 7.5 of the [ALE manual](https://github.com/mgbellemare/Arcade-Learning-Environment/blob/master/doc/manual/manual.pdf) explains this setting in more detail.

The first source is clearly intentional—it seems to me that the second source may be unintentional. The underlying ALE setting can be verified by querying the `ALEInterface` directly:

```
In [3]: env.ale.getFloat('repeat_action_probability')
Out[3]: 0.25
```

The resulting behavior is such that if a user performs `env.step(0)`, then `env.step(1)`, there is a 0.25 probability that the first actual action taken in-game while performing `env.step(1)` will actually be a NOOP (i.e. action 0).

(Incidentally, since this setting is also present for the `Deterministic` variants of the Atari environments, this means that they aren't actually deterministic...)

Just wanted to check whether or not you were aware of this/whether or not this was intended behavior. If not, I'm happy to submit a pull request. (The fix is as simple as calling `self.ale.setFloat('repeat_action_probability', 0.)` somewhere in `AtariEnv.__init__`.)



Discussion:
Yikes, not intentional! Pushed a patch; let me know whether it looks right to you!

Looks good to me. Thanks for the prompt response!

set repeat_action_probability not work
gym: 0.3
according atari_py/ale_interface/src/ale_interface.cpp:loadRom(), setting must call before loadRom,
but in gym.envs.atari.atari_env.py, loadRom will call before setting repeat_action_probability

maybe something wrong in atari emu, I am sure already disabled repeat_action_probability(just print the state in atari_py/ale_interface/src/environment/stella_environment.cpp:act() function), but with following code, when act is NOOP or FIRE, the game paddle still move if last action is LEFT/RIGHT

``` python
import gym
from gym.envs.atari.atari_env import AtariEnv
from atari_py import ALEInterface

def _seed(self, seed=None):
    self.ale.setFloat(b'repeat_action_probability', 0.)
    from gym.utils import seeding
    self.np_random, seed1 = seeding.np_random(seed)
    # Derive a random seed. This gets passed as a uint, but gets
    # checked as an int elsewhere, so we need to keep it below
    # 2**31.
    seed2 = seeding.hash_seed(seed1 + 1) % 2 ** 31
    # Empirically, we need to seed before loading the ROM.
    self.ale.setInt(b'random_seed', seed2)
    self.ale.loadROM(self.game_path)
    return [seed1, seed2]

AtariEnv._seed = _seed

env = gym.make('Breakout-v2')

from time import sleep

space = env.action_space
ale = env.ale
assert(isinstance(ale, ALEInterface))
means = env.get_action_meanings()
while True:
    act = space.sample()
    env.step(act)
    print('act = {} {}'.format(act, means[act]))
    env.render()
    sleep(5)
```

I noticed that behavior -- was wondering if it's just part of the actual
game dynamics (i.e. the paddle picks up momentum). Any chance of that?

On Tuesday, September 27, 2016, ricky1203 notifications@github.com wrote:

> maybe something wrong in atari emu, I am sure already disabled
> repeat_action_probability(just print the state in
> atari_py/ale_interface/src/environment/stella_environment.cpp:act()
> function), but with following code, when act is NOOP or FIRE, the game
> paddle still move if last action is LEFT/RIGHT
> 
> import gymfrom gym.envs.atari.atari_env import AtariEnvfrom atari_py import ALEInterface
> def _seed(self, seed=None):
>     self.ale.setFloat(b'repeat_action_probability', 0.)
>     from gym.utils import seeding
>     self.np_random, seed1 = seeding.np_random(seed)
>     # Derive a random seed. This gets passed as a uint, but gets
>     # checked as an int elsewhere, so we need to keep it below
>     # 2*_31.
>     seed2 = seeding.hash_seed(seed1 + 1) % 2 *_ 31
>     # Empirically, we need to seed before loading the ROM.
>     self.ale.setInt(b'random_seed', seed2)
>     self.ale.loadROM(self.game_path)
>     return [seed1, seed2]
> 
> AtariEnv._seed = _seed
> 
> env = gym.make('Breakout-v2')
> from time import sleep
> 
> space = env.action_space
> ale = env.aleassert(isinstance(ale, ALEInterface))
> means = env.get_action_meanings()while True:
>     act = space.sample()
>     env.step(act)
>     print('act = {} {}'.format(act, means[act]))
>     env.render()
>     sleep(5)
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> https://github.com/openai/gym/issues/349#issuecomment-250055563, or mute
> the thread
> https://github.com/notifications/unsubscribe-auth/AAM7kRgLSCA6CdJMN1VWEOZHiEs55CCiks5qudCcgaJpZM4KBE85
> .

## 

Sent from mobile

@ricky1203 Aren't you supposed to be setting `repeat_action_probability` to 1?

@mthrok I am sure not set repeat_action_probability to 1
i just set the repeat_action_probability to 0, but no effect, so i trace into the atari c++ code, and find the gym not set the repeat_action_probability for the reason described above
@gdb maybe you r right, i will check it more

Cool. @ricky1203 can you confirm that you see a behavior difference when you move the `repeat_action_probability` call before loading the ROM?

i checked setting repeat_action_probability call before/after loading the ROM:
add a line to print the repeat_action_probability in _atari_py/ale_interface/src/environment/stella_environment.cpp:act()_, 
set  repeat_action_probability **before** loading the ROM (just as the code i paste above):
repeat_action_probability = 0.000000, act=11/18, m_act=11/18
repeat_action_probability = 0.000000, act=11/18, m_act=11/18
repeat_action_probability = 0.000000, act=11/18, m_act=11/18
repeat_action_probability = 0.000000, act=11/18, m_act=11/18

set repeat_action_probability **after** loading the ROM (just as in now gym code with "Breakout-v2): 
repeat_action_probability = 0.250000, act=11/18, m_act=11/18
repeat_action_probability = 0.250000, act=11/18, m_act=11/18
repeat_action_probability = 0.250000, act=11/18, m_act=11/18

for Breakout-v2, little behavior difference for setting repeat_action_probability before/after loading the ROM, as @gdb say: paddle picks up momentum
but sure the set repeat_action_probability not working now, maybe cause problem on other games


Resolving patch:
diff --git a/gym/envs/__init__.py b/gym/envs/__init__.py
--- a/gym/envs/__init__.py
+++ b/gym/envs/__init__.py
@@ -281,7 +281,7 @@
         )
 
         register(
-            id='{}-v2'.format(name),
+            id='{}-v3'.format(name),
             entry_point='gym.envs.atari:AtariEnv',
             kwargs={'game': game, 'obs_type': obs_type},
             timestep_limit=100000,
@@ -304,7 +304,7 @@
         )
 
         register(
-            id='{}Deterministic-v2'.format(name),
+            id='{}Deterministic-v3'.format(name),
             entry_point='gym.envs.atari:AtariEnv',
             kwargs={'game': game, 'obs_type': obs_type, 'frameskip': frameskip},
             timestep_limit=100000,
@@ -322,7 +322,7 @@
         # No frameskip. (Atari has no entropy source, so these are
         # deterministic environments.)
         register(
-            id='{}NoFrameskip-v2'.format(name),
+            id='{}NoFrameskip-v3'.format(name),
             entry_point='gym.envs.atari:AtariEnv',
             kwargs={'game': game, 'obs_type': obs_type, 'frameskip': 1}, # A frameskip of 1 means we get every frame
             timestep_limit=frameskip * 100000,
diff --git a/gym/envs/atari/atari_env.py b/gym/envs/atari/atari_env.py
--- a/gym/envs/atari/atari_env.py
+++ b/gym/envs/atari/atari_env.py
@@ -37,6 +37,11 @@ def __init__(self, game='pong', obs_type='ram', frameskip=(2, 5), repeat_action_
         self.ale = atari_py.ALEInterface()
         self.viewer = None
 
+        # Tune (or disable) ALE's action repeat:
+        # https://github.com/openai/gym/issues/349
+        assert isinstance(repeat_action_probability, (float, int)), "Invalid repeat_action_probability: {!r}".format(repeat_action_probability)
+        self.ale.setFloat('repeat_action_probability'.encode('utf-8'), repeat_action_probability)
+
         self._seed()
 
         (screen_width, screen_height) = self.ale.getScreenDims()
@@ -45,11 +50,6 @@ def __init__(self, game='pong', obs_type='ram', frameskip=(2, 5), repeat_action_
         self._action_set = self.ale.getMinimalActionSet()
         self.action_space = spaces.Discrete(len(self._action_set))
 
-        # Tune (or disable) ALE's action repeat:
-        # https://github.com/openai/gym/issues/349
-        assert isinstance(repeat_action_probability, (float, int)), "Invalid repeat_action_probability: {!r}".format(repeat_action_probability)
-        self.ale.setFloat('repeat_action_probability'.encode('utf-8'), repeat_action_probability)
-
         (screen_width,screen_height) = self.ale.getScreenDims()
         if self._obs_type == 'ram':
             self.observation_space = spaces.Box(low=np.zeros(128), high=np.zeros(128)+255)


Test patch: