Repository: openai/gym

Issue:
Base class Wrapper(env) does not allow action_space updates from within original environment
Hi,

I am making a gym wrapper for PyBullet. I extract the action limits from PyBullet to create the `action_space`. However, these limits can only be retrieved from the environment after PyBullet is initialized, which happens after calling `env.reset()`. Therefore, at the moment of calling `gym.make('env-v0')`, the action space is not yet initialized. This should be no problem, as I will set them eventually in the `env.reset()` routine. However, after updating the `self.action_space` in the `env.reset()` routine, I find that `env.action_space=None`. 

I suspect that my issue lies with the `gym.Wrapper(gym.Env)` wrapper.

Because I registered my custom environment `env-v0` with `max_episode_steps=200`, my environment is wrapped in a TimeLimit wrapper, which is a subclass of the `gym.Wrapper(gym.Env)` wrapper (see wrapper code below). This wrapper assigns `self.action_space` in `__init__(self, env)`, which does not make it straightforward to updated `action_space` later on from within the original environment (e.g. by calling `env.reset()`). What is the reason for this implementation?

Why is there no method defined for `action_space` in `gym.Wrapper(gym.Env)`, as given below?
```
    @property
    def action_space(self):
        return self.env.action_space
```

Indeed, when I remove the  `max_episode_steps=200` argument when registering my environment, `self.action_space` is correctly updated after calling `env.reset()` as it is not wrapped in a `gym.Wrapper(gym.Env)` wrapper.


The same problem occurs for `self.observation_space`, `self.reward_range`, and `self.metadata`.
```
class Wrapper(Env):
    """Wraps the environment to allow a modular transformation.

    This class is the base class for all wrappers. The subclass could override
    some methods to change the behavior of the original environment without touching the
    original code.

    .. note::

        Don't forget to call ``super().__init__(env)`` if the subclass overrides :meth:`__init__`.

    """
    def __init__(self, env):
        self.env = env
        self.action_space = self.env.action_space
        self.observation_space = self.env.observation_space
        self.reward_range = self.env.reward_range
        self.metadata = self.env.metadata

    def __getattr__(self, name):
        if name.startswith('_'):
            raise AttributeError("attempted to get missing private attribute '{}'".format(name))
        return getattr(self.env, name)

    @property
    def spec(self):
        return self.env.spec

    @classmethod
    def class_name(cls):
        return cls.__name__

    def step(self, action):
        return self.env.step(action)

    def reset(self, **kwargs):
        return self.env.reset(**kwargs)

    def render(self, mode='human', **kwargs):
        return self.env.render(mode, **kwargs)

    def close(self):
        return self.env.close()

    def seed(self, seed=None):
        return self.env.seed(seed)

    def compute_reward(self, achieved_goal, desired_goal, info):
        return self.env.compute_reward(achieved_goal, desired_goal, info)

    def __str__(self):
        return '<{}{}>'.format(type(self).__name__, self.env)

    def __repr__(self):
        return str(self)

    @property
    def unwrapped(self):
        return self.env.unwrapped
```
 
Base class Wrapper(env) does not allow action_space updates from within original environment
Hi,

I am making a gym wrapper for PyBullet. I extract the action limits from PyBullet to create the `action_space`. However, these limits can only be retrieved from the environment after PyBullet is initialized, which happens after calling `env.reset()`. Therefore, at the moment of calling `gym.make('env-v0')`, the action space is not yet initialized. This should be no problem, as I will set them eventually in the `env.reset()` routine. However, after updating the `self.action_space` in the `env.reset()` routine, I find that `env.action_space=None`. 

I suspect that my issue lies with the `gym.Wrapper(gym.Env)` wrapper.

Because I registered my custom environment `env-v0` with `max_episode_steps=200`, my environment is wrapped in a TimeLimit wrapper, which is a subclass of the `gym.Wrapper(gym.Env)` wrapper (see wrapper code below). This wrapper assigns `self.action_space` in `__init__(self, env)`, which does not make it straightforward to updated `action_space` later on from within the original environment (e.g. by calling `env.reset()`). What is the reason for this implementation?

Why is there no method defined for `action_space` in `gym.Wrapper(gym.Env)`, as given below?
```
    @property
    def action_space(self):
        return self.env.action_space
```

Indeed, when I remove the  `max_episode_steps=200` argument when registering my environment, `self.action_space` is correctly updated after calling `env.reset()` as it is not wrapped in a `gym.Wrapper(gym.Env)` wrapper.


The same problem occurs for `self.observation_space`, `self.reward_range`, and `self.metadata`.
```
class Wrapper(Env):
    """Wraps the environment to allow a modular transformation.

    This class is the base class for all wrappers. The subclass could override
    some methods to change the behavior of the original environment without touching the
    original code.

    .. note::

        Don't forget to call ``super().__init__(env)`` if the subclass overrides :meth:`__init__`.

    """
    def __init__(self, env):
        self.env = env
        self.action_space = self.env.action_space
        self.observation_space = self.env.observation_space
        self.reward_range = self.env.reward_range
        self.metadata = self.env.metadata

    def __getattr__(self, name):
        if name.startswith('_'):
            raise AttributeError("attempted to get missing private attribute '{}'".format(name))
        return getattr(self.env, name)

    @property
    def spec(self):
        return self.env.spec

    @classmethod
    def class_name(cls):
        return cls.__name__

    def step(self, action):
        return self.env.step(action)

    def reset(self, **kwargs):
        return self.env.reset(**kwargs)

    def render(self, mode='human', **kwargs):
        return self.env.render(mode, **kwargs)

    def close(self):
        return self.env.close()

    def seed(self, seed=None):
        return self.env.seed(seed)

    def compute_reward(self, achieved_goal, desired_goal, info):
        return self.env.compute_reward(achieved_goal, desired_goal, info)

    def __str__(self):
        return '<{}{}>'.format(type(self).__name__, self.env)

    def __repr__(self):
        return str(self)

    @property
    def unwrapped(self):
        return self.env.unwrapped
```
 


Discussion:

Resolving patch:
diff --git a/gym/core.py b/gym/core.py
--- a/gym/core.py
+++ b/gym/core.py
@@ -224,10 +224,11 @@ class Wrapper(Env):
 
     def __init__(self, env):
         self.env = env
-        self.action_space = self.env.action_space
-        self.observation_space = self.env.observation_space
-        self.reward_range = self.env.reward_range
-        self.metadata = self.env.metadata
+
+        self._action_space = None
+        self._observation_space = None
+        self._reward_range = None
+        self._metadata = None
 
     def __getattr__(self, name):
         if name.startswith("_"):
@@ -244,6 +245,46 @@ def spec(self):
     def class_name(cls):
         return cls.__name__
 
+    @property
+    def action_space(self):
+        if self._action_space is None:
+            return self.env.action_space
+        return self._action_space
+
+    @action_space.setter
+    def action_space(self, space):
+        self._action_space = space
+
+    @property
+    def observation_space(self):
+        if self._observation_space is None:
+            return self.env.observation_space
+        return self._observation_space
+
+    @observation_space.setter
+    def observation_space(self, space):
+        self._observation_space = space
+
+    @property
+    def reward_range(self):
+        if self._reward_range is None:
+            return self.env.reward_range
+        return self._reward_range
+
+    @reward_range.setter
+    def reward_range(self, value):
+        self._reward_range = value
+
+    @property
+    def metadata(self):
+        if self._metadata is None:
+            return self.env.metadata
+        return self._metadata
+
+    @metadata.setter
+    def metadata(self, value):
+        self._metadata = value
+
     def step(self, action):
         return self.env.step(action)
 


Test patch:
diff --git a/gym/tests/test_core.py b/gym/tests/test_core.py
--- a/gym/tests/test_core.py
+++ b/gym/tests/test_core.py
@@ -1,4 +1,7 @@
-from gym import core
+import pytest
+import numpy as np
+
+from gym import core, spaces
 
 
 class ArgumentEnv(core.Env):
@@ -9,9 +12,99 @@ def __init__(self, arg):
         self.arg = arg
 
 
+class UnittestEnv(core.Env):
+    observation_space = spaces.Box(low=0, high=255, shape=(64, 64, 3), dtype=np.uint8)
+    action_space = spaces.Discrete(3)
+
+    def reset(self):
+        return self.observation_space.sample()  # Dummy observation
+
+    def step(self, action):
+        observation = self.observation_space.sample()  # Dummy observation
+        return (observation, 0.0, False, {})
+
+
+class UnknownSpacesEnv(core.Env):
+    """This environment defines its observation & action spaces only
+    after the first call to reset. Although this pattern is sometimes
+    necessary when implementing a new environment (e.g. if it depends
+    on external resources), it is not encouraged.
+    """
+
+    def reset(self):
+        self.observation_space = spaces.Box(
+            low=0, high=255, shape=(64, 64, 3), dtype=np.uint8
+        )
+        self.action_space = spaces.Discrete(3)
+        return self.observation_space.sample()  # Dummy observation
+
+    def step(self, action):
+        observation = self.observation_space.sample()  # Dummy observation
+        return (observation, 0.0, False, {})
+
+
+class NewPropertyWrapper(core.Wrapper):
+    def __init__(
+        self,
+        env,
+        observation_space=None,
+        action_space=None,
+        reward_range=None,
+        metadata=None,
+    ):
+        super().__init__(env)
+        if observation_space is not None:
+            # Only set the observation space if not None to test property forwarding
+            self.observation_space = observation_space
+        if action_space is not None:
+            self.action_space = action_space
+        if reward_range is not None:
+            self.reward_range = reward_range
+        if metadata is not None:
+            self.metadata = metadata
+
+
 def test_env_instantiation():
     # This looks like a pretty trivial, but given our usage of
     # __new__, it's worth having.
     env = ArgumentEnv("arg")
     assert env.arg == "arg"
     assert env.calls == 1
+
+
+properties = [
+    {
+        "observation_space": spaces.Box(
+            low=0.0, high=1.0, shape=(64, 64, 3), dtype=np.float32
+        )
+    },
+    {"action_space": spaces.Discrete(2)},
+    {"reward_range": (-1.0, 1.0)},
+    {"metadata": {"render.modes": ["human", "rgb_array"]}},
+    {
+        "observation_space": spaces.Box(
+            low=0.0, high=1.0, shape=(64, 64, 3), dtype=np.float32
+        ),
+        "action_space": spaces.Discrete(2),
+    },
+]
+
+
+@pytest.mark.parametrize("class_", [UnittestEnv, UnknownSpacesEnv])
+@pytest.mark.parametrize("props", properties)
+def test_wrapper_property_forwarding(class_, props):
+    env = class_()
+    env = NewPropertyWrapper(env, **props)
+
+    # If UnknownSpacesEnv, then call reset to define the spaces
+    if isinstance(env.unwrapped, UnknownSpacesEnv):
+        _ = env.reset()
+
+    # Test the properties set by the wrapper
+    for key, value in props.items():
+        assert getattr(env, key) == value
+
+    # Otherwise, test if the properties are forwarded
+    all_properties = {"observation_space", "action_space", "reward_range", "metadata"}
+    for key in all_properties - props.keys():
+        assert getattr(env, key) == getattr(env.unwrapped, key)