Repository: openai/gym

Issue:
[Bug Report] Mujoco envs offscreen rendering error for width > 640 or height > 480
Separately track issue "the size >500 making the environment crash" from https://github.com/openai/gym/issues/2989#issuecomment-1202214444 here, and describe it more precisely.

The cause is:
https://github.com/openai/gym/blob/6e2e921b5faaa9c356f589e4e938e718824c7d4a/gym/envs/mujoco/mujoco_rendering.py#L77-L78
https://github.com/openai/gym/blob/6e2e921b5faaa9c356f589e4e938e718824c7d4a/gym/envs/mujoco/mujoco_rendering.py#L63-L67
Precisely, `self.con.free()` will reset the struct(strangely, it will save and restore some fields) thus `con->offFBO` will be `0`, then
https://github.com/deepmind/mujoco/blob/090fe2db3f23ad608f833e3e8e9cb049e4daaa87/src/render/render_gl2.c#L37-L46
will set window buffer instead of offscreen buffer.
In principle, "[call ctx.free(), after which point it is the user’s responsibility to ensure that no further rendering calls are made on the context.](https://mujoco.readthedocs.io/en/latest/python.html#rendering)", we should create a new context after `free`.
## Fix
```python
  def update_offscreen_size(self, width, height):
      if width != self.con.offWidth or height != self.con.offHeight:
          self.model.vis.global_.offwidth = width
          self.model.vis.global_.offheight = height
          self.con.free()
          self.con = mujoco.MjrContext(self.model, mujoco.mjtFontScale.mjFONTSCALE_150)
          self._set_mujoco_buffers()
```
This snippet can fix this issue on `glfw`, but I wonder why `GLContext` with wrong `max_width` and `max_height`(since correct width and height is not passed to `_get_viewer`) doesn't cause any problem. (`egl` doesn't need it, but `glfw` need it)
@pseudo-rnd-thoughts 


Discussion:
@rodrigodelazcano Do you understand this? 
@YouJiacheng Can you fix it? 
I posted the patch that can fix it. (without taking care of viewer's width and height).
But I don't know why `GLContext` with wrong width and height doesn't cause any problem.
Yes, this is necessary. I'm wondering if it will be possible to initialize the width and height of the context only one time in the `__init__` of `RenderContext`. With the new render API the width and height can't be changed at runtime by the user thus it won't be necessary to call `_update_offscreen_size` every time we call `render`. 
 
@YouJiacheng  are you available to make a PR with the changes.

Also by setting width, and height of the `GLContext` what do you mean? The error of changing the width and height size seems to come from the mujoco context `MjrContext`
It is possible to initialize the width and height of the context and keep it constant. We only need to tweak the implementation, and it will also help avoid "`GLContext` with wrong width and height" problem.

By "`GLContext` with wrong width and height", see https://github.com/openai/gym/issues/2989#issuecomment-1225354628
i.e., `GLContext` is not initialize with correct width and height set by user, since width and height is not passed to `_get_viewer` creating the `GLContext`.
Got it, I see what you mean. I can't understand why it doesn't cause any problem but it looks like an issue on the Mujoco side. 

Since we don't require to change the width and height during runtime it would be great if you could make a PR that initializes the size of the offscreen renderer at initialization only.
Okay, let's just do semantically right things: Create `GLContext` with correct size at initialization, set `model.vis.global_.offwidth` and `offheight` at initialization, remove `width` and `height` parameter from `RenderContextOffscreen.render`. I will make a PR for it.

Resolving patch:
diff --git a/gym/envs/mujoco/mujoco_env.py b/gym/envs/mujoco/mujoco_env.py
--- a/gym/envs/mujoco/mujoco_env.py
+++ b/gym/envs/mujoco/mujoco_env.py
@@ -49,7 +49,9 @@ def __init__(
         if not path.exists(self.fullpath):
             raise OSError(f"File {self.fullpath} does not exist")
 
-        self._initialize_simulation()
+        self.width = width
+        self.height = height
+        self._initialize_simulation()  # may use width and height
 
         self.init_qpos = self.data.qpos.ravel().copy()
         self.init_qvel = self.data.qvel.ravel().copy()
@@ -76,8 +78,6 @@ def __init__(
         self.render_mode = render_mode
         render_frame = partial(
             self._render,
-            width=width,
-            height=height,
             camera_name=camera_name,
             camera_id=camera_id,
         )
@@ -126,8 +126,6 @@ def _step_mujoco_simulation(self, ctrl, n_frames):
     def _render(
         self,
         mode: str = "human",
-        width: int = DEFAULT_SIZE,
-        height: int = DEFAULT_SIZE,
         camera_id: Optional[int] = None,
         camera_name: Optional[str] = None,
     ):
@@ -249,11 +247,10 @@ def _step_mujoco_simulation(self, ctrl, n_frames):
     def _render(
         self,
         mode: str = "human",
-        width: int = DEFAULT_SIZE,
-        height: int = DEFAULT_SIZE,
         camera_id: Optional[int] = None,
         camera_name: Optional[str] = None,
     ):
+        width, height = self.width, self.height
         assert mode in self.metadata["render_modes"]
         if mode in {
             "rgb_array",
@@ -350,6 +347,9 @@ def __init__(
 
     def _initialize_simulation(self):
         self.model = mujoco.MjModel.from_xml_path(self.fullpath)
+        # MjrContext will copy model.vis.global_.off* to con.off*
+        self.model.vis.global_.offwidth = self.width
+        self.model.vis.global_.offheight = self.height
         self.data = mujoco.MjData(self.model)
 
     def _reset_simulation(self):
@@ -376,8 +376,6 @@ def _step_mujoco_simulation(self, ctrl, n_frames):
     def _render(
         self,
         mode: str = "human",
-        width: int = DEFAULT_SIZE,
-        height: int = DEFAULT_SIZE,
         camera_id: Optional[int] = None,
         camera_name: Optional[str] = None,
     ):
@@ -406,16 +404,16 @@ def _render(
                     camera_name,
                 )
 
-                self._get_viewer(mode).render(width, height, camera_id=camera_id)
+                self._get_viewer(mode).render(camera_id=camera_id)
 
         if mode in {"rgb_array", "single_rgb_array"}:
-            data = self._get_viewer(mode).read_pixels(width, height, depth=False)
+            data = self._get_viewer(mode).read_pixels(depth=False)
             # original image is upside-down, so flip it
             return data[::-1, :, :]
         elif mode in {"depth_array", "single_depth_array"}:
-            self._get_viewer(mode).render(width, height)
+            self._get_viewer(mode).render()
             # Extract depth part of the read_pixels() tuple
-            data = self._get_viewer(mode).read_pixels(width, height, depth=True)[1]
+            data = self._get_viewer(mode).read_pixels(depth=True)[1]
             # original image is upside-down, so flip it
             return data[::-1, :]
         elif mode == "human":
@@ -427,7 +425,7 @@ def close(self):
         super().close()
 
     def _get_viewer(
-        self, mode, width=DEFAULT_SIZE, height=DEFAULT_SIZE
+        self, mode
     ) -> Union["gym.envs.mujoco.Viewer", "gym.envs.mujoco.RenderContextOffscreen"]:
         self.viewer = self._viewers.get(mode)
         if self.viewer is None:
@@ -443,9 +441,7 @@ def _get_viewer(
             }:
                 from gym.envs.mujoco import RenderContextOffscreen
 
-                self.viewer = RenderContextOffscreen(
-                    width, height, self.model, self.data
-                )
+                self.viewer = RenderContextOffscreen(self.model, self.data)
             else:
                 raise AttributeError(
                     f"Unexpected mode: {mode}, expected modes: {self.metadata['render_modes']}"
diff --git a/gym/envs/mujoco/mujoco_rendering.py b/gym/envs/mujoco/mujoco_rendering.py
--- a/gym/envs/mujoco/mujoco_rendering.py
+++ b/gym/envs/mujoco/mujoco_rendering.py
@@ -44,6 +44,8 @@ def __init__(self, model, data, offscreen=True):
         self.model = model
         self.data = data
         self.offscreen = offscreen
+        self.offwidth = model.vis.global_.offwidth
+        self.offheight = model.vis.global_.offheight
         max_geom = 1000
 
         mujoco.mj_forward(self.model, self.data)
@@ -70,22 +72,10 @@ def _set_mujoco_buffers(self):
             if self.con.currentBuffer != mujoco.mjtFramebuffer.mjFB_WINDOW:
                 raise RuntimeError("Window rendering not supported")
 
-    def update_offscreen_size(self, width, height):
-        if width != self.con.offWidth or height != self.con.offHeight:
-            self.model.vis.global_.offwidth = width
-            self.model.vis.global_.offheight = height
-            self.con.free()
-            self._set_mujoco_buffers()
-
-    def render(self, width, height, camera_id=None, segmentation=False):
+    def render(self, camera_id=None, segmentation=False):
+        width, height = self.offwidth, self.offheight
         rect = mujoco.MjrRect(left=0, bottom=0, width=width, height=height)
 
-        # Sometimes buffers are too small.
-        if width > self.con.offWidth or height > self.con.offHeight:
-            new_width = max(width, self.model.vis.global_.offwidth)
-            new_height = max(height, self.model.vis.global_.offheight)
-            self.update_offscreen_size(new_width, new_height)
-
         if camera_id is not None:
             if camera_id == -1:
                 self.cam.type = mujoco.mjtCamera.mjCAMERA_FREE
@@ -126,7 +116,8 @@ def render(self, width, height, camera_id=None, segmentation=False):
             self.scn.flags[mujoco.mjtRndFlag.mjRND_SEGMENT] = 0
             self.scn.flags[mujoco.mjtRndFlag.mjRND_IDCOLOR] = 0
 
-    def read_pixels(self, width, height, depth=True, segmentation=False):
+    def read_pixels(self, depth=True, segmentation=False):
+        width, height = self.offwidth, self.offheight
         rect = mujoco.MjrRect(left=0, bottom=0, width=width, height=height)
 
         rgb_arr = np.zeros(3 * rect.width * rect.height, dtype=np.uint8)
@@ -231,8 +222,10 @@ def close(self):
 class RenderContextOffscreen(RenderContext):
     """Offscreen rendering class with opengl context."""
 
-    def __init__(self, width, height, model, data):
-
+    def __init__(self, model, data):
+        # We must make GLContext before MjrContext
+        width = model.vis.global_.offwidth
+        height = model.vis.global_.offheight
         self._get_opengl_backend(width, height)
         self.opengl_context.make_current()
 


Test patch: