Repository: openai/gym

Issue:
[Bug Report] Total order of keys for `spaces.Dict`
If you are submitting a bug report, please fill in the following details and use the tag [bug].

**Describe the bug**
A clear and concise description of what the bug is.

Pair of issue google/jax#11871. A total order is required for dictionary keys.

Equal `Dict` spaces `dict1 == dict2` do not imply `dict1.sample() == dict2.sample()` and `flatten_space(dict1) == flatten_space(dict2)`.

```python
In [1]: from gym import spaces

In [2]: dict1 = spaces.Dict({1: spaces.Box(0, 1, shape=(1,)), 'a': spaces.Box(0, 2, shape=(1,))})

In [3]: dict2 = spaces.Dict({'a': spaces.Box(0, 2, shape=(1,)), 1: spaces.Box(0, 1, shape=(1,))})

In [4]: dict1
Out[4]: Dict(1: Box(0.0, 1.0, (1,), float32), a: Box(0.0, 2.0, (1,), float32))

In [5]: dict2
Out[5]: Dict(a: Box(0.0, 2.0, (1,), float32), 1: Box(0.0, 1.0, (1,), float32))

In [6]: dict1 == dict2
Out[6]: True
```

Not equally seeded in subspaces:

```python
In [7]: dict1.seed(0)
Out[7]: [0, 2488343231644625808, 5874934615388537134]

In [8]: dict2.seed(0)
Out[8]: [0, 2488343231644625808, 5874934615388537134]

In [9]: dict1.sample() == dict2.sample()
Out[9]: False
```

Different order while flattening:

```python
In [10]: spaces.flatten_space(dict1)
Out[10]: Box(0.0, [1. 2.], (2,), float32)

In [11]: spaces.flatten_space(dict2)
Out[11]: Box(0.0, [2. 1.], (2,), float32)

In [12]: spaces.flatten_space(dict1) == spaces.flatten_space(dict2)
Out[12]: False
```

The order of keys is important when seeding the subspaces and flattening the space.

------

In `spaces.Dict.__init__` method, we always convert the inputs into an `OrderedDict`:

https://github.com/openai/gym/blob/8b744130bc25d0f4fe2eac8bce8e47527a8eee2c/gym/spaces/dict.py#L82-L92

However, function `sorted` will fail when sorting with uncomparable types (e.g. `int` vs. `str`):

```python
In [1]: d = {1: 1, 'a': 2}

In [2]: sorted(d.items())
TypeError: '<' not supported between instances of 'str' and 'int'
```

So we add a failback choice at line 88 in PR #2491. This means when the keys are not sortable, the keys are ordered by the insertion order (since Python 3.6). However, the order of keys is important when seeding the subspaces and flattening the space.

`seed`: (seed in order of `dict.spaces.values()`)

https://github.com/openai/gym/blob/8b744130bc25d0f4fe2eac8bce8e47527a8eee2c/gym/spaces/dict.py#L131-L135

`flatten_space`: (flatten in order of `dict.spaces.values()`)

https://github.com/openai/gym/blob/8b744130bc25d0f4fe2eac8bce8e47527a8eee2c/gym/spaces/utils.py#L326-L333

This will cause `dict1 == dict2` does not imply `flatten_space(dict1) == flatten_space(dict2)`.

**Code example**
Please try to provide a minimal example to reproduce the bug. Error messages and stack traces are also helpful.

See Description above.

**System Info**
Describe the characteristic of your environment:
 * Describe how Gym was installed: pip
 * What OS/version of Linux you're using: Ubuntu 20.04 LTS
 * Python version: 3.9

**Additional context**
Add any other context about the problem here.

See also:

- google/jax#4085
- google/jax#11871

### Checklist

- [X] I have checked that there is no similar [issue](https://github.com/openai/gym/issues) in the repo (**required**)



Discussion:
As I commented in https://github.com/google/jax/pull/11871#issuecomment-1213068129, a feasible solution is sorting by `key = (obj.__class__.__qualname__, obj)`. It works for all Python built-in types. However, for user-defined types, there is no total order any more.
Thanks for the issue but Im confused what the issue that needs solving is? 
Is it that the `__eq__` function needs to consider the order of the keys, not just that the same keys exist in both dicts? 
Or that we sort no matter if the key type. I should note that in our typing, we assume that the key type is always str so this issue shouldn't occur. In what cases are you using a non-str key? 

Looking back, I disable that we sort the dict keys and not just follow the order of insertion but it seems to late to change. 

> Im confused what the issue that needs solving is?

@pseudo-rnd-thoughts, the issue is:

> Equal `Dict` spaces `dict1 == dict2` do not imply `dict1.sample() == dict2.sample()` and `flatten_space(dict1) == flatten_space(dict2)`.

There are two solutions:

1. find a total order and sort the keys by it. (e.g. `key = (obj.__class__.__qualname__, obj)` works for all Python built-in types, but may not work for user-defined types)
2. consider the order of the keys in `Dict.__eq__` method. 

------

> Is it that the `__eq__` function needs to consider the order of the keys, not just that the same keys exist in both `dict`s?

Python built-in `dict` does not consider the key order either. When we compare `OrderedDict` vs. `dict`, the key order is also ignored.

```python
In [1]: from collections import OrderedDict

In [2]: d1 = {1: 0, 'a': 1}

In [3]: d2 = {'a': 1, 1: 0}

In [4]: d1
Out[4]: {1: 0, 'a': 1}

In [5]: d2
Out[5]: {'a': 1, 1: 0}

In [6]: d1 == d2
Out[6]: True

In [7]: od1 = OrderedDict([(1, 0), ('a', 1)])

In [8]: od2 = OrderedDict([('a', 1), (1, 0)])

In [9]: d1 == od1
Out[9]: True

In [10]: d2 == od1
Out[10]: True
```

When comparing two `OrderedDict`, the key order is under consideration.

```python
In [11]: od1 == od2
Out[11]: False

In [12]: dict(od1) == dict(od2)
Out[12]: True
```

We save the `dict` structure as `OrderedDict` in our `spaces.Dict` implementation. It would be fine to consider the key order when comparing two `Dict` spaces.

------

> Looking back, I disable that we sort the dict keys and not just follow the order of insertion but it seems to late to change.

We do not need to disable the sorting behavior, just updating the `__eq__` method is enough.
Ok that makes sense. 
Would you like to make the PR or I can add it to https://github.com/openai/gym/pull/2977

Resolving patch:
diff --git a/gym/spaces/dict.py b/gym/spaces/dict.py
--- a/gym/spaces/dict.py
+++ b/gym/spaces/dict.py
@@ -81,10 +81,12 @@ def __init__(
 
         if spaces is None:
             spaces = spaces_kwargs
-        if isinstance(spaces, dict) and not isinstance(spaces, OrderedDict):
+        if isinstance(spaces, Mapping) and not isinstance(spaces, OrderedDict):
             try:
                 spaces = OrderedDict(sorted(spaces.items()))
-            except TypeError:  # raise when sort by different types of keys
+            except TypeError:
+                # Incomparable types (e.g. `int` vs. `str`, or user-defined types) found.
+                # The keys remain in the insertion order.
                 spaces = OrderedDict(spaces.items())
         if isinstance(spaces, Sequence):
             spaces = OrderedDict(spaces)
@@ -191,7 +193,17 @@ def __len__(self) -> int:
 
     def __repr__(self) -> str:
         """Gives a string representation of this space."""
-        return "Dict(" + ", ".join([f"{k}: {s}" for k, s in self.spaces.items()]) + ")"
+        return (
+            "Dict(" + ", ".join([f"{k!r}: {s}" for k, s in self.spaces.items()]) + ")"
+        )
+
+    def __eq__(self, other) -> bool:
+        """Check whether `other` is equivalent to this instance."""
+        return (
+            isinstance(other, Dict)
+            # Comparison of `OrderedDict`s is order-sensitive
+            and self.spaces == other.spaces  # OrderedDict.__eq__
+        )
 
     def to_jsonable(self, sample_n: list) -> dict:
         """Convert a batch of samples from this space to a JSONable data type."""


Test patch: