Repository: ray-project/ray

Issue:
[rllib][bug] Some policy-names lead to seemingly unrelated errors

### Some policy-names lead to strange errors 
The following very simple multi-agent script fails because of the names given to the policies

```python
from gym.spaces import Discrete, Box
import numpy as np
import ray
from ray import tune
from ray.rllib.env.multi_agent_env import MultiAgentEnv


class ErrorGame(MultiAgentEnv):
    def __init__(self, env_config):
        pass

    def reset(self):
        obs = self._obs()
        return obs

    def step(self, action_dict):
        rewards = {0: 0, 1: 0}
        obs = self._obs()
        return obs, rewards, {"__all__": False}, {}

    def _obs(self):
        return {0: np.array([0.2, 0.2], dtype=np.float32), 1: np.array([0.2], dtype=np.float32)}


if __name__ == "__main__":

    obs1 = Box(low=np.array([0] * 2), high=np.array([1] * 2))
    obs2 = Box(low=np.array([0] * 1), high=np.array([1] * 1))

    act1 = Discrete(1)
    act2 = Discrete(1)
    # helper_act = Discrete(1)

    # -- POLICIES--
    name1 = "seeker"
    name2 = "helper"


    policies = {
        name1: (None, obs1, act1, {"agent_id": 0}),
        name2: (None, obs2, act2, {"agent_id": 1})
    }


    def policy_name_mapping(x):
        return {0:name1, 1:name2}[x]


    config = {
        "learning_starts": 100,
        "evaluation_interval": 10000,
        "multiagent": {
            "policies": policies,
            "policy_mapping_fn": policy_name_mapping,
        },
    }

    ray.init(
        num_cpus=2,
        # object_store_memory=200 * 1024 * 1024,
    )
    tune.run(
        "contrib/MADDPG",
        stop={
            "timesteps_total": 5000,
        },
        config=dict(config, **{
            "env": ErrorGame,
        }),
        checkpoint_at_end=True,
        checkpoint_freq=10000
        # restore="checkpoint_50000/checkpoint-50000"
    )

```

This will raise the error

` ValueError: Cannot feed value of shape (1, 2) for Tensor 'foo/obs_0:0', which has shape '(?, 1)'` 

it took me a long time to find that this error will not come up if i change the names of my policies
If i replace the lines:

```python
    name1 = "seeker"
    name2 = "helper"
```

just with for example

```python
    name1 = "seeker1"
    name2 = "seeker2"
```

I can not really find a patter when it works and when not: Here are some working and not working examples:

**working**:
    name1 = "seeker1" | name2 = "seeker2"
    name1 = "pol" | name2 = "pool"
    name1 = "ags" | name2 = "asdgasdg"


**not-working**:
    name1 = "seeker" | name2 = "helper"
    name1 = "rom" | name2 = "rem"
    name1 = "foo" | name2 = "bar"


I can't imagine that's intentional. But if there is some pattern behind it that I don't recognize it would be useful to get a more sophisticated error message

Also, as far as I can judge now, the error seems to occur only when the observation_size of the two policies have different sizes. So my guess is that the policies might get mixed up at some point?

And also i only got this error when using `"contrib/MADDPG"` but did not test very much with other setups.

*Ray version and other system information (Python version, TensorFlow version, OS):*
software | version
--- | ---
ray |0.8.4
python | 3.6.9
tensorflow | 1.14.0
OS |  Ubuntu (running in a VM on a Windows OS) Release 18.04


Discussion:

Resolving patch:
diff --git a/rllib/contrib/maddpg/maddpg.py b/rllib/contrib/maddpg/maddpg.py
--- a/rllib/contrib/maddpg/maddpg.py
+++ b/rllib/contrib/maddpg/maddpg.py
@@ -22,6 +22,9 @@
 # yapf: disable
 # __sphinx_doc_begin__
 DEFAULT_CONFIG = with_common_config({
+    # === Framework to run the algorithm ===
+    "framework": "tf",
+
     # === Settings for each individual policy ===
     # ID of the agent controlled by this policy
     "agent_id": None,
diff --git a/rllib/contrib/maddpg/maddpg_policy.py b/rllib/contrib/maddpg/maddpg_policy.py
--- a/rllib/contrib/maddpg/maddpg_policy.py
+++ b/rllib/contrib/maddpg/maddpg_policy.py
@@ -74,12 +74,14 @@ def _make_continuous_space(space):
                     "Space {} is not supported.".format(space))
 
         obs_space_n = [
-            _make_continuous_space(space) for _, (_, space, _, _) in
-            sorted(config["multiagent"]["policies"].items())
+            _make_continuous_space(space)
+            for _, (_, space, _,
+                    _) in config["multiagent"]["policies"].items()
         ]
         act_space_n = [
-            _make_continuous_space(space) for _, (_, _, space, _) in
-            sorted(config["multiagent"]["policies"].items())
+            _make_continuous_space(space)
+            for _, (_, _, space,
+                    _) in config["multiagent"]["policies"].items()
         ]
 
         # _____ Placeholders


Test patch: