Repository: ray-project/ray

Issue:
Inconsistent weight assignment operations in DQNPolicyGraph
<!--
General questions should be asked on the mailing list ray-dev@googlegroups.com.
Questions about how to use Ray should be asked on
[StackOverflow](https://stackoverflow.com/questions/tagged/ray).

Before submitting an issue, please fill out the following form.
-->

### System information
- **OS Platform and Distribution (e.g., macOS 10.14.3)**:
- **Ray installed from (source or binary)**: source
- **Ray version**: 0.7.0.dev2 ab55a1f9
- **Python version**: 3.6.8

<!--
You can obtain the Ray version with

python -c "import ray; print(ray.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->

`DQNPolicyGraph` creates tensorflow assign operations by looping through lists of `self.q_func_vars` and `self.target_q_func_vars` sorted by variable name on lines 390:393. The default sorting is not consistent between the two lists of variable names and as a result the operations can mix up the assignments. The attached code snippet produces the error message below.

```
2019-03-28 18:34:31,402 WARNING worker.py:1397 -- WARNING: Not updating worker name since `setproctitle` is not installed. Install this with `pip install setproctitle` (or ray[debug]) to enable monitoring of worker processes.                                                                                                                                                                                                           
2019-03-28 18:34:31.415440: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
WARNING:tensorflow:From /Users/ristovuorio/miniconda3/envs/ray_fiddle/lib/python3.6/site-packages/tensorflow/python/util/decorator_utils.py:127: GraphKeys.VARIABLES (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.                                                                                                                                                                          
Instructions for updating:
Use `tf.GraphKeys.GLOBAL_VARIABLES` instead.
Traceback (most recent call last):
  File "dqn_fail_demonstrator.py", line 37, in <module>
    trainer = DQNAgent(env="CartPole-v0", config=config)
  File "/Users/ristovuorio/projects/ray_doodle/ray/python/ray/rllib/agents/agent.py", line 280, in __init__
    Trainable.__init__(self, config, logger_creator)
  File "/Users/ristovuorio/projects/ray_doodle/ray/python/ray/tune/trainable.py", line 88, in __init__
    self._setup(copy.deepcopy(self.config))
  File "/Users/ristovuorio/projects/ray_doodle/ray/python/ray/rllib/agents/agent.py", line 377, in _setup
    self._init()
  File "/Users/ristovuorio/projects/ray_doodle/ray/python/ray/rllib/agents/dqn/dqn.py", line 207, in _init
    self.env_creator, self._policy_graph)
  File "/Users/ristovuorio/projects/ray_doodle/ray/python/ray/rllib/agents/agent.py", line 510, in make_local_evaluator
    extra_config or {}))
  File "/Users/ristovuorio/projects/ray_doodle/ray/python/ray/rllib/agents/agent.py", line 727, in _make_evaluator
    async_remote_worker_envs=config["async_remote_worker_envs"])
  File "/Users/ristovuorio/projects/ray_doodle/ray/python/ray/rllib/evaluation/policy_evaluator.py", line 296, in __init__
    self._build_policy_map(policy_dict, policy_config)  File "/Users/ristovuorio/projects/ray_doodle/ray/python/ray/rllib/evaluation/policy_evaluator.py", line 692, in _build_policy_map
    policy_map[name] = cls(obs_space, act_space, merged_conf)
  File "/Users/ristovuorio/projects/ray_doodle/ray/python/ray/rllib/agents/dqn/dqn_policy_graph.py", line 394, in __init__
    update_target_expr.append(var_target.assign(var))
  File "/Users/ristovuorio/miniconda3/envs/ray_fiddle/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 951, in assign
    self._shape.assert_is_compatible_with(value_tensor.shape)
  File "/Users/ristovuorio/miniconda3/envs/ray_fiddle/lib/python3.6/site-packages/tensorflow/python/framework/tensor_shape.py", line 848, in assert_is_compatible_with
    raise ValueError("Shapes %s and %s are incompatible" % (self, other))
ValueError: Shapes (3,) and (11,) are incompatible
```

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->

```
from tensorflow.keras.layers import Dense

import ray
from ray.rllib.models import ModelCatalog, Model
from ray.rllib.agents.dqn import DQNAgent


class DemoNN(Model):
    def _build_layers_v2(self, input_dict, num_outputs, options):
        x = input_dict["obs"]
        x = Dense(1)(x)
        x = Dense(1)(x)
        x = Dense(3)(x)
        x = Dense(1)(x)
        x = Dense(1)(x)
        x = Dense(1)(x)
        x = Dense(1)(x)
        x = Dense(1)(x)
        x = Dense(1)(x)
        x = Dense(1)(x)
        x = Dense(11)(x)
        x = Dense(2)(x)

        return x, x


ray.init(local_mode=True)
ModelCatalog.register_custom_model("demo_nn", DemoNN)
config = {
    "model": {
        "custom_model": "demo_nn",
    },
    "hiddens": [],
    "num_workers": 0,
}
trainer = DQNAgent(env="CartPole-v0", config=config)
```


Discussion:
Would removing the sorted() be the right fix here? According to the TF docs for get_collection(), the variables are already returned in the "order they were collected", which is presumably a consistent order.
Yup, that seems to fix it.

Resolving patch:
diff --git a/python/ray/rllib/agents/dqn/dqn_policy_graph.py b/python/ray/rllib/agents/dqn/dqn_policy_graph.py
--- a/python/ray/rllib/agents/dqn/dqn_policy_graph.py
+++ b/python/ray/rllib/agents/dqn/dqn_policy_graph.py
@@ -387,9 +387,9 @@ def __init__(self, observation_space, action_space, config):
         # update_target_fn will be called periodically to copy Q network to
         # target Q network
         update_target_expr = []
-        for var, var_target in zip(
-                sorted(self.q_func_vars, key=lambda v: v.name),
-                sorted(self.target_q_func_vars, key=lambda v: v.name)):
+        assert len(self.q_func_vars) == len(self.target_q_func_vars), \
+            (self.q_func_vars, self.target_q_func_vars)
+        for var, var_target in zip(self.q_func_vars, self.target_q_func_vars):
             update_target_expr.append(var_target.assign(var))
         self.update_target_expr = tf.group(*update_target_expr)
 


Test patch: