Repository: ray-project/ray

Issue:
[rllib] incorrect model output for DQN with torch and dueling=false 
<!--Please include [tune], [rllib], [autoscaler] etc. in the issue title if relevant-->

### What is the problem?

The output fo the DQN model is not within the action space. 

Something is wrong when constructing the torch model when dueling is off. The output dimension of the model is equal to whatever is passed in "fcnet_hiddens" instead of being of the size of the action space. 

*Ray version and other system information (Python version, TensorFlow version, OS):*
- ray==0.9.0.dev0 
- python 3.6.10
- mac OS

### Reproduction (REQUIRED)

```python
import ray
from ray import tune

ray.init()

config = {
    "env": "CartPole-v1",
    "num_workers": 1,
    "train_batch_size": 128,
    "learning_starts": 128,
    "model": {"fcnet_hiddens": [32]},
    "dueling": False ,
    "framework": "torch"
}

tune.run("DQN", name="MWE", config=config, stop={"training_iteration": 100})
```

- [x] I have verified my script runs in a clean environment and reproduces the issue.
- [x] I have verified the issue also occurs with the [latest wheels](https://docs.ray.io/en/latest/installation.html).



Discussion:

Resolving patch:
diff --git a/rllib/agents/dqn/dqn_torch_model.py b/rllib/agents/dqn/dqn_torch_model.py
--- a/rllib/agents/dqn/dqn_torch_model.py
+++ b/rllib/agents/dqn/dqn_torch_model.py
@@ -53,46 +53,45 @@ def __init__(
         self.dueling = dueling
         ins = num_outputs
 
-        # Dueling case: Build the shared (advantages and value) fc-network.
         advantage_module = nn.Sequential()
-        value_module = None
-        if self.dueling:
-            value_module = nn.Sequential()
-            for i, n in enumerate(q_hiddens):
-                advantage_module.add_module("dueling_A_{}".format(i),
-                                            nn.Linear(ins, n))
-                value_module.add_module("dueling_V_{}".format(i),
+        value_module = nn.Sequential()
+
+        # Dueling case: Build the shared (advantages and value) fc-network.
+        for i, n in enumerate(q_hiddens):
+            advantage_module.add_module("dueling_A_{}".format(i),
                                         nn.Linear(ins, n))
-                # Add activations if necessary.
-                if dueling_activation == "relu":
-                    advantage_module.add_module("dueling_A_act_{}".format(i),
-                                                nn.ReLU())
-                    value_module.add_module("dueling_V_act_{}".format(i),
+            value_module.add_module("dueling_V_{}".format(i),
+                                    nn.Linear(ins, n))
+            # Add activations if necessary.
+            if dueling_activation == "relu":
+                advantage_module.add_module("dueling_A_act_{}".format(i),
                                             nn.ReLU())
-                elif dueling_activation == "tanh":
-                    advantage_module.add_module("dueling_A_act_{}".format(i),
-                                                nn.Tanh())
-                    value_module.add_module("dueling_V_act_{}".format(i),
+                value_module.add_module("dueling_V_act_{}".format(i),
+                                        nn.ReLU())
+            elif dueling_activation == "tanh":
+                advantage_module.add_module("dueling_A_act_{}".format(i),
                                             nn.Tanh())
+                value_module.add_module("dueling_V_act_{}".format(i),
+                                        nn.Tanh())
 
-                # Add LayerNorm after each Dense.
-                if add_layer_norm:
-                    advantage_module.add_module("LayerNorm_A_{}".format(i),
-                                                nn.LayerNorm(n))
-                    value_module.add_module("LayerNorm_V_{}".format(i),
+            # Add LayerNorm after each Dense.
+            if add_layer_norm:
+                advantage_module.add_module("LayerNorm_A_{}".format(i),
                                             nn.LayerNorm(n))
-                ins = n
-            # Actual Advantages layer (nodes=num-actions) and
-            # value layer (nodes=1).
+                value_module.add_module("LayerNorm_V_{}".format(i),
+                                        nn.LayerNorm(n))
+            ins = n
+
+        # Actual Advantages layer (nodes=num-actions).
+        if q_hiddens:
             advantage_module.add_module("A", nn.Linear(ins, action_space.n))
-            value_module.add_module("V", nn.Linear(ins, 1))
-        # Non-dueling:
-        # Q-value layer (use main module's outputs as Q-values).
-        else:
-            pass
 
         self.advantage_module = advantage_module
-        self.value_module = value_module
+
+        # Value layer (nodes=1).
+        if self.dueling:
+            value_module.add_module("V", nn.Linear(ins, 1))
+            self.value_module = value_module
 
     def get_advantages_or_q_values(self, model_out):
         """Returns distributional values for Q(s, a) given a state embedding.
diff --git a/rllib/examples/parametric_actions_cartpole.py b/rllib/examples/parametric_actions_cartpole.py
--- a/rllib/examples/parametric_actions_cartpole.py
+++ b/rllib/examples/parametric_actions_cartpole.py
@@ -48,7 +48,7 @@
             # TODO(ekl) we need to set these to prevent the masked values
             # from being further processed in DistributionalQModel, which
             # would mess up the masking. It is possible to support these if we
-            # defined a a custom DistributionalQModel that is aware of masking.
+            # defined a custom DistributionalQModel that is aware of masking.
             "hiddens": [],
             "dueling": False,
         }


Test patch: