Repository: ray-project/ray

Issue:
[rllib] _get_torch_exploration_action doesn't support tuple action dist
<!--Please include [tune], [rllib], [autoscaler] etc. in the issue title if relevant-->
### System information

*  **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Mac OS 10.15.4
*  **Ray installed from (source or binary)**: binary (via pip)
*  **Ray version**: 0.8.6., but nothing seems to have changed on master
*  **Python version**: 3.7

### What is the problem?

When using tuple action distributions (as advised in #6372) and exploration is disabled, the line:

https://github.com/ray-project/ray/blob/a462ae2747afbeb9047e443cd51e67e3fe0b49e6/rllib/utils/exploration/stochastic_sampling.py#L75

from `_get_torch_exploration_action` raises the following exception:

```
AttributeError: 'tuple' object has no attribute 'size'
```

A simple fix that supports any type of distribution would be:
```python
logp = torch.zeros_like(action_dist.sampled_action_logp())
```

I can submit a PR if it helps. 

### Reproduction (REQUIRED)

Exact command to reproduce: python `rllib_cartpole.py` for the following file

```python
import gym.envs.classic_control
from gym.spaces import Tuple, Discrete

import ray
from ray import tune


class CustomCartpole(gym.envs.classic_control.CartPoleEnv):
    """Add a dimension to the cartpole action space that is ignored."""

    def __init__(self, env_config):
        super().__init__()
        # if override_actions is false this is just the Cartpole environment
        self.override_actions = env_config['override_actions']
        if self.override_actions:
            # 2 is the environment's normal action space
            # 4 is just a dummy number to give it an extra dimension
            self.original_action_space = self.action_space
            self.action_space = Tuple([Discrete(2), Discrete(4)])
            self.tuple_action_space = self.action_space

    def step(self, action):
        # call the cartpole environment with the original action
        if self.override_actions:
            self.action_space = self.original_action_space
            return super().step(action[0])
        else:
            return super().step(action)


def main():
    ray.init()
    tune.run(
        "PPO",
        stop={"episode_reward_mean": 50},
        config={
            "env": CustomCartpole,
            "env_config": {'override_actions': True},
            "num_gpus": 0,
            "num_workers": 1,
            "eager": False,
            "evaluation_interval": 1,
            "evaluation_config": {
                "explore": False,
            },
            "framework": "torch",
        },
    )


if __name__ == '__main__':
    main()
```


- [x] I have verified my script runs in a clean environment and reproduces the issue.
- [ ] I have verified the issue also occurs with the [latest wheels](https://docs.ray.io/en/latest/installation.html).



Discussion:
The proposed fix makes sense to me. We could alternatively try to get the batch dimension of the tuple, but I don't see an existing helper method for that, so your proposal is probably simpler.

And yeah, a PR would be great!

Resolving patch:
diff --git a/rllib/utils/exploration/stochastic_sampling.py b/rllib/utils/exploration/stochastic_sampling.py
--- a/rllib/utils/exploration/stochastic_sampling.py
+++ b/rllib/utils/exploration/stochastic_sampling.py
@@ -72,5 +72,5 @@ def _get_torch_exploration_action(action_dist, explore):
             logp = action_dist.sampled_action_logp()
         else:
             action = action_dist.deterministic_sample()
-            logp = torch.zeros((action.size()[0], ), dtype=torch.float32)
+            logp = torch.zeros_like(action_dist.sampled_action_logp())
         return action, logp


Test patch: