Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Clarity and quality:

The paper is well written and the ideas are motivated clearly both in writing and with block diagram panels.  Also the fact  that the paper considers different  variants of  the idea  adds to the quality of the paper. May main concern is with the quality of results which is limited to some toy/synthetic problems. Also the comparison with the previous work is missing.The paper would benefit from  a more in depth numerical analysis of this approach both by applying it to more challenging/standard domains such as Mujoco and also by comparing the results with prior approaches such as A3C, DDPG and TRPO.

Originality, novelty and Significance:

The paper claims that the approach is novel in the context of policy gradient and Deep RL. I am not sure this is entirely the case since there is a recent work from Google Brain (https://arxiv.org/pdf/1705.05035.pdf ) which consider almost the identical idea with the same variation in the context of DQN and policy gradient (they call their policy gradient approach  Prob SDQN).  The Brain paper also  makes a much more convincing case with their numerical analysis, applied to more challenging domains such as control suite. The paper under review  should acknowledge this prior work and discuss the similarities and the differences. Also since the main idea and the algorithms are quiet similar to the Brain paper I believe the novelty of this work is at best marginal.
review_title: The paper deals with the problem of large-multi-dimensional action space in RL. It proposes an auto-regressive model to represent the policy, in which the value of action at each dimension will be represented as a function of state and the "previous" dimensions. I think the idea is very interesting and  useful but it is already explored in the context of Deep RL before. So It is not entirely novel contrary to the authors claim.

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors suggest introducing dependencies between actions in RL settings with multi-dimensional action spaces by way of two mechanisms (using an RNN and making partial action specification as part of the state); they then introduce entropy pseudo-rewards whose maximization corresponding to joint entropy maximization.

In general, the multidimensional action methods either seem incremental or non novel to me. The combined use of the chain rule and RNNs (LSTM or not) to induce correlations in multi-dimensional outputs is well know (sequence-to-sequence networks, pixelRNN, etc.) and the extension to RL presents no difficulties, if it is not already known. Note very related work in https://arxiv.org/pdf/1607.07086.pdf and https://www.media.mit.edu/projects/improving-rnn-sequence-generation-with-rl/overview/ .

As for the MMDP technique, I believe it is folklore (it can for instance be found as a problem in a problem set - http://stellar.mit.edu/S/course/2/sp04/2.997/courseMaterial/topics/topic2/readings/problemset4/problemset4.pdf). Note that both approaches could be combined; the first idea is essentially a policy method, the second, a value method. The second method could be used to provide stronger, partial action-conditional baselines (or even critics) to the first method.

The entropy derivation are more interesting - and the smoothed entropy technique is as far as I know, novel. The experiments are well done, though on simple toy environments.

Minor:
- In section 3.2, one should in principle tweak the discount factor of the modified MDP to recover behavior identical to the original one with large action space. This should be noted (alternatively, the discount between non-environment transitions should be set to 1).

- From the description at the end of 3.2, and figure 1.b, it seems actions fed to the MMDP feed-forward network are not one-
hot; I thought this was pretty surprising as it would almost certainly affect performance? Note also that the collection of feed-forward network which collectively output the joint vector can be thought of as an RNN with non-learned state transition.

- Since the function optimized can be written as an expectation of reward+pseudo-reward, the proof of theorem 4 can be simplified by using generic score-function optimization arguments (see Stochastic Computation Graphs, Schulman et al).

review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors present two autoregressive models for sampling action probabilities from a factorized discrete action space. On a multi-agent gridworld task and a multi-agent multi-armed bandit task, the proposed method seems to benefit from their lower-variance entropy estimator for exploration bonus. A few key citations were missing - notably the LSTM model they propose is a clear instance of an autoregressive density estimator, as in PixelCNN, WaveNet and other recently popular deep architectures. In that context, this work can be viewed as applying deep autoregressive density estimators to policy gradient methods. At least one of those papers ought to be cited. It also seems like a simple, obvious baseline is missing from their experiments - simply independently outputting D independent softmaxes from the policy network. Without that baseline it's not clear that any actual benefit is gained by modeling the joint distribution between actions, especially since the optimal policy for an MDP is provably deterministic anyway. The method could even be made to capture dependencies between different actions by adding a latent probabilistic layer in the middle of the policy network, inducing marginal dependencies between different actions. A direct comparison against one of the related methods in the discussion section would help better contextualize the paper as well. A final point on clarity of presentation - in keeping with the convention in the field, the readability of the tables could be improved by putting the top-performing models in bold, and Table 2 should almost certainly be replaced by a boxplot.
review_title: Simple autoregressive model for action spaces, but missing some baselines

Meta-review: The paper has some interesting ideas around auto-regressive policies and estimating their entropy for exploration. The use of autoregressive policies in RL is not particularly novel, and the estimate of entropy for such models is straightforward. Finally, the experiments focus on very simple tasks.
Decision: Reject