Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper proposes the use of macro (i.e. aggregated) actions to address reinforcement learning tasks. The inspiration presented is from hierarchical grammar representations, and the method is tested on a subset of Atari games. The paper is overall well written, although many paragraph demonstrate a level of polish inadequate for a top level submission (repetitions, typos, etc.).
The main idea pursued in the work is extremely interesting and with likely important implications to recent DRL. The concept though is far from new: a quick search for "macro action reinforcement learning" points to a NIPS '99 paper from J. Randlov, though on top of my mind there should be even older work on the topic.
The perspective proposed of considering macro actions as atoms in a grammar is certainly intriguing, but the work proposed does not develop the concept. The macro actions are identified as patterns in action sequences, then built in straight hierarchies, without any distinction in type of atoms nor any rule to effectively make up a grammar.
The related work section is extremely lacking, with no work older than 2016. The introduction presents more background, marginally older than that (up to 2012), when grammars make for an entire field of study with decades of history.
The process is interesting and incorporates plenty of useful experience, which I would personally be glad to see published, although in the current context is insufficient as stand-alone contribution.
On a more personal note, I suggest the authors not to get discouraged, as I strongly believe such an avenue of research is worthy investigating. A few research questions which I think should be asked are:
- Are the agents actually learning to play the game? Just render the game with one of your best players. For example, achieving a score of 360 on Qbert barely takes constant down input, and the fact that comparable scores have been published before is of no support.
- Are long action sequences always useful? For Qbert for example an average move length of 8 learned from an initial, untrained policy, is sufficient to get off the screen consistently. While the Abandon Ship protocol can mitigate this, the RL exploration phase is done by random action selection (consider explicit exploration instead), and the action space grows fast from the small initial 6 actions with the addition of all the macro actions, possibly limiting the exploration capability and biasing towards the use of longer macro actions even when sub-optimal.
- Mitigate the claims. I would love to "eventually help make RL a universally practical and useful tool in modern society", but unfortunately I think no single contribution can today make such a claim.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper introduced a way to combine actions into meta-actions through action grammar. The authors trained agents that executes both primitive actions and meta-actions, resulting in better performance on Atari games. Specifically, meta-actions are generated after a period of training from collected greedy action sequences by finding repeated sub-sequences of actions. Several tricks are used to speed up learning and to make the framework more flexible. The most effective one is HAR (hindsight action replay), without which the agent's performance reduces to that of the baseline.

Overall, this paper could be a great contribution for the following reasons: 
1. The paper is well written, with clear performance advantages over the baseline. 
2. The paper provides a different perspective for HRL research, namely that we might not need to have a hierarchical policy to benefit from hierarchical actions that spans over many timesteps. 
3. From this paper's ablation study for HAR, it seems to suggest that even with similar experiences, one can get better performance by substituting actions with temporally abstracted actions, propagating value function errors further back in time. If so, this work can serve as a novel counterexample to the claim made in Nachum et al., 2019.

The authors may want to address the following:
1. They may want to compare and contrast to other works in HRL that also does temporally abstracted actions. e.g. h-DQN, Feudal networks. Or even to repeating the same action N times-- a simple trick commonly used in Atari  -- which can be seen as a very naive form of action grammar.
2. The main claim that having Action Grammar improves sample efficiency is not proved clearly. Apart from the ablation study, it's not immediately clear whether sticking to sub-sequence of actions are inherently beneficial for exploration, or that the agent somehow learned faster with the same set of samples collected.
3. It seems that the algorithm may be the most effective in areas where a baseline algorithm can learn to perform at least some meaningful action sequences already. Otherwise the Action Grammar may not extract meaningful subsequences. Has the algorithm been tried on sparse-reward games?

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The authors propose a method for learning macro-actions in a multi-step manner, where Sequitur, a grammar calculator, is leveraged together with an entropy-minimisation based strategy to find relevant macro-actions. The authors propose a system to bootstrap the weights of these macro-actions when increasing the policy's action space, and a system to increase the amount of data (and bias it towards macro-actions) used to learn a policy for when conditioned on this increased action-space. The authors test against a subset of the Arcade Learning Environment suite.

Overall, I'm conflicted by this paper. On one hand, the framework is interesting, and their method involves the usage and exploration of quite a few nice ideas; on the other hand, (a) the quality of the scientific contribution is hard to judge considering the significant differences between the proposed baselines and and their methods, and (b) the experimental section doesn't provide a lot of qualitative analysis and signal wrt. each component.

Furthermore, I have the following issues / questions:

1. I'm not convinced that the usage of Sequitur to build the macro-actions is sufficient to declare this work novel wrt. other macro-action papers. Sequitur usage in this case seems to be particularly overkill, since ultimately all that the method seems to be doing is finding frequent sequences of actions, which can be done quite fast (at least given the amount of training steps) simply using search and pattern matching. From my point of view, there doesn't seem to be a lot in that work that exploits the fact that the macro-actions are constructed as a "grammar" (beyond, maybe, HAR)

2. The Abandon Ship heuristics is effectively a fixed termination policy, which makes the entire setup somewhat similar to options. In this case, what is traded is learning complexity for a hyperparameter and a significant restriction in how the macro-actions terminate. Did you attempt to learn this function at all? Do you have any insights / experiments that might show how the heuristics behaves with changing values of $z$? Would it be possible to plot the distribution of attempted vs executed move lengths rather than then averages (since I doubt they would be normally distributed)?

3. Given points 1 and 2, the literature review is lacking - there's a lot of prior work done on macro-actions in both RL and robotics (planning, HRI, ...) that goes well beyond the few recent papers mentioned by the authors, and I think it might be necessary to mention work on options where the termination function is structured / biased in some way.

4. I have some doubt the experimental setup for DDQN fairly gives a fair assessment of the method. When using a pretrained features, the problem becomes significantly easier, and thus AG-DDQN potentially doesn't need to deal with the problem of learning extremely bad / noisy macro-actions. I would love to see the method trained for a more reasonable amount of frames without pre-training. Also, did the 8 / 20 atari games get chosen randomly, or were they picked based on some environment features?

5. How do the Q-values for the policy evolve with training time? The proposed methods seem to somewhat imply that the action space grows unboundedly, which might seriously destroy the policy for tasks that require much longer training. Would it be possible to add a paragraph about how the policies evolve in at least some of these environments? Are macro-actions used most of the times after some full iterations? How many <learning -> action distillation> iterations are actually done in the current experiments?

At this point, I cannot recommend the acceptance of this work, however I'd be willing to reconsider my rating if the authors address the above points.

review_title: Official Blind Review #1

Meta-review: The topic of macro-actions/hierarchical RL is an important one and the perspective this paper takes on this topic by drawing parallels with action grammars is intriguing. However, some more work is needed to properly evaluate the significance. In particular, a better evaluation of the strengths and weaknesses of the method would improve this paper a lot.
Decision: Reject