Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: * Summary
This paper introduces architecture modifications for self-attention to stabilize transformers in reinforcement learning.
The new architecture, Gated Transformer-XL, replaces the order of the layer norm blocks to preserve an identity mapping.
Multiple existing gating layers are proposed to replace the residual connections of transformer.
The new architecture is compared against MERLIN and LSTM on DMlab-30, and further ablation studies are done on Numpad and Memory Maze.
It is noted that they use the recent V-MPO objective to train LSTM and transformer.
Their results show that transformers are able to learn in memory-intensive environments, with some gating combinations surpassing LSTM.

* Decision
This paper presents promising empirical results, however the experiments are limited, making it difficult to place in the broader work.
In addition, the contribution is incremental and not well-motivated.
I would recommend a weak rejection.
Still, I think the paper is well written and could be improved upon.

* Reasons
While the empirical results are impressive, they are not put into context.
DMlab-30 is still a relatively new environment suite and it is difficult to place the result of this paper in the context of broader work.
In addition, the comparison is against LSTM on a new objective.
While Transformer is able to beat LSTM on the same objective, it is unclear whether that is a success of the objective or the architecture.
In Numpad, the transformer architecture shows an improvement over LSTM, but no comparisons are made to any other memory-based agents.
The hyperparameter studies on Memory Maze also show improvements in memory-related tasks, but do not help in understanding of the proposed work.

The choice of architecture modification is also not well motivated.
As the paper mentions, initializing near identity has been shown to be important in the supervised learning literature.
For the topic of this paper however, I do not think this adequetly explains the instability of transformers in reinforcement learning.
In the related work section for example, the paper notes that gating mechanisms have been used to handle the vanishing gradients problem.
The paper also notes that vanishing gradients is not an issue in transformers.
Hence, it is unclear why gating would stabilize transformers for reinforcement learning.

The paper is overall well written and the ideas developed are clear.
Unfortunately, the impressive results on DMlab are not sufficient for both the lack of deeper empirical study and better theoretical motivation for the architecture modifications.
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: The paper explores a transformer for reinforcement learning. The authors demonstrate that Canonical Transformer is unstable. The authors introduce two modifications to the Canonical Transformer. The first is to move the layer normalization layer to the input stream. The second is to replace residual connections with gating layers. The experimental results show that (1) the first modification, i.e., moving the layer normalization layer to the input stream significantly stabilizes the training; (2) Gated Recurrent Unit (GRU) gating seems to be most effective gating mechanism.
My decision is Weak Reject, considering the following aspects.

Positive points: (1) The experiments seem solid. The authors have evaluated the overall performance, as well as hyperparameters, seeds, and ablations. (2) Moving the layer normalization layer to the input stream seems to be surprisingly effective. This could be an interesting finding. (3) The paper is well organized.

Negative points: (1) Lack of experiments on benchmark and large environments. The authors did not evaluate their model on the widely used benchmark Atari-57. Also, it is unclear whether the proposed transformer can scale to large environments. (2) Lack of understanding of the layer normalization. The authors provide some explanations about why the reordering works, but they seem not intuitive. More analysis about why the reordering works would significantly enhance this paper. 

Specific questions: (1) Have you tried simply removing the layer normalization layer? (2) TrXL-I moves two layer-normalization layers together. Have you tried only moving one of them? Which modification contributes more? (3) Could you provide more explanations about why the modification of the layer normalization layer works? (4) Have you experimentally validated the proposed hypothesis as to why the Identity Map Reordering, such as recording the evolution of the produced values in the submodules?

review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: This paper is motivated by the unstable performance of Transformer in reinforcement learning, and tried several variants of Transformer to see whether some of them can stabilize the Transformer. The experimental results look good, however, I have problems in understanding the motivation, the intuition of the proposed methods, the experimental design, and the general implication to the research community that is using the Transformer in their day-to-day research.
First, the paper was based on the hypothesis of the authors that the Transformer is not stable, however, there is no comprehensive study on the unstability, and deep understanding on the root cause of it. It would be much more convincing to give a form definition of unstability and to add experimental study and theoretical analysis to the motivation part, instead of just based on a hypothesis.

Second, the proposal of the new structures (e.g., reordering the layer normalization, adding the gating layer) are quite ad hoc. There is not very solid motivation and theoretical analysis on why they could solve the unstable problem of the Transformer.  For example, by changing the order of layer normalization, there are direct identity mapping from the first layer to the last layer, making the information flow smoother. However, why this will make the Transformer more stable? The hypothesis and intuitive analysis are not very convincing. For another example, why replacing the residual connection with the gating layer can make the Transformer more stable? It seems to me that these are mostly heuristics, but not verified or strongly motivated solutions.

Third, if the proposal is sound, it should not be effective only for reinforcement learning. It should be able to improve the performance or stability of the Transformer in general (e.g., in NLP tasks). However, there is no experiments and discussions regarding this.

Fourth, the experiments on the reinforcement learning is a little narrow, and many famous RL benchmarks and environments were not tested. This makes it unclear whether the proposed approach is generally effective.


**I read the author responses, however, they do not really change my assessment on the paper.
review_title: Official Blind Review #3

Meta-review: This paper proposes  architectural modifications to transformers, which are promising for sequential tasks requiring memory but can be unstable to optimize, and applies the resulting method to the RL setting, evaluated in the DMLab-30 benchmark.

While I thought the approach was interesting and the results promising, the reviewers unanimously felt that the experimental evaluation could be more thorough, and were concerned with the motivation behind of some of the proposed changes.

Decision: Reject