Venue: ICLR 2023 Conference

Reviews:

final_score: rating: null
review_content: The authors state that they do not compare their method with SOTA RL algorithms and popular benchmarks just because they want to solve POMDP tasks. This statement is not convincing! Reasons below:

(1) There are many Atari games that are POMDPs (e.g.,  Montezuma's Revenge), but the authors state that "stacking frames is sufficient". If the method is good enough, why did the authors just choose a toy grid world as the testbed? If you only evaluate your method on a simple self-built game, how can you verify the generality of the proposed method?

(2) The MDP is just a special case of POMDPs. If your method can work well on POMDPs, it will also work well on MDP.

(3) The authors say that they do not compare their method with PPO, because PPO can not solve POMDP. That's not correct! We can use RNN units in the neural network of PPO to capture historical information. Even if PPO can only work for MDP, it will be more convincing for the experimental results when the proposed method really outperforms previous SOTA methods.

I will keep my rating, and I suggest that other reviewers and ACs take a closer look at this paper. This paper has bad technical representations and insufficient experimental evaluations, which is definitively not good enough for a top conference.

review_title: Comments on the rebuttals.

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper proposed a transformer-based DQN agent.
strength_and_weaknesses: Weaknesses:
1. Incremental work. Replacing RNN with Transformer in deep reinforcement learning is not a new idea! There are so many works with similar ideas, such as Decision Transformer[1], GATO[2], [3], [4], [5], [6] and [7].
2. Bad writing. The figures in this paper are really bad. For example, in Figure 2(a)(c), the blue curves are out of the figure.
3. Toy benchmarks. The authors claim that they are better than DQN or DRQN, they should conduct experiments on popular RL benchmarks, such as Atari, MuJoco, and DeepMind Lab. 
4. Weak baselines. They should compare their method with real SOTA methods, such as PPO, Rainbow and IMPALA.


- [1] Chen L, Lu K, Rajeswaran A, et al. Decision transformer: Reinforcement learning via sequence modeling[J]. Advances in neural information processing systems, 2021, 34: 15084-15097.
- [2] Reed S, Zolna K, Parisotto E, et al. A generalist agent[J]. arXiv preprint arXiv:2205.06175, 2022.
- [3] Laskin M, Wang L, Oh J, et al. In-context Reinforcement Learning with Algorithm Distillation[J]. arXiv preprint arXiv:2210.14215, 2022.
- [4] Parisotto, Emilio, et al. "Stabilizing transformers for reinforcement learning." International conference on machine learning. PMLR, 2020.
- [5] Banino, Andrea, et al. "Coberl: Contrastive bert for reinforcement learning." arXiv preprint arXiv:2107.05431 (2021).
- [6] Furuta, Hiroki, Yutaka Matsuo, and Shixiang Shane Gu. "Generalized decision transformer for offline hindsight information matching." arXiv preprint arXiv:2111.10364 (2021).
- [7] Micheli, Vincent, Eloi Alonso, and François Fleuret. "Transformers are sample efficient world models." arXiv preprint arXiv:2209.00588 (2022).

clarity,_quality,_novelty_and_reproducibility: - quality: low
- clarity: low
- originality: low
summary_of_the_review: This paper proposed a transformer-based DQN agent.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces deep transformer Q-networks (DTQN) for learning in partially observable environments. The authors train DTQN with a double Q-learning objective using on-policy samples. Given a trajectory of observations till now, observations are encoded using a transformer decoder with a masked attention to generate Q values for every step in trajectory. Typical Q learning loss with a fixed target network is applied to every step and the network is updated using the sum of losses from each step. When evaluated on several synthetic partially observable environments, DTQN is shown to perform better than LSTM variants including attention and traditional DQN. Ablation studies show that predicting Q values of intermediate steps is important for better performance, gating mechanism can help improve performance, and learning positional encodings is overall better but depends on the environment. Visualizing attention probabilities shows that bottleneck states are attended more.
strength_and_weaknesses: **Strengths** Applying transformers to POMDPs is an interesting direction. The empirical results improve upon previous baselines including LSTMs and DQN.

**Weaknesses**
1. The paper improves RNNs using Transformers in a straightforward way. It is not clear what is the main difference of the Transformer studied in this work compared to something like DT and variants. DT uses observations, actions, and rewards while DTQN uses only observations. Later variants improve DT using reward prediction, multi-task learning etc. I believe environments like Atari are also partially observable. Aside from some ablations, new insights that would be worth investigating further could really help.

2. Task that are studied are also synthetic. I think it is important to test the model on more challenging POMDPs even though current model fails to achieve strong performance.

3. Additional baselines that are tailored towards RL is also needed. At least DT, GATO, or variants.

4. Ablation experiments show that intermediate Q value prediction is important but it is not clear if this is due to simply using $k$ times more samples or actual intermediate Q value prediction. Are you using the same number of training samples for both experiments?

clarity,_quality,_novelty_and_reproducibility: The paper is written clearly and results on synthetic POMDP tasks show improvement. I found the model to be derivative of previous work on transformers in RL. The results should be reproducible.
summary_of_the_review: While results on synthetic POMDPs are interesting, the model is derivative of previous work and no realistic task is studied.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new transformer-based architecture to solve POMDP problems. The main contributions include using transformer decoder structure to address partially observable RL domains and utilizing intermediate Q-values for the training. Results on four environment sets are reported.
strength_and_weaknesses: Strength:
1. The proposed transformer-based architecture is novel and the most effective one for DQN baselines.
2. The paper is clearly presented and well-written.
3. The experimental results demonstrate the effectiveness of DTQN in solving POMDP problems.

Weakness:
1. The proposed method is only compared with DQN and DRQN while there lacks evidence to show that DTQN can lead to state-of-the-art performance, which makes it less convincing.
2. More discussions and comparison results between [1] and DTQN should be provided.
3. [2] also presents a very similar architecture (but it uses transformer encoder). More reviews and comparison results should be added.
4. GRU-like gating can lead to better results. Why not adopt the gating structure in the proposed architecture? More discussions are required.

[1] TRANSFORMER BASED REINFORCEMENT LEARNING FOR GAMES.

[2] Stabilizing Transformer-Based Action Sequence Generation For Q-Learning.

Minor: Different structures in Section 5.3 can be illustrated in the supplementary material.
clarity,_quality,_novelty_and_reproducibility: The proposed architecture is novel for solving POMDP problems and the contributions are clearly presented. The code is also provided.
summary_of_the_review: This paper has a good motivation and proposes a new solution for the POMDP problem. Experiments on four environment sets demonstrate its effectiveness. The major concern is the lack of comparison with state-of-the-art/similar approaches.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a deep reinforcement learning method for partially observable environments named Deep Transformer Q-Networks (DTQN). The key feature of DTQN is that, for a given chunk of history sampled from the replay buffer, a loss is taken over predicted Q-values for *all* timesteps in the history (rather than just the last one) by applying a causal mask in the transformer and thus efficiently providing a denser training signal for the agent. The architecture is evaluated on a number of tasks in partially observable environments and compared to a number of deep Q-learning baselines (one with no memory, one RNN-based, and an attention-based), in which DTQN is shown to perform best on average. A number of ablations are also conducted, demonstrating the importance of intermediate Q-value prediction as well as architectural choices such as positional encodings and how the attention outputs are combined.
strength_and_weaknesses: Strengths
- The problem that the method aims to tackle, RL in partially observable environments, is an important one, and the method itself is well-motivated, i.e. leveraging the transformer architecture to predict many Q-values at once for a denser training signal.
- The approach is explained well, along with an informative diagram in Figure 1.
- The related work section is strong, explicitly placing and contrasting the method in context of other transformer-based RL methods.
- The tasks seem to be well-chosen, in the sense that they are partially observable and require some for of memory to complete.
- The choice of baselines is reasonable, particularly when considered in combination with the ablations where, for example, the “Gate and identity” ablation is highlighted to be almost the same architecture as the GTrXL (just without the memory buffer) and thus serves as a relevant comparison to the state of the art in transformer architecture for RL.

Opportunites for improvement
- Ablations / analyses. While a number of ablations were performed and described in the paper, they might be improved in the following ways:
    - In Table 1, it is shown that using gating in the combine step gives a better performance on average than the residual skip connection used in DTQN - given this, why is gating not used in the final DTQN architecture?
    - While the ablations for the combine step and the positional encodings are interesting to see, they seem tangential to the key contribution of the method, which is the intermediate Q-value predictions. Indeed an ablation is performed on the intermediate predictions, demonstrating its significant effect on performance, but there could be more investigation into *how* it helps. For example, how much of the benefit of intermediate Q-value training comes from the extra training signal that comes from training on multiple states at once, versus a potential regularising effect that comes from the Q-value predictions of earlier states in the sequence having far less context than those later in the sequence? Perhaps this could be determined by running experiments that fix the number of predicted Q-values in training, but vary how much temporal context is used for each Q-value prediction.
- While in the introduction, it is clear what the contribution of the method is (the intermediate Q-value prediction), throughout the paper the other architectural choices (which as far as I am aware are not novel, e.g. GTrXL used causal masking, positional encodings, but please correct me if I’m wrong) act as a bit of a distraction to this. I think the paper would be stronger if it focused more on the key contribution and understanding how it helps performance, as mentioned above.

Minor comments
- Table 1 it would be useful to bolden the top performing models on each task
- Section 4.3 last line: “Figure 1” -> “Table 1”
- Legend in Figure 2 quite small
- Appendix E links to higher up in the paper
- The naming of the ‘combine’ step is confusing, why not just call it a residual skip connection?
clarity,_quality,_novelty_and_reproducibility: The paper is clear and well written. The novelty seems to be limited to the intermediate Q-value prediction, which can still be a significant contribution, especially if there were more analysis into why it helps performance. The paper states that code is provided, but I could not see it in the supplementary materials.
summary_of_the_review: This paper proposes a method with a clear motivation for tackling the important problem of efficient RL in partially observable environments. The key contribution of intermediate Q-value prediction can be an important one (and is empirically shown to give a boost to performance in relevant tasks), but more analysis is ideally required to demonstrate how it helps performance, and the ablations, while interesting, serve as a bit of a distraction to the main method.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper addresses the problem of learning in partially observable reinforcement learning settings. The authors propose a deep transformer architecture that builds on deep Q networks. The approach is evaluated in several partially observable domains, including a standardized maze task and a novel memory cards domain.

Reviewers appreciated the importance of the problem addressed in this paper. The proposed approach is well motivated. Empirical results show improvements over LSTM and DQN-based baselines, and results suggest that the proposed intermediate Q-value prediction helps stabilize training.

At the same time, reviewers raise several concerns. The proposal of integrating transformers in an online RL approach is not novel - many variants have been proposed. Reviewers feel that the paper falls short in comparison to the latest state of the art approaches in this area, and that evaluation is performed on a set of domains with little to no overlap with state of the art approaches. As a result, the novelty and contribution of the work falls short. In addition, reviewers suggested that additional analysis could help generate deeper insights, e.g., in the role that intermediate Q-value prediction may play in stabilizing training.

Authors addressed some of the reviewer concerns in the rebuttal phase. In particular, they added two experiments to the appendix, one to better understand the role of intermediate Q-value prediction, and one to compare the proposed agent to a "gate and identity" transformer similar to an existing approach. Further, there was a detailed discussion between the authors and one of the reviewers about the choice of empirical testbed.

In its current state, important concerns remain. Establishing proper comparisons between existing approaches and the present work remains challenging, as test beds differ and the comparison to GTrXL remains a comparison to a variant of the approach. The paper could be substantially strengthened by focusing on the elements of the approach that are novel, and systematically establishing (within the main paper) how these elements contribute to improving performance over state of the art approaches. Overall, the paper is not recommended for acceptance in its present form.
justification_for_why_not_higher_score: Important concerns remain regarding the empirical evaluation and placement in comparison with the state of the art.
Decision: Reject