Deep Transformer Q-Networks for Partially Observable Reinforcement Learning
Authors: Kevin Esslinger
Robert Platt
Christopher Amato
Venue: Submitted to ICLR 2023 (2023)

Abstract: Real-world reinforcement learning tasks often involve some form of partial observability where the observations only give a partial or noisy view of the true state of the world. Such tasks typically require some form of memory, where the agent has access to multiple past observations, in order to perform well. One popular way to incorporate memory is by using a recurrent neural network to access the agent's history. However, recurrent neural networks in reinforcement learning are often fragile and difficult to train, susceptible to catastrophic forgetting and sometimes fail completely as a result. In this work, we propose Deep Transformer Q-Networks (DTQN), a novel architecture utilizing transformers and self-attention to encode an agent's history. DTQN is designed modularly, and we compare results against several modifications to our base model. Our experiments demonstrate the transformer can solve partially observable tasks faster and more stably than previous recurrent approaches.

Under review as a conference paper at ICLR 2023
DEEP TRANSFORMER Q-NETWORKS FOR PARTIALLY OBSERVABLE REINFORCEMENT LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
Real-world reinforcement learning tasks often involve some form of partial observ- ability where the observations only give partial or noisy view of the true state of the world. Such tasks typically require some form of memory, where the agent has access to multiple past observations, in order to perform well. One popular way incorporate memory by using a recurrent neural network access the agent's 's history. However, recurrent neural networks in reinforcement learning are often fragile and difficult to train and sometimes fail completely work, we propose Deep Transformer Q-Networks (DTQN), novel architecture utilizing transformers and self-attention to encode an agent's history. DTQN is designed modularly, and we compare results against several modifications to our base model. Our experiments demonstrate that our approach can solve partially observable tasks faster and more stably than previous recurrent approaches.
result. In this
INTRODUCTION
In recent years, deep neural networks have hecome the computational backhone of reinforcement learning, achieving strong performance across a wide array of difficult tasks including games (Mnih
ct al., 2015 Silver et al., 2016) and robotics (Levine et al. 2018 Gao et al. 2020). In particular, Deep Q-Networks (DQN) (Mnih et al. 2015) revolutionized the field of deep RL by achieving super-human performance on Atari 2600 games in the Atari Learning Environment (Bellemare et al. 2013). Since
then, several advancements have been proposed improve DQN (Hessel al. 2018 and deep RL
has been shown to excel in continuous control tasks as well (Haarnoja et al. 2018 8: Fujimoto et al.
2018
However, most Deep RL methods assume the agent is operating within a fully observable environment; that is, one in which the agent has access to the environment's full state information. But this assumption does not hold for many realistic domains due to components such noisy sensors, occluded images, or additional unknown agents. These domains are partially observable, and pose much bigger challenge for RL compared to the standard fully observable setting. Indeed, naive methods often fail to learn in partially observable environments without additional architectural or
training support (Pinto al. 2017 [g] et al. 2018 Ma et al. 2020).
To solve partially observable domains, RL agents may need to remember (some or possibly all) previous observations (Kaelbling al. 1998). As a result, RL methods typically add some of memory component, allowing them to store or refer back to recent observations in order to make more informed decisions. The current offte-of-the-art approaches integrate recurrent neural networks, like LSTMs (Hochreiter & Schmidhuber fully observable Deep RL architectures to process an agent's history Ni al. 202T). But recurrent neural networks (RNNs) can be fragile and difficult to train, often requiring complicated "warm-up" strategies to initialize its hidden state at the start each training batch (Lample Chaplot, 2017). Conversely, the Transformer has been shown to model sequences much better than RNNs and is ubiquitous in natural language processing (NLP) (Devlin et al. 2018) and increasingly common in computer vision (Dosovitskiy al. 20200.
1997) or GRUs (Cho et al. 2014), in conjunction with
Therefore, we propose Deep Transformer Q-Network (DTQN), a novel architecture using self- attention to solve partially observable RL domains. DTQN leverages transformer decoder architec- ture with learned positional encodings to represent an agent's 's history and accurately predict Q Q-values each timestep. Rather than standard approach that trains single next step for given history,

---

Under review as a conference paper at ICLR 2023
propose training regime called intermediate Q-value prediction, which allows to train DTQN on the Q-values generated for each timestep in the agent's 's observation history and provide more robust learning. DTQN encodes agent's history more effectively than recurrent methods, which we show empirically across several challenging partially observable environments. We evaluate and analyze several architectural components, including: gated skip connections (Parisotto et al., 2020). positional encodings, identity map reordering (Parisotto et al.] (020), and intermediate value predic- tion (AI-Riou et al. 2 Our results provide strong evidence that our approach can successfully represent agents' histories in partially observable domains. We visualize attention weights showing DTQN learns an understanding of the domains as it works solve tasks.
2
BACKGROUND
When an environment does not emit its full state to the agent, the problem can be modeled as a Partially Observable Markov Decision Process (POMDP) Kaelbling et al. (1998). A POMDP is formally described as the 6-tuple A, T, R. Ωt O). S, A. and 52 represent the environment's set of states, actions, and observations, respectively. T is the state transition function (ss, a, 8') P(s'ls, a), denoting the probability of transitioning from state to state s' given action a. R describes the reward function R. S A + I; that is, the resultant scalar reward emitted by the environment for an agent that was in some state E S and took some action a E A. And O is the observation function Os',, a, o) P(o|s', a). the probability of observing when action is taken resulting in state s'. At each time step, f, the agent is in the environment's state 8, E S, takes action E AA, manipulates the environment's 's state to some 81+1 E S based on the transition probability T(s; Gt, 81+1) and receives a reward, r; R(st, a.). The goal of the agent is to maximize E E. Trr] its expected discounted return for some discount factor y E [0, 1) Sutton & Barto 2018
Because agents in POMDPs do not have access to the environment's full state information, they must rely on the observations Og l which relate the state via the observation function, ((1411, 4z2 9r) = P(o (.)) In general, agents acting in partially observable space can- not simply use observations as proxy for state, since several states may be aliased into the same observation. Instead, they often consider some form of their full history of information, h4 = {(o, oo). (1),,.1)..... (o--1, a1 Because the history grows indefinitely as the agent pro- ceeds in a trajectory. various ways of encoding the history exist. Previous work has truncated the history to make it a fixed length (Zhu et al., 2017) or used an agent's belief, which represents the estimate of the current state (Kaelbling et al. 1998). Since the deep learning revolution, others have used forms of recurrency, such as LSTMs and GRUs, to encode the history (Hausknecht & Stone 2015
Yang & Nguyen 2021).
2.1
DEEP RECURRENT Q-NETWORKS
Q-Leariing (Watkins& & Dayan 1992 aims to learn a function Q: S x A + IR which represents the value of each state-action pair in an MDP. Given a state s, action a, reward r, next state s', and learning rate a, the Q Q-function is updated with the equation
Q a) Q(s a) + a(r + max Q(5', a') Q(. a)) d'EA
In more challenging domains, however, the state-action space of the environment is often too large to be able to learn exact Q-value for each state-action pair. Instead of learning a tabular Q-function, DQN (Miih et al. 2015) learns an approximate Q-function featuring strong generalization capabilities over similar states and actions. DQN is trained to minimize the Mean Squared Bellman Error
L(0)= = E(s.nn))) [(r max (('''a';6'' Q(s,a;0))*
(2)
a'EAA
where transition tuples of states, actions, rewards, and future states (s, a, r, s') are sampled uniformly from replay buffer, D, of past experiences while training. The target max Q(s', a'; a':0") 0") invokes DQN's target network (parameterized by 8'), which lags behind the main network (parame- terized by 8) to produce more stable updates.
However, in partially observable domains, DQN may not learn a good policy by simply replacing the network's input from states to observations (i.e., an agent can often perform better by remembering
2

---

Under review as a conference paper at ICLR 2023
Go
O
LayerNorm
Bellman Error
Q'1
Linear
Combine
Q2
Q22
IN times
Feed Forward
Loss
LayerNorm
Argmax
Combine
A
Masked Mutti
Position Encodiing
Self Attention
Observation Embedding
Environment
Observation History
D
Architectural diagram of DTQN. Each observation in the history is embedded independently. and Q-values are generated for each observation sub-history. Only the last set Q-values are used to select the next action, but the other Q-values can be utilized for training.
Figure 1:
some history). To address this challenge, Deep Recurrent Q-Networks (DRQN) (Hausknecht & Stone 2015) incorporated histories into the Q-function by way of a long short-term memory (LSTM) layer (Hochreiter & Schmidhuber 1997). In DRQN's 's training procedure, the sampled states are replaced with histories h&:++ = (or 0s+1 O from timestep to step + k sampled randomly within each episode. The hidden state of the LSTM is zeroed at the start of each update.
2.2 TRANSFORMER DECODERS
The transformer architecture Vaswwani et al., 2017). originally introduced for natural language processing, stacks blocks of attention layers (Bahdanau al. 2014) and is typically used to model sequential data. Intuitively, the transformer's attention module receives as input a sequence of tokens (e.g.,& sequence of observations in episode) and the model learns place stronger weights attention on the most important tokens. For more details about the attention module in transformers, refer Appendix A.3
While the original transformer architecture formed an encoder-decoder structure, recent works often either the encoder (Devlin et al. 2018) or the decoder (Radford al. 2018). The key difference between the two is that the decoder applies causal masking to the attention layer, that is, the ith token cannot attend tokens which come later in the sequence. general, the transformer decoder has been shown to perform better on generative tasks like next token prediction, while the transformer encoder able to learn richer representations and excels on tasks such as language understanding.
DTQN utilizes the transformer decoder structure. Given tensor of shape (B, C, D), where B is the batch size, is the context length, and the model's dimensionality size, the transformer decoder layer returns tensor of the same shape, enabling to stack layers on top of each other. The last transformer layer's 's output can then be projected to the desired shape, or sent as input to another network. To ensure the inputs are of the correct shape, we often prepend feature extraction step, such as a lookup embedding for text or integers, a multilayer perceptron for vectors, or convolutional neural network for images.

---

Under review as a conference paper at ICLR 2023
3
RELATED WORK
Since its inception, several works have built upon DRQN. For example, DRQN was shown to beat human test subjects on the challenging 3D VizDoom video game environment (Kempka al., 2016) when augmented with game feature information Lample & Chaplot 2017). A-tiinn-taastee DRQN (ADRQN) Zhu et al. 2017) conditioned the network on the agent's s full history rather than just its observation history. Deep Distributed Recurrent Q-Networks (DDRQN) (Foerster et al.) 2016) extends DRQN into the multi-agen reinforcement learning setting. Like ADRQN, DDRQN conditioned on actions, but also shared weights between each agent, all while forgoing each agent's 's replay buffer to sample from
The concept of attention is also well-studied in the reinforcement learning setting. The most closely related work to ours using attention in deep Q-learning is Deep Attention Recurrent Q-Network (DARQN) (Sorokin et al. 2015), which used attention to aid an LSTM's representation of the agent's history. Similarly, visual attention has been added to DRQN-like architectures in an effort towards creating more interpretable reinforcement learning algorithms (Mott al. 20T9). Unlike which uses self-attention such that agent's 's history forms the queries, keys, and values, these works use the recurrent network's ' last output state to form the queries, and the environment's most recent observation forms the keys and values. In the multi-agent setting, Multi-Actor- Cetioor-Attetton-CCitt Critiic (lgbal & Sha, 2019) created an attention module in which each agent's query is their own observation, and the keys and values are formed by the other agents' observations. Finally, the Simple Neural Attentlve Learner (SNAIL) Mishrr et al. 2017) used attention to develop meta-learning agent capable of transferring its skills to similar but different environments.
work,
The use of transformers reinforcement learning has become more popular within the last few years. In the offline reinforcement learning setting, Decision Transformer (Chen et al. 202T) and Trajectory Transformer (Janner et al. 202T) concurrently proposed the idea of using transformer decoders for sequence modeling, surpassing the current offline RL state of the art. Online Decision Transformer (Zheng et al. 2022) extended Decision Transformer (Chen et al. 202T) by treating the offline training as a pre-training step, and fine-tuned the transformer in the online setting for even better performance. FlexiBiT (Carroll et al., 2022) trained a transformer encoder to learn a variety of inference tasks, such behavioural cloning, forward and backward modeling, and inferring an agent's 's history given its current state. Contrary to our work, which is trained completely online using reinforcement learning, these works are specialized take advantage of an offline RL dataset, and train their agents in supervised way (Schmidhuber) 2019). Other methods use transformers to learn from scratch in the online RL setting, like GTYXL (Parisotto et al. 2020). which modifies the ordering of components within the transformer block, and introduces a new gating mechanism to replace the residual skip connections. We compare the effects of these modifications to our architecture. Lightweight transformers have shown strong performance in text adventure games (Xu et al. 2020). and the transformer encoder was applied to video games (Upadhyay et al., 2019). In contrast to these works, we utilize multi-layer transformer decoder architecture. Vision transformers have been used in conjuction with DQN to stabilize Q-learning with data augmentation, replacing the standard convolutional neural networks (Hansen et al., 2021). The self-attention block in their work attends to features within a single observation whereas ours attends throughout the agent's history.
DEEP TRANSFORMER Q-NETWORK ARCHITECTURE
Transformers seem like a natural fit represent histories in POMDPs, but there are several open questions regarding how to use them best in deep RL. In particular, it is unclear what form of transformer to use, how to integrate it into deep RL. methods and how they should be trained. We chose to build DTQN using a transformer decoder structure incorporating learned position encodings, and train on the Q-values generated for each timestep in the agent's observation history. DTQN takes as input the agent's previous & observations, /4*++ {or, 0z+1, 0t+k-1 }. linearly projects each observation into the dimensionality of the model, and adds positional encodings to add information about the absolute temporal location of each observation. The embedded history is then passed through N transformer layers, and finally projected to the action space of the environment (see Figure and Algorithm []. DTQON outputs set of Q-values relating each observation in the input.
4

---

Under review as a conference paper at ICLR 2023
Algorithm DDTO
function FORWARD PASS(haa+* 01+1, 0(t E Embedding
+ Pos
for Layer I, = ...... N do
QL-1 W 1. - E =
F² = LayerNorm (Combine (MultiHeadAttennioon kL-1 VL-1), F(-1))
E + LayerNorm (Combine (E"),
end for
Oulpul + FFN (E*)
Project output action space
end function
function TRAIN Sample a minibatch of contexts (hea+k OuttR Tra+ki h44144441) from replay buffer
for i 1,... k do
(0) = BLND [(reti1 + maxa'a Qhe d';0') Q(bu:ts; (+t
end for end function function UPDATE
- 0 a Et
end function
While only use the Q-values from the most recent observation during execution, train the network using all generated Q-values, even those relating to the observations at the beginning of the subhistory using the loss function in Algorithm [ This training regime challenges the network to predict the Q-values in situations where it has little to no context, and produces more robust agent. The remainder of this section expands each contribution of the DTQN architecture.
4.1 OBSERVATION EMBEDDINGS AND POSITIONAL ENCODINGS
Before the observation history is passed to DTQN's transformer layers, each observation in the agent's most recent & observations, h k. is linearly projected to the dimensionality of the transformer via a learned observation embedding (see Figure D. After embedding, we add learned positional encoding to each observation based on its position in the observation history. This result, which we call E in Algorithm|
the input to the first transformer layer DTQN.
Position encodings are common practice in transformers, especially for NLP tasks, where they are well studied (Wang Chen 2020). However, the importance position less clear in the reinforcement learning setting. In some control tasks, the temporal position of observation may not have any effect on its importance or meaning to solve the task. For instance, the importance of when the
the priest observation in the classic HcavenHell domain (Bonet 1998) is not dependent observation occurs in the episode. On the other hand, domains with more dynamic state transitions may benefit greatly from the positional information. For this reason, choose to learn our positional encodings as it gives the agent the most flexibility in terms how it chooses to use them. We ablate this choice by comparing our learned positional encodings to sinusoidal positional encodings (used in the original transformer (Vaswani et al. (017)) as well as not using any positional encodings in section 5.4
4.2 TRANSFORMER DECODER STRUCTURE
Like the original GPT architecture Radford al. 2018 each transformer layer in DTQN features two submodules: masked multi-headed self-attention and a position-wise feedforward network. As described in Algorithm| first we project the output of the previous layer, EL-1 the queries, Q, keys, K, and values, V. through the weight matrices Wº, W and WV respectively. After each submodule, that submodule's 's input and output are combined (see the "Combine" step in Figure I followed by LayerNorm (Ba et al. 2016). Finally, after the last transformer layer, project the final embedding (E Algorithm to the action space our environment to represent the Q-value for each action.
5

---

Under review as a conference paper at ICLR 2023
10
0.
04
044
0.
Tamssdape
Timestege
Tirertass
Hallway
(b) Heaven hell
(c) Gridverse memory 5x5
0.8
0.4
0.1
0.4
0.0
9.0
Thraerage
Timeetope
Tiseddppe
(d) Gridverse memory 7x7
(e) Gridverse memory 9x9
Gridverse four rooms
1.0
0.3
09
0.3
0.3
0.2
les
DTQN (ours)
DRON
(g) Car flag
ATTN
DON
(h) Memory cards
Figure 2: Results showing the success rate of DTQN against baselines. DTQN is shown blue, simple attention network (ATTN) shown in brown, Deep Recurrent Q-Network (DRQN) HHausknecht Stone 2015) shown in orange, and Deep Q-Network (DQN) (Mnih et al. 2015) shown in purple. Lines show the mean and shaded regions represent standard error across 5 random seeds. DTQN excels both terms of learning speed as well as final performance, clearly outperforming baselines nearly all domains. Refer to section 5.2 for discussion of results.
DTQN uses residual skip connection (He ct al. 2016) to combine the two streams, matching the original transformer, in favor of other choices of combination layers such as the GRU gating combi- nation layer Parisotto al. (220)). Another contested decision is the position of LayerNorm with respect to each submodule; the original transformer (Vaswani et al. 2017) and original GPT (Rad) ford et al. 2018) apply LayerNorm after the combine step whereas other works have moved the LayerNorm to immediately before the submodule (Radford et al., 2019: Parisotto et al.) 2020) Xu al. 2020p. DTQN applies the LayerNorm after the combine step, a choice found to simple while also demonstrating strong empirical performance. We ablate our choices of network with the aforementioned variants section
4.3 INTERMEDIATE Q-VALUE PREDICTION
DTQN outputs set of Q-values for each timestep in the agent's observation history. During evaluation, DTQN selects the action with the highest Q-value from the last timestep in its history, would therefore be straightforward to train DTQN using just the last timestep's Q-values, since those have the most context work with and are the most informed select the optimal action. This regime, however, is very wasteful, as only a fraction of the generated Q-values actually get used for training. Instead, train DTQN using all generated Q-values. Originally used in the NL.P

---

Under review as a conference paper at ICLR 2023
setting where each position tasked with predicting the next character and formed auxiliary loss (AI-Rfou et al. 2019), we adapt this training regime to the reinforcement learning setting. as shown in Algorithm Note that the for loop depicted in Algorithm can be done in one forward pass of the network because of the causally- maakeed self-attention mechanism.
We ablate training based all Q-values with training only the last timestep's s Q-values in section 5.5 and show the performance gains Table |
EXPERIMENTS
Our experimental evaluation is designed to compare DTQN not only to previous Q-network baselines, but also to ablate our own method with other architectural choices. We evaluate these methods on range of challenging domains featuring partial observations and requiring memory to solve them. We baseline DTQN against Deep Recurrent Q-Networks (DRQN) (Hausknecht & Stone 2013) to show the transformer is more effective history representation module than RNNs, Deep Q-Networks (DQN) (Mnih et al.) 2015) to demonstrate the need for memory solve the task consistently, and against an attention baseline (called "ATTN" in Figure 2 to show the benefits of our architectural choices. ATTN, like the transformer, has observation and position embeddings, attention and feedforward network modules, but does not have LayerNorm or skip connections, and does not stack multiple blocks.
5.1 DOMAINS
We conduct our experiments on 4 different en- vironment sets designed to challenge DTQN in different ways: classic POMDPs, gym-gridverse
(GV) Baisero & Kat 202T), car flag (Nguyen
2021 and memory cards. Hallway (Littman
al. 1995) and HeavenHell (Bonet 1998) are
!
classic navigation POMDPs requiring the agent to take and remember several information gath- ering steps before it can consistently achieve its goal. Gym-Gridverse offers procedurally gen- erated gridworlds containing difficult partially observable tasks. The agent's 's field of view is restricted such that it only the cells in a x 3 grid in front of it (see Figure which introduces state aliasing and forces the agent to gather localizing information before it can successfully solve the task. We evaluate our agents in gridverse environments "Memory" and "Memory Four Rooms", which require the agent to first find the colored information beacons, and then go to the flag whose color matches the beacon. The colors of the flags and beacons are initialized randomly and, in Memory-Fouu Four-Rooue the locations of the flag and beacons are also initialized randomly, increasing the environment's 's difficulty. Example screenshots on ID line, where the must first drive to an oracle
(a) Gridverse Memory 7x7
(b) Gridverse Memory Four Rooms 7x7
Figure 3: Gym-Griiiia Myn--ridverse Memory domains. The top row depicts the state while the bottom row shows the agent's current observation. The colored beacon informs the agent which flag to reach.
the gridverse domains are
shown in Figure 3 Car Flag features flag to learn which direction the finish line is. Memory cards is a novel domain designed to test how much information pairs of cards are hidden to the agent, with one card revealed at each timestep and the agent must guess the position of that card's pair. We chose this of domains to be representative of challenging partially observable problems. For more information regarding the domains, please see Appendix
agent can memorize. Based the popular children's 's memory card game,
5.2 BASELINE COMPARISON
Our evaluation against baselines is shown in Figure Each learning curve shows the success rate of the agent in the environment, where the line is the mean across 5 random seeds, and the shaded region represents standard error. We search across hyperparameters of interest, and select the best
7

---

Under review as a conference paper at ICLR 2023
performing hyperparameter set, prioritizing consistency across domains. For specific hyperparameters and training details, refer to Appendix A
The results in Figure 2 show DTQN outperforms the baselines in terms of learning speed and final performance on nearly all domains. ATTN learns quickly, but rarely reaches optimal performance, and often becomes unstable. DRQN, featuring an LSTM as its memory module, often performs well in our set of domains, but learns slower than DTQN, and is in general less stable. Sometimes, especially in the gridverse memory domains, DRQN's s performance plummets shortly after begins to learn. It then struggles to regain its initial performance gains, taking as many as one million timesteps in the Gridverse memory 7x7 domain to improve its success rate. DQN, designed for MDPs and without any form of memory in its architecture, fails to achieve higher than 50% success rate on any of our domains, exemplifying the difficulty of the domains and the importance of using memory to solve them. These results highlight the effectiveness of DTQN in solving range of POMDPs.
5.3 GRU-GATES AND IDENTITY MAP REORDERING
In this section, we compare DTQN with different forms of the "Combine" step (see Figure D as well different positions of LayerNorm. DTQN's combine step is residual skip connection, and the LayerNorm occurs after both the attention and the feedforward submodules. In contrast, GTrXL (Pariootto et al., 2020) introduced GRU-like gating in the combine step, and identity map reordering, which moves the LayerNorms directly in front of the masked multi self attention and feedforward sections. We compare which uses GRU-like gating, a version of D'TQN which uses identity map reordering, and a version which uses both GRU-like gating and identity map reordering. When both GRU gates and the identity map reordering is used, the architecture resembles GTrXL. However, we do not use the TransformerXL (Dai et al. 2019) as in GTrXL, therefore we are not comparing an exact replica.
DTQN with residual skip connection to version of DTQN
The results for this ablation are shown in Table D DTQN performs competitively with the ablated versions. The variant with only identity map reordering performs significantly worse than the other versions, and the version with both identity map reordering and GRU-like gating performs worst on hallway. Both DTQN and the GRU-like gating variant perform competitively on all three domains we tested. Although we do not use the TrL in our experiments, we would expect to see the same relative performance across if we replaced our transformer decoder with the TransformerXL. A comparison of different transformer backbones, such as Big Bird Zaheer al., 2020), sparse transformers (Child et al. 2019) or the TransformerXL would be interesting future study.
5.4 POSITIONAL ENCODINGS
DTQN uses learned positional encodings to allow the network to adapt to different domains. Partially observable domains will exhibit a broad range of temporal sensitivity, and we want to provide DTQN the flexibility to learn encodings to match its domain. In this section, compare the use of learned positional encodings with the sinusoidal encodings in the original transformer (Vaswani et al., 2017 as well as no positional encodings. The results for this comparison are shown in Table In the memory cards domain, the variant of DTQN without positional encodings performs sigmificantly worse than both our learned encodings as well as the sinusoidal encodings. However, in the gridverse memory task and hallway, the three styles of positional encodings perform comparably. We analyze the resulting learned positional encodings from our trained DTQN agents across various domains in Appendix
5.5 INTERMEDIATE Q- QAVALUE PREDICTION
DTQN predicts and trains on the Q-values generated for each timestep in the agent's observation history. During evaluation, however, we only consider the last timestep's Q-values when determining which action to take. We could, therefore, train in the same way, only training with the last timesteps' Q-values. We compare these two training regimes, and the results for this are shown in Table [] Our results show the variant trained without intermediate Q-values suffers a significant performance decrease. In the memory cards case, DTQN excels and solves the task with nearly 90% success rate, but the variant without intermediate Q-value prediction can barely solve the task 10% of the time. By training on all generated Q-values, we produce more robust and effective agent.
8

---

Under review as a conference paper at ICLR 2023
Table 1: Ablations. We report the final success rate for each variant, averaged across 5 seeds, with standard error.
GV memory 7x7
Memory cards
Hallway
Average
DTQN (ours)
75.2 + 7.2
89.8 1.9
98.3 + 1.0
87.77
Gate and identity
80.3 = 6.4
90.8 + 3.0
67.5 + 10
79.53
Transformer structure
Gate only
78.3 = 4.4
88.9 + 1.2
100 +
89.07
Identity only
65.3 6.7
88.5 1 1.8
69.8 + 10
74.5
Learned (ours) Sinusoidal
75.2 7.2
89.8 + 1.9
98.3 = 1.0
87.77
Positional encodings
83.1
85.5 1.9
+ 3.9
86.87
None
85.5 0.5=3.1 3.7
70.8 + 2.8
99 - 0.4
85.1
Intermediate Q-value
DTQN (ours)
75.2
89.8 1.9
+ 1.0
87.77
prediction
None
56.27 14
9.0 + 2.2
92.8 + 0.7
52.69
DISCUSSION
DTQN outperforms or is competitive with our baselines in terms of learning speed and final performance on all our domains. One additional advantage of transformers is the ability to visu- alize self-attention weights preting the model. Intuitively, the self-attention mechanism allows the agent to prioritize obser- vations in its history which provide with the most information useful in solving its task. The causal masking ensures the agent cannot attend to observations in its future, resembling how the agent will need to perform during execution. While the use of attention weights tool for explainability is still being studied (Jain & Wat
form of inter-
!
lace 2019; Wiegrelfe & Pinter 2019), does allow us to observe which observations the agent finds most valuable in its history. In Figure 4 we visualize a trained DTQN agent's 's attention weights from a trajectory gridverse memory 7x7. Crucially, the observation including the green beacon (circle with X. magnified on right) strongly attended to by all future observations, indicating the DTQN agent has correctly learned which observations are important solving the task. When the agent the green flag (magnified on left), it attends the observation with green beacon to ensure it selects the correct flag. We provide additional attention visualizations in Appendix D
Figure 4: Attention hars for gridverse memory 7x7. Bars go from left right, and observations go top to bottom (i.e. the second observation attended the first and second observation). Attention weights below 0.2 have been removed for visibility.
7 CONCLUSION
In this work we introduce Deep Transformer Q-Networks, a novel architecture for solving challenging partially observable domains with reinforcement learning. DTQN incorporates the transformer decoder, which excels in generating Q-values for each timestep of the agent's observation history. We train the model on all generated Q-values, enabling an efficient training regime and faster learning. DTQN also utilizes learned positional encodings, empowering the model to learn domain-specific encodings which match the temporal dependencies of the environment. We explore and ablate several architectural structures, and find our choices either outperform or are at least competitive with all tested variants. Finally, we provide modular code implementation DTQN that is easy to extend and modify, which we hope the research community will be able to use as we expect our approach to serve as the basis and benchmark for future transformer-based methods in partially observable reinforcement learning.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rami Al-Rfou, Dokook Choe. Noah Constant, Mandy Guo, and Llion Jones. Character-level language modeling with deeper self-attention. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 3159-3166 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXi:: 1607064550 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0477 2014.
Andrea Baisero and Sammic Katt. gym gym-gridverse: Gridworld domains for fully and partially observable reinforcement learning. pettps:/githie wow.aba sero/ wwweee//ggggrriiveeeeeeee
com/ aba
2021.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents, Journal of Artificial Intelligence Research, 47:253-279, 2013.
Blai Bonet. Solving large pomdps using real time dynamic programming. In In Proc. AAAI Fall Symp. on POMDPs, 1998.
Micah Carroll, Jessy Lin, Orr Paradise, Raluca Georgescu, Mingfei Sun. David Bignell, Stephanie Milani, Katja Hofmann, Matthew Hausknecht, Anca Dragan, al. Towards flexible inference sequential decision problems via bidirectional transformers. arXiv preprint arXiv:2204.13: 2022.
Lili Chen, Kevin La, Aravind Rajeswaran, Kimin I Lee, Aditya Grover, Misha Laskin, Picter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34, 2021.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever: Generating long sequences with sparse transformers. arXiv preprint arXiv: 1904. 10509, 2019.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv: 1259, 2014.
Zihang Dai, Zhilin Yang. Yiming Yang. Jaime Carbonell, Quoe V Le. and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv: 190I.:1901.0000
2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:18/00008805, 2018
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov. Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 11999, 2020.
Jakob N Foerster, Yannis M Assael, Nando Freitas, and Shimon Whiteson. Learning communicate solve riddles with deep distributed recurrent g-networks. arXiv preprint arXiv:1602.0266 2016.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference machine learning. pp. 1587-1596. PMLR, 2018.
Wenbo Gao, Laura Graesser, Krzysztof Choromanski, Xingyou Song, Nevena Lazie, Pannag Sanketi, Vikas Sindhwani, and Navdcep Jaitly. Robotic table tennis with model-free reinforcement learning. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5556-5563, IEEE, 2020.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithus and applications. arXiv preprint arXiv: 2018.
Nicklas Hansen, Hao Su, and Xiaolong Wang- Stabilizing deep q-learning with convnets and vision transformers under data augmentation. Advances in Neural Information Processing Systems, 34, 2021.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. arXiv preprint arXiv:1507.066 06527, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
10

---

Under review as a conference paper at ICLR 2023
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan. Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference on artificial intelligence, 2018.
Sepp Hochreiter and Jürgen Schmidhuher. Long short-term memory. Neural computation, 973):1735- 1997.
Maximilian Igl, Luisa Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson. Deep variational rein- forcement learning for pomdps. International Conference on Machine Learning, pp. 2117-2128 PMLR. 2018.
Shariq Iqbal and Fei Sha. Actor-attentionn-cittic for multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 2961-2970. PMLR, 2019.
Sarthak Jain and Byron C Wallace. Attention not explanation. arXiv preprint arXiv: 1902, 2019.
Michael Janner. Qiyang Li, and Sergey Levine. Offline reinforcement learning one big sequence modeling problem. Advances in neural information processing systems, 34, 2021.
Leslie Pack Kaelbling, Michael Littman, and Anthony Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2)99-134, 1998.
Michal Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojeiech Jaskowski. Vizdoom: A doom- based ai research platform for visual reinforcement learningg In 2016 IEEE Conference on Computational Intelligence and Games (CIG), pp. 8, 2016. doi 10. 201.
DONCIIG.201...00 2.7603833333
Guillaume Lample and Devendra Singh Chaplot. Playing fps games with deep reinforcement learning. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen. Learning hand eye coordination for robotic grasping with deep learning and large-scale data collection. The International journal of robotics research, 37(4-5):221 431.1 2018.
Michael L. Littman, Authony R. Cassandra, and Leslie Pack Kaelbling. Learning policies for partially observable environments: Scaling up. 1995.
Xiao Ma, Peter Karkus, David Hsu, Wee Sun Lee, and Nan Ye. Discriminative particle filter reinforcement learning for complex partial observations. arXiv preprint arXiv:2002.0968 2020.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. arXiv preprint arXi::1707.03144 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529 533, 2015.
Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo Jimenez Rezende. Towards interpretable reinforcement learning using attention augmented agents. Advances in Neural Information Processing Systems, 32, 2019.
Hai Nguyen. Pomdp robot domains, htttps:/ com hai-aahiiaaa
2021.
Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free rl can be strong bascline for many pomdps. arXiv preprint arXiv:2110.05038, 2021.
Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderherg, Raphael Lopez Kaufman, Aidan Clark. Seb Noury, et al. Stabilizing transformers for reinforcement learning. In International Conference on Machine Learning, pp. 7487-7498. PMLR, 2020.
Lerrel Pinto, Marcin Andrychowiez. Peter Welinder, Wojeiech Zaremba, and Pieter Abbeel. Asymmetric actor critic for image-hased robot learning. arXiv preprint ariiv 2017.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models unsupervised multitask learners. OpenAl blog, 188); 2019.
Juergen Schmidhuber. Reinforcement learning upside down: Don't predict rewards-just map them to actions. arXiv preprint 19X.0:19/2.02885, 2019.
11

---

Under review as a conference paper at ICLR 2023
David Silver. Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche. Julian Schrittwieser, Joannis Antonoglou, Veda Pannecrshelvam, Mare Lanctot, et al. Mastering game of go with deep neural networks and tree search. nature, 5 4489, 2016.
Ivan Sorokin, Alexey Seleznev, Mikhail Pavlov, Aleksandr Fedorov, and Anastasiia Ignateva. Deep attention recurrent q-network. arXiv preprint arXiv:15/2.0169. 2015.
Richard Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second net http/boo///htte hto.//thhhhhoooooeeooooooooooo-e 2nd html
edition, 2018. URL / http:///o./.ttpt/t/o/oo////tto..
Uddeshya Upad Nikunj Shah, Sucheta Ravikanti, and Mayanka Medhe. Transformer based reinforcement learning for games. arXiv preprint arXiv: 1912.03918, 2019.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp- 5998 6008, 2017.
Yu-An Wang and Yun-Nung Chen, What do position embeddings leam? an empirical study of pre-trained language model positional encoding. arXiv preprint 2rXiv:2010.04903, 2020.
Christopher J. C. H. Watkins and Peter Dayan. Q-learning. In Machine Learning, pp. 279-292, 1992.
Sarah Wiegreffe and Yuval Pinter. Attention not explanation. arXiv preprint arXiv: pete:::990000222 2019.
Yunqiu Xu, Ling Chen, Meng Fang, Yang Wang, and Chengqi Zhang. Deep reinforcement learning with transformers for text adventure games. 2020 IEFE Conference Games (CoG), pp. 65-72. IEFF, 2020.
Zhihan Yang and Hai Huu Nguyen. Recurrent off-policy baselines for memory-based continuous control. In Deep RL Workshop NeurlPS 2021, 2021.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslic, Chris Albenti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Yang, et al. Big bird: Transformers for longer sequences. Advances Neural Information Processing Systems, 33: 33:17283-17297. 2020.
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. arXiv preprint arXiv:2202.0560) 2022.
Pengfei Zhu, Xin Li, Pascal Poupart, and Guanghui Miao. On improving deep reinforcement learning for pomdps. arXiv preprint arXiv: (tX.:v//04.009900 2017.
12


Reviews:
[{"review_id": "wSCnZ4TkNx", "reviewer": "Reviewer_QsGx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a deep reinforcement learning method for partially observable environments named Deep Transformer Q-Networks (DTQN). The key feature of DTQN is that, for a given chunk of history sampled from the replay buffer, a loss is taken over predicted Q-values for *all* timesteps in the history (rather than just the last one) by applying a causal mask in the transformer and thus efficiently providing a denser training signal for the agent. The architecture is evaluated on a number of tasks in partially observable environments and compared to a number of deep Q-learning baselines (one with no memory, one RNN-based, and an attention-based), in which DTQN is shown to perform best on average. A number of ablations are also conducted, demonstrating the importance of intermediate Q-value prediction as well as architectural choices such as positional encodings and how the attention outputs are combined.", "summary_of_the_review": "This paper proposes a method with a clear motivation for tackling the important problem of efficient RL in partially observable environments. The key contribution of intermediate Q-value prediction can be an important one (and is empirically shown to give a boost to performance in relevant tasks), but more analysis is ideally required to demonstrate how it helps performance, and the ablations, while interesting, serve as a bit of a distraction to the main method.", "strength_and_weaknesses": "Strengths\n- The problem that the method aims to tackle, RL in partially observable environments, is an important one, and the method itself is well-motivated, i.e. leveraging the transformer architecture to predict many Q-values at once for a denser training signal.\n- The approach is explained well, along with an informative diagram in Figure 1.\n- The related work section is strong, explicitly placing and contrasting the method in context of other transformer-based RL methods.\n- The tasks seem to be well-chosen, in the sense that they are partially observable and require some for of memory to complete.\n- The choice of baselines is reasonable, particularly when considered in combination with the ablations where, for example, the “Gate and identity” ablation is highlighted to be almost the same architecture as the GTrXL (just without the memory buffer) and thus serves as a relevant comparison to the state of the art in transformer architecture for RL.\n\nOpportunites for improvement\n- Ablations / analyses. While a number of ablations were performed and described in the paper, they might be improved in the following ways:\n    - In Table 1, it is shown that using gating in the combine step gives a better performance on average than the residual skip connection used in DTQN - given this, why is gating not used in the final DTQN architecture?\n    - While the ablations for the combine step and the positional encodings are interesting to see, they seem tangential to the key contribution of the method, which is the intermediate Q-value predictions. Indeed an ablation is performed on the intermediate predictions, demonstrating its significant effect on performance, but there could be more investigation into *how* it helps. For example, how much of the benefit of intermediate Q-value training comes from the extra training signal that comes from training on multiple states at once, versus a potential regularising effect that comes from the Q-value predictions of earlier states in the sequence having far less context than those later in the sequence? Perhaps this could be determined by running experiments that fix the number of predicted Q-values in training, but vary how much temporal context is used for each Q-value prediction.\n- While in the introduction, it is clear what the contribution of the method is (the intermediate Q-value prediction), throughout the paper the other architectural choices (which as far as I am aware are not novel, e.g. GTrXL used causal masking, positional encodings, but please correct me if I’m wrong) act as a bit of a distraction to this. I think the paper would be stronger if it focused more on the key contribution and understanding how it helps performance, as mentioned above.\n\nMinor comments\n- Table 1 it would be useful to bolden the top performing models on each task\n- Section 4.3 last line: “Figure 1” -> “Table 1”\n- Legend in Figure 2 quite small\n- Appendix E links to higher up in the paper\n- The naming of the ‘combine’ step is confusing, why not just call it a residual skip connection?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MGmqphQKY_", "reviewer": "Reviewer_CoHr", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a transformer-based DQN agent.", "summary_of_the_review": "This paper proposed a transformer-based DQN agent.", "strength_and_weaknesses": "Weaknesses:\n1. Incremental work. Replacing RNN with Transformer in deep reinforcement learning is not a new idea! There are so many works with similar ideas, such as Decision Transformer[1], GATO[2], [3], [4], [5], [6] and [7].\n2. Bad writing. The figures in this paper are really bad. For example, in Figure 2(a)(c), the blue curves are out of the figure.\n3. Toy benchmarks. The authors claim that they are better than DQN or DRQN, they should conduct experiments on popular RL benchmarks, such as Atari, MuJoco, and DeepMind Lab. \n4. Weak baselines. They should compare their method with real SOTA methods, such as PPO, Rainbow and IMPALA.\n\n\n- [1] Chen L, Lu K, Rajeswaran A, et al. Decision transformer: Reinforcement learning via sequence modeling[J]. Advances in neural information processing systems, 2021, 34: 15084-15097.\n- [2] Reed S, Zolna K, Parisotto E, et al. A generalist agent[J]. arXiv preprint arXiv:2205.06175, 2022.\n- [3] Laskin M, Wang L, Oh J, et al. In-context Reinforcement Learning with Algorithm Distillation[J]. arXiv preprint arXiv:2210.14215, 2022.\n- [4] Parisotto, Emilio, et al. \"Stabilizing transformers for reinforcement learning.\" International conference on machine learning. PMLR, 2020.\n- [5] Banino, Andrea, et al. \"Coberl: Contrastive bert for reinforcement learning.\" arXiv preprint arXiv:2107.05431 (2021).\n- [6] Furuta, Hiroki, Yutaka Matsuo, and Shixiang Shane Gu. \"Generalized decision transformer for offline hindsight information matching.\" arXiv preprint arXiv:2111.10364 (2021).\n- [7] Micheli, Vincent, Eloi Alonso, and François Fleuret. \"Transformers are sample efficient world models.\" arXiv preprint arXiv:2209.00588 (2022).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "D68lpCMAEuY", "reviewer": "Reviewer_zcAu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces deep transformer Q-networks (DTQN) for learning in partially observable environments. The authors train DTQN with a double Q-learning objective using on-policy samples. Given a trajectory of observations till now, observations are encoded using a transformer decoder with a masked attention to generate Q values for every step in trajectory. Typical Q learning loss with a fixed target network is applied to every step and the network is updated using the sum of losses from each step. When evaluated on several synthetic partially observable environments, DTQN is shown to perform better than LSTM variants including attention and traditional DQN. Ablation studies show that predicting Q values of intermediate steps is important for better performance, gating mechanism can help improve performance, and learning positional encodings is overall better but depends on the environment. Visualizing attention probabilities shows that bottleneck states are attended more.", "summary_of_the_review": "While results on synthetic POMDPs are interesting, the model is derivative of previous work and no realistic task is studied.", "strength_and_weaknesses": "**Strengths** Applying transformers to POMDPs is an interesting direction. The empirical results improve upon previous baselines including LSTMs and DQN.\n\n**Weaknesses**\n1. The paper improves RNNs using Transformers in a straightforward way. It is not clear what is the main difference of the Transformer studied in this work compared to something like DT and variants. DT uses observations, actions, and rewards while DTQN uses only observations. Later variants improve DT using reward prediction, multi-task learning etc. I believe environments like Atari are also partially observable. Aside from some ablations, new insights that would be worth investigating further could really help.\n\n2. Task that are studied are also synthetic. I think it is important to test the model on more challenging POMDPs even though current model fails to achieve strong performance.\n\n3. Additional baselines that are tailored towards RL is also needed. At least DT, GATO, or variants.\n\n4. Ablation experiments show that intermediate Q value prediction is important but it is not clear if this is due to simply using $k$ times more samples or actual intermediate Q value prediction. Are you using the same number of training samples for both experiments?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8-uFKZmOnp", "reviewer": "Reviewer_oBw9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new transformer-based architecture to solve POMDP problems. The main contributions include using transformer decoder structure to address partially observable RL domains and utilizing intermediate Q-values for the training. Results on four environment sets are reported.", "summary_of_the_review": "This paper has a good motivation and proposes a new solution for the POMDP problem. Experiments on four environment sets demonstrate its effectiveness. The major concern is the lack of comparison with state-of-the-art/similar approaches.\n", "strength_and_weaknesses": "Strength:\n1. The proposed transformer-based architecture is novel and the most effective one for DQN baselines.\n2. The paper is clearly presented and well-written.\n3. The experimental results demonstrate the effectiveness of DTQN in solving POMDP problems.\n\nWeakness:\n1. The proposed method is only compared with DQN and DRQN while there lacks evidence to show that DTQN can lead to state-of-the-art performance, which makes it less convincing.\n2. More discussions and comparison results between [1] and DTQN should be provided.\n3. [2] also presents a very similar architecture (but it uses transformer encoder). More reviews and comparison results should be added.\n4. GRU-like gating can lead to better results. Why not adopt the gating structure in the proposed architecture? More discussions are required.\n\n[1] TRANSFORMER BASED REINFORCEMENT LEARNING FOR GAMES.\n\n[2] Stabilizing Transformer-Based Action Sequence Generation For Q-Learning.\n\nMinor: Different structures in Section 5.3 can be illustrated in the supplementary material.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject