Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions
Authors: Yevgen Chebotar
Quan Vuong
Karol Hausman
Fei Xia
Yao Lu
Alex Irpan
Aviral Kumar
Tianhe Yu
Alexander Herzog
Karl Pertsch
Keerthana Gopalakrishnan
Julian Ibarz
Ofir Nachum
Sumedh Anand Sontakke
Grecia Salazar
Huong T Tran
Jodilyn Peralta
Clayton Tan
Deeksha Manjunath
Jaspiar Singh
Brianna Zitkovich
Tomas Jackson
Kanishka Rao
Chelsea Finn
Sergey Levine
Venue: CoRL 2023 Poster (2023)
Keywords: Reinforcement Learning
Offline RL
Transformers
Q-Learning
Robotic Manipulation

Abstract: In this work, we present a scalable reinforcement learning method for training multi-task policies from large offline datasets that can leverage both human demonstrations and autonomously collected data. Our method uses a Transformer to provide a scalable representation for Q-functions trained via offline temporal difference backups. We therefore refer to the method as Q-Transformer. By discretizing each action dimension and representing the Q-value of each action dimension as separate tokens, we can apply effective high-capacity sequence modeling techniques for Q-learning. We present several design decisions that enable good performance with offline RL training, and show that Q-Transformer outperforms prior offline RL algorithms and imitation learning techniques on a large  diverse real-world robotic manipulation task suite.

Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions
Yevgen Chebotar', Quan Vuong", Alex Irpan, Karol Hausman, Fei Xia, Yao Lu, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, Keerthana Gopalakrishnan, Julian Ibarz, Ofir Nachum, Sumedh Sontakke, Grecia Salazar, Huong T Tran, Jodilyn Peralta, Clayton Tan, Deeksha Manjunath, Jaspiar Singht, Brianna Zitkovich, Tomas Jackson, Kanishka Rao, Chelsea Finn, Sergey Levine
Google DeepMind
Abstract: this work, we present scalable reinforcement learning method for training multi-task policies from large offline datasets that can leverage both hu- man demonstrations and autonomously collected data. Our method uses a Trans- former to provide scalable representation for Q-functions trained via offline tem- poral difference backups. We therefore refer to the method as Q-Transformer. By discretizing each action dimension and representing the Q-value of each tion dimension as separate tokens, we can apply effective high-capacity sequence modeling techniques for Q-learning. We present several design decisions that able good performance with offline RL training, and show that Q- Q-Transformer outperforms prior offline RI algorithms and imitation learning techniques large diverse real-world robotic manipulation task suite. The project's 's website and videos can be found gt rans f former g t.ub.io io
Introduction
Robotic learning methods that incorporate large and diverse datasets in combination with high- capacity expressive models, such as Transform- ers [1, 3, 4, 6], have the potential to acquire generalizable and broadly applicable policies that perform well a wide variety of tasks [1, 2]. For example, these policies can follow natural language instructions [4, 7]. perform multi-stage behaviors [8, 9]. and generalize broadly across environments, objects, and even robot morpholo- gies [10, 3]. However, many of the recently pro- posed high-capacity models in the robotic learn- ing literature are trained with supervised learn- ing methods. As such, the performance of the re- sulting policy is limited by the degree which human demonstrators can provide high-quality demonstration data. This is limiting for sons. First, we would like robotic systems that proficient than human teleoperators, ploiting the full potential of the hardware to per- form tasks quickly, fluently, and reliably. Second, we would like robotic systems that get better with autonomously gathered experience, rather than relying entirely on high-quality demonstrations.
Human demonstrations
Autonomous data
Mixed quality data
Autoregressive
Conservative regularization
Monte-Carlo returns
Q-learnigg
edion dimnir Covelrspee
Q-Transformer
Figure 1: O-Transformer enables training high- capacity sequential architectures on mixed qual- ity data. Our policies are able to improve upon human demonstrations and execute variety of manipulation tasks in the real world.
Reinforcement learning in principle provides both of these capabilities. large-scale robotic RL in varied settings, such robotic grasping and stacking [11, 12]. learning heterogeneous tasks with human-specified rewards [13], learning multi-task policies [14, 15], learn- ing goal-conditioned policies [16. 17, 18, 19], and robotic navigation [20, 21, 22, 23, 24). However,
number of promising recent advances demonstrate the successes of
Equal contribution.
Corresponding emails: chebotar@googeeeee com,
quantevvong@google..coo
7th Conference on Robot Learning (CoRI. 2023), Atlanta, USA.

---

training high-capacity models such as Transformers using RL algorithms has proven more difficult to instantiate effectively at large scale. In this paper, we aim to combine large-scale robotic learning from diverse real-world datasets with modern high-capacity Transformer-basse policy architectures.
smaller convo- lutional neural networks [11, 14]) with Transformer is conceptually straightlorward, devising a methodology that effectively makes use of such architectures High-capacity models only make sense when we train on large and diverse datasets - -malll, narrow datasets simply do not require this much capacity and do not benefit from it. While prior works used simulation to create such datasets [2, 25, 26], the most representative data comes from the real world [12. 11, 14]. Therefore, we focus on reinforcement learning methods that can use Transform- ers and incorporate large, previously collected datasets via offfine RL. Offline RL methods train on prior data, aiming to derive the most effective possible policy from a given dataset. Of course, this dataset can be augmented with additionally autonomously gathered data, but the training is separated from data collection, providing an appealing workflow for large-scale robotics applications [27].
While in principle simply replacing existing architectures (e.g., ResNets [15]
considerably more challenging.
Another issue in applying Transformer models to RL is to design RL systems that can effectively train such models. Effective offline RL methods generally employ Q-function estimation via tem- poral difference updates [28]. Since Transformers model discrete token sequences, we convert the Q-function estimation problem into a discrete token sequence modeling problem, and devise a suit- able loss function for each token in the sequence. Naively discretizing the action space leads to ex- ponential blowup in action cardinality, so we employ a per-dimension discretization scheme, where each dimension of the action space is treated as a separate time step for RI., Different bins in the dis- cretization corresponds to distinet actions. The per-dimension discretization scheme allows us to use simple discrete-action Q-learning methods with conservative regularizer to handle distributional shift [29, 30J. We propose a specific regularizer that minimizes values of every action that was not taken in the dataset and show that our method can learn from both narrow demonstration-Llkk data and broader data with exploration noise. Finally, we utilize a hybrid update that combines Monte Carlo and step returns with temporal difference backups [3 I] and show that doing so improves the performance of our Transformer-base offline RL method on large-scale robotic learning problems.
In summary, our main contribution is the Q-Transformer, Transformer-based architecture for robotic offline reinforcement learning that makes use of per-dimension tokenization of Q-values and can readily be applied to large and diverse robotic datasets, including real-world data. We summarize the components of Q-Transformer in Figure 1. Our experimental evaluation validates the Q-Transformer by learning large-scale text-conditioned multi-task policies, both in simulation for rigorous comparisons and in large-scale real-world experiments for realistic validation. Our real-world experiments utilize dataset with 38,000 successful demonstrations and 20,000 failed autonomously collected episodes on more than 700 tasks, gathered with a fleet of 13 robots. Q- Transformer outperforms previously proposed architectures for large-scale robotic RL [15, 14], as well as previously proposed caansformer-bssee models such as the Decision Transformer [32, 33).
2 Related Work
Offline RL has been extensively studied in recent works [34, 35, 36, 37, 35, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 391. Conservative Q-learning (CQL) [29] learns policies constrained to a conservative lower bound of the value function. Our goal is not to develop new algorithmic principle for offline RL, but to devise an offline RL system that can integrate with nighccppccity Transformers, and scale to real-world multi-task robotic learning. We thus develop a version of CQL particularly effective for training large Transformer-based Q-functions on mixed quality data. While some works have noted that imitation learning outperforms offline RI. on demonstration data [49], other works showed offline RL techniques to be effective with demonstrations both in theory and in practice [50, 15]. Nonetheless, setting that combines "narrow" demonstration data with "broad" sub-optimal (c.g., autonomously collected) data is known to be particularly difficult [51, 52, 53], though it is quite natural in many robotic learning settings where might want to augment core set of demonstrations with relatively inexpensive low-quality autonomously collected data. We believe that the effectiveness of our method in this setting is of particular interest practitioners.
Transformer-based architectures [54] have been explored in recent robotics research, both to learn generalizable task spaces [55, 56, 57, 58, 8, 59] and to learn multi-task or even multi-domain se- quential policies directly [2, 1, 6, 3]. Although most of these works considered Transformers in a supervised learning setting, e.g., learning from demonstrations [4, 5], there are works on employ-

---

Legend
max
dimension
Q- Q-Targeis
0 R
Target Q Q-values of dataset actions
++0
+
Target Q-valuss of unobserved actions
MC
Q-Transformer steps
Environment
History of states St St-wit wif
steps
conservative each action dimension
Monte-Carlo
Return
Figure 2: Q-values update for each action dimension at timestep t. Given a history of states, we update the Q-values all bins in all action dimensions. The Q-values of the discrete action bins of the dataset actions are trained via the Bellman update (green boxes). The values of action bins not observed in the dataset are minimized towards zero (red boxes). The Q-targets of all action dimen- sions except the last one are computed using maximization over the next action dimension within the same time step. The Q-target of the last action dimension computed using the discounted max- imization of the first dimension of the next time step plus the reward. We also incorporate Monte Carlo returns by taking the maximum of the computed Q-targets and the return-to-go.
ing Transformers for RL and conditional imitation learning [32, 20, 60, 33]. In experiments, compare to Decision Transformer (DT) particular [32], which extends conditional imitation learn- ing with reward conditioning [61, 62] to sequence models, and structurally resembles imitation learning methods that have been used successfully for robotic control. Although DT incorporates elements of RL (namely, reward functions), it does not provide mechanism to improve over the demonstrated behavior or recombine parts of the dataset to synthesize more optimal behaviors, and indeed is known to have theoretical limitations [63]. On the other hand, such imitation- bastdd recipes are popular perhaps due to the difficulty of integrating Transformer architectures with more powerful temporal difference methods (e.g., Q-learning). We show that several simple but important design decisions are needed to make this work, and our method significantly outperforms non-TD methods such as DT. as well imitation learning, on our large-scale multi-task robotic control evaluation. Extending Decision Transformer, Yamagata et al. [64] proposed to use Q-function in combination policy. but the Q-function itself did not use Transformer-bar bassed architec- ture. Our Q-function could in principle be combined with this method, but our focus is specifically on directly training Transformers to represent Q-values.
with Transformer-basse
To develop a Transformer-based Q-learning method, we discretize each action space dimension, with each dimension acting as distinet time step. Autoregressive generation of discrete actions has been explored by Metz et al. [65], who propose a hierarchical decomposition of an MDP and then utilize LSTM [66] for autoregressive discretization. Our discretization scheme similar but simpler. in that we do not use any hierarchical decomposition but simply treat each dimension as a time step. However, since our goal to perform offline RL scale with real-world image based tasks (vs. the smaller state tasks learned via online RL by Metz et al. [65]). we present a number of additional design decisions to impose a conservative regularizer, enabling training our Transformer- based offline Q-learning method at scale, providing a complete robotic learning system.
Background
In RL, we learn policies ग that maximizes the expected total reward in a Markov decision process (MDP) with states s, actions a. discount factor 7 E (0. 1], transition function T(s'|s,,) a) and reward function R(s,a), Actions a have dimensionality d Value-based RL approaches learn Q-function ((s,, a) representing the total discounted return Σ. R(s,, az), with policy x (a]s) arg max, Q a). The Q-function can be learned by iteratively applying the Bellman operator [67]:
B ("Q(s,, ax) R(st, az) + max Q(++11 a++1),
ac+1
approximated via function approximation and sampling. The offline RL setting assumes access to an offline dataset of transitions or episodes, produced by some unknown behavior policy (T(aass) but does not assume the ability perform additional online interaction during training. This is appealing

---

Carrers
images
EfficienNeet
Pasitional encoding
Transformer
Q-vatues each action
One-hot action
Sef-Allarion Layara (8x)
concat
AEg
Pick
Encoder
Hrmndiion
Action
prnsicuely predicted
embedding
Figure 3: Q-Transformer network architecture, as applied to our multi-task robotic control setting. The encoding of the observations concatenated with embeddings of the previous predicted action dimensions and processed by Transformer layers. We apply a sigmoid the Transformer output to produce Q-values (normalized lie in the range [0, 1J) for each of the action value bins. Finally, one-hot action vectors are constructed by taking the arg max over all bins and are fed back the network to predict the Q-values of the next action dimensions. The language instruction is encoded with Universal Sentence Encoder [68] and then fed to FiLM EfficientNet [69, 70] network together with the robot camera images.
for real-world robotic learning, where on-policy data collection time-consuming. Learning from offline datasets requires addressing distributional shift, since in general the action that maximizes ([$++1,0++]] might lie outside of the data distribution. One approach to mitigate this is to add a conservative penalty [29, 52] that pushes down the Q-values Q a) for any action outside of the dataset, thus ensuring that the maximum value action in-distribution.
In this work, we consider tasks with sparse rewards, where a binary reward R E (0. 1} (indicating success or failure) is assigned at the last time step of episodes. Although our method not specific to this setting, such reward structure is common in robotic manipulation tasks that either succeed or fail on each episode, and can be particularly challenging for RL due to the lack of reward shaping.
Q-Transformer
In this section, introduce Q- Transformer, architecture for offline Q-learning with Transformer models, which based on three main ingredients. First, we describe how we apply discretization and autoregression enable TD-learning with Transformer architectures. Next, introduce particular conservative Q-function regularizer that enables learning from offline datasets. Lastly, we show how Monte Carlo and n-step returns can be used to improve learning efficiency.
4.1 Autoregressive Discrete Q-Learning
Using Transformers with Q-learning presents two challenges: (1) we must tokenize the inputs to ef- fectively apply attention mechanisms, which requires discretizing the action space; (2) we must per- form maximization Q-values over discretized actions while avoiding the curse dimensionality. Addressing these issues within the standard Q-learning framework requires The intuition behind our autoregressive Q-lcarning update is to treat each action dimension as essen- tially separate time step. That way, we can discretize individual dimensions (ID quantities), rather than the entire action space, avoiding the curse of dimensionality. This can be viewed as a simplified version of the scheme proposed in [65], though we apply this to high-capacity Transformer models, extend it to the offline RL setting, and scale it up to real-world robotic learning.
modeling decisions.
Let (8), a1,...,. 8T,88T aT) be a trajectory of robotic experience of length T from an offline dataset D. For a given time-step I, and the corresponding action a, in the trajectory, we define per-dimension view of the action d,. Let at denote the vector of action dimensions from the first dimension aj until the i-th dimension aj, where can range from 1 to the total number of action dimensions, that we denote d Then, for time window of state history, define the Q-value of the action of in the i-th dimension using an autoregressive Q-function conditioned on states from this time window st-w:t and previous action dimensions for the current time step a 1:i-1 To train the Q-function, we define per-dimension Bellman update. For all dimensions i {1,.... d.):
max Q(s--wit: of i+1)
if E {1...., ......ddy da 1} R(se, ur) + 7max Q(s a if = d olt1
Q( az d: .a)) +
at
(1)

---

The reward is only applied on the last dimension (second line in the equation), as we do not receive any reward before executing the whole action. In addition, we only discount Q-values between the time steps and keep discounting at 1.0 for all but the last dimension within each time step, to ensure the same discounting as in the original MDP. Figure 2 illustrates this process, where each yellow box represents the Q-target computation with additional conservatism and Monte Carlo returns described in the next subsections. It should be noted that by treating each action dimension as a time step for the Bellman update, we do not change the general optimization properties of Q-learning algorithms and the principle of the Bellman optimality still holds for a given MDP as we maximize over an action dimension given the optimality of all action dimensions in the future. We show that this approach provides a theoretically consistent way to optimize the original MDP in Appendix A, with proof of convergence in the tabular setting in Appendix B.
4.2
Conservative Q-Learning with Transformers
Having defined Bellman backup for running Q-learning with Transformers, we now develop a tech- nique that enables learning from offline data, including human demonstrations and autonomously collected data. This typically requires addressing over-estimation due to the distributional shift, when the Q-function for the target value is queried at an action that differs from the one on which it was trained. Conservative Q-learning (CQL.) [29] minimizes the Q-function on quitt--distritution actions, which can result in Q-values that are significantly smaller than the minimal possible cumu- lative reward that can be attained any trajectory. When dealing with sparse rewards R E {0, 1). results in [27] show that the Q-function regularized with a standard conservative objective can take on negative values, even though instantaneous rewards are all non non-negative. This section presents a modified version of conservative Q-learning that addresses this issue in our problem setting.
The key insight behind our design is that, rather than minimizing the Q-values on actions not in the data, we can instead regularize these Q-values to be close to the minimal attainable possible cumulative reward. Concretely, denoting the minimal possible reward on the task as Rmins and the time horizon of the task as T, our approach regularizes the Q-values on actions not covered by the dataset towards Ruinn T, which in our problem setting is equal to (i.e., Rmin- 0). For simplicity of notation, we omit the action dimension indices in presenting the resulting objective, but remark that the training objective below is applied to Bellman backups on all action dimensions as described in the previous section. Let Ty be the behavioral policy that induced given dataset D, and let (als)) z(a (1.0 (a)ss) be the distribution over all actions which have very low density under Ts(aas). Our objective to train the Q-function is:
1 J
2 Eg-Daanxaall)
(" ta 2 RonDaawastala) [(Q(s.a) =) (2)
(3), crror
(1i), conservative regularization Co
where the first term (i) trains the Q-function by minimizing the temporal difference error objective
as defined in Eq. 1, and the second term (ii) regularizes the Q-values to the minimal possible Q- value of 0 in expectation under the distribution of actions induced by #g. which denote
conservative regularization term Lc- Term (ii) is also weighted by a multiplier a, which modulates the strength of this conservative regularization. We discuss the choice of a in our implementation in Appendix D.2 and analyze the behavior of the conservatism term in Appendix C, providing simple characterization of how this regularizer modifies the learned Q-function in tabular settings.
4.3 Improving Learning Efficiency with Monte Carlo and n-step Returns
When the dataset contains some good trajectories (c.g., demonstrations) and some suboptimal trajec- tories (e.g., autonomously collected trials), utilizing Monte Carlo return-to-go estimates to accelerate Q-learning can lead to significant performance improvements, as the Monte Carlo estimates along the better trajectories lead to much faster value propagation. This has also been observed in prior work [31]. Based on this observation, we propose a simple improvement to Q-Transformer that we found to be quite effective in practice. The Monte Carlo return is defined by the cumulative reward within the offline trajectory т: MC
" a,). This matches the Q-value of the be- havior policy πF. and since the optimal Q*(xx a) is larger than the Q-value for any other policy, we have Q" (st, ax) 2 MC..T. Since the Monte Carlo return is a lower bound of the optimal Q-function, we can augment the Bellman update to take the maximum between the MC-return and the current Q-lluee max (MCrr, Q(s,, ), without changing what the Bellman update will converge to.
5

---

Task category
of tasks
Q-T
DT
IQL
RT-1
drawer pick and place open and close drawer move object near target
18
64%
49%
11 %
17%
7
33%
11%
11%
0%
47
71%
40%
60%
58%
Average success rate
56%
33%
27%
25%
Right: Real world performance comparison. RT-1 [1] is imitation learning demonstrations. Q-Transformer (Q-T). Decision Transformer (DT) [32], Implicit Qllearnin (IQL) [40] learn from both demonstrations and autonomous data.
Figure 4: Left: Real world manipulation tasks.
Although this does not change convergence, including this maximization speeds up learning (see Section 5.3). We present hypothesis why this occurs. In practice, Q-values for final timesteps (r) are learned first and then propagated backwards in future gradient steps. It can take multiple gradients for the Q-value to propagate all the way to ($1, a1). The max Q) allows us to apply useful gradients to ( a1) at the start of training before the Q-values have propagated.
In our experiments, we also notice that additionally employing n-step returns [71, 72] over action dimensions can significantly help with the learning speed. We pick n such that the final Q-value of the last dimension of the next time step is used as the Q-target. This is because and reward only after inferring and executing the whole action as opposed to parts of it, meaning that intermediate rewards remain 0 all the way until the last action dimension. While this introduces bias to the Bellman backups, as is always the case with off-policy learning with n-step returns, we find in our ablation study in Section 5.3 that the detrimental effects of this bias are small, while the speedup in training is significant. This is consistent with previously reported results [72]. More details about our Transformer sequence model architecture (depicted in Figure 3) conservative Q- learning implementation, and the robot system can be found in Appendix D.
get new state
5 Experiments
In our experiments, we aim to answer the following questions: (1) Can Q- Q-Tansformer learn from a combination of demonstrations and sub-optimal data? (2) How does Q-Transformer compare to other methods? (3) How important are the specific design choices in Q- Q-Tansfformr?? (4) Can Q-
Transformer be applied large-scale real world robotic manipulation problems?
5.1 Real-world language-conditionee manipulation evaluation
Training dataset. The offline data used in our experiments was collected with a fleet of 13 robots, and consists of a subset of the demonstration data described by Brohan et al. [1], combined with lower quality autonomously collected data. The demonstrations were collected via human teleoper- ation for over 700 distinet tasks, each with separate language description. We 100 demonstrations per task, for a total of about 38,000 demonstrations. All of these demonstrations succeed on their respective tasks and receive a reward of 1.0. The rest of the dataset was collected by running the robots autonomously, executing policies learned via behavioral cloning.
maximum of
To ensure fair comparison between Q- Q-Tansfforerr and imitation learning methods, discard all successful episodes in the autonomously collected data when we train our method, to ensure that by including the autonomous data the Q-Transformer does not get to observe more successful trials than the imitation learning baselines. This leaves us with about 20,000 additional autonomously collected failed episodes, each with reward of 0.0, for dataset size of about 58,000 episodes. The episodes are on average 35 time steps in length. Examples of the tasks are shown in Figure 4.
Performance evaluation. To evaluate how well Q- Q-Transformer real-world offline datasets while effectively incorporating autonomously collected failed episodes, we evaluate Q- Q-Transformer as "drawer pick and place". "open and close drawer", "move object near target", each consisting of 18, 7 and 48 unique tasks instructions respectively to specify different object combinations and drawers. As such, the average success rate in Table 4 is the average over 72 tasks.
perform when learning from
72 unique manipulation tasks, and a variety of different skills, such
Since each task in the training set only has a maximum of 100 demonstrations, we observe from Figure that an imitation learning algorithm like RT-1 [1], which also uses a similar Transformer architecture, struggles to obtain good performance when learning from only the limited pool of

---

successful robot demonstrations, Existing offline RL methods, such as IQL [40] and a Transformer- based method such Decision Transformer [32], can learn from both successful demonstrations and failed episodes, and show better performance compared to RT-1, though by relatively small margin. Q-Transformer has the highest success rate and outperforms both the behavior cloning baseline (RT-1) and offline RL baselines (Decision Transformer, IQI., exceeding the average performance of the best-performing prior method by about 70%. This demonstrates that Q-Transformer can effectively improve upon human demonstrations using autonomously collected sub-optimal data.
Appendix G also shows that Q-Transformer can be successfully applied in combination with a re- cently proposed language task planner [8] to perform both affordance estimation and robot action execution. Q- Transformer outperforms prior methods for planning and executing long manghorizon tasks.
5.2 Benchmarking in simulation
In this section, we evaluate Q- Q-Transformer on challenging simulated offline RL task that re- quire incorporating sub-optimal data to solve the task. In particular, use visual simu- lated picking task depicted in Figure 5, where have small amount of position controlled human demonstrations (~8% of the data). The demonstrations are replayed with noise gen- erate more trajectories (~992% of the data). Fig- ure 5 shows comparison several offline al- gorithms, such as QT-Opt with CQL [11, 29], IQL [40]. AW-Opt [73], and Decision Trans- former [32]. along with RT-1 using Behavioral Cloning [1] on demonstrations only. As we see, algorithms that can effectively perform TD- learning combine optimal and sub-optimal data (such as Q- Q-Transformer and QT-Optt) per- form better than others. BC with RT- is not able to take advantage of sub-optimal data. Decision Transformer is trained on both demonstrations and sub-optimal data, but is not able to leverage the noisy data for policy improvement and does not end up performing well as method. Although IQL and AW-Opt perform TD-Icamnngg the actor remains too close to the data and can not fully leverage the sub-optimal data. Q-Transformer is able to both bootstrap the policy from demonstrations and also quickly improve through propagating information with TD-learning. We also analyze the statistical significance of the results by training with multiple random seeds in Appendix F.
Inmeeoamee (oun)
Decaiie Trandformer
RI-1 BC
QT-OstOOL
604
1:0K
120%
190k
Training s0aps
Figure 5: Performance comparison on simulated picking task.
5.3 Ablations
We perform series of ablations of our method design choices in simulation, with results presented in Figure (left). First, we demonstrate that our choice of conservatism for Q-Transformer per- forms better than the standard CQL regularizer, which corresponds to a seftmax layer on top the Q-function outputs with a cross- cross-entropy loss between the dataset action and the output of this soft- max [29]. This regularizer plays similar role to the one we propose, decreasing the Q-values for out-of-distribution actions and staying closer to the behavior policy.
As in Figure (left), performance with softmax conservatism drops around the fraction of demonstration episodes (~8%). This suggests a collapse to the behavior policy as the conservatism penalty becomes too good constraining to the behavior policy distribution. Due to the nature of the softmax, pushing Q-values down for unobserved actions also pushes Q-values up for the observed actions, and we theorize this makes it difficult to keep Q-values low for sub-optimal in-distribution actions that fail to achieve high reward. Next, we show that using conservatism is important. When removing conservatism entirely, we observe that performance collapses. Actions that are rare in the dataset will have overestimated Q-values, since they are not trained by the offline Q-learning procedure. The resulting overestimated values will propagate and collapse the entire Q-function, as described in prior work [38]. Finally, we ablate the Monte-Carlo returns and again observe performance collapse. This demonstrates that adding information about sampled future returns significantly helps in bootstrapping the training of large architectures such as Transformers.
7

---

Inmrstomme with acttrae
(nun)
I-mnenateme whout coraartatem
s-step ablation
ne-teep
1-step
1-step
# of gradient steps Training duration (hours)
137480
582960
136920
32
163
40
pick object
94%
97%
92%
move object near target
88%
80%
67%
Large offline dataset Q-T DT
RT-1
Average success rate 88% 78% 82%
Training stapa
Figure 6: Left: Ablations: changing to softmax conservatism decreases performance. Removing MC returns or conservatism completely collapse performance. Top Right: The n-step return ver- sion of our method reaches similar performance to the standard version with 4 times fewer steps, indicating that the added bias from n-step returns is small compared to the gain in training speed. Using n-step return also leads to better performance tasks that have longer horizon, e.g. move object near target. Bottom Right: Success rates on real world task categories with a larger dataset.
We also ablate the choice of n-step returns from the Section 4.3 on real robots and observe that using n-step returns leads to significantly faster training speed as measured by the number of gradient steps and wall clock time compared to using 1-step returns, with a minimal loss in performance, as shown in Figure 6 (top right).
5.4 Massively scaling up Q-Transformer
The experiments in the previous section used large dataset that included successful demonstrations and failed autonomous trials, comparable in size to some of the largest prior experiments that utilized demonstration data [74, 15, 58]. We also carry out a preliminary experiment with a much larger dataset to investigate the performance of Q- Q-Transformerr as we scale up the dataset size.
This experiment includes all of the data collected with 13 robots and comprises of the demonstrations used by RT-1 [1] and successful autonomous episodes, corresponding about 115,000 successful trials, and an additional 185,000 failed autonomous episodes, for a total dataset size of about 300,000 trials. Model architecture and hyperparameters were kept exactly the same, as the computational cost of the experiment made further hyperparameter tuning prohibitive (in fact, we only train the models once). Note that with this number of successful demonstrations, even standard imitation learning with the RT-1 architecture already performs very well, attaining 82% success rate. However, as shown in Figure (bottom right), Q- O-Transformer was able to improve even on this very high number. This experiment demonstrates that Q-Transformer can continue to scale to extremely large dataset sizes, and continues to outperform both imitation learning with RT-1 and Decision Transformer.
Limitations and Discussion
In this paper, we introduced the Q-Transformer, an architecture for offline reinforcement learning with high-capacity Transformer models that is suitable for large-scale multi-task robotic RL. Our framework does have several limitations. First, we focus on sparse binary reward tasks correspond- ing to success or failure for each trial. While this setup is reasonable for a broad range of episodic robotic manipulation problems, it is not universal, and we expect that Q Q-Transformer could be ex- tended to more general settings as well in the future.
Second, the per-dimension action discretization scheme that we employ may become cumber- some in higher dimensions (e.g., controlling a humanoid robot), as the sequence length and inference time for our model increases with action dimensionality. Although n-step returns mitigate this to a degree, the length of the sequences still increases with action dimensionality. For such higher- dimensional action space, adaptive discretization methods might also be employed, for example by training a discrete autoencoder model and reducing representation dimensionality. Uniform action discretization can also pose problems for manipulation tasks that require a large range of motion granularities, e.g. both coarse and fine movements. In this case, adaptive discretization based on the distribution of actions could be used for representing both types of motions.
Finally, in this work we concentrated on the offline RI setting. However, extending Q- Q-Transformer to online finetuning is exciting direction for future work that would enable even more effective autonomous improvement of complex robotic policies.
8

---

References
[1] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022.
[2] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:2210.03042 2022.
[3] A. Gupta, L. Fan, S. Ganguli, and L. Fei-Fei. Metamorph: Learning universal controllers with transformers, arXiv preprint urXiv:2203. 17931, 2022.
[4] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actors A multi-task transformer for robotic manipulation. arXiv preprint arXiv:2209005551, 2022. 2022.
[5] N. M. M. Shafiullah, Z. J. Cui, A. Altanzaya, and L. Pinto. Behavior transformers: Cloning modes with one stone. arXiv preprini urXiv:2206. 1251, 2022.
[6] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06115. 2022.
[7] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In Proceedings of the 5th Conference on Robot Learning (CoRL), 2021.
[8] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, Herzog, et al. Do as can, not as say: Grounding language in robotic 2022.
affordances. arXiv preprint arXiv:
[9] D.-A. Huang, Y.W. Chao, C. Paxton, X. Deng, L. Fei-Fei, J. C. Niebles, A. Garg, and D. Fox. Motion reasoning for goal-based imitation learning. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4878-4884. IEEE, 2020.
[10] D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine. Gnm: A general navigation model to drive any robot. arXiv preprint 0rX0v:22/0.03330, 2022.
[11] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. Conference on Robot Learning, pages 651-673. PMLR, 2018.
[12] A. X. Lee, C. M. Devin, Y. Zhou, T. Lampe, K. Bousmalis, J. T. Springenberg, A. Byravan, A. Abdolmaleki, N. Gileadi, D. Khosid, et al. Beyond placce::: Tackling robotic stack- ing of diverse shapes. In 5th Annual Conference on Robot Learning, 2021.
[13] S. Cabi, S. G. Colmenarejo, A. Novikov, K. Konyushkova, S. Reed, R. Jeong, K. Zolna, Y. Ay- tar, D. Budden, M. Vecerik, et al. Scaling data-driven robotics with reward sketching and batch reinforcement learning. Xrviv preprint arXiv: 12200, 2019.
[14] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. MT-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv, 2021.
[15] A. Kumar, A. Singh, F. Ebert, Y. Yang, C. Finn, and S. Levine. Pre-training for robots: Offline enables learning new tasks from a handful of trials. arXiv preprint arXiv:2210.05178, 2022.
[16] L. P. Kaelbling. Learning to achieve goals. In R. Bajesy, editor, IJCAI, pages 1094-1099. Morgan Kaufmann, 1993. ISBN 1-55860-300-X
[17] Y. Chebotar, K. Hausman, Y. La, T. Xiao, D. Kalashnikov, J. Varley, A. Irpan, B. Eysenbach, R. Julian, C. Finn, al. Actionable models: Unsupervised offline reinforcement learning of 2021.
robotic skills. arXiv preprint

---

[18] T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal value function approximators. In Bach and D. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1312-1320, Lille, France, 07-09 Jul 2015. PMIR.
[19] M. Andrychowicz, D. Crow, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, Abbeel, and W. Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, pages 5048-55 (488-505,,40 2017.
[20] K. Fang, A. Toshev, L. Fei-Fei, and S. Savarese. Scene memory transformer for embodied agents long-horizon tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 538-547, 2019.
[21] Anderson, A. X. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, Kosecka, J. Malik, R. Mottaghi, M. Savva, and A. R. Zamir. On evaluation of embodied naviga- tion agents. CoRR, abs/1807.067555 2018. URL http: // /dblp......////.p///... uni-trier. de / /bb/ journals/corr// cooor//oorr88777 html
[22] P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. D. Reid, S. Gould, and A. van den Hengel. Vision-and seioo-and-langee aad--lnguage navigation: Interpreting visually-grounde navigation instructions in real environments. CoRR, abs/1711.07280, 2017. URL http: //a //arxiv. wtwpg//////.../////// 07280.
[23] P. Mirowski, R. Pascanu, F. Viola, H. Soyer, A. J. Ballard, A. Banino, M. Denil, R. Goroshin, L. Sifre, K. Kavukcuoglu, D. Kumaran, and R. Hadsell. Learning to navigate in complex environments. CoRR, abs/1611 2016. URL http: /ttp://dbllppu/ uni-trier de/db/ Kournals/cort/coree 611, html www.owwiiP/SBBBC.....
[24] Y. Zhu, R. Mottaghi, Kolve, J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In ICRA, pages 3357- 3364. IEEE, 2017. ISBN 978-1 1778-1-5000 150900-463--1 URL http://dblp uni-trier. con wtn./iccaa/ccra2000
html #2huMKLGFF17.
[25] Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference robot learning, pages 1094-1100. PMLR, 2020.
[26] Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[27] A. Kumar, A. Singh, Tian, C. Finn, and Levine. workflow for offline model-free robotic reinforcement learning, arXiv preprint arXiv:2109, 10813, 2021.
[28] R. S. Sutton and A. G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998. ISBN 0262193981.
[29] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforce- ment learning, Advances in Neural Information Processing Systems, 17:111911 2020.
[30] Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 131.5.13,,.560,, 2013.
[31] A. Wilcox, A. Balakrishna, J. Dedieu, W. Benslimane, D. S. Brown, and K. Goldberg. Monte carlo augmented actor-critic for sparse reward deep reinforcement learning from suboptimal demonstrations. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neu- wttpeiiwwwweeeeeeeeeee.e.
ral Information Processing Systems, 2022. URL https: hwwp///ppeeee.e////o//e/p. net/ / forum? id=FLzTj 4118B...
[32] I. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 15:44484-1 2021.
10

---

[33] K.-H. Lec, O. Nachum, M. Yang, L. Lee, D. Freeman, W. Xu, S. Guadarrama, I. Fis- cher, Jang, H. Michalewski, et al. Multi-game decision transformers. arXiv preprint arXiv:2205. 15241, 2022.
[34] N. Jaques, A. Ghandeharioun, J. H. Shen, C. Ferguson, A. Lapedriza, N. Jones, S. Gu, and R. Picard. Way off policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv: 00000006, 2019.
[35] Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. arXiv 2019.
preprint arXiv:
[36] X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-weighted regression: Simple and scalable off off-policy reinforcement learning. arXiv preprint arXiv:1910.0000 2019.
[37] N. Y. Siegel, T. Springenherg. F. Berkenkamp, A. Abdolmaleki, M. Neunert, T. Lampe, R. Hafner, and M. Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint Stt::2002.08396, 2020.
[38] A. Kumar, Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off- q-learning via hootstrapping error reduction. In Advances in Neural Information Processing Systems, pages 11761-11771, 2019.
[39] I. Kostrikov, J. Tompson, R. Fergus, and O. Nachum. Offline reinforcement learning with fisher divergence critic regularization. arXiv preprint arXiv:2103.08050, 2021.
[40] Kostrikov, A. Nair, and Levine. Offline reinforcement learning with implicit q-learning. arXiv preprini 1 2021.
[41] Z. Wang, A. Novikov, K. Zołna, J. T. Springenberg. S. Reed, B. Shahriari, N. Siegel, J. Merel,
C. Gulcehre, N. Heess, et al. Critic regularized regression. arXiv preprint arXiv:2006. 2020.
[42] S. Fujimoto and S. S. Gu. A minimalist approach to offline reinforcement learning. arXiv preprint 2XXi:2106.06660 2021.
[43] X. Chen, Z. Zhou, Z. Wang, C. Wang, Y. Wu, and K. Ross. Bail: Best-action imitation learning for batch deep reinforcement learning, 2019. URL https://arxix org. abt///ab////////// C. 12179.
[44] H. Furuta, Y. Matsuo, and S. S. Gu. Generalized decision transformer for offline hindsight information matching. In International Conference on Learning Representations, 2022. URL putps://openreett
net/ forum?id=CAAxvodd.
[45] Y. Jang, J. Lee, and K.-E. Kim. GPT-critic: Offline reinforcement learning for end-to-end task- oriented dialogue systems. In International Conference on Learning Representations, 2022.
URL htw.::ttss/// /ttp:///////eeeieiieeee net. www.teeeooruu forum? d= UUaS.
[46] L. Meng, M. Wen, Y. Yang, chenyang le, X. yun Li, Zhang, Wen, W. Zhang, J. Wang, and B. XU. Offline pre-trained multi-agen decision transformer, 2022. URL https:// openreview n forum? rid=008 MLMMMer.
[47] Daoudi, M. Barlier, L. D. Santos, and A. Virmaux. Density estimation for conservative q-learning 2022. URL https:/ www./eveeeeeeeeeeeeeeeeeeeeeeeee net wwt.:eet///tte wewee//ooommddddii-eoeee f.
[48] L. Liu, Z. Tang, L. Li, and D. Luo. Robust imitation learning from corrupted demonstrations, 2022. URL https: opw.revieeeeeeeeeoooee net. forum? deunmmiriuEECC
[49] A. Mandlekar, D. Xu. J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, F.-. Li, S. Savarese, Y. Zhu, and R. Martin-Martin. What matters learning from offline human demonstrations for robot manipulation. In 5th Anmual Conference on Robot Learning, 2021. URL https: /pwprevveew....m.. ret/fforuu wwr.m/orumm///mm himmmidd==rrBBtr
[50] A. Kumar, J. Hong, A. Singh, and S. Levine. When should we prefer offline reinforcement learning over behavioral cloning? arXiv preprint arX0v:2204.05666 2022.

---

[51] A. Singh, A. Kumar. Q. Vuong, Y. Chebotar, and S. Levine. Offline rl with realistic datasets: Heteroskedasticity and support constraints. arXiv preprint arXiv:2211.01052 2022.
[52] Q. Vuong, A. Kumar, S. Levine, and Y. Chebotar. DASCO: Dual-generator adversarial sup- port constrained offline reinforcement learning. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https: / /pppnrevieeeeeeeeeeeeeeeeeeeeeee net/ (oorummid=jBB@QQ
9qA-
[53] Li, X. Zhan, H. Xu, X. Zhu, J. Liu, and Y.-Q. Zhang. Distance-sensitive offline reinforcement learning. arXiv preprint TrXiv:225..11022 2022.
[54] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, t., Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[55] Y. Zhang and J. Chai. Hierarchical task learning from language instructions with unified trans- formers and self-monitorin arXiv preprint arXiv:2106. 20..0306777777 03427, 2021.
[56] A. Pashevich, C. Schmid, and C. Sun. Episodic transformer for aaion-and-langueee naviga- tion. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15942-15952, 2021.
[57] A. Silva, N. Moorman, W. Silva, Z. Zaidi, N. Gopalan, and M. Gombolay. Lancon-learn Learning with language to enable generalization in multi-task manipulation. IEEE Robotics and Automation Letters, 1635:66351 2021.
(58) F. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Bc-7: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learn- ing, pages 991-1002. PMLR, 2021.
[59] S. Nair, E. Mitchell, K. Chen, S. Savarese, C. Finn, et al. Learning language-conditioned robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning. pages 1315-1315. PMLR, 2022.
[60] M. Janner, Q. Li, and S. Levine. Reinforcement learning as one big sequence modeling prob- lem. ICML 202/ Workshop on Unsupervised Reinforcement Learning, 2021.
[61] A. Kumar, X. B. Peng, and S. Levine. Reward-conditioned policies. arXiv preprint arXiv:1912.13465 2019.
[62] R. K. Srivastava, P. Shyam, F. Mutz, W. Jaskowski, and J. Schmidhuber. Training agents using upside-down reinforcement learning. arXiv preprint arXiv:1912.02287. 2019.
[63] D. Brandfonbrener, A. Bietti, J. Buckman, R. Laroche, and J. Bruna. When does return- conditioned supervised learning work for offline reinforcement learning? arXiv preprint TTT:2220600107 2022.
[64] T. Yamagata, A. Khalil, and R. RodriszRo Q-learning decision transformer: Leverag- ing dynamic programming for conditional sequence modelling in offline rl, 2023.
[65] I.. Metz, J. Ibarz, N. Jaitly, and J. Davidson. Discrete sequential prediction of continuous actions for deep rl CoRR, www/sa/sss../////// 005055, 2017. URL http::/tt////. //tp:p/dbbpppppp uni-trier. de/ db/ hbb/jounalss/aee corrlll///oorrroooo corr /tt///crllli/// 7055. .tmll #Met
[66] S. Hochreiter and J. Schmidhuber. Long Short-Term Memory. 988): 1997.
[67] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. Second edition, 2018.
[68] D. Cer, Y. Yang, S.y Kong, N. Hua, N. Limtiaco, R. S. John, N. Constant, M. Guajardo- Cespedes, S. Yuan, C. Tar, et al. Universal sentence encoder. arXiv preprint arXiv: //15, 2018.
[69] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
12

---

[70] M. Tan and Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114. PMLR, 2019.
[71] R. S. Sutton. Learning to predict by the methods of temporal differences. Machine learning, (1):9-44, 1988.
[72] M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thhrryy-Second AAAI Conference Artificial Intelligence, 2018.
[73] Y. Lu, K. Hausman, Y. Chebotar, M. Yan, E. Jang, Herzog, T. Xiao, A. Irpan, M. Khansari, D. Kalashnikov, and S. Levine. Aw-opt: Learning robotic skills with imitation andreinforce- ment at scale. 5th Annual Conference on Robot Learning, 2021.
[74] Dasari and Gupta. Transformers for one-shot visual imitation. Conference on Robot Learning, pages 2071-2084. PMLR, 2021.
[75] C. J. Watkins and P. Dayan. Q-learning. Machine Learning, 8:279-292. 2979-292,11 1992.
[76] T. Xiao, E. Jang, D. Kalashnikov, S. Levine, Ibarz, K. Hausman, and A. Herzog. Think- ing while moving: Deep reinforcement learning with concurrent control. arXiv preprint arXiv:2004.066 06089, 2020.
[77] A. Chowdhery, Narang, Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, Barnes, Tay, N. Shazeer, Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghemawat, Dev, Michalewski, X. Garcia, V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito, D. Luan, H. Lim, B. Zoph, A. Spiridonov, R. Sepassi, D. Dohan, S. Agrawal, M. Omernick,
A. M. Dai, T. Pillai, M. Pellat, Lewkowycz, E. Moreira, R. Child, O. Polozov, K. Lee, 7. Zhou, X. Wang, B. Saeta, M. Diaz, O. Firat, M. Catasta, J. Wei, K. Meier-Hellstern, D. Eck, Dean, Petrov, and Fiedel. Palm: Scaling language modeling with pathways, 2022.
URL https:/ / /arxiv. org/ abs 2204 02311.
13

---

A Proof of MDP optimization consistency
To show that transforming MDP into a per-action-dimensiie form still ensures optimization of the original MDP, we show that optimizing the Q-function for each action dimension equivalent to optimizing the Q-function for the full action.
If consider the full action 01:da and that switch to the state s' at the next timestep, the Q- function for optimizing over the full action MDP would be:
max a1:d)) = max R(s, a1dda) + max Q(s', andd)) R1
and
R[ss, didda max Q(s', alid))) aldd
(3)
where R(s, (idd)) is the reward we get after executing the full action.
The optimization over each action dimension using our Bellman update is:
max Q(s, -is-1, a.) max B* Q(1,Q(sssss) (1., 4:
84
max (ss, ais-i, ai (++1)
max
max
a,, max B Q ais,
a:l1
max (s,, ai 1. as a++1 0:+2) ax42
max
max
9.+1
R(s, (idd,) + 7 max Q a1)
R(s, y max B ( a1)
21
RR(s, (Gdds) max
max Q a1 0z)
R(ss, + max [B' (s' a1 02)
R(s, + max 01.02
max Q(s 01, a3)
0g
-(s, max Q a1dd))
ai.d
which optimizes the original full action MDP as in Eq. 3.
B Proof of convergence
Convergence of Q-learning has been shown in the past [67, 75]. Below we demonstrate that per-action dimension Q-function converges as well, by providing a proof almost identical to the standard Q-learning convergence proof, but extended account for the per-action dimension maximization.
Let d be the dimensionality of the action space, a indicates a possible sequence of actions, whose dimension is not necessarily equal the dimension of the action space. That is:
E {ai:1, Vi < dr}
To proof convergence, demonstrate that the Bellman operator applied the per-action di- mension Q-function is a contraction, i.e.:
B (2)2(s. a) I (s, Q2(s,
14

---

where
R(s,al a) + max a(s,a,a) (²Q(,,0)= R(s, +7 max E QQ(s', a the dimension of is equal to d,a
the dimension of is less than d
a'
a' is the next action dimension following the sequence a, 8 is the next state of the MDP, 7 is the discounting factor, and S
Proof: We can show that this is the case as follows:
Case I: For action sequence whose dimension is less than the dimension of the action space.
3" (*,)(,,))) - B" Q2(,,,))
R(s,a) a) + max (i,, R(s,a) - max Q2(s,
= max (Qi(s, Q2(s,a,
S susppQi -Q((s)))))
8,
= 1 (i(,,)) - Q2 a) S yl1Q1(s, 92(s,a)lle
where sup,,a the supremum over all action sequences, with S 7 S and || sup,,
Case 2: For action sequence whose dimension is equal to the dimension of the action space
B B"Qi(s, a) - B (2
R(s,a) + max FIQ R(s,a) max FlQ2(8', a'
a' 2
max F. FIQn (2(''
S ssup(Qi(s.)) Q2(s.a)
= Q1 - (2 S (),Qi(s,a) - Q2(8,
C Analysis of the conservatism term
With the goal of understanding the behavior of our training procedure, we theoretically analyze the solution obtained by Eq. 2 for the simpler cases when is represented table, and when the objective in Eq. can be minimized exactly. We derive the minimizer of the objective in Eq. by differentiating J with respect to Q:
dJ
Vs dQ(s,a) =0
To(a|s) (Q(s.a) B 1* + aas(s))))
((s,a) (To(als) + ang(als)) (
Te(a|s) B (*.aa) (s,
@ Ta(a|s) + (als)
:=m(s.)
Eq. implies that training with the objective in Eq. performs a weighted Bellman backup: un- like the standard Bellman backup, training with Eq. multiplies large Q-value targets by a weight m(s,a) a). This weight m(s, a) takes values between and 1, with larger values close to for in- distribution actions where (s, DD and very small values close to 0 for out-of-distribution actions a at any state (i.e,, actions where (a)() small). Thus, the Bellman backup induced via Eq. should effectively prevent over-estimation of Q-values for unseen actions.
15

---

D Q-Transformer Architecture & System
In this section, we describe the architecture of Q- Q-ranfformer as well as the important implementa- tion and system details that make it an effective Q-learning algorithm for real robots.
D.1 Transformer sequence model architecture
Our neural network architecture is shown in Figure 3. The architecture is derived from the RT-1 design [1], adapted accommodate the Q-Transformer framework, and consists of Transformer backbone that reads in images via a convolutional encoder followed by tokenization. Since we apply Q- Q-Transformer to a multi-task robotic manipulation problem where each task is specified by a natural language instruction, we first embed the natural language instruction into an embedding vector via the Universal Sentence Encoder [68]. The embedding vector and images from the robot camera are then converted into a sequence of input tokens via a FiLM EfficientNet [69, 70]. In the standard RT-1 architecture [1], the robot action space is discretized and the Transformer sequence model outputs the logits for the discrete action bins per dimension and per time step. In this work, we extend the network architecture to use Q-learning by applying a sigmoid activation to the output values for each action, and interpreting the resulting output after the sigmoid as Q-values. This representation particularly suitable for tasks with sparse per-episode rewards R E [0, 1]. since the Q-values may be interpreted as probabilities of task success and should always lie in the range [0, 1J. Note that unlike the standard softmax, this interpretation of Q- Q-values does not prescribe normalizing across actions (i.e., each action output can take on any value in [0, 1)..
Since our robotic system, described in Section D.3, has 8-dimensional actions, we end up with dimensions per time step and discretize each one into N = 256 value bins. Our reward function is a sparse reward that assigns value 1.0 at the last step of an episode if the episode is successful and 0.0 otherwise. We use discount rate = 0.98. As is common in deep RL, we use a target network to estimate target Q-values Q*, using an exponential moving average of Q-network weights to update the target network. The averaging constant is set to 0.01.
D.2 Conservative Q-learning implementation
The conservatism penalty in Section 4.2 requires estimating expectations under TT(aas) and (a|s)) x (1 (als), with the latter being especially non-trivial to estimate. We employ simple and crude approximation that we found work well in practice, replacing (a(a))) with the empir- ical distribution corresponding, for each sampled state-action tuple (sj, a,) E D. to a Dirac delta centered on aj. such that T (a(s)) 5(a az). This results in simple expression for +g(a|ss) corresponding to the uniform distribution over all other actions, such that (a(a)))) x 8(a + az). After discretizing the actions, there N 1 bins per dimension to exhaustively iterate over when computing the conservatism term in Eq. 2, which is the same as taking the average over targets for all unseen action values. In experiments, we find that simply setting the conservatism weight to = 1.0 worked best, without additional tuning.
D.3 Robot system overview
The robot that we use in this work is mobile manipulator with a 7-DOF arm with a 2 jaw parallel gripper, attached mobile base with a head-mounted RGB camera, illustrated in Figure 1. The RGB camera provides a 640 x 512 RGB image, which is downsampled to 320 x 256 before being consumed by the Q-Transformer. See Figure for images from the robot camera view. The learned policy is set to control the arm and the gripper of the robot. Our action space consists of 8 dimensions: 3D position, 3D orientation, gripper closure command, and an additional dimension indicating whether the episode should terminate, which the policy must trigger to receive positive reward upon successful task completion. Position and orientation are relative to the current pose, while the gripper command is the absolute closedness fraction, ranging from fully open to fully closed. Orientation is represented via axis-angles, and all actions except whether to terminate are continuous actions discretized over their full action range in 256 bins. The termination action is binary, but we pad it to be the same size as the other action dimensions to avoid any issues with unequal weights. The policy operates at 3 Hz, with actions executed asynchronously [76].
16

---

Algorithm Temporal difference error and loss computation for one action dimension i at timestep t, d..
Input Sequence of state in time window of size w, St-wet-
Input Language embedding of task instruction
Input The state at timestep /. + 1, 8
Input Dataset action up to dimension i, {Da) } j==
Output The loss to optimize Q-Transformer.
Q Compute maximum Q-values of the next action dimension using Eq. 1
// Compute the maximum between Q-target and Monte Carlo return. Qtarg + max Qlarg)
// Compute the temporal difference error.
TDError 2 (O-Tranffommrr(1 st st-wa, (a))
// Compute the conservative regularizer.
// The sum is over all action bins not equal to the tokeni zed dataset action.
// N is the number of discretizatio bin.
Reg 2(N 1) a#D!;
Σ (Q-Transformer((, &t-wth u
Compute the loss function
L = TDError+ Reg
Return L as the loss function to optimize Q-Transformer with.
E Pseudo-code
Algorithm shows the loss computation for training each action dimension of Q-Transformer. We first use Eq. to compute the maximum Q-values compute the Q-target for the given dataset action by using the Bellman update with an additional maximization over the Monte-Carlo return and predicted maximum Q-value at the next time step. The TD-error then computed using the Mean-Squared Error. Finally, we set a target of for all discretized action bins except the dataset action and add the averaged Mean-Squared Error over these dimensions to the TD-Error, which results in the total loss L.
the next action dimensions. Then
17

---

Running training for multiple random seeds
RT-1
0.8
0.6
Success Rate
20000
40000
60000
80000
100000
Step
Figure 7: Mean and variance of Q-Transformer and RT-1 performance in simulation when running the training for different random seeds.
In addition to performing large amount of evaluations, also analyze the statistical significance of our learning results by running our training of O-Transformer and RT-1 on multiple seeds in simulation. In particular, run the training for 5 random seeds in Figure 7. As see, Q- Transformer retains its improved performance across the distribution of the random seeds.
G Q-Transformer value function with language planner experiments
close drawer open top dra pick rxbar chocolate pick
QT-Opt sim-to-real
Q-Transformer w relabel
pick green can
pick rxbar Bueberry
pick
pick
can
pick orange can
pick green rice chip bag green jalapeno
pick coke can pick brown chip bag pick blue blue pick applé pick 7up can
0.6
Q-value
close top drawer open top chorolate
sim-to-roal O-Transformer w/ relabel
pick water bottle
pick sponge
pick green blueberry pick radbull can
pick popsi con
pick orange can
pick green jalapene chip bag
pick coke can pick chip plastic bottle
pick blue chip bag pick apple pick 7up
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Qvalue
Figure 8: Qualitative comparisons of Q-values from QT-Opt (sim-to-real) and O-Transformer. Q- Transformer outputs sharper Q- Q-values for objects close to the robot, which can be grasped faster and more easily than the objects farther away.
Recently, the SayCan algorithm [8] was proposed as way to combine large language models (LLMs) with learned policies and value functions to solve long-horizon tasks. this framework, the value function for each available skill is used to determine the "affordance" of the current state
18

---

for that skill, and a large language model then selects from among the available affordances to take a step towards performing some temporally extended task. For example, the robot is commanded to bring all the items on table, the LLM might propose a variety of semantically meaningful items, and select from among them based on the item grasping skill that currently has a high value (cor- responding to items that the robot thinks it can grasp). SayCan uses QT-Optt in combination with sim-to-real transfer to train Q-functions for these affordances. In the following set of experiments, we demonstrate that the Q-Transformer outperforms QT-Opt for affordance estimation without us- ing any sim-to-real transfer, entirely using the real world dataset that we employ in the preceding experiments.
We first benchmark Q- Transformer on the problem of correctly estimating task affor- dances from the RT-1 dataset [1]. In addi- tion to the standard training on demonstra- tions and autonomous data, we introduce training with relabeling, which we found particularly useful for affordance estima- tion. During relabeling, we sample a ran- dom alternate task for a given episode. We relabel the task name of the episode to the newly sampled task, and set reward to 0.0. This ensures that the boundaries between tasks are more clearly learned during train- ing. Table 1 shows comparison of performance of model with and without relabeling as well as the sim-to-real QT-Opt model used in SayCan [8]. Both of our models outperform the QT-Opt model on Fl score, with the relabeled model outperforming it by large margin. This demonstrates that our Q-function can be effectively used for affordance estimation, even without training with sim-to-real transfer. Visualization of the Q-values produced by our Q-function can be found in Figure 8.
Model
Precision
Recall
F1
QT-Opt (sim-to-real)
0.61
0.68
0.64
Q-T w/ relabel
0.76
0.89
0.82
Q-T w/o relabel
0.58
0.93
0.71
Affordance estimation comparison: precision, recall and Fl score when using Q-values to determine if a task feasible. Q-Transformer (Q-T) with multi- task relabeling consistently produces better affordance estimates.
Table I:
We then use Q- Q-Trnsformer in a long hori- zon SayCan style evaluation, replacing both the sim-to-real QT-Opt model for af- fordance estimation, and the RT-1 policy
Method
Success Rate
Affordance
Execution Planning Execution
for low-level robotic control. During this evaluation, PaLM language model [77] is used to propose task candidates given a user query. Q-values are then used to pick the task candidate with the highest affor- dance score, which is then executed on the robot using the execution policy. The Q- Transformer used for affordance estima- tion is trained with relabeling. The Q- Transformer used for low-level control is trained without relabeling, since we found relabeling episodes at the task level did not improve ex- ecution performance. SayCan with Q-Transformer is better at both planning the sequence of tasks and executing those plans, illustrated in Table 2.
Q-T w/ relabel
Q-T
93
93
QT-Opt (sim-to-real)
RT-1
87
67
Table 2: Performance on SayCan style long-horizon tasks: SayCan queries Q(s, a) in planning pick language instruction, then runs a policy to execute the plan. Q-Transformer outperforms RT-1 with QT-Opt in both planning and execution.
H Real robotic manipulation tasks used in our evaluation
We include the complete list of evaluation tasks in our real robot experiments below.
Drawer pick and place: pick 7up can from top drawer and place on counter, place 7up can into top drawer, pick brown chip bag from top drawer and place on counter, place brown chip bag into top drawer, pick orange can from top drawer and place on counter, place orange can into top drawer, pick coke can from middle drawer and place on counter, place coke can into middle drawer, pick orange from middle drawer and place on counter, place orange into middle drawer, pick green rice chip bag from middle drawer and place on counter, place green rice chip bag into middle drawer, pick blue plastic bottle from bottom drawer and place on counter, place blue plastic bottle into bottom drawer, pick water bottle from bottom drawer and place on counter, place water bottle into bottom drawer,
19

---

pick rxbar blueberry from bottom drawer and place on counter, place rxbar blueberry into bottom drawer.
Open and close drawer: open top drawer, close top drawer, open middle drawer, close middle drawer, open bottom drawer, close bottom drawer.
Move object near target: move 7up can near apple, move 7up can near blue chip bag, move apple near blue chip bag, move apple near 7up can, move blue chip bag near 7up can, move blue chip bag near apple, move blue plastic bottle near pepsi can, move blue plastic bottle near orange, move pepsi orange, move pepsi can blue plastic bottle, move orange blue plastic bottle, move orange near pepsi can, move redbull can near rxbar blueberry, move redbull can near water bottle, move rxbar blueberry near water bottle, move rxbar bluebenry near redbull can, move water bottle near redbull can, move water bottle near rxbar blueberry, move brown chip bag near coke can, move brown chip bag near green can, move coke bag, move green can near brown chip bag, move green can near coke can, move green jalapeno chip bag near green rice chip bag, move green jalapeno chip bag near orange can, move green rice chip bag near orange can, move green rice chip bag near green jalapeno chip bag, move orange can near green jalapeno chip bag, move orange can near green rice chip bag, move redbull can near sponge, move sponge near water bottle, move sponge near redbull can, move water bottle near sponge, move 7up can near blue blastic bottle, move 7up can near green can, move blue plastic bottle near green can, move blue plastic bottle near 7up can, move green can near 7up can, move green can near blue plastic bottle, move apple near brown chip bag, move apple near green jalapeno chip bag, move brown chip bag near green jalapeno chip bag. move brown chip bag near apple, move green jalapeno chip bag near apple, move green jalapeno chip bag near brown chip bag.
green can,
coke
brown chip
20


Reviews:
[{"review_id": "iyuuTibjJaY", "reviewer": "Reviewer_5C59", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors present \"Q-Transformer\" which is multi-task algorithm based on offline data. It leverages both human demonstrations and autonomous collected data. The authors argue that discretizing the action space naively leads to exponential blowup in the size of the action space. As a result, the authors propose to discretizing each action dimension separately and treat it in different time step. This enables the authors to use a a simple Q-learning method (unlike QT-Opt for example). In addition they use a regularizer that minimizes values of actions not appearing in the off-line data so they can handle offline RL distributional shift phenomenon. Lastly, they use Monte-Carlo N-step returns with TD that is shown to assist in perfrormance. The experiments are applied on 700 tasks where the training was done on 38K successful demonstrations and 20K failed autonomously collected episodes. These experiments show that the Q-transformer beats previous robotic-RL approaches.", "summary_of_recommendation": "I thank the authors for the work! I tend to accept the paper, but conditioned on that the authors will provide more details as requested in \"questions for rebuttal\". ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "(1) Do you have results regarding not splitting the actions by the axes? I would appreciate see how much gain can one have in this context.\n(2) Can you please show the technique performance on contact reach tasks such as insertion? Such tasks are more \"closed loop\" in high frequency tasks that I suspect the given technique cannot tackle.\n(3) What are the actual system performance in terms of:\n - Latency\n - Cycle time\n - Compute needed in inference time?\n(4) What is the SW stack used in the experiments? ROS? What is the motion planner?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In my opinion this work is interesting, in the right direction but with limited impact. On the up side, there are 3 good points to this paper. First, it based on two input modals which is a new standard in robotics: language and images. Second, the system architecture has some advantages in comparison to previous work. It is based on both Film EfficientNet and a Q-Transformer where using the output of the network, the chosen action is propagated back to the Q-Transformer input in a closed loop fashion. Third, we note that the technical challenge that the Q-Transformer is solving is manifested in the three principles implemented by the Q-Transformer, namely, the discretization, conservative Q-function, and the usage of the Monte-Carlo n-step returns for improving the efficiency. These principles are proven to solve the tasks in hand.\n\nOn the down side, I argue that the experiments that the authors provide are not really 700, but of much lower scale. Actually, as the authors themselves provide in the appendix, it is roughly can be classified into 3 types: drawer pick and place, open and close drawer, and move object near target. In addition, I argue that what the authors really want to say is that their system can generalize over this three types. I value these experiments and the variability of in the text prompts, but I argue that this is expected outcome in this case.\n\nQuality\nThe Quality of paper is satisfactory. First, in the chosen topic of how to use transformer in order to have a closed loop controller that can implement the architecture. Second, the considerations provided are logical. Specifically, I find the authors methodology (of disassembling the action space into its dimensions and training on each dimension separately) smart, specially when the experiments validate this although I'm not sure that for 6 degrees of continuous movement the alternative method of discretizing in these 6 dimensions would achieve less.\n\nClarity\nThe paper is written clearly although many real robot details are missing.\n\nOriginality\nThe work is build on foundations suggested in other domains such as the Monte-Carlo n-step backup, using the EfficientNet, Universal Sentence Encoder, adopting the Metz et al. scheme of discrete sequential prediction of continuous actions for deep RL.  On the other hand, as far as I know, some of the techniques are original as the architecture. \n\nStrengths\nTo summarize: the strengths of the paper are the architecture, using SOTA components, demonstrate the method in experiments, the usage of both vision and language for operating the robot, and finally, make all these mentioned things work together.\n \nWeaknesses\nThe experiments are limited and some design choices are not grounded enough. I must say that I know how hard is to make things work so I'm not position that as a must be, but rather a \"I wish it was explored more profoundly\".  ", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "AoziQjGWCR", "reviewer": "Reviewer_SvFm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents Q-transformers, a transformer-based representation of Q networks designed to improve the performance of high-capacity models on multi-task offline RL problems when learning from large datasets. The proposed method employs a number of adjustments to the typical RL optimization structure to make it compatible with transformer architectures and the need for discrete tokens. The presented results indicate that, for more modest training datasets, performance gains over previous transformer-based supervised learning techniques can be significant, and also provides evidence to suggest that these gains also translate to improved performance when working with large datasets.", "summary_of_recommendation": "I think, overall, this is a good paper  but that it is limited by its lack of statistical evaluation. ~~While I understand the computational limitation, with just a single run's worth of experimental data, I remain skeptical, particularly given that RL performance typically invites skepticism. I also maintain that I believe the paper would benefit from more direct contrasting with QDTs - ideally experimentally, but at least with some meaningful discussion. For now, I will mark this as a weak reject but am willing to raise my score if other reviewers are accepting of the limited scope of experimentation and do not raise other critical concerns.~~\n\nWith the added experimentation and clarifications, I believe the paper is sufficiently improved for me to raise my score to a 'weak accept'.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1. ~~How would you address the points raised in the 'Weaknesses' section in my review?~~\n\n2. ~~Could you clarify what $a^0$ is in eq 1? Is the a vector not $d_A$-dimensional? It appeared that it was 1-indexed and was defined for $i \\in [1, d_A]$ and yet a 0th index appears in eq. 1.~~\n\n3. ~~Based on the results in Fig. 6, it would appear that both the innovations proposed here - the monte-carlo lower bound and the approach to conservatism - are important. However, it is curious that the softmax conservatism works so poorly by comparison. Could you provide some more intuition for this? With just one test, I would require convincing that this is not an issue with implementation but rather a fundamental limitation of the baseline approach.~~\n\n4. ~~The table on the top right of Fig 6 has 2 columns labeled 1-step. What is the difference between them? I am not sure I am parsing this clearly.~~\n\n### Post-rebuttal update:\n\nAll questions sufficiently answered.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Strengths\n\n1. Overall, I found this paper to be generally well-written and well-organized and communicates its ideas clearly.\n2. Transformers have the potential to provide significant gains in RL, much as they have in other sequence modeling tasks, but this aspect of their potential is still relatively under-explored. The steps this paper takes to make transformer architectures compatible with more standard RL optimization pipelines may be an important step in enabling more such research.\n3. I appreciate that the paper is able to abstract away enough of the specifics of transformers to be able to have a meaningful discussion on how they may be integrated into RL, without getting bogged down in specifics, as I feel that that may have been distracting for this type of work.\n\n### Weaknesses\n\n1. ~~Given that we are dealing with RL problems, the fact that models are trained only once leads me to question the repeatability of the results.~~ How reliably can one expect performance gains - and what are the margins on performance differences with larger datasets (would, for example, the gap be narrowed between Q-T vs RT-1 with more statistical significance?). **UPDATE**: Given the added experimentation, I am less concerned about repeatability. The analysis is not as thorough as it could be, but is sufficient for me to not consider it a major limiting weakness.\n\n2. ~~As a choice of baseline, I find that the decision transformer (DT) makes for a poor choice relative to existing work - specifically, if one were to do a web search for \"Q learning transformer RL,\" one of the top hits is \"Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL\" by Yamagata et al. from 2022, which also addresses the limitations of DTs by augmenting training with RL. Given that the space of transformers applied directly to RL is relatively small, the omission seems particularly stark (especially given how, that paper was the top hit for searches on multiple search engines, is almost a year old, and seems highly relevant to the work in this paper). All that said,~~ I recognize the differences between the Q-transformer and QDT developed by Yamagata et al. and am not suggesting a lack of novelty in this work - merely that I feel an important point of discussion has been left out. **UPDATE**: I maintain that the paper would benefit from more discussion of existing work on QDTs (which the authors have committed to including), but I no longer regard it a weakness as strongly as I previously did.\n\n3. ~~I find that the way lines are plotted in Figs. 5 and 6 can be misleading. It would seem that the training curves are being smoothed for clarity, which is in itself fine, but I would propose sticking to either the raw curves or the smoothed curves (with indications that they have been smoothed) - instead of potentially leading readers to believe that the darker lines may represent more meaningful aggregated statistics (like from multiple runs).~~\n\n### Post-rebuttal notes:\n\nWith the additional experiments and clarifications, I believe the quality of the paper is improved. While there are still points that could be improved, I think the authors have done enough to cross the threshold into having an acceptable paper.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "AC9PdfIzyT", "reviewer": "Reviewer_VHvc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a method to enable the use of transformers for learning action-value functions in reinforcement learning. The proposed method, called Q-transformer, discretize each dimension of the action space and optimizes each dimension separately. The method additionally leverages conservative Q-learning for dealing with offline data. Experimental results are positive in challenging high-dimensional real-world manipulation tasks.", "summary_of_recommendation": "The paper is well written and the proposed method is potentially a significant contribution to enabling the use of transformers in reinforcement learning. However, the lack of soundness of the proposed method is a critical issue that forces me to provide a very negative overall score. As stated before, in case the authors will provide a valid formal proof to support their claims, I'd be willing to increase my score.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Can you formally prove the correctness of your maximization scheme over each dimension of the action space?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I have reviewed this paper already in the version submitted to RSS and this version seems substantially identical. As for the previous version, I am positive about the quality of writing, presentation, and potential significance of this work. Experimental results are also convincing. However, I am still concerned about the soundness of the action-value maximization. In my opinion, optimizing each dimension separately does not lead to a joint optimal action. In other words, given $a_1^*=argmax_{a_1}Q(s_{1:2},a_1)$ and $a_2^*=argmax_{a_2}Q(s_{1:2},a_1^*,a_2)$, I do not think that $(a_1^*,a_2^*)=argmax_{a_1,a_2}Q(s_{1:2},a_1,a_2)$ holds. The reason is that $a_1^*$ is found without considering the other dimension $a_2$, which could affect the optimality of action $a_1^*$. The paper states that\n\n> It should be noted that by treating each action dimension as a time step for\n168 the Bellman update, we do not change the general optimization properties of Q-learning algorithms\n169 and the principle of the Bellman optimality still holds for a given MDP as we maximize over an\n170 action dimension given the optimality of all action dimensions in the future.\n\nhowever, this claim should be supported by a rigorous proof. Even if I consider the paper of good quality, I have to express a very negative overall evaluation due to this lack of mathematical soundness. In case a valid proof will be provided, I'd be willing to increase my score.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "2ngMR3I7t8", "reviewer": "Reviewer_9Xry", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors present a transformer-based discretized Q-learning method for offline reinforcement learning.  Their model improves over several offline RL baselines by significant margins.  They present several practical modifications necessary to achieve these results.", "summary_of_recommendation": "The authors method is practical and interesting with possibly future applications in a wide array of robotic control problems.  The experiments show strong performance on challenging problems.  This passes an important test: \"would I want to read this paper when starting a project in this domain?\" The answer is yes.\n\nPost rebuttal: I liked this paper before and still like it.  Seems like some other authors' issues have been resolved.  Still strong accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "None at this time.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n+ The system performs well on real-world robotics problems\n+ The method has high practical utility\n+ The paper is written clearly and is easy to understand\n+ The paper contains many practical modifications to prior work that may aid further development.\n\nCons:\n- Figure 6 indicates that the system requires all of the modifications presented here in order to function, and that removing any of them dramatically hurts performance.  This may indicate some brittleness in the system and hyperparameters and is worth exploring further.\n- (MINOR) The experiments cover robotic pick-and-place tasks, which is definitely sufficient for this paper, however it would be worth exploring these ideas in other domains to determine how general these improvements are.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is one of the first approaches using large-scale transformers in combination with offline reinforcement learning to learn complex tasks from a set of demonstrations. The approach shows considerable improvements w.r.t. the presented baselines.

The rebuttal was extremely beneficial to this paper, the discussion between the authors and the reviewers clarify the soundness of the methods, solving the most problematic concerns of the reviewers. All reviewers now share a positive opinion of the paper, therefore I recommend accepting this work.

However, most reviewers, believe that this paper still has some shortcomings. The main concern is the reproducibility of the work, that in its current status appears to be quite limited. Also, during the discussion, it was evident that the method is mostly composed of minor incremental advances. While these components are important for deploying the method in complex environments and allowing effective learning of transformers-based policies, it is probably not enough for an oral presentation. 

Indeed, the current score is close, but not above, the 93% percentile, and no reviewer has argued in favor of an oral presentation, despite my requests to provide support arguments in favor of this decision. Therefore, I suggest a poster presentation for this work.
Decision: Accept (Poster)