Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper presents Q-transformers, a transformer-based representation of Q networks designed to improve the performance of high-capacity models on multi-task offline RL problems when learning from large datasets. The proposed method employs a number of adjustments to the typical RL optimization structure to make it compatible with transformer architectures and the need for discrete tokens. The presented results indicate that, for more modest training datasets, performance gains over previous transformer-based supervised learning techniques can be significant, and also provides evidence to suggest that these gains also translate to improved performance when working with large datasets.
review: ### Strengths

1. Overall, I found this paper to be generally well-written and well-organized and communicates its ideas clearly.
2. Transformers have the potential to provide significant gains in RL, much as they have in other sequence modeling tasks, but this aspect of their potential is still relatively under-explored. The steps this paper takes to make transformer architectures compatible with more standard RL optimization pipelines may be an important step in enabling more such research.
3. I appreciate that the paper is able to abstract away enough of the specifics of transformers to be able to have a meaningful discussion on how they may be integrated into RL, without getting bogged down in specifics, as I feel that that may have been distracting for this type of work.

### Weaknesses

1. ~~Given that we are dealing with RL problems, the fact that models are trained only once leads me to question the repeatability of the results.~~ How reliably can one expect performance gains - and what are the margins on performance differences with larger datasets (would, for example, the gap be narrowed between Q-T vs RT-1 with more statistical significance?). **UPDATE**: Given the added experimentation, I am less concerned about repeatability. The analysis is not as thorough as it could be, but is sufficient for me to not consider it a major limiting weakness.

2. ~~As a choice of baseline, I find that the decision transformer (DT) makes for a poor choice relative to existing work - specifically, if one were to do a web search for "Q learning transformer RL," one of the top hits is "Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL" by Yamagata et al. from 2022, which also addresses the limitations of DTs by augmenting training with RL. Given that the space of transformers applied directly to RL is relatively small, the omission seems particularly stark (especially given how, that paper was the top hit for searches on multiple search engines, is almost a year old, and seems highly relevant to the work in this paper). All that said,~~ I recognize the differences between the Q-transformer and QDT developed by Yamagata et al. and am not suggesting a lack of novelty in this work - merely that I feel an important point of discussion has been left out. **UPDATE**: I maintain that the paper would benefit from more discussion of existing work on QDTs (which the authors have committed to including), but I no longer regard it a weakness as strongly as I previously did.

3. ~~I find that the way lines are plotted in Figs. 5 and 6 can be misleading. It would seem that the training curves are being smoothed for clarity, which is in itself fine, but I would propose sticking to either the raw curves or the smoothed curves (with indications that they have been smoothed) - instead of potentially leading readers to believe that the darker lines may represent more meaningful aggregated statistics (like from multiple runs).~~

### Post-rebuttal notes:

With the additional experiments and clarifications, I believe the quality of the paper is improved. While there are still points that could be improved, I think the authors have done enough to cross the threshold into having an acceptable paper.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I think, overall, this is a good paper  but that it is limited by its lack of statistical evaluation. ~~While I understand the computational limitation, with just a single run's worth of experimental data, I remain skeptical, particularly given that RL performance typically invites skepticism. I also maintain that I believe the paper would benefit from more direct contrasting with QDTs - ideally experimentally, but at least with some meaningful discussion. For now, I will mark this as a weak reject but am willing to raise my score if other reviewers are accepting of the limited scope of experimentation and do not raise other critical concerns.~~

With the added experimentation and clarifications, I believe the paper is sufficiently improved for me to raise my score to a 'weak accept'.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: The authors present a transformer-based discretized Q-learning method for offline reinforcement learning.  Their model improves over several offline RL baselines by significant margins.  They present several practical modifications necessary to achieve these results.
review: Pros:
+ The system performs well on real-world robotics problems
+ The method has high practical utility
+ The paper is written clearly and is easy to understand
+ The paper contains many practical modifications to prior work that may aid further development.

Cons:
- Figure 6 indicates that the system requires all of the modifications presented here in order to function, and that removing any of them dramatically hurts performance.  This may indicate some brittleness in the system and hyperparameters and is worth exploring further.
- (MINOR) The experiments cover robotic pick-and-place tasks, which is definitely sufficient for this paper, however it would be worth exploring these ideas in other domains to determine how general these improvements are.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The authors method is practical and interesting with possibly future applications in a wide array of robotic control problems.  The experiments show strong performance on challenging problems.  This passes an important test: "would I want to read this paper when starting a project in this domain?" The answer is yes.

Post rebuttal: I liked this paper before and still like it.  Seems like some other authors' issues have been resolved.  Still strong accept.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The authors present "Q-Transformer" which is multi-task algorithm based on offline data. It leverages both human demonstrations and autonomous collected data. The authors argue that discretizing the action space naively leads to exponential blowup in the size of the action space. As a result, the authors propose to discretizing each action dimension separately and treat it in different time step. This enables the authors to use a a simple Q-learning method (unlike QT-Opt for example). In addition they use a regularizer that minimizes values of actions not appearing in the off-line data so they can handle offline RL distributional shift phenomenon. Lastly, they use Monte-Carlo N-step returns with TD that is shown to assist in perfrormance. The experiments are applied on 700 tasks where the training was done on 38K successful demonstrations and 20K failed autonomously collected episodes. These experiments show that the Q-transformer beats previous robotic-RL approaches.
review: In my opinion this work is interesting, in the right direction but with limited impact. On the up side, there are 3 good points to this paper. First, it based on two input modals which is a new standard in robotics: language and images. Second, the system architecture has some advantages in comparison to previous work. It is based on both Film EfficientNet and a Q-Transformer where using the output of the network, the chosen action is propagated back to the Q-Transformer input in a closed loop fashion. Third, we note that the technical challenge that the Q-Transformer is solving is manifested in the three principles implemented by the Q-Transformer, namely, the discretization, conservative Q-function, and the usage of the Monte-Carlo n-step returns for improving the efficiency. These principles are proven to solve the tasks in hand.

On the down side, I argue that the experiments that the authors provide are not really 700, but of much lower scale. Actually, as the authors themselves provide in the appendix, it is roughly can be classified into 3 types: drawer pick and place, open and close drawer, and move object near target. In addition, I argue that what the authors really want to say is that their system can generalize over this three types. I value these experiments and the variability of in the text prompts, but I argue that this is expected outcome in this case.

Quality
The Quality of paper is satisfactory. First, in the chosen topic of how to use transformer in order to have a closed loop controller that can implement the architecture. Second, the considerations provided are logical. Specifically, I find the authors methodology (of disassembling the action space into its dimensions and training on each dimension separately) smart, specially when the experiments validate this although I'm not sure that for 6 degrees of continuous movement the alternative method of discretizing in these 6 dimensions would achieve less.

Clarity
The paper is written clearly although many real robot details are missing.

Originality
The work is build on foundations suggested in other domains such as the Monte-Carlo n-step backup, using the EfficientNet, Universal Sentence Encoder, adopting the Metz et al. scheme of discrete sequential prediction of continuous actions for deep RL.  On the other hand, as far as I know, some of the techniques are original as the architecture. 

Strengths
To summarize: the strengths of the paper are the architecture, using SOTA components, demonstrate the method in experiments, the usage of both vision and language for operating the robot, and finally, make all these mentioned things work together.
 
Weaknesses
The experiments are limited and some design choices are not grounded enough. I must say that I know how hard is to make things work so I'm not position that as a must be, but rather a "I wish it was explored more profoundly".  
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I thank the authors for the work! I tend to accept the paper, but conditioned on that the authors will provide more details as requested in "questions for rebuttal".
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper introduces a method to enable the use of transformers for learning action-value functions in reinforcement learning. The proposed method, called Q-transformer, discretize each dimension of the action space and optimizes each dimension separately. The method additionally leverages conservative Q-learning for dealing with offline data. Experimental results are positive in challenging high-dimensional real-world manipulation tasks.
review: I have reviewed this paper already in the version submitted to RSS and this version seems substantially identical. As for the previous version, I am positive about the quality of writing, presentation, and potential significance of this work. Experimental results are also convincing. However, I am still concerned about the soundness of the action-value maximization. In my opinion, optimizing each dimension separately does not lead to a joint optimal action. In other words, given $a_1^*=argmax_{a_1}Q(s_{1:2},a_1)$ and $a_2^*=argmax_{a_2}Q(s_{1:2},a_1^*,a_2)$, I do not think that $(a_1^*,a_2^*)=argmax_{a_1,a_2}Q(s_{1:2},a_1,a_2)$ holds. The reason is that $a_1^*$ is found without considering the other dimension $a_2$, which could affect the optimality of action $a_1^*$. The paper states that

> It should be noted that by treating each action dimension as a time step for
168 the Bellman update, we do not change the general optimization properties of Q-learning algorithms
169 and the principle of the Bellman optimality still holds for a given MDP as we maximize over an
170 action dimension given the optimality of all action dimensions in the future.

however, this claim should be supported by a rigorous proof. Even if I consider the paper of good quality, I have to express a very negative overall evaluation due to this lack of mathematical soundness. In case a valid proof will be provided, I'd be willing to increase my score.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is well written and the proposed method is potentially a significant contribution to enabling the use of transformers in reinforcement learning. However, the lack of soundness of the proposed method is a critical issue that forces me to provide a very negative overall score. As stated before, in case the authors will provide a valid formal proof to support their claims, I'd be willing to increase my score.
review_title: null

Meta-review: This paper is one of the first approaches using large-scale transformers in combination with offline reinforcement learning to learn complex tasks from a set of demonstrations. The approach shows considerable improvements w.r.t. the presented baselines.

The rebuttal was extremely beneficial to this paper, the discussion between the authors and the reviewers clarify the soundness of the methods, solving the most problematic concerns of the reviewers. All reviewers now share a positive opinion of the paper, therefore I recommend accepting this work.

However, most reviewers, believe that this paper still has some shortcomings. The main concern is the reproducibility of the work, that in its current status appears to be quite limited. Also, during the discussion, it was evident that the method is mostly composed of minor incremental advances. While these components are important for deploying the method in complex environments and allowing effective learning of transformers-based policies, it is probably not enough for an oral presentation. 

Indeed, the current score is close, but not above, the 93% percentile, and no reviewer has argued in favor of an oral presentation, despite my requests to provide support arguments in favor of this decision. Therefore, I suggest a poster presentation for this work.
Decision: Accept (Poster)