Q-Transformer: Scalable Offline Reinforcement Learning via
  Autoregressive Q-Functions
Authors: name: Yevgen Chebotar, name: Quan Vuong, name: Alex Irpan, name: Karol Hausman, name: Fei Xia, name: Yao Lu, name: Aviral Kumar, name: Tianhe Yu, name: Alexander Herzog, name: Karl Pertsch, name: Keerthana Gopalakrishnan, name: Julian Ibarz, name: Ofir Nachum, name: Sumedh Sontakke, name: Grecia Salazar, name: Huong T Tran, name: Jodilyn Peralta, name: Clayton Tan, name: Deeksha Manjunath, name: Jaspiar Singht, name: Brianna Zitkovich, name: Tomas Jackson, name: Kanishka Rao, name: Chelsea Finn, name: Sergey Levine

In this work, we present a scalable reinforcement learning method for
training multi-task policies from large offline datasets that can leverage both
human demonstrations and autonomously collected data. Our method uses a
Transformer to provide a scalable representation for Q-functions trained via
offline temporal difference backups. We therefore refer to the method as
Q-Transformer. By discretizing each action dimension and representing the
Q-value of each action dimension as separate tokens, we can apply effective
high-capacity sequence modeling techniques for Q-learning. We present several
design decisions that enable good performance with offline RL training, and
show that Q-Transformer outperforms prior offline RL algorithms and imitation
learning techniques on a large diverse real-world robotic manipulation task
suite. The project's website and videos can be found at
https://q-transformer.github.io