Q-Transformer: Scalable Offline Reinforcement Learning via
  Autoregressive Q-Functions
Authors: name: Yevgen Chebotar; name: Quan Vuong; name: Alex Irpan; name: Karol Hausman; name: Fei Xia; name: Yao Lu; name: Aviral Kumar; name: Tianhe Yu; name: Alexander Herzog; name: Karl Pertsch; name: Keerthana Gopalakrishnan; name: Julian Ibarz; name: Ofir Nachum; name: Sumedh Sontakke; name: Grecia Salazar; name: Huong T Tran; name: Jodilyn Peralta; name: Clayton Tan; name: Deeksha Manjunath; name: Jaspiar Singht; name: Brianna Zitkovich; name: Tomas Jackson; name: Kanishka Rao; name: Chelsea Finn; name: Sergey Levine
Published: 2023-09-18 21:00:38

Abstract: In this work, we present a scalable reinforcement learning method for
training multi-task policies from large offline datasets that can leverage both
human demonstrations and autonomously collected data. Our method uses a
Transformer to provide a scalable representation for Q-functions trained via
offline temporal difference backups. We therefore refer to the method as
Q-Transformer. By discretizing each action dimension and representing the
Q-value of each action dimension as separate tokens, we can apply effective
high-capacity sequence modeling techniques for Q-learning. We present several
design decisions that enable good performance with offline RL training, and
show that Q-Transformer outperforms prior offline RL algorithms and imitation
learning techniques on a large diverse real-world robotic manipulation task
suite. The project's website and videos can be found at
https://q-transformer.github.io