BeTAIL: Behavior Transformer Adversarial Imitation Learning from Human
  Racing Gameplay
Authors: name: Catherine Weaver; name: Chen Tang; name: Ce Hao; name: Kenta Kawamoto; name: Masayoshi Tomizuka; name: Wei Zhan
Published: 2024-02-22 00:38:43

Abstract: Imitation learning learns a policy from demonstrations without requiring
hand-designed reward functions. In many robotic tasks, such as autonomous
racing, imitated policies must model complex environment dynamics and human
decision-making. Sequence modeling is highly effective in capturing intricate
patterns of motion sequences but struggles to adapt to new environments or
distribution shifts that are common in real-world robotics tasks. In contrast,
Adversarial Imitation Learning (AIL) can mitigate this effect, but struggles
with sample inefficiency and handling complex motion patterns. Thus, we propose
BeTAIL: Behavior Transformer Adversarial Imitation Learning, which combines a
Behavior Transformer (BeT) policy from human demonstrations with online AIL.
BeTAIL adds an AIL residual policy to the BeT policy to model the sequential
decision-making process of human experts and correct for out-of-distribution
states or shifts in environment dynamics. We test BeTAIL on three challenges
with expert-level demonstrations of real human gameplay in Gran Turismo Sport.
Our proposed residual BeTAIL reduces environment interactions and improves
racing performance and stability, even when the BeT is pretrained on different
tracks than downstream learning. Videos and code available at:
https://sites.google.com/berkeley.edu/BeTAIL/home.