Title: Recursive Deep Inverse Reinforcement Learning
Primary area: reinforcement learning
Keywords: Inverse Reinforcement learning, imitation learning, Kalman filtering

Abstract: Inferring an adversary's goals from exhibited behavior is crucial for counterplanning and non-cooperative multi-agent systems in domains like cybersecurity, military, and strategy games. Deep IRL methods based on maximum entropy principles show promise in recovering adversaries' goals but are typically offline, require large batch sizes with gradient descent, and rely on first-order updates, limiting their applicability in real-time scenarios. We propose an online RDIRL approach to recover the cost function governing the adversary actions and goals. 
Specifically, we minimize a moment matching loss function using sequential second-order Newton updates, akin to the EKF, leading to a fast (in terms of convergence) learning algorithm. We demonstrate that RDIRL is able to recover cost and reward functions of expert agents in standard and adversarial benchmark tasks. Experiments on benchmark tasks show that our proposed approach outperforms several leading IRL algorithms.