Title: Learning to Incentivize on the Fly: Leader-Follower Games with Policy Recommendation
Primary area: reinforcement learning
Keywords: Leader-Follower Games, Policy Recommendation, Incentive Compatibility, Multi-agent Reinforcement Learning

Abstract: In dynamic and strategic interactions, ensuring incentive compatibility (IC) is essential for achieving stable and predictable outcomes. This paper explores online reinforcement learning (RL) under a novel IC constraint in the context of leader-follower Stackelberg games with policy recommendation. In these games, the leader announces a policy to commit to while recommending a policy for the follower to adopt. The leader's optimal strategy is captured by the Stackelberg equilibrium, where the leader's announced policy maximizes her rewards, and the recommended policy is incentive-compatible—serving as the follower's optimal response given the leader's commitment. We study the online learning problem from the leader's perspective, where she has no prior knowledge of the follower's reward function and hence must infer it solely through observed follower actions. To address this problem, we develop a theory for such games, and propose a provably efficient algorithm that minimizes regrets with respect to both the leader's rewards and violations of the IC constraint. The algorithm integrates the maximum likelihood estimation of the follower’s response model with optimistic planning over an estimated IC constraint. To the best of our knowledge, this is the first provably efficient RL algorithm for incentive-compatible decision-making, highlighting the potential of RL in addressing challenges in misaligned multi-agent systems.

TL;DR: This paper presents a provably efficient RL algorithm for leader-follower Stackelberg games under an incentive compatibility constraint, aiming to optimize the leader’s rewards while ensuring incentive compatibility in online learning.