Title: TurboReBeL: 250$\times$ Accelerated Belief Learning for large Imperfect-Information Extensive-Form Games
Primary area: reinforcement learning
Keywords: Game Theory, Imperfect Information Games, Counterfactual Regret Minimization, Deep Reinforcement Learning, Depth-limited Solving

Abstract: Recursive Belief-based Learning (ReBeL) provides a general framework for large-scale Imperfect-Information Extensive-Form Games (IIEFGs) by integrating self-play reinforcement learning with search. However, ReBeL suffers from prohibitive computational costs during training: each Public Belief State (PBS) sample requires $T$ (typically $T=250$) iterations of Counterfactual Regret Minimization (CFR), and the PBS state space necessitates billions of samples for convergence. For example, training ReBeL on games such as Heads-Up No-Limit Texas Hold'em (HUNL) demands $4.5$ billion samples and $2$ million GPU hours. To address this, we propose \emph{TurboReBeL}, which achieves a $250\times$ acceleration in training through two key innovations: (i) Single-Sample Multi-Iteration Generation: This core innovation fixes subgame strategies to CFR-averaged policies, generating data for all $T$ iterations in one sampling pass and yielding a theoretical $O(T)$ speedup. (ii) Isomorphic Data Augmentation: This technique enhances sample diversity through game-theoretic invariants (suit and chip isomorphism) with minimal overhead and no performance loss. Evaluations show that TurboReBeL matches ReBeL's exploitability in Turn Endgame Hold'em using less than $0.4$% of the training cost, and achieves comparable performance on HUNL with $450\times$ fewer samples. TurboReBeL is the first depth-limited solving framework that combines ultra-fast training, low exploitability, strong convergence guarantees, human-data-free training, and fast real-time decision-making, distinguishing it from all existing techniques.

TL;DR: We introduce a novel general depth-limited solving framework for large IIEFGs, achieving unprecedented training efficiency without sacrificing the strengths