# Techniques and Paradigms in Modern Game AI Systems
## 5 Paradigms and Trends
### Common Paradigms
#### 5.1.2 CFR Series  
DeepStack, Libratus, and Pluribus follow a unique paradigm to solve HUNL by CFR algorithms. The core of these systems is to simplify the original game by abstraction and use nested safe subgame solving for real-time planning. Specifically, DeepStack decomposes HUNL into stages and trains value networks at each stage, which is combined with depth-limit sparse look-ahead trees to re-solve subgame policies. In this way, DeepStack always calculates real-time responses to opponent off-tree actions but suffers from approximation error of the value networks. Libratus and Pluribus directly apply abstraction to the whole game and compute a tabular blueprint strategy and the corresponding counterfactual values. Subgame policies are later re-solved based on the value of blueprint strategies. Such abstraction can introduce error when rounding off-tree betting size to the course-grained blueprint strategy but is more robust when the opponent mistakenly chooses an action far from optimal, which could cause unprecedented ranges and value estimation in DeepStack.  
However, this paradigm is only a success limited to HUNL, and no further works successfully apply it to larger imperfect-information games. The reason lies in the compromise of computational cost and approximation error. Specifically, the computational cost under this paradigm comes from the CFR iterations both in the training and inference stages, and experiments show that a large number of CFR iterations are needed to produce a high-quality solution [5]. The blueprint strategies are in tabular forms, which are also limited by the available memory resources. Meanwhile, the process of subgame solving expands a search tree of limited depth and branching factor, so more abstraction is needed in larger games. Pluribus uses a very strong assumption that each player can only choose among four strategies to bring down the size of search trees, which introduces large errors in finding optimal policies. In fact, HUNL is relatively small in imperfect-information games considering its average size of information sets and the short time horizon. It can be concluded that the CFR-based paradigm is not scalable to larger games. It achieves superhuman performance on HUNL mainly because of the limited complexity of the game and suboptimality of human strategies.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**AI System** & **Prior Knowledge** & **Training Pipeline** & **Inference** & **RL Algorithm** \\ \hline AlphaGo & policy network & SL + RL & & & PG \\  & rollout policy & SL & & & \\  & value network & SL & & MCTS + NN & \\ \hline AlphaGo Zero & policy-value network & RL & & MCTS-RL \\ \hline DeepStack & DCFV network & SL & Continual re-solving + NN & & \\ \hline Libratus & blueprint strategy & Abstraction + MCCFR & Nested-safe subgame solving & & N/A \\ \hline Pluribus & blueprint strategy & RL & NN & Trinal-clip PPO \\ \hline AlphaHoldem & policy-value network & RL & NN & Trinal-clip PPO \\ \hline Suphx & \begin{tabular}{c} policy networks \\ global reward predictor \\ \end{tabular} & SL + RL & pMCPA finetune + NN & PG with entropy \\ \hline Juelong & policy-value network & RL & NN & ACH \\ \hline DouZero & value network & RL & One-step greedy + NN & DMC \\ \hline PerfectDou & policy-value network & RL & NN & PPO \\ \hline AlphaStar & policy-value network & SL+RL & NN & UFOO \\ \hline OpenAI Five & policy-value network & RL & \
\begin{tabular}{c} Minimax drafting \\ NN \\ \end{tabular} & PPO \\ \hline JueWu &
\begin{tabular}{c} policy-value network \\ drafting value network \\ \end{tabular} & RL+SL+RL & MCTS drafting & & \\ \hline \hline \end{tabular}
\end{table}
Table 2: The main components of AI system milestones.