## Algorithms for Game AI
For example, REINFORCE [22] samples full episode trajectories with Monte Carlo methods to estimate returns as the loss function. However, such pure policy-based algorithms suffer from high variance, and actor-critic algorithms [23] have been proposed, which use actors to learn parameterized policies and critics to learn value functions, allowing the policy updates to consider the value estimates to reduce the variance. Actor-critic algorithms include DDPG [24], A3C [25], IMPALA [26], TRPO [27], and PPO [28], to name but a few. * Regret-based algorithms seek to find the Nash equilibrium in games with imperfect information. The basic idea is to choose actions to minimize the regret of not having chosen other actions in previous games. Counterfactual regret minimization (CFR) [29] traverses the game tree for multiple iterations to calculate the cumulative regrets of each state-action pair. Many variants of CFR, such as Discounted CFR [30] and MCCFR [31], have been proposed to improve efficiency by incorporating sampling, discounting, and reweighting techniques. For games with large state space, algorithms such as Deep CFR [32] and DREAM [33] adopt neural networks to approximate the regrets and policies. * Self-play algorithms are often used in competitive multi-agent environments. In fictitious play (FP) [34], each agent calculates its best response to the opponent's average policies, which has been proven to converge to the Nash equilibrium in theory. When applied to games with large state space, neural fictitious self-play (NFSP) [35] adopts neural networks as the policy model, using reinforcement learning to calculate best responses and supervised learning to learn average policies. Double oracle (DO) [36] starts from a small policy subset, where each agent iteratively calculates the Nash equilibrium under the current strategy set and adds it to the set. Policy-space response oracles (PSRO) [37] provide a unified view of them using a policy pool to train new policies to be added, which has become the common practice of multi-agent RL training. * Multi-agent RL (MARL) algorithms extend single-agent RL algorithms to multi-agent settings, usually following a training paradigm called centralized training decentralized execution (CTDE). CTDE jointly trains multiple agents in a centralized manner but keeps their independence in execution, which can provide a mechanism of communication to eliminate the problems of unstable dynamics in independent training. For example, value-based algorithms like Value Decomposition Network (VDN) [38] and QMIX [39] are variants of DQN in cooperative multi-agent settings that adopt centralized state-action value functions, using summation and mixing networks to combine individual Q-networks. Multi-agent DDPG (MADDPG) [40] is a policy-based algorithm that generalizes DDPG to multi-agent settings, and many variants of it have been proposed to improve its performance and efficiency. This Special Issue presents eleven papers covering a wide range of game AI topics, including the quantification of non-transitivity in chess, the expressiveness of level generators in Super Mario Bros, Mahjong as a new game AI benchmark, new MARL algorithmsto reduce Q-value bias, surveys of various AI algorithms in cyber defense, energy areas and games, the application of MCTS in Amazons, the application of deep reinforcement learning in autonomous vehicle driving, and the application of transformers in both offline RL and imitation learning. The list of contributions to this Special Issue is as follows:  
Contribution 1 [41] examines the issue of cyber defense from the perspective of decision-making and proposes a framework to study the problem characteristics of different cyber defense contexts. The framework identifies four stages in the life cycle of threats--pentest, design, response, and recovery--integrating both the attacker's and defender's perspectives.