## Algorithms for Game AI
Different from Decision Transformer (DT), which predicts next states as well as rewards and actions, ATT only predicts actions to maximize cumulative rewards, given partial trajectories containing states, actions, and rewards. Positional encoding typically used in NLP tasks is also devised to adapt to new input elements of RL. Experiments performed in several Mujoco environments show that ATT performs better than other offline RL algorithms such as DT. Contribution 7 [47] discusses the issue of Q-value overestimation in MADDPG, and proposes two variants to reduce estimation bias, called Multi-Agent Mutual Twin Delayed Deep Deterministic Policy Gradient (M2 ATD3) and Multi-Agent Softmax Twin Delayed Deep Deterministic Policy Gradient (MASTD3). Both methods introduce a second Q-network for each agent and incorporate either the minimum or maximum value of two Q-values to reduce bias in the opposite direction. MASTD3 further extends the Softmax Bellman operation to multi-agent settings by fixing others' actions and only changing its own actions. Experiments on two multi-agent environments, Particles and Tank, show some performance improvement and Q-value bias reduction compared to baseline methods such as MATD3. Contribution 8 [48] investigates the quality of PCG and specifically evaluates the expressive range of three level generators for Super Mario Bros (SMB) under different algorithms, including Genetic Algorithms (GA), Generative Adversarial Networks (GAN), and Markov Chains (MC). By defining nine metrics for each SMB level covering different types of characteristics, this paper visualizes the expressive ranges based on the metric distribution of generated levels and concludes that GA and MC have a much wider expressive range than GAN. The practice of expressive range analysis (ERA) presented in this paper can help game developers to recognize potential problems early in the PCG process to guarantee high-quality game content. Contribution 9 [49] explores the application of the MCTS algorithm in Amazons, and proposes MG-PEO, a combination of several optimization strategies called \"Move Groups\" and \"Parallel Evaluation\", to improve performance and efficiency. Move groups is a method used to divide a move into two parts, which can largely reduce the branching factor of the game tree and focus the visited nodes of MCTS more on promising subtrees. Parallel evaluation uses multiple threads to speed up the evaluation of leaf nodes. A technique called \"Shallow Rollout\" is also used to expand the small, fixed depth of nodes and apply the evaluation function to them. Ablation experiments show that MG-PEO agents achieve higher win rate than vanilla MCTS agents, indicating the effectiveness of the proposed optimization strategies. Contribution 10 [50] explores the application of vision transformer (ViT) to imitate the strategy of human Go players. While AlphaGo has achieved superhuman performance in Go, it often comes up with moves beyond human players' comprehension. This paper adopts supervised learning to train a policy network with vision transformer as its backbone architecture, which achieves professional-level play while mimicking the decision-making of human players. Experiments show that this ViT model achieves higher accuracy than models based on convolutional neural networks (CNNs). Contribution 11 [51] develops a vehicle driving game and explores the application of different reinforcement learning, including PPO and SAC [52], to train autonomous driving agents. By designing a series of tasks with rising complexity as a curriculum, the agent can gradually learn to navigate through multiple targets within lane boundaries without running into obstacles. Rewards are carefully designed in each task for effective and robust driving. Experiments show that though SAC agents learn faster in early phase, PPO agents achieve better performance and adaptability than SAC, shedding light on the applications of autonomous systems in real-world scenarios.