Title: Can Large Language Models Master Complex Card Games?
Venue: NeurIPS 2025
Accepted: yes
Keywords: Language Model

Evaluation
Abstract: Complex games have long been an important benchmark for testing the progress of artificial intelligence algorithms. AlphaGo, AlphaZero, and MuZero have defeated top human players in Go and Chess, garnering widespread societal attention towards artificial intelligence. Concurrently, large language models (LLMs) have exhibited remarkable capabilities across various tasks, raising the question of whether LLMs can achieve similar success in complex games. In this paper, we explore the potential of LLMs in mastering complex card games. We systematically assess the learning capabilities of LLMs across eight diverse card games, evaluating the impact of fine-tuning on high-quality gameplay data, and examining the models' ability to retain general capabilities while mastering these games. Our findings indicate that: (1) LLMs can approach the performance of strong game AIs through supervised fine-tuning on high-quality data, (2) LLMs can achieve a certain level of proficiency in multiple complex card games simultaneously, with performance augmentation for games with similar rules and conflicts for dissimilar ones, and (3) LLMs experience a decline in general capabilities when mastering complex games, but this decline can be mitigated by integrating a certain amount of general instruction data. The evaluation results demonstrate strong learning ability and versatility of LLMs. The code is available at 
https://github.com/THUDM/LLM4CardGame


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article is an empirical study that extensively explores the learning capabilities of Large Language Models (LLMs) in complex card games. The author systematically evaluates 8 different complexity levels of card games and studies the potential of LLMs to master complex games through supervised fine-tuning on high-quality game data. The experimental results indicate that: (1) LLMs can approximate the performance of strong game AI through supervised fine-tuning; (2) Compared to previous RL-Based AI, LLMs can simultaneously master multiple complex card games; (3) Mastering complex games leads to a decline in general capabilities, but this can be mitigated by integrating general instruction data.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. Comprehensive experimental design: The author selected 8 card games of varying complexities, providing a comprehensive evaluation of LLMs' game learning capabilities.\n\n2. Detailed ablation studies: The paper conducted various ablation experiments, including performance of commercial models, the impact of model size on performance in supervised fine-tuning, the influence of training data volume, and the interplay between tasks in multitask learning.\n\n3. Practical value: Validates that LLMs with a parameter scale of 8 billion, after supervised fine-tuning with data generated by dedicated reinforcement learning AI, can achieve high performance in various card games.\n\n4. Multi-dimensional analysis: Not only evaluated game performance but also analyzed the impact on general capabilities, as well as the knowledge transfer effects between different games.\n\n5. Training cost comparison: The paper clearly compared the training resources and time costs between RL-Based and LLM-Based approaches.\n\n**Weaknesses:**\n\n1. Comparison benchmark limited: Primarily compared with rule-based models, lacking direct performance comparisons with original RL-based strategies.\n\n2. Lack of reasoning model evaluation: Although understandable given time constraints at the project's initiation, there is a lack of evaluation of current mainstream reasoning models (such as O3, R1, etc.), which could be supplemented when time permits.\n\n3. Unexplored potential of reinforcement learning: For card games with clear rules and rewards, the direct possibility of applying reinforcement learning to LLMs has not been explored. This could be an important research direction and may not require additional experiments.\n\n4. Insufficient explanation for abnormal performance of the 14B model: The author attributes the performance decline of the 14B model to overfitting, but this explanation lacks persuasiveness and requires a more in-depth analysis.\n\n5. Why can't general knowledge be trained together with the card game tasks? Will the sequence of training first on card games and then on general tasks reduce the performance of the card game tasks?\n\n6. In the limitations section, there is no detailed explanation of the differences in reasoning time.\n\nQUESTIONS:\n1. Could the authors provide a direct comparison of win rates between LLM-based and RL-based strategies? Currently, the comparisons mainly involve rule-based models.\n\n2. Have the authors considered evaluating the performance of reasoning-based models (such as O3, R1) on these card tasks?\n\n3. Have you considered directly applying reinforcement learning to LLMs for card tasks? What are the potential advantages of this approach?\n\n4. Regarding the performance decline of the 14B model, could you provide a more detailed analysis? For example, how does it perform on multi-task data? Can it better handle different tasks compared to the 7B model?\n\nLIMITATIONS:\nSee Strengths and Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper evaluates the learning ability of Large Language Models (LLMs) by fine-tuning them on a hight-quality dataset spanning 8 diverse card games, and compare their performances against the state-of-the-art agents like AlphaGo, AlphaZero, and MuZero. It also further examines how the fine-tuning impacts the models' general abilities and explore the strategies to restore the general capabilities of LLMs.\n\nSTRENGTHS AND WEAKNESSES:\n#### **Strengths**\n1. The authors utilizes strong game AIs to build a large, high-quality dataset that spans diverse scenarios.\n2. The paper is well-written, analyses and discussions are comprehensive and easy to read.\n3. Provides insightful experiments correlating fine-tuning data with in-game performance.\n4. Explores how high-quality fine-tuning impacts both specialized gaming skills and broader general capabilities, and also the possibility of recovering the general capabilities of LLMs.\n#### **Weaknesses**\n1. Only open models are being fine-tuned, though closed models cannot be fine-tuned by LoRA, it can still be fine-tuned with the produced dataset to verify the quality of the data.\n2. The paper omits comparisons with leading reasoning models, leaving the performance gap between the fine-tuned card-game models and top-tier baselines unquantified.\n3. The highly uneven distribution of game samples in the merged 3.1 M point dataset (ranging from 50 K to 950 K per game) may prevent the model from converging on all tasks simultaneously.\n4. After mixed-dataset fine-tuning restores some general capabilities, the paper fails to report per-game performance, so it remains unclear whether game-specific skills were preserved or sacrificed in the process.\n\nQUESTIONS:\n1. The merged dataset spans from 50K to 950K samples per game. I wonder how sensitive is convergence to this imbalance? Have you tried balanced or curriculum sampling to ensure underrepresented games aren’t underfit?\n2. You show that mixed-dataset tuning recovers general capabilities, can you provide per-game win rates post-mix to confirm that individual game skills weren’t degraded?\n3. Why top reasoning LLMs are excluded? Including any of them could have clarified the true performance gap and dataset quality.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper evaluates LLMs across eight complex card games by finetuning them on high-quality gameplay data. The paper investigates three research questions: (i) can LLMs master complex card games? (ii) Can they master multiple games simultaneously? And (iii) do their general-purpose capabilities degrade if they are fine-tuned on gameplay data? The paper conducts an empirical investigation of these questions, and the results show that LLMs can master complex card games, but their general-purpose capabilities deteriorate as a result of finetuning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* The research questions are sensible and interesting and form a complete study.\n* The experiments are comprehensive and executed well.\n* The paper is well-written, easy to follow, and contains sufficient information to reproduce the results.\n\n**Weaknesses**\n\n* Some claims are misleading, incomplete, or incorrect\n  * The paper claims that the base models “lack the strategic knowledge required for games, resulting in poor initial performance”. However, the paper provides no evidence that base models do not contain such “strategic knowledge”; it just shows that the models perform poorly when playing. Instead, the poor performance can be attributed to the “knowing-doing gap”, which was documented in other game environments (Paglieri et al., 2025; Ruoss et al., 2025) and states that LLMs perform poorly *even though* they contain a lot of (strategic) knowledge about these games.\n  * The paper claims that LLMs “can master multiple complex card games simultaneously” and that training on multiple games can “mutually enhance each other’s performance”. However, the results in Table 3 actually show the opposite, since the models finetuned on a single game almost always perform better than the models trained on data from all games (except for DouDizhu), and sometimes quite substantially so, even though they were trained on roughly the same amount of data (“the amount of data required for each game to converge”). Consequently, if I have a FLOPS budget of X, I will obtain a better performance by training eight individual models, one for each game, with roughly X/8 FLOPS per model, than spending the entire budget on finetuning a single model across all games. Thus, since the individual models score quite a bit higher than the joint one, the LLMs cannot “master multiple games simultaneously”, and since there is no (or very little) transfer between games, it is better to train individual models than a general one.\n* The experiment showing that some of the general-purpose capabilities can be recovered is ill-defined. Firstly, it is not very surprising that the performance of a fixed set of evals can be improved by finetuning on relevant data. However, the effects of that training process on other general-purpose capabilities are not shown. Secondly, the paper does not show how the model performs on the card games after some of the general-purpose capabilities have been “recovered”.\n* The paper studies imitation learning on imperfect information games, which is susceptible to delusions (Ortega et al., 2021).\n* The paper should discuss relevant related work on evaluating LLMs in game environments (Paglieri et al., 2025; Ruoss et al., 2025) as mentioned above.\n\nI am very happy to increase my score if these weaknesses and the questions below are reasonably addressed.\n\n\n**References**\n\nOrtega et al. Shaking the foundations: delusions in sequence models for interaction and control. 2021.\n\nPaglieri et al. BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games. ICLR 2025.\n\nRuoss et al. LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations. ICML 2025.\n\nQUESTIONS:\n* How do you ensure that the training and evaluation data are different?\n* Why do you attribute the poor performance of the 14B model in Figure 4 to overfitting and not some other problem? What does its performance on the training distribution look like?\n* Why do all the finetuned models obtain a score of 1.24 for Leduc in Table 2?\n* L121: Does a standard deck not have 52 cards?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper conducts an empirical study to evaluate the capability of LLMs to handle card games. By giving the instruction prompt and supervisely fine-tuning LLMs with downstream datasets using LoRA, the paper shows that LLMs can master diverse card games and are able to simultaneously master multiple games by mixing trajectories together. Finetuning under a card game will hurt LLMs' general capabilities, while such decline can be marginally mitigated by integrating general instruction data.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper implemented extensive empirical studies, covered diverse card games, and multiple models with various model parameter scales. Some reasonable observations and conclusions come from experiments. However, I have several concerns:\nThe paper focuses on card games, which are less complicated than certain chess games like Go, and primarily evaluates the existing methods instead of proposing a new model or training strategy, which leads to a relatively narrow scope with limited insights.\nThe fine-tuning process conducted in the paper is solely using LoRa with fixed hyperparameters, making the paper less comprehensive. More fine-tuning strategies should be discussed.\n\nQUESTIONS:\nDescriptions of trajectory generation and the fine-tuning process are unclear; more details and a workflow figure might be more helpful.\nIn Figure 3, the authors pointed out that the reason for a low performance under Framer’s role in the DouDizhu game is due to the low quality of training data. While the assumption is reasonable, can the authors include further experiments to verify?\nThe authors talk about the influence between different games, starting from line 290, but did not specify the model they used for the experiment. All models should be tested to verify the conclusion.\nThe authors claimed that if the rules of two games are more similar, there tends to be greater knowledge transfer between them. However, while a model finetuned over GuanDan improved the DouDizhu’s performance, a model finetuned over DouDizhu still achieves a low performance on GuanDan. Does this imply that the transfer is not mutual?\nThe meaning of bolded values in tables should be mentioned in the caption.\nA typo in line 162. Instead of Table 2, it should be Table 1. \n\nI am mainly questioning the novelty and insights of this work. Whether the questions asked by the paper are valuable and whether the conclusion is general enough to be future development guidance.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}