# A Rule-Based AI Method for an Agent Playing Big Two
## 4 Experimental Results and Discussion  
We have performed three experiments to evaluate the performance of our rule-based AI agent. Big Two is a four-player card game; therefore, two rule-based AI agents played against two opponents and play log data were recorded in each of these experiments. The data presented in this study are available online in our repository (see Supplementary Materials). For the first and second experiments, two types of AI opponents were built: randomized AI and conventional AI, which have been implemented in Python as the rule-based agents. Note that we generated all possible moves of an agent (single, pair, and five-card) in each turn. Randomized AI randomly chooses a card combination from the possible moves to play. The first experiment was the baseline for verifying whether rule-based AI showed any intelligence in playing Big Two. Conventional AI played the game according to the systematic approach [7] and probability calculations [8] which some experts have tried when formalizing how to play Big Two. The main idea of this AI is to play the lowest possible card combination to maximize the chances of winning. We generated all possible card combinations (single, pair, and five-card) of the agent. The value of possible combinations can be determined based on the probability of opponents having a single, pair, and five-card of a certain rank. In each turn, the agent will select the lowest possible card combination. If the agent does not have a valid combination to play, then it will pass its turn. The second experiment is designed to evaluate whether rule-based AI outperforms the existing approaches. For the third experiment, our rule-based AI connected to our web-based Big Two [6] and played against human players. Figure 3 shows a screenshot of our web-based Big Two game (our web-based Big Two game is freely available to try using Google Chrome at [[http://web.bigtwo.gamelab.com.tw/](http://web.bigtwo.gamelab.com.tw/)]([http://web.bigtwo.gamelab.com.tw/](http://web.bigtwo.gamelab.com.tw/)), accessed on 28 December 2020). The green 19 in Figure 3 is the timer countdown. If the active player does not discard a valid combination at the specified time (20 s), then it will pass its turn. We selected 50 graduate students and assured that there was no cheating between human players. Every two random participants played against two rule-based AI agents in different game rooms. Before starting the experiment, each player played more than 20 games to become familiar with the game. Then, each player played 80 games against rule-based AI agents via our web-based Big Two. We stated the following hypotheses:  
**Hypothesis 1 (H1)**. _Rule-based AI significantly outperforms randomized AI._
**Hypothesis 2 (H2)**. _Rule-based AI significantly outperforms conventional AI._
**Hypothesis 3 (H3)**. _Rule-based AI significantly outperforms human players._  
We used a Wilcoxon signed-rank test [44; 45] to measure the effect size (\(r\)) and check whether each hypothesis was accepted and statistically significant using \(p\)-values. The hypothesis was accepted and statistically significant if the \(p\)-value < 0.01; otherwise, the hypothesis was rejected. Furthermore, we used Pearson's \(r\) to interpret the effect size [46] as small (0.10 < \(r\) < 0.30), medium (0.30 < \(r\) < 0.50), and large (\(r\) > 0.50). Figure 4 shows a histogram of rule-based AI's win-loss records achieved after playing 2000 games in each experiment against various opponents. The number above the blue bar indicates the number of wins, and the number above the orange bar indicates the number of losses. In the first experiment against randomized AI, the rule-based AI won 1792 games and lost 208 games. In the second experiment against conventional AI, the rule-based AI won 1460 games and lost 540 games. In the third experiment against human players, the rule-based AI won 1101 games and lost 899 games. Overall, rule-based AI played 6000 games, winning 4353 games and losing 1647 games.