# Policy-Based Reinforcement Learning Approach in Imperfect Information Card Game
## 1 Introduction  
Artificial intelligence (AI) is a common and powerful tool that allows people to produce effective and satisfactory solutions to complex problems. Many effective AI systems were invented based on the Reinforcement Learning (RL) approach. Examples often refer to Go, chess, and other board games. The MuZero [1] agent's results, published in early 2020 by the DeepMind team, show the robustness of RL-like models surpassing superhuman AI players. In the Stratego game, an agent also reached an expert level by learning a game strategy through self-play [2]. Video games are an exciting test-bed for RL, too. It is worth mentioning that Agent57 [3] can play all 57 Atari2600 games, outperforming not only the human but also the previously mentioned MuZero solution in this field of research. Card games, however, remain a relatively underexplored domain for AI research despite their potential to test complex decision-making strategies. They often bring challenging problems to solve due to the large states' space and complexity of available strategies or rulesets [4]. They are usually imperfect information games (GII), which makes the work difficult. Nevertheless, it boosts the desire to satisfy researchers' curiosity to find a robust solution. In our research, we focused on a game popular in Poland, the two-player version of the game Thousand, which brings some difficulties to overcome. Firstly, dealing with imperfect information over the played deals is needed; the player knows only his or her hand and the played tricks in the current game before--so he or she does not have all the information about the current gameplay. Secondly, the bidding problem is that the bidder can only assess his or her estimated bids at the end of the episode. Therefore, one can obtain only sparse rewards during this stage. The last but not least obstacle to overcome is adjusting the model to avoid invalid actions--they do not bring any information during the game. The player can only play the cards that are in their hand and conform to the game's rules. The analysed game is not widely discussed in terms of applying AI techniques to surpass human players. Currently, in the literature, there are only a few articles related to this card game [5; 6; 7]. All of them refer to the three-player version. In [5], the Perfect Information Monte Carlo Tree Search (MCTS) algorithm was applied to create an AI agent to play the German ancestor of Thousand Schnapsen, or the so-called 66. The three-player version of the game Thousand was considered in [6; 7]. The first paper compares the MCTS, Cheating Perfect Information MCTS, and Supervised Deep Learning Neural Networks trained by the set of played games obtained from MCTS rollouts. The Deep Counterfactual Regret Minimisation (DCFR) algorithm was applied in the latter one. However, these approaches are computationally impractical for the two-player game due to its significantly larger state space--potentially up to 1000 times larger in certain scenarios. As a result, it hinders the application of methods that require a traversal of the game tree, as mentioned earlier, MCTS and DCFR. Our research addresses these limitations by leveraging deep reinforcement learning techniques. The RL approach via self-play seems to be a potentially good way to avoid involving expert knowledge in the AI agent development process. The spectrum of possible RL algorithms is vast, as is in the case of trick-based card games. Solutions that support stochastic environments are considered the best. This paper's developed method relies on the customised REINFORCE [8] algorithm. We have chosen this algorithm because of its simplicity and adaptability. The suggested solution is prepared for the trick-taking stage of Thousand Schnapsen. It is the first attempt to apply deep reinforcement learning algorithms to this game. In this paper, we aimed to advance reinforcement learning techniques by introducing and evaluating the adjusted REINFORCE policy gradient algorithm.