A Solution to China Competitive Poker Using Deep Learning

1 INTRODUCTION . China Competitive Poker ( CCP ) is a poker game known as easy to learning but hard to master , requiring the mathematical and strategic thinking as well as carefully planned execution . CCP is played among three people with one pack of cards , including two jokers . After bid , one player would be the “ Landlord ( or Declarer ) ” , the other players are “ Peasant ( or Defenders ) ” , they are a team competing against the “ Landlord ” . Starting from the Landlord , players take turns to play sets/groups of cards . The objective of the game is to be the first player to play out all the cards . More details about CCP are described on Wikipedia ( 2018b ) . CCP is certified competitive sport by General Administrator of Sport of China . Dou dizhu has 15.4 million DAU ( Daily Active User ) in Tencent game platform , which is the largest online platform in China ( Baijiahao , 2018 ) . Compared to Go , CCP is a kind of imperfect information game , starts from a random state . A player gets different cards each game , does not know the cards distribution of the other two players . Like bridge , Peasants must coordinate with each other , otherwise they could hardly beat the Landlord . CCP is a good problem for game AI . We choose CNN to solve the problem in CCP due to the following reasons : First , CNN has achieved superhuman performance in perfect information games . Second , there is semi-translational invariance in CCP , e.g . there are two sets of cards in the same category but with different ranks ( like “ 34567 ” and “ 45678 ” , see more information in section 3 ) , if we add each card ’ s rank , “ 34567 ” become “ 45678 ” , this is translational invariance . The player can play out “ 45678 ” after another one played out “ 34567 ” , but it is illegal if we swap the order , this is the reason for “ semi ” . There is no AI about CCP using deep neural network so far . It is still to be proved that the network can output a proper action with an imperfect information input in CCP . The same factor between CCP and Go is blurred when making a decision . It is very hard to evaluate the value of alternative actions in Go , also CCP . As there are teammates in CCP , one problem is to teach the network to cooperate , e.g . how to help teammate become the first one to play out all cards , while his cards is not good enough . Another problem we are interested in is inference , whether the network is able to analyze like human experts ’ logic . The DeepRocket , which is developed by us , gets the state of the art performance in CCP . We prove that the network can learn cooperate and inference in imperfect information game . In section 2 , we introduce related work about CCP . In section 3 , introduce some notations in CCP , while is used through this paper . We explain each component in section 4 , including details about the policy network and the kicker network . We show how to prepare the experiment , the results of competing with the best previous AI , and top human amateur players in section 5 . Finally , the summary and the problems to be improved is shown in section 6 . 2 RELATED WORK . As so far , Libratus and DeepStack are the best AI programs in Texas Hold ’ em , which is also imperfect information game , developed by Carnegie Mellon University and University of Alberta respectively ( Brown & Sandholm , 2017 ; Moravčı́k et al. , 2017 ) . Only DeepStack use the deep neural network , and the problem is both of them can only be used in the one-to-one mode . University Computer Games Championship & National Computer Games Tournament is the largest AI competition in China . CCP is opened to not only universities but also the public . More than ten teams participate the tournament each year , all are based on logic code , rather than deep neural network , the champion can not beat the human amateur players until now . 3 NOTATIONS . This section introduces the notations in CCP , as well as the notations we use throughout this paper . The Specified Player : is one player we stand in his perspective . Down Player : is the player who does action after the Specified Player . Up Player : is the player who does action before the Specified Player . Landlord : is the player who bid the highest score . Down Peasant : is the player who does action after Landlord . Up Peasant : is the player who does action before Landlord . Public Cards : Each player get 17 cards . Landlord can get the last 3 cards , but have to show them to all players . These 3 cards are Public Cards . Round : All players have done actions once sequentially , started from the specified player . “ 33 ; 99 ; PASS ” is a round information , it represents “ 33 ” is played out by the Specified Player , next “ 99 ” is played out by Down Player , finally , Up Player choose “ PASS ” . Category : All legal types of cards in CCP , like “ 33344 ” is legal while “ 33345 ” is not . Main Group and Kicker Card : a solo or a pair is allowed to be played out with Trio . Trio is named as “ Main Group ” , the solo or the pair is named as “ Kicker Card ” . The rule is also used in Trio with Chain . For example , when “ 3334 ” or “ 33355 ” is played out , “ 333 ” is Main Group , “ 4 ” or “ 55 ” are Kicker Cards . Two solos or two pairs are played out with Four ( A kind of category in CCP ) . Set or Group : is one kind of categories , e.g . “ 334567 ” is divided into two sets , “ 3 ” and “ 34567 ” . Active Mode : A player can play any set available , if the following occurs : • Landlord first play in the game ; • Both the last two actions are “ PASS ” . Passive Mode : Except Active Mode , subsequent player must play a set of the same category but higher rank ( bomb and rocket are not included ) . Duplicate Mode : Analogous to Duplicate Bridge ( Wikipedia , 2018a ) ( also known as Tournament Bridge ) , it provides a more robust evaluation for players ’ performance . More details is provided in Appendix B . 4 DEEPROCKET FRAMEWORK . The system named DeepRocket consists of three parts : the Bid Module , the Policy Network and the Kicker Network . When game starts , the Bid Module is called in order to make DeepRocket output score , if necessary . The Policy Network is called before DeepRocket will do action , the output may contain Kicker Card type ( solo or pair ) in some labels . If there is a Kicker card need to be predicted , then the Kicker Network is called , as shown in Figure 1 and Figure 2 . 4.1 THE BID MODULE . First of all , the game need to make sure who is Landlord by bid after players get cards , so we design the module for bid . The key factors of bid is cards . The Bid module is based on logic code . Like human experts , the output of module depends on control cards ( like “ A ” , “ 2 ” , and “ jokers ” ) and tidiness ( that means less sets combined by cards ) . The module have been used on many online platform in business in China , and it is proved that works well . Bid is not used in Duplicate Mode , also in test , we do not introduce more details and focus on other components of DeepRocket . 4.2 THE POLICY NETWORK . Like AlphaGo ( Maddison et al. , 2015 ; Silver et al. , 2016 ) , we trained the Policy Network to predict human expert actions using supervised learning . The Policy Network consists of 10-layer CNN and 1 full connection layer , use Relu as activation function . A final softmax layer outputs probability distribution over all legal action a . The input represents current game state . The Policy Network is trained on randomly sampled state-action pairs ( s , a ) , using stochastic gradient ascent to maximize the likelihood of the human expert action . We use 8 million game records for training the Policy Network . One record represents a complete game , and can be divided into many state-action pairs , from several rounds to more than twenty rounds , depending on length of the records . Here is a example , see Appendix A for more details about the game record . Cards:4456777889JKKAA2B ; 335567899TTJJKAA2 ; 4456689TTJQQQK22S ; 33Q Game process:0,33 ; 1,55 ; 2,66 ; 0,77 ; 1 , AA ; 1,6 ; 2 , T ; 0 , J ; 1 , K ; 0,2 ; 2 , S ; 2,44 ; 0 , KK ; 2,22 ; 2,89TJQK ; 2 , QQ ; 0 , AA ; 0,56789 ; 1,789TJ ; 1,3 ; 2,5 If we want to learn Down Peasant , the record should be divided into the state-action pairs shown in Table 1 ( the order is shuffled ) . The input to the Policy Network is a 15×19×21 3-dimensional binary tensor ( we name the corresponding dimension X , Y , and Z respectively ) . Each dimension along X-axis represents the ranks of cards , from 3 to big joker . Y-axis represents the number of each rank ( from 1 to 4 ) , and features in CCP , such as solo , pair , trio , etc . Z-axis represents the sequential information of each rounds , this design learned from AlphaGo , make the variable length to fixed length in game . Finally , the most recent 6 rounds are chosen . Si , j , k standards the binary feature for current state of the game . Details are shown in table 2 . 512 filters are the most suitable after models are tested . 10-layers CNN achieved the best performance after repeated testing , different strides are used in each layer . The Policy Network outputs 309 action probabilities after we add the Kicker Network to DeepRocket , See table 3 . It is noteworthy that the maximum number of cards is 20 , it restricts the longest length of category . 4.3 THE KICKER NETWORK . A combination of the Bid Module and the Policy Network suffices to play a game in principle . However , there is a problem dealing with Kicker Cards . If there are n choice for the Main Group and m for the Kicker Cards , then there are n×m available actions in total . In some other case , if the player has trio with chain ( e.g . “ 333444 ” is a trio with chain ) , the length is i ( the length of ” 333444 ” is 2 ) , and j for available Kicker Cards , then there are Cij choice in total . For example , “ 333444555666789J ” is a legal action , but the sample is hardly found in even 8 million game records . Furthermore , Main Group with different Kicker Cards is labeled as different actions , and thereby a combinatorial number of different actions to be predicted by the Policy Network . In order to address this problem , we add an extra network for predicting the Kicker Card . The Policy Network predict the Main Group and Kicker Cards type , like solo or pair , then the Kicker Network predict the kicker cards . This make the Policy Network only output 309 probabilities rather than thousands . For example , “ 333444555666789J ” and “ 333444555666789Q ” are the same output in the Policy Network ( outputs “ 333444555666 ” and solo kicker cards ) while “ 789J ” or “ 789Q ” is predicted by the Kicker Network . The input to the Kicker Network , which include the remaining cards and the output of the Policy Network is a 15×9×3 3-dimensional binary tensor . Each dimension along X-axis represents the cards , is the same as the Policy Network . Table 4 shows the details about Y-axis and Z-axis . The Kicker Network has 28 outputs , include 15 kinds of solo and 13 kinds of pair . The Kicker Network consists of 5-layer CNN and 1 fully connection layer and outputs the probabilities of Kicker Cards . The Kicker Network outputs one type kicker each time . If the Policy Network predict “ 333444 ” and 2 solo Kicker Cards , the Kicker Network should be called twice .

Summary: This paper provides a system to play CCP using some deep learning. The system consists of  three modules - the bid module, which is rule based, and the policy and kicker networks, which are simple convolutional neural networks. The authors use a dataset of 8 million game records consisting of 80 million state action pairs, and train the network in a supervised fashion. The resulting model is able beat MicroWe, the current state of the art in playing CCP, and even are able to beat a few "top amateur players"