Liar’s Dice by Self-Play. With Counterfactual Regret and Neural... | by Thomas Dybdahl Ahle | Towards Data Science

I’ve been meaning to learn about AIs for games like Poker or Liar’s Dice for a while. Recently while reading Deepmind’s article Player of Games, I thought I might be able to make something really simple that works well enough to be fun. “It shouldn’t take more than an afternoon,” I thought. Of course, I ended up spending much more time on it, but it did turn out to be fun and very simple.
The project allowed me to dive into the exciting concepts of Counterfactual Regret Minimization, Reinforcement Learning, serving PyTorch models in the browser and a few other fun topics, so there are a bunch of things to cover in this blog post.
I understand. It’s a game of deception. But your bet includes all the dice, not just your own. — Will Turner, Pirates of the Caribbean
Liar’s Dice is a game I personally liked to play back when we could still go to bars. It is simpler than Poker, but has many of the same concepts of betting, bluff and hidden information, making it a fun challenge for AI.
The game is also famous from Pirates of the Caribbean, but I will quickly sketch the rules if you don’t know the game.
Each player has a set of dice known only to them. Players take turns bidding on the minimum number of dice that the player believes are showing a given value.
Each turn, the player must bid a higher quantity of any particular face or the same quantity of a higher face. E.g. “2 ⚃” and “3 ⚂” are both valid bets after “3 ⚁”, but “2 ⚂” isn’t.
Alternatively, the player can challenge the previous bid. If the challenge is successful — e.g. the previous bid was a bluff — the challenger wins the round; otherwise, the player who made the bet wins.
Often ⚀’s are considered jokers and count for any face when summing up the dice in the end.
Example: Alice and Bob may have the hands ⚀ ⚁ and ⚂ ⚃. Alice bids “2 ⚁” (which she can support from her own hand alone), Bob says “2 ⚂” (anticipating that Alice has either a ⚂ or a ⚀ (joker)). Alice calls Bob, but Bob was right and wins.
It takes a bit of theory to explain the algorithm — even though it is ultimately very simple. If you want to jump straight to the neural network section or the evaluation of the model in terms of ELO rating and exploitability, feel free to skip the following three sections.
Why are imperfect-information games so hard? After all, we had Deep Blue, we had AlphaGo. Why can’t we take those techniques and apply them out of the box to a game like Poker? — Noam Brown NIPS2017 best paper talk.
Liar’s Dice is a finite game: There are only so many dice combinations you can roll, and so many sequences of bets that can be made. (We call the combination a state, technically it is often called an info-set) However, unlike games of perfect information like chess, there is no single perfect move (or action) we can compute for each state. Instead, as shown by John von Neumann and John Nash, we need a “Mixed Strategy”: For each state, we pick a move from a probability distribution over the set of actions.
Intuitively this is because having a fixed strategy gives away information about our private dice. If we always make a high bet when we roll a lousy roll, our opponent will learn this and start calling our bluff. If we only bluff half the time, and play safe the other half, it gets harder for our opponent to figure us out.
The Nash equilibrium can be computed using Linear Programming, e.g. the method of Daphne Koller and Nimrod Megiddo, which takes polynomial time in the number of states (or “information sets” as is the more common term.) I actually used precisely this method to solve Liar’s Dice years ago. This is possible because with 1 die each, the maximum bid that makes sense is 2 ⚅’s. That makes a total of 12 possible bids. Combined with the dice information, that’s 6*212 ~400,000 states. This is just within what’s possible with modern Linear Algebra solvers like Google or-tools.
Unfortunately, this method doesn’t scale to multiple dice. With 6 dice each, we get ~1026 states, which is comparable to the total legal positions in chess. And while chess could theoretically be solved by a simple depth-first search — taking constant space and linear time in the number of states — Liar’s Dice is much much harder. This is why AIs could play chess quite well in 1998, but only recently got to the same level in Poker.
Okay, time I actually start explaining how this AI actually works, and how machine learning comes into play.
In the past, a lot of people would model decisions by maximizing your payoff. You want to try and win the most, right? But what people found was that what you actually want to try and do is minimize your regret. — Bryan Pellegrino, Poker Pro and AI Researcher
The favoured algorithm for solving imperfect information games is called Counterfactual Regret Minimization. I added some relevant links at the end of this post, but in my casual version, it works like this:
Assign all the final states (where one player has called the other a liar) the value +1 if the starting player wins, and −1 if the second player wins.For all other states, we’ll maintain a value v(state) (between -1 and 1) that approximates the average value observed in previous games that passed through the given state.(In real CFR the values should be inversely weighted by the player’s probability of going into the state, but for simplicity, and since the weighting is already kinda messed up by using Adam and momentum for training the neural network, I just skipped this part.)Define the “counterfactual regret” r(state, i) = v(state+i)−v(state), that is how much worse (or better) we would have scored (in average) if we had made the action i every time we were in the given state.(Here state+i is the resulting state after taking action i.)Finally, define the strategy (probability of picking action i in a given state) by
Assign all the final states (where one player has called the other a liar) the value +1 if the starting player wins, and −1 if the second player wins.
For all other states, we’ll maintain a value v(state) (between -1 and 1) that approximates the average value observed in previous games that passed through the given state.(In real CFR the values should be inversely weighted by the player’s probability of going into the state, but for simplicity, and since the weighting is already kinda messed up by using Adam and momentum for training the neural network, I just skipped this part.)
Define the “counterfactual regret” r(state, i) = v(state+i)−v(state), that is how much worse (or better) we would have scored (in average) if we had made the action i every time we were in the given state.(Here state+i is the resulting state after taking action i.)
Finally, define the strategy (probability of picking action i in a given state) by
where r(state)+ = max(r(state), 0). This ensures there are no negative probabilities.
It might be tempting to always play the action with the highest expected value v(state+i). However, that would be a deterministic strategy and likely exploitable by our opponent. Instead, the above algorithm — known as Regret Matching — is guaranteed to give us the Nash Equilibrium as our estimates of the state values get more precise.
In real CounterFactual Regret Minimization (CFR for short), we repeatedly improve our v(state) estimates by running a depth-first search on the entire game tree and storing the average score seen so far.
This has two main problems:
It uses too much time.It uses too much space.
It uses too much time.
It uses too much space.
To solve the time problem, we replace the depth-first search with game samples (also known as Monte Carlo CFR). Basically, instead of updating every child state — including those following a probability 0 action — we just play a game and update that path up through the game tree with +1 or −1 depending on the game result. In other words, we are doing reinforcement learning.
To ensure we explore all options, we add some noise to the probabilities when playing. The noise also teaches the AI to take advantage of bad moves from its opponent, since it isn’t trained with an opponent always making perfect choices. Ideally we should reduce the amount of noise added over time during training — if we want to eventually get a Nash Equilibrium — but I didn’t do this.
To solve the memory problem, we don’t store the exact mean score for every possible state. We instead use a neural network that computes a good approximation to the average value, v(state). To mimic the CFR update rule, we use a learning rate of 1/T, where T is the current time step. This ensures v(state) learns the average value over the training games, and not just the most recent outcome. (I actually tried other learning rates like 1/sqrt(T) and 1/T2, and amazingly 1/T was the only one that converged to something near the Nash Equilibrium.)
The complete training code is available in github.com/thomasahle/liars-dice/blob/main/code/train.py, and the most critical part — the self-play code — is just 27 lines.
The code saves triples of (private state, public state, game result) to a list called replay_buffer, which is passed to PyTorch for training the neural network (invoked in game.sample_action). After a step of gradient descent, we play another batch of games with the improved policy and repeat.
The neural network is a function v(private, public) that returns a value between −1 and 1. There are lots of ways we might define this, so I just chose the most straightforward thing I could come up with, and it seems to work well enough.
The private input is a 6 × “number of dice” one-hot representation of the player’s hand. We can view it as a histogram of how many dice we have of each face. This encoding is taken from DouZero [2021].
As the maximum call we can make in a game with two players is 2 × “number of dice”,the public state is a 2 × 6 × (2 × num-dice) one-hot representation of the calls that have been made so far by each player.
Each one-hot input is embedded as a 500-dimensional vector and the two are multiplied together. Other ways of pooling, like concatenation, also work fine, but element-wise multiplication mimics compressed bilinear pooling, for which I have a personal bias.
After combining the private and public states, we run 4 fully connected layers, 100 units wide, with rectified linear units in each layer. Eventually we reduce down to a single scalar value, which we map to [−1,+1] using tanh (like AlphaZero did.)
The training loss is simply MSE: (v(private, public) − game_result)2. Again just copying AlphaZero.
Sometimes I beat it. Sometimes it beats me. Liar’s Dice is a game involving a lot of chance and it would take a lot of games to know for sure who’s the better player. So let’s play a lot of games!
During the first week playing against people online at dudo.ai, the AI won 5031 rounds and lost 4416. This corresponds to a 53.3% win-rate across dice numbers. Games of Liar’s Dice consist of multiple rounds with different number of dice. The AI won 846 complete games and lost 640 corresponding to a 56.9% win-rate.
We can convert these percentages into a rating difference using the classical ELO formula:
Plugging in the win-rates from above, the AI has an advantage of somewhere between 23 and 48 ELO points over the human opponents.
Another interesting test is to see how quickly the algorithm improves as it plays against itself. I set up a tournament where different versions of the model played 10,000 games against every other version. I then used a maximum likelihood estimate to compute the ELO for each player.
The models clearly improve faster in the beginning, but they all keep improving for a long time. It took about 24 hours to train the 5-dice model to 230,000 iterations. The training itself doesn’t take much time, but playing 23,000,000 games took a while, since I didn’t implement any parallelization.
An interesting point is that ELO is usually used for games involving little to no luck, like chess. We might add the element of chance to the ELO formula as follows:
If we use this formula to estimate ELO ratings for the tournament between the different versions, it turns out that the model that best explains the data has r ~ 0.20. This means Liar’s Dice is about 80% pure luck. 🤷
There is a more principled approach to computing exactly how close our AI gets to the Nash Equilibrium. This is called measuring the exploitability of the strategy.
The exploitability is the maximum amount anybody can win against the AI even knowing exactly what probabilities the AI is using and having access to infinite computing power.
Actual CFR is guaranteed to eventually produce an unexploitable strategy. We can’t expect our method to be quite that good since the neural network may not even be able to represent this perfect strategy. However, if we get close to 0, it means the AI is objectively pretty good and not just playing well against itself and humans online.
Computing exploitability is done by computing the optimal response strategy for our opponent. We tell our opponent our strategy (our probabilities for each state) which transform the game turns into a Markov decision process (MDP). The opponent can solve this (in linear time with a depth-first search), and the value of the MDP is the most any opponent playing any strategy could win against us.
While the exploitability can be computed in linear time in the number of states, that is still too expensive for games with 4 or more dice. However I managed to compute the 1v1 exploitability (above), as well as 1v2 and 2v1 (below). Coincidentally these are the same games I managed to compute the exact Nash Equilibrium of (using Linear Programming) 5 years ago. The results were as follows:
Apparently, my model is quite good at learning the best strategy for Player 2. Here the exploitability is not far from the optimum. On the other hand, in both games, the strategy for Player 1 isn’t as good. It seems the AI prefers to judge the humans call before making up its own mind.
Does this mean it is just much harder to play well as Player 1? In the 1v1 setting, this didn’t seem to be a problem... Is there a bug 🐛 somewhere in my algorithm, architecture, or best-response exploiter that only comes up with more dice and for Player 1? If I figure it out, I’ll update this post.
Overall the algorithm seems to learn Liar’s Dice very well & very fast. I’m pretty encouraged by this, given how simple the code and architecture is. It is certainly strong enough to win most of the time against people online. Feel free to play around with the code on Github and make it better!
I decided that having only a terminal interface wouldn’t encourage many of my friends to test the model. I needed something browser-based, but I didn’t want to create an entire server/client setup for running the model.
Luckily I found out about ONNX, particularly this great video by Elliot Waite. In two lines of code, the PyTorch model can be converted into an onnx file, which can be used directly from javascript:
torch.onnx.export(model, path=’out.onnx’)
I created a small static javascript frontend, bought a domain name, and you can now play against it on dudo.ai.
This article is inspired by Erik Bernhardsson’s Deep learning for... chess from 2014. Besides Deepmind’s Player of Games, I took inspiration from other exciting poker papers, like DouZero, DeepStack and ReBeL.
The more advanced methods have two main improvements:
They use two neural networks instead of one: One network (like ours) for the expected game value, and another one that predicts the optimal policy/probability distribution. This second network can be trained particularly well because of another innovation:They use search. Search means that the strategy (probability distribution) for a given state is iteratively improved as the AI is given time to “think”. AIs for classical games — like chess — have always done this, but in CFR the strategy is entirely computed at training time, and more “thinking time” during play doesn’t help the AI. Only in recent years was it discovered how to apply this to games like Liar’s Dice, and the methods are much more complicated than what I thought I could do in an afternoon.
They use two neural networks instead of one: One network (like ours) for the expected game value, and another one that predicts the optimal policy/probability distribution. This second network can be trained particularly well because of another innovation:
They use search. Search means that the strategy (probability distribution) for a given state is iteratively improved as the AI is given time to “think”. AIs for classical games — like chess — have always done this, but in CFR the strategy is entirely computed at training time, and more “thinking time” during play doesn’t help the AI. Only in recent years was it discovered how to apply this to games like Liar’s Dice, and the methods are much more complicated than what I thought I could do in an afternoon.
For more information on CFR, you can see
The original paper Regret Minimization in Games with Incomplete Information from NIPS 2007.The Monte Carlo version from NIPS 2009.A more principled paper on Deep Counterfactual Regret Minimization from ICLR 2020.Chapter 12.4.2.2 on Policy-Based Methods in Handbook of Reinforcement Learning and Control from 2021.
The original paper Regret Minimization in Games with Incomplete Information from NIPS 2007.
The Monte Carlo version from NIPS 2009.
A more principled paper on Deep Counterfactual Regret Minimization from ICLR 2020.
Chapter 12.4.2.2 on Policy-Based Methods in Handbook of Reinforcement Learning and Control from 2021.
Thanks to Kosti for reading an earlier draft of the article, and to Troels Bjerre Lund for criticizing my frivolous use of CFR terms.