Reinforcement Learning & Sushi Go!
By: Martin Liu
Tags: Machine Learning, Data Science, Card Game, Sushi, Reinforcement Learning

Reinforcement Learning & Sushi Go!

Use Reinforcement Learning algorithm to solve the popular card drafting game Sushi Go!

Reinforcement learning is an area of machine learning concerned with how an agent takes action based on its environment to maximize its long-term reward. Although the concept has been out for a while, its application has not been nearly as successful as supervised and unsupervised learning. RL tends to be hard to converge and even solving simple problem requires some careful architectural design. Today, we will train some RL agents to play the game of SushiGo.

The Game

Photo by Martin Liu

SushiGo is a card-drafting game, where 2–5 players aim to grab the best combination of sushi dishes with the highest score. Each player has an initial hand of 6–9 cards (depending on the player number). Players pick a card from the hand, put it on their board, then pass it on to the next player. The next player picks a card from that hand (now 5–8 cards), put it on their board, and so on. If the initial hand size is 6, then after 6 turns, each player will have picked 6 cards. The players now calculate the score from their board and compare against each other. In the original game, 3 rounds are played, and whoever has the highest sum of scores wins the game. You can also watch the video below to get a sense how to play the game.

https://www.youtube.com/watch?v=o0jdqucqqfQ

The game includes the following cards:

Sashimi: 10pt for set of 3. 0 otherwise.

Tempura: 5pt for set of 2. 0 otherwise.

Nigiri: 3pt for Squid, 2pt for Salmon, and 1pt for Egg.

Wasabi: Triples the value of the next Nigiri. 0 if no Nigiri afterwards.

Dumplings: 1pt for the first dumpling, 2pt for the second one, 3pt for the third, 4pt for the fourth, and 5pt for other ones.

Maki Rolls: For each Maki card you take, you will add the number of Maki Rolls on that card to your board. This can be 1, 2, or 3, depending on the card. At the end of each round, every player calculates the number of rolls they have. The most gets 6 points, and 3 points for second. Ties will split the reward.

There are two additional cards in the game, but not included in this version of implementation due to complexity. They are:

Pudding: At the end of the game , the player with the most puddings get 6 points, and the player with the least get -6 points. Unlike other cards, puddings are not discarded each around.

, the player with the most puddings get 6 points, and the player with the least get -6 points. Unlike other cards, puddings are not discarded each around. Chopsticks: Chopsticks can be used to swap for two cards during a pick later. Since a player now has the ability to pick two cards at a time if it does have a pair of chopsticks, the action space becomes quadratic.

Representing action, state, agent, and environment

After understanding the game from a human perspective, it’s time to think of it in machine learning framework. In RL, regardless of the method we choose, we need to define the agent and environment, and how the agent chooses its action based on the state.

What are the “actions” in the game? A player only has one type of action, that is to pick a card from the hand that is given to her (or 2 if using chopsticks). An agent only needs to learn how to pick a card.

The next question is, what information about the environment does a player know at the time of decision making? While picking a card, the player knows:

The combination of cards in her hand

The combination of cards on her board (picked cards)

The combination of cards on other players’ board

The cards on previous hands (that she saw on previous turns)

From a human perspective, the first three can be seen at real-time, while the fourth one requires the player to memorize the hands from previous picks.

Implementation of game logic

This implementation simplifies the original game in a couple of ways. Instead of playing three rounds, players will play one round in a game. Since there are no chopsticks, the action space is limited to choosing a card instead of a pair. Also, if two players both get the highest score, they will share victory with no discount.

For each round, the state class draws from the deck randomly, and then each player get a starting hand. For each pick, a player picks a card given its hand, board, and other players’ board.

for p in self.players:

p.pick_a_card(all_player_boards)

self.pass_around()

At the end of each game, players will “reflect” on their performance and adjust their knowledge system. The player uses a reward of 1 for win, -and 1 for loss. The players will loop through all the encountered states during that game and update values based on their logic.

for i, p in enumerate(self.players):

if self.scoreboard[i] == max_score:

p.feed_reward(1)

else:

p.feed_reward(-1)

How does the players choose their action based on the reward information? That’s where different player differ in their behaviors.

Players

Random Player

Choose a card randomly given each hand. Can be used as a baseline opponent.

Rule-based Player

The player pick cards based on simple priority. Since there are cards that are strictly better than others (3 Maki better 1 Maki, Squid Nigiri better than Egg Nigiri, etc.), this could be a major improvement from picking randomly. Based on my personal experience with the game, the priority is:

Squid Nigiri -> Sashimi -> Tempura -> Salmon Nigiri -> 3 Maki -> Dumpling -> 2 Maki -> 1 Maki -> Egg Nigiri -> Wasabi

Wasabi can sometimes be good, but that would require a player to pick Nigiri after picking Wasabi, so I left it the least priority for simplicity purpose.

Q-Learning Player

Q-learning seeks to learn a policy that maximize reward, by choosing action given certain state. Given the episodic nature of the game, and that there is only one reward (win or lose) in the end, we can translate this into evaluating the value of the state after our action. Since the agent needs to memorize every possible state value pair, having too many distinct states are obviously hard for agent to learn. Therefore, I only uses post-action board as the state. That is, after picking a card, what does my board look like? What is the value of that state of the board?

To update the value of the state, a decay gamma is applied for each step before the end. For example, if a player wins with some state, the value for that state will move towards 1. The state one step before the final state move towards 1 * decay_gamma. If state is not seen before, it will be initialized as 0. When choosing action, the agent iterate through all the possible next states by adding any available card to the current board, and pick the card that leads to the state with the highest value. In code, it would translate to

for possible_next_card in set(self.hand):

board = copy.copy(self.board)

add_a_card_to_board(board, possible_next_card)

value = self.model_dict.get(str(board), 0) + random.random() / 1e6

if value > max_value:

max_value = value

action = possible_next_card

A small randomness is added to the value so it does not always choose the first card initially.

Deep Learning Player

In order the resolve the issue large state space, we can use a deep learning model to estimate the value of each state. This allows to include more features at decision time since we longer have to memorize all the distinctive states.

At first, I tried to use a similar learning logic as q-learning. I use features including the player’s board, the player’s hand, and other players’ board, and use this predict the best possible next state. However, given the relative small change of input (picking a different card would only result a single different column), the model does not learn too well. Instead, I modified the logic so that the model does not output a scalar value, but rather an array of values for each different actions. This is more similar to human’s approach, where we evaluate our options and try to pick a card that is best for us. Some cards are inherently of higher value than others, so this method allows the model to capture the “intrinsic value” of cards more easily.

At decision time, the agent passes the current state as the input and output a value for each action. The agent chooses the available action that has the highest value output. During the learning, a player will compare its model output to the actual value. Since the agent does not explore other actions, we will use the model output value for action that are not chosen. For the chosen action, the value will move towards the (reward * decay) product.

model_input = self.board + convert_hand_to_counter(self.hand) + other_player_board_feature action_values = self.model(torch.FloatTensor(model_input))

action_value_rank = torch.argsort(action_values, descending=True)

for r in action_value_rank:

if r.item() in self.hand:

action = r.item()

break

Configuration & Parameter Selection

For the exploration vs. exploitation balance, I chose 0.3 as the exploration rate. That is, the agent explores 30% of the time and chooses action randomly instead of following its model. I found that exploration rate does not make a significant impact in this game, likely due to dynamic nature of the game. Each game provides you very different cards to start so you are exploring anyways.

I chose 0.01 as learning rate for both agents, and reward is +1 for winning and -1 for losing. This should allow quick learning although it might need to be decreased during later epochs.

I chose 0.9 as the decay gamma. That is, for each step away from the final state, reward is multiplied by 0.9. This should allow the agent to learn early states more slowly and carefully.

For simulation, I use 200 training games and 200 eval games in one epoch (round). During training games, players will explore at 0.3 frequency. During eval games, players does not explore but still learns based on the result of the game.

Result

Initially, I did some 1 on 1 games to see if the models are learning. However, since the game is usually played with 3–5 people, I decided to put different agents together and see how they perform against each other. Here is the result.

4-Player Game Stats, Round = 200 Train + 200 Eval Games

Here, we can see that Rule-based player is dominating in first few rounds, and then the deep learning player and q-learning player starts to catch up. Deep Learning player maintain its win rate for a few rounds, before q-learning player starts to get better. Rule-based player’s win rate starts to decline steadily, while the random player hardly wins any games.

Additional, I checked the model to see if it can provide any result for gameplay insight. For simplicity purpose, I checked the model output, if the boards are all clear and there is one card for each kind in the kind. In this case, the value estimation is:

DL Model, Value Estimation on Empty Board

Since the player is playing against 3 other opponents, most of values are negative since the player cannot maintain a 50% win rate. However, the order is used to determine the player’s action. Specifically, we can see that the player really prioritizes Wasabi over others. It would also prefer Sashimi over Squid Nigiri, which is interesting as well. When I play, Squid is usually a must-take on first turn. The prioritization on Wasabi is something interesting, but it could be that since not all the opponents are very strong, the agent might be able to get a Squid Nigiri for a combo 9pt in the first few turns.

Code

I’m currently working on implementing pudding and chopsticks logic and see how it changes the agent behavior. Also, I plan to work on adding memory of previous hands to the model input to see if that can increase win rate. Let me know if you have any other ideas!