Reinforce for a multiplayer game

username: CesMak
cooked: <p>Hey there,</p>
<p>I want to use Policy Gradients (see <a href="https://github.com/pytorch/examples/blob/master/reinforcement_learning/reinforce.py" rel="nofollow noopener">REINFORCE</a> and <a href="https://pytorch.org/docs/stable/distributions.html" rel="nofollow noopener">probability distributions</a>) to train a very simple 4-player card game. I have a lot of questions on this algorithm (part of the code is below, full code is available <a href="https://github.com/CesMak/mcts_cardgame/tree/master/modules" rel="nofollow noopener">reinforcement_learning.py</a>.</p>
<p><strong>Questions</strong></p>
<ol>
<li>How to design the network?</li>
</ol>
<ul>
<li>I have only little experience on this and would like to hear your suggestions</li>
<li>How many hidden layers, conv2d or linear etc?</li>
</ul>
<ol start="2">
<li>How to calculate the discounted rewards?</li>
</ol>
<ul>
<li>I am currently using not discounted rewards</li>
<li>Why should I use discounted rewards? (I have 2 discounted reward functions in the code below which of them would you use?)</li>
</ul>
<ol start="3">
<li>The algorithm learns very slowly (or nothing at all?)</li>
</ol>
<ul>
<li>after 10000 games the result is: [-61199.0, -52777.0, -58842.0, -60025.0]), Player 2 is the reinforcement player.</li>
<li>How to measure if the algorithm learns something? -&gt; monitor losses.mean() ?</li>
<li>How long should I train?</li>
<li>What improvements can I do to learn faster? (Including more Reinforcement players ?, sharing policys?)</li>
</ul>
<ol start="4">
<li>Batches</li>
</ol>
<ul>
<li>I do not use any batches as you can see in the code</li>
<li>In other examples I saw that many are using batches - why?</li>
</ul>
<p><strong>Rules of the game explained:</strong></p>
<p>[8] 0 Laura	RANDOM	Card 9 of R	Hand Index 3<br>
[8] 1 Alfons	REINFO	Card 13 of R	Hand Index 5<br>
[8] 2 Frank	RANDOM	Card 6 of R	Hand Index 4<br>
[8] 3 Lea	RANDOM	Card 5 of R	Hand Index 4<br>
Update rewards:  [0, -4, 0, 0]</p>
<ul>
<li>Round 8, Player Index 0 = Laura plays a Random possible card</li>
<li>Alfons Player Index 1 plays as Reinforcement Player he wins this round (highest card)</li>
<li>In this round he earns 4 minus points</li>
</ul>
<p><strong>General Rules</strong></p>
<ul>
<li>All red cards give -1 point</li>
<li>60 Cards in total (15 rounds has one game)</li>
</ul>
<p><strong>State Vector</strong> 180x1 binary vector consists of</p>
<ul>
<li>60x1 binary vector of played cards</li>
<li>60x1 binary vector of cards in hand of the ai player</li>
<li>60x1 binary vector of cards currently played</li>
</ul>
<p><strong>Code</strong></p>
<pre><code class="lang-auto"># tested with python 3.7.5
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from torch.distributions import Categorical
import numpy as np

import stdout

from gameClasses import card, deck, player, game

class PolicyGradientLoss(nn.Module):
    def forward(self, log_action_probabilities, discounted_rewards):
        # log_action_probabilities -&gt; (B, timesteps, 1)
        # discounted_rewards -&gt; (B, timesteps, 1)
        losses = -discounted_rewards * log_action_probabilities # -&gt; (B, timesteps, 1)
        loss = losses.mean()
        print("Mean Loss  :" ,round(loss.item(), 5), "Shape losses:", losses.shape)
        return loss

### TODO GAME HERE
class TestReinforce:
    def __init__(self, parent=None):
        #self.playingPolicy = PlayingPolicy()
        self.witchesPolicy  = WitchesPolicy()
        self.options = {}
        self.options_file_path =  "../data/reinforce_options.json"
        with open(self.options_file_path) as json_file:
            self.options = json.load(json_file)
        self.my_game     = game(self.options)

    def notifyTrick(self, value):
        # der schlimmste wert ist -17 (g10, g5, r1, r2)
        # ausser wenn noch mal2 hinzukommt?! dann ist es wohl 21?!
        #value +=21
        normalizedReward = value / 21 # 21 zuvor sonst 26
        if abs(normalizedReward)&gt;1:
            stdout.enable()
            print(normalizedReward)
            print(eeee)
        #self.playingPolicy.feedback(normalizedReward)
        self.witchesPolicy.feedback(normalizedReward)

    def selectAction(self):
        '''
        the returned action is a hand card index no absolut index!
        '''
        current_player = self.my_game.active_player
        if "RANDOM" in self.my_game.ai_player[current_player]:
            action = self.my_game.getRandomOption_()
        elif "REINFO"  in self.my_game.ai_player[current_player]:
            # get state of active player
            active_player, state, options = self.my_game.getState()
            #print("Options", options)
            #print("State: [Ontable, hand, played]\n", state)

            #torch_tensor = self.playingPolicy(torch.tensor(state).float()   , torch.tensor(options))
            torch_tensor = self.witchesPolicy(torch.tensor(state).float()   , torch.tensor(options))
            # absolut action index:
            action_idx   = int(torch_tensor[:, 0])
            log_action_probability = torch_tensor[:, 1]
            card   = self.my_game.players[current_player].getIndexOfCard(action_idx)
            action = self.my_game.players[current_player].specificIndexHand(card)
        return action

    def play(self):
        total_points = [0, 0, 0, 0]
        for j in range(0, 200):
            i=0
            nuGames = 100
            while i&lt;nuGames:
                action = self.selectAction()
                current_player = self.my_game.active_player
                card   = self.my_game.players[current_player].hand[action]
                print("[{}] {} {}\t{}\tCard {}\tHand Index {}".format(self.my_game.current_round, current_player, self.my_game.names_player[current_player], self.my_game.ai_player[current_player], card, action))
                rewards, round_finished = self.my_game.step_idx(action, auto_shift=False)
                if round_finished:
                    # player idx of Reinforce
                    self.notifyTrick(rewards[1])
                    print("Update rewards: ", rewards, "\n")
                    if len(self.my_game.players[current_player].hand) == 0: # game finished
                        print("update policy at end of one game!")
                        #self.playingPolicy.updatePolicy()
                        self.witchesPolicy.updatePolicy()
                        stdout.enable()
                        if i == nuGames-1:
                            print("game finished with:::", self.my_game.total_rewards, "\n")
                        stdout.disable()
                        self.my_game.reset_game()
                        i+=1
            if j&gt;100:
                for i in range(len(total_points)):
                    total_points[i] += self.my_game.total_rewards[i]
            self.my_game.total_rewards = [0, 0, 0, 0]
        stdout.enable()
        print(total_points)

if __name__ == "__main__":
    trainer = TestReinforce()
    trainer.play()

</code></pre>

username: iffiX
cooked: <ol>
<li>
<p>most of the time for most gym environments three Linear layers is enough, maybe 64 ~ 500 neurons per level and I would suggest you use a pyramid like structure. Conv2d is only necessary for visual inputs.</p>
</li>
<li>
<p>Must use discount &lt; 1, otherwise there is no garantee on convergence, because the convergence of magic like RL algorithms relies on a simple math principle, you must have learned it in your freshman math analysis class: <a href="https://en.wikipedia.org/wiki/Convergent_series" rel="nofollow noopener">converging series</a> or alittle bit more advanced <a href="https://en.wikipedia.org/wiki/Compactification_(mathematics)" rel="nofollow noopener">compaction</a></p>
</li>
<li>
<p>Because the naive REINFORCE algorithm is bad, try use DQN, RAINBOW, DDPG,TD3, A2C, A3C, PPO, TRPO, ACKTR or whatever you like. Follow the train result reference <a href="https://github.com/openai/gym/wiki/Leaderboard" rel="nofollow noopener">openai gym train reference</a>, normally you need to let the agent interact with the environment for 100K or even 1M steps, for extremely complex real life scenes, you need stacks of servers and massively distributed algorithms like IMPALA. There are <strong>many many many many</strong> methods to learn faster, but I would recommend you to start with PPO. But PPO is not a solution once and for all, it cannot solve some scenes like the hardcore bipedalwalker from openai gym.</p>
</li>
<li>
<p>You will know that it is learning, by looking at its behavior changing from a total noise, to a fool, to an expert, and a fool again. Reward and loss might be good indicators, but be careful of your reward design, as your networks will exploit it and take lazy counter measures! Thats’ called reward shaping.</p>
</li>
<li>
<p>Please use batches, because it can stablelize training, <strong>because</strong> pytorch will average the gradients computed from each sample from the batch, You see more, you can judge better, right? normally 100~256 samples per batch, but some studies says monsterous batchsize could be helpful like 20000+</p>
</li>
</ol>

username: CesMak
cooked: <p>Thanks for this extensive very good answer! This is what I was looking for.</p>
<p>I successfully got PPO for a multiplayer card game working. I also let the network learn correct moves and it has learned 16.95 of 17 after 8h of training (on single i7 cpu no gpu used). After 8h training is slowed down dramastically and is not proceeding much. Any ideas on what I can do next? Any suggestion for an algo I should try next?</p>
<p>This is my repro:<br>
Most recent version will be (soon) here:<br>
</p><aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/CesMak/mcts_cardgame/tree/master/modules" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/20240718?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/CesMak/mcts_cardgame/tree/master/modules" target="_blank" rel="nofollow noopener">CesMak/mcts_cardgame</a></h3>

<p>Reinforcement Learning (PPO) applied to a multiplayer simple card game (Witches) - CesMak/mcts_cardgame</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: iffiX
cooked: <p>I have read your code, and from my personal perspective, you may adjust your hyper-parameters, network structures etc. Training results depends on your algorithm type, your test scenario, and how you define the rewards.</p>
<p>PS: layer initialization method is also possible to change, since pytorch use xavier to initilize weights, may be consider about using orthorgonal methods.</p>