Training a card game classifier

CesMak: <p>Hey there,<br>
I currently used Monte Carlo Tree Search (MCTS) to predict good actions for a card game (4 players, each 15 cards). This works quite nice, but is computationally expensive. That is why I thought about training a Neuronal Network with that data.<br>
My goal is that this nn should predict me very fast an action for an input state vector.</p>
<p>My data (generated by MCTS) for one batch is as follows:</p>
<ul>
<li>x: <strong>input vector:</strong> 180x1
<ul>
<li>60x1 binary vector for card is on the table</li>
<li>60x1 binary vector for card is already played</li>
<li>60x1 binary vector for card is on the players hand.<br>
*y: <strong>output vector:</strong> 60x1 (has only one 1 for the card index of the action that is played by the MCTS player).</li>
</ul>
</li>
</ul>
<p><strong>Questions:</strong></p>
<ol>
<li>As you can see I have binary input and binary output vector. Is this advisable to use binary inputs and binary outputs?</li>
<li>I have no doubt on how to design my nn (number of hidden layers, relu or linear etc.) What would you suggest?</li>
<li>Also the loss function I do not know how I should choose it.</li>
</ol>
<p>I currently searched on classifier examples but I mostly found classifiers for images and also in linear case (no binary inputs / outpus)<br>
Hope you can point me to some examples which suit my case.</p>

KFrank: <p>Hi Markus!</p>
<aside class="quote no-group" data-username="CesMak" data-post="1" data-topic="70625" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cesmak/48/21445_2.png" class="avatar"> CesMak:</div>
<blockquote>
<ul>
<li>x: <strong>input vector:</strong> 180x1
<ul>
<li>60x1 binary vector for card is on the table</li>
<li>60x1 binary vector for card is already played</li>
<li>60x1 binary vector for card is on the players hand.<br>
*y: <strong>output vector:</strong> 60x1 (has only one 1 for the card index of the action that is played by the MCTS player).</li>
</ul>
</li>
</ul>
<p><strong>Questions:</strong></p>
<ol>
<li>As you can see I have binary input and binary output vector. Is this advisable to use binary inputs and binary outputs?</li>
</ol>
</blockquote>
</aside>
<p>There is no problem in general with having binary inputs and/or<br>
outputs.  Also, to first approximation, you don’t get to choose your<br>
inputs and outputs – the problem you are trying to solve gives<br>
them to you.</p>
<p>Note, however, that your <em>target</em> vector – the value of your y vector<br>
that you are given (rather than predict) and use to train with – is<br>
not a general binary vector.  It is a class label, equivalent to an<br>
integer class label that takes it values in [0, 59] (inclusive). However,<br>
it is <em>one-hot encoded</em> – that is, it is a binary vector with exactly<br>
one 1 value (as you specified).</p>
<aside class="quote no-group">
<blockquote>
<ol start="2">
<li>I have no doubt on how to design my nn (number of hidden layers, relu or linear etc.) What would you suggest?</li>
</ol>
</blockquote>
</aside>
<p>I would start by treating this as a conventional classification problem.<br>
You have an input of 180 (binary) “features.”  You are trying to predict<br>
which of 60 classes that input corresponds to.</p>
<p>A good starting point would be a one- or two-hidden-layer network.<br>
So, for a single-hidden-layer network,  you might have a 180x120<br>
fully-connected layer followed by a rectified-linear activation function<br>
(“ReLU”), followed by a 120x60 fully-connected layer that produces<br>
a vector of 60 raw scores (<em>logits</em>) as your output.</p>
<p>For two hidden layers you could add a second ReLU followed by,<br>
say, a 60x60 fully-connected layer.</p>
<p>You could also try other nonlinear activations in place of ReLU (such<br>
as <code>tanh</code>).</p>
<aside class="quote no-group">
<blockquote>
<ol start="3">
<li>Also the loss function I do not know how I should choose it.</li>
</ol>
</blockquote>
</aside>
<p>Then feed your output vector and target into <code>CrossEntropyLoss</code>.<br>
(For this, you will need to recast your one-hot target vector into a<br>
single integer class label, because this is what <code>CrossEntropyLoss</code><br>
expects.)</p>
<p>Now for some more speculative comments:</p>
<p>I assume that in a given position in each of your three 60x1 binary<br>
vectors at most one of the binary values can be 1.  That is, you<br>
have 60 different cards, and there is only one copy of any given<br>
card.  (And there are three other players whose hands are “hidden”<br>
and therefore are not encoded in any vectors.)</p>
<p>This is structure that your network will have to “learn.”  <em>Hypothetically</em><br>
you could encode this structure in your inputs (so the network won’t<br>
have to learn it), for example, by having your input be a single length-60<br>
vector with entries that take on four values: on-table, played, in-hand,<br>
and in-hidden-hands.  However, my intuition tells me that this would be<br>
no better (and likely worse).</p>
<p>Also, I assume that there is a constraint that the class label (the one-hot<br>
target vector in your original description) (and hence the prediction)<br>
can only be one of the cards currently in the player’s hand – that is, a<br>
card for which the player’s hand has a 1 in the corresponding position.</p>
<p>This is also structure that the network must learn.  I don’t see an<br>
attractive way of telling the network this structure, but you could<br>
make it easier for the network to learn this structure.  To do this,<br>
you could add one more fully connected layer (to either your one- or<br>
two-hidden-layer network) where you concatenate the player’s-hand<br>
length-60 binary vector with the length-60 (former) output vector,<br>
and pass it through a 120x60 fully-connected layer to form your<br>
(new) length-60 output vector.</p>
<p>Giving your network a second “late look” at the player’s hand should<br>
make it easy (I think) for the network to learn the constraint that you<br>
can only play a card you have.</p>
<p>Of course, this is only a bunch of talk about what <em>might</em> work.  The<br>
name of the game in this neural-network business is to try it out and<br>
see what actually <em>does</em> work.</p>
<p>This sounds like an interesting (and potentially doable) problem.</p>
<p>Good luck!</p>
<p>K. Frank</p>

CesMak: <p>Hey Frank,<br>
thanks for your very helpful and long answer and all your explanations. It would be cool if we could discuss more concrete on the code. I am not sure if I understood everything right. Here is the code based on your answer - would be nice if you can have a short look on it:</p>
<p>Using a constraint is a cool idea (unfortunately I am not sure how to incooperate that).</p>
<pre><code class="lang-auto"># tested with python 3.7.5
from torch.utils.data import Dataset
import ast
import torch
import torch.nn as nn
import matplotlib.pyplot as plt
from torch.autograd import Variable

# Links:
# https://www.kaggle.com/danieldagnino/training-a-classifier-with-pytorch
# https://discuss.pytorch.org/t/training-a-card-game-classifier/70625/2

class MyDataSet(Dataset):
    '''A line in the dataset consists of
    [bin_options+bin_on_table+bin_played]+[action_output+[round(bestq, 2)]]
    Input: 180x1
    Output: one-hot encoded 60x1
    The bestq comes from the mcts estimation of how good the played action is
    This values is currently not used.
    '''
    def __init__(self, data_root):
        self.samples = []
        with open(data_root,"r") as f:
            self.samples = [ [ast.literal_eval(ast.literal_eval(elem)[0]), ast.literal_eval(ast.literal_eval(elem)[1])] for elem in f.read().split('\n') if elem]
        print(len(self.samples)) # ca. 18000

    def __len__(self):
        return len(self.samples)

    def __getitem__(self, idx):
        # Function that returns one input and one output (label)
        # as one dim output use:
        # torch.Tensor(self.samples[idx][0]), torch.Tensor([self.samples[idx][1]].index(1))
        return torch.Tensor(self.samples[idx][0]), torch.Tensor([self.samples[idx][1]])

class my_model(nn.Module):
    '''
    '''
    def __init__(self, n_in=180, n_hidden=1, n_out=60):
        super(my_model, self).__init__()
        self.n_in  = n_in
        self.n_out = n_out

        self.fc1 = nn.Linear(self.n_in, 120)
        # TODO
        # insert fully con layer here direclty connect
        # hand with output
        self.relu1 = nn.ReLU(inplace=True)
        self.fc2 = nn.Linear(120, self.n_out)

    def forward(self, x):
        x = self.fc1(x)
        x = self.relu1(x)
        x = self.fc2(x)
        return x

if __name__ == '__main__':
    from torch.utils.data import DataLoader
    my_data = MyDataSet('actions__.txt')

    my_loader = DataLoader(my_data, batch_size=1, num_workers=0)

    model = my_model()

    criterium = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-4)

    # Taining.
    loss_values = []
    running_loss = 0.0
    epoch        = 0

    for k, (data, target) in enumerate(my_loader):

        data   = Variable(data,          requires_grad=False) # input
        target = Variable(target.long(), requires_grad=False) # output

        #squeeze the target here?!
        # s your target has a channel dimension of 1, which is not needed using nn.CrossEntropyLoss or nn.NLLLoss.
        #target = target.squeeze(1)
        #TODO recast target  one hot to single!

        # Set gradient to 0.
        optimizer.zero_grad()

        # Feed forward.
        pred = model(data)
        print(pred.shape, target.shape) # torch.Size([1, 60]) torch.Size([1, 1, 61])

        #ValueError: Expected input batch_size (1) to match target batch_size (61).
        # Fails:
        loss = criterium(pred, target.view(-1))

        # Gradient calculation.
        loss.backward()

        # print statistics
        running_loss += loss.item()
        if k % 100 == 99:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, k + 1, running_loss / 100 ))
            loss_values.append(running_loss / 100 )
            running_loss = 0.0

        # Model weight modification based on the optimizer.
        optimizer.step()
</code></pre>
<p>Further Notes:<br>
I have currently 18786 batches -&gt; I think I need more or?</p>
<p>One Batch (line) in actions__.txt looks like:<br>
[’[0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]’, ‘[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, -9.96]’]</p>

KFrank: <p>Hello Markus!</p>
<p>I haven’t looked at your code in detail, but here are comments on some<br>
things I noticed:</p>
<aside class="quote no-group quote-modified" data-username="CesMak" data-post="3" data-topic="70625" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cesmak/48/21445_2.png" class="avatar"> CesMak:</div>
<blockquote>
<pre><code class="lang-auto">class my_model(nn.Module):
    def __init__(self, n_in=180, n_hidden=1, n_out=60):
        super(my_model, self).__init__()
        self.n_in  = n_in
        self.n_out = n_out

        self.fc1 = nn.Linear(self.n_in, 120)
        self.relu1 = nn.ReLU(inplace=True)
        self.fc2 = nn.Linear(120, self.n_out)
</code></pre>
</blockquote>
</aside>
<p>First note that this version of <code>my_model</code> only supports one hidden<br>
layer.  So I would just get rid of the <code>n_hidden</code> argument to the<br>
constructor – it’s not used at all, so it just confuses things.</p>
<p>It’s not a big deal, but you might consider not hard-coding the value<br>
of <code>120</code> for the number of hidden neurons.  It’s a value you might<br>
want to play with, so you could make it an argument to your<br>
<code>my_model</code> constructor, e.g.,</p>
<pre data-code-wrap="python"><code class="lang-python">    def __init__(self, n_in=180, n_middle=120, n_out=60):
</code></pre>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">    optimizer = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-4)
</code></pre>
</blockquote>
</aside>
<p><code>Adam</code> is a perfectly reasonable optimizer, and often works better<br>
than <code>SGD</code>.  But you might also want to experiment with <code>SGD</code> as it<br>
has fewer moving parts, so it can make things a little easier to<br>
understand when you’re developing a new model.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">        #TODO recast target  one hot to single!
</code></pre>
</blockquote>
</aside>
<p>Indeed.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">        pred = model(data)
        print(pred.shape, target.shape) # torch.Size([1, 60]) torch.Size([1, 1, 61])

        #ValueError: Expected input batch_size (1) to match target batch_size (61).
        # Fails:
        loss = criterium(pred, target.view(-1))
</code></pre>
</blockquote>
</aside>
<p>Several things here:</p>
<p>First (and not so important), your batch size – the number of<br>
individual input samples you process per optimizer step – is 1.</p>
<p>As a general rule (and, as a general rule, general rules for neural<br>
networks are often wrong),  you want to have “many” samples<br>
in a batch.  I don’t really have any intuition for your specific problem,<br>
but I think playing with batch sizes ranging from 50 to 500 might<br>
make sense.  (Powers of 2 look cool.)</p>
<p>Now for the error:  Your <code>target</code> shape is wrong.  <code>data.shape</code> is<br>
presumably <code>[nBatch, 180]</code> (where you have <code>nBatch = 1</code>).<br>
<code>pred.shape = [nBatch, 60]</code> is correct.</p>
<p>For <code>CrossEntropyLoss</code> you need to have <code>target.shape = [nBatch]</code>,<br>
that is, a single integer class label for each sample in your batch.<br>
<code>target.shape = [1, 1, 61]</code>is wrong, and taking <code>.view (-1)</code><br>
doesn’t fix that because <code>target.view (-1).shape = [61]</code>, which<br>
is still wrong.</p>
<p>As I understand it, when you read in <code>target</code>, you read in 61 numbers:<br>
a length-60 one-hot encoded class label, followed by what you call<br>
<code>bestq</code>.  First, you need to get rid of <code>bestq</code> – your model has no way<br>
to know to ignore it, so if you include it you will get an error.  Second,<br>
you need to convert your one-hot encoded (59 '0’s and one ‘1’)<br>
class-label vector into a single integer (in the range [0, 59], inclusive)<br>
class label.  You can do this as follows:</p>
<pre data-code-wrap="python"><code class="lang-python">dummy, integer_class_batch = one_hot_class_batch.max (dim = 1)`
</code></pre>
<p>This version of <code>max()</code> returns the tuple <code>(max, argmax)</code>.<br>
<code>dummy = max = 1</code>, and we discard it.  <code>argmax</code> is how we get the<br>
single integer class label from the one-hot vector.</p>
<p>Here, <code>one_hot_class_batch.shape = [nBatch, 60]</code>, and<br>
<code>integer_class_batch.shape = [nBatch]</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>Further Notes:<br>
I have currently 18786 batches → I think I need more or?</p>
</blockquote>
</aside>
<p>I don’t have intuition about your specific problem.  In general,<br>
20,000 samples for a (straightforward) 60-class classification<br>
problem should be plenty.  But the structure of your card game<br>
could make the problem hard.  How easy or hard is it for a<br>
person to choose the right move just looking at the hands?</p>
<p>A couple more comments:</p>
<p>You might consider first building a similar model for a well-studied<br>
problem (for example MNIST digit classification).  That way you’ll<br>
be able to separate bugs in your code (or suboptimal models) from<br>
the problem itself being hard to solve.</p>
<p>As a modified approach, you might consider training your model not<br>
with a target that is the single best move, but with scores for how<br>
good each possible move is (or, for practical purposes, perhaps how<br>
good each of the ten best moves is).</p>
<p>Suppose you have five possible moves: (bad, bad, good, good, best).<br>
<em>Categorical</em> cross-entropy (that is, <code>target</code> is the single correct class<br>
label) will score the two following predictions equally:</p>
<pre><code class="lang-plaintext">   (33% (bad), 33% (bad),  0% (good),  0% (good), 33% (best))
   ( 0% (bad),  0% (bad), 33% (good), 33% (good), 33% (best))
</code></pre>
<p>But a model that makes the second prediction will play a much better<br>
game (even if not perfect) than a model that makes the first.</p>
<p>You talk about <code>bestq</code>.  If you have <code>q</code> values for all possible moves<br>
(or for the best several moves), you could use (values derived from)<br>
your <code>q</code> values as your target, and use what I’ll call (for lack of a<br>
better term) “full” cross-entropy for your loss function.</p>
<p>Last, you mention:</p>
<aside class="quote no-group">
<blockquote>
<p>Using a constraint is a cool idea (unfortunately I am not sure how to incooperate that).</p>
</blockquote>
</aside>
<p>I think you might be referring to the constraint I speculated about that<br>
you can only play a card you actually have in your current hand.</p>
<p>My point was not that you would somehow impose such a constraint.<br>
(That is, you wouldn’t ever make such a constraint explicit or build it<br>
into your model.)  But if such a constraint is part of the rules of your<br>
game, your model with have to <em>learn</em> that rule.  My suggestion to add<br>
to the model a “late look” at what’s in the current hand is a proposal<br>
for making it easier for the model to learn that rule.</p>
<p>Good luck!</p>
<p>K. Frank</p>

CesMak: <p>Hey Frank,<br>
I changed my code in the dataloader to just give back the class index as target (With that modification the code ran through without any errors).</p>
<pre><code class="lang-auto">def __getitem__(self, idx):
        return torch.Tensor(self.samples[idx][0]), torch.Tensor([self.samples[idx][1][:-1].index(1)])
</code></pre>
<p>My problem now was that I am not sure how to interpret the output if I give an 180x1 input vector to the trained model. I get back the following:<br>
learning_rate = 1e-5</p>
<pre><code class="lang-auto">tensor([ 0.0982,  0.0182,  0.0380,  0.0131, -0.1438, -0.1540, -0.1128, -0.1452,
         0.0520, -0.0235, -0.0581, -0.1654,  0.0238, -0.0367, -0.0718,  0.0265,
        -0.0240,  0.0926, -0.0354, -0.1772,  0.0667,  0.0795, -0.1556, -0.1301,
        -0.1172,  0.1152,  0.1159, -0.0975,  0.0838, -0.0328, -0.0970, -0.0034,
        -0.0415,  0.0879,  0.0251,  0.0339, -0.0162, -0.0447, -0.1247, -0.0288,
        -0.0193,  0.0407,  0.0237, -0.1138, -0.0198,  0.0213,  0.0173,  0.0107,
         0.0198,  0.0166, -0.0446, -0.1113, -0.0847, -0.1868, -0.0057, -0.0450,
        -0.1188, -0.1111, -0.0694,  0.0380], grad_fn=&lt;AddBackward0&gt;)
</code></pre>
<p>or learning rate=0.1:</p>
<pre><code class="lang-auto">tensor([20.7308, -0.3333, -0.3342, -0.3399, -0.3359, -0.3355, -0.3352, -0.3441,
        -0.3427, -0.3442, -0.3405, -0.3442, -0.3394, -0.3452, -0.3483, -0.3503,
        -0.3363, -0.3425, -0.3441, -0.3375, -0.3388, -0.3365, -0.3406, -0.3325,
        -0.3509, -0.3480, -0.3365, -0.3329, -0.3407, -0.3453, -0.3416, -0.3386,
        -0.3493, -0.3467, -0.3386, -0.3418, -0.3363, -0.3493, -0.3445, -0.3397,
        -0.3425, -0.3497, -0.3367, -0.3371, -0.3520, -0.3403, -0.3394, -0.3397,
        -0.3496, -0.3333, -0.3350, -0.3397, -0.3334, -0.3502, -0.3414, -0.3430,
        -0.3341, -0.3643, -0.3401, -0.3340], grad_fn=&lt;AddBackward0&gt;)
</code></pre>
<p>What is now the action index I should play -&gt; Taking e.g. 21 did not work (playing moves that are not possible)</p>
<p>Well currently I have in my dataset the bestmove index and the q value which is the expected output value(reward) if the player plays this card.</p>
<p>Hm ok I could train my Classifier also on Q-Values. Now the interpretation of the 60 outputs would reflect its q value (reward if game finished) is that correct? I then would know for which action I get the best q value and I simply have to find the max. q for each of the possible actions among the 60 outputs. - Is that correct?</p>
<p>Referring to the constraint. Yes it is a constraint a player can only play a card it has on its hand.<br>
I think a NN like this one would be nice ( output value is the q value):<br>
<img src="https://raw.githubusercontent.com/CesMak/witches_ai/master/imgs/network.png" alt width="627" height="354"></p>
<p>I am not sure how to construct this model and how to train it. I will work on this one as well.</p>

KFrank: <p>Hi Markus!</p>
<aside class="quote no-group quote-modified" data-username="CesMak" data-post="5" data-topic="70625" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cesmak/48/21445_2.png" class="avatar"> CesMak:</div>
<blockquote>
<p>My problem now was that I am not sure how to interpret the output if I give an 180x1 input vector to the trained model. I get back the following:<br>
learning_rate = 1e-5</p>
<pre><code class="lang-auto">tensor([ 0.0982,  0.0182,  0.0380,  0.0131, -0.1438, -0.1540, -0.1128, -0.1452,
...
</code></pre>
<p>What is now the action index I should play → Taking e.g. 21 did not work (playing moves that are not possible)</p>
</blockquote>
</aside>
<p>As to how to interpret the output, the output of your network means<br>
(as always) what you train it to mean.</p>
<p>You are training a conventional 60-class classifier using cross-entropy<br>
(with categorical class labels) as the loss function.  What does this<br>
mean?</p>
<p>First, you are using <code>CrossEntropyLoss</code>.  This takes raw-score <em>logits</em><br>
as its predictions (the output of your network).  Internally to<br>
<code>CrossEntropyLoss</code> these are converted to probabilities by (in effect)<br>
passing them through <code>softmax()</code>.</p>
<p>Cross-entropy then measures how dissimilar two probability distributions<br>
are.</p>
<p>What are your two probability distributions?  They are the probability<br>
of “0” being the right, move, the probability of “1” being the right<br>
move, and so on.  That is, they are <em>discrete</em> probability distributions<br>
on the space [0, 59] (inclusive).  As probabilities, they are between<br>
0 and 1, and sum to 1, <code>sum{i = 0, 59} (P(i)) = 1</code>.</p>
<p>Your first probability distribution is your prediction, that is, the output of<br>
your network (more precisely, the output of your network after passing<br>
it through <code>softmax()</code>).  Your second distribution is your target.  In this<br>
case your target probability distribution has a special form – exactly<br>
one of the 60 probabilities is 1, with the other 59 being 0.</p>
<p>So you are training your network to output the probability of of each<br>
of the 60 moves being the right move.  The simplest rule will be to<br>
play the move that has the largest predicted probability of being right.<br>
(Again, your network outputs logits, but the larger (more positive) the<br>
logit, the larger the probability, and the smaller (more negative) the<br>
logit, the smaller the probability, so choosing the move with the largest<br>
logit is the same as choosing the move with the largest probability.)</p>
<p>Again, the output of your network means what you train it to mean,<br>
and you’ve trained yours (by using <code>CrossEntropyLoss</code>) to output<br>
the predicted probability of each of the 60 moves being right.</p>
<p>Now, as a practical matter, in order to play the game, you know that<br>
only some moves are actually possible.  Your network – not being<br>
perfect – may sometimes predict an impossible move.  It’s perfectly<br>
reasonable to play the <em>possible</em> move with the largest predicted<br>
logit (probability), rather than playing an impossible move with a<br>
larger logit.  Here, you would be taking the output of your (imperfect)<br>
network and combining it with some simple knowledge you have that<br>
your network hasn’t (fully) learned.  In practice, that’s okay.</p>
<p>Best.</p>
<p>K. Frank</p>

CesMak: <p>Hey Frank,</p>
<p>training worked, however the result is not so good the mcts trained player is only a little bit better than a random player.</p>
<p>I think the next step is to go to real learning. However for this multiplayer game learning is not so easy, as action space can vary depending on the cards. Moreover cards of other players are not known (imperfect information). For this reason I would try a Policy Gradient Method to learn in such a case.</p>
<p><strong>What would you suggest</strong>? The only similar example I found is <a href="https://github.com/bornabesic/belot" rel="nofollow noopener">here</a>, which is not documented at all <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>You can checkout the game here:</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/CesMak/mcts_cardgame" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/20240718?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/CesMak/mcts_cardgame" target="_blank" rel="nofollow noopener">CesMak/mcts_cardgame</a></h3>

<p>Tests with Monte Carlo Tree Search on a multi player card game - CesMak/mcts_cardgame</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


KFrank: <p>Hello Markus!</p>
<aside class="quote no-group" data-username="CesMak" data-post="7" data-topic="70625" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cesmak/48/21445_2.png" class="avatar"> CesMak:</div>
<blockquote>
<p>training worked, however the result is not so good</p>
</blockquote>
</aside>
<p>I would ask how well the training worked, and how you measured that.</p>
<p>Common practice is to break your dataset into two pieces: a training<br>
dataset that you use to train your network, and a test / validation<br>
dataset that you do <em>not</em> train on and that you use to test your trained<br>
network.</p>
<p>You want to track your loss function, usually for every batch, since<br>
you have it per-batch anyway, and your “accuracy” function, again,<br>
usually for every batch, since you already have the per-batch<br>
prediction, and the extra cost of calculating the accuracy is not<br>
that much.</p>
<p>For a conventional classification problem, the accuracy function is<br>
the fraction of predictions that are correct – that is, how often you<br>
predicted the best card to play for the sample hand.  If you always<br>
make the correct prediction, your accuracy would be 1 (100%, if<br>
you prefer that).  If your predictions were effectively random, your<br>
accuracy would be 0.01667 – you would guess the right move one<br>
time in sixty.</p>
<p>If your network is training well, then you should see your loss fall<br>
to a number that is close (in some units) to zero, and your accuracy<br>
rise up close to one.  If this isn’t happening, your network isn’t really<br>
training well.  (It might be training as well as it can, given the difficulty<br>
of the problem, but not really training well.)</p>
<p>You also want to track the loss and accuracy of your network on your<br>
test dataset – that is, on samples that were <em>not</em> used in training the<br>
network.  If it’s not too expensive, you can evaluate your test loss and<br>
accuracy every batch.  In any event, you should probably run your<br>
test evaluation at least once every epoch.  Your network can “train”<br>
well – low loss, high accuracy – on your training set, but still not<br>
perform well on your test set.  This is often referred to as “overfitting”<br>
or “not generalizing.”</p>
<p>If your network is overfitting, there are some things you can try.  Try<br>
fewer parameters and try regularization / weight decay.  There is<br>
some thinking that <code>ReLU</code> activations are likely to generalize better<br>
than things like <code>tanh()</code>.  You can also reduce overfitting by training<br>
on more data, if, of course, you have it.</p>
<p>If your network is not training well, try running the training longer.<br>
The training process sometimes hits “plateaus.”  Try changing<br>
the learning rate – both larger and smaller.  Try using a different<br>
optimizer, e.g., switch SGD for Adam (or vice versa).  It can also be<br>
worth changing your network architecture, in particular, adding more<br>
and/or wider layers (both of which translate to more parameters).<br>
These “bigger” networks are likely to train more slowly, but may end<br>
up performing better after the slower training.</p>
<p>Lastly, if your network is training and generalizing well – that is, if<br>
it is performing well on your test dataset, in particular, achieving a<br>
high accuracy – but not playing a good game, then there is something<br>
unrealistic with the data you are training on.  After all, if your network<br>
is doing a good job predicting the moves chosen by your “MCTS”<br>
scheme, but not playing well, then presumably your MCTS system<br>
shouldn’t be playing well either.</p>
<p>Charts as your training progresses of your loss and accuracy on<br>
both your training and test datasets would help give a sense of<br>
which scenario is likely to be playing out.</p>
<p>(As for other learning methods that might be useful for your card game,<br>
I don’t have much to say.  I don’t know much about potentially relevant<br>
learning methods, and I don’t have much intuition about your specific<br>
use case.)</p>
<p>Good luck.</p>
<p>K. Frank</p>