Problems about achieving 'fizzbuzz' game using 2-layer nn

Kerwind: <p>I’m a starter learning pytorch based on python 3.7. As I tried to implete ‘fizzbuzz’ game(details about this little game is in the code) using nn, I found the loss decreased quite slowly. And I also noticed that the dimension of y_pre and batchY is different. Is this the reason why the model seems to be unefficient? How can I change my code?</p>
<pre><code class="lang-auto">### FuzzBuzz game. If the given number can be divided by 15, print 'fizzbuzz'; 5, print 'buzz'; 3, print 'fizz', else print itself

def fizz_buzz_encode(i):
    if i % 15 == 0: return 0 
    elif i % 5 == 0: return 1 
    elif i % 3 == 0: return 2 
    else: return 3 

def fizz_buzz_decode(i,prediction):
    return ['fizzbuzz','buzz','fizz',i][prediction]

for i in range(50):
    print(fizz_buzz_decode(i,fizz_buzz_encode(i)))


### impletment using nn

import numpy as np
import torch
NUM_DIGITS = 10 
def binary_encode(i,num_digits):
    return np.array([i&gt;&gt;d&amp;1 for d in range(num_digits)][::-1])

trX = torch.Tensor([binary_encode(i,NUM_DIGITS) for i in range(101,2**NUM_DIGITS)])
trY = torch.LongTensor([fizz_buzz_encode(i) for i in range(101,2**NUM_DIGITS)])


NUM_HIDDEN = 100 
model = torch.nn.Sequential(torch.nn.Linear(NUM_DIGITS,NUM_HIDDEN),
                           torch.nn.ReLU(),
                           torch.nn.Linear(NUM_HIDDEN,4))

device=torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model=model.to(device)

loss_fn = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(),lr = 0.01)

BATCH_SIZE = 256 
for epoch in range(10000):    # 进行10000次epoch
    for start in range(0,len(trX),BATCH_SIZE):
        end = start + BATCH_SIZE    # 每隔128个数分段
        batchX =trX[start:end].to(device)
        batchY =trY[start:end].to(device)
        
        y_pre = model(batchX)
        loss = loss_fn(y_pre,batchY)
        print('Epoch',epoch,loss.item())
        
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>
<p>The output are as flow:<br>
Epoch 0 1.33250093460083<br>
Epoch 0 1.3155981302261353<br>
Epoch 0 1.3044551610946655<br>
Epoch 0 1.2906633615493774<br>
Epoch 1 1.3085103034973145<br>
Epoch 1 1.290393352508545<br>
Epoch 1 1.2784980535507202<br>
Epoch 1 1.2634493112564087<br>
Epoch 2 1.2881089448928833<br>
Epoch 2 1.2691847085952759<br>
Epoch 2 1.256813645362854<br>
Epoch 2 1.2410774230957031<br>
…<br>
Epoch 9998 0.3821529746055603<br>
Epoch 9998 0.38773271441459656<br>
Epoch 9998 0.44764772057533264<br>
Epoch 9998 0.35775092244148254<br>
Epoch 9999 0.38201063871383667<br>
Epoch 9999 0.3875814378261566<br>
Epoch 9999 0.4476431608200073<br>
Epoch 9999 0.357889324426651</p>
<p>I also get confused that my CPU utilization is 100% but my GPU’s is 0, but I already transfered the whole process to cuda. What’s wrong with my code? Thanks a lot.</p>

beaupreda: <aside class="quote no-group" data-username="Kerwind" data-post="1" data-topic="62004">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/85e7bf/48.png" class="avatar"> Kerwind:</div>
<blockquote>
<p>And I also noticed that the dimension of y_pre and batchY is different. Is this the reason why the model seems to be unefficient?</p>
</blockquote>
</aside>
<p>If you look at the <a href="https://pytorch.org/docs/1.3.0/nn.html#crossentropyloss" rel="noopener nofollow ugc">nn.CrossEntropyLoss</a> documentation, you can see that <code>y_pre</code> (shape <code>B x C</code>) and <code>batchY</code> (shape <code>C</code>) have the correct shape, so no problem here (<code>B = batch_size</code> and <code>C = number_classes</code>).</p>
<aside class="quote no-group" data-username="Kerwind" data-post="1" data-topic="62004">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/85e7bf/48.png" class="avatar"> Kerwind:</div>
<blockquote>
<p>I also get confused that my CPU utilization is 100% but my GPU’s is 0, but I already transfered the whole process to cuda.</p>
</blockquote>
</aside>
<p>You might want to make sure that <code>torch.cuda.is_available()</code> returns <code>True</code>. I tried your code and it went on my GPU no problem, so you may not be using yours.</p>

ptrblck: <aside class="quote no-group" data-username="beaupreda" data-post="2" data-topic="62004">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/beaupreda/48/17583_2.png" class="avatar"> beaupreda:</div>
<blockquote>
<p>You might want to make sure that <code>torch.cuda.is_available()</code> returns <code>True</code> . I tried your code and it went on my GPU no problem, so you may not be using yours.</p>
</blockquote>
</aside>
<p>This is one possibility, however I also think you cannot achieve a high utilization on the GPU with such a tiny model, so I would assume to see only a few spikes for this workload.</p>

Kerwind: <p>Thanks for your reply. Actually I’ve already checked torch.cuda.is_available() and the return is True, so I omit it from my code.<br>
What I do not understand is that, my CPU gets 100% when running the code and I can hardly do others things on my computer:( but my GPU is empty.<br>
Also somebody told me that this code is quite efficient running on his computers as the loss drop fast, while mine seems to be useless. Changing learning_rate does not help.</p>

beaupreda: <p>Ok, this seems weird to me. What I can tell you is that running your code takes 53.99 seconds (for the 10 000 epochs). Also, it uses around 511 MiB of memory on my GPU. You can get information about you GPU by running this command in the terminal:</p>
<pre><code class="lang-auto">watch -n 0.5 nvidia-smi
</code></pre>
<p>Maybe you could check if you get the same thing as me on your GPU?</p>

DanaWang: <p>I also met the same problem. Have you solved it? <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>