Simple numeric handling DNN

Woosung: <p>Hi guys,</p>
<p>I have a hard time during training and need help.</p>
<p>I’m trying to make DNN which just simply act as one hot encoding.<br>
So, the input of DNN is just a normal vector. and output of DNN will be the one hot vector of input.</p>
<p>I made DNN with simple linear layers and relu. But, unable to train my model…<br>
What could be the problem in here?</p>
<p>I attach my full code.</p>
<pre><code class="lang-auto">class onehot(nn.Module):
    def __init__(self, input_dim):
        super(onehot, self).__init__()
        self.input_dim = input_dim
        self.layers = nn.Sequential(
            nn.Linear(self.input_dim, 32),
            nn.ReLU(inplace=False),
            nn.Linear(32, self.input_dim),
        )
        for m in self.modules():
          if isinstance(m, nn.Linear):
            torch.nn.init.uniform_(m.weight.data)  
    def forward(self, x):
      return self.layers(x)

criterion = nn.CrossEntropyLoss().cuda()
optimizer = optim.SGD(encoder_9.parameters(), lr=0.001, momentum=0.9)

encoder_9 = onehot(9).cuda().train()

for epoch in range(10): 
    running_loss = 0.0
    for i, data in enumerate(x_list_c1):
        # get the inputs; data is a list of [inputs, labels]
        inputs = data.cuda()
        labels = y_list_c1[i].cuda().float()

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = encoder_9(inputs)
        logits = nn.functional.softmax(outputs, dim = -1)
        loss = criterion(logits, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:
            print(f'[{epoch + 1}, {i + 1:5d}] loss: {running_loss / 2000:.6f}')
        running_loss = 0.0

print('Finished Training')

training_epoch = 100
</code></pre>
<p>My inputs and labels are like this<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed3d9a31cf33f358ed88d27393f7c326193c9cad.png" data-download-href="https://discuss.pytorch.org/uploads/default/ed3d9a31cf33f358ed88d27393f7c326193c9cad" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed3d9a31cf33f358ed88d27393f7c326193c9cad_2_690x126.png" alt="image" data-base62-sha1="xQJ8RaqNBfbJ1YSp0l4tppu58C1" width="690" height="126" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed3d9a31cf33f358ed88d27393f7c326193c9cad_2_690x126.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed3d9a31cf33f358ed88d27393f7c326193c9cad_2_1035x189.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed3d9a31cf33f358ed88d27393f7c326193c9cad_2_1380x252.png 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1612×296 19.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>and training result like this</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa981788c637663011387cbff3e2123af06c8f9a.png" data-download-href="https://discuss.pytorch.org/uploads/default/fa981788c637663011387cbff3e2123af06c8f9a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa981788c637663011387cbff3e2123af06c8f9a.png" alt="image" data-base62-sha1="zKRdPu5kMjMPkjxtw63B4pvcMc2" width="690" height="499" data-dominant-color="E5E6E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">702×508 11.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>loss does not decreasing… <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

KFrank: <p>Hi Woosung!</p>
<p>What you have posted doesn’t seem consistent.</p>
<aside class="quote no-group quote-modified" data-username="Woosung" data-post="1" data-topic="198466" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/bb73d2/48.png" class="avatar"> Woosung:</div>
<blockquote>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss().cuda()
...
        outputs = encoder_9(inputs)
        logits = nn.functional.softmax(outputs, dim = -1)
        loss = criterion(logits, labels)
</code></pre>
</blockquote>
</aside>
<p><code>CrossEntropyLoss</code> has <code>log_softmax()</code> inside of it, so your call to<br>
<code>softmax()</code> means that <code>softmax()</code> is being applied <em>twice</em> to <code>outputs</code>.</p>
<p>This is an error and could certainly make your model train poorly.</p>
<p>But, furthermore, even if <code>outputs</code>, the prediction of your model, were a<br>
perfect prediction, the incorrect call to <code>softmax()</code> would make the prediction<br>
relatively poor.  However, you report <code>loss</code> values of the order <code>0.001</code>, which<br>
which is really quite low and, based on the code you posted, you can’t get<br>
a <code>loss</code> value anywhere near that low with the incorrect call to <code>softmax()</code>.</p>
<p>Try printing out <code>outputs</code>, <code>logits</code>, and <code>loss</code> (as well as <code>labels</code>, just to be<br>
complete), and double-check what’s going on.</p>
<p>Aside from this issue, my intuition is that your two-layer model probably<br>
won’t have enough “structure” (whatever that may mean) to do a good<br>
job of performing your desired one-hot computation.  (I could be wrong,<br>
though; I’ve never tried it.)  Once you get your code working properly, you<br>
might consider experimenting with models that are both wider (larger<br>
internal dimensions) and deeper (more layers).</p>
<p>As an aside, please don’t post screenshots of textual information.  Doing<br>
so breaks accessibility, searchability, and copy-past.</p>
<p>Best.</p>
<p>K. Frank</p>