The NN model is not converging!

username: Nazila-H
cooked: <p>Hi everyone,<br>
I have a code as following. The accuracy and loss of validation (I am training and validation on the same dataset) are not converging!<br>
idx_l is a list with a size of 96.<br>
I really appreciate any help you can provide.</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self, num_input, num_hidden, num_output, dropout,
                 activation='tanh'):
        super(Net, self).__init__()
        self.dropout = nn.Dropout(dropout)
        self.fc1 = nn.Linear(num_input, num_hidden)
        self.fc2 = nn.Linear(num_hidden, num_output)

        if activation == 'tanh':
            self.activation_f = torch.tanh
        elif activation == 'relu':
            self.activation_f = torch.relu

    def forward(self, x):
        x = self.activation_f(self.fc1(x))
        x = self.dropout(x)
        x = torch.sigmoid(self.fc2(x))
        return x
</code></pre>
<p>model_net= Net(num_input=8, num_hidden=4, num_output=2, dropout=0.0, activation=‘tanh’)</p>
<pre><code class="lang-auto">def model(data, label, idx_l, model_net):

    dataset = TensorDataset(data, label)
    data_loader = DataLoader(dataset, batch_size=len(idx_l), shuffle=False, drop_last=True)

    model_net.train()
    optimizer = torch.optim.SGD(model_net.parameters(), lr=0.0001, momentum=0.9)

    max_nr_batches = 1
    iteration_count = 0
    for batch_idx, (data, label) in enumerate(data_loader): 
        data, label = data.to(device), label.to(device)
        optimizer.zero_grad()
        data = data.clone().detach().requires_grad_(True)
        output= model_net(data)
        model_net.to(device)
        loss = loss_fn(pred=output, target=label)
        loss.backward(retain_graph=True)
        grad = torch.autograd.grad(outputs=loss, inputs=data)
        optimizer.step()

    return grad, model_net
</code></pre>
<pre><code class="lang-auto">def validation(data, label,idx_l, model_net):

    dataset = TensorDataset(data, label)
    data_loader = DataLoader(dataset, batch_size=32, shuffle=False, drop_last=True)

    model_net.eval()
    iteration_count = 0
    val_loss = 0.0
    correct = 0
    max_nr_batches = 3
    with torch.no_grad():
        for batch_idx, (data, label) in enumerate(data_loader): 
            data, label = data.to(device), label.to(device)
            output= model_net(data)
            val_loss += loss_fn(output, label).item()  # sum up batch loss
            pred = output.argmax(dim=1, keepdim=True)  # get the index of the max log-probability
            correct += pred.eq(label.view_as(pred)).sum().item()
            if iteration_count &gt;= max_nr_batches &gt;= 0:
                break
    val_loss /= len(data_loader.dataset)
    accuracy = 100.0 * correct / len(data_loader.dataset)

    return val_loss, correct, len(data_loader.dataset), accuracy
</code></pre>

username: InnovArul
cooked: <aside class="quote no-group quote-modified" data-username="Nazila-H" data-post="1" data-topic="107331">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/48db29/48.png" class="avatar"> Nazila-H:</div>
<blockquote>
<pre><code class="lang-auto">    optimizer = torch.optim.SGD(model_global.parameters(), lr=0.0001, momentum=0.9)
</code></pre>
</blockquote>
</aside>
<p>Can you try with some <code>weight_decay</code>? maybe <code>weight_decay=5e-4</code>?</p>

username: Nazila-H
cooked: <p><a class="mention" href="/u/innovarul">@InnovArul</a> Thanks for your reply.<br>
I applied it, but it did not work.</p>

username: InnovArul
cooked: <aside class="quote no-group quote-modified" data-username="Nazila-H" data-post="1" data-topic="107331">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/48db29/48.png" class="avatar"> Nazila-H:</div>
<blockquote>
<p><code>        model_global.to(device)</code></p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="Nazila-H" data-post="1" data-topic="107331">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/48db29/48.png" class="avatar"> Nazila-H:</div>
<blockquote>
<p><code>        grad = torch.autograd.grad(outputs=loss, inputs=data)</code></p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="Nazila-H" data-post="1" data-topic="107331">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/48db29/48.png" class="avatar"> Nazila-H:</div>
<blockquote>
<pre><code class="lang-auto">        if iteration_count &gt;= max_nr_batches &gt;= 0:
           break
</code></pre>
</blockquote>
</aside>
<p>I am not sure what task you are working on. But the lines above seems not needed.<br>
Can you clarify why you need those?</p>

username: Nazila-H
cooked: <p>Sure.<br>
<code>model_global.to(device)</code> is a typo error it would be <code>model_net.to(device)</code>. I rewrote them.</p>
<p>I want to use grad to sending it to other virtual machines to update the models there. Moreover, with loss.backward(retain_graph=True) the model would update in terms of all leaf nodes (data and weights).</p>
<p>The following part is just using if I would pass some data to be trained not all the data. However, in the model () function I want to pass all data so the following part is not needed.</p>
<pre><code class="lang-auto">if iteration_count &gt;= max_nr_batches &gt;= 0:
    break
</code></pre>

username: InnovArul
cooked: <aside class="quote no-group" data-username="Nazila-H" data-post="5" data-topic="107331">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/48db29/48.png" class="avatar"> Nazila-H:</div>
<blockquote>
<p><code>model_global.to(device)</code> is a typo error it would be <code>model_net.to(device)</code>. I rewrote them.</p>
</blockquote>
</aside>
<p>You do not need <code>model_net.to(device)</code> inside training loop.<br>
Push the model to GPU before creating the optimizer.</p>
<aside class="quote no-group" data-username="Nazila-H" data-post="5" data-topic="107331">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/48db29/48.png" class="avatar"> Nazila-H:</div>
<blockquote>
<p>with loss.backward(retain_graph=True) the model would update in terms of all leaf nodes (data and weights).</p>
</blockquote>
</aside>
<p>I am not sure what do you mean by this. Even without <code>retain_graph=True</code>, the model update will do what you mentioned. No?</p>
<p>It seems like you have a larger distributed training mechanism. Maybe if you try simple one-node training to overfit the model with small batches of data, you might be able to find the cause.</p>

username: Nazila-H
cooked: <aside class="quote no-group" data-username="InnovArul" data-post="6" data-topic="107331">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/innovarul/48/5282_2.png" class="avatar"> InnovArul:</div>
<blockquote>
<p>I am not sure what do you mean by this</p>
</blockquote>
</aside>
<p>If I call <code>.backward()</code> I will get the gradient for all the leafs (the input as well as all the weights in the net) then I can optimize the net for all the leafs.<br>
Moreover, I need the gradient w.r.t input data which I get it by <code>data.grad</code>.</p>

username: Nazila-H
cooked: <aside class="quote no-group" data-username="InnovArul" data-post="6" data-topic="107331">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/innovarul/48/5282_2.png" class="avatar"> InnovArul:</div>
<blockquote>
<p>Even without <code>retain_graph=True</code>, the model update will do what you mentioned. No?</p>
</blockquote>
</aside>
<p>I guess if I call backward twice I should use <code>retain_graph=True</code> first time.</p>

username: Nazila-H
cooked: <aside class="quote no-group" data-username="InnovArul" data-post="6" data-topic="107331">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/innovarul/48/5282_2.png" class="avatar"> InnovArul:</div>
<blockquote>
<p>Maybe if you try simple one-node training to overfit the model with small batches of data, you might be able to find the cause.</p>
</blockquote>
</aside>
<p>I will do it, Thanks for your suggestions.</p>

username: localh
cooked: <p>What do you mean by not coverging, that loss is not decreasing? If so, the primary culprit is a bad learning rate most likely.</p>

username: neuralpat
cooked: <p>lr seems pretty low for SGD. How many epochs are you training this for?</p>

username: Nazila-H
cooked: <p>Thanks for your reply.<br>
I tried different values of lr, but it did not change.</p>

username: Nazila-H
cooked: <p>Thanks for your answer,<br>
I tried with 10 epochs.<br>
Actually I am using a data which I did not work on it myself, I know just one hot encoding and normalization are done on it, may one hot encoding cases this?</p>