Model learns very slowly but what is this solution?!

gandolfin69420: <p>i stumbled upon<br>
<a href="https://datascience.stackexchange.com/questions/53033/why-my-network-needs-so-many-epochs-to-learn" rel="noopener nofollow ugc">this post</a> and all is well, learning rate too low etc., but then i read:</p>
<hr>
<p>I thus modified the pytorch implementation by moving the 3 lines: <code>optimizer.zero_grad()</code> , <code>loss.backward()</code> , <code>optimizer.step()</code> inside the (training) batches loop instead of running them after the loop. As a result, this is the edited code that performs training and testing at each epoch:</p>
<pre><code class="lang-auto"># Iterate over epochs
for epoch in range(1, n_epochs+1):
    train_loss = 0
    model.train()

    train_predictions = []
    train_true_labels = []

    # Iterate over training batches
    for i, (inputs, labels) in enumerate(train_loader):
        inputs, labels = Variable(inputs).to(device), Variable(labels).to(device)

        optimizer.zero_grad() # set gradients to zero

        preds = model(inputs)
        preds.to(device)

        # Compute the loss and accumulate it to print it afterwards
        loss = loss_criterion(preds, labels)
        train_loss += loss.detach()

        pred_values, pred_encoded_labels = torch.max(preds.data, 1)
        pred_encoded_labels = pred_encoded_labels.cpu().numpy()

        train_predictions.extend(pred_encoded_labels)
        train_true_labels.extend(labels)

        loss.backward()       # backpropagate and compute gradients
        optimizer.step()      # perform a parameter update


    # Evaluate on development test
    predictions = []
    true_labels = []
    dev_loss = 0

    model.eval()
    for i, (inputs, labels) in enumerate(dev_loader):
        inputs, labels = Variable(inputs).to(device), Variable(labels).to(device)

        preds = model(inputs)
        preds.to(device)

        loss = loss_criterion(preds, labels)
        dev_loss += loss.detach()

        pred_values, pred_encoded_labels = torch.max(preds.data, 1)
        pred_encoded_labels = pred_encoded_labels.cpu().numpy()

        predictions.extend(pred_encoded_labels)
        true_labels.extend(labels)
</code></pre>
<p>I thus trained again the network using my default configuration (Adam, lr=0.001) and surprisingly I obtained a convergence at epoch 22 (see images below). I think the issue was there, do you agree? Do you have any additional advice? Thanks again!</p>
<hr>
<p>what? what? so this person reordered optimizer.zero_grad() into the next loop iteration just after the datapoint has been loaded? how should this affect convergence at all? that can’t be real, right?</p>

vdw: <p>Just some quick comments:</p>
<ul>
<li>
<p>You’re not even stating what you’re training to learn. For example, a basic classifier is generally easier to train than, say, a Variational Autoencoder.</p>
</li>
<li>
<p>Depending on your setup (i.e., what you’re trying to learn), are you sure you’re using the correct <code>loss_criterion</code>. Most likely, but it is an easy trap to fall into.</p>
</li>
<li>
<p>You didn’t post the code of you’re model. There are a bunch of things that can go wrong even if there’s not error and the model is at least somewhat training (e.g., see <a href="https://discuss.pytorch.org/t/for-beginners-do-not-use-view-or-reshape-to-swap-dimensions-of-tensors/75524">this post</a>)</p>
</li>
<li>
<p>Your code looks “old”. From the <a href="https://pytorch.org/docs/stable/autograd.html#variable-deprecated" rel="noopener nofollow ugc">docs</a>: “The Variable API has been deprecated: Variables are no longer necessary to use autograd with tensors. Autograd automatically supports Tensors with <code>requires_grad</code> set to <code>True</code>.”</p>
</li>
</ul>

gandolfin69420: <p>to address the comments:<br>
primarily this question is based on the linked SO post, the line (-----) separated section is an excerpt from the post illustrating what was really surprising to me.</p>
<blockquote>
<p>You’re not even stating what you’re training to learn</p>
</blockquote>
<p>relation classification task for NLP<br>
convolutional neural network using PyTorch, and I’m trying to select the best hyper-parameters […]</p>
<blockquote>
<p>are you sure you’re using the correct <code>loss_criterion</code></p>
</blockquote>
<p>i am not sure what loss OP used, but the validation loss seemed to still decrease from epoch to epoch</p>
<blockquote>
<p>there’s not error and the model is at least somewhat training</p>
</blockquote>
<p>the grime OP had was mostly about why the model was training so slow</p>
<blockquote>
<p>Your code looks “old”</p>
</blockquote>
<p>yes, the post is from almost 4 years ago according to SO</p>
<p>my question is mainly concerned about the line separated section, which can also be found in context in update 2 of the SO post. OP was allegedly experimenting with moving <code>optimizer.zero_grad()</code> around.</p>
<p>to my understanding, the resulting training loop is basically semantically equivalent to the more common sequence where the calculated grads are zero’ed at the END of each step, but OP allegedly achieved significantly faster convergence.</p>
<p>Q: is there a misunderstanding on my part, or did the OP probably change around something else that actually fixed the problem, and his reordering of <code>optimizer.zero_grad()</code> was pointless?</p>