Question about optimizer

spike354: <p>Hello,</p>
<p>is it right that optimizer.step() only does: p_{n+1} = p_n - alpha*gradient(p_n)? It only performs one step?<br>
If it does so, would’nt it better to perform more steps during one training?<br>
For Example if I had a classifier wouldnt it make sense to do the following:</p>
<pre><code class="lang-auto">def train_loop(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    for batch, (X, y) in enumerate(dataloader):
        # Compute prediction and loss
        X=X.to(device=0)
        y=y.cuda()
        for i in range(1000):
        pred = model(X.to(device=0))
         loss = loss_fn(pred, y)
        # Backpropagation
         for i in range(1000):
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

        if batch % 100 == 0:
            loss, current = loss.item(), (batch + 1) * len(X)
            print(f"loss: {loss:&gt;7f}  [{current:&gt;5d}/{size:&gt;5d}]")
</code></pre>
<p>Instead of only doing:</p>
<pre><code class="lang-auto">def train_loop(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    for batch, (X, y) in enumerate(dataloader):
        # Compute prediction and loss
        X=X.to(device=0)
        y=y.cuda()

        pred = model(X.to(device=0))
        loss = loss_fn(pred, y)
        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch % 100 == 0:
            loss, current = loss.item(), (batch + 1) * len(X)
            print(f"loss: {loss:&gt;7f}  [{current:&gt;5d}/{size:&gt;5d}]")
</code></pre>
<p>one parameter update for each input?</p>

ptrblck: <aside class="quote no-group" data-username="spike354" data-post="1" data-topic="176207">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e5b9ba/48.png" class="avatar"> spike354:</div>
<blockquote>
<p>is it right that optimizer.step() only does: p_{n+1} = p_n - alpha*gradient(p_n)? It only performs one step?</p>
</blockquote>
</aside>
<p>A very simple <code>SGD</code> optimizer would use this update rule, yes.</p>
<aside class="quote no-group" data-username="spike354" data-post="1" data-topic="176207">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e5b9ba/48.png" class="avatar"> spike354:</div>
<blockquote>
<p>If it does so, would’nt it better to perform more steps during one training?</p>
</blockquote>
</aside>
<p>No, I don’t think so, since the gradients were computed from the initial parameter set (let’s call it p0) while you would apply them to the already updated parameters p1, p2, p3, etc.<br>
The second approach will create new outputs using the updated parameter set, would then calculate the new loss, and thus the new and corresponding gradients.</p>
<p>In a very simple example you could check this illustration from Wikipedia:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/4/84e14637e3255d15b2a9dabff96168229f59ee6c.png" alt="image" data-base62-sha1="iXvFN0EZrsfEEU6EjLpIGUzUCZu" width="350" height="375"></p>
<p>In your first example you would apply the first “arrow” multiple times and might thus not converge to the local minimum.</p>

Joschka: <p>Even with <code>retain_graph=True</code>, this kind of code</p>
<pre><code class="lang-python">pred = model(X)
loss _= loss_fn(pred, y)
for i in range(1000):
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>should throw an error, because <code>step</code> modifies the leaf nodes (your model parameters) required for <code>backward</code> in-place.</p>
<p>Doing something like this (when using plain SGD)</p>
<pre><code class="lang-python">pred = model(X)
loss _= loss_fn(pred, y)
optimizer.zero_grad()
loss.backward()
for i in range(1000):
    optimizer.step()
</code></pre>
<p>is just a very inefficient way of multiplying your learning rate by a factor of 1000 <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<p>As a side note: Increasing the step size / learning rate is not always a bad idea. It can also increase training speed and even improve final model performance, but initializing your optimizer with a higher learning rate is the preferred way to do it. What exact learning rate / step size you should go for depends a lot on your problem at hand and finding a good value usually requires some degree of trial-and-error.</p>

ptrblck: <aside class="quote no-group" data-username="Joschka" data-post="3" data-topic="176207">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/4af34b/48.png" class="avatar"> Joschka:</div>
<blockquote>
<p>should throw an error, because <code>step</code> modifies the leaf nodes (your model parameters) required for <code>backward</code> in-place.</p>
</blockquote>
</aside>
<p>That’s also a good point as PyTorch will not even allow the gradient re-calculation using the stale forward activations.</p>

spike354: <p>Let me get that clear. So you say, that in my first approch, I would also use the Vector from x0 to x1 all the time and miss the local Minimum, is that right?</p>

ptrblck: <p>Yes, if you call <code>optimizer.step()</code> inside the loop and move the <code>backward()</code> out as this will already raise a runtime error due to an invalid gradient calculation as pointed out by <a class="mention" href="/u/joschka">@Joschka</a>.</p>

spike354: <p>Thank you very much!</p>