Add one tensor from dataloader to optimiser

alwynmathew: <p>I want to optimize my <code>init_tensor</code> instead of optimizing the <code>model</code>. My data loader <code>train_dataloader</code> gives the initial value of <code>init_tensor</code>.</p>
<pre><code class="lang-auto">model.eval()
for inputs in train_dataloader:
     init_tensor = inputs["init_tensor"]
     optimizer = optim.SGD(init_tensor, learning_rate)
     output = model(init_tensor)
     loss = loss_fn(output)
     model.zero_grad()
     loss.backward()
     optimizer.step()
</code></pre>
<ol>
<li>
<p>Defining the optimizer inside the training loop doesn’t look right. Will this hinder optimizing the <code>init_tensor</code>?</p>
</li>
<li>
<p>How can I move the <code>optimizer</code> initialization outside the training loop but still add <code>init_tensor</code> to <code>optimizer</code>?</p>
</li>
</ol>
<p>Any help would be appreciated.</p>

JuanFMontesinos: <p>What’s wrong with initilizing the optimizer inside training loop?<br>
In the end if you want to optimize the input you will need to initilize an optimizer per input I would say.<br>
Once you have optimized the input, old optimizer is no longer useful.<br>
If you want to optimize both, model and init tensor, you may think about having 2 optimizers, one for the model and another one for each tensor.</p>

alwynmathew: <p>But when we initialize the <code>optimizer</code> as per above mentioned, <code>optimizer</code> could only take one step per <code>input_tensor</code>. Thus would you recommend taking multiple step for each <code>input_tensor</code> until a condition as shown below:</p>
<pre><code class="lang-auto">model.eval()
for inputs in train_dataloader:
     init_tensor = inputs["init_tensor"] # tensor to be optimized
     optimizer = optim.SGD(init_tensor, learning_rate)
     input_tensor = inputs["input_tensor"] # different images from loader
     input = init_tensor + input_tensor
     loss = 1
     while(loss &gt; threshold):
          output = model(input)
          loss = loss_fn(output)
          model.zero_grad()
          loss.backward()
          optimizer.step()
</code></pre>

JuanFMontesinos: <p>I mean, as far as I understand, if you are optimizing a tensor that means somehow that tensor is gonna “remain” during several iterations.  It’s a bit diff to discuss about this not knowing what are you trying to do but yes, iterating over that tensor until you reach your target would be the expected pipeline.</p>
<p>Whether the adecuate criteria were loss or any other metric, I cannot know.</p>

tux: <p>To me, what you are doing is a gradient flow. I would suggest you to have a look at the very good post on Wasserstein gradient flow here : <a href="https://www.kernel-operations.io/geomloss/_auto_examples/comparisons/plot_gradient_flows_2D.html#sphx-glr-auto-examples-comparisons-plot-gradient-flows-2d-py" rel="nofollow noopener">https://www.kernel-operations.io/geomloss/_auto_examples/comparisons/plot_gradient_flows_2D.html#sphx-glr-auto-examples-comparisons-plot-gradient-flows-2d-py</a></p>
<p>In the gradient flow function, they basically do:</p>
<pre><code class="lang-auto">for i in range(Nsteps):
        # Compute loss and gradient
        L_αβ = loss(x, y)
        [g]  = torch.autograd.grad(L_αβ, [x])
        x.data -= learning_rate * len(x) * g
</code></pre>
<p>This way, you do not need to define an optimizer as you do it yourself ! Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

alwynmathew: <p>Thank you <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a> and <a class="mention" href="/u/tux">@tux</a>.</p>
<p>In my above code snippet, I have done <code>model.zero_grad()</code>, but I don’t think that is correct. As I’m optimizing <code>init_tensor</code>, I should zero grad the <code>optimizer</code> as below:</p>
<pre><code class="lang-auto">model.eval()
for inputs in train_dataloader:
     init_tensor = inputs["init_tensor"] # tensor to be optimized
     optimizer = optim.SGD(init_tensor, learning_rate)
     input_tensor = inputs["input_tensor"] # different images from loader
     input = init_tensor + input_tensor
     loss = 1
     while(loss &gt; threshold):
          output = model(input)
          loss = loss_fn(output)
          optimizer.zero_grad()
          loss.backward()
          optimizer.step()
</code></pre>

JuanFMontesinos: <p>In fact, you should zero grad both, model and tensor. Realize that you are doing:</p>
<pre><code class="lang-auto">     while(loss &gt; threshold):
          output = model(input)
          loss = loss_fn(output)
          optimizer.zero_grad()
          loss.backward()
          optimizer.step()
</code></pre>
<p>That way gradients are being accumulated on model. As model isn’t wrapped by the optimizer, its gradients are never zeroed. (In pytorch if you don’t make gradients zero they accumulate to the previous ones)</p>

alwynmathew: <p>Thanks <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a>. Cleaning it up, the final pipeline will look like this:</p>
<pre><code class="lang-auto">model.eval()
for batch_id, inputs in enumerate(train_dataloader):
     init_tensor = inputs["init_tensor"] # tensor to be optimized
     optimizer = optim.SGD(init_tensor, learning_rate)
     input_tensor = inputs["input_tensor"] # different images from loader
     input = init_tensor + input_tensor
     loss = 1
     while(loss &gt; threshold):
          output = model(input)
          loss = loss_fn(output)
          model.zero_grad()
          optimizer.zero_grad()
          loss.backward()
          optimizer.step()
</code></pre>
<p>I still have a doubt,</p>
<p>How can we use the learned info of <code>optimizer</code> from <code>inputs</code> of <code>batch_id=1</code> to improve the optimization of <code>inputs</code> of <code>batch_id=2</code>? As we are initializing <code>optimizer</code> for each <code>inputs</code> we are throwing out all it has learned from <code>inputs</code> of <code>batch_id=1</code>. So that the optimizer should spend almost same amount of time optimizing <code>inputs</code> with <code>batch_id=2</code> as it did with <code>inputs</code> of <code>batch_id=1</code>. How can we improve this?</p>

alwynmathew: <p>According to the link below, we don’t need to do <code>model.zero_grad()</code> as the <code>model</code> is in <code>eval</code> mode.</p>
<aside class="quote quote-modified" data-post="5" data-topic="1628">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/backprop-in-pytorch/1628/5">Backprop in PyTorch?</a> 
  </div>
  <blockquote>
    Hi, 
So pytorch is very different from tensorflow in the sense that your don’t create the graph once, and then just backprop through it. 
The whole framework is built such that the graph is super cheat to build and so it is built on the fly. 
That means that for every input for which you want to compute backprop, you should execute your computation with them, and then call backward on the output. 
In your case, you should have: 
import torch
import torch.nn as nn
from torch.autograd import Varia…
  </blockquote>
</aside>


Pooria: <p>Did you find a solution/explanation for this?</p>