Training slows down with loop like x=f(x)

username: sigma_x
cooked: <p>Here <code>x</code> is a tensor, <code>f</code> is an <code>nn.Module</code> model. Both are on cuda:</p>
<pre><code class="lang-auto">x = x.to(torch.device('cuda'))
f = f.to(torch.device('cuda'))
</code></pre>
<p>The main loop looks like this:</p>
<pre><code class="lang-auto">while(some condition):
      x = f(x)
</code></pre>
<p>I noticed that pretty quickly training slows down. I believe this feedforward mechanism could be a problem. Do I need to use a different approach, like create an intermediary tensor for <code>x</code> and input it into <code>f</code>?</p>

username: JuanFMontesinos
cooked: <p>The problem may be that, by doing that, you are increasing the computational graph (if you never call backward). I would say it depends on how many iterations your loop runs until being broken by the condition. I would have a look on that with a counter. Note that if you feed forward lot of times your graph will be huge, thus, will backpropagate way slower.</p>
<pre><code class="lang-auto">c=-1
while(some condition)
     c+=1
     x = f(x)
</code></pre>

username: sigma_x
cooked: <p>OK then maybe I don’t understand how Pytorch works well enough. When I set<br>
<code>x=f(x)</code><br>
does the ‘new’ x replace the ‘old’ x in the graph?</p>

username: JuanFMontesinos
cooked: <p>Nope, you are creating a junction between both forward passes in a siamese-like fashion. All your runs will have an effect in the backpropagation.</p>

username: sigma_x
cooked: <p>Got it, so how should I do it instead? something like</p>
<pre><code class="lang-auto">
y=f(x)
x=y
</code></pre>

username: JuanFMontesinos
cooked: <p>If I properly understand you want to run that network until it fits a condition? But backpropagating only for those results which fits the condition?</p>
<p>I don’t know if there is an elegant way to do that.<br>
Only possible fast way would be. This approach would by-pass the graph computation, thus, would be faster.</p>
<pre><code class="lang-auto">x_old ...
while(1):
     with torch.no_grad():
          x = f(x_old)
     if condition:
         x = f(x_old)
         break
     else:
         x_old=x
</code></pre>
<p>Another way would be cutting the computational graph. This would probably use more memory and would be slower as all the graph would stay over there (but wouldn’t affect backprop)</p>
<pre><code class="lang-auto">while(1):
     x = f(x)
     if condition:
         break
     else:
         x = x.detach()
</code></pre>
<p>There is a 3rd option that would be deleting the graph but I don’t really know if that’s possible.</p>
<p>Anyway, this pipeline looks a bit strange to me.</p>

username: sigma_x
cooked: <p>Condition is just loop until reaching the predefined iteration number. Apart from that it seems that you did what I thought was right:</p>
<pre><code class="lang-auto">x = f(x_old)
old_x = x
</code></pre>
<p>The idea is to get the network’s output, compute some loss function on it, backprop into the network and then take this output as the network’s input in the next iteration. Does this make sense?</p>

username: JuanFMontesinos
cooked: <p>Well, it’s not the same at all.<br>
So that would be</p>
<pre><code class="lang-auto">while(some condition):
      x = f(x)
     loss = function(x)
     loss.backward()
     ...
     x = x.detach()
</code></pre>
<p>Last part will convert x into a leaf variable and should work</p>

username: sigma_x
cooked: <p>Thanks. So <code>x=detach(x)</code> creates a new variable <code>x</code> and adds it to the graph. What happens to the old <code>x</code> then?</p>

username: JuanFMontesinos
cooked: <p>Backpropagation backpropagates from the variable you call backprop from to leaf variables.</p>
<p>Leaf variables are input nodes to the graph.  Intermediate variables keep graph history.<br>
When you call detach you are kind of breaking the graph and creating a new fresh leaf variable so that memory is reused. Else, it stills carry old graph I would say.</p>
<p>And it’s not x=detach(x) but x=x.detach() <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Detach is a tensor method. I dunno if it exists are a standalone function in torch</p>

username: sigma_x
cooked: <p>Yeah sorry my mistake. So x.detach() removes the ‘old’ x from the graph and inserts the new one, right? Other (inner) tensors stay the same.</p>

username: JuanFMontesinos
cooked: <p>yep</p><aside class="quote quote-modified" data-post="2" data-topic="226">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/48/21_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/help-clarifying-repackage-hidden-in-word-language-model/226/2">Help clarifying repackage_hidden in word_language_model</a> 
  </div>
  <blockquote>
    Every variable has a .creator attribute that is an entry point to a graph, that encodes the operation history. This allows autograd to replay it and differentiate each op. So each hidden state will have a reference to some graph node that has created it, but in that example you’re doing BPTT, so you never want to backprop to it after you finish the sequence. To get rid of the reference, you have to take out the tensor containing the hidden state h.data and wrap it in a fresh Variable, that has n…
  </blockquote>
</aside>