Why is it recommended to wrap your data with `Variable` each step of the iterations rather than before training starts?

Brando_Miranda: <p>I recently asked on the pytorch beginner forum if it was good practice to wrap the data with <code>Variable</code> each step or pre-wrap the data before training starts. It seems that its better (for some unknown reason to me) to wrap each step rather than before the training starts. Do people know why thats true?</p>
<hr>
<p>Context, saw the practice here:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/vinhkhuc/PyTorch-Mini-Tutorials/blob/master/2_logistic_regression.py#L20" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/vinhkhuc/PyTorch-Mini-Tutorials/blob/master/2_logistic_regression.py#L20" target="_blank" rel="nofollow noopener">vinhkhuc/PyTorch-Mini-Tutorials/blob/master/2_logistic_regression.py#L20</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="10" style="counter-reset: li-counter 9 ;">
<li>def build_model(input_dim, output_dim):</li>
<li># We don't need the softmax layer here since CrossEntropyLoss already</li>
<li># uses it internally.</li>
<li>model = torch.nn.Sequential()</li>
<li>model.add_module("linear",</li>
<li>                 torch.nn.Linear(input_dim, output_dim, bias=False))</li>
<li>return model</li>
<li>
</li>
<li>
</li>
<li>def train(model, loss, optimizer, x_val, y_val):</li>
<li class="selected">x = Variable(x_val, requires_grad=False)</li>
<li>y = Variable(y_val, requires_grad=False)</li>
<li>
</li>
<li># Reset gradient</li>
<li>optimizer.zero_grad()</li>
<li>
</li>
<li># Forward</li>
<li>fx = model.forward(x)</li>
<li>output = loss.forward(fx, y)</li>
<li>
</li>
<li># Backward</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


albanD: <p>Hi,</p>
<p>Few points:</p>
<ul>
<li>As soon as you wrap a <code>Tensor</code> in a <code>Variable</code>, it will start saving all computation history for it. So each operation is slightly more costly (less and less true with recent changes).  Moreover, when you call backward, it will go through all the history, and so the bigger it is, the longer it’s gonna take.</li>
<li>It is quite easy to do an operation on your <code>Variable</code> outsite of the training loop (moving to gpu for example). And in that case, you will end up  with and error saying that at the second iteration, you try to backpropagate through the graph a second time.</li>
</ul>
<p>That being said, <code>Variable</code>s are going to disapear soon as they will be merged with <code>Tensor</code>s, so don’t overthing this <img src="//discuss.pytorch.org/images/emoji/apple/smiley.png?v=5" title=":smiley:" class="emoji" alt=":smiley:"></p>

Brando_Miranda: <p>when are they disappearing?</p>

albanD: <p>For the 0.4 release if I am not mistaken. Not sure when this is due though.</p>

wszp: <p>Sorry to resurrect this topic. I’m new to PyTorch, switching from TF, and trying to learn it.<br>
So is this still the case? Something like<br>
<code>inputs = Variable(inputs, requires_grad=False)</code> is still considered best practice?<br>
Cheers!</p>

albanD: <p>Hi,</p>
<p>Not at all <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
<code>Variable</code> doesn’t exist anymore (it’s a noop).<br>
What do you want to do?<br>
If you want a new Tensor that does not share history with the current one, you can use <code>inputs = inputs.detach()</code>.</p>
<p>Also feel free to open a new thread if you have question about how to write specific stuff or if you want to double check with us that you’re doing things the right way!</p>