Autograd independently on entries of a single tensor

Danimator: <p>Hello,</p>
<p>I am trying to do a computation which follows the following pattern:</p>
<pre data-code-wrap="python"><code class="lang-python">output = torch.zeros((len,), device=...)
for i in range(len):
    output[i] = do_computation_on(output[:i])
</code></pre>
<p>To maintain compatibility with autograd (i.e., avoiding the <code>one of the variables needed for gradient computation has been modified by an inplace operation</code> message), it is necessary to use <code>output[:i].clone()</code>.</p>
<p>In the interest of optimizing memory usage, I would like to avoid this call to <code>clone()</code> as it is clearly conceivable to implement the above pattern without any cloning: just maintain a list of singleton tensors and do the computation that way instead of using a single tensor to maintain the whole list, but this clearly sacrifices memory contiguity (cache-friendliness) and slicing properties I’d like to keep.</p>
<p>Is there any way to maintain a list of independent variables (in the eyes of autograd) which are still contiguous in GPU memory? And if so, are there any drawbacks to such an approach?</p>
<p>Any help is greatly appreciated.</p>
<p>P.S.: I tried to do the above using <code>.chunk()</code> (as it returns a tuple of sub-tensors), and it seems PyTorch was one step ahead of me as this does not work either:<br>
<code>RuntimeError: A view was created in no_grad mode and is being modified inplace with grad mode enabled. This view is the output of a function that returns multiple views. Such functions do not allow the output views to be modified inplace. You should replace the inplace operation by an out-of-place one.</code></p>

KFrank: <p>Hi Danimator!</p>
<aside class="quote no-group quote-modified" data-username="Danimator" data-post="1" data-topic="219258" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/danimator/48/76713_2.png" class="avatar"> Danimator:</div>
<blockquote>
<p>I am trying to do a computation which follows the following pattern:</p>
<pre data-code-wrap="python"><code class="lang-python">output = torch.zeros((len,), device=...)
for i in range(len):
    output[i] = do_computation_on(output[:i])
</code></pre>
<p>…<br>
maintain a list of singleton tensors and do the computation that way instead of using a single tensor to maintain the whole list, but this clearly sacrifices memory contiguity (cache-friendliness) and slicing properties I’d like to keep.</p>
</blockquote>
</aside>
<p>I don’t believe that there is any way to do what your want.</p>
<p>At a higher level, bear in mind that a <code>Tensor</code> not only contains values, but also other<br>
stuff.  In your particular case, you need the connections to the computation graph that<br>
autograd builds and uses.</p>
<p>I believe (but am not sure) that you could build a list of singleton tensors whose contained<br>
values were contiguous in memory (by manipulating the tensors’ storage appropriately), but<br>
I don’t see how you could control the placement of the tensors’ other data, so you would lose<br>
the full version of cache-friendliness that you probably want.  Furthermore, pytorch wouldn’t<br>
know that the data values were contiguous in memory, so I don’t see how you could keep<br>
any built-in slicing functionality.</p>
<p>(Also, just to be clear, autograd acts on whole tensors, not on elements of tensors.  So to<br>
perform the computation you describe in a way compatible with autograd would seem to<br>
require singleton tensors – even if they were assembled into larger aggregate tensors later.)</p>
<p>Best.</p>
<p>K. Frank</p>

Danimator: <p>Hi K. Frank,</p>
<p>Thanks for your detailed response! That’s a bit of a shame to find out, but not the end of the world. I suppose there is nothing more to do then to eat the quadratic cost of all these copies, and hopefully the cache-friendliness (and speed of copies) works out to be better (than the alternative of maintaining a list of singleton tensors) for all reasonable lengths I’d care for.</p>
<p>Best,<br>
Danimator</p>