Slice a tensor inplace correctly

username: vadimkantorov
cooked: <p>resize_ doesn’t seem to work, while set_ seems to work. Any idea why?</p>
<pre><code class="lang-python">import torch

B = 2
T = 4

x = torch.rand(B, T)
y = x[:, :T - 1].clone()
print('y', y)
#y tensor([[0.6385, 0.3710, 0.0730],
#        [0.7255, 0.4496, 0.9298]])
# ground truth

b = x.clone()
b.resize_(B, T - 1)
print('b', b)
# b tensor([[0.6385, 0.3710, 0.0730],
#        [0.2039, 0.7255, 0.4496]])
# incorrect

c = x.clone()
reshaped = c[:, :T - 1]
c.set_(c.storage(), 0, reshaped.size(), reshaped.stride())
print('c', c)
# c tensor([[0.6385, 0.3710, 0.0730],
#        [0.7255, 0.4496, 0.9298]])
# correct
</code></pre>

username: albanD
cooked: <p>Resize_ is just a low level construct that makes b a contiguous Tensor with the given size (increasing the storage if needed).<br>
Note that it does not initialize the memory or guarantee re-used of the existing storage. So you can get arbitrary values after calling it.<br>
Resize is <em>not</em> doing slicing!</p>
<p>set_ changes the Tensor and metadata of the Tensor. In this case, you set a new storage, offset, size and stride. And they happen to be what you expect?</p>

username: vadimkantorov
cooked: <p>I thought that resize_ does a similar set_ under the hood (and adjusting the strides). If it’s not doing, that I guess there should be an op that does that… Or at least some explicit mention in docs about this usecase.</p>

username: albanD
cooked: <p>I am not sure to understand what you’re trying to do here? What would be the matching out of place code?<br>
Also what is the benefit of doing a view op inplace??</p>

username: vadimkantorov
cooked: <p>I’m calling <code>resize_to_min_size_(tensorA, tensorB, *many_more_tensors, dim = -1)</code></p>
<pre><code class="lang-auto">def resize_to_min_size_(*tensors, dim = -1):
    tensor_dim_slice = lambda tensor, dim, dim_slice: tensor[(dim if dim &gt;= 0 else dim + tensor.dim()) * (slice(None), ) + (dim_slice, )]
    size = min(t.shape[dim] for t in tensors)
    for t in tensors:
        if t.shape[dim] &gt; size:
            reshaped = tensor_dim_slice(t, dim, slice(size))
            t.set_(t.storage(), 0, reshaped.size(), reshaped.stride())
</code></pre>
<p>This is to save for errors when adding tensors to the call line. This is a workaround for absence of SAME padding mode while doing some audio processing.</p>
<p>Basically this is some inplace slicing.</p>

username: albanD
cooked: <p>But why doesn’t this return a new list of updated Tensors?<br>
It will be most efficient as you skip the set call.<br>
It will be less dangerous as if anything else has a reference to an element of tensors in this function, they will all see the new size.</p>

username: vadimkantorov
cooked: <p>Just to save keystrokes and avoid typing down all the tensor names twice - this is just some code for hacking <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>Without this it would be: <code>a, b, c, d, e, f, g = slice_to_size(a, b, c, d ,e, f, g)</code>. Very cumbersome if names are long and similar-ish.</p>

username: vadimkantorov
cooked: <p>Of course I agree that this is a dangerous operation (as is resize_ or set_). This is just to mention that resize_ semantic was not very clear (wrt to strides). Maybe it should accept a flag that would also adjust the strides.</p>

username: albanD
cooked: <p>I think <code>resize_()</code> should be retired <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"><br>
And the doc already mentions this semantic very clearly in the doc: <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.resize_">https://pytorch.org/docs/stable/tensors.html#torch.Tensor.resize_</a> (see the warning)</p>
<p>Another point is that autograd semantic for inplace view is weird at best <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> And we currently don’t have any inplace view which would be a brand new concept we would need to introduce and support.</p>

username: vadimkantorov
cooked: <p><img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> I always have to scratch my head when I need to decide if I need to change strides or not <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> so I didn’t give a very careful thought to the warning <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: vadimkantorov
cooked: <p>But for other people like me, maybe it’s worth expanding the warning to explain that in this case modifying strides would be needed</p>