As_strided on a subview has surprising offset

JohnCarmack: <p>I was surprised that tensor.as_strided() doesn’t correct for the offset when the tensor is not at the base of the underlying storage:</p>
<pre><code class="lang-auto">import torch
matrix = torch.arange(20).view(4,5)
print(matrix)
row = matrix[2]
print(row)
patchOnRow = row.as_strided( (2,), (1,), 0)
print(patchOnRow)
patchOnMatrix = matrix.as_strided( (2,), (1,), 0)
print(patchOnMatrix)
</code></pre>
<pre><code class="lang-auto">tensor([[ 0,  1,  2,  3,  4],
        [ 5,  6,  7,  8,  9],
        [10, 11, 12, 13, 14],
        [15, 16, 17, 18, 19]])
tensor([10, 11, 12, 13, 14])
tensor([0, 1])
tensor([0, 1])
</code></pre>
<p>The documentation says “the offset in the underlying storage”, which is indeed what it does, but I can’t think of a time when you would not want it to be from the offset of the tensor you called it with.</p>
<p>Perhaps including a note in the docs that you should add tensor.storage_offset() if there is any chance you are operating on a tensor that isn’t at the start of the storage.</p>

ptrblck: <aside class="quote no-group" data-username="JohnCarmack" data-post="1" data-topic="160062">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/c0e974/48.png" class="avatar"> JohnCarmack:</div>
<blockquote>
<p>The documentation says “the offset in the underlying storage”, which is indeed what it does, but I can’t think of a time when you would not want it to be from the offset of the tensor you called it with.</p>
</blockquote>
</aside>
<p>Yes, I agree that it’s confusing as you are now working on a view of the original tensor.<br>
It’s also more confusing as the <a href="https://pytorch.org/docs/stable/generated/torch.as_strided.html?highlight=as_strided#torch.as_strided">docs</a> are wrong in the current <code>1.12</code> relase and show that a default offset of <code>0</code> is used.<br>
<a href="https://github.com/pytorch/pytorch/issues/77730">This PR</a> fixed it and the <a href="https://pytorch.org/docs/master/generated/torch.as_strided.html?highlight=as_strided#torch.as_strided">docs from <code>master</code></a> should show that the offset of the input will be used.</p>
<p>If you remove the expicit <code>storage_offset</code> it will do the right thing:</p>
<pre><code class="lang-auto">row.as_strided((2,), (1,), row.storage_offset())
# tensor([10, 11])

row.as_strided((2,), (1,)) # default offset is None
# tensor([10, 11])
</code></pre>

JohnCarmack: <p>It is still confusing when you specify an offset – two tensors that looks identical by elements, shape, and stride, would give different results for as_strided if one is an offset view.  I would have expected it to just automatically add the storage_offset(), but that is probably not ok to change for compatibility reasons, so spelling out the hazard in the docs would be nice.</p>

ptrblck: <aside class="quote no-group" data-username="JohnCarmack" data-post="3" data-topic="160062">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/c0e974/48.png" class="avatar"> JohnCarmack:</div>
<blockquote>
<p>I would have expected it to just automatically add the storage_offset(), but that is probably not ok to change for compatibility reasons,</p>
</blockquote>
</aside>
<p>I think this is exactly what is happening now if you don’t explicitly specify the <code>storage_offset</code> or am I missing the point?</p>
<pre data-code-wrap="python"><code class="lang-python">row.as_strided((2,), (1,))
# tensor([10, 11])
</code></pre>
<p>Without specifying <code>storage_offset</code> the <code>row.storage_offset()</code> will be used.<br>
Let me know, if I misunderstand the issue.</p>
<aside class="quote no-group" data-username="JohnCarmack" data-post="3" data-topic="160062">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/c0e974/48.png" class="avatar"> JohnCarmack:</div>
<blockquote>
<p>so spelling out the hazard in the docs would be nice.</p>
</blockquote>
</aside>
<p>Yeah, I think maybe an example with a view (maybe just stealing your example) might be helpful.</p>

JohnCarmack: <p>The case where I ran into this was when I <em>was</em> explicitly specifying an offset – I’m pulling patches out of random places inside images.</p>
<pre><code class="lang-auto">batchOfImages = torch.tensor( 128, 3, 64, 64 )
image2 = batchOfImages[2]
stride = image2.stride()
patch = image2.as_strided( (3,16,16), stride, dy * stride[1] + dx * stride[2] )
</code></pre>
<p>Because image2 is a view with a non-zero storage offset, this creates a patch from data that isn’t even in image2, but rather from batchOfImages[0], which is unintuitive.  It can be fixed with:</p>
<p><code>patch = image2.as_strided( (3,16,16), stride, dy * stride[1] + dx * stride[2] + image2.storage_offset())</code></p>

ptrblck: <p>Ah OK, that’s a good use case and I agree that it’s not intuitive at all.<br>
We should add a warning to the docs then.</p>