Mental model for tensor.stride()

PersonalFinance004: <p>I’m trying to understand <code>tensor.stride()</code>.</p>
<p>Let’s compare three tensors with the same underlying data: <code>[0, 1, 2]</code>.</p>
<p>Making a tensor <code>T</code> of shape <code>(3,1)</code> (i.e. a ‘column vector’ x a singleton dimension) from that data will have <code>stride=(1,3)</code>. This makes sense because moving from component <code>T[i,j]</code> to <code>T[i+1,j]</code> corresponds to ‘going 1 forward’ in the underlying data <code>[0,1,2]</code>. Similarly, moving from <code>T[i,j]</code> to <code>T[i,j+1]</code> means ‘going 3 forward’, when we start at the beginning again after reaching the end of the underlying data.</p>
<p>For a tensor of shape <code>(1,3)</code> (‘row vector’ x singleton dimension), we get <code>stride=(3,1)</code>. This also makes sense because now moving from <code>T[i,j]</code> to <code>T[i+1,j]</code> means ‘going 3 forward’ in the underlying data, and analogously ‘going 1 forward’ for the second dimension.</p>
<p>However, I don’t understand the case for shape <code>(3,1,1)</code> (‘column vector’ x singleton dimension x singleton dimension). By the above reasoning, the stride should look like this: <code>stride=(1,3,3)</code> because moving from <code>T[i,j,k]</code> to <code>T[i,j+1,k]</code> would require ‘going 3 forward’ in the data <code>[0,1,2]</code>. However, the actual stride is <code>[1,1,1]</code>. How does this make sense?</p>

PersonalFinance004: <p>I might have made a mistake in converting the numpy arrays to tensors before querying the stride, because now I get <code>stride=(1,1)</code> for the <code>shape=(1,3)</code> case, which makes all the above examples consistent with each other if one sets the stride to 1 for every singleton dimension. The only thing I don’t really understand then is why for a singleton dimension the stride should be 1 instead of 0. Moving 1*8 forward in storage brings me to the next component in one of the other dimensions, instead of remaining where I am (because the singleton dimension has only size 1).</p>

ptrblck: <p>I’ve tried to write a general explanation of tensor shapes and strides in <a href="https://discuss.pytorch.org/t/contigious-vs-non-contigious-tensor/30107/2">this post</a> so you might want to take a look at it.</p>
<aside class="quote no-group" data-username="PersonalFinance004" data-post="3" data-topic="120567">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/d6d6ee/48.png" class="avatar"> PersonalFinance004:</div>
<blockquote>
<p>The only thing I don’t really understand then is why for a singleton dimension the stride should be 1 instead of 0.</p>
</blockquote>
</aside>
<p>A stride of 0 would indicate that you are not “moving” and would be reading the same value from memory. This is the case, if you <code>expand</code> a tensor, so increase the size without copying data:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.tensor([1])
y = x.expand(10)
print(y.size(), y.stride())
&gt; torch.Size([10]) (0,)
y = y.contiguous() # trigger a copy
print(y.size(), y.stride())
&gt; torch.Size([10]) (1,)
</code></pre>