Contigious vs non-contigious tensor

username: Manik_Hossain
cooked: <p>whats the difference between contigious vs non-contigious tensor and functionality ?</p>

username: ptrblck
cooked: <p>It’s a flag indicating, if the memory is contiguously stored or not.<br>
Let use an example to see, how we can get a non-contiguous tensor.</p>
<pre><code class="lang-python"># Create a tensor of shape [4, 3]
x = torch.arange(12).view(4, 3)
print(x, x.stride())
&gt; tensor([[ 0,  1,  2],
        [ 3,  4,  5],
        [ 6,  7,  8],
        [ 9, 10, 11]]) 
&gt; (3, 1)
</code></pre>
<p>As you can see, the tensor has the desired shape.<br>
The strides are also interesting in this case. They basically tell us, how many “steps” we skip in memory to move to the next position along a certain axis.<br>
If we look at the strides, we see, that we would have to skip 3 values to go to the new row, while only 1 to go to the next column. That makes sense so far. The values are stored sequentially in memory, i.e. the memory cells should hold the data as <code>[0, 1, 2, 3, ..., 11]</code>.</p>
<p>Now lets transpose the tensor, and have again a look at the strides:</p>
<pre><code class="lang-python">y = x.t()
print(y, y.stride())
print(y.is_contiguous())
&gt; tensor([[ 0,  3,  6,  9],
        [ 1,  4,  7, 10],
        [ 2,  5,  8, 11]]) 
&gt; (1, 3)
&gt; False
</code></pre>
<p>The print statement of the tensor yields the desired transposed view of <code>x</code>.<br>
However, the strides are now swapped. In order to go to the next row, we only have to skip 1 value, while 3 to move to the next column.<br>
This makes sense, if we recall the memory layout of the tensor:<br>
<code>[0, 1, 2, 3, 4, ..., 11]</code><br>
In order to move to the next column (e.g. from <code>0</code> to <code>3</code>, we would have to skip 3 values.<br>
The tensor is thus non-contiguous anymore!</p>
<p>That’s not really a problem for us, except, that some operations won’t work.<br>
E.g. if we try to get a flattened view of <code>y</code>, we will run into a <code>RuntimeError</code>:</p>
<pre><code class="lang-python">try:
    y = y.view(-1)
except RuntimeError as e:
    print(e)
&gt; invalid argument 2: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Call .contiguous() before .view().
</code></pre>
<p>So let’s call <code>.contiguous()</code> before the <code>view</code> call:</p>
<pre><code class="lang-python">y = y.contiguous()
print(y.stride())
&gt; (4, 1)
y = y.view(-1)
</code></pre>
<p>Now the memory layout is contiguous once again (have a look at the strides) and the <code>view</code> works just fine.<br>
I’m not completely sure, but I assume the <code>contiguous</code> call copies the memory to make it continuous again.</p>
<p>That being said, continuous arrays are necessary for some vectorized instructions to work. Also generally they should have some performance advantages, as the memory access pattern on modern CPUs will apparently be used in an optimal way, but I’m really not an expert on this topic, so take these last information with a grain of salt. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"></p>

username: Alex4
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>. Thank you for your explanation. This is quite helpful!</p>

username: rashad101
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks you for your clear and concise explanation <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>

username: akashs
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for such clear and concise explanation.</p>

username: nithiroj
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>  Very clear. <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"></p>

username: Brando_Miranda
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="30107">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>In order to move to the next column (e.g. from <code>0</code> to <code>3</code> , we would have to skip 3 values.<br>
The tensor is thus non-contiguous anymore!</p>
</blockquote>
</aside>
<p>Apologies for being picky with the language. But</p>
<blockquote>
<p><code>The tensor is thus non-contiguous anymore!</code> is confusing to me.</p>
</blockquote>
<p>Did you mean:</p>
<blockquote>
<p><code>The tensor is thus non-contiguous now!</code>?</p>
</blockquote>
<p>It seems like the crux of your whole answer to there is a lot of value, in my opinion, to clarify &amp; correct the grammar if possible.</p>
<p>Thanks for your help &amp; time.</p>

username: Brando_Miranda
cooked: <p>From what I understand this a more summarized answer:</p>
<blockquote>
<p>Contiguous is the term used to indicate that the memory layout of a tensor does not align with its advertised meta-data or shape information.</p>
</blockquote>
<p>In my opinion the word contiguous is a confusing/misleading term since in normal contexts it means when memory is not spread around in disconnected blocks (i.e. its “contiguous/connected/continuous”).</p>
<p>Some operations might need this contiguous property for some reason (most likely efficiency).</p>
<p>Resource:</p>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/48915810/what-does-contiguous-do-in-pytorch">
  <header class="source">

      <a href="https://stackoverflow.com/questions/48915810/what-does-contiguous-do-in-pytorch" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/7483494/mbt" target="_blank" rel="noopener nofollow ugc">
    <img alt="MBT" src="https://i.stack.imgur.com/tRSwM.png?s=256&amp;g=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/48915810/what-does-contiguous-do-in-pytorch" target="_blank" rel="noopener nofollow ugc">What does .contiguous() do in PyTorch?</a>
</h4>

<div class="tags">
  <strong>python, memory, pytorch, contiguous</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/7483494/mbt" target="_blank" rel="noopener nofollow ugc">
    MBT
  </a>
  on <a href="https://stackoverflow.com/questions/48915810/what-does-contiguous-do-in-pytorch" target="_blank" rel="noopener nofollow ugc">09:30PM - 21 Feb 18 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: ptrblck
cooked: <aside class="quote no-group" data-username="Brando_Miranda" data-post="7" data-topic="30107">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/48/14355_2.png" class="avatar"> Brando_Miranda:</div>
<blockquote>
<p>It seems like the crux of your whole answer to there is a lot of value, in my opinion, to clarify &amp; correct the grammar if possible.</p>
</blockquote>
</aside>
<p>What would be the difference, if I change <code>anymore</code> to <code>now</code>?<br>
<code>is_contiguous()</code> returned <code>True</code> before and returns now <code>False</code>, thus I think <code>anymore</code> is the right wording.</p>
<aside class="quote no-group" data-username="Brando_Miranda" data-post="8" data-topic="30107">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/48/14355_2.png" class="avatar"> Brando_Miranda:</div>
<blockquote>
<p>In my opinion the word contiguous is a confusing/misleading term since in normal contexts it means when memory is not spread around in disconnected blocks (i.e. its “contiguous/connected/continuous”).</p>
</blockquote>
</aside>
<p>That might be the source of the confusion. I’m talking strictly, how <code>tensor.is_contiguous()</code> is defined.</p>

username: Brando_Miranda
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="9" data-topic="30107">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>What would be the difference, if I change <code>anymore</code> to <code>now</code> ?<br>
<code>is_contiguous()</code> returned <code>True</code> before and returns now <code>False</code> , thus I think <code>anymore</code> is the right wording.</p>
</blockquote>
</aside>
<p>If that’s the case I think the right wording is to say say that it’s non-contiguous now. Not sure what anymore would mean. Perhaps you meant “not contiguous anymore” I think that makes sense grammatically as far as I understand english.</p>

username: Brando_Miranda
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="9" data-topic="30107">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>That might be the source of the confusion. I’m talking strictly, how <code>tensor.is_contiguous()</code> is defined.</p>
</blockquote>
</aside>
<p>You mean how it’s defined in pytorch and not in standard computer hardware? Perhaps I have the terms confused, hence my motivation to seek clarification. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>I see the confusion now between “non-contiguous anymore” and “not contiguous anymore”.<br>
Thanks for clarifying <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Brando_Miranda
cooked: <p>Can’t tell if you are being sarcastic or funny (or both! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> <img src="https://discuss.pytorch.org/images/emoji/apple/laughing.png?v=9" title=":laughing:" class="emoji" alt=":laughing:">) but I think it’s clear now thanks to the discussion. thanks ptrblck! I appreciate the help, your posts across the forum are really helpful <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: talipturkmen
cooked: <p>Thanks for your clear explanation</p>

username: tanvir_siddique
cooked: <p>That’s a damn good explanation. Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: HashRocketSyntax
cooked: <p>So basically contiguous means the stride == width of a row? So that the operations can glide over the dimensions neatly?</p>

username: Brando_Miranda
cooked: <p>maybe this answer helps you: <a href="https://stackoverflow.com/questions/48915810/pytorch-contiguous" class="inline-onebox" rel="noopener nofollow ugc">neural network - PyTorch - contiguous() - Stack Overflow</a> for a different explanation. Hope it helps!</p>

username: iacob
cooked: <blockquote>
<p><strong>A tensor whose values are laid out in the storage starting from the rightmost dimension onward (that is, moving along rows for a 2D tensor) is defined as <code>contiguous</code>.</strong> <a href="https://stackoverflow.com/a/67021086/9067615" rel="noopener nofollow ugc">Contiguous tensors</a> are convenient because we can visit them efficiently in order without jumping around in the storage (improving data locality improves performance because of the way memory access works on modern CPUs). This advantage of course depends on the way algorithms visit.</p>
<p>Some tensor operations in PyTorch only work on contiguous tensors, such as <code>view</code>, […]. In that case, PyTorch will throw an informative exception and require us to call contiguous explicitly. It’s worth noting that calling <code>contiguous</code> will do nothing (and will not hurt performance) if the tensor is already contiguous.</p>
</blockquote>
<ul>
<li><a href="https://pytorch.org/assets/deep-learning/Deep-Learning-with-PyTorch.pdf" rel="noopener nofollow ugc">Deep Learning with PyTorch</a></li>
</ul>
<p>Note this is a more specific meaning than the general use of the word “contiguous” in computer science (i.e. contiguous <em>and ordered</em>).</p>
<p>e.g given a tensor:</p>
<pre data-code-wrap="python"><code class="lang-python">[[1, 2]
 [3, 4]]
</code></pre>
<div class="md-table">
<table>
<thead>
<tr>
<th>Storage in memory</th>
<th>PyTorch <code>contiguous</code>?</th>
<th>Generally “contiguous” in memory-space?</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>1 2 3 4 0 0 0</code></td>
<td><img src="https://discuss.pytorch.org/images/emoji/apple/white_check_mark.png?v=12" title=":white_check_mark:" class="emoji only-emoji" alt=":white_check_mark:" loading="lazy" width="20" height="20"></td>
<td><img src="https://discuss.pytorch.org/images/emoji/apple/white_check_mark.png?v=12" title=":white_check_mark:" class="emoji only-emoji" alt=":white_check_mark:" loading="lazy" width="20" height="20"></td>
</tr>
<tr>
<td><code>1 3 2 4 0 0 0</code></td>
<td><img src="https://discuss.pytorch.org/images/emoji/apple/x.png?v=12" title=":x:" class="emoji only-emoji" alt=":x:" loading="lazy" width="20" height="20"></td>
<td><img src="https://discuss.pytorch.org/images/emoji/apple/white_check_mark.png?v=12" title=":white_check_mark:" class="emoji only-emoji" alt=":white_check_mark:" loading="lazy" width="20" height="20"></td>
</tr>
<tr>
<td><code>1 0 2 0 3 0 4</code></td>
<td><img src="https://discuss.pytorch.org/images/emoji/apple/x.png?v=12" title=":x:" class="emoji only-emoji" alt=":x:" loading="lazy" width="20" height="20"></td>
<td><img src="https://discuss.pytorch.org/images/emoji/apple/x.png?v=12" title=":x:" class="emoji only-emoji" alt=":x:" loading="lazy" width="20" height="20"></td>
</tr>
</tbody>
</table>
</div>

username: Brando_Miranda
cooked: <p>in short:</p>
<blockquote>
<p>when is a case that we <strong>do</strong> need to call <code>contiguous</code>?</p>
</blockquote>

username: ashari4
cooked: <blockquote>
<p>I’m not completely sure, but I assume the <code>contiguous</code> call copies the memory to make it continuous again.</p>
</blockquote>
<p>Yes, it does.</p>
<pre><code class="lang-auto">t1 = torch.arange(9).view(3,3)
t2 = torch.as_strided(t1, size=(3,3),stride=(1,3))
t3 = t2.contiguous()
assert t2.storage().data_ptr() != t3.storage().data_ptr()
</code></pre>