Making a slice contiguous

thyeros: <p>Hi, community.</p>
<p>I’m trying to find a way to make a slice of a tensor contiguous. One way I tried is the following using permuate.</p>
<pre><code class="lang-auto">import torch

x = torch.zeros([16, 8, 1024, 128], device='cuda:0')                
print(x[:,:, 128:256].is_contiguous()) #False


y = torch.zeros([1024, 16, 8, 128], device='cuda:0')              
print(y[128:256].is_contiguous()) #True

z = y.permute(1,2,0,3)
assert z.size()==x.size()
print(z[:,:,128:256].is_contiguous()) #False
</code></pre>
<p>Why is the last print False? I confirmed the data_ptr() still the same, so assume no copy has been made. If this is not the right way to get a contiguous slice, can someone share a trick for that?</p>
<p>Thank you!</p>

KFrank: <p>Hi Thyeros!</p>
<aside class="quote no-group" data-username="thyeros" data-post="1" data-topic="194733" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thyeros/48/12827_2.png" class="avatar"> thyeros:</div>
<blockquote>
<p>I’m trying to find a way to make a slice of a tensor contiguous.</p>
</blockquote>
</aside>
<p>Call <code>.contiguous()</code> on the slice in question.  If the slice is already contiguous,<br>
it will be a no-op.  If not, it will return a contiguous copy of the slice.</p>
<aside class="quote no-group">
<blockquote>
<p>One way I tried is the following using permuate.</p>
</blockquote>
</aside>
<p>Taking a (non-trivial) permutation of a (contiguous) tensor makes it<br>
non-contiguous.</p>
<aside class="quote no-group">
<blockquote>
<p>Why is the last print False?</p>
</blockquote>
</aside>
<p>There are two things going on here.  Slicing a set of rows (that is, its first<br>
dimension) out of a tensor gives you a contiguous subtensor, while slicing<br>
out a set of columns (or anything other than the first dimension) gives you<br>
a non-contiguous tensor.</p>
<p>This is the key issue.</p>
<p>But you’re trying to fix this issue by using <code>.permute()</code> to make your slice<br>
apply to the first dimension.  However, <code>.permute()</code>, itself, renders the tensor<br>
non-contiguous.</p>
<aside class="quote no-group">
<blockquote>
<p>If this is not the right way to get a contiguous slice, can someone share a trick for that?</p>
</blockquote>
</aside>
<p>The subtensor (slice) you want simply isn’t stored contiguously in memory<br>
so you have to make a contiguous <em>copy</em> of the data to get your contiguous<br>
slice.  <code>.contiguous()</code> does this for you.  (I wouldn’t call this a trick – it just<br>
makes a brute-force copy.)</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.1.2'
&gt;&gt;&gt; t = torch.zeros (4, 4)
&gt;&gt;&gt; t.is_contiguous()
True
&gt;&gt;&gt; t[1:3, :].is_contiguous()                # the two rows in the slice are still contiguous
True
&gt;&gt;&gt; t[:, 1:3].is_contiguous()                # not contiguous -- have to jump over the missing columns
False
&gt;&gt;&gt; t.permute (1, 0).is_contiguous()         # no longer row-major -- not contiguous
False
&gt;&gt;&gt; t[:, 1:3].contiguous().is_contiguous()   # .contiguous() creates a new contiguous copy
True
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

thyeros: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="194733">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p><code>ous()                # not</code></p>
</blockquote>
</aside>
<p>Thanks KFrank. Actually, I’m trying to avoid calling “contiguous()” for the slice, as it becomes expensive in my case (which means every slice I create is NOT contiguous()).</p>