RuntimeError: input is not contiguous

username: yunjey
cooked: <p>When i reshape a tensor (if the rank is changed), i get the following error.</p>
<pre><code>x = ...  # Tensor of shape (100, 20)
x.view(-1)   # expect a tensor of shape (2000)

RuntimeError: input is not contiguous
</code></pre>
<p>What does ‘contiguous’ mean and why does this error occur?</p>

username: apaszke
cooked: <p>It means that your tensor is not a single block of memory, but a block with holes. <code>view</code> can be only used with contiguous tensors, so if you need to use it here, just call <code>.contiguous()</code> before.</p>
<p>You can find some more details on the memory layout <a href="https://docs.scipy.org/doc/numpy/reference/arrays.ndarray.html#internal-memory-layout-of-an-ndarray">in numpy docs</a>. Torch uses the same representation.</p>

username: wasiahmad
cooked: <p>Can you tell when we get this error? I am facing this problem but not always. I am just wondering why this is happening occasionally?</p>

username: parth126
cooked: <p>Thanks this helped.<br>
Can you tell what causes this? In what cases the variables are not contiguous and if there is a way to ensure apriori that a variable takes up contiguous memory.</p>

username: colesbury
cooked: <p>Use the <code>tensor.contiguous()</code> function. If tensor non-contiguous, it’ll return a contiguous copy. If it’s already contiguous, it’ll return the original tensor.</p>

username: alexis-jacq
cooked: <p>A classical way to obtain this bug is to use transposition. If you do</p>
<pre><code class="lang-python">x = torch.Tensor(5,2)
y = x.t()
</code></pre>
<p>Then, the storage of y is still the same than the one of x. You can check:</p>
<pre><code class="lang-python">x.fill_(0)
 0  0
 0  0
 0  0
 0  0
 0  0
[torch.FloatTensor of size 5x2]

y
 0  0  0  0  0
 0  0  0  0  0
[torch.FloatTensor of size 2x5]
</code></pre>
<p>That way, y[0,4] exists, but it does not exist <em>in the storage</em> of y, that is the storage of x. So y is not contiguous (y[0,4] is missing in the storage). If you try:</p>
<pre><code class="lang-python">y.view(-1)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: input is not contiguous at /b/wheel/pytorch-src/torch/lib/TH/generic/THTensor.c:231
</code></pre>
<p>So you have to make y contiguous with <code>tensor.contiguous()</code></p>

username: parth126
cooked: <p>Thanks. That makes complete sense now.</p>

username: ducha-aiki
cooked: <aside class="quote no-group" data-username="alexis-jacq" data-post="6" data-topic="930">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alexis-jacq/48/180_2.png" class="avatar"> alexis-jacq:</div>
<blockquote>
<p>So you have to make y contiguous with tensor.contiguous()</p>
</blockquote>
</aside>
<p>Why is it not good idea to add automatic  tensor.contiguous() to transpose function?</p>

username: alexis-jacq
cooked: <p>I guess, because it saves a lot of memory to keep one storage for each tensor, while you don’t apply any reshaping transformation.</p>

username: zakaria_laskar
cooked: <p>So, in the following case</p>
<p>z=y.contiguous().view(-1) , will the gradients of some error with respect to z backpropgate to x?.</p>

username: colesbury
cooked: <p><a class="mention" href="/u/zakaria_laskar">@zakaria_laskar</a>: yes, gradient back propagate correctly through the view(-1) and through the contiguous call.</p>

username: alphadl
cooked: <p>thanks for your clear and useful reply ~~~</p>

username: Brando_Miranda
cooked: <aside class="quote no-group" data-username="colesbury" data-post="5" data-topic="930">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/colesbury/48/26_2.png" class="avatar"> colesbury:</div>
<blockquote>
<p>tensor.contiguous()</p>
</blockquote>
</aside>
<p>is calling <code>.view</code> on tensors also a reason why this bug appears?</p>
<p>I am getting this bug from an RNN…I am doing a bunch of reshapings with <code>view</code>:</p>
<pre><code class="lang-auto">        xn_lstm = torch.cat((loss_prep, err_prep, grad_prep), 1).unsqueeze(0) # [n_learner_params, 6]
        # normal lstm([loss, grad_prep, train_err]) = lstm(xn)
        n_learner_params = xn_lstm.size(1)
        (lstmh, lstmc) = hs[0] # previous hx from first (standard) lstm i.e. lstm_hx = (lstmh, lstmc) = hs[0]
        if lstmh.size(1) != xn_lstm.size(1): # only true when prev lstm_hx is equal to decoder/controllers hx
            # make sure that h, c from decoder/controller has the right size to go into the meta-optimizer
            expand_size = torch.Size([1,n_learner_params,self.lstm.hidden_size])
            lstmh, lstmc = lstmh.squeeze(0).expand(expand_size), lstmc.squeeze(0).expand(expand_size)
        lstm_out, (lstmh, lstmc) = self.lstm(input=xn_lstm, hx=(lstmh, lstmc))
</code></pre>
<p>error:</p>
<pre><code class="lang-auto">RuntimeError: rnn: hx is not contiguous
</code></pre>