For beginners: Do not use view() or reshape() to swap dimensions of tensors!

username: vdw
cooked: <p>Being more of an NLP person and dealing regulary with LSTMs or GRUs – but this is a general issue, I think – I’ve noticed that many people make a fundamental mistake. I’ve seen it many Github projects I’ve tried to reproduce but also here in the forum (usually something like: “My network runs but does not train/learn properly”, even for arguably simple networks).</p>
<p>When using LSTMs or GRUs, the input and/or output do not have the right shape. For example, the input batch has the shape <code>[batch_size, seq_len, hidden_size]</code>, but an LSTM without <code>batch_first=True</code> assumes an input of shape <code>[seq_len, batch_size, hidden_size]</code>. Many people incorrectly use <code>view()</code> or <code>reshape()</code> to fix the shape. While it does fix the shape, it messes up the data and essentially prohibits proper training (e.g., the loss is not going down).</p>
<p>The correct way here is to use <code>transpose()</code> or <code>permute()</code> to swap dimensions.</p>
<p>To illustrate the problem, let’s create an example tensor with a <code>batch_size</code> and <code>seq_len</code> dimension; I omit the <code>hidden_size</code> dimension to keep things simple</p>
<pre><code>batch_size, seq_len = 3, 5
A = torch.zeros((batch_size, seq_len))
A[0,0:] = 1
A[1,0:] = 2
A[2,0:] = 3
</code></pre>
<p>This gives a tensor with shape <code>[batch_size, seq_len]</code> looking like that:</p>
<pre><code>tensor([[1., 1., 1., 1., 1.],
        [2., 2., 2., 2., 2.],
        [3., 3., 3., 3., 3.]])
</code></pre>
<p>Now let’s say you have defined an <code>nn.LSTM</code> layer with <code>batch_first=False</code>. In this case, you need two swap the two dimensions of A to make an a valid input for the LSTM layer (again, the <code>hidden_size</code> dimension is omitted here). You can do this with the following commands:</p>
<pre><code>A1 = a.transpose(1,0)
A2 = a.permute(1,0)
</code></pre>
<p>If you print either A1 or A1 you get the correct result:</p>
<pre><code>tensor([[1., 2., 3.],
        [1., 2., 3.],
        [1., 2., 3.],
        [1., 2., 3.],
        [1., 2., 3.]])
</code></pre>
<p>However, if you use <code>view()</code> or <code>reshape()</code> like</p>
<pre><code>A3 = a.view(seq_len, -1)
A4 = a.reshape(seq_len, -1)
</code></pre>
<p>you get the wrong result:</p>
<pre><code>tensor([[1., 1., 1.],
        [1., 1., 2.],
        [2., 2., 2.],
        [2., 3., 3.],
        [3., 3., 3.]])
</code></pre>
<p>Note that for <code>A1</code>, <code>A2</code>, <code>A3</code>, <code>A4</code> the shape is <code>[5, 3]</code>. So with respect to the shape all results are correct, i.e., <code>[seq_len, batch_size]</code>, the LSTM layer will accept each result without throwing any error.</p>
<p>However, <code>A3</code> and <code>A4</code> have messed up the batch semantically. The network might still learn something – after all, there’s usually a pattern to be found in everything – but it will learn the wrong pattern.</p>
<p><code>view()</code> and <code>reshape()</code> obviously have their purpose, for example, to flatten tensors. But for swapping dimensions they are generally the wrong methods. I hope this avoids some head-scratching, particularly for beginners. And I can point to this post when I see the next question that most likely suffer from that issue :).</p>

username: pinocchio
cooked: <p>Thanks for the answer! This seems really useful.</p>
<p>I am curious, what would be uses cases for <code>.view</code> then? (perhaps making a list would be useful)</p>
<p>The only places I’ve ever seen it used without it causing issues is when flattening (as you mentioned) and when doing a squeeze (despite a squeeze function already existing). e.g.</p>
<pre><code class="lang-auto">x = torch.randn(3, 4, 5, 6)
x.view(batch_size, -1)  # flat!
</code></pre>
<p>or (saw it on the imagenet code for compute topk accuracy I believe)</p>
<pre><code class="lang-auto">x.view(1,-1)
</code></pre>
<p>It just seems that .view is weird since it only arranges the tensor to the right view but starts from the beginning and enumerates “in order” (ref: <a href="https://discuss.pytorch.org/t/tensor-view-is-misleading/20553/2" class="inline-onebox">Tensor.view is misleading - #2 by rasbt</a>).</p>

username: tom
cooked: <p>So <code>x.view(batch_size, -1)</code> is an example of the broader pattern of <em>combining dimensions</em>. There may be other applications, for example when you want a matrix-multiplication-like operation where the three matrix “axes” you have in vanilla matrix multiplications have different shapes. Fun fact: the very versatile einsum reduces to batch matrix multiplications. It does so using permute and view/reshape.</p>
<p>The other use, dual if you want, is <em>splitting dimensions</em>. In addition to this happing as the backward of <em>combining dimensions</em> this can also be useful in things like very simple downscaling (I sometimes use <code>x.view(batch_size, c, h // 2, 2, w // 2, 2)</code> to then do something with the dimensions of size 2 - if you take the max over both, you get a maxpool, but you could also do a logsumexp-pool or somesuch – your imagination is the limit!</p>
<p>Best regards</p>
<p>Thomas</p>

username: leckofunny
cooked: <p>Thanks for the elaboration on this matter. I was struggling with this issue for 6 months in the end!</p>
<p>There is still another use-case left. What if your training data is flat in the first place (batch_size, data)?<br>
This scenario could come from a visual encoder. Its result could be fed into an LSTM and therefore needs to be shaped accordingly.</p>
<p>A reshape to (num sequences, sequence length, data) is just fine, but the other way around to a shape of (sequence length, num sequences, data) is causing the troubles that you describe.</p>
<p>So what would be good way to reshape the tensor to sequence length first in this case?</p>

username: vdw
cooked: <p>It depends what <code>data</code> in your batch exactly refers to. Since you want to push it through I LSTM, I assume that <code>data</code> contains some sequence somewhere. If so, it really depends on the content/structure of <code>data</code>.</p>
<p>Or do you have a sequence of, say, 32 frames of a video clip you first encode and then push these 32 frames as a sequence through an LSTM. In this case, <code>batch_size</code> for the visual encoding becomes your <code>seq_len</code> for the LSTM with a <code>batch_size=1</code>. So you don’t need to reshape anything but “unsqueeze” your batch dimension. Say <code>h</code> is your batch after visual encoding with shape (32, data). Then you can do:</p>
<pre><code>h = h.unsqueeze(0)
</code></pre>
<p>resulting in shape of (1, 32, data) which works just fine as input for your LSTM.</p>

username: tom
cooked: <aside class="quote no-group" data-username="leckofunny" data-post="4" data-topic="75524">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leckofunny/48/25059_2.png" class="avatar"> leckofunny:</div>
<blockquote>
<p>So what would be good way to reshape the tensor to sequence length first in this case?</p>
</blockquote>
</aside>
<p>The conversion between sequence first and batch first is a usually a <code>.permute(1, 0, 2)</code> (could also be written as transpose, but to me permute better captures the intent).</p>

username: leckofunny
cooked: <p>My particular use-case originates from reinforcement learning with pixel observation. Lets assume that I sample 16 agent-environment-interactions and use a sequence length of 8.</p>
<p>I’d feed a tensor of shape (16, 3, 84, 84) to a stack of convolutional layers. After that the data needs to be reshaped into sequences of length 8. Doing so by using sequence first will mess up the data. (reshape(8, 2, -1)). Using the batch size as first dimension works well concerning the reshape.</p>

username: zhengyang_xu
cooked: <aside class="quote no-group" data-username="tom" data-post="6" data-topic="75524">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>.permute(1, 0, 2)</p>
</blockquote>
</aside>
<p>hello, your answer is very useful for me who are a beginner in AI.But in cv, we always need to flatten the matrix([M,C,H,W]) to a row vector([M,C<em>H</em>W]). so how can I flatten this matrix without in-place option. It brother me long time, I’m so grateful if you can reply</p>

username: Md_Zahidul_Islam
cooked: <p>Hi <a class="mention" href="/u/vdw">@vdw</a> thank you for your answer and referring this post in my post. I was wondering if the data is a 128*128 matrix will this approach still work? Or do I need to flatten the matrix? Thank you so much.</p>

username: vdw
cooked: <p>I’m not sure if I fully understand your questions.</p>
<p>The issue of reshaping/transforming the matrix is not a issue of the shape of the matrix or tensor, but how the matrix or tensors is actually organized in memory. For example a 2d matrix is not exactly stored as a 2d grid in memory but as an array of length m*n (assuming m and n are the dimensions of the matrix). Which element in the array represent a specific cell in the matrix depends if the matrix is stored <a href="https://en.wikipedia.org/wiki/Row-_and_column-major_order" rel="noopener nofollow ugc">row- and column-major order</a>.</p>
<p>In short, when all you want to do is to transpose (i.e., “rotate” a tensor by swapping dimensions around), you should never use <code>reshape()</code> or <code>view()</code>. Under certain conditions it might work, but no generally.</p>
<p>However, I’m not sure what your task is and what you want to do with your 128x128 matrix.</p>

username: Md_Zahidul_Islam
cooked: <aside class="quote no-group" data-username="vdw" data-post="5" data-topic="75524">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vdw/48/10074_2.png" class="avatar"> vdw:</div>
<blockquote>
<p>Or do you have a sequence of, say, 32 frames of a video clip you first encode and then push these 32 frames as a sequence through an LSTM. In this case, <code>batch_size</code> for the visual encoding becomes your <code>seq_len</code> for the LSTM with a <code>batch_size=1</code>. So you don’t need to reshape anything but “unsqueeze” your batch dimension. Say <code>h</code> is your batch after visual encoding with shape (32, data). Then you can do:</p>
<pre><code class="lang-auto">h = h.unsqueeze(0)
</code></pre>
<p>resulting in shape of (1, 32, data) which works just fine as input for your LSTM.</p>
</blockquote>
</aside>
<p>Hi <a class="mention" href="/u/vdw">@vdw</a> referring to this question, let my input sequence is 24 images and my input is organised as a sequence 32<em>24</em>128<em>128 where 32 is the bathsize. 24 is the sequence length and 128</em>128 is an image. I want to use LSTM to encode the temporal dependence in the data. How should I format the LSTM input?</p>

username: vdw
cooked: <p>Ok, I got you now. Yes, if you have something like <code>(batch_size, seq_len, width, height)</code> you need to transform into <code>(batch_size, seq_len, width*height)</code> to serve as valid input for the <code>nn.LSTM</code>. This means you have to flatten the image, i.e., flatten your last 2 dimensions. The easiest way should be to do:</p>
<pre><code class="lang-auto">h = h.view(batch_size, seq_len, -1)
</code></pre>
<p>But again, this assume that <code>h.shape</code> was initially <code>(batch_size, seq_len, width, height)</code>.</p>
<p>If for some reasons <code>h</code> starts out with <code>(seq_len, batch_size, width, height)</code> you need to the additional step:</p>
<pre><code class="lang-auto">h = h.transpose(1,0) # or: h = h.permute(1,0,2)
h = h.view(batch_size, seq_len, -1)
</code></pre>

username: Md_Zahidul_Islam
cooked: <p>Thank you so much <a class="mention" href="/u/vdw">@vdw</a></p>