When and why do we use Contiguous()?

username: Shisho_Sama
cooked: <p>I have seen codes such as this:</p>
<pre><code class="lang-auto">class lstm_char(nn.Module):
    def __init__(self, unique_input_tokens, inputsize, hiddensize, outputsize, num_layer, dropout_prob=0.2):
        super().__init__()

        self.hiddensize = hiddensize
        self.num_layer = num_layer
        self.int2char = dict(enumerate(unique_input_tokens))
        self.char2int = {char:int for (int, char) in int2char.items()}

        self.lstm = nn.LSTM(input_size=inputsize,
                             hidden_size = hiddensize,
                             num_layers = num_layer,
                             batch_first=True, #since we are using batches of data we set this to true!
                             dropout=dropout_prob)

        self.dropout = nn.Dropout2d(dropout_prob)
        self.fc = nn.Linear(hiddensize, outputsize)
    
    def forward(self, x, hidden):
        """
        hidden contains hiddenstate and cellstate
        """
        output, hidden = self.lstm(x, hidden)

        output = self.dropout(output)
        output = output.contiguous().view(-1, self.hiddensize)
        output = self.fc(output)
        return output, hidden
</code></pre>
<p>But why would someone want to that?! What would they gain by making for example, a tensor of shape <code>(3,5, 83)</code> which is (batch, sequences, features) to a tensor of shape <code>(15, 83)</code> ?</p>
<p>Wouldn’t we want the output be <code>(3,5,83)</code>? As we have 3 input samples, we need to have 3 outputs as well!?<br>
Can someone please explain to me what the coder had in mind by using <code>contiguous()</code> prior to using <code>fc</code> layer?<br>
Thank you very much in advance</p>

username: jerinphilip
cooked: <p>If I remember correctly, these typically used to happen with old codes. Earlier <code>nn.Linear</code> perhaps didn’t support multidimensional inputs <code>(*,*, H)</code>. So a hack was to convert <code>(T, B, H)</code> to <code>(TxB, H</code>) to pass through the <code>Linear</code> layer. It doesn’t make a difference to <code>Linear</code> so long as the final dimension was <code>(*, H)</code> in the computation and backprop as due to PyTorch’s dynamic graph. Frameworks like Keras had a <code>TimeDistributedDense</code> for this case. In this case, after linear operations, you could get back the <code>(T, B, H)</code> with another <code>view(...)</code>.</p>
<p><a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Linear" rel="nofollow noopener">nn.Linear docs</a> indicate it can take multi-dimensional inputs now and handle them. So the <code>contiguous().view(...)</code> is not  required now. Your example however may have code later on which works with the dimensions output with this model.</p>
<p>And as for <code>contiguous(..)</code>, it’s typically called because most cases <code>view(...)</code> would throw an error if <code>contiguous(..)</code> isn’t called before. Normally some changes like <code>view(..)</code>, <code>transpose(...)</code> or <code>permute(..)</code> would just change the metadata (being lazy) and not the underlying storage. This create issues with parallel computations. Inorder to consolidate it into a contiguous memory as expected by other ops, <code>contiguous()</code> is called.</p>

username: Shisho_Sama
cooked: <p>Thank you very much sir <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
It all makes sense now, its greatly appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>