Do linear layer after GRU saved the sequence output order?

laro: <p>I’m dealing with the following senario:</p>
<ul>
<li>
<p>My input has the shape of: <code>[batch_size, input_sequence_length, input_features]</code><br>
where:</p>
<p>input_sequence_length = 10</p>
<p>input_features = 3</p>
</li>
<li>
<p>My output has the shape of: <code>[batch_size, output_sequence_length]</code><br>
where:</p>
<p>output_sequence_length = 5</p>
</li>
</ul>
<p>i.e: for each time slot of 10 units (each slot with 3 features) I need to predict the next 5 slots values.</p>
<p>I built the following model:</p>
<pre><code>import torch
import torch.nn as nn
import torchinfo

class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        
        self.GRU = nn.GRU(input_size=3, hidden_size=32, num_layers=2, batch_first=True)
        self.fc  = nn.Linear(32, 5)
        
    def forward(self, input_series):
        
        output, h = self.GRU(input_series)                
        output    = output[:,  -1, :]       # get last state                
        output    = self.fc(output) 
        output    = output.view(-1, 5, 1)   # reorginize output        
        return output
    
torchinfo.summary(MyModel(), (512, 10, 3))  



==========================================================================================
Layer (type:depth-idx)                   Output Shape              Param #
==========================================================================================
MyModel                                  [512, 5, 1]               --
├─GRU: 1-1                               [512, 10, 32]             9,888
├─Linear: 1-2                            [512, 5]                  165
==========================================================================================
</code></pre>
<p>I’m getting good results (very small <code>MSE</code> loss, and the predictions looks good),</p>
<p>but I’m not sure if the model output (5 sequence values) are really ordered by the model ?<br>
i.e the second output based on the first output and the third output based on the second output …</p>
<p>I know that the <code>GRU</code> output based on the learned sequence history.<br>
But I’m also used linear layer, so is the output (after the linear layer) still sorted by time ?</p>

ptrblck: <p>No, I don’t think the output is “sorted by time” since you have explicitly removed the time axis by indexing the last time step:</p>
<pre><code class="lang-python">output    = output[:,  -1, :]       # get last state  
</code></pre>
<p><code>output</code> now only contains the feature tensor of the last time step which is then passed to the linear layer.<br>
Due to the recurrent nature of <code>GRU</code> the last time step could contain information from the previous steps, but the actual tensor does not have a temporal dependency anymore.</p>

laro: <p>So although I’m getting good results, it seems it works with no sense and it better to chage the model ?</p>

ptrblck: <p>No, I don’t think the model does anything wrong as the last state could still contain enough information from the previous time steps.</p>

laro: <p>Can I count on the model outputs (tensor of shape 5) to be ordered ?<br>
I.e the output[4]  based on output[3] and output[3] based on output[2]… output[1] based on output[0] ?</p>

ptrblck: <p>No, that’s not how a linear layer works.<br>
Right now, you are using the 32 features of the last time step and pass them to the linear layer. The linear layer will apply a weight matrix (and bias) to this input and will “map” the 32 features to 5. The 5 output features are not ordered or sorted in any way but represent what you’ve defined as the model output, e.g. class logits.</p>