Outputs of LSTM Model all Very Similar or the Same

username: Michael-Equi
cooked: <p>I am working on a project that requires inputting an image and a sequence of actions and predicting the future positions of the robot as well as any collisions. I am currently having an issue with the model producing only a single set output no matter the input provided (from training set, from testing set, or random). Before the model is even trained it seems to have only one small set of outputs and only slowly diverges from its initial set of outputs during training until it has a new fixed set of outputs that again does not change depending on the input. Because of this, I do not think that it is a training problem or an over-fitting problem. I am not able to figure out what is going on so any help is greatly appreciated. Below is my model code, some test code, and some of the outputs/labels.</p>
<p>Model</p>
<pre><code class="lang-auto">class NavNet(nn.Module):
    def __init__(self, input_shape=(3,180,320), hidden_dims=128):
        super(NavNet, self).__init__()
        
        # Constants for applying activations and loss functions only at specific locations in output tensor        
        collision_mask = torch.tensor([0,0,0,1]).float()
        self.register_buffer('collision_mask_const', collision_mask)
        bumpiness_mask = torch.tensor([0,0,1,0]).float()
        self.register_buffer('bumpiness_mask_const', bumpiness_mask)
        position_mask = torch.tensor([1,1,0,0]).float()
        self.register_buffer('position_mask_const', position_mask)
        
        self.input_conv = nn.Sequential(
            nn.Conv2d(3,32, kernel_size=5,stride=2, padding=0),
            nn.ReLU(),
            nn.Conv2d(32,32,5,2),
            nn.ReLU(),
            nn.Conv2d(32,64,3,2),
            nn.ReLU(),
            nn.Flatten()
        )
        
        fc_dims = self.input_conv(torch.zeros(input_shape).unsqueeze(0)).shape[1]
        
        self.input_fc = nn.Sequential(
            nn.Linear(fc_dims,hidden_dims*3),
            nn.ReLU(),
            nn.Linear(hidden_dims*3, hidden_dims*2),
            nn.ReLU(),
            nn.Linear(hidden_dims*2, hidden_dims),
            nn.ReLU(),
            nn.Linear(hidden_dims, hidden_dims)
        )
                
        self.action_fc = nn.Sequential(
            nn.Linear(2, 16),
            nn.ReLU(),
            nn.Linear(16, 16)
        )
        
        self.lstm = nn.LSTM(16, hidden_dims, batch_first=True) # 16 input size, 128 hidden size

        
        self.output_fc = nn.Sequential(
            nn.Linear(hidden_dims, 32),
            nn.ReLU(),
            nn.Linear(32, 4) # collision, bumpiness, x, y
        )
        
    # seq is the action sequence to be evaluated [[action 1],[action 2],[action 3]]
    def forward(self,x,seq):
        
        x = self.input_conv(x)
        x = self.input_fc(x).unsqueeze(0)
        
        actions = self.action_fc(seq)
                
        g, hidden = self.lstm(actions, (x, x))
                
        output = self.output_fc(g)
        
        # apply sigmoid activation to collision probability
        output = torch.sigmoid(output)*self.collision_mask_const + output*(self.bumpiness_mask_const + self.position_mask_const)
        
        return output
    
    # custom loss function https://github.com/gkahn13/badgr/blob/master/configs/bumpy_collision_position.py
    def navnet_loss(self, model_output, target): 

        loss  = F.binary_cross_entropy(model_output*self.collision_mask_const, target*self.collision_mask_const)
        loss += F.mse_loss(model_output*self.bumpiness_mask_const, target*self.bumpiness_mask_const)
        loss += F.mse_loss(model_output*self.position_mask_const, target*self.position_mask_const)

        return loss
</code></pre>
<p>Simple Test Function</p>
<pre><code class="lang-auto">test_batch_size = 2
action_seq_length = 7 

test_input = torch.rand((test_batch_size, 3,180,320))
print("Input Image : ", test_input.shape)

test_actions = torch.rand((test_batch_size, action_seq_length,2))
print("Input Actions : ", test_actions.shape)

output = navnet(test_input, test_actions)
print("Output : ", output.shape)
print(output)

torchviz.make_dot(output).render("navnet", format="png")
</code></pre>
<p>Output  of test</p>
<pre><code class="lang-auto">Input Image :  torch.Size([2, 3, 180, 320])
Input Actions :  torch.Size([2, 7, 2])
Output :  torch.Size([2, 7, 4])
tensor([[[ 0.0184, -0.0096,  0.0313,  0.4624],
         [ 0.0205, -0.0130,  0.0311,  0.4639],
         [ 0.0211, -0.0146,  0.0303,  0.4646],
         [ 0.0190, -0.0147,  0.0314,  0.4652],
         [ 0.0187, -0.0154,  0.0307,  0.4652],
         [ 0.0201, -0.0162,  0.0316,  0.4653],
         [ 0.0174, -0.0156,  0.0317,  0.4654]],

        [[ 0.0187, -0.0098,  0.0298,  0.4623],
         [ 0.0202, -0.0129,  0.0289,  0.4637],
         [ 0.0202, -0.0142,  0.0282,  0.4645],
         [ 0.0194, -0.0147,  0.0285,  0.4649],
         [ 0.0189, -0.0151,  0.0279,  0.4650],
         [ 0.0189, -0.0157,  0.0311,  0.4652],
         [ 0.0208, -0.0164,  0.0312,  0.4652]]], grad_fn=&lt;AddBackward0&gt;)
</code></pre>
<p>After a short training period the output ends up something like the following when visualized. On the top are the labels and the bottom is the output of the network. I have double and triple checked to make sure that everything in way of the data is being processed and passed in correctly. Before any training, the output has a similar fixedness but is usually just more bunched up around a particular point.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/2/22e7cf42db3cb84503ecd68079f4d8a8cb41982a.png" data-download-href="https://discuss.pytorch.org/uploads/default/22e7cf42db3cb84503ecd68079f4d8a8cb41982a" title="Screenshot from 2020-06-05 22-46-20"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/22e7cf42db3cb84503ecd68079f4d8a8cb41982a_2_690x163.png" alt="Screenshot from 2020-06-05 22-46-20" data-base62-sha1="4YMSNkSYNddaUmgtyOKiqJgSib8" width="690" height="163" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/22e7cf42db3cb84503ecd68079f4d8a8cb41982a_2_690x163.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/22e7cf42db3cb84503ecd68079f4d8a8cb41982a_2_1035x244.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/22e7cf42db3cb84503ecd68079f4d8a8cb41982a_2_1380x326.png 2x" data-dominant-color="F3F6F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-06-05 22-46-20</span><span class="informations">1594×377 90.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dc8893583c749cbbe3ea0476db9844600fd8e8a4.png" data-download-href="https://discuss.pytorch.org/uploads/default/dc8893583c749cbbe3ea0476db9844600fd8e8a4" title="Screenshot from 2020-06-05 22-46-41"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dc8893583c749cbbe3ea0476db9844600fd8e8a4_2_690x168.png" alt="Screenshot from 2020-06-05 22-46-41" data-base62-sha1="vsVECXKZQKIN7ZKKwMP2p38bDFi" width="690" height="168" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dc8893583c749cbbe3ea0476db9844600fd8e8a4_2_690x168.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dc8893583c749cbbe3ea0476db9844600fd8e8a4_2_1035x252.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dc8893583c749cbbe3ea0476db9844600fd8e8a4_2_1380x336.png 2x" data-dominant-color="F6F8F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-06-05 22-46-41</span><span class="informations">1596×389 87.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
edit 1 : Here is the networks output from different inputs without any training<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9de76c65e24df478497f9902b1ffbf0a68f778d4.png" data-download-href="https://discuss.pytorch.org/uploads/default/9de76c65e24df478497f9902b1ffbf0a68f778d4" title="Screenshot from 2020-06-06 10-57-05"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9de76c65e24df478497f9902b1ffbf0a68f778d4_2_690x167.png" alt="Screenshot from 2020-06-06 10-57-05" data-base62-sha1="mwSMGAfunHhUWzMgnys4ySKTOxC" width="690" height="167" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9de76c65e24df478497f9902b1ffbf0a68f778d4_2_690x167.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9de76c65e24df478497f9902b1ffbf0a68f778d4_2_1035x250.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9de76c65e24df478497f9902b1ffbf0a68f778d4_2_1380x334.png 2x" data-dominant-color="F8F8F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-06-06 10-57-05</span><span class="informations">1573×381 70.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: viet98lx
cooked: <p>I have same problem! Did you solved?</p>

username: sigma_x
cooked: <p>Try initializing the weights in the model using</p>
<blockquote>
<p>kaiming_normal_</p>
</blockquote>
<p>instead of Normal. Worked for me.</p>