Infinite validation loss, even without shuffling data

username: Amith_Adiraju
cooked: <p>I’m developing a <code>LSTM AutoEncoder</code> to encode text data. I’ve trained <code>Flair's DocumentRNNEmbeddings</code> to embed sequence of sentences and using the saved model.</p>
<p>Each batch of my training data is of the shape <code>[12,5,2048]</code>; i.e; each sample contains <code>5 sentences</code> having dimensions <code>2048</code>.</p>
<p><strong>PROBLEM</strong></p>
<p>I’ve tried both <code>BCEWithLogitLoss</code> and   <code>MSELoss</code>   along with <code>Adam Optimizer</code> ; in both the cases my training loss was fairly good <code>0.31 on average with BCE</code> and <code>0.011 on average with MSE</code> but my validation loss mostly ends up being <code>inf</code> or in some rare cases <code>really huge: 8.025489465e+32</code>. I’m just trying to validate my approach of Auto Encoder, so my entire training data contains only <code>335 Sentences</code>  (6 batches of data ) and test data <code>60 Sentences</code> (only two batches of data) ; I’m not sure if that’s what is causing the issue or if there’s something wrong with the way I’m performing <code>forward pass</code>.</p>
<p><strong>Important parts of network initialization:</strong></p>
<pre><code>    self.lstm_enc = nn.LSTM(self.input_dim, self.hidden_dim, 
                            self.num_layers, batch_first = True,
                            dropout = self.dropout_rate,
                            bidirectional = self.bidirectional)
            
        self.lstm_dec = nn.LSTM(self.hidden_dim*2, self.output_dim // 2, 
                        self.num_layers, batch_first = True,
                        dropout = self.dropout_rate,
                        bidirectional = self.bidirectional)

    self.hidden_rep = get_hidden_rep
    self.activation = nn.ELU()
    
    self.hidden_enc_weights = self._init_hidden_enc_weights()
            
    self.hidden_dec_weights = self._init_hidden_dec_weights()   
    
   # self.loss_func = nn.BCEWithLogitsLoss()

    self.loss_func  = nn.MSELoss(reduction = 'mean')
</code></pre>
<p><strong>Here’s how my forward pass looks like:</strong></p>
<pre><code>        encoder_output, self.hidden_enc_weights = self.lstm_enc(X_batch, self.hidden_enc_weights)
        self.hidden_enc_weights[0].clamp(min = 1e-1);self.hidden_enc_weights[1].clamp(min = 1e-1) 
        self.hidden_enc_weights[0].detach_(); self.hidden_enc_weights[1].detach_()
        
        
        encoder_output = self.activation(encoder_output)
        
        
        if self.step_through_linear:
            encoder_output = self.step_linear(encoder_output)
            encoder_output = self.activation(encoder_output) 
                
        decoder_output, self.hidden_dec_weights = self.lstm_dec(encoder_output, self.hidden_dec_weights)

        self.hidden_dec_weights[0].clamp(min = 0);self.hidden_dec_weights[1].clamp(min = 0) 
        self.hidden_dec_weights[0].detach_(); self.hidden_dec_weights[1].detach_()
        
        
        if self.hidden_rep:
            return encoder_output
        
        else:
            
            decoder_output = self.activation( decoder_output )
            
            #return self.loss_func(decoder_output, torch.flip(X_batch, [0])) # BCEWithLogitLoss
            return self.loss_func(X_batch, decoder_output) # MSELOSS
</code></pre>
<p>I’ve applied some new techniques, such as <code>reversing the input data</code> while comparing the loss, and also other techniques such as initializing with <code>xavier weights</code> and clamping weights of encoder and decoder weights. But, I don’t think they’d affect my loss in a negative way.</p>
<p><strong>WHAT I’VE TRIED TO COUNTER</strong></p>
<p>I did find similar issues such as : <a href="https://discuss.pytorch.org/t/infinitive-or-nan-batch-loss-encountered-when-shuffling-the-training-data/1970">Similar Issue Infinite Loss</a> ; I implemented some checks in my code from suggested answers, such as checking for <code>inf</code> values in my input and validation values, ex:</p>
<pre><code class="lang-auto">torch.isfinite(eval_batch).all().item()
</code></pre>
<p>The return value always suggested that there were no <code>inf</code> values in my data in both the sets. So, I’m not able to figure out what’s causing the issue. Any help is much appreciated.</p>
<p>P.S: I moved to pytorch recently from Keras.<br>
TIA !</p>

username: ptrblck
cooked: <p>Please don’t tag specific people, as this might discourage others to post an answer.</p>