Model's output during training is totally different than when loading saved modeln loading weights

username: Alexandros_Drymoniti
cooked: <p>Reading <a href="https://discuss.pytorch.org/t/different-output-from-the-same-trained-model/191484">this</a> topic, I see that I face the same problem. I do have a dropout layer, so my issue is most likely related to the topic I posted. How can I solve it thought? In the topic, a solution is highlighted, but how to implement it is not mentioned in any of the posts.</p>

username: ptrblck
cooked: <p>From the linked post:</p>
<blockquote>
<p>Did you disable any randomness via <code>model.eval()</code>?</p>
</blockquote>
<p>so call <code>model.eval()</code> to disable dropout layers.</p>

username: Alexandros_Drymoniti
cooked: <p>I am doing this, but it’s not working. Here’s the code I have for the network:</p>
<pre><code class="lang-auto">class RNNModel(nn.Module):
    def __init__(self, vocab_size, embedding_size, hidden_dim, n_layers, drop_rate=0.2):
        
        super(RNNModel, self).__init__()

        # Defining some parameters
        self.hidden_dim = hidden_dim
        self.embedding_size = embedding_size
        self.n_layers = n_layers
        self.vocab_size = vocab_size
        self.drop_rate = drop_rate
        self.char2int = None
        self.int2char = None


        #Defining the layers
        # Define the encoder as an Embedding layer
        #self.encoder = nn.Embedding(vocab_size, embedding_size)
            
        # Dropout layer
        self.dropout = nn.Dropout(drop_rate)
        # RNN Layer
        self.rnn = nn.LSTM(embedding_size, hidden_dim, n_layers, dropout=drop_rate, batch_first = True)
        # Fully connected layer
        self.decoder = nn.Linear(hidden_dim, vocab_size)
    
    def forward(self, x, state):
        
        # input shape: [batch_size, seq_len, embedding_size]
        # Apply the embedding layer and dropout
        #embed_seq = self.dropout(self.encoder(x))
            
        #print('Input RNN shape: ', embed_seq.shape)
        # shape: [batch_size, seq_len, embedding_size]
        rnn_out, state = self.rnn(x, state)
        #print('Out RNN shape: ', rnn_out.shape)
        # rnn_out shape: [batch_size, seq_len, rnn_size]
        # hidden shape: [2, num_layers, batch_size, rnn_size]
        rnn_out = self.dropout(rnn_out)

        # shape: [seq_len, batch_size, rnn_size]
        # Stack up LSTM outputs using view
        # you may need to use contiguous to reshape the output
        rnn_out = rnn_out.contiguous().view(-1, self.hidden_dim)

        logits = self.decoder(rnn_out)
        # output shape: [seq_len * batch_size, vocab_size]
        #print('Output model shape: ', logits.shape)
        return logits, state
    
    def init_state(self, device, batch_size=1):
        """
        initialises rnn states.
        """
        #return (Variable(torch.zeros(self.n_layers, batch_size, self.hidden_dim)),
        #        Variable(torch.zeros(self.n_layers, batch_size, self.hidden_dim)))
        return (torch.zeros(self.n_layers, batch_size, self.hidden_dim).to(device),
                torch.zeros(self.n_layers, batch_size, self.hidden_dim).to(device))

    def predict(self, input):
        # input shape: [seq_len, batch_size]
        logits, hidden = self.forward(input)
        # logits shape: [seq_len * batch_size, vocab_size]
        # hidden shape: [2, num_layers, batch_size, rnn_size]
        probs = F.softmax(logits)
        # shape: [seq_len * batch_size, vocab_size]
        probs = probs.view(input.size(0), input.size(1), probs.size(1))
        # output shape: [seq_len, batch_size, vocab_size]
        return probs, hidden
</code></pre>
<p>and I’m creating a model with this line</p>
<pre><code class="lang-auto">model = RNNModel(dict_size,embedding_size, hidden_dim, n_layers, drop_rate=0.0)
model.eval()
</code></pre>
<p>I have copied the model code from some project, but can’t remember which one it was and can’t find it.<br>
Apart from a dropout kwarg in the LSTM class, this network also uses a Dropout layer before that. Could that be the issue? Reading its <a href="https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html" rel="noopener nofollow ugc">documentation</a>, I see that this layer randomly zeros outputs during training. What happens when not in training mode? How do I go about that?</p>