Forward propagation returns different logits on same samples

username: kkonstantinidis
cooked: <p>Consider the following LeNet model for MNIST</p>
<pre><code class="lang-auto">import torch
from torch import nn
import torch.nn.functional as F

class LeNet(nn.Module):
    def __init__(self):
        super(LeNet, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5, 1)
        self.conv2 = nn.Conv2d(20, 50, 5, 1)
        self.fc1 = nn.Linear(4*4*50, 500)
        self.fc2 = nn.Linear(500, 10)
        self.ceriation = nn.CrossEntropyLoss()
    def forward(self, x):
        x = self.conv1(x)
        x = F.max_pool2d(x, 2, 2)
        x = F.relu(x)
        x = self.conv2(x)
        x = F.max_pool2d(x, 2, 2)
        x = F.relu(x)
        x = x.view(-1, 4*4*50)
        x = self.fc1(x)
        x = self.fc2(x)
        return x
</code></pre>
<p>Now, I use this model to do a single forward step on a batch of samples like</p>
<pre><code class="lang-auto">network=LeNet()
optimizer = torch.optim.SGD(self.network.parameters(), lr=0.001, momentum=0.9)
device = torch.device("cpu")
network.to(device)
network.train()
optimizer.zero_grad()
# X_batch= ... some batch of 50 samples pulled from a train_loader defined as
# torch.manual_seed(42)
# training_set = datasets.MNIST('./mnist_data', train=True, download=False, 
#                               transform=transforms.Compose([
#                                   transforms.ToTensor(),
#                                   transforms.Normalize((0.1307,), (0.3081,))]))
# train_loader = torch.utils.data.DataLoader(training_set, 
#                                            batch_size=50, 
#                                            shuffle=False)
logits = network(X_batch)
</code></pre>
<p>Note that <code>shuffle=False</code> and <code>download=False</code> for the loader since the data set is already downloaded and I don’t want to shuffle. My problem is that if I run this code twice I will get different values for <code>logits</code> and I don’t understand why since everything else seems to be unchanged. For an extra check, I also extract <code>X_batch</code> to a <code>numpy</code> array and verify that the batch of samples is exactly the same as of previous execution. I do this check with <code>numpy.array_equal()</code> function.</p>
<p>I really can’t figure out what I am missing here unless there are precision issues.</p>

username: KFrank
cooked: <p>Hello Konstantinos!</p>
<aside class="quote no-group quote-modified" data-username="kkonstantinidis" data-post="1" data-topic="88827" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/74df32/48.png" class="avatar"> kkonstantinidis:</div>
<blockquote>
<pre><code class="lang-auto">class LeNet(nn.Module):
    def __init__(self):
        super(LeNet, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5, 1)
        self.conv2 = nn.Conv2d(20, 50, 5, 1)
        self.fc1 = nn.Linear(4*4*50, 500)
...
</code></pre>
<pre><code class="lang-auto">network=LeNet()
...
logits = network(X_batch)
</code></pre>
<p>My problem is that if I run this code twice I will get different values for <code>logits</code></p>
</blockquote>
</aside>
<p>When you construct your <code>LeNet</code> (two separate times), you<br>
construct its layers, e.g.,<br>
<code>self.fc1 = nn.Linear(4*4*50, 500)</code>, (two separate times).</p>
<p>By default, pytorch randomly initializes the layer weights, so you<br>
get two differing sets of weights, and hence differing output<br>
values for <code>logits</code>.</p>
<p>You can check this by printing out some example weights from<br>
<code>network</code> <em>each individual time</em> you initialize it.</p>
<p>Best.</p>
<p>K. Frank</p>

username: harsha_g
cooked: <p>You can get exact results if you fix the random seed. More on that here: <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/randomness.html</a></p>

username: kkonstantinidis
cooked: <p>Thanks a lot for your answer, this helped me narrow down the problem.</p>