Why the results is different when the init function in nn.modulehas been modifed?

Anikily: <p>I defined a neural network with the <strong>init</strong> and <strong>forward</strong> function,</p>
<pre><code class="lang-auto">class Model(nn.Module)
    def __init__():
        self.sub1 = Module1
        self.sub2 = Module2
</code></pre>
<p>when i add some layers such as Conv2d into the self.sub1 or self.sub2,  i found the performance after one epoch is different!<br>
I can’t figure out the reason above, could you help me?</p>

AlphaBetaGamma96: <p>Can you share a minimal reproducible example and explain what this “difference” in behavior is?</p>

ptrblck: <p>Additionally to what <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a> already asked:<br>
if you are concerned about the reproducibility of your code and determinism, note that a layer initialization will call into the pseudorandom number generator and will thus change all following calls (even if the layer is not used in the forward).</p>

Anikily: <p>Hi, thanks for you reply, i have a simple code here:</p>
<pre><code class="lang-auto">import torch
import torchvision
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import random
import os

class conv_module(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)

    def forward(self,x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = torch.flatten(x, 1) # flatten all dimensions except batch
        return x
class mlp_module(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = conv_module()
        self.mlp = mlp_module()

    def forward(self, x):
        x = self.conv(x)
        x = self.mlp(x)
        return x
# seeds
random.seed(int(1024))
os.environ['PYTHONHASHSEED'] = str(1024)
np.random.seed(int(1024))
torch.manual_seed(int(1024))
torch.cuda.manual_seed(int(1024))
torch.cuda.manual_seed_all(int(1024))
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.enabled = False

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

batch_size = 4

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size,
                                          shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size,
                                         shuffle=False, num_workers=2)

classes = ('plane', 'car', 'bird', 'cat',
           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

# get some random training images
dataiter = iter(trainloader)

net = Net().cuda()

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs.cuda())
        loss = criterion(outputs, labels.cuda())
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print(f'[{epoch + 1}, {i + 1:5d}] loss: {running_loss / 2000:.3f}')
            running_loss = 0.0

print('Finished Training')

############# Test ########################
correct = 0
total = 0
# since we're not training, we don't need to calculate the gradients for our outputs
with torch.no_grad():
    for data in testloader:
        images, labels = data
        # calculate outputs by running images through the network
        outputs = net(images.cuda())
        # the class with the highest energy is what we choose as prediction
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels.cuda()).sum().item()

print(f'Accuracy of the network on the 10000 test images: {100 * correct // total} %')

correct_pred = {classname: 0 for classname in classes}
total_pred = {classname: 0 for classname in classes}

# again no gradients needed
with torch.no_grad():
    for data in testloader:
        images, labels = data
        outputs = net(images.cuda())
        _, predictions = torch.max(outputs, 1)
        # collect the correct predictions for each class
        for label, prediction in zip(labels.cuda(), predictions):
            if label == prediction:
                correct_pred[classes[label]] += 1
            total_pred[classes[label]] += 1


# print accuracy for each class
for classname, correct_count in correct_pred.items():
    accuracy = 100 * float(correct_count) / total_pred[classname]
    print(f'Accuracy for class: {classname:5s} is {accuracy:.1f} %')

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
print(device)
</code></pre>
<p>this code modified from the pytorch tutorials.<br>
And when i add a conv2d layer into the conv_module such as</p>
<pre><code class="lang-auto">class conv_module(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.conv_test = nn.Conv2d(3, 3, 1)

    def forward(self,x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = torch.flatten(x, 1) # flatten all dimensions except batch
        return x
</code></pre>
<p>The performance is different with original model.</p>

Anikily: <p>Thanks for your reply.<br>
I have propose a simple example for my case as in above, and there is no initialization at all except the default func in the nn.Module.</p>

ptrblck: <aside class="quote no-group" data-username="Anikily" data-post="5" data-topic="148705">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/a5b964/48.png" class="avatar"> Anikily:</div>
<blockquote>
<p>and there is no initialization at all except the default func in the nn.Module.</p>
</blockquote>
</aside>
<p>The “default” initialization will call into <code>.reset_parameters()</code> and then also into the pseudorandom number generator.<br>
As already described, you are changing the PRNG and thus cannot expect to get the same random values without re-seeding.</p>

Anikily: <p>Hi,<br>
Thanks for your reply, which helps me a lot.<br>
But how can i set the seed in PRNG to ensure the reproducibility of my code?<br>
I have set the numpy seed, torch seed and cuda seed in the example, all them seems does not wok.</p>
<p>Best,<br>
Yu</p>

ptrblck: <p>I think the seeds do work, but since you are changing the order of calls into the PRNG you cannot expect to see the same results.<br>
If you want to ignore the additional calls to the newly initialized layers, you could try to re-seed the code.</p>

Anikily: <p>Sorry, could you specify the meaning of “re-seed”?<br>
Do you mean set the seeds within the network init function?<br>
Looking forward your reply.</p>

ptrblck: <p>I mean you would have to re-seed the script at the point where you can guarantee the same order or calls into the PRNG.</p>
<p>Seeding the PRNG guarantees that the sequence of randomly generated numbers will be equal between runs for the same order of calls into the PRNG.<br>
In your case you are changing the order of calls into the PRNG by initializing additional layers in your script, but expect to see the same random numbers, which is a wrong expectation.</p>
<p>Here is a small example:</p>
<pre><code class="lang-python"># initial seed
torch.manual_seed(seed1)

# init layers in both scripts
layer1 = nn.Linear(...) # will have the same parameters in both scripts as you've seeded the code
layer2 = nn.Linear(...)

# init layer3 only in second script
layer3 = nn.Linear(...) # does not exist in first script!

# continue with random operations in both scripts
x = torch.randn(...) # !!! will NOT yield the same random numbers, since the calls into the PRNG diverged
# if you want to sample the same values, re-seed the code
torch.manual_seed(seed2)
x = torch.randn(...) # will have the same random numbers in both scripts
</code></pre>