ResNet reproducibility

snowe: <p>Hi everyone <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I have two models that are essentially the same (same architecture, same number of parameters) but they yield different results. The first model is one from the PyTorch model selection (a ResNet18 without pretrained weights) and the other one is essentially copy pasted code a bit reformatted (I want to later try some stuff with the ResNet architecture which is why I had to code it myself).</p>
<p>Somehow they yield different results even if I seed my code to make it reproducible…Does anyone know why that is the case? Is it because altough conceptually they are the same, PyTorch initialises different weights for them because they are different instances?</p>
<p>Any help is very much appreciated!</p>
<p>All the best<br>
snowe</p>
<p>Edit: I just realised that when I instantiate two ResNets from PyTorch they also yield different results, even though they are the same. Is that behaviour to be expected? Is this because of some randomness in the batchnorm or so?</p>

AladdinPerzon: <p>You should be able to get the same results between runs, at least I’m pretty sure of that. What did you set seed on? I tried with</p>
<pre><code class="lang-python">import torch
import torchvision.models as models
seed=42
torch.manual_seed(seed)
resnet18 = models.resnet18(pretrained=False)
x = torch.randn((1, 3, 224, 224))
print(resnet18(x)[0:10])
</code></pre>
<p>And I seem to get same results on different runs. If you are running on CUDA then you should add</p>
<pre><code class="lang-auto">torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
</code></pre>
<p>When training I think it is also important to send seed to <code>DataLoader</code>. Perhaps you could share some small code that give you different results?</p>

snowe: <p>Hi <a class="mention" href="/u/aladdinperzon">@AladdinPerzon</a>, thank you for your response!</p>
<p>I seed like this:</p>
<pre><code class="lang-auto">torch.manual_seed(seed)
np.random.seed(seed)
random.seed(seed) 
# for cuda
torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.enabled = False
</code></pre>
<p>I am able to reproduce the results if I load the ResNet from PyTorch and use this one over and over again. But when I use my own implementation of the ResNet with the same architecture and the same number of parameters I don’t get the same results as I did with the loaded one.</p>
<p>In other words, loading and own implementation yield different results although the network is essentially the same…</p>

AladdinPerzon: <p>Ah, got it. It seems to me that it must weight initialization which seems strange since we’ve set the seed. This seems a bit odd and is indeed what I also get:</p>
<blockquote>
<p>Edit: I just realised that when I instantiate two ResNets from PyTorch they also yield different results, even though they are the same.</p>
</blockquote>
<p>Maybe someone can clarify this <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <aside class="quote no-group" data-username="snowe" data-post="1" data-topic="103113">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e68b1a/48.png" class="avatar"> snowe:</div>
<blockquote>
<p>Edit: I just realised that when I instantiate two ResNets from PyTorch they also yield different results, even though they are the same. Is that behaviour to be expected? Is this because of some randomness in the batchnorm or so?</p>
</blockquote>
</aside>
<p>I cannot reproduce this issue:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.manual_seed(2809)

modelA = models.resnet18()
modelB = models.resnet18()
modelB.load_state_dict(modelA.state_dict())

x = torch.randn(8, 3, 224, 224)

outA = modelA(x)
outB = modelB(x)
print((outA - outB).abs().max())
&gt; tensor(0., grad_fn=&lt;MaxBackward1&gt;)
</code></pre>
<p>Could you clarify your use case or post a code snippet to produce this issue?</p>

Alexey_Demyanchuk: <p>If I understand correctly, topic author is trying to understand why his own implementation and implementation of resnet18 from torchvision give different results. First, i would try to catch some obvious mistakes like instantiating pretrained version from torchvision. Second, I would go through the source code of torchvision model and make sure I did all the layer instantiations and weights initialization in my own model the same as they do in ‘reference’ model. Without looking into ‘own’ model code it is to broad topic to discuss, right?</p>

ptrblck: <p>That’s right and a good idea to solve the original issue. I would start with making sure the <code>torchvision</code> implementation yields the same results first as a smoke test, as I guess there might be a misunderstanding how seeding works or another issue.</p>

AladdinPerzon: <p>Could you explain why we are doing this line:</p>
<p><code>modelB.load_state_dict(modelA.state_dict())</code></p>
<p>If I do</p>
<pre><code class="lang-python">seed=42
torch.manual_seed(seed)
resnetA = models.resnet18(pretrained=False)
resnetB = models.resnet18(pretrained=False)
x = torch.randn((1, 3, 224, 224))
print((resnetA(x) - resnetB(x)).abs().max())
</code></pre>
<p>I obtain different results. I’m assuming this is expected but I guess why they are different is not clear to me</p>

ptrblck: <p>Your code should yield different results, since you are only seeding the code once.<br>
Each call into the <a href="https://en.wikipedia.org/wiki/Pseudorandom_number_generator">pseudorandom number generator</a> would yield a new random number.<br>
Seeding will make sure that the sequence of these pseudorandom numbers is reproducible, but won’t yield the same numbers for random calls:</p>
<pre><code class="lang-python">torch.manual_seed(2809)
print(torch.randn(2))
&gt; tensor([-2.0748,  0.8152])
print(torch.randn(2))
&gt; tensor([-1.1281,  0.8386])

torch.manual_seed(2809)
print(torch.randn(2))
&gt; tensor([-2.0748,  0.8152])
print(torch.randn(2))
&gt; tensor([-1.1281,  0.8386])
</code></pre>
<p>If you want to use the seed to initialize both models with the same random values, you would have to re-seed the code:</p>
<pre><code class="lang-python">
torch.manual_seed(2809)
modelA = models.resnet18()
torch.manual_seed(2809)
modelB = models.resnet18()


x = torch.randn(8, 3, 224, 224)

outA = modelA(x)
outB = modelB(x)
print((outA - outB).abs().max())
&gt; tensor(0., grad_fn=&lt;MaxBackward1&gt;)
</code></pre>

Alexey_Demyanchuk: <p>Great explanation! Thanks ))</p>

AladdinPerzon: <p>That makes sense, thank you <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=9" title=":pray:" class="emoji" alt=":pray:"></p>

snowe: <p>Thank you all for your inputs! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>So I can reproduce the results of two ResNets I load from PyTorch. However, my own implementation still yields different results compared to the loaded ones and I’ve checked it multiple times and cannot figure out why it behaves different. I can reproduce the results of my own implementation as well, so there does not seem to be some weird random stuff happening…</p>
<p>Here is my ResNet implementation:</p>
<pre><code class="lang-auto"># this type of block is used to build ResNet18 and ResNet34
class BasicBlock(nn.Module):
    expansion = 1
    
    def __init__(self, in_channels, out_channels, stride=1, downsample=None):
        super(BasicBlock, self).__init__()
        
        self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1, stride=stride, bias=False)
        self.bn1 = nn.BatchNorm2d(out_channels)
        
        self.relu = nn.ReLU(inplace=True)
        
        self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, padding=1, stride=1, bias=False)
        self.bn2 = nn.BatchNorm2d(out_channels)
        
        self.downsample = downsample
        self.stride = stride
        
    def forward(self, x):
        identity = x
        
        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)
        
        out = self.conv2(out)
        out = self.bn2(out)
        
        if self.downsample is not None:
            identity = self.downsample(identity)
            
        out += identity
        out = self.relu(out)
        
        return out
    
class ResNet(nn.Module):
    def __init__(self, block, layers, num_classes):
        super(ResNet, self).__init__()
        
        self.in_channels = 64 
        
        self.conv1 = nn.Conv2d(in_channels=3, out_channels=self.in_channels, kernel_size=7, 
                               padding=3, stride=2, bias=False)
        self.bn1 = nn.BatchNorm2d(self.in_channels)
        self.relu = nn.ReLU(inplace=True)
        self.maxpool = nn.MaxPool2d(kernel_size=3, padding=1, stride=2)
        
        self.layer1 = self._make_layer(block, layers[0], out_channels=64, stride=1)
        self.layer2 = self._make_layer(block, layers[1], out_channels=128, stride=2)
        self.layer3 = self._make_layer(block, layers[2], out_channels=256, stride=2)
        self.layer4 = self._make_layer(block, layers[3], out_channels=512, stride=2)
        
        self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
        self.fc = nn.Linear(512 * block.expansion, num_classes)
        
        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
            elif isinstance(m, (nn.BatchNorm2d, nn.GroupNorm)):
                nn.init.constant_(m.weight, 1)
                nn.init.constant_(m.bias, 0)
        
    def _make_layer(self, block, num_blocks, out_channels, stride):
        downsample = None
        
        if stride != 1 or self.in_channels != out_channels * block.expansion:
            downsample = nn.Sequential(
                nn.Conv2d(self.in_channels, out_channels*block.expansion, kernel_size=1, stride=stride, bias=False),
                nn.BatchNorm2d(out_channels*block.expansion)
            )
        
        layers = []
        layers.append(block(self.in_channels, out_channels, stride, downsample))
        self.in_channels = out_channels * block.expansion
        for _ in range(1, num_blocks):
            layers.append(block(self.in_channels, out_channels))
            
        return nn.Sequential(*layers)
    
    def forward(self, x):
        
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)
        
        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)
        
        x = self.avgpool(x)
        x = torch.flatten(x, 1)
        x = self.fc(x)
        
        return x
</code></pre>
<p>And here how I test it:</p>
<pre><code class="lang-auto">def set_seed(seed):
    torch.manual_seed(seed)
    np.random.seed(seed)
    random.seed(seed) 
    # for cuda
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.enabled = False

set_seed(0)
modelA = PyTorchModels.resnet18()
in_ = modelA.fc.in_features
classes = 10
modelA.fc = nn.Linear(in_features=in_, out_features=classes)

set_seed(0)
modelB = PyTorchModels.resnet18()
in_ = modelB.fc.in_features
classes = 10
modelB.fc = nn.Linear(in_features=in_, out_features=classes)

set_seed(0)
modelC = ResNet(BasicBlock, [2, 2, 2, 2], 10)

t = torch.rand(32, 3, 32, 32)

outA = modelA(t)
outB = modelB(t)
outC = modelC(t)

print(outA[0])
print('\n')
print(outB[0])
print('\n')
print(outC[0])

tensor([-0.0160, -0.0413,  0.5379, -0.3654, -0.0620, -0.7079, -0.9632, -0.9346,
         1.5941,  1.0369], grad_fn=&lt;SelectBackward&gt;)

tensor([-0.0160, -0.0413,  0.5379, -0.3654, -0.0620, -0.7079, -0.9632, -0.9346,
         1.5941,  1.0369], grad_fn=&lt;SelectBackward&gt;)

tensor([ 0.1272,  0.1153, -0.4902, -0.2696, -0.4524, -0.4243, -0.5799, -0.0227,
         0.5023,  0.8597], grad_fn=&lt;SelectBackward&gt;)
</code></pre>
<p>So the two loaded ResNets behave the same but diffreent to my own…</p>

ptrblck: <p>I get the same results, if I try to make sure to use the same calls into the PRNG:</p>
<pre><code class="lang-python">torch.manual_seed(2809)
modelA = ResNet(BasicBlock, [2, 2, 2, 2], 1000)
in_ = modelA.fc.in_features
classes = 10
modelA.fc = nn.Linear(in_features=in_, out_features=classes)
torch.manual_seed(2809)
modelB = ResNet(BasicBlock, [2, 2, 2, 2], 1000)
in_ = modelB.fc.in_features
classes = 10
modelB.fc = nn.Linear(in_features=in_, out_features=classes)

torch.manual_seed(2809)
modelC = models.resnet18()
in_ = modelC.fc.in_features
modelC.fc = nn.Linear(in_features=in_, out_features=classes)


x = torch.randn(8, 3, 224, 224)

outA = modelA(x)
outB = modelB(x)
outC = modelC(x)
print((outA - outB).abs().max())
&gt; tensor(0., grad_fn=&lt;MaxBackward1&gt;)

print((outA - outC).abs().max())
&gt; tensor(0., grad_fn=&lt;MaxBackward1&gt;)
</code></pre>
<p>In your example you are using <code>modelC = ResNet(BasicBlock, [2, 2, 2, 2], 10)</code>, which will directly create a linear layer with 10 output classes.<br>
While it’s a valid approach for your use case, this will break the comparison using seeds, since the calls to the PRNG in the <code>torchvision</code> implementation are:</p>
<pre><code class="lang-python">-&gt; init layer1
-&gt; init layer2
...
-&gt; init fc with 1000 output classes
-&gt; init custom nn.Linear with 10 output classes
</code></pre>
<p>while you would skip the penultimate step.<br>
If you use my code, you should get the same results.</p>
<p>That being said, I would recommend not to use the seeding approach to compare models, as you would need to be familiar which layers are initialized in which order.<br>
The better approach is just to load the <code>state_dict</code> from one model into the other and test both models.</p>

snowe: <p>Thank you so much <a class="mention" href="/u/ptrblck">@ptrblck</a>, it is working now! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Usually I wouldn’t use seeding to compare models but this time I just wanted to make sure that my implementation was correct. Therefore, I figured I just check the results using the same seed. But I see how this can lead to issues, as in this case for example.</p>
<p>Anyways, once again, thank you for the support!</p>
<p>All the best<br>
snowe</p>