Different results when manually optimizing a network's parameters

Omar_AlSuwaidi: <p>I’ve been playing around with the autograd engine, attempting to “manually” update a network’s parameters via the autograd mechanism, replicating SGD. However, in doing so, I’ve got different results compared to the automatic/conventional way of updating a network’s parameters (using opt.step()). I’ve had the following settings to ensure reproducibility when running the same network using the different updating schemes:</p>
<pre><code class="lang-auto">device = torch.device('cuda')
torch.cuda.empty_cache()
torch.manual_seed(0)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
np.random.seed(0)
</code></pre>
<p>The network used was a simple multi-layer FFN (MNIST classification):</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Sequential(nn.Linear(784, 1440, bias=False), nn.BatchNorm1d(1440), nn.ReLU())
        self.fc2 = nn.Sequential(nn.Linear(1440, 1440, bias=False), nn.BatchNorm1d(1440), nn.ReLU())
        self.fc3 = nn.Sequential(nn.Linear(1440, 784, bias=False), nn.BatchNorm1d(784), nn.ReLU())
        self.fc4 = nn.Sequential(nn.Linear(784, 784, bias=False), nn.BatchNorm1d(784), nn.ReLU())
        self.fc5 = nn.Linear(784, 10)

    def forward(self, x):
        return nn.Sequential(*list(self.children()))(x)
</code></pre>
<p>Updating the network using the “manual” way was done in the following manner:</p>
<pre><code class="lang-auto">m2 = Model().to(device)
for e in trange(epochs):
    for x, y in train_loader:
        x, y = x.to(device, non_blocking=True), y.to(device, non_blocking=True)
        loss = F.cross_entropy(m2(x), y)
        grad = torch.autograd.grad(loss, m2.parameters())
        for p, g in zip(m2.parameters(), grad):
            p.data -= LR * g
</code></pre>
<p>While updating the network using the “automatic/conventional” way was as follows:</p>
<pre><code class="lang-auto">m1 = Model().to(device)
opt = torch.optim.SGD(m1.parameters(), lr=LR)
for e in trange(epochs):
    for x, y in train_loader:
        x, y = x.to(device, non_blocking=True), y.to(device, non_blocking=True)
        loss = F.cross_entropy(m1(x), y)
        loss.backward()
        opt.step()
        opt.zero_grad()
</code></pre>
<p>The learning rates and epochs used are the same in both cases.</p>
<p>Here is a visualization of the results I got:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/0/c056f4e4f9606a434a93fbf98195f70e1d94d52b.png" data-download-href="https://discuss.pytorch.org/uploads/default/c056f4e4f9606a434a93fbf98195f70e1d94d52b" title="fig1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c056f4e4f9606a434a93fbf98195f70e1d94d52b_2_690x474.png" alt="fig1" data-base62-sha1="rrvYDwfgKhEywN94jclMEiyaQwb" width="690" height="474" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c056f4e4f9606a434a93fbf98195f70e1d94d52b_2_690x474.png, https://discuss.pytorch.org/uploads/default/original/3X/c/0/c056f4e4f9606a434a93fbf98195f70e1d94d52b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/0/c056f4e4f9606a434a93fbf98195f70e1d94d52b.png 2x" data-dominant-color="DFE8E3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">fig1</span><span class="informations">800×550 47.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/5/653d1c2cab6d274209f9143c85aad37d40f9386e.png" data-download-href="https://discuss.pytorch.org/uploads/default/653d1c2cab6d274209f9143c85aad37d40f9386e" title="Figure_2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/653d1c2cab6d274209f9143c85aad37d40f9386e_2_690x474.png" alt="Figure_2" data-base62-sha1="erB9r5nyzMMz41j5sfrsYBaBGVw" width="690" height="474" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/653d1c2cab6d274209f9143c85aad37d40f9386e_2_690x474.png, https://discuss.pytorch.org/uploads/default/original/3X/6/5/653d1c2cab6d274209f9143c85aad37d40f9386e.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/5/653d1c2cab6d274209f9143c85aad37d40f9386e.png 2x" data-dominant-color="EBEDEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Figure_2</span><span class="informations">800×550 35.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
While the accuracies were:</p>
<p>Accuracy (Automatic): 88.0%<br>
Accuracy (Manual): 86.7%</p>
<p>Can someone explain to me why these differences arise? Is there something extra going on behind the hood?</p>

ptrblck: <p>Based on the posted images it looks as if the very first loss value is already different before the parameter update was performed. Could you compare the initial parameters as well as the input and loss value before comparing the parameter updates?</p>

Omar_AlSuwaidi: <p>Thanks for the response; yes, you’re totally right. After comparing the initial state of the parameters of the two models, it turned out that they differed in every <code>nn.Linear()</code> weight initialization but had the same weight and bias initializations for the BatchNorms (<code>nn.BatchNorm1d()</code>).</p>
<p>I guess the question now is how to enforce the same weight initialization for all the <code>nn.Linear()</code> modules in both models?</p>

ptrblck: <p>You could either load the <code>state_dict</code> from one model to the other or seed the code carefully before initializing the models and make sure the order of calls into pseudorandom number generator is the same.</p>

Omar_AlSuwaidi: <p>Yeah, that sounds like a great idea; however, I’m getting this weird behavior now where after I did the following:</p>
<pre><code class="lang-auto">m1 = Model().to(device)
m2 = Model().to(device)
m2.load_state_dict(m1.state_dict())
</code></pre>
<p>And then manually checked whether the initial state of the parameters in both models was equivalent (which it was), the initial loss value was <strong>still</strong> different (and throughout)! Not only that, but the Accuracy (Automatic) went to 88.21% from 88.0%. Somehow loading <code>m1</code>'s state dictionary into <code>m2</code> affected <code>m1</code>?</p>
<p>I even tried the following:</p>
<pre><code class="lang-auto">m1 = Model().to(device)
m2 = deepcopy(m1)
</code></pre>
<p>Yet again, manually checking all the parameters showed they were indeed equivalent throughout the whole network. However, I’m still getting different results (though the Accuracy (Automatic) this time matches the original one (i.e., 88.0%)).</p>

ptrblck: <p>If the parameters are equal but the results still differ, your input data is either not the same or your model is applying some random operations (e.g. dropout) which can be disabled via calling <code>model.eval()</code>.</p>

Omar_AlSuwaidi: <p>After forcing the to inputs to be exactly the same, and in the exact same order sequentially (disabled shuffling for extra measure), I finally started getting same <strong>initial</strong> loss evolution, though they slightly differ towards the end and give slightly different results, but at least it’s much better than before. Thanks a lot!</p>