2 models ( with similar hyper parameters) loaded from same checkpoint giving different results during training

username: sarvani
cooked: <p><strong>COLAB LINK</strong></p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20210526-060130-RC00_375891265" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1qQhyTWNzCgMYn8t0ZtIZilLgk4JptbJG?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/1qQhyTWNzCgMYn8t0ZtIZilLgk4JptbJG?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I trained a model (Lenet-5) for 10 epochs and saved the model as follows<br>
state = {<br>
‘model’: model.state_dict(),<br>
‘optimizer’: optimizer.state_dict(),<br>
‘scheduler’: scheduler.state_dict()<br>
}<br>
th.save(state,’/Documents/saved_model.pt’)</p>
<p><strong>loaded into 2 models ‘new_model’, ‘new_model2’ as below</strong><br>
new_model= model2().cuda()<br>
new_model2= model2().cuda()<br>
checkpoint = th.load(’/home/hareesh/Documents/sar/information_theory/lenet/saved_model.pt’)<br>
checkpoint2 = th.load(’/home/hareesh/Documents/sar/information_theory/lenet/saved_model.pt’)</p>
<p>new_optimizer=th.optim.SGD(new_model.parameters(), lr=0.1,momentum=0.9,weight_decay=5e-4)<br>
new_scheduler = MultiStepLR(new_optimizer,milestones=[10,20], gamma=0.1)</p>
<p>new_optimizer2= th.optim.SGD(new_model2.parameters(), lr=0.1,momentum=0.9,weight_decay=5e-4)<br>
new_scheduler2 = MultiStepLR(new_optimizer2,milestones=[10,20],  gamma=0.1)</p>
<p>new_model.load_state_dict(checkpoint[‘model’])<br>
new_model2.load_state_dict(checkpoint[‘model’])</p>
<p>new_optimizer.load_state_dict(checkpoint[‘optimizer’])<br>
new_optimizer2.load_state_dict(checkpoint[‘optimizer’])</p>
<p>new_scheduler.load_state_dict(checkpoint[‘scheduler’])<br>
new_scheduler2.load_state_dict(checkpoint[‘scheduler’])</p>
<p>trained the new models for 5 epochs,but the results are different.<br>
Also when i continue training the original model for 5 more epochs ,the results are also different from the training results of 2 new models.</p>
<p>Is it possible that the test and train accuracies of original model(15 epochs), 2 new models(5 epochs after loading from the checkpoint) will be same?</p>
<p>(From loaded checkpoint im getting same test accuracy for all 3 models, if not training any of the models)</p>

username: Bilal
cooked: <p>Didn’t go through the code but, loading a model into two different models doesn’t necessarily ends in the both having identical end result since the batches they’re trained on are different</p>

username: sarvani
cooked: <p>Thank you  for the reply <a class="mention" href="/u/bilal">@Bilal</a>.<br>
Yes it seems true.</p>
<p>But for reproducibility I have used below code snippet</p>
<p>"seed =1234<br>
random.seed(seed)<br>
os.environ[‘PYTHONHASHSEED’] = str(seed)<br>
th.manual_seed(seed)<br>
th.cuda.manual_seed(seed)<br>
th.cuda.manual_seed_all(seed)<br>
th.backends.cudnn.deterministic = True "</p>
<p>which is giving me epoch wise same test and train accuracy for all 3 models every time I experiment.</p>
<p>SO, I think this would happen only if the training batches are loaded in similar way every time for original model and 2 new models.</p>
<p>Am I  correct?<br>
If it is correct same batches should be loading for each of the models every time right ?</p>

username: Bilal
cooked: <p>Hi <a class="mention" href="/u/sarvani">@sarvani</a> ,</p>
<p>You might be right, but it all depends on the working of the <code>random.seed()</code> function call, maybe since you are training the model’s individually twice is why they’re receiving different but same batches throughout</p>

username: sarvani
cooked: <p><a class="mention" href="/u/bilal">@Bilal</a> I’m training each of them only once.<br>
Just to check I tested saving ,loading and training of 2 new models in same file so that even random.seed() would be common for all models now.<br>
Still the 2 new models and continuing the training of old model are giving different accuracies.</p>