How to get the same results from a code before/after modularization

username: dkobayas-cyber
cooked: <p>Hello,<br>
I have a <code>main.py</code> where several models (U-net, GAN, …etc) are defined and used inside. I cannot share the actual code but I can show the overall structure of the code:</p>
<ol>
<li>Load the input dataset</li>
<li>Function1 to preprocess the dataset
<ul>
<li>Run the Function1</li>
</ul>
</li>
<li>Class1 that defines the convolutional network using PyTorch
<ul>
<li>Load a pre-trained weight to Class1 and create a network model.</li>
</ul>
</li>
<li>Class2 that defines the convolutional network using PyTorch
<ul>
<li>Load a pre-trained weight to Class2 and create a network model.</li>
</ul>
</li>
<li>Class3 that defines the convolutional network using Class1 and Class2
<ul>
<li>Train the network defined by Class3</li>
</ul>
</li>
<li>Save the results from the training</li>
</ol>
<p>However, because everything is defined in one file, this code is too long. To make it more reader-friendly, I defined the functions and classes in separate files so that I can import them like below:</p>
<pre><code class="lang-auto"># main_new.py

# Set seed for reproducibility 
seed = 0
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
np.random.seed(seed)
random.seed(seed)

# Load packages
from tools.file1 import Function1
from tools.file2 import Class1, Class2, Class3 

###################################################
# Processing using Function1, Class1, Class2, Class3
###################################################
</code></pre>
<p>This new code runs without any errors. However, <strong>the problem is, the original <code>main.py</code> and the new <code>main_new.py</code> give different results</strong>. I want to get the same results as that of the original one to convince that I’m not messing up anything during the modularization. Note that both <code>main.py</code> and <code>main_new.py</code> are deterministic, so I don’t think the problem is due to the randomness of PyTorch libraries.</p>
<ul>
<li>Can the modularization (defining classes and functions in separate files) change the results?</li>
<li>If so, is there anyway to prevent that?</li>
</ul>
<p>I am using Python=3.7.10 and PyTorch=1.7.1.</p>

username: InnovArul
cooked: <aside class="quote no-group" data-username="dkobayas-cyber" data-post="1" data-topic="152951">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dkobayas-cyber/48/41999_2.png" class="avatar"> dkobayas-cyber:</div>
<blockquote>
<p><strong>the original <code>main.py</code> and the new <code>main_new.py</code> give different results</strong>.</p>
</blockquote>
</aside>
<p>Can you elaborate on what you mean by different results?<br>
Is it different accuracy or loss etc?</p>
<aside class="quote no-group" data-username="dkobayas-cyber" data-post="1" data-topic="152951">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dkobayas-cyber/48/41999_2.png" class="avatar"> dkobayas-cyber:</div>
<blockquote>
<p>Can the modularization (defining classes and functions in separate files) change the results?</p>
</blockquote>
</aside>
<p>AFAIK, this won’t be the case.</p>
<p>Can you try to pass the same example through both networks and check the output layer-by-layer and see if they are equal?</p>

username: dkobayas-cyber
cooked: <p><a class="mention" href="/u/innovarul">@InnovArul</a> I used the following function (credit: <a href="https://discuss.pytorch.org/t/check-if-models-have-same-weights/4351">Check if models have same weights</a>) to compare the trained weights from the two codes layer-by-layer. The weights and biases were totally different even when the inputs are the same.</p>
<pre><code class="lang-auto">import torch 

def compare_weights(path_save1, path_save2):
    weights1 = torch.load(path_save1, map_location=torch.device('cpu'))
    weights2 = torch.load(path_save2, map_location=torch.device('cpu'))
    
    models_differ = 0
    for key_item1, key_item2 in zip(weights1.items(), weights2.items()):
        if torch.equal(key_item1[1], key_item2[1]):
            pass
        else:
            models_differ += 1
            if (key_item1[0] == key_item2[0]):
                print('Mismatch found at', key_item1[0])
            else:
                raise Exception 
    
    if models_differ == 0:
        print('Weights match perfectly!')

</code></pre>

username: InnovArul
cooked: <p>Are these models similar performance-wise/loss-wise?<br>
Can you check if you are starting with the same model weights in both of them?</p>
<p>In my understanding, modularization won’t introduce the change in results. but you have to make sure the old file and new file follow the same steps in the same order, so that weight initialization doesn’t differ.</p>
<aside class="quote no-group" data-username="dkobayas-cyber" data-post="3" data-topic="152951">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dkobayas-cyber/48/41999_2.png" class="avatar"> dkobayas-cyber:</div>
<blockquote>
<p><code>        if torch.equal(key_item1[1], key_item2[1]):</code></p>
</blockquote>
</aside>
<p>can you print and check the values instead of <code>torch.equal</code>?<br>
For comparing float tensors, you may need to use <code>torch.allclose()</code>.</p>

username: dkobayas-cyber
cooked: <p>Answers to your comments:</p>
<ul>
<li>Their training loss differ. See below.</li>
<li>I’m sure the two codes are using the same initial weights since I’m giving the pre-trained weights explicitly to the models.</li>
<li>I printed one of then tensor values of the trained weights. See below.</li>
</ul>
<p>A few things I found:</p>
<ul>
<li>
<p>Until the training starts, the two codes are using the same random stream.<br>
In the code before the training phase, I added the following:</p>
<pre><code class="lang-auto">print('--Test--')
print(torch.bmm(torch.randn(2, 2, 2).to_sparse().cuda(), torch.randn(2, 2, 2).cuda()))
</code></pre>
<p>The results from the two codes are the same, so until this point, the random number streams must be consistent.</p>
</li>
<li>
<p>The batch data chosen for each iteration in the two codes is also the same.<br>
At every iteration, I generate a set of random indices to pick the batch data from training dataset. I printed out the indices and confirmed that the same set of indices are used in the two codes.</p>
</li>
<li>
<p>However, in the training phase, the inconsistency starts from the very first iteration. The image below compares the training loss and trained weights of the two codes:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/0/a0e6e8222bc0c70982fb7d01179bbcef32bfbc85.png" data-download-href="https://discuss.pytorch.org/uploads/default/a0e6e8222bc0c70982fb7d01179bbcef32bfbc85" title="Picture1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a0e6e8222bc0c70982fb7d01179bbcef32bfbc85_2_690x357.png" alt="Picture1" data-base62-sha1="mXp6RIlqgptcVRV93oecBTLhFS5" width="690" height="357" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a0e6e8222bc0c70982fb7d01179bbcef32bfbc85_2_690x357.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a0e6e8222bc0c70982fb7d01179bbcef32bfbc85_2_1035x535.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a0e6e8222bc0c70982fb7d01179bbcef32bfbc85_2_1380x714.png 2x" data-dominant-color="000000"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Picture1</span><span class="informations">4366×2265 303 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The new code gives larger training loss for some reason…</p>
</li>
</ul>

username: dkobayas-cyber
cooked: <p>Sorry, you were right! After checking the model again, I realized that one of the model weights was not loaded correctly! After correcting this, the problem has been solved!</p>