Just initializing (and not using) an auxiliary model varying the loss of primary model

username: Yashika
cooked: <p>I’m noticing that initializing any random model, and not using that random model anywhere further in the code, is causing variations in the loss and AUC as compared to not initializing that random model.</p>
<p>For instance, in the below code snippet, just initializing the unused auxiliary model <code>model_aux</code> is causing variations in the loss and AUC of <code>model</code>.</p>
<pre><code class="lang-auto">model = Model(n_heads, num_layers, device)
model_aux = Model_aux(n_heads_aux, num_layers_aux, device)

model = model.to(device)
# model_aux = model_aux.to(device)

params = list(model.parameters()) 
# params_aux = list(model_aux.parameters())

optimizer = torch.optim.Adam(params, lr=lr, weight_decay=weight_decay)  
# optimizer_aux = torch.optim.Adam(params_aux, lr=lr, weight_decay=weight_decay)
</code></pre>
<p>Has anyone else experienced this issue, and if so, do you have any insights as to why this might be happening?</p>

username: srishti-git1110
cooked: <p>Hey,<br>
This seems strange.<br>
Did you set the seed correctly to ensure a fair comparison?</p>
<p>I would also think that the data loading order might also create such differences. That to say, in case you used a random sampler for your dataloader with <code>shuffle=True</code> rather than a custom sampler class, the samples included in the batches for each run shall be different and might be a potential reason for this difference.</p>
<p>Did you checked this difference multiple times and if yes, is this difference seen every time?</p>

username: Yashika
cooked: <p><a class="mention" href="/u/srishti-git1110">@srishti-git1110</a>, Thanks for the reply!<br>
Indeed, I had set the seed using the following function-</p>
<pre><code class="lang-auto">def seed_everything(seed: int):
    random.seed(seed)
    os.environ["PYTHONHASHSEED"] = str(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = True
</code></pre>
<p>I tried setting <code>shuffle=False</code>, yet a significant difference persists.<br>
Yes, I checked this multiple times and the difference is seen every time.<br>
Also, the difference varies with altering the architecture of the auxiliary model.</p>

username: srishti-git1110
cooked: <p>I see.<br>
Unfortunately, I cannot seem to understand why this must be happening and whether the behaviour is expected. Couldn’t find anything similar in GitHub issues as well.</p>
<p>We can maybe wait for someone from the community to provide some insight.<br>
cc: <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: KFrank
cooked: <p>Hi Yashika!</p>
<aside class="quote no-group" data-username="Yashika" data-post="1" data-topic="179300" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yashika/48/44745_2.png" class="avatar"> Yashika:</div>
<blockquote>
<p>just initializing the unused auxiliary model <code>model_aux</code> is causing variations in the loss and AUC of <code>model</code>.</p>
<pre><code class="lang-auto">model = Model(n_heads, num_layers, device)
model_aux = Model_aux(n_heads_aux, num_layers_aux, device)
</code></pre>
</blockquote>
</aside>
<p>Almost certainly you are using some sort of randomization in your training<br>
and initializing <code>model_aux</code> is consuming some random numbers, putting<br>
your training at a different point in the (pseudo)random-number stream.</p>
<p>I know you said that you’ve set <code>shuffle = False</code>, but there could be other<br>
sources of randomization such as <code>Dropout</code> layers or random transformations<br>
such as <code>RandomCrop</code> if you are using augmentation.</p>
<p>Try this experiment:</p>
<pre data-code-wrap="python"><code class="lang-python"># to make runs repeatable
torch.manual_seed (12345)
model = Model(n_heads, num_layers, device)

# either do or don't instantiate Model_aux
model_aux = Model_aux(n_heads_aux, num_layers_aux, device)

# set random-number stream to the same well-defined state
# regardless of whether Model_aux was instantiated
torch.manual_seed (98765)

# run some training ...
</code></pre>
<p>First verify that your runs are repeatable when you perform multiple runs<br>
<em>without</em> <code>Model_aux</code>.</p>
<p>Assuming this test passes, try running <em>with</em> <code>Model_aux</code> (making sure to<br>
include the second call to <code>torch.manual_seed (98765)</code>) and see if you<br>
continue to get the same results.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Yashika
cooked: <p><a class="mention" href="/u/kfrank">@KFrank</a>, Thank you very much for the help!<br>
Setting the random seed again after instantiating <code>Model_aux</code> has resolved the issue.<br>
The loss and AUC score now remain the same regardless of <code>Model_aux</code> 's initialization.</p>