Strange behaviour in PyTorch

MrPositron: <p>I have a classifier, and I want to compute outputs from the model in the training and validation modes. I am also checking their respective accuracies. However, I found one interestingly strange behaviour.</p>
<p>I have already pre-trained the model (ResNet-18) on Cifar-10 and saved the best model on the validation set with accuracy 85.54%. When I run <code>get_outputs</code> function on the same data, I get same accuracy in the val set. However, if I will change the order of mode, s.t. instead of <code>for mode in ['val', 'train']</code> I will write <code>for mode in ['train', 'val']</code>, val accuracy will be 85.66%. I do not understand why this happens.</p>
<pre><code class="lang-auto">@torch.no_grad()
def get_outputs(model, loader_dict, device):
	batches = {'train': [], 'val': []}
	correct = {'train': 0, 'val': 0}
	total = {'train': 0, 'val':0}
	for params in model.parameters():
		params.requires_grad = False

	for mode in [ 'val', 'train']:
		if mode == 'train':
			model.train()
		else:
			model.eval()
		for _, (inputs, targets) in enumerate(tqdm(loader_dict[mode])):
			inputs, targets = inputs.to(device), targets.to(device)
			outputs = model(inputs)
			batches[mode].append(outputs.detach().cpu())
			_, predicted = outputs.max(1)
			predicted = predicted.cpu()
			total[mode] += targets.size(0)
			correct[mode] += predicted.eq(targets.cpu()).sum().item()
	return correct, total
</code></pre>
<p>The way I create a <code>loader_dict</code>:</p>
<pre><code class="lang-auto">	train_size = int(0.9 * len(dataset))
	val_size = len(dataset) - train_size
	train_data, val_data = random_split(dataset, [train_size, val_size], generator = torch.Generator().manual_seed(42))

	train_loader = DataLoader(train_data, batch_size = 128, shuffle = True, 
		num_workers = 8, pin_memory = True, generator = torch.Generator().manual_seed(42))
	val_loader = DataLoader(val_data, batch_size = 128, shuffle = False, 
		num_workers = 8, pin_memory = True, generator = torch.Generator().manual_seed(42) )

	
	loader_dict = {'train': train_loader, 
				   'val': val_loader}
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/5/950d22537815b233fd96ae6fa4fa27326c4940c6.png" data-download-href="https://discuss.pytorch.org/uploads/default/950d22537815b233fd96ae6fa4fa27326c4940c6" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/5/950d22537815b233fd96ae6fa4fa27326c4940c6.png" alt="image" data-base62-sha1="lgzhwSrVIzh4BI6vc0D63siUWCa" width="690" height="142" data-dominant-color="454545"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1289×266 10.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
In the picture above you can see that if mode <code>'val'</code> is before <code>'train'</code>, then the number of correct images is 4277. However, when the mode <code>'val'</code> is after <code>'train'</code>, then the number of correct images is 4283. As for the training set, they are same.</p>

MrPositron: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> could you look at this?</p>

MrPositron: <p>I printed the squared output of the val set in different orders.</p>
<p>When <code>'val'</code> follows <code>'train'</code> the squared output is 4932.34765625. Otherwise it is 4933.31591796875.</p>

ptrblck: <p>Swapping the training and validation loop might change the order or calls into the pseudorandom number generator and might thus yield a different final accuracy.<br>
You could rerun both approaches with different seeds and check this behavior.</p>

MrPositron: <p>Thanks for you reply!</p>
<p>I am seeding everything as follows:</p>
<pre><code class="lang-auto">	torch.manual_seed(42)
	torch.cuda.manual_seed(42)
	torch.cuda.manual_seed_all(42)
</code></pre>
<p>Furthermore, if I will put it like this <code>for mode in ['val', 'val']</code>. The answer will be 85.54% each time</p>

MrPositron: <aside class="quote no-group quote-modified" data-username="MrPositron" data-post="5" data-topic="113768">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrpositron/48/8671_2.png" class="avatar"> MrPositron:</div>
<blockquote>
<p>for mode in [‘val’, ‘val’]</p>
</blockquote>
</aside>
<p>However, setting it like this: <code>for mode in ['val', 'train', 'val']</code> will yield different results, 85.54% and 85.66% respectively.</p>

MrPositron: <p>Choosing different seeds doesn’t change the behavior.</p>

MrPositron: <p>I think I found the mistake.</p>
<p>When I use <code>for mode in ['val', 'train', 'val']</code> accuracies are different during validation sets, 85.54% and 85.66%. However, If I will remove <code>model.train()</code>, everything works fine. Thus, I think that <code>model.train()</code> affects the network. I checked the value of parameters, and they are identical. I think that somehow dropout layer is affected after we do <code>model.train()</code></p>

ptrblck: <p>In training mode dropout layers will be activated and batchnorm layers will normalize the input batch using the batch statistics and will also update their internal running stats.<br>
I’m not sure where you’ve called <code>.train()</code> and <code>.eval()</code>, but these operations should be called in the corresponding training and evaluation loops, respectively.</p>

MrPositron: <p>In the first post I showed how I apply <code>model.eval()</code> and <code>model.train()</code>. Below what happens when I loop the mode over <code>['val', 'train', 'val']</code>:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1bf1eac1ac4a4b6df7379ecec18917a2fd27abd9.png" alt="image" data-base62-sha1="3ZdcbqJOSCOPZ5xLcoOwxuJnwnL" width="329" height="155"><br>
When I remove <code>mode.train()</code> and there will be only <code>model.eval()</code>. The result is following:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/c/fce561507eb2b56be5ffda4bae55220f9b6b2ebf.png" alt="image" data-base62-sha1="A5dLzd9ILCLYgnp1uydJHhW428D" width="258" height="163"></p>
<p>The rightmost values are accuracies.</p>

MrPositron: <p>Interestingly when I put it like this:</p>
<pre><code class="lang-auto">for mode in ['val', 'train', 'val', 'train', 'val']:
</code></pre>
<p>It returns 85.54%, 85.66% and 85.44%. I am not sure what is happening. But anyway, thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/6/e684498c1c29dde02b6ef88af17e3f31b1c5a867.png" alt="image" data-base62-sha1="wTff1SoHpqfvMVE6JLPWs7wBYmH" width="318" height="330"></p>

MrPositron: <p>Could you look at it? <a class="mention" href="/u/alband">@albanD</a> <a class="mention" href="/u/smth">@smth</a> <a class="mention" href="/u/tom">@tom</a></p>
<p>Thanks in advance!</p>

tom: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> actually already mentioned what’s going on there: The BatchNorm statistics are updated when you run in train mode, so you have a different network.<br>
Note that you need to check the state dict or the buffers, not just the parameters.<br>
To abstract the situation: You have something where you get behaviour A or B depending on “the state” but you don’t know what exactly is causing it. It might be natural to actually see where the state, here the state_dict, actually differs. So you’d want to save the state dict before the val run and then load the two and compute the maximum absolute difference or so between the weights for each key. You’ll see that it’s the batch norm statistics.</p>
<p>Best regards</p>
<p>Thomas</p>

MrPositron: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>  and <a class="mention" href="/u/tom">@tom</a>! I just understood it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>