The confusing difference in training () and eval() mode

Wong_Marco: <h2><a name="content-1" class="anchor" href="#content-1"></a>content</h2>
<p>The metrics of train() is not bad. But when it comes to validating/eval mode, the metrics result is disaster. So I try to print the predict value as the graph below.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8dbf8f787e8ffcd5b26cd43898f36ad3f4fcf4db.png" data-download-href="https://discuss.pytorch.org/uploads/default/8dbf8f787e8ffcd5b26cd43898f36ad3f4fcf4db" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8dbf8f787e8ffcd5b26cd43898f36ad3f4fcf4db.png" alt="image" data-base62-sha1="kdXJRhjd2v8XlM2mFctR8C2KV2b" width="690" height="488" data-dominant-color="EEECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">835×591 51.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<blockquote>
<p>the blue one is the ground truth, and the orange one is my prediction.</p>
</blockquote>
<p>From the graph, the output of the model in validating mode is almost the same every time. So I’m wondering that why it happened and what can I do to fix it? Here is parts of my code. In case of overfitting, I use the same dataset in training and validating actually. But the output of validating is still the same.</p>
<h2><a name="pseudo-code-2" class="anchor" href="#pseudo-code-2"></a>Pseudo code:</h2>
<pre><code class="lang-auto">def train(train_inf, model, criterion, optimizer):
	# some not important code is ignored
	model.train()
	for idx, movie in enumerate(train_video_names):
		movie_path = os.path.join(video_path, movie)
		train_dataset = video_dataset(movie_path, annot_path)
		train_loader = DataLoader(train_dataset, batch_size=4, num_workers=4)

		for i, data_batch in enumerate(train_loader):
			img_data, label = data_batch
			img_data = img_data.type(‘torch.FloatTensor’).cuda()
			label = label.type(‘torch.FloatTensor’).cuda()
			
			output = model(img_data)
			loss = criterion(output, label)

			optimizer.zero_grad()
			loss.backward()
nn.utils.clip_grad_norm_(model.parameters(), max_norm=20, norm_type=2)
optimizer.step()
</code></pre>
<pre><code class="lang-auto">def validate(val_inf, model, criterion, optimizer):
	# some not important code is ignored
	model.eval()
	for idx, movie in enumerate(val_video_names):
		movie_path = os.path.join(video_path, movie)
		train_dataset = video_dataset(movie_path, annot_path)
		train_loader = DataLoader(val_dataset, batch_size=4, num_workers=4)

		for i, data_batch in enumerate(val_loader):
			img_data, label = data_batch
			img_data = img_data.type(‘torch.FloatTensor’).cuda()
			label = label.type(‘torch.FloatTensor’).cuda()
			
			output = model(img_data)
			loss = criterion(output, label)
</code></pre>
<pre><code class="lang-auto">def main():
	train(train_inf, model, criterion, optimizer)
	valdate(val_inf, model, criterion, optimizer)
</code></pre>
<h2><a name="notes-3" class="anchor" href="#notes-3"></a>Notes:</h2>
<ol>
<li>I have already make sure that the data in validation mode is not the same, even not close, though the result is still almost the same.</li>
<li>Because I use the same data in train mode and eval mode, so it should not be the problem of overfitting.</li>
<li>I use the transformer architecture. I don’t think that it gonna make this thing happen, just trying to provide more information.</li>
</ol>
<p>Thanks for your help. Any kind of suggestion will be my pleasure.</p>

supalak: <p>I was also facing such a problem before.</p>

Wong_Marco: <p>Any kind of suggestion will help a lot.</p>

eqy: <p>If you remove the <code>model.eval()</code> for testing purposes does it improve the results? It might be useful to print or track some tensors layer by layer to see where the divergence begins during training and validation.</p>

Wong_Marco: <p>Sorry for replying late. I did some experiment in the day.<br>
It is the same if i remove <code>the model.eval()</code>.<br>
And I also try to print some parameters of specific layers. It’s the same parameters.</p>
<p>After further debugging, I find that when I stop at the last batch in <code>train()</code>, nomater what I input it comes out almost the same result.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1c430780d72b6c922333c81cab502aea6a8032b8.png" alt="image" data-base62-sha1="420YGwWyg9dRdfBEXM2H31mhUAE" width="556" height="372"></p>
<blockquote>
<p>As you can see, when I stop at the last batch in <code>train()</code>, the result is the same. But in the last batch, the result will change.</p>
</blockquote>
<p>Do you have any more suggestions?</p>

Wong_Marco: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bb2c3c0b90cf1af87a4b59faa59d189e44f9c6f7.png" data-download-href="https://discuss.pytorch.org/uploads/default/bb2c3c0b90cf1af87a4b59faa59d189e44f9c6f7" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bb2c3c0b90cf1af87a4b59faa59d189e44f9c6f7.png" alt="image" data-base62-sha1="qHO3F2t5Csv2V86X1rqn80L4SZ9" width="690" height="317" data-dominant-color="272624"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1092×502 30.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here is part of my code, I don’t have any idea what I do that will cause this happen.</p>