Possible reasons for regression problem with almost same prediction result

Hong_Cheng: <p>I have a model that predict four similar category numerical data (target values).</p>
<p>I use a CNN to extract feature for each category target value, before FC layers, I also add two values to the flattened extracted feature maps.</p>
<p>This works in Keras. I rewrite my codes with Pytorch. But the result of prediction for each category are almost same, with a little fluctuation.</p>
<p>I worry that this is caused by the back propagation.<br>
The loss my model is RMSE of four target values of a training batch.</p>
<p>Any ideas?</p>

albanD: <p>Hi,</p>
<p>This does not sounds like something you should not do.<br>
Could you share some code? In particular, how do your define your model. And how do you add the extra values before the FC?</p>

Hong_Cheng: <p>concatenate the extra value with the feature map</p>

Hong_Cheng: <p>Here is my model output</p>
<pre><code class="lang-auto">		out = torch.cat((x1, x2, x3, x4), 1)
		# print('after_concat', x1.shape)  # (batch_size,4)
</code></pre>
<p>Here is my loss function</p>
<pre><code class="lang-auto">def RMSELoss_new(yhat, y):
	return torch.mean(torch.sqrt(torch.mean((yhat - y) ** 2, 0)))
criterion = RMSELoss_new
</code></pre>
<p>And I use <code>SGD</code> optimizer and in the main training loop, I calculate the loss and optimize the model.</p>
<pre><code class="lang-auto">			optimizer.zero_grad()

			outputs = net(inputs, tr_angles)   # (batch_size,4)

			# loss = torch.sqrt(criterion(outputs, labels))
			loss = criterion(outputs, labels)
			# print(loss)  # tensor(107.8965, device='cuda:0', grad_fn=&lt;SqrtBackward&gt;)

			loss.backward()
			optimizer.step()
</code></pre>

Hong_Cheng: <p>Here is my understanding:</p>
<pre><code class="lang-auto">for i in range(1,epoch+1):
  model.train()
  optimizer.zero_grad()
  outputs = model(inputs)
  loss = criterion(outputs, labels)
  loss.backward()
  optimizer.step()
  if val_print_show_time == i:
     with torch.no_grad():
        model.eval()
        val_outputs = model(val_inputs)
        # early stoping ......
</code></pre>
<p>I think I may make a mistake that I should move the <code>model.eval()</code> outside of the <code>with torch.no_grad():</code></p>
<pre><code class="lang-auto">  if val_print_show_time == i:
     model.eval()
     with torch.no_grad():
        val_outputs = model(val_inputs)
        # early stoping ......
</code></pre>
<p>But my model’s performance is not good.</p>
<p>For the <em><strong>regression</strong></em>, especially for this kind of <strong>multi-value prediction problem</strong>, is there any better <strong>loss function</strong> or <strong>optimizer</strong> than <strong>RMSE</strong> and <strong>SGD</strong>(or Adam)?</p>
<p><strong>OTHER DETAILS</strong></p>
<p>Here is my <strong>model output</strong></p>
<pre><code class="lang-auto">		out = torch.cat((x1, x2, x3, x4), 1)
		# print('after_concat', x1.shape)  # (batch_size,4)
</code></pre>
<p>Here is my <strong>loss function</strong></p>
<pre><code class="lang-auto">def RMSELoss_new(yhat, y):
	return torch.mean(torch.sqrt(torch.mean((yhat - y) ** 2, 0)))
criterion = RMSELoss_new
</code></pre>
<p>And I use <code>SGD</code> <strong>optimizer</strong> and in the main training loop, I <em>calculate the loss and optimize the model.</em></p>
<pre><code class="lang-auto">			optimizer.zero_grad()

			outputs = net(inputs, tr_angles)   # (batch_size,4)

			# loss = torch.sqrt(criterion(outputs, labels))
			loss = criterion(outputs, labels)
			# print(loss)  # tensor(107.8965, device='cuda:0', grad_fn=&lt;SqrtBackward&gt;)

			loss.backward()
			optimizer.step()
</code></pre>

albanD: <p>All these look good.</p>
<ul>
<li>Moving the <code>.eval</code> outside of the <code>no_grad()</code> block won’t change anything.</li>
<li>If you use batchnorm, you might want to double check how the model behaves without the <code>model.eval()</code>. The saved statistics for the eval mode might not be very good in the middle of the training when the model changes a lot.</li>
<li>You can check that the network initialization is what you expect. That could explain the difference with Keras.</li>
</ul>

Hong_Cheng: <p>If we don’t use validation dataset do the evaluation, I am not sure if my model learns or not.<br>
In this case, how can we know the learning progress during the training? or this kind of evaluation is for debug?</p>
<p>Anyway, I will take your suggestions</p>

albanD: <p>You can also do evaluation with large batch-size, with the network in training mode. That might help with getting a better idea while the weights are not stable.</p>

Hong_Cheng: <p>you mean I can evaluate the model without using the <code>model.eval()</code>.<br>
Is my understanding right? Thanks a lot!</p>
<p>Actually, I use one-batch validation dataset <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"> <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>

albanD: <p><code>.eval()</code> changes the behavior of some modules. For example, Batchnorm uses the saved statistics instead of the current batch’s ones, dropout becomes an identity, etc</p>