Custom loss function using two different data sets

Prak_kas: <p>I am trying to develop a model with a custom loss function which involves using of two different data sets simultaneously …</p>
<p>But I am getting many errors…Also, i am getting many NaNs, for following code…</p>
<pre><code class="lang-python">def my_loss(dX, target1, x2, target2):
  target1=torch.tensor(target1, dtype=torch.double)
  target2=torch.tensor(target2, dtype=torch.double)

  out1=model(dX)
  out2=model(x2)


  l1= ((target1-out1)**2).mean()

  l2= ((target2-out2)**2).mean()

  return torch.mean(l1+l2)
</code></pre>

albanD: <p>Hi,</p>
<p>If you want helpful answer you might want to ask precise questions (what is the error you have, what is happening that you did not expect, etc) and point to particular places in your code.<br>
Also you can format you code nicely by adding ``` before and after it.</p>

Prak_kas: <p>Sir, I have edited the post…</p>
<p>I have specified my custom loss function, which takes two sets of datasets (and 2 sets of class labels) as input.</p>
<p>When i call  backward() function on this loss , I am getting no output. When traced with intermediate values. I am getting NaNs</p>

albanD: <p>(I reformated the code with ``` to make it easier to read)</p>
<p>The code looks fine.<br>
The final torch.mean() is not very useful as l1+l2 is already a single number.</p>
<p>Could you elaborate on the “errors” you have?</p>

Prak_kas: <p>Thank You, I am just new to pytorch, and tensor flow.</p>
<p>I used to get few syntax errors, after rectifying them, I am getting NaNs</p>
<p>tensor([[-0.0967, -0.1131,  0.1547, -0.1049,  0.0389, -0.0262,  0.0479,  0.0782,<br>
0.0131, -0.1127, -0.0521,  0.0997, -0.0067,  0.0571,  0.0448, -0.0665,<br>
0.1008, -0.0876,  0.1350,  0.0513,  0.0555, -0.1171, -0.0818, -0.1276,<br>
-0.0125, -0.0328,  0.0354, -0.0660, -0.1319,  0.0002,  0.0743,  0.1331,<br>
0.0225,  0.0970],<br>
[ 0.0827, -0.1100,  0.1147, -0.0378, -0.1213, -0.0037,  0.0196, -0.1361,<br>
0.1561, -0.0331,  0.0005, -0.0352,  0.0591,  0.1045,  0.0053,  0.0299,<br>
0.1678, -0.0411, -0.0481, -0.0484,  0.1273,  0.1627,  0.0127,  0.0643,<br>
0.0254, -0.1005,  0.0228, -0.0644, -0.1644, -0.0429, -0.1133,  0.0151,<br>
0.0835,  0.1594]])</p>
<p>tensor([-0.1383, -0.1448])<br>
Iteration: 0. Loss: nan.Correct:361. total:382. Accuracy: 94.50261780104712.<br>
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:23: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).<br>
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:24: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).<br>
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:26: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).<br>
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:27: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).<br>
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:58: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).<br>
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:59: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).<br>
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:65: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).<br>
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:66: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).<br>
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:71: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).<br>
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:72: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).<br>
Iteration: 10. Loss: nan.Correct:382. total:382. Accuracy: 100.0.<br>
Iteration: 20. Loss: nan.Correct:382. total:382. Accuracy: 100.0.<br>
Iteration: 30. Loss: nan.Correct:382. total:382. Accuracy: 100.0.<br>
Iteration: 40. Loss: nan.Correct:382. total:382. Accuracy: 100.0.<br>
Iteration: 50. Loss: nan.Correct:382. total:382. Accuracy: 100.0.<br>
Iteration: 60. Loss: nan.Correct:382. total:382. Accuracy: 100.0.<br>
Iteration: 70. Loss: nan.Correct:382. total:382. Accuracy: 100.0.</p>

albanD: <p>This can come from many things.</p>
<p>Have you verified that the values are correct at the first iteration?<br>
Have you tried with a very low learning rate? models that diverge usually lead to nans after a while.<br>
You can try and remove the warnings above as well for better readability.</p>