Saving a Variable with requires_grad=True in a dictionary is not being updated

muammar: <p>Hi, I am still very new to PyTorch and after searching google for this doubt I could not find anything. Hopefully, someone could help me.</p>
<p>I have a forward() method that looks as follows:</p>
<blockquote>
<pre><code>  def forward(self, feature_vector):
      activation_function = {'tanh': torch.tanh, 'relu': F.relu}

      symbol, X = feature_vector

      X = self.backend.from_numpy(X)

      for i, l in enumerate(self.linears[symbol]):
          if i != self.out_layer_indices[symbol]:
              X = activation_function[self.activation_function](l(X))
          else:
              X = l(X)

       X = (self.slope[symbol] * X) + self.intercept[symbol]
      return X
</code></pre>
</blockquote>
<p>The problem I have is with the line before the <code>return</code>. I also have a method called <code>train</code> where I defined <code>self.slope</code> and <code>self.intercept</code> using a python dictionary:</p>
<pre><code>      self.slope = {}
      self.intercept = {}
      
       
      for symbol in unique_element_symbols:
          linears = []
      
          intercept = some operation
          intercept = self.backend.from_numpy(intercept)
          self.intercept[symbol] = Variable(intercept, requires_grad=True)
       
          slope = some operation
          slope = self.backend.from_numpy(slope)
          self.slope[symbol] = Variable(slope, requires_grad=True)
          # I create a neural network here with Linear() + nn.ModuleDict

      # Model is trained in this loop
      for epoch in range(self.epochs):     
          self.forward(tensor)
          #  some more code to have the outputs
          criterion = nn.MSELoss()
          loss = torch.sqrt(criterion(outputs, targets))
          self.optimizer.zero_grad()  # clear previous gradients
          loss.backward()
          self.optimizer.step()
</code></pre>
<p>If my understanding of the documentation is correct, the creation of <code>Variable</code> with <code>requires_grad=True</code> should make <code>autograd</code> aware of the existence of those tensors. I can see the <code>weights</code> of the layers are being updated but those variables I created inside the dictionary not. As they are part of the <code>forward()</code> method they have to affect the output and then they have to change according to the gradient. Are those variables not updated because they are in a python dictionary?</p>
<p>I would be glad if someone could help me to understand what would be the issue with my code.</p>

ptrblck: <p>Most likely there parameters are not properly registered and thus unknown to the optimizer.</p>
<p>You could try to use a <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.ModuleDict" rel="nofollow noopener">nn.ModuleDict</a> instead, but you would need to wrap your parameters in a <code>nn.Module</code>, since plain <code>nn.Parameters</code> won’t be recognized as far as I know.</p>
<p>PS: <code>Variables</code> are deprecated since <code>0.4.0</code>. Just use <code>torch.tensor(..., requires_grad=True)</code> instead.</p>

chenglu: <p>The core reason of “those variables not updated” is because they are not registed in the optimizer. If you look at any optimizer’s constructor, there will be a parameter named <code>params</code>, which is expected to hold the parameters you want to automatically updated by grad.</p>
<p>I’m not sure how you initialized your optimizer. If you do create the optimizer in a most common way, you should make whatever you want to updated automatically as a parameter of the model by using <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Module.register_parameter" rel="nofollow noopener">self.register_parameter</a> in the model’s constructor. And then, you could do <code>torch.optim.SGD(model.parameters()</code> since now the <code>model.parameters()</code> will return the variables you just registed.</p>
<p>If you have a hard time to put <code>slop</code> or <code>intercept</code> in the model’s parameters, you could also register those tensors directly to the optimizer by using <a href="https://pytorch.org/docs/stable/optim.html#torch.optim.Optimizer.add_param_group" rel="nofollow noopener">self.optimizer.add_param_group</a></p>

muammar: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> and <a class="mention" href="/u/chenglu">@chenglu</a>. Both of you were right. The parameters were not correctly registered when using a python dictionary.</p>
<p>I tried using <code>nn.ModuleDict</code> and <code>nn.Parameters</code> without success. However, the <code>self.register_parameter()</code> did the trick for me.</p>
<p>I removed the dictionaries for those parameters and added this instead:</p>
<pre><code>self.register_parameter(intercept_name, intercept)
self.register_parameter(slope_name, slope)
</code></pre>
<p>To access them in the <code>forward()</code> method I had to do:</p>
<pre><code>for name, param in self.named_parameters():
    if intercept_name == name:
        intercept = param 
    elif slope_name == name:
        slope = param 
       
X = slope * X + intercept 
</code></pre>
<p>Now intercept and slope are changing according to the gradient. Is there any way to access these parameters without using the loop over <code>name, param</code> shown above?</p>

ptrblck: <p>I’m glad it’s working!<br>
You can just use the name you’ve used to register the parameters, e.g.:</p>
<pre><code class="lang-python">X = self.slope_name * X + self.intercept_name
</code></pre>

muammar: <p>Thanks. That worked.</p>

muammar: <p>This is me, again… Something weird is happening. When I register the variables as we discussed above in <a href="https://gist.github.com/muammar/6d298f7fa5700efdbd01ae697f63e089" rel="nofollow noopener">this class</a> (I pasted it in a gist because it is long), not all variables are being seen by autograd. There should be a total of 8 tensors but at the end of optimization only 6 of them are shown (relevant part of the output below):</p>
<pre><code class="lang-auto">outputs
tensor([-14.5772266388, -14.5772266388])
targets
tensor([-14.5868730545, -14.5640010834])
No diff in intercept_Cu
No diff in slope_Cu
Diff in linears.Cu.0.weight
No diff in linears.Cu.0.bias
Diff in linears.Cu.2.weight
No diff in linears.Cu.2.bias
Diff in linears.Cu.4.weight
No diff in linears.Cu.4.bias

Optimized parameters for Cu symbol
Index 0
Parameter containing:
tensor([[ 1.5998219169e-05, -1.1084647089e-11,  3.4887983702e-07,
         -5.9022102505e-05,  1.5358797100e-05, -2.2421262713e-07,
          3.9578364522e-05, -2.5841705792e-05],
        [-1.2979450847e-11,  2.4929600051e-11, -4.3761640145e-11,
          8.7308825414e-07,  6.8003464548e-07, -6.9001464453e-07,
         -3.0529092328e-05, -1.9285680537e-06],
        [-3.4112128677e-09, -2.0672181415e-12,  1.0248225879e-12,
          1.3090937500e-05, -1.9991681199e-08, -1.2244654499e-05,
          1.1959917501e-09, -1.1793726173e-07],
        [ 9.3959987360e-12, -2.8132822081e-06, -7.1578106144e-06,
         -1.5608311514e-06,  7.4273208156e-05, -6.5615221589e-13,
          1.0243820725e-04,  2.6734230119e-07],
        [-2.8905316867e-05,  1.7972409978e-06,  2.8471620681e-05,
          1.1441625247e-06, -4.3263348743e-06,  9.2861837402e-06,
         -7.3636897469e-08, -6.2427188823e-06],
        [ 1.8716022510e-08, -4.3462468966e-06, -7.1537678559e-11,
          4.4766447493e-13, -4.2634189867e-07,  6.2688843006e-10,
         -1.5413985643e-09, -1.9352362415e-06],
        [-4.0789027480e-06,  1.7624552484e-08, -5.8772937336e-05,
          1.3928577259e-12,  1.4477242303e-06, -6.5660731252e-07,
          1.3057894830e-04,  1.0623334674e-06],
        [ 2.8627397342e-07,  7.6879496191e-07, -1.5201392500e-07,
          9.4639290182e-08,  1.7211885250e-09, -3.1544458712e-10,
         -3.1436915742e-04, -9.5523216004e-09],
        [ 5.4327131238e-07,  5.3367260989e-05,  3.0272097329e-11,
         -2.5873794129e-06, -2.5613280741e-07,  4.1264866013e-05,
          1.3438809527e-12, -5.6481166411e-09],
        [-6.4899657445e-05, -4.3667625960e-08, -6.4955729684e-10,
          7.9043999790e-08, -7.7281238191e-06,  1.7655082047e-05,
         -1.6245309098e-07, -1.7478591019e-08]], requires_grad=True)
Gradient tensor(0.0126342149)

Index 1
Parameter containing:
tensor([ 0.0846629143,  0.2052433789,  0.1129320264,  0.1384415329,
         0.2349925339, -0.1073408127,  0.2195934355,  0.3364700377,
         0.1929847300, -0.0893238783], requires_grad=True)
No gradient?

Index 2
Parameter containing:
tensor([[-1.4006408492e-05, -1.3260194009e-06,  1.4346720434e-07,
         -6.5448512032e-07,  2.9784255275e-06,  4.5995878547e-13,
          6.7223256337e-05,  6.4453017576e-12,  1.0301571401e-10,
         -1.2009696349e-08],
        [-2.2814828071e-07, -5.8791869151e-08, -3.9165245835e-04,
         -2.5221936539e-06,  1.1180619595e-06, -2.6514657293e-05,
         -1.4766897038e-07,  2.7023989242e-04, -2.9795790401e-12,
          3.4368467823e-06],
        [ 3.6120570712e-06, -3.7223298568e-04,  7.1171717408e-09,
         -4.0368172449e-06, -1.1812019807e-07, -9.0479334176e-06,
         -9.7775303479e-12,  3.3027842505e-07, -2.2225761143e-07,
          1.7060537516e-07],
        [ 4.7848516260e-05,  1.4109857602e-06, -4.7986867813e-09,
         -1.1886934145e-11, -1.5743089534e-06, -1.9210867777e-06,
          2.5946489401e-10,  7.1065740485e-05, -7.2540847214e-06,
         -2.9720404740e-13],
        [ 7.8338234744e-07,  2.9897366403e-05,  1.0493286936e-05,
         -1.2905216806e-07, -5.0532015905e-08, -1.4369081327e-05,
          5.9140187659e-05,  1.8394788640e-05,  2.8736901004e-04,
         -7.9514339557e-11],
        [-3.5491411109e-04,  3.9472433855e-06, -3.6779524635e-06,
          1.3279050108e-05,  1.0775630388e-09,  2.0076269536e-09,
          2.2207383154e-05,  1.0671607924e-05,  3.5179223801e-07,
          8.3256582002e-06],
        [-4.0831773518e-09,  3.4044984204e-05,  3.9824635678e-07,
         -5.4254252291e-07, -8.2707781596e-12,  7.9960360555e-10,
          1.6246242751e-07, -1.5748057303e-09, -4.6191617002e-05,
          1.4769234986e-04],
        [ 6.0335892158e-06,  4.0175755203e-06,  2.3420781872e-05,
         -1.4100555745e-07,  4.3824256863e-06, -1.9676244847e-05,
         -4.2883926653e-05,  2.6943742341e-05,  1.5044579982e-07,
          3.4529236359e-08],
        [-2.4134715204e-05,  3.6303499655e-05, -1.0801615247e-07,
          8.3609793364e-06,  3.0849619179e-06, -8.6793288574e-06,
          2.4900288554e-04,  8.5335452355e-14, -3.4220584699e-11,
         -4.0262357288e-06],
        [-3.2995540096e-06, -9.5245795251e-08,  2.4340472748e-08,
         -3.7661432133e-13, -4.4606429661e-09, -7.5562275015e-06,
         -6.9999718107e-05,  1.4586039470e-04,  1.0552175809e-06,
         -6.1385714220e-12]], requires_grad=True)
Gradient tensor(-0.0002918996)

Index 3
Parameter containing:
tensor([-0.0368886292, -0.1048975587, -0.2438423038, -0.2089971900,
         0.2615807354,  0.0241439044, -0.1016014665,  0.2302859128,
        -0.2738550305, -0.2952967882], requires_grad=True)
No gradient?

Index 4
Parameter containing:
tensor([[-1.6471599520e-04,  5.0920876674e-05,  1.6964193492e-05,
         -7.2204138633e-06, -7.4410144713e-11,  1.3845928848e-09,
          2.6772568162e-07,  4.4445322422e-11,  3.0647162930e-05,
         -4.6163746447e-05]], requires_grad=True)
Gradient tensor(-0.0134047084)

Index 5
Parameter containing:
tensor([-0.1564691514], requires_grad=True)
No gradient?
</code></pre>
<p>I understand biases are not counted, but only the weights of the layers are retained. Additionally, the loss seems to be decreasing in error with each epoch but the outputs of the model remain the same.</p>
<p>Do you see any problem in the class I have built? what would you recommend to check? I would really appreaciate any suggestions. I am lost here.</p>

ptrblck: <p>Could you just for the sake of debugging set the learning rate quite high, e.g. <code>100</code> and run a single update step to check if the parameters get updated?<br>
I would like to make sure that the gradients are not too small and we are thus not seeing any updates in the parameters even though the code should generally work.</p>
<p>If that still doesn’t work, could you point me to some lines of code in your Gist so that debugging would be a bit faster? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

muammar: <p>I tried it and this is the output:</p>
<pre><code class="lang-auto">outputs
tensor([-14.7144765854, -14.7144765854])
targets
tensor([-14.5868730545, -14.5640010834])
No diff in intercept_Cu
No diff in slope_Cu
Diff in linears.Cu.0.weight
No diff in linears.Cu.0.bias
Diff in linears.Cu.2.weight
No diff in linears.Cu.2.bias
Diff in linears.Cu.4.weight
No diff in linears.Cu.4.bias

Optimized parameters for Cu symbol
Index 0
Parameter containing:
tensor([[ 8.5844440460e+00,  5.7265067101e-01,  9.7642364502e+00,
         -6.3340994529e-05, -6.5706312656e-01, -1.0317638516e-01,
         -8.1990205217e-04,  3.8604885340e-02],
        [ 5.6350793839e+00, -2.3107304573e+00,  3.0389562016e-04,
          4.8599953651e+00,  6.9255195558e-03, -1.2411396503e+00,
         -1.9741505384e-01,  1.5696491755e-04],
        [-5.0464026572e-05, -3.3354687691e+00,  8.9420490265e+00,
          3.8780815601e+00,  2.3376888130e-03, -5.5896580219e-02,
          1.3885598630e-02, -1.0426228866e-02],
        [ 9.5443532337e-04,  6.6207236052e-01, -8.9323358536e+00,
          6.4504299164e+00, -5.6406411204e-06, -1.3022724390e+00,
         -6.7651176453e-01, -4.0060423315e-02],
        [-7.3771867752e+00, -2.8842912674e+01, -1.2261917114e+01,
          2.0418181084e-03, -3.8052463531e+00, -4.1274856776e-02,
          7.0336312056e-02,  7.7507920563e-02],
        [ 1.1060595512e-01, -3.2884517312e-01,  1.3592758179e+00,
         -3.1894344091e-01, -1.7296176404e-02, -6.2223523855e-02,
         -1.0991756916e+00, -1.0611775797e-03],
        [-1.0529378653e+00,  8.7992340326e-02, -3.9956837893e-02,
          5.1080572605e-01,  7.3425645828e+00, -1.4347046090e-05,
          4.9415111542e-02,  1.4767070770e+01],
        [-6.3522720337e-01, -3.3820127137e-03,  1.0707162857e+01,
          1.5198823530e-03, -5.2807319164e-01,  5.2644854784e-01,
         -1.2110622600e-02, -2.9190010391e-03],
        [ 3.0526014045e-02, -1.3536047190e-03, -3.8478989154e-04,
          2.9252339154e-03,  5.4483871460e+00,  7.9564154148e-03,
         -1.8055616617e+00, -6.4464583993e-03],
        [-4.6962329745e-01,  8.6185136752e-06,  2.4837136269e-02,
         -4.1209143092e-05,  4.2492513657e+00,  8.4312686920e+00,
          1.9236560433e-07,  3.2226529717e-01]], requires_grad=True)
Gradient tensor(-0.0052632340)

Index 1
Parameter containing:
tensor([ 0.3178096116,  0.0436611772, -0.2040621340, -0.0848887563,
         0.2899220884, -0.2525188029,  0.3507566750, -0.1945837736,
         0.1707542241, -0.0507352650], requires_grad=True)
No gradient?

Index 2
Parameter containing:
tensor([[ 9.4454865903e-03, -3.8685989380e-01,  2.8510479927e+00,
          1.6451107513e-05,  7.2450813605e-05, -2.4000716209e-01,
         -1.0067681968e-01, -6.8808451295e-02, -1.3941727579e-02,
         -1.1572503299e-01],
        [-4.0232582251e-05,  1.5237447619e-01, -8.4863287952e-08,
         -3.3062148094e-01,  3.1492298841e-01,  7.1657931805e-01,
          6.5576374531e-02,  5.8732334524e-02, -1.4156305790e-01,
         -1.1431868374e-01],
        [ 3.6709681153e-01, -5.6241098791e-03, -1.8890530029e-08,
          2.2205217101e-04, -3.8731803894e+00, -8.4317040443e-01,
         -3.5567022860e-03,  7.6645493507e-02, -1.7931096554e+00,
         -2.0117998123e+00],
        [-2.6692817919e-03,  2.3045387268e+00, -1.9369858503e-01,
          1.1653967202e-02, -1.5044789314e+00,  2.6386910677e-01,
         -1.8918566406e-02,  2.4579927325e-02, -8.7022192020e-05,
          1.4020656636e-07],
        [ 2.8100815415e-01, -1.4995394740e-03,  3.7862854004e+00,
          2.3118360519e+01, -1.2707098722e+00, -8.9394124225e-03,
         -7.3012824942e-06,  6.0733418650e-07, -6.8714976311e-02,
         -1.7940466932e-04],
        [-1.1861760616e+00, -1.7072351277e-01,  7.4709236622e-02,
         -1.6057054698e-01,  1.0028474033e-01,  4.4707970619e+00,
         -3.2747825980e-01,  1.8114055820e-06, -6.0276460648e-01,
         -2.9894538879e+01],
        [-2.0331549644e-01, -9.2998981476e-01, -2.3422073573e-03,
          6.5794992447e-01, -4.0772670507e-01, -1.7908929586e+00,
         -4.3703973293e-02, -2.3664340377e-02,  3.4835241735e-02,
          7.3881530762e-01],
        [ 5.4340696335e-01,  1.3241521083e-04, -3.2028186321e-01,
         -1.6411489248e-01, -8.0035102367e-01, -1.0085972399e-01,
         -2.3231016099e-01,  9.6048679352e+00, -1.3925330162e+01,
         -8.2148885727e-01],
        [-6.0046720505e-01,  1.0296676308e-02, -5.9643266723e-03,
         -2.2244569845e-04,  1.5874393284e-03,  9.7708535194e-01,
         -7.4371069670e-02,  8.7442662334e-05, -2.0362114906e-01,
          1.5027550697e+01],
        [ 1.2441553175e-02,  3.5354614258e+00, -4.9783945084e-01,
          1.0338279605e-01,  2.9940547943e+00, -1.0266765952e-01,
          1.2045311928e-01, -3.1238024235e+00,  3.3330893517e+00,
         -4.7617787123e-01]], requires_grad=True)
Gradient tensor(0.0078644780)

Index 3
Parameter containing:
tensor([-0.0413947999,  0.2711434066,  0.0748769045,  0.1031675935,
         0.0756872594,  0.3022760451,  0.2172745764, -0.2653046250,
         0.2037093341, -0.0445466638], requires_grad=True)
No gradient?

Index 4
Parameter containing:
tensor([[-4.1193764657e-02, -3.5373184830e-02,  1.8808110617e-03,
         -3.8154840004e-03, -2.7028546333e+00, -1.3087383270e+01,
         -3.1675234437e-02, -7.3683762550e-01, -4.4051003456e-01,
         -1.4208417851e-03]], requires_grad=True)
Gradient tensor(0.0099373152)

Index 5
Parameter containing:
tensor([-0.1635424048], requires_grad=True)
No gradient?
</code></pre>
<p>Now, by downloading the gist the changes can be made in <a href="https://gist.github.com/muammar/6d298f7fa5700efdbd01ae697f63e089#file-neuralnetwork-py-L342" rel="nofollow noopener">L-342</a>. Thank you very much for your help. I am embarrassed but honestly lost.</p>

ptrblck: <p>Thanks for the info!<br>
I tried to debug your code and stumbled upon <a href="https://gist.github.com/muammar/6d298f7fa5700efdbd01ae697f63e089#file-neuralnetwork-py-L200" rel="nofollow noopener">this line of code</a>. You are detaching <code>image_energy</code> by calling <code>.item()</code>. Later in your <code>get_loss</code> method, you are using <code>outputs</code> to calculate the gradients.<br>
However, since <code>image_energy</code> was detached, the computation graph shouldn’t compute any valid gradients before this point.<br>
I’m not completely understanding the code, so let me know, if I’m on the wrong path. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"></p>

muammar: <blockquote>
<p>Thanks for the info!<br>
I tried to debug your code and stumbled upon <a href="https://gist.github.com/muammar/6d298f7fa5700efdbd01ae697f63e089#file-neuralnetwork-py-L200" rel="noopener nofollow ugc">this line of code</a>. You are detaching  <code>image_energy</code>  by calling  <code>.item()</code> . Later in your  <code>get_loss</code>  method, you are using  <code>outputs</code>  to calculate the gradients.<br>
However, since  <code>image_energy</code>  was detached, the computation graph shouldn’t compute any valid gradients before this point.</p>
</blockquote>
<p>I think I understand what you mean about the problem I create when detaching the <code>image_energy</code> with <code>.item()</code>. I <a href="https://gist.github.com/muammar/6d298f7fa5700efdbd01ae697f63e089/revisions#diff-cb70bad5acc04a28940d30ec97742299" rel="noopener nofollow ugc">proceeded to change</a> the line of code you referred above removing the <code>.item()</code> and changed <a href="https://gist.github.com/muammar/6d298f7fa5700efdbd01ae697f63e089#file-neuralnetwork-py-L202" rel="noopener nofollow ugc">L-202</a> to <code>outputs = torch.tensor(outputs, requires_grad=True)</code>. I still get the same problem. Then it means that the tensors I am using to compute the loss function are broken. Is that correct?</p>
<blockquote>
<p>I’m not completely understanding the code, so let me know, if I’m on the wrong path. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
</blockquote>
<p>Your analysis is right. Let me just give a brief idea about what is going on.  These <code>targets</code> are two energies (scalars) of two molecules with 4 atoms each. The inputs, in this case, are 4 vectors for each each molecule because there are 4 atoms (<a href="https://gist.github.com/muammar/6d298f7fa5700efdbd01ae697f63e089#file-neuralnetwork-py-L304" rel="noopener nofollow ugc">see here</a>). Applying <code>forward</code> to those features will return atomic energies. That is why I sum them, add their sum (<code>image_energy</code>) to the <code>outputs</code> array, and then pass them to the loss. I probably am failing to see how to avoid summing those 4 atomic energies to recover the total energy and avoid detaching and breaking the graph calculation.</p>

ptrblck: <p>Thanks for the explanation!<br>
Your experiment sounds really interesting.</p>
<p>Rewrapping <code>output</code> in a <code>tensor</code> also detaches the graph.<br>
Your code should work with the following changes:</p>
<pre><code class="lang-python">...
     outputs.append(image_energy)
outputs = torch.stack(outputs)
loss, rmse = self.get_loss(outputs, targets, 4)
</code></pre>
<p>Could you check for valid results?</p>

muammar: <p>I just checked my commits and I was using  <code>torch.stack</code> but dropped it at some point. With that change, now I am getting this (thanks for spotting that!);</p>
<pre><code class="lang-auto">outputs
tensor([[-14.5754413605],
        [-14.5754413605]], grad_fn=&lt;StackBackward&gt;)
targets
tensor([-14.5868730545, -14.5640010834])
Diff in intercept_Cu
Diff in slope_Cu
Diff in linears.Cu.0.weight
Diff in linears.Cu.0.bias
Diff in linears.Cu.2.weight
Diff in linears.Cu.2.bias
Diff in linears.Cu.4.weight
Diff in linears.Cu.4.bias

Optimized parameters for Cu symbol
Index 0
Parameter containing:
tensor([[-3.2861328236e-06,  2.6473844628e-05,  3.5650995045e-08,
          1.4864326658e-06, -1.0720622959e-04,  3.0406141605e-08,
          1.3593539488e-11, -1.9859398570e-09],
        [-5.8556226534e-10, -1.1646034137e-10, -3.3600910682e-12,
          6.7106892265e-10,  1.8430428624e-13,  1.9599950107e-08,
          2.4708364435e-05,  3.2043487863e-07],
        [ 1.9733222143e-04,  1.3596744564e-10,  1.6436105810e-08,
         -3.0531379647e-09, -6.3785437305e-06,  7.4555270811e-13,
          8.1752958067e-05, -1.1181630725e-05],
        [-1.2528111881e-09,  4.6580535127e-06,  2.3549859979e-12,
          2.8091712984e-11,  1.7993905931e-04, -1.4886735508e-12,
         -9.2567507479e-12,  2.8859590202e-06],
        [ 3.4168685943e-06, -5.6807679357e-05,  1.2368669559e-05,
         -2.1798576399e-12, -1.4500128600e-05, -3.1362407071e-07,
          1.7807322283e-09,  9.7959136838e-06],
        [ 8.9927290503e-08, -5.2266013739e-09, -9.1957379753e-14,
         -2.9821003977e-07, -1.7513568764e-06,  1.0372443600e-13,
          1.2319574694e-07, -3.6574114347e-05],
        [ 5.3554540500e-06, -1.8524660845e-05,  1.1853338037e-05,
         -2.1492420638e-04,  2.3621556466e-05, -8.0939061009e-11,
         -6.9240194023e-08,  1.5314364646e-05],
        [-2.0754782781e-08, -1.9774879547e-05, -1.3601642422e-05,
          5.2368657634e-05,  3.3497635741e-05, -5.8766081565e-06,
         -6.5623047703e-05,  3.9108752389e-05],
        [-1.5761332861e-06, -7.3087621786e-06, -2.9493878628e-07,
         -4.5463502829e-07, -3.2682427786e-07, -1.4819252101e-05,
          2.6041425372e-05, -1.0358776308e-06],
        [-6.6475655558e-07, -1.3438479496e-10,  1.8068027430e-07,
         -2.5042306007e-09, -5.2879945542e-06, -6.9557786446e-06,
          2.9763690179e-08, -2.8302894425e-14]], requires_grad=True)
Gradient tensor(-6.9187924964e-05)

Index 1
Parameter containing:
tensor([ 0.0316366255,  0.2617242932,  0.3015798032, -0.0021922502,
        -0.0615932010, -0.2602513433, -0.0311477333, -0.2361671627,
         0.1662444025, -0.0660640150], requires_grad=True)
Gradient tensor(2.4240243301e-16)

Index 2
Parameter containing:
tensor([[ 9.7365699503e-08,  4.2661922635e-06,  1.9631031591e-06,
         -4.2697833123e-05,  6.2208728195e-06,  1.1655485604e-12,
         -6.4603467763e-05, -1.3209832117e-09,  1.1756450391e-07,
         -7.9867913882e-07],
        [ 6.2096376041e-07, -6.1568898673e-06, -7.3711348136e-10,
          3.8170369088e-09, -1.4567660855e-06, -1.9914123186e-06,
          2.3581033020e-06, -1.4400919781e-06,  1.2830110308e-09,
         -9.6331113753e-08],
        [-1.5564822888e-06,  2.5510303203e-06,  1.5743958670e-07,
         -1.4166996607e-06, -1.3405845323e-07, -9.9951203083e-06,
          1.0537170965e-05,  7.7086369856e-06,  2.7015998813e-11,
          4.8960191457e-10],
        [-1.7156789909e-06, -2.3564821277e-06,  1.5615292262e-13,
          3.7418202217e-13, -4.3808788178e-05, -1.6505031454e-05,
         -8.4225684986e-06,  4.8483889259e-06, -5.0767212656e-08,
          2.3069132737e-07],
        [ 2.8512771678e-05, -7.5202997323e-06,  1.7333918549e-06,
          1.2562820473e-07,  5.2780844271e-05, -1.1339360562e-09,
          3.8166854659e-13, -1.2022780993e-07,  7.6206299127e-05,
          1.7692066194e-06],
        [-1.8082403130e-07,  1.8067876226e-06,  1.5731624337e-10,
         -3.9476603410e-12, -1.6683844706e-07,  1.6850806333e-06,
         -2.5483970489e-10, -1.8325088604e-05, -3.7972899918e-06,
          1.8083280651e-08],
        [ 1.2292147552e-12,  9.3183753052e-06, -1.0420450280e-07,
          3.0822411645e-07,  6.6431852019e-07,  9.7349062145e-10,
          3.3600823372e-05, -2.3434172908e-04, -4.7051515462e-11,
          3.7607719605e-08],
        [ 2.9142852020e-10,  1.6880323983e-07,  3.4797506032e-06,
          2.2227823138e-08, -8.4504938513e-07, -1.0985943663e-04,
          2.5039498723e-05,  8.6511966211e-13,  1.6281462740e-04,
         -2.8856720746e-07],
        [-2.9593747968e-06, -5.8458951457e-08, -5.6971380502e-08,
         -1.2519759184e-04, -7.4558295735e-13, -2.9341401842e-07,
          4.2668673927e-08,  4.3059226300e-06, -2.8244965478e-08,
         -5.0044291129e-06],
        [ 3.0511776004e-06, -1.0126113713e-15,  5.8472587625e-06,
         -7.7287486420e-06, -1.2084972241e-06, -4.0337028162e-10,
          7.3831834015e-05,  1.5755430240e-06,  1.3774927379e-12,
          1.3072969159e-04]], requires_grad=True)
Gradient tensor(-0.0001017772)

Index 3
Parameter containing:
tensor([ 0.1047996879, -0.2729528248,  0.2065724581,  0.0899153724,
        -0.1029036865, -0.3049356043, -0.2874263823,  0.1054596379,
        -0.2320392430, -0.0505143851], requires_grad=True)
Gradient tensor(-1.4201264947e-12)

Index 4
Parameter containing:
tensor([[ 1.3515332284e-06,  1.9205540269e-08,  1.9853887352e-05,
          2.2419371817e-06,  1.0494016323e-08, -4.0517086745e-05,
          1.4675807324e-04,  5.4397496285e-08, -6.9757788879e-08,
          3.3906806038e-06]], requires_grad=True)
Gradient tensor(0.0001229179)

Index 5
Parameter containing:
tensor([-0.2859740555], requires_grad=True)
Gradient tensor(-7.3175310256e-09)

</code></pre>
<p>The two variables that were not changing before now they are changing. However, outputs remained the same even though the loss function value at epoch 1000 was <code>1.643620e-05</code>. Do you know any idea why? Doesn’t that mean that outputs are very near to the targets? Meanwhile, I am playing with learning rate and weight decay and see what happens.</p>

ptrblck: <p>So while the loss decreases the outputs stay approx. the same? Let me know if that’s the case and I do some more debugging.</p>
<p>One issue that comes to my mind is that we’ve recently had similar problems using <code>nn.MSELoss</code> when the model output and target had a shape mismatch and we’re silently broadcasted. Could you check the shapes of all tensors used in <code>nn.MSELoss</code>?</p>

muammar: <aside class="quote no-group" data-username="ptrblck" data-post="14" data-topic="34947">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>So while the loss decreases the outputs stay approx. the same? Let me know if that’s the case and I do some more debugging.</p>
</blockquote>
</aside>
<p>That is exactly the case. While the loss function is decreasing the outputs remain the same.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="14" data-topic="34947">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>One issue that comes to my mind is that we’ve recently had similar problems using <code>nn.MSELoss</code> when the model output and target had a shape mismatch and we’re silently broadcasted. Could you check the shapes of all tensors used in <code>nn.MSELoss</code> ?</p>
</blockquote>
</aside>
<p>OK! I checked, and the shapes are not the same!</p>
<pre><code class="lang-auto">output = torch.Size([2, 2])
target = torch.Size([2])
</code></pre>
<p>Why stack is changing the shape of the output tensor??? What can I use instead of <code>stack</code>? I really appreciate all your help!!</p>

ptrblck: <p>Great! If you don’t want an additional dimension, you could use <code>torch.cat</code>, but you would end up with 4 values nevertheless so there is a shape mismatch indeed.<br>
If you are dealing with energies for 4 atoms, I would expect the target to also have 4 values. Does it make sense or what am I still missing something? <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"></p>

muammar: <aside class="quote no-group" data-username="ptrblck" data-post="16" data-topic="34947">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Great! If you don’t want an additional dimension, you could use <code>torch.cat</code> , but you would end up with 4 values nevertheless so there is a shape mismatch indeed.<br>
If you are dealing with energies for 4 atoms, I would expect the target to also have 4 values. Does it make sense or what am I still missing something? <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
</blockquote>
</aside>
<p>It makes sense, but I don’t have access to energies per atom :(. The energy of a molecular system comes from something we call the wave function and solving the Schrödinger equation using that function returns only total energies and not atomic ones.</p>
<p>Now with <code>cat</code> sizes match:</p>
<pre><code class="lang-auto">torch.Size([2])
torch.Size([2])
</code></pre>
<p>But the loss keeps decreasing without the outputs changing. Do you think I should change the forward method?</p>

ptrblck: <p>Thanks for the explanation. I remember some things from high school about the energy levels etc.</p>
<p>I’m not completely sure what’s going on as I would assume your <code>output</code> should now contain 4 elements.<br>
I would like to debug it a bit later this evening as I currently don’t have access to my machine.</p>

muammar: <p><img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"> thanks! I will keep  changing things here and there. I am very interested in understanding what is causing this issue.</p>

ptrblck: <p>I tried to debug the code a bit more and in my code the shapes of the output and target were <code>[2, 1]</code> and <code>[2]</code>, respectively.<br>
To fix this, I just created the target as:</p>
<pre><code class="lang-python">targets = [[-14.586873530850994], [-14.56400104603344]]
</code></pre>
<p>Also, it seems you are passing the input in a shape of <code>[8]</code>. A linear layer would expect an input of <code>[batch_size, nb_features]</code>, so I unsqueezed the tensor in dim0:</p>
<pre><code class="lang-python">X = X.unsqueeze(0)
X = self.linears[symbol](X)
</code></pre>
<p>The output values are really close to each other and sometimes have even the same values.<br>
If I just change the targets randomly to have a larger distance, the output values also diverge a bit, but generally your input and target seem to contain the “signal” in a low precision range.<br>
Have you thought about some normalization scheme?<br>
I’m a bit afraid that we are currently running into floating point precision issues (<code>~1e-6</code> would be the limit).<br>
However, even with float64 precision, I couldn’t really fit the data.</p>
<pre><code class="lang-python">loss 9.11804408893418e-6, rmse 0.011436155542466114
</code></pre>