Parameters not changed after optimized

username: chenglu
cooked: <p>I try to optimize some manually created parameters to see how the grad system work. But the parameters never changed for the same input, here is the code:</p>
<pre><code class="lang-auto">#include &lt;string&gt;
#include &lt;iostream&gt;
#include &lt;torch/torch.h&gt;

torch::Tensor forward_and_optimize_parameter(
    torch::Tensor&amp;&amp; input,
    std::vector&lt;torch::Tensor&gt;&amp; parameters,
    torch::optim::SGD&amp; optim) {
  auto loss = torch::sum(torch::pow(input + parameters.front(), 2));
  optim.zero_grad();
  loss.backward();
  optim.step(); // I expect the parameters being changed after this, but it's not, why?
  return loss;
}

int main() {
  auto options = torch::TensorOptions().requires_grad(true);
  std::vector&lt;torch::Tensor&gt; parameters{torch::rand(5, options)};
  torch::optim::SGD optimizer(parameters, /*lr=*/1);

  for (size_t i = 0; i &lt;= 10; i++) {
    auto loss = forward_and_optimize_parameter(torch::ones({5}), parameters, optimizer);
    std::cout&lt;&lt;loss&lt;&lt;std::endl; // the loss stays the same for the same input, the optimize is not working?
  }

  return 0;
}
</code></pre>

username: tom
cooked: <p>You win the “exceptionally bad luck in picking a small example” award <img src="https://discuss.pytorch.org/images/emoji/apple/trophy.png?v=9" title=":trophy:" class="emoji" alt=":trophy:">.</p>
<p>Your parameters do change, just your loss does not.</p>
<p>So what happens is that the gradient of <code>(x-1)**2</code> is <code>2*(x-1)</code> and with an SGD lr or 1, you get an oscillation between your initial value and 2-x which both happen to have the same loss value. With an lr of 0.1 or anything else below 1, your loss decreases as you would expect.</p>
<p>If you allow two unsolicited comments:</p>
<ul>
<li>Many people claim that <code>using std</code> is a bad habit. Given that many people will read your code on the forums, I’d advise against doing this.</li>
<li>Just 10 iterations would be fully sufficient so see the loss doesn’t change. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">
</li>
</ul>
<p>I will admit that I spent quite a while looking whether you somewhere made the parameter a non-leaf tensor by accident and only when I printed the gradient (to see if there was any at all) I noticed what happens.</p>
<p>Best regards</p>
<p>Thomas</p>

username: chenglu
cooked: <p>Thanks Tom you’re a life saver. Just as you said I changed the learning rate to 0.1 and everything is fine.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/c4fdc6dc875e0ce0a98762e347c3cda909f80abf.png" data-download-href="https://discuss.pytorch.org/uploads/default/c4fdc6dc875e0ce0a98762e347c3cda909f80abf" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/c/c4fdc6dc875e0ce0a98762e347c3cda909f80abf.png" alt="image" data-base62-sha1="s6FivqRdyZxEg39FmxbJMte36x1" width="305" height="499" data-dominant-color="374253"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">434×710 11.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Also have dropped the <code>using namespace std</code> and changed iteration number in the original question, thanks again for these great tips. I’m kind a newbie in CPP <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>