Different samples use different learning rates

username: shirui-japina
cooked: <p>I want to use different learning rates for different input samples to carry out model training. Any good suggestions?<img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>

username: shirui-japina
cooked: <p>The way I think of is:</p>
<p><em>Assuming using <code>nn.BCELoss()</code> as loss function.</em><br>
As <a href="https://pytorch.org/docs/master/nn.functional.html#loss-functions" rel="noopener nofollow ugc">here</a> said,</p>
<blockquote>
<p><code>'none'</code> : no reduction will be applied</p>
</blockquote>
<p>So I can get the loss value of each sample by:<br>
<code>loss_reduction_none = nn.BCELoss(reduction='none')</code></p>
<p>Then I can adjust the loss value of each sample like:</p>
<pre><code class="lang-auto">loss_reduction_none[1] = criterion_reduction_none[1] * scaling_factor_1,
loss_reduction_none[0] = criterion_reduction_none[2] * scaling_factor_2,
...
loss_reduction_none[n] = criterion_reduction_none[n] * scaling_factor_n,
</code></pre>
<p>Because the <em><strong>learning rate</strong></em> works in this way.</p>
<blockquote>
<p>w_new = w_old - <em><strong>learning_rate</strong></em> * (∂ loss / ∂ weight)</p>
</blockquote>
<p>As I mentioned above, I adjust the loss function value of each sample.<br>
That indirectly solves the problem of learning rate adjustment for each sample.</p>
<p>Then I can use backward and step function:<br>
<code>loss_reduction_none.backward()</code><br>
<code>optimizer.step()</code></p>
<p><strong>But I am not sure if PyTorch will use the loss function value which I have modified.</strong><br>
<strong>Actually, I don’t know the parameter <em>reduction</em> is designed for what kind of demand.</strong></p>
<p>Is this the right answer?<img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>
<p>Maybe this question is something like below:</p><aside class="quote quote-modified" data-post="1" data-topic="7350">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/bcef8e/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-are-optimizer-step-and-loss-backward-related/7350">How are optimizer.step() and loss.backward() related?</a> 
  </div>
  <blockquote>
    Hi. 
I am pretty new to Pytorch and keep surprised with the performance of Pytorch <img width="20" height="20" src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title="slight_smile" alt="slight_smile" class="emoji"> 
I have followed tutorials and there’s one thing that is not clear. 
How the optimizer.step() and loss.backward() related? 
Does optimzer.step() function optimize based on the closest loss.backward() function? 
When I check the loss calculated by the loss function, it is just a Tensor and seems it isn’t related with the optimizer. 
Here’s my questions: 
(1) Does optimzer.step() function optimize bas…
  </blockquote>
</aside>


username: Oli
cooked: <p>I second <a class="mention" href="/u/shirui-japina">@shirui-japina</a>’s suggestions. Do a <code>nn.LossFn(reduction='none')</code>, weigh the different samples to your liking and then do your own reduction (e.g. mean).</p>

username: shirui-japina
cooked: <p>Thanks for your reply.</p>
<blockquote>
<p>Do a  <code>nn.LossFn(reduction='none')</code> , weigh the different samples to your liking</p>
</blockquote>
<p>If the method above is correct (In fact, I think that’s correct),<br>
but why I need to <strong>do reduction (e.g. mean)</strong>?<br>
If I don’t do that, what will happen?</p>
<p>From mathematical theory</p>
<ul>
<li>
<p><em><strong>Batch Gradient Descent</strong></em><br>
When the number of samples is large, the training process will be slow.</p>
</li>
<li>
<p><em><strong>Stochastic Gradient Descent</strong></em><br>
Decreased accuracy, not globally optimal.</p>
</li>
<li>
<p><em><strong>Mini-batch Gradient Descen</strong></em><br>
Use a subset of samples to update each parameter.</p>
</li>
</ul>
<p>Usually we use <em><strong>Mini-batch Gradient Descen-like</strong></em> to update the parameters. Which means we get the average value of the loss function to update parameters.<br>
<strong>Is the process of averaging (do reduction (e.g. mean)) equivalent to this process?</strong><br>
Or <strong>the process of averaging (do reduction (e.g. mean))</strong> is just for PyTorch operating mechanism?</p>

username: Oli
cooked: <aside class="quote no-group" data-username="shirui-japina" data-post="4" data-topic="58234">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shirui-japina/48/15285_2.png" class="avatar"> shirui-japina:</div>
<blockquote>
<p>Usually we use <em><strong>Mini-batch Gradient Descen-like</strong></em> to update the parameters. Which means we get the average value of the loss function to update parameters.<br>
<strong>Is the process of averaging (do reduction (e.g. mean)) equivalent to this process?</strong></p>
</blockquote>
</aside>
<p>Yup <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20">  We have a loss for each sample in your batch, then mean/average that to get <strong>one</strong> loss-float which we can use to backpropagate. (edit: doesn’t need to be mean, but other reduction techniques such a sum)</p>

username: shirui-japina
cooked: <p>Thanks for your reply.<img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
I think I thoroughly understood this problem.<img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: shirui-japina
cooked: <p>I have reconsidered it. And found some problems.</p><aside class="quote quote-modified" data-post="2" data-topic="7350">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/898d66/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-are-optimizer-step-and-loss-backward-related/7350/2">How are optimizer.step() and loss.backward() related?</a> 
  </div>
  <blockquote>
    optimizer.step is performs a parameter update based on the current gradient (stored in .grad attribute of a parameter) and the update rule. As an example, the update rule for SGD is defined here: 
<a href="https://github.com/pytorch/pytorch/blob/cd9b27231b51633e76e28b6a34002ab83b0660fc/torch/optim/sgd.py#L63" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/cd9b27231b51633e76e28b6a34002ab83b0660fc/torch/optim/sgd.py#L63</a>. 


Calling .backward() mutiple times accumulates the gradient (by addition) for each parameter. This is why you should call optimizer.zero_grad() after each .step() call. Note that following the f…
  </blockquote>
</aside>
<p>
Topic above describes the operating mechanism of .backward() and optimizer.step() in PyTorch.<br>
Simple understanding is:</p>
<ul>
<li>
<p><em><strong>.backward()</strong></em><br>
Get the gradient for the parameters in model.</p>
</li>
<li>
<p><em><strong>optimizer.step()</strong></em><br>
performs a parameter update based on the  <em>current</em>  gradient (stored in <code>.grad</code> attribute of a parameter) and the update rule.</p>
</li>
</ul>
<p>I mean, the function  <em><strong>optimizer.step()</strong></em> considers the update method ,and actually update the parameters, but not PyTorch users to consider the update method. (<strong>We don’t have to do reduction (e.g. mean) or something by manual</strong>)</p>
<p>What we should do is,</p>
<ol>
<li>
<p>Before the function .backward(), adjust the loss function value of each sample.</p>
</li>
<li>
<p>Then get gradient for the parameters in model by .backward().</p>
</li>
<li>
<p>At the end, update the parameters based on the  <em>current</em>  gradient by optimizer.step().</p>
</li>
</ol>
<p>The parameter in LossFn() <em><strong>reduction</strong></em> can set as: <code>'none'</code>  |  <code>'mean'</code>  |  <code>'sum'</code>.<br>
But not here to get the average of loss function value for the parameters updating.<br>
(Although I don’t know what these <code>'mean'</code>  |  <code>'sum'</code> are for.)</p>

username: Oli
cooked: <p>If you set the reduction flag to ‘none’ and try to do the backward() on the un-reduced loss, you get this error message</p>
<p><code>RuntimeError: grad can be implicitly created only for scalar outputs</code></p>

username: shirui-japina
cooked: <p><img src="https://discuss.pytorch.org/images/emoji/apple/scream.png?v=9" title=":scream:" class="emoji" alt=":scream:"><img src="https://discuss.pytorch.org/images/emoji/apple/scream.png?v=9" title=":scream:" class="emoji" alt=":scream:"><img src="https://discuss.pytorch.org/images/emoji/apple/scream.png?v=9" title=":scream:" class="emoji" alt=":scream:"></p>