Adding a float variable as an additional loss term works?

username: TheIllusion
cooked: <p>Suppose I have a loss term as shown below:</p>
<pre><code>G_sample = G(z)
D_fake = D(G_sample)

# Generator loss
G_loss = 0.5 * torch.mean((D_fake - 1)**2)

G_loss.backward()
G_solver.step()
reset_grad()
</code></pre>
<p>If I add any ordinary float variable to G_loss before .backward(),<br>
for example,<br>
G_loss = G_loss + avg_pixel_value</p>
<p>Does avg_pixel_value has any effect when doing back prop?<br>
Or it is completely ignored?</p>
<p>Since avg_pixel_value itself cannot use .backward() api, I have a doubt of its effectiveness.</p>
<p>My question is can I add ordinary float terms to PyTorch Loss?</p>
<p>Many thanks.</p>

username: jpeg729
cooked: <p>To differentiate a sum of two terms, you differentiate each term separately. Hence adding a constant value to your loss has no effect whatsoever on the gradients produced by backpropagation.</p>

username: TheIllusion
cooked: <p>Thanks for the reply.</p>
<p>Even though the additional term I mentioned doesn’t support .backward(), it’s not a constant but a variable(python float).</p>
<p>In my sample code,<br>
I think avg_pixel_value might affect the value of G_loss.<br>
(avg_pixel_value changes every iteration)</p>
<p>G_loss = G_loss + avg_pixel_value</p>
<p>Is there any faults on my understanding?</p>

username: jpeg729
cooked: <p>avg_pixel_value is calculated from the input image without using any of the model parameters or outputs, so for backpropagation with respect to the model parameters, it is ignored.</p>
<p>Might I suggest a <a href="https://youtu.be/GZTvxoSHZIo?list=PLlJy-eBtNFt6EuMxFYRiNRS07MCWN5UIA&amp;t=192">lecture on backpropagation by Andrej Karpathy</a>.</p>

username: TheIllusion
cooked: <p>Thanks for the answer.</p>
<p>What if I use model’s output as blow:<br>
G_loss = G_loss + mean_output_pixel_value</p>
<p>But mean_output_pixel_value is merely a float variable that doesn’t have graph information.</p>
<p>Can mean_output_pixel_value affect back propagation?</p>

username: jpeg729
cooked: <p>Well, when you do G_loss.backward() pytorch first looks at what operation produced G_loss, in this case</p>
<pre><code class="lang-auto">G_loss = G_loss + mean_output_pixel_value
</code></pre>
<p>so pytorch retrieves the values that were summed and basically runs backward on both of them.<br>
But <code>mean_output_pixel_value</code> is either a float value which has no <code>.backward()</code> method, or it is a Variable calculated from <code>input_image</code> which is a Variable with requires_grad=False. Therefore <code>mean_output_pixel_value.backward()</code> does nothing.</p>
<p>I shall try another explanation… The reason for running <code>G_loss.backward()</code> is to calculate a measure of how each parameter affected the value of G_loss, this measure is stored in <code>param.grad</code> for each parameter.</p>
<p>Now obviously, the model parameters have no effect whatsoever on the <code>mean_output_pixel_value</code>, so adding it to <code>G_loss</code> will not change the calculated <code>param.grad</code> values in any way at all.</p>

username: TheIllusion
cooked: <p>I’m still confused.</p>
<p>After run<br>
G_loss = G_loss + mean_output_pixel_value,</p>
<p>G_loss will be updated.</p>
<p>Back propagation will be computed via multiplying G_loss and the input values of each later layers.</p>
<p>That’s why I thought that the updated G_loss will have some effect during backprop.</p>

username: jpeg729
cooked: <p>When you write <code>G_loss = G_loss + mean_output_pixel_value</code> python first calculates <code>G_loss + mean_output_pixel_value</code> and stores the result in a <em>new</em> tensor, then python updates the name G_loss to point to this result.</p>
<p>That line does not update the data stored in G_loss. It basically acts like this: <code>new_G_loss = old_G_loss + mean_output_pixel_value</code>.</p>
<p>If you wanted to do an inplace addition, you could do</p>
<pre><code class="lang-auto">G_loss += mean_output_pixel_value
</code></pre>
<p>but that would either be non-differentiable, or pytorch would assume you meant <code>new_G_loss = old_G_loss + mean_output_pixel_value</code></p>

username: TheIllusion
cooked: <p>Thanks for the great answer</p>

username: Surya_Narayanan
cooked: <p>I have a question along these lines</p>
<p>I have a loss and a penalty term</p>
<p>so something like <code>loss = function(inputs) + lambda * penalty(inputs)</code> where function and penalty are both functions of my inputs and <code>lambda</code> is a constant</p>
<p>However, I notice that my loss term behaves independently of lambda, i.e. it doesn’t change the learning trajectory despite me changing lambda.</p>
<p>Any thoughts why?</p>

username: ptrblck
cooked: <p>Are you able to call <code>loss.backward()</code> if only <code>penalty(inputs)</code> is used?<br>
If not, this would indicate that <code>penalty</code> is detaching the <code>inputs</code> and might create a constant which will not influence the gradients.<br>
On the other hand, if <code>loss.backward()</code> still works and creates valid gradients in the parameters of the model, could you post a minimal and executable code snippet to reproduce the issue?</p>