Cost function repeats itself at each epoch... Why?

Capo_Mestre: <p>Hello,</p>
<p>I have defined a <a href="https://discuss.pytorch.org/t/runtime-error-does-not-make-sense/91997/3">DenseNet</a> (please follow the link for details) and a custom Loss-function <code>MSE_mod</code> as shown below:</p>
<pre><code class="lang-auto"># mean squared error with explicit const and linear terms
def MSE_toOptimize(params,yHat,y):
    y0,y1 = params
    x = [i for i in range(yHat.size()[1])]
    x = torch.tensor(x).to('cuda:0')
    size = yHat.size()[0]*yHat.size()[1]
    diff = yHat + y0 + y1*x - y
    res_ = torch.sum(diff**2) / size
    return res_.cpu().detach().numpy()

def optimizedParametersAre(yHat,y):
    guess = [0.,0.]
    res = minimize(MSE_toOptimize, guess, args=(yHat,y))
    params = res.x
    return torch.tensor(params).to('cuda:0')

def MSE_mod(yHat,y):
    params = optimizedParametersAre(yHat,y)
    loss__ = MSE_toOptimize(params,yHat,y)
    loss__ = torch.tensor(loss__).to('cuda:0').requires_grad_(True)
    return loss__
</code></pre>
<p>This function <code>MSE_mod</code> integrates two other functions.</p>
<ol>
<li>
<p><code>MSE_toOptimize</code> is the function bearing the expression for the Loss-function itself. As seen, the Loss-function is of the form <code>sum((yHat + y0 + y1*x - y)**2)</code>, where <code>x</code> is the vector of pixels, and <code>y0</code> and <code>y1</code> are parameters which are yet undefined. I want to have such values of these two parameters, so that the current expression for the Loss-function was minimized for given <code>yHat</code> and <code>y</code> (<code>yHat</code> is the predicted vector, and <code>y</code> is the ground-truth vector).</p>
</li>
<li>
<p><code>optimizedParametersAre</code> is the function where the optimization takes place. This function returns the values of <code>y0</code> and <code>y1</code> that minimize the Loss-function for given input vectors <code>yHat</code> and <code>y</code>.</p>
</li>
</ol>
<p>After training for a while (on a GPU) with such a Loss-function, I am getting the following results (the batch size is 100, the number of batches per epoch is 150):</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/0/a0a0009e362e9015ce1793b4bf0af369112705bc.png" data-download-href="https://discuss.pytorch.org/uploads/default/a0a0009e362e9015ce1793b4bf0af369112705bc" title="Capture2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a0a0009e362e9015ce1793b4bf0af369112705bc_2_690x201.png" alt="Capture2" data-base62-sha1="mUXckqE27FpKCn5pFGiNNhrXCb2" width="690" height="201" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a0a0009e362e9015ce1793b4bf0af369112705bc_2_690x201.png, https://discuss.pytorch.org/uploads/default/original/3X/a/0/a0a0009e362e9015ce1793b4bf0af369112705bc.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/0/a0a0009e362e9015ce1793b4bf0af369112705bc.png 2x" data-dominant-color="E4ECF1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture2</span><span class="informations">709×207 37.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The L-labeled axis shows the actual values of this custom Loss-function for each of the consecutive batch, whereas J-labeled axis shows the overall cost-function defined as the normalized sum of all Loss-functions over one epoch (there are 7 epochs in total).</p>
<p>The question is why the Loss-values within an epoch repeat each other from one epoch to another without showing any evolution over epochs?</p>

KFrank: <p>Hello Capo!</p>
<p>The short answer is that you have to rewrite your loss function to<br>
properly use autograd so that loss-function gradients can flow back<br>
through your model and can be used to update your model’s parameters</p>
<p>(The most straightforward way to do this is to implement the entirety of<br>
your loss-function computation with pytorch tensor operations, in which<br>
case you get the autograd machinery “for free.”)</p>
<p>[quote=“Capo_Mestre, post:1, topic:95790, full:true”]</p>
<pre><code class="lang-auto"># mean squared error with explicit const and linear terms
def MSE_toOptimize(params,yHat,y):
    ...
    x = torch.tensor(x).to('cuda:0')
    ...
    return res_.cpu().detach().numpy()

def optimizedParametersAre(yHat,y):
    ...
    return torch.tensor(params).to('cuda:0')

def MSE_mod(yHat,y):
    ...
    loss__ = torch.tensor(loss__).to('cuda:0').requires_grad_(True)
    return loss__
</code></pre>
<p>If you are indeed using the above loss function for the loss you<br>
backpropagate by calling <code>loss.backward()</code>, the backpropagation<br>
won’t work.</p>
<p>The various <code>.to('cuda:0')</code> calls “break the computation graph”<br>
(unless the tensors in question are already on <code>'cuda:0'</code>, in which<br>
case they’re a no-op), and <code>.detach().numpy()</code> in any event breaks<br>
(that part of) the computation graph.</p>
<p>Lastly, calling <code>.requires_grad_(True)</code> in:</p>
<pre><code class="lang-auto">loss__ = torch.tensor(loss__).to('cuda:0').requires_grad_(True)
</code></pre>
<p>doesn’t fix the problem.  It gives you a nice, new pytorch tensor that<br>
will track its gradient <em>going forward,</em> but the damage has already<br>
been done.</p>
<aside class="quote no-group">
<blockquote>
<p>The question is why the Loss-values within an epoch repeat each other from one epoch to another without showing any evolution over epochs?</p>
</blockquote>
</aside>
<p>As a consequence of the above, non-trivial gradients never flow back<br>
to your model parameters, so your call to <code>optimizer.step()</code> (which<br>
you don’t show, but I’m guessing you have) doesn’t actually update<br>
your model – your model remains unchanged throughout your training<br>
run.</p>
<p>You don’t show how you prepare your batches, but if you don’t shuffle,<br>
or otherwise randomize your training data when you create the batches,<br>
then all the epochs will be cycling through the same set of batch with<br>
the same model, hence giving you the same loss values over and over<br>
again.</p>
<p>(But because, within an epoch, one batch is different from the next,<br>
your loss function will vary within an epoch, but repeat from epoch<br>
to epoch.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

Capo_Mestre: <p>Thanks a lot <a class="mention" href="/u/kfrank">@KFrank</a>  for a great reply! Everything seems to make sense.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="95790">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>As a consequence of the above, non-trivial gradients never flow back<br>
to your model parameters, so your call to <code>optimizer.step()</code> (which<br>
you don’t show, but I’m guessing you have) doesn’t actually update<br>
your model – your model remains unchanged throughout your training<br>
run.</p>
</blockquote>
</aside>
<p>Yes, I do use this command in the training process.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="95790">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>You don’t show how you prepare your batches, but if you don’t shuffle,<br>
or otherwise randomize your training data when you create the batches,<br>
then all the epochs will be cycling through the same set of batch with<br>
the same model, hence giving you the same loss values over and over<br>
again.</p>
</blockquote>
</aside>
<p>Indeed, I don’t shuffle the training data.</p>
<p>I will try to think how to rewrite the code, so that my custom loss-function would work properly having in mind the understanding that your reply brought.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="95790">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>and <code>.detach().numpy()</code> in any event breaks<br>
(that part of) the computation graph.</p>
</blockquote>
</aside>
<p>The reason why I did <code>detach().numpy()</code> is that the function <code>minimize</code> was not happy about the <code>torch.tensor()</code> data format as the output of the function <code>MSE_toOptimize</code>.</p>

Capo_Mestre: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="95790">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>The various <code>.to('cuda:0')</code> calls “break the computation graph”<br>
(unless the tensors in question are already on <code>'cuda:0'</code> , in which<br>
case they’re a no-op), and <code>.detach().numpy()</code> in any event breaks<br>
(that part of) the computation graph.</p>
</blockquote>
</aside>
<p>Looking at the documentation, it seems that I should make use of the tensor.clone() function to keep connection with the computation graph. Would you confirm?</p>

Capo_Mestre: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="95790">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<pre><code class="lang-auto">def MSE_toOptimize(params,yHat,y):
    ...
    x = torch.tensor(x).to('cuda:0')
    ...
    return res_.cpu().detach().numpy()
</code></pre>
</blockquote>
</aside>
<p>Only when the function returns the result transfered on cpu, detached and converted to numpy, the <code>minimize</code> function can accept <code>MSE_toOptimize</code> as its first argument. This probably breaks the flow of computation graph, but I am not sure how to make <code>minimize</code> work otherwise. Would you be aware of a simple workaround solution?</p>

Capo_Mestre: <p><a class="mention" href="/u/kfrank">@KFrank</a></p>
<aside class="quote no-group" data-username="Capo_Mestre" data-post="5" data-topic="95790">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/capo_mestre/48/26690_2.png" class="avatar"> Capo_Mestre:</div>
<blockquote>
<p>Only when the function returns the result transfered on cpu, detached and converted to numpy, the <code>minimize</code> function can accept <code>MSE_toOptimize</code> as its first argument. This probably breaks the flow of computation graph, but I am not sure how to make <code>minimize</code> work otherwise. Would you be aware of a simple workaround solution?</p>
</blockquote>
</aside>
<p>I found a code <a href="https://gist.github.com/gngdb/a9f912df362a85b37c730154ef3c294b" rel="noopener nofollow ugc">here</a> that sounds like a workaround solution, but for now it is not quite transparent how to use it in my situation.</p>

KFrank: <p>Hello Capo!</p>
<aside class="quote no-group" data-username="Capo_Mestre" data-post="5" data-topic="95790" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/capo_mestre/48/26690_2.png" class="avatar"> Capo_Mestre:</div>
<blockquote>
<p>This probably breaks the flow of computation graph,</p>
</blockquote>
</aside>
<p>You are correct; this does break the computation graph.</p>
<aside class="quote no-group">
<blockquote>
<p>but I am not sure how to make <code>minimize</code> work otherwise.</p>
</blockquote>
</aside>
<p>In order to be able to backpropagate through the <code>minimize()</code> part<br>
of your loss-function computation, pytorch has to be able to get the<br>
gradient of <code>minimize()</code>'s output with respect to its input.</p>
<p>You have two choices:</p>
<p>You can write the <code>minimize()</code>* logic entirely with pytorch tensor<br>
operations, and let pytorch’s autograd calculate the gradients for<br>
you.</p>
<p>Or you can wrap the computation that uses <code>minimize()</code> in a<br>
<code>torch.autograd.Function</code>, work out the gradient of <code>minimize()</code><br>
“by hand,” and implement that gradient computation in the<br>
<code>.backward()</code> method of your <code>torch.autograd.Function</code>.</p>
<p>(Both approaches are likely to entail significant work on your part.<br>
If I were doing it, I would be lazy and try to cook up a loss function<br>
that doesn’t use <code>minimize()</code>, but still captures the important part<br>
of what <code>minimize()</code> is doing for you in some approximate way.)</p>
<p>*)  With the caveat that <code>minimize()</code> is a relatively complicated<br>
iterative algorithm, which can require care writing so that it builds a<br>
proper computation graph, and is likely to generate large, expensive<br>
computation graphs.</p>
<p>Good luck.</p>
<p>K. Frank</p>