Sobolev training loss

gallo: <p>I would like to test the loss proposed in the <a href="https://arxiv.org/pdf/1706.04859.pdf" rel="noopener nofollow ugc">Sobolev training paper</a>. For example consider: <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/32d78e8c285a4c09e97f57bbb3f17deea37743a9.png" data-download-href="https://discuss.pytorch.org/uploads/default/32d78e8c285a4c09e97f57bbb3f17deea37743a9" title="loss"><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/32d78e8c285a4c09e97f57bbb3f17deea37743a9.png" alt="loss" data-base62-sha1="7fLHw4DGXBzfquBDufctpAgWI2d" width="690" height="92" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss</span><span class="informations">824×110 8.72 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>where m(x_i,\theta) is the network evaluated at x_i.</p>
<p>The first part is the usual MSE but I have no idea how to go for the second piece.</p>

JuanFMontesinos: <p>In practical examples they use first order derivative which actually is what you get with the framework right now. If you are trying to learn a function f (which in this case is a  already trained network) you can compute gradients of the output wrt input. and m(x|O) is the network which is learning to approximate f.</p>
<p>In short, they apply an MSE not only over output but over gradients too.</p>

Tony-Y: <aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/Tony-Y/PotFit" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/11532812?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/Tony-Y/PotFit" target="_blank" rel="nofollow noopener">Tony-Y/PotFit</a></h3>

<p>A PyTorch Example of Potential Fitting . Contribute to Tony-Y/PotFit development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>This is my example that employs the derivative error as well as the MSE.</p>

gallo: <p>Thank you. This is what I was looking for.</p>
<p>At line 95 in <em>potfit.py</em> you have:</p>
<p><code>output0.sum().backward(retain_graph=True, create_graph=True)</code></p>
<p>why the  <code>.sum()</code> is needed?</p>
<aside class="quote no-group quote-modified" data-username="Tony-Y" data-post="3" data-topic="44867" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tony-y/48/8373_2.png" class="avatar"> Tony-Y:</div>
<blockquote>
<p><a href="https://github.com/Tony-Y/PotFit" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Tony-Y/PotFit: A PyTorch Example of Potential Fitting</a></p>
<p>This is my example that employs the derivative error as well as the MSE.</p>
</blockquote>
</aside>

Tony-Y: <p><a href="https://pytorch.org/docs/stable/autograd.html#torch.Tensor" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/autograd.html#torch.Tensor</a></p>
<p>If the sum() is not used, the backward() requires a gradient of the scalar function wrt Self.</p>