Recreating CURE curvature regularizer

username: Nate
cooked: <p>Hello, I am attempting to recreate the CURE regularizer (<a href="https://arxiv.org/abs/1811.09716" class="inline-onebox" rel="noopener nofollow ugc">[1811.09716] Robustness via curvature regularization, and vice versa</a>) and have a question about how to compute the gradient of the regularization term. Specifically, the regularizer is</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/5/65f1435b2e2ef9d2c04731435df821c7c51e8307.png" alt="image" data-base62-sha1="exP7Xm8vX6IHio5ESBp303vm7TV" width="343" height="42"></p>
<p>and is used in conjunction with the standard cross entropy loss of a DNN classifier. I am following the implementation (from snippet) and may be misunderstanding the key line for the gradient computation. Here is how the repo computes the regularizer:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/F-Salehi/CURE_robustness/blob/daeef24ea443c37a55f87f93679a44d8504d88e7/CURE/CURE.py#L200-L217">
  <header class="source">

      <a href="https://github.com/F-Salehi/CURE_robustness/blob/daeef24ea443c37a55f87f93679a44d8504d88e7/CURE/CURE.py#L200-L217" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/F-Salehi/CURE_robustness/blob/daeef24ea443c37a55f87f93679a44d8504d88e7/CURE/CURE.py#L200-L217" target="_blank" rel="noopener nofollow ugc">F-Salehi/CURE_robustness/blob/daeef24ea443c37a55f87f93679a44d8504d88e7/CURE/CURE.py#L200-L217</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="200" style="counter-reset: li-counter 199 ;">
          <li>def regularizer(self, inputs, targets, h = 3., lambda_ = 4):</li>
          <li>    '''</li>
          <li>    Regularizer term in CURE</li>
          <li>    '''</li>
          <li>    z, norm_grad = self._find_z(inputs, targets, h)</li>
          <li>    </li>
          <li>    inputs.requires_grad_()</li>
          <li>    outputs_pos = self.net.eval()(inputs + z)</li>
          <li>    outputs_orig = self.net.eval()(inputs)</li>
          <li></li>
          <li>    loss_pos = self.criterion(outputs_pos, targets)</li>
          <li>    loss_orig = self.criterion(outputs_orig, targets)</li>
          <li>    grad_diff = torch.autograd.grad((loss_pos-loss_orig), inputs, grad_outputs=torch.ones(targets.size()).to(self.device),</li>
          <li>                                    create_graph=True)[0]</li>
          <li>    reg = grad_diff.reshape(grad_diff.size(0), -1).norm(dim=1)</li>
          <li>    self.net.zero_grad()</li>
          <li></li>
          <li>    return torch.sum(self.lambda_ * reg) / float(inputs.size(0)), norm_grad</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>My question is, is this:</p>
<pre><code class="lang-auto">grad_diff = torch.autograd.grad((loss_pos-loss_orig), inputs, grad_outputs=torch.ones(targets.size()).to(self.device), create_graph=True)[0]
</code></pre>
<p>the correct statement for the gradient computation involved in the regularization term? At first glance, it appears to be allowing for the computation of the gradient of the difference of the losses, rather than the difference of the gradients. But, I may be misinterpreting this line completely.</p>
<p>Any comments on if this is the correct/incorrect way to compute the regularization term and potential solutions or improvements would be greatly appreciated.</p>
<p>Thanks,<br>
Nate</p>

username: tom
cooked: <aside class="quote no-group" data-username="Nate" data-post="1" data-topic="58214">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nate/48/16675_2.png" class="avatar"> Nate:</div>
<blockquote>
<p>the correct statement for the gradient computation involved in the regularization term? At first glance, it appears to be allowing for the computation of the gradient of the difference of the losses, rather than the difference of the gradients. But, I may be misinterpreting this line completely.</p>
</blockquote>
</aside>
<p>The gradient operator ∇ is in itself linear as an operator. This means that ∇a - ∇b = ∇(a-b).<br>
In fact, if you write a minus function m(x1, x2) = x1 - x2 and some loss l(m) calculated from that, then the backpropagation step for this calculation is just feeding dl/dm as grad_out into the backward of x1 and -dl/dm as grad_out into the backward of x2. So really, the two not only agree mathematically but also computationally, with some efficiency gain if the two have common intermediates, so that the graph of the calculation of the intermediates has to be only backpropagated through once.</p>
<p>Best regards</p>
<p>Thomas</p>