How to implement gradient penalty in PyTorch

username: chenyuntc
cooked: <p>I was reading <a href="https://arxiv.org/abs/1704.00028" rel="noopener nofollow ugc">Improved Training of Wasserstein GANs</a>, and thinking how it could be implemented in PyTorch. It seems not so complex but how to handle <strong>gradient penalty</strong> in loss troubles me.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/1X/133cfbfe07034d4bfa92d883cabff23061abe9df.png" data-download-href="https://discuss.pytorch.org/uploads/default/133cfbfe07034d4bfa92d883cabff23061abe9df" title="2017-04-05.png"><img src="https://discuss.pytorch.org/uploads/default/original/1X/133cfbfe07034d4bfa92d883cabff23061abe9df.png" width="690" height="121" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2017-04-05.png</span><span class="informations">709×125 6.71 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In the tensorflow’s implementation, the author use <code>tf.gradients</code>.</p><aside class="onebox githubblob" data-onebox-src="https://github.com/igul222/improved_wgan_training/blob/master/gan_cifar.py#L132-L136">
  <header class="source">

      <a href="https://github.com/igul222/improved_wgan_training/blob/master/gan_cifar.py#L132-L136" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/igul222/improved_wgan_training/blob/master/gan_cifar.py#L132-L136" target="_blank" rel="noopener nofollow ugc">igul222/improved_wgan_training/blob/master/gan_cifar.py#L132-L136</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="132" style="counter-reset: li-counter 131 ;">
          <li>gradients = tf.gradients(Discriminator(interpolates), [interpolates])[0]</li>
          <li>slopes = tf.sqrt(tf.reduce_sum(tf.square(gradients), reduction_indices=[1]))</li>
          <li>gradient_penalty = tf.reduce_mean((slopes-1.)**2)</li>
          <li>disc_cost += LAMBDA*gradient_penalty</li>
          <li></li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I wonder if there is an easy way to handle the gradient penalty.<br>
here is my idea of implementing it, I don’t know whether it will work and work in the way I think:</p>
<pre data-code-wrap="python"><code class="lang-python">
optimizer_D = optim.adam(model_D.parameters())
x = Variable()
y = Variable()
x_hat = (alpha*x+(1-alpha)*y).detach()
x_hat.requires_grad = True

loss_D = model_D(x_hat).sum()
loss_D.backward()
x_hat.grad.volatile = False

loss = model_D(x).sum() - model_D(y).sum() + ((x_hat.grad -1)**2 * LAMBDA).sum()
loss.backward()
optimizer_D.step()
</code></pre>

username: fmassa
cooked: <p>For the moment, it’s not yet possible to have gradients of gradients in PyTorch, but there is a <a href="https://github.com/pytorch/pytorch/pull/1016">pending PR</a> that will implement that and should be merged soon.</p>

username: chenyuntc
cooked: <p>Great! Can’t wait for that.</p>

username: adrian
cooked: <p><a class="mention" href="/u/chenyuntc">@chenyuntc</a>, did you figured out a way of doing this using pytorch? Thanks</p>

username: chenyuntc
cooked: <p><code>torch.autograd.grad</code>  would help.</p>

username: adrian
cooked: <p>Thank you, I had a look into this but from what I see torch doesn’t have yet support for higher-order derivates of Non-linear functions present in the DCGAN model. Or am I wrong?</p>

username: chenyuntc
cooked: <p>You are right, most function are still old-style which don’t support grad of grad.<br>
There is a temporary fix: use difference rather than differential<br>
<img src="https://discuss.pytorch.org/uploads/default/original/1X/c0708eec6775d377c5695eec7d1863453b552720.png" width="546" height="41"><br>
x_1,x_2 are sampled from x_hat<br>
idea from 郑华滨</p>

username: TheQuickBrownFox
cooked: <p>Have been struggling with this as well, could you provide an example of how it can be used?</p>

username: tom
cooked: <p>Ajay and I discussed that a bit a while ago and there is a link to a blog post and Jupyter notebook doing the toy examples from the <em>improved training</em> article in pytorch:</p><aside class="quote" data-post="24" data-topic="1275">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/wasserstein-loss-layer-criterion/1275/24">Wasserstein loss layer/criterion</a> 
  </div>
  <blockquote>
    Hi <a class="mention" href="/u/ajaytalati">@AjayTalati</a>, 
thanks for the pointers! I will definitely look into implementing an application or two. 
In the meantime, I <a href="http://lernapparat.de/improved-wasserstein-gan/" rel="noopener nofollow ugc">jotted down a few thoughts</a> regarding the <a href="https://www.arxiv.org/abs/1704.00028" rel="noopener nofollow ugc">Improved Training</a> paper for your amusement while we’re waiting for the grad of grad to be merged. <img width="20" height="20" src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title="slight_smile" alt="slight_smile" class="emoji"> Or maybe we find something to do with the original WGAN code, too. 
You have good holidays, 
best regards 
Thomas
  </blockquote>
</aside>

<p>Best regards</p>
<p>Thomas</p>

username: chenyuntc
cooked: <aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/caogang/wgan-gp" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/2537027?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/caogang/wgan-gp" target="_blank" rel="nofollow noopener">caogang/wgan-gp</a></h3>

<p>A pytorch implementation of Paper "Improved Training of Wasserstein GANs" - caogang/wgan-gp</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
<a class="mention" href="/u/caogang">@caogang</a> is working on it, looking forward to that.

username: caogang
cooked: <p>I have finish the toy dataset. You can refer to the implementation</p>

username: caogang
cooked: <p>Now I am working on the gan_language, gan_toy is finished. Hope it will be helpful<br>
</p><aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/caogang/wgan-gp" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/2537027?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/caogang/wgan-gp" target="_blank" rel="nofollow noopener">caogang/wgan-gp</a></h3>

<p>A pytorch implementation of Paper "Improved Training of Wasserstein GANs" - caogang/wgan-gp</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: orashi
cooked: <p>The idea seems more likely from Thomas’s <a href="https://lernapparat.de/improved-wasserstein-gan/" rel="nofollow noopener">Semi-Improved Training of Wasserstein GANs</a> or it’s just a coincidence?</p>

username: tom
cooked: <p>Hi <a class="mention" href="/u/orashi">@orashi</a>,</p>
<p>thank you for the credit. I might be among the first to discuss this in detail in this specific context and with a pytorch implementation, but certainly the identification of 1-Lipschitz (in the classical definition) with unit sphere in $W_{1,\infty}$ in the Sobolev scale (which is the fancy mathematician talk for the gradient being bounded by 1) is very standard just as the approximation of the derivative by finite differences (actually, one could fancy-talk that into a different norm, but let’s not), so I would expect many other people to have the same idea independently, so I’d go for coincidence. (Actually sampling two points is a bit different to sampling the center and using the two side points as I did, too.)<br>
What struck me as particularly curious in this case is why the authors of <em>Improved Training</em> chose to do a point-wise derivative test instead of testing the Lipschitz constant directly, but I have not asked them yet, so I don’t know.</p>
<p>Best regards</p>
<p>Thomas</p>

username: chenyuntc
cooked: <p>I first find the idea from <a href="https://www.zhihu.com/question/52602529/answer/158727900" rel="nofollow noopener">zhihu(Chinese Quora)</a>. The author seems to simply use the difference as an approximation of differential. But someone commented in the article that the difference is actually a better way for Kantorovich dual problem.<br>
The blog from <a class="mention" href="/u/tom">@tom</a> seems both more insightful and more intuitive. Excellent work!</p>

username: tom
cooked: <p>Hi,</p>
<p>just a quick update:<br>
A discussed in the <a href="http://lernapparat.de/more-improved-wgan/#slogan" rel="noopener nofollow ugc">SLOGAN blog post</a>, the difference in the <img src="https://discuss.pytorch.org/uploads/default/original/1X/c0708eec6775d377c5695eec7d1863453b552720.png" alt="above equation" width="546" height="41"> is generally not the gradient, but a projection onto x_hat. Thus it would seem to be more prudent to use a one-sided penalty in this formulation.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Yash_Rathi
cooked: <p>Could someone please explain, how would you find a gradient of a gradient mathematically?</p>