Backprop through Discrete Wavelet Transform (DWT) on GPU

username: Veril
cooked: <p>It there an efficient way to perform this operation? 2D.</p>
<p>The naive approach is to either loop through the rows and columns or construct the matrix and multiply.</p>
<p>Oddly I was unable to find an example of how to do this for either theano or tensorflow.</p>

username: tom
cooked: <p>You can use conv layers to have single levels (using coeffs eg from <code>pywavelets</code> and setting <code>requires_grad</code> to <code>False</code>.<br>
Also check out</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://assets-cdn.github.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/edouardoyallon/pyscatwave/" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/4263222?s=400&amp;v=4" class="thumbnail onebox-avatar" width="250" height="250">

<h3><a href="https://github.com/edouardoyallon/pyscatwave/" target="_blank" rel="nofollow noopener">edouardoyallon/pyscatwave</a></h3>

<p>pyscatwave - Fast Scattering Transform with CuPy/PyTorch</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>that seems to be the most waveletty neural network thing I have recently seen.</p>
<p>I think I saw Edouard around here, but I forgot his username.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Veril
cooked: <p>I’m not sure I understand how to reduce DWT to convolutions as you propose.<br>
Edit: Are you proposing to basically construct nlogn conv filters? since the higher frequency wavelets are just translated and mostly 0 everywhere I suppose it’s possible. Do you have an example? Not sure how I would stitch it all up.</p>
<p>The scattering transform is indeed interesting but not what I’m immediately looking for, will check it out though thanks. Seems to have high potential, too bad you can’t backprop through the fast implementation though.</p>

username: tom
cooked: <p>As talk is cheap, I wrote up a quick <a href="https://github.com/t-vi/pytorch-tvmisc/blob/master/misc/2D-Wavelet-Transform.ipynb" rel="nofollow noopener">notebook on using 2D Wavelet transformations with PyTorch</a> and hope it is useful for you.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Veril
cooked: <p>It certainly is, you seem like you have gone through a bit of effort.<br>
Thanks.</p>

username: verified.human
cooked: <p>I recently implemented a wavelet filter bank in PyTorch. Although my focus here was to write a fast temporal convolution library for wavelets, this might be of interest to you: <a href="https://github.com/tomrunia/PyTorchWavelets" rel="nofollow noopener">https://github.com/tomrunia/PyTorchWavelets</a></p>

username: fbcotter
cooked: <p><a class="mention" href="/u/tom">@tom</a> Great work on the notebook. I already had a repo to do a dual tree DWT in pytorch (<a href="https://en.wikipedia.org/wiki/Complex_wavelet_transform" rel="nofollow noopener">https://en.wikipedia.org/wiki/Complex_wavelet_transform</a>), and inspired by your code, I have now added support for the DWT and Inverse DWT in 2 dimensions. You can check it out at<br>
<a href="https://github.com/fbcotter/pytorch_wavelets" rel="nofollow noopener">https://github.com/fbcotter/pytorch_wavelets</a> (<a class="mention" href="/u/verified.human">@verified.human</a> sorry for the similar name!)</p>
<p>The repo has tests checking that gradients pass nicely through the dual tree DWT, but I am yet to write tests to check the gradients for the DWT. I would wager they work nicely, but do need to confirm. The perfect reconstruction works, as well as getting the same wavelet coefficients as pywt.</p>

username: tom
cooked: <p>Thanks for sharing, <a class="mention" href="/u/fbcotter">@fbcotter</a>!</p>
<p>It’s not terribly efficient, but wouldn’t the tracing work for backpropagation until you have an explicit backward?</p>

username: chicanagram
cooked: <p>Thanks so much for writing this up! Do you have any quick tips for how to adapt this for 1D DWT and inverse DWT?</p>

username: alon_Tones
cooked: <p>Hi <a class="mention" href="/u/tom">@tom</a> <a class="mention" href="/u/fbcotter">@fbcotter</a>.<br>
One thing that appears to be different between both of your implementations is the requires_grad parameter. As far as I can understand <a class="mention" href="/u/tom">@tom</a>’s solution does not explicitly state that the convolution layer parameters have requires_grad = False and thus they would get updated at backprop (as would any kernel of a conv2d or transposeconv2d layer). Similarly in <a class="mention" href="/u/fbcotter">@fbcotter</a>’s repo, the parameters are explicitly marked with requires_grad = False.</p>
<p>For my implementation I need a 1D wavelet based loss, so I would require the parameters to have requires_grad = False, it seems like it might be slightly easier for me to adapt tom’s version, so I wanted to make sure I understand this before going ahead and using the code.</p>
<p>Thanks again for these awesome contributions!</p>

username: tom
cooked: <p>So I finally removed all the long deprecated <code>Variable</code> bits. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
My notebook makes no assumption about whether you pass in weights which require grads or not - if you do, it will happily backpropagate into them.<br>
I did use a similar approach for audio (1d).</p>
<p>Best regards</p>
<p>Thomas</p>

username: alon_Tones
cooked: <p>Thanks for the update <a class="mention" href="/u/tom">@tom</a>  So essentially, to make sure that the DWT is fixed =, I would simply set dec_hi.requires_grad = False (and same for the others off course). And then everything should work.<br>
Perhaps I’ll write this as a pytorch module so it can be easily integrated into a network.</p>
<p>Thanks again!</p>

username: tom
cooked: <p>Not requiring grad is the default, so no need to set things.<br>
My personal philosophy here is to keep stateless things as functions, but having it in a module is fine, too.</p>
<p>Best regards</p>
<p>Thomas</p>