Using skimage transform inside pytorch model

ttodic: <p>Hello,<br>
I was wondering if there is a way to use skimage transforms as part of cnn in pytorch?<br>
The model should have two branches, one is a simple convolutional neural network, and the other should perform transform, then the result should go through some convolutional layers, then follows the inverse transform, and the output should be concatenated with output of the first branch, and then this concatenated tensor should pass a couple more convolutional layers.<br>
Since the transform performs only on numpy arrays, prior to the call I should move the tensor to cpu and convert it to numpy array, right? But I’m not sure how to deal with batch of images, since the network processes whole batch at a time, not just one sample, like the transform works with one sample at a time. The first thing that comes to my mind is to use for loop across all the samples in a batch, could it work this way? Also, should I use no_grad() when computing transform? Will this cut the graph or affect training some way?<br>
Thanks in advance!</p>

ptrblck: <aside class="quote no-group" data-username="ttodic" data-post="1" data-topic="115358">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/91b2a8/48.png" class="avatar"> ttodic:</div>
<blockquote>
<p>Since the transform performs only on numpy arrays, prior to the call I should move the tensor to cpu and convert it to numpy array, right?</p>
</blockquote>
</aside>
<p>Yes, but this would also detach the array from the computation graph and Autograd won’t be able to track these operations (in case you want to use them as differentiable methods). You could either implement the backward pass manually using custom <code>autograd.Function</code>s or check out <a href="https://github.com/kornia/kornia">kornia</a>, which provides differentiable transformations (CC <a class="mention" href="/u/edgarriba">@edgarriba</a> as the core dev).</p>
<aside class="quote no-group" data-username="ttodic" data-post="1" data-topic="115358">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/91b2a8/48.png" class="avatar"> ttodic:</div>
<blockquote>
<p>But I’m not sure how to deal with batch of images, since the network processes whole batch at a time, not just one sample, like the transform works with one sample at a time. The first thing that comes to my mind is to use for loop across all the samples in a batch, could it work this way?</p>
</blockquote>
</aside>
<p>Yes, a loop would work, but could be slow. Since you are already moving the data to the CPU (and thus synchronizing the code), the performance penalty from the loop might not be the bottleneck.</p>
<aside class="quote no-group" data-username="ttodic" data-post="1" data-topic="115358">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/91b2a8/48.png" class="avatar"> ttodic:</div>
<blockquote>
<p>Also, should I use no_grad() when computing transform? Will this cut the graph or affect training some way?</p>
</blockquote>
</aside>
<p>Yes, it will detach these operations from the computation graph as explained before.</p>

ttodic: <p>KK, thank you very much! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>