Depthwise convolution using nn.functional.conv3d

username: shyu
cooked: <p>Hello,</p>
<p>I’m implementing the CNN to align multiple video frames using the dynamic filter network (<a href="https://arxiv.org/abs/1605.09673" rel="nofollow noopener">DFN</a>).</p>
<p>In my implementation, the DFN estimates the convolution filters of size <code>[batch x 3 (depth, # of input frames) x filter_height x filter_width]</code> and the estimated filters are used to align three input frames by a convolution.</p>
<p>Given the tensor of size <code>[8 x 128 x 3 x 32 x 32]</code> , which is generated using <code>nn.Conv3D</code> and three input frames, and the estimated filters of size <code>[8 x 3 x 9 x 9]</code>, I tried to use <code>nn.functional.conv3d</code> function to perform the depthwise convolution to the feature maps of each input frame as</p>
<pre><code class="lang-auto"># estimated filters
filters = torch.unsqueeze(filters, dim=1)       # [8, 1, 3, 9, 9] 
filters = filters.repeat(1, 128, 1, 1, 1)       # [8, 128, 3, 9, 9] 
filters = filters.permute(1, 0, 2, 3, 4)       # [128, 8, 3, 9, 9] 

f_sh = filters.shape
filters = torch.reshape(filters, (1, f_sh[0] * f_sh[1], f_sh[2], f_sh[3], f_sh[4]))       # [1, 128*8, 3, 9, 9] 
filters = filters.permute(1, 0, 2, 3, 4)       # [128*8, 1, 3, 9, 9] 

# input tensor to be aligned
sh = x_in.shape        # [8, 128, 3, 32, 32]
x_in = torch.reshape(x_in, (1, sh[0] * sh[1], sh[2], sh[3], sh[4]))       # [1, 8*128, 3, 32, 32]
temp_out = nn.functional.conv3d(x_in, filters, stride=(1, 1, 1), padding=(0, 2, 2), groups=128*8)
</code></pre>
<p>It works, but I want to know that it’s correctly implemented or not.<br>
Could you give me an idea to effectively perform the depthwise convolution using <code>nn.functional.conv3d</code>?</p>
<p>Thank you.</p>

username: ptrblck
cooked: <p>I’m not sure, if reshaping the batch size into the channels is the right approach here.<br>
Currently you are using an own filter for each channel in each batch (128*8).<br>
This would set your batch size to a fixed value, which seems a bit unusual.</p>
<p>Could you explain your use case a bit?</p>

username: shyu
cooked: <p>I refer <a href="https://arxiv.org/pdf/1907.06490.pdf" rel="noopener nofollow ugc">this paper</a>, which performs the image registration. I’m trying to the convolution to each feature map using the estimated dynamic filters as<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/61799ed3705e1b8b4ae223376014a8bc3b064f39.png" data-download-href="https://discuss.pytorch.org/uploads/default/61799ed3705e1b8b4ae223376014a8bc3b064f39" title="%EC%BA%A1%EC%B2%98"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/6/61799ed3705e1b8b4ae223376014a8bc3b064f39_2_345x175.png" alt="%EC%BA%A1%EC%B2%98" data-base62-sha1="dUiSSApyD5wTBOO9VV6FdzXHyZP" width="345" height="175" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/6/61799ed3705e1b8b4ae223376014a8bc3b064f39_2_345x175.png, https://discuss.pytorch.org/uploads/default/optimized/2X/6/61799ed3705e1b8b4ae223376014a8bc3b064f39_2_517x262.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/6/61799ed3705e1b8b4ae223376014a8bc3b064f39_2_690x350.png 2x" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">%EC%BA%A1%EC%B2%98</span><span class="informations">738×376 23.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In this figure, <code>N</code> is the number of input images (in my case, input frames) and <code>F</code> is the number of feature maps to each input image.</p>
<p>Besides, I found the TensorFlow code released by the authors and I’m implementing <a href="https://github.com/diegovalsesia/deepsum/blob/51943081bb42863385aa217dd5ae3ffd959209c9/DeepSUM/DeepSUM_network.py#L476" rel="noopener nofollow ugc">the registration part</a> using Pytorch. In this code, they performed the convolution to the misaligned feature maps using <code>tf.nn.depthwise_conv2d</code> function (Since Tensorflow doesn’t have a function for the 3D depthwise convolution.)</p>
<p>Compared to the authors’ code, I’m not sure my approach is right or not.</p>
<p>Thank you.</p>