Conv2d with multiple channels or Conv3d with 1?

username: 80HighDef
cooked: <p>Hello,</p>
<p>I am trying to write a Binary Classification for pairs of images taken one after the other from a video. My frames are in black and white and thus only have one channel. I’ve been using 2 Conv3d() layers with one channel to moderate success (~85% accuracy), however it occurred to me that I could just be using Conv2d() layers with multiple channels instead.</p>
<p>What would be the advantages/disadvantages of using either or for this scenario? I’m thinking I should stick with Conv3d() layers due to the ability to run convolution in the third dimension, but as I’m rather new to this I don’t know how useful this is.</p>
<p>Note, I did come across <a href="https://discuss.pytorch.org/t/video-frames-as-conv2d-channels-or-1-channel-conv3d/130810">this thread</a> before making this post, but I was hoping someone could give a more in-depth answer and I wasn’t sure if I should bump the thread or not, thus I am making a new post.</p>
<p>All help is appreciated, thanks in advance!</p>

username: KFrank
cooked: <p>Hi Alex!</p>
<aside class="quote no-group" data-username="80HighDef" data-post="1" data-topic="152821" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/80highdef/48/50052_2.png" class="avatar"> 80HighDef:</div>
<blockquote>
<p>I’m thinking I should stick with Conv3d() layers due to the ability to run convolution in the third dimension</p>
</blockquote>
</aside>
<p>If your use case is restricted to having just pairs of images as your<br>
network input (as compared to using more than two frames of the<br>
video, say, 8 or 16), then using <code>Conv2d</code> with <code>in_channels = 2</code><br>
and <code>kernel_size = (k, k)</code> and <code>Conv3d</code> with <code>in_channels = 1</code>,<br>
<code>kernel_size = (2, k, k)</code>, and a tensor with a depth dimension of 2<br>
passed in are essentially equivalent (assuming that <code>out_channels = 1</code>).</p>
<p>With a depth dimension (and <code>kernel_size</code>) of 2 , you don’t really<br>
have anything to convolve over, so you get the same result as the<br>
2-channel computation.</p>
<p>Since they’re equivalent, in terms of the result, neither one gives you<br>
a better network.  To my mind, <code>Conv2d</code> is stylistically better, because<br>
it better fits the way I think about what is going on.</p>
<aside class="quote no-group">
<blockquote>
<p>I wasn’t sure if I should bump the thread or not, thus I am making a new post.</p>
</blockquote>
</aside>
<p>A profound thanks to you for not resurrecting that zombie thread.<br>
Exhuming old threads from their rightful interment just adds to the<br>
noise.</p>
<p>Best.</p>
<p>K. Frank</p>

username: 80HighDef
cooked: <p>Hi K. Frank,</p>
<p>Thank you for your response!</p>
<p>If the out channels were greater than 1 for both Conv2d and Conv3d, are they still equivalent?</p>
<p>-Alex</p>

username: KFrank
cooked: <p>Hi Alex!</p>
<aside class="quote no-group" data-username="80HighDef" data-post="3" data-topic="152821" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/80highdef/48/50052_2.png" class="avatar"> 80HighDef:</div>
<blockquote>
<p>If the out channels were greater than 1 for both Conv2d and Conv3d, are they still equivalent?</p>
</blockquote>
</aside>
<p>Yes.</p>
<p>The shapes of the relevant weights suggest that they could do the same<br>
thing, and a test computation shows that this is the case:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.10.2'
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt; conv2 = torch.nn.Conv2d (in_channels = 2, out_channels = 5, kernel_size = (3, 3), bias = False)
&gt;&gt;&gt; conv3 = torch.nn.Conv3d (in_channels = 1, out_channels = 5, kernel_size = (2, 3, 3), bias = False)
&gt;&gt;&gt; conv2.weight.shape
torch.Size([5, 2, 3, 3])
&gt;&gt;&gt; conv3.weight.shape
torch.Size([5, 1, 2, 3, 3])
&gt;&gt;&gt; with torch.no_grad():
...     _ = conv3.weight.copy_ (conv2.weight.unsqueeze (1))
...
&gt;&gt;&gt; input2 = torch.randn (1, 2, 7, 7)
&gt;&gt;&gt; input3 = input2.unsqueeze (1)
&gt;&gt;&gt; result2 = conv2 (input2)
&gt;&gt;&gt; result3 = conv3 (input3)
&gt;&gt;&gt; torch.equal (result2, result3.squeeze (2))
True
</code></pre>
<p>Best.</p>
<p>K. Frank</p>