Why grouped convolution so much slower?

username: overloader7
cooked: <p>I was testing convolution of 32 channel maps in the following ways:</p>
<ol>
<li>full Pointwise→ReLU→Depthwise convolution</li>
<li>16 group PW→channel shuffle, ReLU→2 groups 3x3 convolution</li>
<li>1 group 3x3 convolution</li>
</ol>
<p>They each took 1.13s, 3.62s and 0.83s on average respectively. I find it strange as the options require:</p>
<ol>
<li>32x32+3x3x32 = 1312</li>
<li>32x16+3x3x32x2 = 1088</li>
<li>3x3x32x32 = 9216</li>
</ol>
<p>mul-add operations per pixel respectively (assuming naive approach). While i can understand full convolution about as much time as DW-separable one due to possible memory bottlenecks and dispatch overheads, I find the 2nd option being very slow - strange, as it requires two conv calls just like the 1st, yet is more than 4.5x slower than a full convolution. The only major difference is addition of channel shuffle, but I think that it can’t reasonably take as much time as an entire convolution operation (though I didn’t measure it separately.)</p>
<p>using torch 2.7.0 on arch linux with cuda, cc 6.1</p>

username: KFrank
cooked: <p>Hi Overloader!</p>
<aside class="quote no-group quote-modified" data-username="overloader7" data-post="1" data-topic="222478">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/overloader7/48/78003_2.png" class="avatar"> overloader7:</div>
<blockquote>
<ol>
<li>full Pointwise→ReLU→Depthwise convolution</li>
<li>16 group PW→channel shuffle, ReLU→2 groups 3x3 convolution</li>
<li>1 group 3x3 convolution</li>
</ol>
<p>I find it strange as the options require:</p>
<ol>
<li>32x32+3x3x32 = 1312</li>
<li>32x16+3x3x32x2 = 1088</li>
<li>3x3x32x32 = 9216</li>
</ol>
</blockquote>
</aside>
<p>As written, your second option, “16 group PW→channel shuffle, ReLU→2 groups 3x3 convolution,”</p>
<p>would seem to me to be:</p>
<p><code>Conv2d (32, 32, 1, groups = 16)</code></p>
<p>(“no math” shuffle and <code>ReLU</code>)</p>
<p><code>Conv2d (32, 32, 3, groups = 2)</code></p>
<p>If so, you have your operation counts wrong.  By my count, “16 Group PW” has 2x2x16 = 64</p>
<p>operations and “2 groups 3x3” has 16x16x2x3x3 = 4608 operations.</p>
<p>So your second option has slightly more than half as many operations as a plain (1 group)</p>
<p>3x3 convolution.</p>
<p>(The numbers you give correspond to a “2 group PW” followed by a “16 groups 3x3”.)</p>
<p>This makes the large timing for the second option seem much less surprising.  Also, when I</p>
<p>compare the timing of <code>Conv2d (32, 32, 3, groups = 2)</code> with <code>Conv2d (32, 32, 3)</code>, I find</p>
<p>the 2-group convolution is only about 25% slower than the plain, 1-group convolution, so the</p>
<p>additional steps in option 2 do add significantly to the cost.</p>
<p>(What is interesting – although maybe not surprising – is how fast the plain 3x3 convolution</p>
<p>is compared to the other two options, as well as to a 2-group 3x3 convolution (without extra</p>
<p>steps), even though it has the highest operation count.    I would attribute this to the floating-point</p>
<p>pipelines and kernels being especially highly optimized for “plain” (no groups) convolutions,</p>
<p>arguably an appropriate design choice.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: Sene1944
cooked: <p>Grouped convolution isn’t inherently slower; the observed slowness stems from memory bandwidth limitations and suboptimal kernel implementations within deep learning frameworks, not reduced computational complexity. While grouped convolutions theoretically require fewer multiply-accumulate (MAC) operations, they introduce channel-wise processing and require a specialized kernel and memory format for efficient execution, which is not always well-supported. Furthermore, the additional channel shuffle or aggregation operations needed between groups can add to the overall processing time.</p>

username: KFrank
cooked: <p>Hi Sene (and Overloader)!</p>
<aside class="quote no-group quote-modified" data-username="Sene1944" data-post="3" data-topic="222478" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/919ad9/48.png" class="avatar"> Sene1944:</div>
<blockquote>
<p>Grouped convolution isn’t inherently slower;<br>
…<br>
Furthermore, the additional channel shuffle or aggregation operations needed between groups can add to the overall processing time.</p>
</blockquote>
</aside>
<p>Here are some additional timings for your delectation:</p>
<p>These are all for single pytorch <code>Module</code>s, e.g. <code>Conv2d (32, 32, 3)</code>, without<br>
any <code>Sequential</code> wrappers or <code>ReLU</code>s.</p>
<p>In the timings below, <code>k</code> is the kernel size and <code>g</code> is the number of groups. So,<br>
for example, <code>tconvk3g02</code> is the timing for a <code>Conv2d (32, 32, 3, groups = 2)</code><br>
All of the <code>Conv2d</code>s have <code>in_channels = out_channels = 32</code> and the  input tensor<br>
has shape <code>[16, 32, 256, 256]</code>.  Last, <code>tshuffle16</code> is for <code>ChannelShuffle (16)</code>.</p>
<p>(Therefore <code>tconvk1g01</code> is the timing for what is often called a pointwise<br>
convolution and <code>tconvk3g32</code> is that for a (3x3) depthwise convolution.)</p>
<p>These timings were all performed on the same laptop that supports <code>cuda</code> with a<br>
discrete nvidia gpu, <code>xpu</code> with its integrated intel arc graphics, and <code>cpu</code> with<br>
its intel cpu.  (Timings and relative timings are expected to be different on<br>
other hardware.)</p>
<p>Here are the timings:</p>
<pre><code class="lang-plaintext">2.8.0+cu129
device: cuda
12.9
NVIDIA RTX 3000 Ada Generation Laptop GPU
_CudaDeviceProperties(name='NVIDIA RTX 3000 Ada Generation Laptop GPU', major=8, minor=9, total_memory=7933MB, multi_processor_count=36, uuid=f7fb8822-dc6a-36ef-c5e3-2665db44af42, pci_bus_id=1, pci_device_id=0, pci_domain_id=0, L2_cache_size=32MB)
tconvk1g01:    2.68 msec
tconvk1g02:    3.32 msec
tconvk1g16:    5.54 msec
tconvk1g32:    2.00 msec
tconvk3g01:    6.39 msec
tconvk3g02:    7.91 msec
tconvk3g16:    4.47 msec
tconvk3g32:    2.33 msec
tshuffle16:    1.49 msec

2.8.0+xpu
device: xpu
20250101
Intel(R) Arc(TM) Graphics
_XpuDeviceProperties(name='Intel(R) Arc(TM) Graphics', platform_name='Intel(R) oneAPI Unified Runtime over Level-Zero', type='gpu', driver_version='1.6.33276+22', total_memory=29184MB, max_compute_units=128, gpu_eu_count=128, gpu_subslice_count=8, max_work_group_size=1024, max_num_sub_groups=128, sub_group_sizes=[8 16 32], has_fp16=1, has_fp64=1, has_atomic64=1)
tconvk1g01:   12.82 msec
tconvk1g02:   12.82 msec
tconvk1g16:   14.88 msec
tconvk1g32:   12.78 msec
tconvk3g01:   17.49 msec
tconvk3g02:   16.77 msec
tconvk3g16:   18.90 msec
tconvk3g32:   14.79 msec
tshuffle16:    3.71 msec

2.8.0+cu129
device: cpu
tconvk1g01:   37.60 msec
tconvk1g02:   39.49 msec
tconvk1g16:   35.54 msec
tconvk1g32:   36.18 msec
tconvk3g01:   80.73 msec
tconvk3g02:   75.99 msec
tconvk3g16:   51.50 msec
tconvk3g32:   57.20 msec
tshuffle16:   15.05 msec
</code></pre>
<p>The timings basically speak for themselves.  It’s worth noting in regard to<br>
Overloaded’s original question that while the cost of <code>ChannelShuffle</code> is,<br>
in all cases, the smallest, it is not non-trivial, especially for <code>cuda</code>.</p>
<p>Best.</p>
<p>K. Frank</p>