akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Does pytorch optimize the group parameter in convs?

username: Sora
cooked: <p>Does pytorch optimize the group parameter? So can efficient the MobileNet architecture.</p>

username: smth
cooked: <p>no, group parameter right now is done via a naive for loop.</p>

username: lolongcovas
cooked: <p>does <a href="https://github.com/pytorch/pytorch/blob/master/torch/lib/THNN/generic/SpatialDepthWiseConvolution.c" rel="nofollow noopener">this</a> do depth wise convolution by setting group parameter = input data depth?</p>
<p>From pytorch doc, I couldnt find any information about this new feature, I can imagine that if I do</p>
<pre><code class="lang-auto">model = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3, stride=2),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2), # output is 64 depth
            nn.Conv2d(64, 64, kernel_size=3, group=64), # will this layer use SpatialDepthWiseConvolution instead of group loop?
            ... # next layers
)
</code></pre>
<p>will this model use SpatialDepthWiseConvolution?</p>
<p>Best</p>

username: fmassa
cooked: <p>No, it won’t use that depthwise function, just the standard codepath for groups.<br>
also, note that that implementation of depthwise convolutions is very naive, and will be as slow as setting groups to 64 in your case.</p>

username: falmasri
cooked: <p>I didn’t really get you point here.<br>
so if he used group=64 this doesn’t mean that each output channel is doing convolution with only one channel ?<br>
I’m testing this function but the network is not converging so I really want to know if the code is working or not</p>

username: gquenot
cooked: <p>Hello,</p>
<p><code>Conv2d(256, 256, kernel_size=(3, 3), padding=(1, 1), groups=4, bias=False)</code><br>
is almost twice slower (in a sequence of similar calls) than:<br>
<code>Conv2d(128, 128, kernel_size=(3, 3), padding=(1, 1), groups=1, bias=False)</code><br>
while the number of operations is about the same. Is there a way to do any better? Is there a chance that in a near future a more efficient implementation than a naive loop becomes available?</p>
<p>Best regards,</p>
<p>Georges Quénot.</p>

username: ptrblck
cooked: <p>Assuming you are using cudnn, then the answer is: yes, more fast kernel will be implemented for more workloads in the future.<br>
You could use <code>torch.backends.cudnn.benchmark = True</code> at the beginning of your script to use the cudnn heuristics, which should pick the currently fastest available kernel.<br>
I would also recommend to use the latest cudnn version (<code>7.6.5.32</code>).</p>
<p>Note that <code>benchmark=True</code> will profile the kernels in the first run for each new input shape, so your profiling should start after a warmup phase.</p>
<p>You could also compare the current speed to the native implementation by disabling cudnn via <code>torch.backends.cudnn.enabled = False</code>.</p>