How to modify a Conv2d to Depthwise Separable Convolution?

forcefulowl: <p>I just read the paper about MobileNet v1.<br>
I’ve already known the mechanism behind that.<br>
But in pytorch, how can I achieve that base on Conv2d?</p>

forcefulowl: <p>should I just use 2 Conv2d to achieve that?</p>

apsvieira: <p>You would normally set the <code>groups</code> parameter of the Conv2d layer. From the docs:</p>
<blockquote>
<p>The configuration when groups == in_channels and out_channels = K * in_channels where K is a positive integer is termed in literature as depthwise convolution.</p>
</blockquote>
<p>You can find more information <a href="http://pytorch.org/docs/0.3.1/nn.html#torch.nn.Conv2d" rel="noopener nofollow ugc">here</a>.</p>

forcefulowl: <p>thanks for you information!! but I do not think they’re same things.<br>
From my perspective,<br>
<code>group</code> means to separate the channels.<br>
For e.g., the input of the image is DFxDFxM, the output is DFxDFxN, the original convolution is: DKxDKxMxN<br>
What I mean <code>Depthwise Separable Convolution</code> can be divided into 2 parts:<br>
part 1: Depthwise, the convolution of this part is DKxDKx1xM<br>
part 2: Pointwise, the convolution of this part is 1x1xMxN</p>
<p>If the situation is like that, should I just use 2 Conv2d to achieve that?</p>

forcefulowl: <p>Does anyone have some ideas?</p>

apsvieira: <p>I believe <a href="https://discuss.pytorch.org/t/depthwise-and-separable-convolutions-in-pytorch/7315/2">this answer</a> is a more complete reply to your question.</p>
<blockquote>
<p>If groups = nInputPlane, then it is Depthwise. If groups = nInputPlane, kernel=(K, 1), (and before is a Conv2d layer with groups=1 and kernel=(1, K)), then it is separable.</p>
</blockquote>
<p>In short, you can achieve it using Conv2d, by setting the groups parameters of your convolutional layers. Hope it helps.</p>

shicai: <p>Here is a simple example:</p>
<pre><code class="lang-auto">class depthwise_separable_conv(nn.Module):
    def __init__(self, nin, nout):
        super(depthwise_separable_conv, self).__init__()
        self.depthwise = nn.Conv2d(nin, nin, kernel_size=3, padding=1, groups=nin)
        self.pointwise = nn.Conv2d(nin, nout, kernel_size=1)

    def forward(self, x):
        out = self.depthwise(x)
        out = self.pointwise(out)
        return out
</code></pre>

AlexV: <p>For reference: see numpy implementations of <a href="https://github.com/jalexvig/cnns/blob/master/cnns/cnn_depthwise.py" rel="nofollow noopener">depthwise convolutions</a> vs. <a href="https://github.com/jalexvig/cnns/blob/master/cnns/cnn_grouped.py" rel="nofollow noopener">grouped convolutions</a> to see how this works.</p>

WilliamOu: <p>I am checking the shape of weights in a Conv2d layer, let’s say, the conv para is Conv2d(nin, 4*nin, groups=nin). The shape of weights in this layer is (4*nin, 1, 3, 3). Do you know how to interpret this shape?<br>
(4 kernels for channel 1, 4 kernels for channel 2, …) or (1 kernel for channel 1, 1 kernel for channel 2, …, 1 kernel for channel 1,  1 kernel for channel 2, …)</p>
<p>Thanks,</p>

guSTARlove: <p>I think the first understanding is right</p>

James_Howard: <p>Hi, in reply to <a class="mention" href="/u/shicai">@shicai</a> 's example, this means you have only 1 kernel per layer, which is unlikely to be helpful.</p>
<p>Maybe something like this:</p>
<pre><code class="lang-python">class depthwise_separable_conv(nn.Module):
    def __init__(self, nin, kernels_per_layer, nout):
        super(depthwise_separable_conv, self).__init__()
        self.depthwise = nn.Conv2d(nin, nin * kernels_per_layer, kernel_size=3, padding=1, groups=nin)
        self.pointwise = nn.Conv2d(nin * kernels_per_layer, nout, kernel_size=1)

    def forward(self, x):
        out = self.depthwise(x)
        out = self.pointwise(out)
        return out
</code></pre>

Zion_Huang: <p>You can just set group=nin/kernels_per_layer instead of nin*kernels_per_layer.</p>

Chirag_Kalpesh_Shah: <p>Can you help me in figuring out how do I modify the Depthwise convolution to full convolution? I have them as follows -</p>
<pre><code class="lang-auto">class ConvBNReLU(nn.Sequential):
    def __init__(self, in_planes, out_planes, kernel_size, stride=1, groups=1):
        padding = (kernel_size - 1) // 2
        super(ConvBNReLU, self).__init__(
            nn.Conv2d(in_planes, out_planes, kernel_size, stride, padding, groups=groups, bias=False),
            nn.BatchNorm2d(out_planes),
            nn.ReLU6(inplace=True)
        )
</code></pre>
<pre><code class="lang-auto">
class InvertedResidual(nn.Module):
    def __init__(self, inp, oup, kernel, stride, expand_ratio, res_connect):
        super(InvertedResidual, self).__init__()
        self.stride = stride
        assert stride in [1, 2]
        hidden_dim = int(round(inp * expand_ratio))
        self.use_res_connect = res_connect
        layers = []
        if expand_ratio != 1:
            # pw
            layers.append(ConvBNReLU(inp, hidden_dim, kernel_size=1))
        layers.extend([
            # depth-wise
            ConvBNReLU(hidden_dim, hidden_dim, kernel_size=kernel, stride=stride, groups=hidden_dim),
            # pw-linear
            nn.Conv2d(hidden_dim, oup, 1, 1, 0, bias=False),
            nn.BatchNorm2d(oup),
        ])
        self.conv = nn.Sequential(*layers)

    def forward(self, x):
        if self.use_res_connect:
            return x + self.conv(x)
        else:
            return self.conv(x)
</code></pre>

ptrblck: <p>I’m not sure if I understand the question correctly, but if you don’t want to use the depthwise comvolution but a “vanilla” convolution, you could remove the <code>groups</code> argument or set it to <code>1</code>.</p>

anguoyang: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , this thread is about nn.Conv2d, however in styleGAN, it used F.Conv2d, is it possible to implement separable convolution for F.Conv2d instead of nn.Conv2d? (like <a class="mention" href="/u/james_howard">@James_Howard</a>  did ),  thank you</p>

ptrblck: <p>Yes, that’s possible since internally the <code>nn.Conv2d</code> modules will also just call into the functional API <code>F.conv2d</code>. You would have to create the parameters (<code>weight</code> for the filters and <code>bias</code>) in the correct shapes and could then call <code>F.conv2d</code> with the right <code>groups</code> argument.</p>

anguoyang: <p>hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , thank you for your suggestion, yes, sometimes, we could do it according to your suggestion:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/rosinality/stylegan2-pytorch/blob/master/model.py#L94">
  <header class="source">

      <a href="https://github.com/rosinality/stylegan2-pytorch/blob/master/model.py#L94" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/rosinality/stylegan2-pytorch/blob/master/model.py#L94" target="_blank" rel="noopener nofollow ugc">rosinality/stylegan2-pytorch/blob/master/model.py#L94</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="84" style="counter-reset: li-counter 83 ;">
          <li>        self.register_buffer("kernel", kernel)</li>
          <li>
          </li>
<li>        self.pad = pad</li>
          <li>
          </li>
<li>    def forward(self, input):</li>
          <li>        out = upfirdn2d(input, self.kernel, pad=self.pad)</li>
          <li>
          </li>
<li>        return out</li>
          <li>
          </li>
<li>
          </li>
<li class="selected">class EqualConv2d(nn.Module):</li>
          <li>    def __init__(</li>
          <li>        self, in_channel, out_channel, kernel_size, stride=1, padding=0, bias=True</li>
          <li>    ):</li>
          <li>        super().__init__()</li>
          <li>
          </li>
<li>        self.weight = nn.Parameter(</li>
          <li>            torch.randn(out_channel, in_channel, kernel_size, kernel_size)</li>
          <li>        )</li>
          <li>        self.scale = 1 / math.sqrt(in_channel * kernel_size ** 2)</li>
          <li>
      </li>
</ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
Because we don’t need to update the weight manually<br>
however, for this one:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/rosinality/stylegan2-pytorch/blob/master/model.py#L277">
  <header class="source">

      <a href="https://github.com/rosinality/stylegan2-pytorch/blob/master/model.py#L277" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/rosinality/stylegan2-pytorch/blob/master/model.py#L277" target="_blank" rel="noopener nofollow ugc">rosinality/stylegan2-pytorch/blob/master/model.py#L277</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="267" style="counter-reset: li-counter 266 ;">
          <li>)</li>
          <li>
          </li>
<li>if self.upsample:</li>
          <li>    input = input.view(1, batch * in_channel, height, width)</li>
          <li>    weight = weight.view(</li>
          <li>        batch, self.out_channel, in_channel, self.kernel_size, self.kernel_size</li>
          <li>    )</li>
          <li>    weight = weight.transpose(1, 2).reshape(</li>
          <li>        batch * in_channel, self.out_channel, self.kernel_size, self.kernel_size</li>
          <li>    )</li>
          <li class="selected">    out = conv2d_gradfix.conv_transpose2d(</li>
          <li>        input, weight, padding=0, stride=2, groups=batch</li>
          <li>    )</li>
          <li>    _, _, height, width = out.shape</li>
          <li>    out = out.view(batch, self.out_channel, height, width)</li>
          <li>    out = self.blur(out)</li>
          <li>
          </li>
<li>elif self.downsample:</li>
          <li>    input = self.blur(input)</li>
          <li>    _, _, height, width = input.shape</li>
          <li>    input = input.view(1, batch * in_channel, height, width)</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
We have to do something additionally to the weight in the forward function before the convolution, so we have to use weight as one of the input parameter,  that is not possible, because we could set weight for separable convolution class as it has 2 convs, could you please tell me how to do it? thank you so much</p>

ptrblck: <p>I don’t fully understand the issue you are mentioning. The linked code shows some scaling and reshaping of the <code>weight</code> parameter. Wouldn’t it be possible to apply the same logic to your custom depthwise conv layer?</p>

anguoyang: <p>Sorry for the confusion, yes, the code did some reshape and scaling on weight, thanks for the suggestion, let me try to apply same operation on my customized depthwise convolution, I just wondering how about the pointwise convolution?</p>

anguoyang: <p>I tried to modify my separable convolution as follow:</p>
<pre><code class="lang-auto">class dsc(nn.Module):
    def __init__(self, in_ch = 3, out_ch = 3, kernel_size=3, stride=1, padding=1, bias = True):
        super(dsc, self).__init__()
        self.depthwise = nn.Conv2d(in_ch, in_ch, kernel_size=kernel_size, stride = stride, padding = padding, bias= bias, groups=in_ch)
        self.pointwise = nn.Conv2d(in_ch, out_ch, kernel_size=1, bias=bias)

        self._initialize_weights()
        
    def _initialize_weights(self):
        .........init code.............
    
    def forward(self, x, weight = None):
        if weight is not None:
            self.depthwise.weight = torch.nn.Parameter(weight)
        out = self.depthwise(x)
        out = self.pointwise(out)
        return out
</code></pre>
<p>I added weight to the forward function so I could easily replace the F.conv2d operations, but I got an error like:</p>
<pre><code class="lang-auto">Exception has occurred: RuntimeError
Given groups=3, expected weight to be divisible by 3 at dimension 0, but got weight of size [[64, 3, 1, 1]] instead
</code></pre>
<p>So I think the self.depthwise.weight = torch.nn.Parameter(weight) is wrong, but I didn’t figure out how to solve it</p>