Depthwise 1D convolution with shared filter

Dewei_Hu: <p>I want to use Conv1D depthwise on the text and the filters should be the same with each other. Is there a way we can do this?</p>

ptrblck: <p>Could you explain the use case a bit more? Repeating a kernel would yield the same outputs, so I’m not sure if I understand the question correctly.</p>

Dewei_Hu: <p>Hi, <a class="mention" href="/u/ptrblck">@ptrblck</a>! Thanks for interested in this question.</p>
<p>I’m doing a multi-label classification task, and the label space is about 8900. The classifier needs to make predictions about  what labels the input text corresponds to (generally, an input text might correspond to 5~10 labels).</p>
<p>As described in this paper: <a href="https://arxiv.org/pdf/1909.11386.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1909.11386.pdf</a>, I’m gonna do a per-label mask of the input embedding [B, L, D] (batch size, input length, embed dimension), after the mask, the embedding would become a 4-D tensor [B, L, T, D]. Then I will do convolution.</p>
<p>The original paper suggests that all embedding share the same convolution layer, which means all label embedding should be convolved by the same weights. For simplicity, we could stack the 4-D tensor at the embedding dimension, then it has the shape [B, L, T*D], which is suitable for depthwise convolution.</p>
<p>However, if we directly use 1-D convolution, there will be one unique filter for each label embedding, and there will be 8900 different filters in total, which can be a disaster for GPU memory.  I’m wondering if there is a method to make the filters share the same parameters.</p>

ptrblck: <p>Is your input shape <code>[B, L, T*D]</code> corresponding to a channels-last memory format, i.e. would <code>T*D</code> represent the channels? If so, you would have to <code>permute</code> the data, but each kernel would still use all input channels in the default layout.<br>
Could you post the input shape and desired output shapes (with the description what the temporal and channel dimensions would be), please?</p>

Dewei_Hu: <p>Yes, <code>T*D</code> represents channels, so the code should be like <code>conv1d([B, T*D, L])</code>. While initializing the convolution layer, it would be <code>layer = nn.Conv1d(T*D,filter_maps*T,kernel_size,groups=T)</code></p>

ptrblck: <p>This setup:</p>
<pre><code class="lang-python">T, D, filter_maps = 2, 3, 4
kernel_size=5
layer = nn.Conv1d(T*D,filter_maps*T,kernel_size,groups=T)
print(layer.weight.shape)
&gt; torch.Size([8, 3, 5])
</code></pre>
<p>would use 8 filters (defined by <code>filter_maps*T</code>) where each will use 3 input channels (defined by <code>T*D</code> and the groups). Would you explain a bit more which filters should now be shared?<br>
Would you like to use a single output filter only?</p>

Dewei_Hu: <p>The code you wrote above is exactly what I mean. Sorry for not explaining clearly.<br>
Well, as you can see, we set groups=T, then there would be T kernels (filters) in total. What I want is, all the filters should be initialized with the same weight (share the weights), and in gradient descent, they could be updated to the same value.</p>

ptrblck: <aside class="quote no-group" data-username="Dewei_Hu" data-post="7" data-topic="118888">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dewei_hu/48/34228_2.png" class="avatar"> Dewei_Hu:</div>
<blockquote>
<p>What I want is, all the filters should be initialized with the same weight (share the weights)</p>
</blockquote>
</aside>
<p>You could certainly initialize all 8 filters to the same value either by directly set the values:</p>
<pre data-code-wrap="python"><code class="lang-python">T, D, filter_maps = 2, 3, 4
kernel_size=5
layer = nn.Conv1d(T*D,filter_maps*T,kernel_size,groups=T)

with torch.no_grad():
    ref = layer.weight[0:1]
    layer.weight.copy_(ref.repeat(8, 1, 1))

print(layer.weight)
</code></pre>
<p>or by using the functional API and stacking the filters.</p>
<aside class="quote no-group" data-username="Dewei_Hu" data-post="7" data-topic="118888">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dewei_hu/48/34228_2.png" class="avatar"> Dewei_Hu:</div>
<blockquote>
<p>and in gradient descent, they could be updated to the same value.</p>
</blockquote>
</aside>
<p>Unfortunately that won’t work directly, since you are using 2 groups. Even though all filters are equal, they would use different input channels and would thus also create different outputs and gradients:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(2, 6, 24)
out = layer(x)
out.mean().backward()
print(layer.weight.grad)
</code></pre>

Dewei_Hu: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a> bro, I came up with an idea where we can initialize the weights and biases in different groups with the same value and update them during training with the same value as well.</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
       # some other codes

        # initialize the weights
        np.random.seed(123)

        a = np.random.randn(self.num_filter_maps,self.embed_size,kernel_size)
        a = torch.from_numpy(a).type(torch.FloatTensor)
        self.cnn_weight = nn.Parameter(a)
        
        b = np.random.randn(64)
        b = torch.from_numpy(b).type(torch.FloatTensor)
        self.cnn_bias = nn.Parameter(b)

        #linear output
        self.fc = nn.Linear(num_filter_maps, label_space)
        xavier_uniform_(self.fc.weight)


    def forward(self,x):
        # pdb.set_trace()
        batch_size = x.shape[0]
        max_len = x.shape[1]
        with torch.no_grad():
            lengths = torch.count_nonzero(x,dim=-1).cpu()
        with torch.no_grad():
            conv = nn.Conv1d(50*self.embed_size,
                            self.num_filter_maps*50,
                            kernel_size=self.kernel_size,
                            padding=int(self.kernel_size//2))
        conv.weight.data = self.cnn_weight.repeat(50,1,1).data.clone()
        conv.bias.data = self.cnn_bias.repeat(50).data.clone()

        return something
</code></pre>
<p>and then during training we only update the <code>self.cnn_weights</code> and <code>self.cnn_bias</code> instead of the whole CNN layer.</p>