Conv1d layer for text classification

Nouf: <p>Hello everyone,<br>
I have a question regarding the Conv1d in torch,<br>
the simple model below, which works with text classification, has a ModuleList containing three Conv1d layers (each one dedicated to a specific filter size)</p>
<blockquote>
<p>import torch<br>
import torch.nn as nn</p>
<p>class TextClassifier(nn.Module):<br>
def <strong>init</strong>(self, vocab_size, embedding_dim, num_classes):<br>
super(TextClassifier, self).<strong>init</strong>()</p>
<pre><code>    self.embedding = nn.Embedding(vocab_size, embedding_dim)
    self.conv_layers = nn.ModuleList([
        nn.Conv1d(embedding_dim, 100, kernel_size=2),
        nn.Conv1d(embedding_dim, 100, kernel_size=3),
        nn.Conv1d(embedding_dim, 100, kernel_size=4)
    ])
    self.fc = nn.Linear(300, num_classes)
</code></pre>
</blockquote>
<p>my question is: it is necessary to use ModuleList and have Conv1d layer for each filter size, I mean is it work to have just one Conv1d layer that has the different filter sizes (2,3,4)</p>

J_Johnson: <p>A ModuleList is just one way of organizing layers and is especially useful if you plan to use a for loop. What you do with those layers will be determined in the forward pass. (i.e. make them sequential or have them parallel process the same inputs).</p>
<p>Suppose you wanted 100x Conv1d layers with kernel sizes of 1 to 100, each taking the same embedded text input. You could do:</p>
<pre><code class="lang-auto">self.kernels = nn.ModuleList([])
for i in range(100):
    self.kernels.append(nn.Sequential(
            nn.Conv1d(embedding_dim, 100, kernel_size = i + 1), 
            nn.BatchNorm1d(100), 
            nn.ReLU()
            )
            )
...

#in the forward pass, with x as input
contexts = []
for i in range(100):
    y = self.kernels[i](x)
    contexts.append(y)
...
</code></pre>
<p>An individual Conv1d layer can only have one kernel_size.</p>

Nouf: <p>Thank you <a class="mention" href="/u/j_johnson">@J_Johnson</a> for your reply,<br>
this is the forward function</p>
<pre><code>def forward(self, x):
    embedded = self.embedding(x)  # shape: (batch_size, seq_len, embedding_dim)
    
    conv_outputs = []
    for conv in self.conv_layers:
        conv_out = torch.relu(conv(embedded.transpose(1, 2)))  # shape: (batch_size, num_filters_i, seq_len - kernel_size + 1)
        pooled_out = torch.max_pool1d(conv_out, conv_out.size(2)).squeeze(2)  # shape: (batch_size, num_filters_i)
        conv_outputs.append(pooled_out)
    
    concat_output = torch.cat(conv_outputs, dim=1)  # shape: (batch_size, sum(num_filters))
    
    logits = self.fc(concat_output)  # shape: (batch_size, num_classes)
    
    return logits
</code></pre>
<p>I can understand from your answer, we can not have a single Conv1d layer containing different kernel sizes , is that right?</p>

J_Johnson: <p>Right. If you want multiple kernels, use s ModuleList to store the convolutional layers with the specific kernel/padding/stride/etc you want for each.</p>

Nouf: <p>Thank you <a class="mention" href="/u/j_johnson">@J_Johnson</a></p>

vdw: <p>If you want to look at another <a href="https://github.com/chrisvdweth/ml-toolkit/blob/master/pytorch/models/text/classifier/cnn.py" rel="noopener nofollow ugc">example code</a>, I have an old implementation of the architecture shown below.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/b/fba6993773b4754dfd5c008d3e2d06e337dc8084.png" data-download-href="https://discuss.pytorch.org/uploads/default/fba6993773b4754dfd5c008d3e2d06e337dc8084" title="cnn-sentence-classification"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fba6993773b4754dfd5c008d3e2d06e337dc8084_2_546x500.png" alt="cnn-sentence-classification" data-base62-sha1="zUcMp7jEw2Bo7JFfc0VVMNFtGII" width="546" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fba6993773b4754dfd5c008d3e2d06e337dc8084_2_546x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fba6993773b4754dfd5c008d3e2d06e337dc8084_2_819x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fba6993773b4754dfd5c008d3e2d06e337dc8084_2_1092x1000.png 2x" data-dominant-color="EAE9E3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cnn-sentence-classification</span><span class="informations">1400×1281 193 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The only difference is that I don’t use 1-max pooling but normal max-pooling. This implementation uses a <code>nn.ModuleDict</code> for a change :).</p>
<p>Please note that the code is rather verbose not make model very configurable, by specifying different <code>conv_kernel_sizes</code> as input parameter.</p>