1D convolution on 1D data

FATTOMCAT: <p>Not sure if I understod it correctly but souldnt be it possible to convolve 1dimensional input, like I have 4096 Datasets with 45 floats ?<br>
Is convolution on such an input even possible, or does it make sense to use convolution.<br>
If yes how do I setup this ?<br>
If not how yould you approach this problem ?</p>

ptrblck: <p>You can use <code>nn.Conv1d</code> to apply a convolution on a time signal.<br>
The input shape should be <code>[batch_size, channels, sequence_length]</code>.<br>
Based on your description, it seems you are dealing with 4096 samples, each containing 45 time steps and a single channel?</p>

FATTOMCAT: <p>If i reshape to [4096, 1, 27] (I removed 18 input values) I get this error:<br>
<code>RuntimeError: Given groups=1, weight of size [64, 27, 2], expected input[4096, 1, 27] to have 27 channels, but got 1 channels instead</code><br>
when changing to [4096, 27, 1] it goes crazy like <code>RuntimeError: cuDNN error: CUDNN_STATUS_BAD_PARAM</code></p>
<p>Propably the network isnt setup correctly.</p>
<p>I think I didn’t give enough information:</p>
<p>I have 3 3x3 pixel blocks, block 1/2 are different in time but block 3 is the same time as block 2.<br>
Currently they are all flat shaped [4096, 27].</p>
<p>What would a single valide convolution layer look like and whats the best way to shape this data.</p>

spanev: <p>What are your Conv1d arguments?</p>
<aside class="quote no-group" data-username="FATTOMCAT" data-post="3" data-topic="54661">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/ed8c4c/48.png" class="avatar"> FATTOMCAT:</div>
<blockquote>
<p>I have 3 3x3 pixel blocks, block 1/2 are different in time but block 3 is the same time as block 2.<br>
Currently they are all flat shaped [4096, 27].</p>
</blockquote>
</aside>
<p>I am not sure to understand here, but if you mean that you want to convolve over 3 time steps and your input data is <code>[4096, 1, 27]</code>, this should work:</p>
<pre><code class="lang-auto">&gt;&gt; output_channels = 1
&gt;&gt; t = torch.rand(4096, 1, 27)
&gt;&gt; conv = torch.nn.Conv1d(1, output_channels, kernel_size=3, padding=1)
&gt;&gt; conv
Conv1d(1, 1, kernel_size=(3,), stride=(1,), padding=(1,))
&gt;&gt; conv(t).shape
torch.Size([4096, 1, 27])
</code></pre>

FATTOMCAT: <p>Thank you, it somehow works.<br>
I still not really understand the convolution/input thing but at least no errors get raised now.</p>

Muhammad_Izaz: <p>sir i have a data set of 2968 rows and 100 columns and i can’t pass my the data to CNN please help me to prepare my data for passing and also explain the structure of 1d convolutional layers.</p>

Aldebaran: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Taking advantage of your answer, I would like to know how to apply a CNN for text processing.</p>
<p>Imagine that I have the pre-processed text (tokenized + PAD):</p>
<pre><code class="lang-python">input = torch.tensor(
    [
     [1, 2, 0, 0],
     [3, 4, 5, 6],
     [7, 0, 0, 0]
    ]
)
</code></pre>
<p>What would the <code>Conv1D</code> layer look like for this scenario? I mean, what would the <code>in_channels</code>, <code>out_channels</code>, and <code>kernel_size</code> parameters be configured?</p>
<p>Thanks in advance</p>

ptrblck: <p>It depends a bit how you would like to process this input.<br>
Currently your input has a shape of <code>[3, 4]</code>, which is invalid for <code>nn.Conv1d</code> as well as <code>nn.Conv2d</code>.<br>
If you want to use these two dimensions as the “spatial size”, i.e. similar to an input image, you would have to unsqueeze the batch and channel dimensions as:</p>
<pre><code class="lang-python">input = input[None, None]
print(input.shape)
&gt; torch.Size([1, 1, 3, 4])
</code></pre>
<p>Now you could use an <code>nn.Conv2d</code> layer with <code>in_channels=1</code> and a <code>kernel_size</code>, which would have the same size or which would be smaller than the padded input of <code>3x4</code>.</p>
<p>Note that you could also use one dimension as the channel dimension, which would then change the conv layer as well as the unsqueezing, so let me know what you would like to achieve. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Aldebaran: <p>The tensor:</p>
<pre data-code-wrap="python"><code class="lang-python">input = torch.tensor(
    [
     [1, 2, 0, 0],
     [3, 4, 5, 6],
     [7, 0, 0, 0]
    ]
)
</code></pre>
<p>is a batch of 3 preprocessed sentences in which I must represent as a dense vector. For instance <code>[1, 2, 0, 0]</code> represents the tokens id (and pad token) for the first sentence. Now I want to do something like it:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/2/52566e6d8022b4ada051c6983902bf2205563c75.png" data-download-href="https://discuss.pytorch.org/uploads/default/52566e6d8022b4ada051c6983902bf2205563c75" title="cnn_text"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52566e6d8022b4ada051c6983902bf2205563c75_2_546x500.png" alt="cnn_text" data-base62-sha1="bKokw0ErH5Vz14MJP0g5drpvkSp" width="546" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52566e6d8022b4ada051c6983902bf2205563c75_2_546x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52566e6d8022b4ada051c6983902bf2205563c75_2_819x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52566e6d8022b4ada051c6983902bf2205563c75_2_1092x1000.png 2x" data-dominant-color="EBE9E4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cnn_text</span><span class="informations">1504×1376 150 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>for each sentence in the batch.</p>
<p>I am using PyTorch Lightning (which helps a lot) but I am completely confused about how a CNN can be used for text representation.</p>
<p>I think that before passing the input through a convolution block, I could go through an embedding layer, which would produce a shaped tensor <code>[3,4,768]</code> (batch_size, sentence_size, representation_size).</p>

ptrblck: <p>If the input represents word indices then an embedding layer sounds reasonable.<br>
I don’t fully understand the figure and how the convolution should be applied.<br>
Are 3 sets of filters with different sizes used? If so, you would need three conv layers or would need to pad some filters.</p>

Aldebaran: <aside class="quote no-group" data-username="Aldebaran" data-post="9" data-topic="54661">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aldebaran/48/23963_2.png" class="avatar"> Aldebaran:</div>
<blockquote>
<p>representation_size</p>
</blockquote>
</aside>
<p>With this new image I think my objective becomes clearer:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5faf64f4eb86dd37121774c720877b1d44f7f617.gif" alt="conv_maxpooling_steps" data-base62-sha1="dEt8HCCfBEDp1R9j0NnlCBI8Cnd" width="690" height="340" class="animated"></p>
<p>The filters/kernels are like a sliding window of different sizes. In the previous image, there are 2 filters of shape <code>[2, 5]</code>, 2 filters of shape <code>[3, 5]</code>, and other 2 filters of shape <code>[4, 5]</code>.</p>
<p>For now, the model has only one embedding layer:</p>
<pre data-code-wrap="python"><code class="lang-python"># batch of 3 sentences
input = torch.tensor(
    [
     [1, 2, 0, 0], # tokenized sentence 1
     [3, 4, 5, 6], # tokenized sentence 2
     [7, 0, 0, 0]  # tokenized sentence 3
    ]
)

embedding_layer = torch.nn.Embedding(num_embeddings = 8, # vocabulary size
                                     embedding_dim=5, # representation size
)

emb_out = embedding_layer(input) # torch.Size([3, 4, 5]) (batch_size, sentence_size, representation_size)
conv = torch.nn.Conv1d(in_channels=?,out_channels=?, kernel_size=?)
</code></pre>
<p>and, what I need to know is then, how to pass the embedding layer output into the convolutional layer as shown in the figure above.</p>
<p>Thanks in advance.</p>

ptrblck: <p>Thanks for the update.<br>
In that case you would need to unsqueeze the channel dimension via:</p>
<pre><code class="lang-python">emb_out = emb_out.unsqueeze(1)
</code></pre>
<p>and I would use 3 different conv layers in the first step via:</p>
<pre><code class="lang-python">conv1 = nn.Conv2d(in_channels=1, out_channels=2, kernel_size=(2, 5))
conv1 = nn.Conv2d(in_channels=1, out_channels=2, kernel_size=(3, 5))
conv1 = nn.Conv2d(in_channels=1, out_channels=2, kernel_size=(4, 5))
</code></pre>
<p>After applying these conv layers, you could pass the outputs through the pooling layers and concatenate the the final features as given in the figure.</p>
<p>Note that your first figure indicates that no padding is used and thus the output activation would be smaller, while your current 3x3 filter approach is using a padding value of 1, since the feature tensor has the same number of rows.</p>

Aldebaran: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
The final implemented model is doing great. Check it out:</p>
<pre><code class="lang-python">class CNNSentenceEncoder(nn.Module):
    """Represents a text as a dense vector."""

    def __init__(self, vocabulary_size, representation_size, out_channels, kernel_sizes, sentence_length):
        """
        :param vocabulary_size: number of tokens in corpus. 
        :param representation_size: dense vector length.
        :param out_channels: number of kernels.
        :param kernel_sizes:  list of kernel sizes ([2,3,5], for instance)
        :param sentence_length: max number of tokens for sentences.
        """
        super(CNNEncoder, self).__init__()

        # embedding layer
        self.embedding = nn.Embedding(
            num_embeddings=vocabulary_size,
            embedding_dim=representation_size
        )

        # convolutional layers
        self.convs = nn.ModuleList([
            self.get_conv_layer(representation_size, out_channels, kernel_size, sentence_length)
            for kernel_size in kernel_sizes])

      

        self.linear = nn.Linear(len(convs) * out_channels, representation_size)

    def get_conv_layer(self, representation_size, out_channels, kernel_size, sentence_length):
        """
        Defines a convolutional block.
        """
        return nn.Sequential(
            nn.Conv1d(in_channels=representation_size, out_channels=out_channels, kernel_size=kernel_size),
            nn.ReLU(),
            nn.MaxPool1d(sentence_length - kernel_size + 1, stride=1),
            nn.BatchNorm1d(out_channels)
        )

    def forward(self, x):
        r1 = self.embedding(x)
        r1 = torch.transpose(r1, 2, 1)

        conv_outputs = []
        for conv in self.convs:
            conv_outputs.append(conv(r1))

        # concatenates the outputs from each convolutional layer
        cat = torch.cat(conv_outputs, 1)
        
        # flatten
        flatten_cat = torch.flatten(cat, start_dim=1)
        return self.linear(flatten_cat)

</code></pre>