How to treat variable length sequence in Conv1D layers?

cerebralnetworkdev: <p>I’ve got a set of sequences that are shaped like this:</p>
<pre><code class="lang-auto"># batch_size x features x length
seq = tensor([[[-1.6664,  1.3793, -1.0168, -0.2382,  0.3806, -1.4693,  0.0000,
           0.0000],
         [ 0.9992, -0.7673, -0.4289, -0.6654, -1.0930, -0.7454,  0.0000,
           0.0000],
         [ 1.4257,  0.9920, -0.1028,  0.8902,  0.5898,  0.1072,  0.0000,
           0.0000],
         [ 0.1998,  1.0295, -0.2461,  0.1640,  0.5859,  0.6036,  0.0000,
           0.0000]],

        [[ 0.7872,  0.3735, -0.5858, -0.6331, -1.7811, -0.2623,  0.8302,
          -1.5729],
         [-1.7866, -0.9432, -0.0326,  0.5870,  0.9642,  2.0408, -0.8691,
          -1.8870],
         [-2.1594, -0.4498,  1.0198,  0.7867,  0.9520, -0.8631, -0.5116,
           0.2256],
         [ 2.0375,  0.5512, -1.4802, -0.5710,  0.8688,  0.8487,  1.3697,
           1.0811]]])
</code></pre>
<p>As you can see, the last 2 positions of the 0th sequence is just <code>0.000</code>s to represent padding.<br>
What would be the recommendation here? Even a layer like this “compatible”:</p>
<pre><code class="lang-auto">conv = nn.Conv1D(n_input_features, n_output_features, kernel_size = 3)
o = conv(seq) # this works
</code></pre>
<p>For example, if I had a <code>seq</code> tensor of <code>N x C x L</code> with a kernel of, say, 3, then I get a new tensor with <code>N x C x L-2</code>, and this applies even for sequences with 0 padding.</p>
<p>I’ve read <a href="https://discuss.pytorch.org/t/deal-with-variable-length-sequence-for-1d-convolution-batch-normalization/6748">here</a> to try and clip the output? Would the idea then be to do something like this?</p>
<pre><code class="lang-auto">o = conv(seq)
# o[:, :, :-seq_lengths]  something along these lines?
</code></pre>
<p>I’m asking as I’m trying to migrate a Keras 1D convolution into PyTorch. The Keras layer is written as a “MaskedConvolution” that involves a <code>Masking</code> layer from Keras (presumably to handle variable lengths?). From my understanding it’s not a true Masked convolution <em>per se</em> where the kernel has a 0…</p>
<p>In addition to above about handling variable length sequences in <code>nn.Conv1d</code>, if anyone has tips for Keras’ <code>Masking</code> equivalent would be for PyTorch, I’d also be keen to hear!</p>

ptrblck: <p>Your slicing operation would remove the padding indices in all samples, so I’m unsure if this fits your use case.</p>
<aside class="quote no-group" data-username="cerebralnetworkdev" data-post="1" data-topic="110242">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cerebralnetworkdev/48/29874_2.png" class="avatar"> cerebralnetworkdev:</div>
<blockquote>
<p>In addition to above about handling variable length sequences in <code>nn.Conv1d</code>, if anyone has tips for Keras’ <code>Masking</code> equivalent would be for PyTorch, I’d also be keen to hear!</p>
</blockquote>
</aside>
<p>Could you explain, what this layer is doing, i.e. how the “masking” is performed and how the outputs and gradients are changed by it?</p>

cerebralnetworkdev: <p>Hi Piotr,</p>
<p>Thanks – so the code is a bit unusual, but here we are</p>
<pre><code class="lang-auto">## this is all Keras that I'm trying to understand
class MaskedConvolution1D(Convolution1D):
    def __init__(self, *args, **kwargs):
        super(MaskedConvolution1D, self).__init__(*args, **kwargs)

    def compute_mask(self, input, input_mask=None):
        return input_mask

    def call(self, x, mask=None):
        assert mask is not None
        mask = K.expand_dims(mask, axis=-1)
        x = super(MaskedConvolution1D, self).call(x)
        return x * K.cast(mask, K.floatx())

class MaskingByLambda(Layer):
    def __init__(self, func, **kwargs):
        self.supports_masking = True
        self.mask_func = func
        super(MaskingByLambda, self).__init__(**kwargs)

    def compute_mask(self, input, input_mask=None):
        return self.mask_func(input, input_mask)

    def call(self, x, mask=None):
        exd_mask = K.expand_dims(self.mask_func(x, mask), axis=-1)
        return x * K.cast(exd_mask, K.floatx())

def lambda_func(tensor):
    return lambda input, mask: tensor

def model(length):
    input = Input(shape=(length, n_features))
    mask = Input(shape=(length,))
    seq = MaskingByLambda(lambda_func(mask))(input)
    conv = MaskedConvolution1D(28, 3, padding='same', activation='relu')(seq)
</code></pre>
<p>The dataset I’m working on is a set of sequences, such as:</p>
<pre><code class="lang-auto">x = ["This is a sentence", "This is another longer sentence", "Short sentence", "AHHH!"]
</code></pre>
<p>This is then tokenised into a 1-0 encoded tensor to be fed into the Conv1D layer. I’m just wondering most importantly how a <code>nn.Conv1d</code> can handle variable length sequences, and how it can replicate the functionality of the <code>MaskedConvolution1D</code> implementation above.</p>
<p>Thanks!!</p>

ptrblck: <aside class="quote no-group" data-username="cerebralnetworkdev" data-post="3" data-topic="110242">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cerebralnetworkdev/48/29874_2.png" class="avatar"> cerebralnetworkdev:</div>
<blockquote>
<p>I’m just wondering most importantly how a <code>nn.Conv1d</code> can handle variable length sequences</p>
</blockquote>
</aside>
<p>Conv layers can handle variable sized inputs as long as the kernel size is larger than the temporal or spatial size of the input. The output shape will be determined by the kernel size, stride, padding, and dilation as given in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv1d.html#torch.nn.Conv1d">docs</a>.</p>
<aside class="quote no-group" data-username="cerebralnetworkdev" data-post="3" data-topic="110242">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cerebralnetworkdev/48/29874_2.png" class="avatar"> cerebralnetworkdev:</div>
<blockquote>
<p>and how it can replicate the functionality of the <code>MaskedConvolution1D</code> implementation above.</p>
</blockquote>
</aside>
<p>Based on the code it seems you are creating a mask and multiplying it with the output activation of the conv layer. If that’s the case, you could use exactly the same workflow in the <code>forward</code> method of your custom conv layer.</p>

cerebralnetworkdev: <p>Piotr, that was <em>exactly</em> the solution. Thanks for posting, I forgot to write this down after my Eureka moment yesterday <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>
<p>For anyone else wanting to do something similar, it was really a case of something like this:</p>
<pre><code class="lang-auto">class MaskedConvolution(nn.Module):
    # ...
    def forward(self, x: torch.Tensor, mask: torch.BoolTensor) -&gt; torch.Tensor:
        o = self. .... # do fancy operations here
        assert o.shape == mask.shape, "Mask shape should be the same as the output shape"
        return o * mask
</code></pre>