How can i run CNN encoder for tensors of different shape?

jubick: <p>I want to have CNN encoder for my texts.<br>
Here is what i have now:</p>
<pre><code class="lang-auto">class CNNEncoder(nn.Module):
    def __init__(self, embed_dim: int, vocab_size: int):
        super(CNNEncoder, self).__init__()

        self.embed = nn.Embedding(vocab_size, embed_dim)
        self.conv1 = nn.Conv1d(embed_dim, 120, 1)
        self.pool = nn.AvgPool1d(1, 1)

    def forward(self, x):
        x = self.embed(x)
        if len(x.shape) == 2:
            x = x.unsqueeze_(1).permute([0, 2, 1])
        x = self.conv1(x)
        x = torch.tanh(x)
        x = self.pool(x)
        return x
</code></pre>
<p>When i pass it tensor of size [120] it works fine.<br>
But i need to encode a matrix NxK of sentences. It has size [5, 20, 120].<br>
How can i deal with that?</p>

ptrblck: <p><code>nn.Embedding</code> will output a tensor in the shape <code>[*, H]</code>, where <code>*</code> is the input shape and <code>H</code> the <code>embedding_dim</code>.<br>
Given your input shape, you’ll end up with a 4-dimensional tensor.<br>
Could you explain a bit, in which dimension the conv layer should operate?<br>
I.e. should the filters convole the “sentence dimension” or the “word dimension”?</p>

jubick: <p>If we talk about one sentence i.e. x.shape = [batch_size, sentence_length] after embedding i get x.shape = [batch_size, sentence_length, embedding_dim].<br>
I want to convolve over words in this sentence.<br>
So as it’s told in one of CNN tutorials in pytorch i need to remute and end up with [batch_size, embedding_dim, sentence_length] before convolution.<br>
After convolution i’m getting strange shaped tensor  [1, 100, 118]. 1 is batch_size in my current setting but i don’t get where other dims came from as i have embedding_dim=300 and sentence_length=120.<br>
It seems like i’m using it wrong. I know what convolutional layer is as i worked with it in tf but torch api seems hard to me.<br>
Correct me if i’m using it wrong please.<br>
Also here’s my updated code.</p>
<pre><code class="lang-auto">class CNNEncoder(nn.Module):
    def __init__(self, embed_dim: int, vocab_size: int):
        super(CNNEncoder, self).__init__()

        self.embed = nn.Embedding(vocab_size, embed_dim)
        self.conv1 = nn.Conv1d(embed_dim, 100, 3)
        self.pool = nn.AvgPool1d(1, 1)

    def forward(self, x):
        x.unsqueeze_(0)
        x = self.embed(x)
        x = x.permute(0, 2, 1)
        x = self.conv1(x)
        x = torch.tanh(x)
        x = self.pool(x)
        x.unsqueeze_(1)
        return x
</code></pre>
<p>P.S.<br>
Main question is actually about using same model for sentence (shape [batch_size, sentence_length]) and for matrix of sentences (shape [K, N, sentence_length])<br>
But i think i need to figure out first if i’m using this model right just for sentence and then deal with matrix question.</p>

jubick: <p>Actually i think i solved main question by doing this:</p>
<pre><code class="lang-auto">        encoded_y_query = []

        for classs in y_query.split(1):
            encoded_sents = []
            for sentence in classs.squeeze_().split(1):
                encoded_sents.append(self.cnn_encoder.forward(sentence.squeeze_()))
            encoded_y_query.append(encoded_sents)
        y_tensors = []
        for i in range(y_query.shape[0]):
            temp_tens = []
            for j in range(y_query.shape[1]):
                temp_tens.append(torch.tensor(encoded_y_query[i][j]))

            y_tensors.append(torch.cat(temp_tens))

        y_query = torch.cat(y_tensors)
</code></pre>
<p>This code does forward pass for each sentence in tensor [K,N, sentence_length].<br>
Although it needs a little of fine tuning due to batch_size. But i’ll sure handle that.<br>
But i still would want ti know if my cnn works as i want it to work (described in previous post).<br>
I’d be readlly glad if you could help me with that.</p>

ptrblck: <aside class="quote no-group" data-username="jubick" data-post="3" data-topic="75770">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jubick/48/22454_2.png" class="avatar"> jubick:</div>
<blockquote>
<p>After convolution i’m getting strange shaped tensor [1, 100, 118].</p>
</blockquote>
</aside>
<p>You’ll get this output, since you are passing an input of  <code>[batch_size=5, emb_dim=300, seq_len=120]</code> to your conv layer, which will output <code>[5, 100, 118]</code>.<br>
<code>dim1=100</code> is defined by the number of filters (<code>out_channels</code>), while the sequential dimension is decreased due to the kernel size without padding.</p>
<p>I’m not completely sure, what your last code snippet does.<br>
Are you splitting the input into sentences and pass each sentence separately?</p>

jubick: <p>Yes, i split tensor [4,5, 120] and pass each sentence to encoder.<br>
Here’s final version of correct encoder:</p>
<pre><code class="lang-auto">class CNNEncoder(nn.Module):
    def __init__(self, embed_dim: int, vocab_size: int):
        super(CNNEncoder, self).__init__()
        self.embed = nn.Embedding(vocab_size, embed_dim)
        self.conv1 = nn.Conv2d(1, 100, (2, embed_dim))

    def forward(self, x):
        x = self.embed(x)
        x = x.unsqueeze(0)
        x = x.unsqueeze(0)
        x = self.conv1(x)
        x = x.squeeze(3)
        x = torch.tanh(x)
        x = F.avg_pool1d(x, x.size()[2]).squeeze(2)
        return x
</code></pre>
<p>I’m not sure if this method of splitting tensor is ok.<br>
Here it is (latest update):</p>
<pre><code class="lang-auto">for class_row in support_set.split(1):
    encoded_sents = []
    for sentence in class_row.squeeze_().split(1):
        encoded_sents.append(self.cnn_encoder.forward(sentence.squeeze_()))
     encoded_support_set.append(encoded_sents)

class_tensors = []
for i in range(support_set.shape[0]):
    sentence_tensors = []
    for j in range(support_set.shape[1]):
        sentence_tensors.append(torch.tensor(encoded_support_set[i][j].clone().detach().requires_grad_(True)))
    class_tensors.append(torch.cat(sentence_tensors))

support_set = torch.stack(class_tensors)
</code></pre>