Dynamic layers combination

username: Aiman_Mutasem-bellh
cooked: <p>Hi all <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>I’m trying to <strong>combine encoder layers output dynamically</strong> as below figure :</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/e/7e5826aa579e7b4533ffefe0f99bd29878404ae2.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/7e5826aa579e7b4533ffefe0f99bd29878404ae2" title="WeChat Image_20210309171047"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/e/7e5826aa579e7b4533ffefe0f99bd29878404ae2_2_690x449.jpeg" alt="WeChat Image_20210309171047" data-base62-sha1="i1H1ughucvbq3qYiSRFYBCSSw4W" width="690" height="449" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/e/7e5826aa579e7b4533ffefe0f99bd29878404ae2_2_690x449.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/7/e/7e5826aa579e7b4533ffefe0f99bd29878404ae2.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/e/7e5826aa579e7b4533ffefe0f99bd29878404ae2.jpeg 2x" data-dominant-color="EBEBEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">WeChat Image_20210309171047</span><span class="informations">800×521 40.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have followed an answer <a href="https://discuss.pytorch.org/t/concatenate-layer-output-with-additional-input-data/20462">here</a> using <code>torch.cat()</code>, and my code as below:</p>
<pre><code class="lang-auto">        #---------------------------------------- Start dynamic combination ----------------------------
        
        for layer in self.layers:
            src = layer(src, src_mask)
            
            # I have three layers and I expect 3 vectors
            src = torch.cat([src]).view(-1)
        
        src = self.fc_o(src)
        #src = [batch size, src len, hid dim]
</code></pre>
<pre><code class="lang-auto">class Encoder(nn.Module):
    def __init__(self, 
                 input_dim, 
                 hid_dim, 
                 n_layers, 
                 n_heads, 
                 pf_dim,
                 dropout, 
                 device,
                 max_length = 100):
        super().__init__()

        self.device = device
        
        self.n_layers = n_layers
        
        self.tok_embedding = nn.Embedding(input_dim, hid_dim)
        self.pos_embedding = nn.Embedding(max_length, hid_dim)
        
        self.layers = nn.ModuleList([EncoderLayer(hid_dim, 
                                                  n_heads, 
                                                  pf_dim,
                                                  dropout, 
                                                  device) 
                                     for _ in range(n_layers)])
        self.fc_o = nn.Linear(BATCH_SIZE, input_dim, hid_dim)
        self.dropout = nn.Dropout(dropout)
        
        self.scale = torch.sqrt(torch.FloatTensor([hid_dim])).to(device)
 
        #self.fc_o = nn.Linear(BATCH_SIZE, input_dim, hid_dim)
        
    def forward(self, src, src_mask):
        
        #src = [batch size, src len]
        #src_mask = [batch size, src len]
        
        batch_size = src.shape[0]
        src_len = src.shape[1]
        
        pos = torch.arange(0, src_len).unsqueeze(0).repeat(batch_size, 1).to(self.device)
        
        #pos = [batch size, src len]
        
        src = self.dropout((self.tok_embedding(src) * self.scale) + self.pos_embedding(pos))
        
        #src = [batch size, src len, hid dim]
        
        #---------------------------------------- Start dynamic combination ----------------------------
        
        outputs = []
        
        for layer in self.layers:
            outputs.clear()
            for layer in self.layers:
                src = layer(src, src_mask)
                outputs.append(src)
                
            src = torch.cat(outputs, dim=1) 
            src = self.fc_o(src)


        return src
</code></pre>
<p>Actually, I’m not sure about my implementation also I got an error:</p>
<p><code>RuntimeError: size mismatch, m1: [1 x 753664], m2: [256 x 256] at /opt/conda/conda-bld/pytorch_1573049301898/work/aten/src/THC/generic/THCTensorMathBlas.cu:290</code></p>
<p>Any suggestions to fix this error and to improve the implementation?</p>
<p>Kind regards,<br>
Aiman Solyman</p>

username: Unity05
cooked: <p>Hi,</p>
<p>in your linear layer definition, you should not specify the batch size (<code> *in_features* , *out_features* , *bias=True*</code> <a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html" rel="noopener nofollow ugc">[Docs]</a>). This does most likely cause your error. However, your concatentation does not make sense to me as your just do <code>view(-1)</code> effectively. You could concat compressed layer outputs (as you have to pay attention to not increase the size to fast) seperatly. (Just regarding the encoder part right now.)</p>
<p>Regards,<br>
Unity05</p>

username: Aiman_Mutasem-bellh
cooked: <p>Thank you sir for your valuable comment.</p>
<p>I have tried to apply the encoder without forward layer forward <code>src = self.fc_o(src)</code>, and I got the same error. Here, the issue is how to concatenate layers output dynamically.  The updated code as below:</p>
<pre><code class="lang-auto">        outputs = []
        
        for layer in self.layers:
            outputs.clear()
            for layer in self.layers:
                src = layer(src, src_mask)
                outputs.append(src)
            x = torch.cat(outputs, dim=1) 
            x = self.fc_o(x)
</code></pre>
<p>Moreover, the objective of this type of combination is that I’m trying to apply routing by agreement using capsule network following this <a href="https://arxiv.org/abs/1902.05770" rel="noopener nofollow ugc">paper</a>. The first step is to aggregate the encoder layers output dynamically, then feed them to CapsNet.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/4/9429dce3e8b468d33af3ccd0c5efee1a7e1c0e59.png" alt="Untitled_2" data-base62-sha1="l8Im5U1VGOLsvlUNskx0Te661xD" width="430" height="478"></p>
<p>Could you help with this case?</p>

username: Unity05
cooked: <p>Hi,</p>
<p>Ah okay, now I see what you try to do. Your updated code snipped looks fine to me. Now, implementing a routing mechanism like the <code>Dynamic Routing</code> mentioned in the paper should be pretty straight forward.</p>
<p>Regards,<br>
Unity05</p>