Where to add pos_embedding and pos_drop for two parallet network separately?

qm-intel: <p>I have a model, which is extracting features from two different networks (<code>SwinTransformer3D()</code> and <code>MyNetwork(...)</code>) in parallel and then concatenates two obtained features from two networks. The code is similar to <a href="https://discuss.pytorch.org/t/combine-two-model-on-pytorch/47858/6">this post</a>:</p>
<pre><code>class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.features1 = SwinTransformer3D(pretrained=None,
                 pretrained2d=False,
                 patch_size=self.patch_size, 
                 in_chans=1,
                 embed_dim=dim,
                 depths=[2, 2, 6, 2],
                 num_heads=[3, 6, 12, 24],
                 window_size=window_size, #(20,7,7),
                 mlp_ratio=4.,
                 qkv_bias=True,
                 qk_scale=None,
                 drop_rate=0.,
                 attn_drop_rate=0.,
                 drop_path_rate=0.2,
                 norm_layer=torch.nn.LayerNorm,
                 patch_norm=True,
                 frozen_stages=-1,
                 use_checkpoint=False)
        
        self.features2 = MyNetwork(...)
        self.dropout = nn.Dropout(emb_dropout)
        
        self.fc1 = nn.Linear(self.num_features, self.num_features)
        self.fc2 = nn.Linear(self.num_features, self.num_features)
        
        self.fc_out = nn.Linear(2*self.num_features, self.num_features)
        
    def forward(self, x):
        x = self.to_patch_embedding(x)  #ln 1

        **x = x + pos_embed**  
        **x = self.dropout(x)**

        x1 = self.features1(x)   #ln2
        x1 = x1.view(x1.size(0), -1)
        x1 = F.relu(self.fc1(x1))
        
        x2 = self.features2(x)
        x2 = x2.view(x2.size(0), -1)
        x2 = F.relu(self.fc2(x2))

        # Concatenate in dim1 (feature dimension)
        x = torch.cat((x1, x2), 2)
        x = self.fc_out(x)
        return x
</code></pre>
<p>I have a few questions:</p>
<ol>
<li>
<p>Since <a href="https://github.com/haofanwang/video-swin-transformer-pytorch/blob/aa1afa07b159f44e29c1cb43d4a7894bb77583ff/video_swin_transformer.py#L131" rel="noopener nofollow ugc">SwinTransformer</a> is computing parameters for the position bias, what is <code>pos_drop</code> is for in <a href="https://github.com/haofanwang/video-swin-transformer-pytorch/blob/aa1afa07b159f44e29c1cb43d4a7894bb77583ff/video_swin_transformer.py#L531C7-L531C48" rel="noopener nofollow ugc">this line</a>?</p>
</li>
<li>
<p>and the second network <code>MyNetwork(...)</code> has</p>
<pre><code> class MyNetwork(nn.Module):
        def __init__(self):
            super(MyNetwork, self).__init__()
            self.pos_embed = nn.Parameter(torch.zeros(1, num_patches, embed_dim))
            self.pos_drop = nn.Dropout(p=drop_rate)
            ...
        def forward_features(self, x):
            ....
            x = x + self.pos_embed
            x = self.pos_drop(x)
            ....
            return x
</code></pre>
</li>
</ol>
<p>and it is added in its <code>forward_features()</code> method, does not mismatch with the position of the token in <code>SwinTransformer()</code> and should I remove it from this class of network?</p>
<ol start="3">
<li>
<p>Since I have the same input patches (x) for both networks, should I add the position embedding to <code>MyModel</code>, <code>forward()</code> method (between ln1 and ln2)and remove the <code>self.pos_embed</code> and <code>self.pos_drop</code> from <strong>MyNetwork()</strong>  and also remove the</p>
<p><code>self.pos_drop = nn.Dropout(p=drop_rate)</code></p>
</li>
</ol>
<p>from <strong>SwinTransformer()</strong>?</p>
<p>How this may affect training?</p>
<p>I would really appreciate if you give your expert opinion on this. where should I add <code>pos_embed</code> and <code>pos_drop</code> when we have a model that is combined from two different models (in parallel) and each extracting two different features?</p>