Combination of features of convolutional layers channel-by-channel in a multi-branch model

username: Tm4
cooked: <p>The convolutional model presented below, has two branches and each branch  (for example) has two stages (convolutional layers).<br>
My aim is to combine the weighted feature maps (channels) of the first convolutional layer from the second branch with the channels of the first convolutional layer from the first branch.</p>
<p>I want to extract the channels from the first convolutional layer in the second branch, multiply it by a weight (weight is a class in the code that makes the output a weighted version of its input) and stack it with the channels of its counterpart convolutional layer from the first branch. Afterwards, by utilizing a 1x1 conv2d, the size of the stacked feature maps will be changed to its initial size and this combined channels should be used by the first branch and the next convolutional layers will be computed based on these combined channels. After that, I want to have this kind of combination between the second convolutional layers of the branches. (In other words, I want to combine features channel-by-channels between branches.)</p>
<p>Please find the <code>main_class</code> (the whole model that consists of two branches) and the <code>first_branch</code> and <code>second_branch</code> below:</p>
<pre><code class="lang-auto">class main_class(nn.Module):
    def __init__(self, pretrained=False):
        super(main_class, self).__init__()
        self.input=input_data()  # input_data is a class the provides the input data for the each branch
        
        self.conv_t2 = BasicConv3d(...........)
        self.second_branch=second_branch(512, out_sigmoid=True)
        
        self.conv_t1 = BasicConv3d(..............)
        self.first_branch=first_branch(512, out_sigmoid=True)
  
        self.last = nn.Conv2d(4, 1, kernel_size=1, stride=1)
        
        self.sigmoid = nn.Sigmoid()

        def forward(self, x, par = False):
        
         x1, x2 = self.input(x)
        
        #second branch
        y2 = self.conv_t2(x2)
        out2 = self.second_branch(y2)
        
        #first branch
        y1 = self.conv_t1(x1)
        out1 = self.first_branch(y1)
        
        x = torch.cat((out2, out1), 1)
        x = self.last(x)
        out = self.sigmoid(x)
        
        if par:
            return  out1, out2, out
        
        return out
</code></pre>
<p>The <code>first_branch</code>:</p>
<pre><code class="lang-auto">class first_branch(nn.Module):
    def __init__(self, in_channel=512, out_channel=[380, 200], out_sigmoid=False):
        super(first_branch, self).__init__()
        
        self.out_sigmoid=out_sigmoid

        self.deconvlayer1_2 = self._make_deconv(in_channel, out_channel[0], num_conv=3)
        self.upsample1_2=Upsample(scale_factor=2, mode='bilinear')
        self.combined1_2 = nn.conv2d(720, 380, kernel_size=1, stride=1, padding=0)

        self.deconvlayer1_1 = self._make_deconv(out_channel[0], out_channel[1], num_conv=3)
        self.upsample1_1=Upsample(scale_factor=2, mode='bilinear')
        self.combined1_1 = nn.conv2d(400, 200, kernel_size=1, stride=1, padding=0)

    def forward(self, x):
        x=self.deconvlayer1_2(x)
        x = self.upsample1_2(x)
        x=self.deconvlayer1_1(x)
        x = self.upsample1_1(x)
        
        if self.out_sigmoid:
            x=self.sigmoid(x)
        
        return x
</code></pre>
<p>The <code>second_branch</code>:</p>
<pre><code class="lang-auto">class second_branch(nn.Module):
    def __init__(self, in_channel=512, out_channel=[380,200], out_sigmoid=False):
        super(second_branch, self).__init__()
        
        self.out_sigmoid=out_sigmoid
        
        self.weight = weight()  # weight is a class that weighted its input

        self.deconvlayer2_2 = self._make_deconv(in_channel, out_channel[0], num_conv=3)
        self.upsample2_2=Upsample(scale_factor=2, mode='bilinear')
        self.deconvlayer2_! = self._make_deconv(out_channel[0], out_channel[1], num_conv=3)
        self.upsample2_1=Upsample(scale_factor=2, mode='bilinear')
        
    
    def forward(self, x):
        x=self.deconvlayer2_2(x)
        x = self.upsample2_2(x)
        weighted2_2 = self.weight(x)

        x=self.deconvlayer2_1(x)
        x = self.upsample2_1(x)
        weighted2_1 = self.weight(x)

        
        if self.out_sigmoid:
            x=self.sigmoid(x)
        
        return x, weighted2_1, weighted2_2
</code></pre>
<p>For implementing the mentioned idea, I modified the <code>main_class</code> as follows (instead of using the <code>first_branch</code> class in the forward function of the <code>main_class</code>, I wrote the script lines of the forward function of the <code>first_branch</code> in the forward function of the <code>main_class</code>):</p>
<pre><code class="lang-auto">class main_class(nn.Module):
    def __init__(self, pretrained=False):
        super(main_class, self).__init__()
        self.input=input_data()  # input_data is a class the provides the input data for the each branch
        
        self.conv_t2 = BasicConv3d(....................)
        self.second_branch=second_branch(512, out_sigmoid=True)
        
        self.conv_t1 = BasicConv3d(............)
        self.first_branch=first_branch(512, out_sigmoid=True)
  
        self.last = nn.Conv2d(4, 1, kernel_size=1, stride=1)
        
        self.sigmoid = nn.Sigmoid()

        def forward(self, x, par = False):
        
        x1, x2 = self.input(x)
        
        #second branch
        y2 = self.conv_t2(x2)

        out2, weighted2_1, weighted2_2 = self.second_branch(y2)

        
        #first branch
        y1 = self.conv_t1(x1)

       # instead of using from class first_branch, again I write the script lines of first_branch.forward() in below:
        x=self.deconvlayer1_2(y1)
        x = self.upsample1_2(x)
        stacking_2 = torch.stack(x, weighted2_2)
        x = self.frist_branch.combined1_2(stacking_2)


        x=self.deconvlayer1_1(x)
        x = self.upsample1_1(x)
        stacking_1 = torch.stack(x, weighted2_1)
        x = self.frist_branch.combined1_1(stacking_1)
        
        out1=self.sigmoid(x)


        
        x = torch.cat((out2, out1), 1)
        x = self.last(x)
        out = self.sigmoid(x)
        
        if par:
            return  out1, out2, out
        
        return out
</code></pre>
<p>I faced with the following error:</p>
<blockquote>
<p>TypeError: Cannot create a consistent method resolution order (MRO) for bases Module, second_branch</p>
</blockquote>
<p>How can I fix this problem and how can I make the code able to have the interactions between new branches that may be added later to the model (for example if I have three branches, how can I have this kind of data combination between the third branch and the second one, and between the output of the previous combination and the first branch)?</p>

username: aiGeekProgrammer
cooked: <aside class="quote no-group quote-modified" data-username="Tm4" data-post="1" data-topic="149302">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/b38774/48.png" class="avatar"> Tm4:</div>
<blockquote>
<pre><code class="lang-auto">class my_main_class(nn.Module):
    def __init__(self, pretrained=False):
        super(main_class, self).__init__()
</code></pre>
</blockquote>
</aside>
<p>Did you try:<br>
super(my_main_class, self).<strong>init</strong>()</p>
<p>Not saying this will help, but…</p>

username: Tm4
cooked: <p>Thanks for your reply. In the question I named the modified class to my_main_class to be more clear but I have used your idea in the source code. In other words, my_main_class is the main_class that the idea is applied on that. The name of the the modified class is change to avoid ambiguity.</p>