Combine two model on pytorch?

Daniel_Joseph: <p>Hello everyone,<br>
How to combine two models parameter of two different datasets to generate one model like :</p>
<pre><code class="lang-auto">class NetworkA(nn.Module):
    def __init__(self, Input, Output):
        super(NetworkA, self).__init__()
    def forward(self, x):
        return x

class NetworkB(nn.Module):
    def __init__(self, Input, Output):
        super(NetworkB, self).__init__()
    def forward(self, x):
        return x
</code></pre>
<p>In the Training Step combine the two parameters , then, for Testing i use only the combined resulted model.<br>
Any suggestion, ideas would be much appreciated.</p>

ptrblck: <p>How would you like to combine the parameters of the two models?<br>
Could you give an example of the use case you are thinking about?</p>

Daniel_Joseph: <p>I have datasets that has many channels(hyperspectral images) where each pixel has a cube of value “Pi(1,1,200)”, i would like to create a model for spatial features and other for spectral features for the same datasets but reshaped in different way, and then combine these models.</p>

ptrblck: <p>Thanks for the information.<br>
In this case, you could use two submodules (each working on the specific data samples) and concatenate these features later in the model.<br>
The concatenation might be applied e.g. on the activation volumes or the linear activation outputs.</p>

Daniel_Joseph: <p>can you kindly provide some example oh how to concatenate them , just to get it clearly, still could not work it out.<br>
Thanks Ptrblck for helping <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>

ptrblck: <p>Sure! Here is a small example concatenating the outputs of two linear layers:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.features1 = nn.Sequential(
            nn.Conv2d(3, 6, 3, 1, 1),
            nn.ReLU(),
            nn.MaxPool2d(2),
            nn.Conv2d(6, 12, 3, 1, 1),
            nn.ReLU(),
            nn.MaxPool2d(2)
        )
        
        self.features2 = nn.Sequential(
            nn.Conv2d(1, 6, 3, 1, 1),
            nn.ReLU(),
            nn.MaxPool2d(2),
            nn.Conv2d(6, 12, 3, 1, 1),
            nn.ReLU(),
            nn.MaxPool2d(2)
        )
        
        self.fc1 = nn.Linear(12*6*6, 64)
        self.fc2 = nn.Linear(12*6*6, 64)
        
        self.fc_out = nn.Linear(128, 10)
        
    def forward(self, x1, x2):
        x1 = self.features1(x1)
        x1 = x1.view(x1.size(0), -1)
        x1 = F.relu(self.fc1(x1))
        
        x2 = self.features2(x2)
        x2 = x2.view(x2.size(0), -1)
        x2 = F.relu(self.fc2(x2))

        # Concatenate in dim1 (feature dimension)
        x = torch.cat((x1, x2), 1)
        x = self.fc_out(x)
        return x

model = MyModel()
x1 = torch.randn(2, 3, 24, 24)
x2 = torch.randn(2, 1, 24, 24)
output = model(x1, x2)
</code></pre>
<p>Note that you could also concatenate/sum/average the outputs of the <code>features</code> modules.<br>
I’m not sure what would work best in your use case, so you might want to play around a bit. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Daniel_Joseph: <p>Thank you so much Ptrblck , much helpful <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>

raed1991: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="47858">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>self.fc1</code></p>
</blockquote>
</aside>
<p>guys I have similar issue if you could help me please.<br>
I have two different models. I trained the first model (AE). Then, I want to feed the output of the AE into the second model. while doing that, I freeze the parameters of AE. I concatenate them similar to the above example and has no issue. my question how to make the second model (traditional cnn) look at/focus more on the features (weights) coming from AE? what is efficient way to do it? because I want the first model to improve the accuracy of my second model.</p>

HJX-zhanS: <p>Hi, ptrblck</p>
<p>I try to use the structure you mentioned to build a model, but the model parameters did not change, and <code>grad_value</code> is None. Can you help me? Thanks!</p>
<p>Please see <a href="https://discuss.pytorch.org/t/how-to-assemble-two-models-into-one-big-model/157027">How to assemble two models into one big model - FAQ - PyTorch Forums</a></p>