How to add nodes to pretrained model input?

KyleOng: <p>Good day,</p>
<p>Question as in the title, how can I add new input nodes to a pretrained model?</p>
<p>An example as you can see in the image below. Let’s say I have trained <code>ModelA</code> and I would like to add new input nodes to pretrained <code>ModelA</code> as in <code>ModelB</code>.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/4/54d18c7c7f5b74e95305179af1b57d288986b5bc.png" data-download-href="https://discuss.pytorch.org/uploads/default/54d18c7c7f5b74e95305179af1b57d288986b5bc" title="ask"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/4/54d18c7c7f5b74e95305179af1b57d288986b5bc.png" alt="ask" data-base62-sha1="c6l3WQKX8T7F6KENSeCYCu8Igzy" width="690" height="424" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ask</span><span class="informations">701×431 36.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Example code.</p>
<pre><code class="lang-auto">import torch

class ModelA(torch.nn.Module):
    def __init__(self):
        self.fc1 = torch.nn.Linear(4,3)
        self.fc2 = torch.nn.Linear(3,1)

    def forward(self, x):
         return self.fc2(self.fc1(x))
 
class ModelB(torch.nn.Module):
     pass
     # how do I contruct ModelB?
</code></pre>
<p>Thanks in advance.</p>

ariG23498: <p>I guess there is a better way to do it and would eventually be updated in the thread, but as far as I think adding a Linear layer in front of the input node of the pre-trained model should do the trick.</p>
<pre><code class="lang-python">class ModelB(torch.nn.Module):
    def __init__(self):
        self.fc0 = torch.nn.Linear(new_shape,4)
        self.pretrained = ModelA()

    def forward(self, x):
         return self.pretrained(self.fc0(x))
</code></pre>

KyleOng: <p>Thanks for the reply <a class="mention" href="/u/arig23498">@ariG23498</a></p>
<p>But wouldn’t that be adding a new layer, instead of adding new nodes to the pretrained input layer? Maybe can you explain more on it?</p>

ariG23498: <p>I haven’t seen a problem where I had to change the input node of a pretrained model. And come to think of it, if we apply a <code>Linear</code> infront of the pretrained model, we do not affect the model architecture, as we are eventually learning a linear equation.</p>

KyleOng: <p>I see. After googling for awhile, I thought the solution might be something like this<br>
(<a href="https://discuss.pytorch.org/t/how-to-add-new-nodes-at-the-last-layer-fully-connected-layer/40164/4" class="inline-onebox">How to add new nodes at the last layer (fully connected layer)? - #4 by MariosOreo</a>),<br>
where we pad the old model input weights to the shape of the new model input weights with random values and then <code>load_state_dict</code>.</p>
<p>or</p>
<p>maybe replacing the first linear layer of the new model with new input shape, and then assign its weights with pretrained weights like so (<a href="https://discuss.pytorch.org/t/how-to-transfer-the-pretrained-weights-for-a-standard-resnet50-to-a-4-channel/52252/2" class="inline-onebox">How to transfer the pretrained weights for a standard ResNet50 to a 4-channel - #2 by ptrblck</a>)</p>
<p>Can you have a look? and let me know your thoughts. Thank you very much.</p>

ariG23498: <p>I think the second option should be the best.</p>