Creating a model which weights are the sum of weights of 2 different neural networks

username: CasellaJr
cooked: <p>I am doing an experiment of transfer learning.<br>
I trained 2 CNNs that have exactly the same structure, one for MNIST and one for SVHN.<br>
I obtained the parameters (weights  and bias) of the 2 models.<br>
Now, I want to combine (sum, or other operations) these weights. A thing like this:</p>
<pre><code class="lang-auto">modelMNIST.parameters()
modelSVHN.parameters()

#now the new model
model3 = MyCNN(1)
model3.parameters = modelMNIST.parameters()+modelSVHN.parameters()
</code></pre>
<p>If I do in this way, I obtain this error:<br>
<code>SyntaxError: can't assign to function call</code></p>
<p>And in this way:</p>
<pre><code class="lang-auto">model3.block_1[0].weight = modelMNIST.block_1[0].weight + modelSVHN.block_1[0].weight
</code></pre>
<p>I get this error:</p>
<pre><code class="lang-auto">TypeError: cannot assign 'torch.cuda.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)
</code></pre>
<p>I have to use in some way load_state_dict?</p>

username: KFrank
cooked: <p>Hi Bruno!</p>
<p>I have both a conceptual comment and a couple of technical comments.</p>
<p>On the conceptual side:</p>
<aside class="quote no-group" data-username="CasellaJr" data-post="1" data-topic="119491" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/50afbb/48.png" class="avatar"> CasellaJr:</div>
<blockquote>
<p>Now, I want to combine (sum, or other operations) these weights.</p>
</blockquote>
</aside>
<p>I have concerns about combining weights like this from two<br>
independently-trained models.  (I have heard reports that this<br>
works or can be made to work, but I’m skeptical.)</p>
<p>The issue is that there is a lot of arbitrariness and redundancy<br>
in the weights of a neural-network model.  If I take two identical<br>
models, but give them different (but equivalent) initializations,<br>
and train them on the same training data (but probably batched<br>
up into different (but equivalent) random batches), there is no<br>
reason for “weight-17” in model A to have the same value as<br>
“weight-17” in model B.</p>
<p>Training model A takes it down some particular path along the<br>
“loss-surface” to some reasonable location in weight-configuration<br>
space.  But because of the redundancy and randomness, training<br>
model B takes it down some entirely different (but comparably<br>
worthwhile) path to some entirely different location (that is comparably<br>
good in making predictions).</p>
<p>Speaking somewhat figuratively, let’s say that, by happenstance,<br>
the weights on the “left-hand” side of model A learn to recognize<br>
straight edges, while weights on the “right-hand” side learn curved<br>
edges.  But now assume that, by happenstance, this is reversed in<br>
model B, so the left side learns curved edges and the right, straight<br>
edges.</p>
<p>So weight-17 in model A means straight, but weight-17 in model B<br>
means curved.  These two weights mean two different things, so it<br>
doesn’t make sense to add, or average, or otherwise combine then<br>
together.</p>
<p>On the technical side:</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">model3.parameters = modelMNIST.parameters()+modelSVHN.parameters()
</code></pre>
<p>If I do in this way, I obtain this error:<br>
<code>SyntaxError: can't assign to function call</code></p>
</blockquote>
</aside>
<p><code>.parameters</code> is a method (“member function”) of <code>Module</code>, while<br>
<code>.parameters()</code> is the result of invoking that method (calling that<br>
function).  You are trying to assign something to the function as the<br>
error message indicates.</p>
<p>But furthermore, the result of calling <code>.parameters()</code> is also not<br>
something you can assign to.  It’s a “generator” – something you<br>
can really just iterate over.</p>
<p>So you could simultaneously iterate over the result of calling<br>
<code>.parameters()</code> on your three models.</p>
<p>But …</p>
<aside class="quote no-group">
<blockquote>
<p>And in this way:</p>
<pre><code class="lang-auto">model3.block_1[0].weight = modelMNIST.block_1[0].weight + modelSVHN.block_1[0].weight
</code></pre>
<p>I get this error:</p>
<pre><code class="lang-auto">TypeError: cannot assign 'torch.cuda.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)
</code></pre>
</blockquote>
</aside>
<p>A <code>Parameter</code> is not a <code>Tensor</code> – it’s a wrapper for a <code>Tensor</code> that<br>
has some added functionality.  When you add two <code>Parameter</code>s<br>
together, you don’t get a new <code>Parameter</code> – you get a new <code>Tensor</code><br>
that is the sum of the two wrapped <code>Tensors</code>.  If you assign a <code>Tensor</code><br>
to a (reference to a) <code>Parameter</code>, you don’t get a (reference to a)<br>
<code>Parameter</code> – you get a (reference to a) <code>Tensor</code> (and a <code>Module</code><br>
is smart enough not to let you do this).</p>
<p>You would need to do something like this:</p>
<pre data-code-wrap="python"><code class="lang-python">l1 = torch.nn.Linear (2, 3)
l2 = torch.nn.Linear (2, 3)
l3 = torch.nn.Linear (2, 3)
with torch.no_grad():
    l1.weight.copy_ (l2.weight + l3.weight)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: CasellaJr
cooked: <p>Hi Frank!</p>
<blockquote>
<p>I have concerns about combining weights like this from two<br>
independently-trained models. (I have heard reports that this<br>
works or can be made to work, but I’m skeptical.)</p>
</blockquote>
<p>Can you send me these papers/reports, if you can find? Because I am doing this experiment for my thesis, and I just started with this.</p>
<blockquote>
<p>The issue is that there is a lot of arbitrariness and redundancy<br>
in the weights of a neural-network model. If I take two identical<br>
models, but give them different (but equivalent) initializations,<br>
and train them on the same training data (but probably batched<br>
up into different (but equivalent) random batches), there is no<br>
reason for “weight-17” in model A to have the same value as<br>
“weight-17” in model B.</p>
</blockquote>
<p>Yes, I know, but my supervisor assigned me this work, and we want to obtain these bad results. My supervisor has several ideas on how to try to improve later.</p>
<p>For the technical solution: I have modelMNIST, and mondelSVHN<br>
You mean, I have to do this?</p>
<pre><code class="lang-auto">with torch.no_grad():
    model3.weight.copy_ (modelMNIST.block_1[0].weight + modelSVHN.block_1[0].weight
</code></pre>
<p>Doing it for all the blocks (I have 4 blocks + 1 classifier) and for each layer of the blocks?</p>

username: CasellaJr
cooked: <p>Okay, I found the solution:</p>
<pre><code class="lang-auto">class VGG16SUM(nn.Module):
    
    def __init__(self, model1, model2, num_classes):
        super(VGG16SUM, self).__init__()

        # calculate same padding:
        # (w - k + 2*p)/s + 1 = o
        # =&gt; p = (s(o-1) - w + k)/2

        self.block_1 = nn.Sequential(
            nn.Conv2d(in_channels=1,
                      out_channels=64,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      # (1(32-1)- 32 + 3)/2 = 1
                      padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.Conv2d(in_channels=64,
                      out_channels=64,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=(2, 2),
                         stride=(2, 2))
        )

        self.block_2 = nn.Sequential(
            nn.Conv2d(in_channels=64,
                      out_channels=128,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(),
            nn.Conv2d(in_channels=128,
                      out_channels=128,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=(2, 2),
                         stride=(2, 2))
        )
        
        self.block_3 = nn.Sequential(
            nn.Conv2d(in_channels=128,
                      out_channels=256,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(),
            nn.Conv2d(in_channels=256,
                      out_channels=256,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(),
            nn.Conv2d(in_channels=256,
                      out_channels=256,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=(2, 2),
                         stride=(2, 2))
        )

        self.block_4 = nn.Sequential(
            nn.Conv2d(in_channels=256,
                      out_channels=512,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(),
            nn.Conv2d(in_channels=512,
                      out_channels=512,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(),
            nn.Conv2d(in_channels=512,
                      out_channels=512,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=(2, 2),
                         stride=(2, 2))
        ) 


        self.classifier = nn.Sequential(
            nn.Linear(2048, 4096),
            nn.ReLU(True),
            nn.Dropout(p=0.25),
            nn.Linear(4096, 4096),
            nn.ReLU(True),
            nn.Dropout(p=0.25),
            nn.Linear(4096, num_classes),
        )

        for p_out, p_in1, p_in2 in zip(self.parameters(), model1.parameters(), model2.parameters()):
            p_out.data = nn.Parameter(p_in1 +p_in2);

    def forward(self, x):

        x = self.block_1(x)
        x = self.block_2(x)
        x = self.block_3(x)
        x = self.block_4(x)
        # x = self.avgpool(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x
        #logits = self.classifier(x)
        #probas = F.softmax(logits, dim=1)
        # probas = nn.Softmax(logits)
        #return probas
        # return logits
</code></pre>

username: KFrank
cooked: <p>Hi Bruno!</p>
<aside class="quote no-group quote-modified" data-username="CasellaJr" data-post="3" data-topic="119491" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/50afbb/48.png" class="avatar"> CasellaJr:</div>
<blockquote>
<blockquote>
<p>(I have heard reports that this<br>
works or can be made to work, but I’m skeptical.)</p>
</blockquote>
<p>Can you send me these papers/reports, if you can find? Because I am doing this experiment for my thesis, and I just started with this.</p>
</blockquote>
</aside>
<p>I don’t have any specific references.  I have a vague recollection that<br>
I expressed similar skepticism in another thread on this forum and the<br>
poster ended up replying that he got his scheme working.</p>
<aside class="quote no-group">
<blockquote>
<p>Yes, I know, but my supervisor assigned me this work, and we want to obtain these bad results. My supervisor has several ideas on how to try to improve later.</p>
</blockquote>
</aside>
<p>Some thoughts:</p>
<p>First, a disclaimer:  I haven’t tried any relevant experiments myself,<br>
and I don’t know of any literature on this issue.</p>
<p>But let’s say you want to try this and make it work – that is, you want<br>
to train two models with identical architectures independently on two<br>
similar, but distinct problems, and then average the weights together<br>
somehow.</p>
<p>Why might doing something like this have value?  My intuition runs as<br>
follows:  The upstream weights of a network tend to learn “lower-level”<br>
features that are likely to be common to both problems.  For example,<br>
upstream convolutions might learn to detect edges.  In contrast, the<br>
downstream weights learn “higher-level” features more tuned to the<br>
specific problem being trained on.  For example, even though similar,<br>
a downstream fully-connected layer trained on zipcodes might learn<br>
handwritten digits, but one trained on house numbers, might do better<br>
with block-letter-style printed or engraved digits.</p>
<p>By “averaging” together the upstream weights, you might do a better<br>
(and perhaps more generalizeable) job recognizing the more generic,<br>
lower-level features.  (It would seem harder to gain any advantage with<br>
the more-problem-specific features.)</p>
<p>Let’s say this is all true and can be made to work somehow.  The<br>
concern I expressed above still remains:  After independent training,<br>
because of the redundancies in the network weights, there’s no reason<br>
that weight-17 in model A plays the same role or has the same meaning<br>
as weight-17 in model B.  So it doesn’t make sense to combine them<br>
together, using an average or otherwise.</p>
<p>However, what if, by (extremely unlikely) happenstance, model A and<br>
model B, while being trained, did end up following similar paths, and<br>
ended up at similar locations in weight-configuration space.  Now it<br>
could make sense to combine the two weight-17s together, as they<br>
could now have similar meaning.</p>
<p>One approach could be to have the two models guide one another<br>
along similar paths while training.</p>
<p>The idea would be that you could start the two models at the same<br>
(random) location – i.e., use the same random initialization for weights<br>
of both models – and then add a loss term that nudges the two models<br>
to prefer similar paths.</p>
<p>Concretely, you could use <code>((weights_A - weights_B)**2).sum()</code><br>
as an added loss term for training both models, where, when taking<br>
the optimization step for model A, <code>weights_B</code> are viewed as fixed<br>
(non-trainable) parameters, and vice versa.</p>
<p>If you accept the intuition that the upstream weights are more likely<br>
than downstream weights to play similar roles in the two models, you<br>
might choose to weight the upstream weights more heavily in the<br>
proposed added loss term.</p>
<p>Taking this idea to its logical extreme, you could weight the upstream<br>
weights so heavily that the upstream weights of the two models are<br>
forced to be identical, and give zero weight to the downstream weights<br>
so that they train completely independently.</p>
<p>Of course doing this would just be doing what people already do when<br>
they train two-headed networks where there is a common upstream<br>
part to the network, but two downstream heads that make two sets of<br>
predictions for two different problems that feed to different loss functions<br>
that are added together into a single loss that is then backpropagated.</p>
<p>From this perspective using such a (potentially-weighted) loss term to<br>
guide the two models along similar paths could be viewed as a way<br>
to interpolate between training the two models fully independently,<br>
combining the weights together after the fact. and training a single<br>
model with two heads.</p>
<p>Best.</p>
<p>K. Frank</p>

username: KFrank
cooked: <p>Hi Bruno!</p>
<aside class="quote no-group" data-username="CasellaJr" data-post="4" data-topic="119491" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/50afbb/48.png" class="avatar"> CasellaJr:</div>
<blockquote>
<pre><code class="lang-auto">        for p_out, p_in1, p_in2 in zip(self.parameters(), model1.parameters(), model2.parameters()):
            p_out.data = nn.Parameter(p_in1 +p_in2);
</code></pre>
</blockquote>
</aside>
<p>As an answer to your original technical question, this looks fine to me.</p>
<p>Best.</p>
<p>K. Frank.</p>

username: CVLab_005
cooked: <pre><code class="lang-auto">
</code></pre>
<p>The above solution does not work for me. I mean I am doing this<br>
with torch.no_grad():<br>
for p_out, p_in1, p_in2 in zip(teacher.parameters(), student.parameters(), teacher.parameters()):<br>
p_out.data = nn.Parameter(p_in1 +p_in2);<br>
But when I am using teacher for my inference this is giving zero outputs. Like this,<br>
tensor([], device=‘cuda:0’, size=(0, 4))<br>
tensor([], device=‘cuda:0’, size=(0, 4))<br>
tensor([], device=‘cuda:0’, size=(0, 4))<br>
tensor([], device=‘cuda:0’, size=(0, 4))<br>
One thing I want to add these teacher and student are same model. The model is “torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)”</p>