Multiply a model by trainable scalar

victorc25: <p>Hello,</p>
<p>Is it possible to multiply a trained model by a single scalar that is trainable and backpropagate the loss to that scalar?</p>
<p>I want to do this for two networks, so the idea is to train those two scalars to find how to ideally combine the two models’ weights (not the outputs).</p>
<p>Thanks in advance for any guidance.</p>

KFrank: <p>Hello Victor!</p>
<aside class="quote no-group" data-username="victorc25" data-post="1" data-topic="76308" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/13edae/48.png" class="avatar"> victorc25:</div>
<blockquote>
<p>Is it possible to multiply a trained model by a single scalar that is trainable and backpropagate the loss to that scalar?</p>
</blockquote>
</aside>
<p>Yes.  In general you can train additional parameters (e.g., things that<br>
aren’t weights in <code>Linear</code>s) by adding them as <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Parameter" rel="noopener nofollow ugc">Parameter</a>s to your<br>
model (which should be a <code>torch.nn.Module</code>) or to your <code>Optimizer</code>.<br>
Pytorch will do the rest.</p>
<aside class="quote no-group">
<blockquote>
<p>I want to do this for two networks, so the idea is to train those two scalars to find how to ideally combine the two models’ weights (not the outputs).</p>
</blockquote>
</aside>
<p>I’m not sure exactly what you have in mind here.  But here is one idea<br>
that might be relevant to your use case:</p>
<p>You have model A and model B already trained.  Freeze their layers<br>
(<code>.requires_grad = False</code>).  Create an <code>Optimizer</code> that contains your<br>
weight parameter as a single trainable parameter:</p>
<pre data-code-wrap="python"><code class="lang-python">weight_param = torch.FloatTensor ([1.234])
optim = torch.optim.SGD ([weight_param])
</code></pre>
<p>(where <code>1.234</code> is your initial value of <code>weight_param</code>).</p>
<p>Then do something like:</p>
<pre data-code-wrap="python"><code class="lang-python">xA = layer1A (input)
xB = layer1B (input)
x = weight_param * xA + (1.0 - weight_param) * xB
x = activation1 (x)
xA = layer1A (x)
xB = layer1B (x)
x = weight_param * xA + (1.0 - weight_param) * xB
x = activation2 (x)
...
loss = criterion (x, target)
optim.zero_grad()
loss.backward()
optim.step()
</code></pre>
<p>(This code can be rearranged in various ways, but this is the basic<br>
idea.)</p>
<p>Is this the line along which you were thinking?</p>
<p>Best.</p>
<p>K. Frank</p>

victorc25: <p>Hello Frank! Thank you very much for your reply!</p>
<p>So, the overall idea is that I want to combine two models that were fine-tuned independently into a single one by interpolating their weights, but I would like to do this using the PyTorch framework (and, most importantly, the loss functions and autograd).</p>
<p>Originally, I was thinking about a single “coefficient” for each model that could just multiply all of the model’s trained weight, but I think I understand your code and it makes sense to me!</p>
<p>If I understand correctly, it would be adding an activation after every layer of the original models that combines the corresponding layer of the two original models, is that right? What could be a good activation to test with in this case?</p>
<p>This sounds great!</p>

KFrank: <p>Hi Victor!</p>
<aside class="quote no-group" data-username="victorc25" data-post="4" data-topic="76308" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/13edae/48.png" class="avatar"> victorc25:</div>
<blockquote>
<p>it would be adding an activation after every layer of the original models that combines the corresponding layer of the two original models, is that right? What could be a good activation to test with in this case?</p>
</blockquote>
</aside>
<p>I was imagining that you had two models with the same architecture,<br>
that is, that all of the layers and activations match up.  The only<br>
difference is that they have been trained (or fine tuned) differently,<br>
so that the trained weights differ.</p>
<p>In this case, model A’s <code>activation1</code> and model B’s <code>activation1</code><br>
are the same, so I was thinking that you would simply use<br>
<code>activation1</code> after interpolating the results of <code>layer1A</code> and <code>layer1B</code>.<br>
(Note, the result of interpolating these two results is the same as<br>
what you would get by interpolating the <em>linear</em> layers <code>layer1A</code> and<br>
<code>layer1B</code>, and then applying the interpolated layer to the upstream<br>
input.)</p>
<p>(As an aside, I’m not convinced your scheme will work.  If model A<br>
and model B are trained independently to do the same thing, the<br>
values of the internal “hidden” neurons may still be very different,<br>
and the layer weights may be very different, so averaging them might<br>
not make sense.  If one model was first trained, and then fine tuned<br>
separately to get models A and B, the weights in A and B might line<br>
up sensibly, but I could imagine that averaging the models together<br>
could simply serve to average away the results of the fine tunings.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

victorc25: <p>Hello Frank!</p>
<aside class="quote no-group" data-username="KFrank" data-post="5" data-topic="76308">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>I was imagining that you had two models with the same architecture,<br>
that is, that all of the layers and activations match up. The only<br>
difference is that they have been trained (or fine tuned) differently,<br>
so that the trained weights differ.</p>
</blockquote>
</aside>
<p>Yes, that’s exactly the case!</p>
<aside class="quote no-group" data-username="KFrank" data-post="5" data-topic="76308">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>In this case, model A’s <code>activation1</code> and model B’s <code>activation1</code><br>
are the same, so I was thinking that you would simply use<br>
<code>activation1</code> after interpolating the results of <code>layer1A</code> and <code>layer1B</code> .<br>
(Note, the result of interpolating these two results is the same as<br>
what you would get by interpolating the <em>linear</em> layers <code>layer1A</code> and<br>
<code>layer1B</code> , and then applying the interpolated layer to the upstream<br>
input.)</p>
</blockquote>
</aside>
<p>I understand now! Thanks for explaining it to me, it makes sense now. I’ll try that today to see how it goes, but it really sounds like what I wanted to test.</p>
<aside class="quote no-group" data-username="KFrank" data-post="5" data-topic="76308">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>(As an aside, I’m not convinced your scheme will work. If model A<br>
and model B are trained independently to do the same thing, the<br>
values of the internal “hidden” neurons may still be very different,<br>
and the layer weights may be very different, so averaging them might<br>
not make sense. If one model was first trained, and then fine tuned<br>
separately to get models A and B, the weights in A and B might line<br>
up sensibly, but I could imagine that averaging the models together<br>
could simply serve to average away the results of the fine tunings.)</p>
</blockquote>
</aside>
<p>And you are exactly correct. The models line up (the filters are correlated) and I’ve tested doing this combination manually and it works, but I wanted to test using Pytorch’s automatic differentiation to optimize the search automatically and see how it behaves using some custom loss functions as well, so I think your suggestion will work perfectly!</p>
<aside class="quote no-group" data-username="KFrank" data-post="5" data-topic="76308">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Good luck.</p>
</blockquote>
</aside>
<p>Thanks again!!</p>

victorc25: <p>“I’m making a note here: huge success.”</p>
<p>I had to write 1875 lines of code just to disassemble and reassemble everything as I wanted and test it step by step, but it worked <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>
<p>Thanks, again!</p>