Training linear combination of Pytorch models

victorc25: <p>Hello everyone! I’m trying to find a way to train a linear layer (vector) that will optimize the way a number of pytorch models are combined, but I’m not sure where to start.</p>
<p>The idea would be something like: having 4 independently trained models (same network, shapes and number of weights), a linear layer that has a weight that corresponds to each input model and what I would like to do is having the resulting combination evaluated with a training dataset and backpropagate the error to modify only the linear layer, so it can be optimized given a loss function.</p>
<p>It’s a bit similar to the question asked in <a href="https://discuss.pytorch.org/t/combining-trained-models-in-pytorch/28383" class="inline-onebox">Combining Trained Models in PyTorch</a>, but in my case I’m working with CNN networks, and instead would like to optimize how to combine the models instead of combining the results of the models.</p>
<p>Is there a way this is feasible?</p>

ptrblck: <aside class="quote no-group" data-username="victorc25" data-post="1" data-topic="75349">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/13edae/48.png" class="avatar"> victorc25:</div>
<blockquote>
<p>but in my case I’m working with CNN networks</p>
</blockquote>
</aside>
<p>The posted approach is independent of the model architecture and just combines the model outputs before feeding them to a final classifier.<br>
You could also manipulate the sumbodels to output the penultimate activation, if needed.</p>
<aside class="quote no-group" data-username="victorc25" data-post="1" data-topic="75349">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/13edae/48.png" class="avatar"> victorc25:</div>
<blockquote>
<p>and instead would like to optimize how to combine the models instead of combining the results of the models.</p>
</blockquote>
</aside>
<p>Could you explain, what the first approach would do in comparison the combining the results, please?</p>

victorc25: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="75349">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Could you explain, what the first approach would do in comparison the combining the results, please?</p>
</blockquote>
</aside>
<p>Yes, basically I want to combine the 4 models’ weights, to eventually end up with a single model, instead of combining the results, which would be an ensemble.</p>

ptrblck: <p>In that case you could get the <code>state_dicts</code> from each model, average all parameters (or use another reduction instead of the mean), and reload the <code>state_dict</code> to a single model.</p>
<p>However, I’m very skeptical if that approach will give you good results.<br>
Given that each model might have converged to another local minimum, I don’t think that e.g. the average of all parameters representing local minima will give you another minimum.</p>
<p>Let us know, how the experiment goes and if you were able to achieve a good performance using this approach.</p>

victorc25: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="75349">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>In that case you could get the <code>state_dicts</code> from each model, average all parameters (or use another reduction instead of the mean), and reload the <code>state_dict</code> to a single model.</p>
</blockquote>
</aside>
<p>I’ve done that, but now what I want to try to do is to optimize the linear combination of the models. I’ve been able to do something like that manually (very inefficient), but I wanted to see if there’s a way to do it using Pytorch. The struggle I have is finding a way to have the vector that represents the coefficient for each model in the combination inside the path of the backprop, so I can use Pytorch to tune those coefficients. I’m at a loss there, I don’t know if it’s even possible.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="75349">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Given that each model might have converged to another local minimum, I don’t think that e.g. the average of all parameters representing local minima will give you another minimum.</p>
</blockquote>
</aside>
<p>And you are absolutely correct. I think there’s a possibility for it to work, given my previous manual tests, but that’s why I want to keep testing <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

victorc25: <p>To illustrate, this would be something like the pseudo model I would like to create.</p>
<pre><code class="lang-auto">class Combinator(nn.Module):
    def __init__(self, modelA, modelB):
        super(Combinator, self).__init__()
        #Linear coefficient vector
        #[Wa, Wb]
        self.linweights = nn.Parameter(torch.ones(2))
        
        #Combined model
        #self.model = Wa*modelA + Wb*modelB
             
    def forward(self, x):
        x = self.model(x)
        return x
</code></pre>
<p>The problem I have is with creating that resulting model combining the others and if it makes sense to, for example, setting the models parameters to “requires_grad = False” and wanting only the coefficients I’ve named “linweights” to be trainable and changed by the backprop.</p>

victorc25: <p>Hello! Sorry to write again, but I haven’t been able to advance much, I was wondering if you may have an idea about what to do.</p>
<p>So far, the only way I’ve found to multiply what I call the “coefficients” to train is by creating a nn.Parameter for each model I want to combine (say, <em>self.linwA = nn.Parameter(abs(torch.randn(1)),requires_grad=True)</em> ) and multiply it by each of the parameters of the model, something like this:</p>
<pre><code class="lang-auto">def mulpar(self, coefficient, model):    
    for k, v in model.state_dict().items():
        model.state_dict()[k][:] = coefficient.expand_as(v)*model.state_dict()[k][:]
    return model
</code></pre>
<p>However, I think it doesn’t work, because the resulting model is no longer trainable (requires_grad = False) and I think that’s causing a “<em>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</em>” error later when I want to calculate the loss, there’s nothing to backpropagate.</p>
<p>Any pointers or ideas?</p>

aman_neelappa: <p>Hi Victor. Did you manage to get this to work? I am trying to do something similar. Thanks.</p>

victorc25: <p>Hello Aman! Yes, I was able to somewhat get it working using the suggestions by <a class="mention" href="/u/kfrank">@KFrank</a>  here: <a href="https://discuss.pytorch.org/t/multiply-a-model-by-trainable-scalar/76308" class="inline-onebox">Multiply a model by trainable scalar</a></p>
<p>It does work, I just didn’t automate the process to generate the mirror paths. Might or might not be useful in your case, but I hope at least it gives you an idea <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

aman_neelappa: <p>Thanks Victor. WIll try it out.</p>

ChrisLiu2: <p>I had the same question, and for my use in 1d CNN, I wrote a custom convolution layer that creates multiple weights and uses F.conv1d to calculate. This approach might be faster than saving and loading state_dict if performance is in your concern.</p>