Optimizing based on another model's output

username: TrevP_1
cooked: <p>Hi sorry I’m new to Pytorch: if I have Model1 producing some output, which is fed into Model2 (which is pre-trained), is there a simple way to optimize Model1’s weights based on Model2’s outputs? I could of course just backprop through Model2 into Model1, but I want to assume Model2 is not accessible to Model1 other than it’s outputs.</p>
<p>If I do:</p>
<pre><code>for p in Model2.parameters():
    p.requires_grad = False
</code></pre>
<p>Does that have the desired effect? - Model2’s weights and gradient are not accessible to Model1 during backprop</p>

username: albanD
cooked: <p>Hi,</p>
<p>If I understand properly, you have this:</p>
<pre><code class="lang-auto">input = # ...
out1 = Model1(input)
out2 = Model2(out1)
loss = LossFunc(out2)
</code></pre>
<p>If you want to optimize the parameters of <code>Model1</code>, you can just use <code>loss.backward()</code> and create your optimizer to only update the first model with <code>optimizer = torch.optim.SGD(Model1.parameters(), other_args)</code>. That way, <code>Model1</code> will be updated but not <code>Model2</code>.</p>
<p>Note that whatever you’re gonna do, to be able to get gradients for <code>Model1</code>, you will have to backpropagate through <code>Model2</code> (this is how backprop works).</p>

username: TrevP_1
cooked: <p>Thanks very much! Are there any gradient estimation techniques I could use then, if assume I don’t have access to Model2’s internals?</p>

username: albanD
cooked: <p>If <code>Model2</code> is completely unknown, then you have to start using “Black Box Optimization”. Which is a branch of optimization that studies the optimization of functions for which you have absolutely no information (you can just evaluate them). Unfortunately the quality of these approaches is worst (as expected) compared to problems where you have informations about the internal of your function.</p>

username: TrevP_1
cooked: <p>Fantastic thank you, last question - is there an easy method to pass this gradient approximation to Model1? i.e. combine the approximated gradient of Model2 and then continue with real backprop for Model1?</p>

username: albanD
cooked: <p>These methods are not really implemented in pytorch so there is no builtin way.<br>
If you have a blackbox optimization method that give you <code>d(loss) / d(out1)</code>: how you loss varies wrt the input of <code>Model2</code>, then you can simply do:</p>
<pre><code class="lang-auto">grad = BBOpt(out1, out2, loss, ...)
optimizer.zero_grad()
out1.backward(grad)
optimizer.step()
</code></pre>
<p>Basically, specify directly what are the gradients that correspond to <code>out1</code>.</p>

username: zakaria_laskar
cooked: <p>Hi, just an extended question to you reply. If one wants to update both the models, one needs to create 2 optimizers, one for each model?. And thereby use opt1.step() and opt2.step() after loss.backward()?</p>

username: albanD
cooked: <p>You can do that or create an optimizer that works with both models:<br>
<code>full_opt = torch.optim.SGD(list(Model1.parameters())+list(Model2.parameters()), other_args)</code> and then you can just call <code>full_opt.step()</code> to update both models.</p>

username: jeffreycordero
cooked: <p>Hello!  So mostly related, I am attempting to setup mutual learning with pytorch and am unclear about how.  I have two models, whose loss values are related (affect the loss value of the other model, but not the same) and each model needs to be back-propagated with its corresponding loss value.</p>
<p>Is there a way to backprop each loss value to its corresponding model?</p>

username: Deeply
cooked: <aside class="quote no-group" data-username="albanD" data-post="8" data-topic="6935">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>list(Model1.parameters())+list(Model2.parameters())</p>
</blockquote>
</aside>
<p>You really saved my day!<br>
Just one question, would the + sign between the two models’ parameters work, or I’ll have to use <code>itertools.chain</code>, as follows?</p>
<p><code>full_opt = torch.optim.Adam( itertools.chain( model1.parameters(), model2.parameters(), model3.parameters() ), other_args )</code></p>
<p>Thanks</p>

username: albanD
cooked: <p>You can chain the iterators to be clean or be lazy and just create lists that you can then  add up <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
Both will work !</p>

username: YoonYeong
cooked: <p>Hi, I have a related question regarding this problem. If I want to update the parameters of model1 while fixing model2, then should I have to set requires_grad=False for params in model2.parameters()? Actually, I guess that setting optimizer as torch.optim.SGD(model1.parameters()) will not change model2.parameters. I’m so confused <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

username: albanD
cooked: <p>Both will work. But there are subtle differences.<br>
The short answer is do both: set requires_grad to False and do <strong>not</strong> give these paramaters to the optimizer.</p>
<p>If you only set the requires_grad field but give both to the optimizer, the weight might still be updated even though the gradient is 0 due to l2 regularization or momentum.<br>
If you only not give it to the optimizer, then you might be doing extra computation to compute the gradients of some Tensors for which you don’t need them.</p>

username: Qu_Yukun
cooked: <p>Hi, I have a related question regarding this problem. Could you help me?<br>
I have two models, and I hope I can use the sum output of the two models to update the two models.</p>
<p>The code is as follows:</p>
<blockquote>
<p>spat_out = spat_model(spat_data)<br>
temp_out = temp_model(temp_data)</p>
<p>spat_loss = spat_criterion(spat_out, labels)<br>
temp_loss = temp_criterion(temp_out, labels)</p>
<p>loss = spat_loss + temp_loss<br>
loss.backward()</p>
<p>spat_optimizer.step()<br>
temp_optimizer.step()</p>
</blockquote>
<p>loss.backward() will update wich models’ gradient? The last one or both of them?</p>

username: albanD
cooked: <p>If you don’t use <code>.detach()</code> or <code>torch.no_grad()</code>, it will backprop on everything you used. So here both.</p>

username: arinaruck
cooked: <p>Hi!<br>
I am trying to train two models in stages:</p>
<pre><code class="lang-auto">model1 = Model1()
model2 = Model2()
optimizer1 = Adam(model1.parameters())
optimizer2 = Adam(model2.parameters())
for epoch in range(epochs):
     for stage in range(stages):
         model2.eval()
         model1.train()
         for step in range(steps1):
              ....
              pred = model1()
              ....
              loss1.backward() # depends on model1 only
              optimizer1.step()
              optimizer1.zero_grad()
        model2.train()
        for step in range(steps2):
              ....
              with torch.no_grad():
                  encoded = model1.encoder(inp)
              projection = model1.proj(encoded)
              pred = model2(projection)
              ....
              loss2.backward()
              optimizer1.step()
              optimizer2.step()
              optimizer1.zero_grad()
              optimizer2.zero_grad()
</code></pre>
<p>And I am getting an CUDA error: an illegal memory access was encountered (embedding_dense_backward_cuda at /pytorch/aten/src/ATen/native/cuda/Embedding.cu:267)<br>
(model1 is a BERT)<br>
Is this an incorrect way to do this? How should I approach updating the model’s weights in this setting?</p>

username: ptrblck
cooked: <p>If you are using PyTorch <code>1.5</code>, update to <code>1.6</code> or the nightly binaries, as <code>assert</code> statements were not properly working in <code>1.5</code>.<br>
Based on the illegal memory access in the embedding layer, I guess you are passing an input containing invalid indices to this layer. Instead of this memory violation you should get a proper indexing error.</p>

username: arinaruck
cooked: <p>Thanks a lot! That actually helped me trace the error<br>
Indeed, one of asserts in the loss function started falling as soon as I updated the torch version</p>

username: Hongjun_Choi
cooked: <aside class="quote no-group" data-username="Qu_Yukun" data-post="14" data-topic="6935">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/qu_yukun/48/25048_2.png" class="avatar"> Qu_Yukun:</div>
<blockquote>
<p>e last one or both of</p>
</blockquote>
</aside>
<p>As an additional question for the above one, let’s suppose that we have this.</p>
<p>spat_out = spat_model(spat_data)<br>
temp_out = temp_model(spat_out.detach())</p>
<p>spat_loss = spat_criterion(spat_out, labels)<br>
temp_loss = temp_criterion(temp_out, labels)</p>
<p>loss = spat_loss + temp_loss<br>
loss.backward()</p>
<p>spat_optimizer.step()<br>
temp_optimizer.step()</p>
<p>Basically, the output of the first model becomes the input for the second model. If I create input for the second model by using ‘spat_out.detach()’ to treat it being an independent input, in this case this will backprop on the first model and second model independently? i.e the second model’s gradient is not flowing into the first model.?</p>