Update overlapping parameters using different losses

username: Torres
cooked: <p>Hi,</p>
<p>I have a question on how to update overlapping parameters using different losses. For example,</p>
<pre><code class="lang-auto">hidden = encoder(imgs)
reconstructed = decoder(hidden)
prediction = classifier(hidden)

optimizer1 = Adam(encoder.parameters())
optimizer2 = Adam(decoder.parameters())
optimizer3 = Adam(classifer.parameters())

loss1 = Loss1(imgs, reconstructed)
loss2 = Loss2(prediction, labels)
</code></pre>
<p>In this case, I’d like to</p>
<ol>
<li>minimise loss1 and only update the parameters of encoder and decoder.</li>
<li>minimise loss2 and only update the parameters of encoder and classifer.</li>
<li>maximise loss2 and only update the parameters of encoder.</li>
</ol>
<p>I tried to call backward() for each loss separately, but I’ve got the following error:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 32, 7, 7]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
<p>I’m not sure if it’s the correct way. Or do I need to sum the loss and call backward() only once?</p>
<p>Thanks in advance for any help!</p>

username: ecdrid
cooked: <p>I think, the shouldn’t share parameters (atleast in my experience we separate the parameters heads accordingly to our loss functions)</p>
<p>And usually we just do <code>(loss_1 + loss_2 + .... + loss_n).backward()</code> (or mean/weighted approach etc)</p>

username: Torres
cooked: <p>Thanks! If doing backward() once, is there a way to update parameters according to each sub loss?</p>

username: ecdrid
cooked: <p>Ahh, i am not sure but i think my initial comment might be a bit misleading. Let’s see via an example.</p>
<p>Let’ss ay we are working on Image Classification / Bounding Box (locating where the object of interest lies within an image) + Generating the caption as well at the same time. Now these tow things will share a common backbone pretty much and then they will diverge into two separate “heads”. And naturally we will have two separate losses as well for the respective heads.</p>
<p>So, we have <code>loss_from_classification_head</code> and <code>loss_from_captioning_head</code> computed with us and then we can compute the total loss directly by <code>loss_from_classification_head</code> + <code>loss_from_captioning_head</code> and only calling backward once. PyTorch will take care of stuffs for you pretty much.</p>
<p>In your initial example, you are creating multiple optimizers, so do you plan to sue different learning_rate for them? if not, you can pretty much just do <code>main_model</code>.parameters()` and pass it to an optimizer and have that model output data as per the heads etc that you want to compute the loss for.</p>
<pre data-code-wrap="python"><code class="lang-python">loss1 = Loss1(imgs, reconstructed)
loss2 = Loss2(prediction, labels)

total_loss = loss1 + loss2
total_loss.backward() # that's it!

</code></pre>
<blockquote>
<p>Thanks! If doing backward() once, is there a way to update parameters according to each sub loss?</p>
</blockquote>
<p>If by this, Do you mean that you want to use separate learning rate for each separate sub-model components that you have defined above? (e.g. encoder’s LR is 1e-3, decoder’s LR is 1e-4 and classifier’s LR is 1e-2?)</p>
<p>If that’s what you want to do, then just pass the LR in a single optimizer by separating the model parameters.</p>
<p>e.g.</p>
<pre data-code-wrap="python"><code class="lang-python">
# lrs is a list of 2 elements basically.
opt = optim.Adam([{'params': model.body.parameters(), 'lr':lrs[0]},
                {'params': model.head.parameters(), 'lr': lrs[1]}])
</code></pre>
<p>Please let me know if this helps!</p>

username: Torres
cooked: <p>Thanks! It helps a lot!</p>