Using a combined loss to update two different models

username: chantk
cooked: <p>Hi all, I’m trying to accomplish an event detection task with 2 models where model_a will produce the event tag and model_b will produce the event localization (aka label at each time frame). Basically, I’m trying to do update the two models according to the combined loss of these two models.</p>
<p>I tried the following code which run successfully but I am not entirely sure about the loss.backward() part:</p>
<blockquote>
<p>model_a = tkmodel_a()<br>
model_a = tkmodel_b()</p>
</blockquote>
<blockquote>
<p>criterion = nn.BCELoss() <span class="hashtag-raw">#binary</span> cross entropy</p>
</blockquote>
<blockquote>
<p>optimizer_a = optim.Adam(model_a.parameters(), lr=0.001, betas=(0.9, 0.999),<br>
eps=1e-08, weight_decay=0., amsgrad=True)</p>
</blockquote>
<blockquote>
<p>optimizer_b = optim.Adam(model_b.parameters(), lr=0.001, betas=(0.9, 0.999),<br>
eps=1e-08, weight_decay=0., amsgrad=True)</p>
</blockquote>
<blockquote>
<p>if torch.cuda.is_available():<br>
model_a.cuda()<br>
model_b.cuda()<br>
“Function to do prediction and etc”<br>
loss_a = criterion(predicted_a, event_label)<br>
loss_b = criterion(predicted_b, localization_label)</p>
</blockquote>
<blockquote>
<p>combined_loss_a = loss_a + loss_b<br>
combined_loss_b = loss_a + loss_b</p>
</blockquote>
<blockquote>
<p>optimizer_a.zero_grad(retain_graph=True)<br>
combined_loss_a.backward()<br>
optimizer_a.step()</p>
</blockquote>
<blockquote>
<p>optimizer_b.zero_grad()<br>
combined_loss_b.backward()<br>
optimizer_b.step()</p>
</blockquote>
<p>I have read on several posts that there is no need to do it this way and I can actually use</p>
<blockquote>
<p>combined_loss = loss_a + loss_b<br>
combined_loss.backward()</p>
</blockquote>
<p>and based on the discussion at <a href="https://discuss.pytorch.org/t/optimizing-based-on-another-models-output/6935" class="inline-onebox">Optimizing based on another model's output</a> if i do the following</p>
<blockquote>
<p>optimizer_strong.zero_grad()<br>
optimizer_weak.zero_grad()<br>
combined_strong_loss.backward()<br>
optimizer_strong.step()<br>
optimizer_weak.step()</p>
</blockquote>
<p>I would be updating both model_a and model_b with the combined_loss by using backward only once. But is there anyone who can confirm this?</p>

username: ptrblck
cooked: <p>Both approaches will yield the same result, but the method using two losses is wasteful, since you need to call the backward method twice.<br>
Here is a small code snippet to demonstrate the behavior:</p>
<pre><code class="lang-python"># Setup
torch.manual_seed(2809)

modelA = nn.Linear(1, 1)
modelB = nn.Linear(1, 1)

criterion = nn.MSELoss()

xA = torch.randn(1, 1)
targetA = torch.randn(1, 1)
xB = torch.randn(1, 1)
targetB = torch.randn(1, 1)

# Standard approach
outA = modelA(xA)
outB = modelB(xB)

lossA = criterion(outA, targetA)
lossB = criterion(outB, targetB)

loss = lossA + lossB
loss.backward()

print(modelA.weight.grad)
print(modelB.weight.grad)

# other approach
modelA.zero_grad()
modelB.zero_grad()

outA = modelA(xA)
outB = modelB(xB)

lossA = criterion(outA, targetA)
lossB = criterion(outB, targetB)

clossA = lossA + lossB
clossB = lossA + lossB

clossA.backward(retain_graph=True)
print(modelA.weight.grad)

modelB.zero_grad()
clossB.backward()
print(modelB.weight.grad)
</code></pre>

username: chantk
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="73925">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Both approaches will yield the same result, b</p>
</blockquote>
</aside>
<p>Okay thanks a lot for your reply.</p>

username: Mr.Bank
cooked: <p>In the beginning of the training block, should I write?</p>
<pre><code class="lang-auto">modelA.train() 
modelB.train()
</code></pre>

username: ptrblck
cooked: <p>Yes, that would be necessary, if you’ve previously called <code>.eval()</code>, e.g. in a validation loop.<br>
However, I would add these lines nevertheless to avoid mistakes just in case some other code part calls <code>model.eval()</code>.</p>

username: Evan_Hsu
cooked: <p>If I only want to update modelA by modelB’s lossB?<br>
How to do this?</p>

username: ptrblck
cooked: <p>Assuming modelB’s loss was calculated without the usage of modelA’s parameters, you won’t be able to directly update modelB using this loss and could instead e.g. copy the gradients from modelB to modelA manually (and update modelA afterwards).<br>
However, you would have to make sure this approach actually makes sense.</p>

username: asura
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, I have model A (say, MLP), and model B (say, CNN).<br>
Now, I want to update model A using model B’s loss. Model A was not used for computing anything in model B. How can it be done in this case? Because now I don’t have the same architecture, so can’t copy the gradients from model B to model A.</p>

username: ptrblck
cooked: <p>I don’t know how this use case could work, since (as you’ve already explained) you won’t be able to copy the gradients from A to B.</p>

username: asura
cooked: <p>Is there no any way around this?<br>
Basically, my usecase is like this:<br>
I have model 1 which predicts which layers to freeze in model 2.<br>
Then using the loss of model 2, I want to update model 1.</p>

username: ptrblck
cooked: <p>Maybe, but I don’t know as I don’t see how this use case could work.<br>
E.g.:</p>
<pre><code class="lang-python">modelA = resnet50()
modelB = nn.Linear(1, 1)
</code></pre>
<p>Both models have a different number of layers and of course overall structure, so copying the gradient from <code>A</code> to <code>B</code> won’t work.<br>
Also, I claim it’s even impossible for <code>modelB</code> to achieve the loss created by <code>modelB</code> on any dataset (say <code>ImageNet</code>).<br>
You can’t directly use the "loss of <code>modelB</code>" to update <code>modelA</code> since <code>modelA</code> was never used and would thus have to come up with your custom mapping strategy e.g. how the gradients should be copied.<br>
Maybe there are other valid approaches which I’m unaware of.</p>

username: asura
cooked: <p>Thanks for the quick reply! I will think on this, maybe I will have to change the approach if this is the case.</p>

username: ptrblck
cooked: <p>Let’s also wait for others to chime in, as they might have some ideas or might know similar use cases which you could reuse.</p>

username: Anwarvic
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , sorry but I have to ask a slightly different question.</p>
<p>Consider this use-case; I have an encoder-decoder architecture where there are two loss functions; one for the whole model while the other is for just the decoder. And I created two optimizers; one responsible for updating the model’s parameters while the other responsible for just the decoder’s.</p>
<p>Now, when I come to this part of the code:</p>
<pre><code class="lang-auto">decoder_criterion.backward()
criterion.backward()
</code></pre>
<p>It throws the following error:</p>
<pre><code class="lang-auto">RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons:

1) Use of a module parameter outside the `forward` function.
Please make sure model parameters are not shared across multiple concurrent
forward-backward passes. or try to use _set_static_graph() as
a workaround if this module graph does not change during training loop.

2) Reused parameters in multiple reentrant backward passes.
For example, if you use multiple `checkpoint` functions to wrap the same part of your model,
it would result in the same set of parameters been used by different reentrant backward passes
multiple times, and hence marking a variable ready multiple times. DDP does not support such use
cases in default. You can try to use _set_static_graph() as a workaround
if your module graph does not change over iterations.
Parameter at index 95 has been marked as ready twice. This means that multiple autograd engine 
hooks have fired for this particular parameter during this iteration. You can set the environment variable
TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print parameter names for further
debugging.
</code></pre>
<p>Could you please help me?</p>

username: ptrblck
cooked: <p>Is this error only raised in a distributed setup? Could you post a minimal, executable code snippet to reproduce the error, please?</p>

username: Anwarvic
cooked: <p>Yes, the error only occurs when using multiple GPUs. I’m so sorry, but it’s so hard to create a minimal code for this. I use the <strong>fairseq</strong> framework and there are a lot of pieces attached to it.</p>
<p>If you don’t mind, let’s keep discussing this in the <a href="https://discuss.pytorch.org/t/updating-two-sets-of-parameters-using-two-optimizers-fails/144003">question’s thread</a>.</p>