Multiple outputs, losses, and optimizers

username: Joshua_Clancy
cooked: <p>I am working on a visual model with multiple outputs and thus multiple losses. I was under the impression that I could simply add the losses together and backpropagate over the aggregate. This school of thought seems quite common throughout the forums, for example <a href="https://discuss.pytorch.org/t/tracking-loss-through-parallel-paths/64018">here</a> and <a href="https://discuss.pytorch.org/t/what-does-the-backward-function-do/9944">here</a>.</p>
<p>But I came across this <a href="https://stackoverflow.com/questions/46774641/what-does-the-parameter-retain-graph-mean-in-the-variables-backward-method" rel="noopener nofollow ugc">StackOverflow</a> thread that says there is an advantage with keeping the two losses separate if you use two different optimizers. For example, multiple Adam optimizers would each optimize the different output pathways. In that thread they say:</p>
<blockquote>
<p>“Let’s say loss 1 varies rapidly with your parameters but is small in magnitude. You’d need small steps to optimize it, because it’s not smooth. And loss 2 varies slowly, but is big in magnitude. <span class="hashtag-raw">#2</span> will dominate their sum, so one shared ADAM will choose a big learning rate. But if you keep them separate, ADAM will chose a big learning rate for loss <span class="hashtag-raw">#2</span> and a small learning rate for loss <span class="hashtag-raw">#1</span>”</p>
</blockquote>
<p>I’m wondering how this would look in practice. Would this work?</p>
<pre><code class="lang-auto">optimizer1 = optim.Adam(model.parameters(), lr=lr, momentum=momentum)
optimizer2 = optim.Adam(model.parameters(), lr=lr, momentum=momentum)

for data, target in train_loader:
  optimizer1.zero_grad()
  optimizer2.zero_grad()
  data = data.to(device)
  target = target.to(device)
  output = model(data)
  loss1 = loss_fn(output, target)
  loss2 = loss_fn2(output, target)

  loss1.backward(retain_graph = True)
  loss2.backward(retain_graph = True)
  optimizer1.step()
  optimizer2.step()
</code></pre>
<p>Retain_graph = True seems to be needed. based on that thread.</p>
<p>This seems to me to be advantageous, does anyone see any compute problems with this? Specifically what if I scaled up, and had many different output pathways and optimizers? <strong>for example 100?</strong> would this still be feasible?</p>

username: Dipam_Vasani
cooked: <p>This is a really good. I am not an experienced practitioner but I would really like to see how this compares against adding the losses together in some way. Did you try it on a prototype? What are the results?</p>

username: Joshua_Clancy
cooked: <p>Yeah I have been trying to implement but have run into problems. Specifically:</p>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [3, 10, 3, 3]] is at version 3; expected version 2 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>
<p>Still looking into a fix (speak up if you have an idea of how to <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:">)</p>

username: futscdav
cooked: <p>Unless your loss / model modifies the value in place, as the error says, it could be caused by calling <code>optimizer1.step()</code> before calculating <code>loss2.backward()</code>. Make sure it works with 1 optimizer before doing two.</p>

username: Joshua_Clancy
cooked: <p>you are good! thank you I was doing that the wrong way around.</p>
<p>I ended up with this (for varying amounts of outputs)</p>
<pre><code class="lang-auto">optimizerList = []
for n in range(output_num):  
  optimizer = getOptimizer(model.parameters(), run)
  optimizerList.append(optimizer)

for data, target in train_loader:
  data = data.to(device)
  target = target.to(device)

  output = model(data)

  lossList = []
  for n in range(output_num):  
    loss = loss_fn(output, target)
    lossList.append(loss)
  for n in range(output_num):  
    lossList[n].backward(retain_graph = True)
  for n in range(output_num):  
    optimizerList[n].step()

</code></pre>
<p>This seems to work.<br>
I am still unsure about how compute-intensive this method is though. My gut is telling me that retaining multiple backward-pass graphs before the optimizer step could be a problem.</p>