Updating two sets of parameters using two optimizers FAILS

username: Anwarvic
cooked: <p>I’m trying to train a model under two different loss functions using two optimizers; where the first updates the whole model’s parameters and the other updates only the decoder. The following is a simple code to explain what I mean:</p>
<pre><code class="lang-auto">opt1 = Adam(model.parameters())
opt2 = Adam(model.decoder.parameters())

opt1.step()
opt2.step()
</code></pre>
<p>The problem is when the model is initialized, I see the following error:</p>
<pre><code class="lang-auto">RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons:

1) Use of a module parameter outside the `forward` function.
Please make sure model parameters are not shared across multiple concurrent
forward-backward passes. or try to use _set_static_graph() as
a workaround if this module graph does not change during training loop.

2) Reused parameters in multiple reentrant backward passes.
For example, if you use multiple `checkpoint` functions to wrap the same part of your model,
it would result in the same set of parameters been used by different reentrant backward passes
multiple times, and hence marking a variable ready multiple times. DDP does not support such use
cases in default. You can try to use _set_static_graph() as a workaround
if your module graph does not change over iterations.
Parameter at index 95 has been marked as ready twice. This means that multiple autograd engine 
hooks have fired for this particular parameter during this iteration. You can set the environment variable
TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print parameter names for further
debugging.
</code></pre>
<p>I know this is because the <code>decoder.parameters()</code> were used twice. I don’t know how to fix this error.</p>
<p>Any help is very much appreciated</p>

username: Mathemagician
cooked: <p>Are you trying to have a larger effective learning rate for the decoder of your model? In that case multiple param groups would be more suitable?</p>
<pre><code class="lang-auto">opt = Adam([{'params' : model.encoder.parameters(), 'lr' : encoder_lr}, {'params' : model.decoder.parameters(), 'lr' : decoder_lr}])
</code></pre>

username: Anwarvic
cooked: <aside class="quote no-group" data-username="Mathemagician" data-post="2" data-topic="144003">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/a698b9/48.png" class="avatar"> Mathemagician:</div>
<blockquote>
<p><code>{'params' : model.encoder.parameters(), 'lr' : encoder_lr</code></p>
</blockquote>
</aside>
<p>No, I’m using two optimizers to update two sets of parameters based on two different loss functions.</p>

username: Mathemagician
cooked: <p>Why is summing the two losses and using a single optimizer not an option here?</p>

username: Anwarvic
cooked: <p>I want to update the (encoder+decoder) parameters based on the first loss… and update the (decoder) parameters based on the second loss. Can I do that using one optimizer?</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/anwarvic">@Anwarvic</a> will it work if you manually exclude decoder parameters from the first optimizer? Sth like</p>
<pre><code class="lang-python">decoder_param_ids = [id(p) for p in decoder.parameters()]

opt1 = Adam([p for p in model.parameters() if id(p) not in decoder_param_ids])
opt2 = Adam(decoder.parameters())
</code></pre>

username: Anwarvic
cooked: <p>Thank you for your answer. But in your code, <code>opt1</code> has only the parameters of the encoder while <code>opt2</code> has the parameter of the decoder. In my use case, I need <code>opt1</code> to have all (encoder + decoder) parameters.</p>
<p>I should have added this to my question, but my code works just fine on CPUs. The problem occurs when I use multiple GPUs.</p>

username: Anwarvic
cooked: <p>Yes, I have finally made it work thanks to God.</p>
<p>The problem was in using two optimizers. I don’t know why though. However, the reason behind using them is to update two sets of parameters separately. And this can be done using just one optimizer. Here is what I did:</p>
<p>First, I created two loss functions:</p>
<pre><code class="lang-auto">criterion = Criterion()
decoder_criterion = AnotherCriterion()
</code></pre>
<p>Then, I created one optmizer that monitors the whole model parameters:</p>
<pre><code class="lang-auto">opt = Optim(model.parameters())
</code></pre>
<p>Then, you can get the two losses separately like so:</p>
<pre><code class="lang-auto">model_loss = criterion(model_output, model_target)
decoder_loss = decoder_criterion(decoder_output, decoder_target)
</code></pre>
<p>Finally, you can perform the backward propagation over the two-loss functions at the same time like so:</p>
<pre><code class="lang-auto">loss = model_loss + decoder_loss
loss.backward()
opt.step()
</code></pre>
<p>This is how I fixed my problem. The following is a few more details in case you were interested:</p>
<ol>
<li>
<code>loss.backward()</code>: all it does is to calculate the gradient of all parameters that were used in the <code>loss</code>'s <code>forward()</code> method that has <code>require_grad=True</code>. Given a parameter <code>x</code>, this method saves its gradient wrt the <code>loss</code> inside the <code>x.grad</code> variable.</li>
<li>
<code>opt.step()</code>: all it does is to update these parameters using the gradient.</li>
<li>When you use <code>(loss1 + loss2).backward()</code>: this performs the backward() over <code>loss1</code> and <code>loss2</code> separately and accumulate the gradient. You don’t have to worry.</li>
</ol>
<p>This is what I learned so far. Please, don’t hesitate to correct me if I’m wrong.</p>