Training multiple models independently

username: cijerezg
cooked: <p>Hello,</p>
<p>I am implementing the actor-critic algorithm. I have my actor and critic that are both neural nets but I want to train simultaneously and independently.</p>
<p>What I currently have looks more or less like this:</p>
<pre><code class="lang-auto">actor = Net1()
critic = Net2()
optim_a = optim.SGD(actor.parameters(),lr=0.1, momentum=0.9)
optim_c = optim.SGD(critic.parameters(),lr=0.1, momentum=0.9)
optim_a.zero_grad()
optim_c.zero_grad()
output_c = actor(input)
output_a = critic(output_c, other_inputs)
loss = output_c+output_a
loss.backward()
optim_a.step()
optim_c.step()
</code></pre>
<p>But when I train, the weights are not changing even though the loss is not 0, that is, there is some issue when the backpropagatiion is carried out. Furthermore, I would ideally like to have this</p>
<pre><code class="lang-auto">output_c.backward()
optim_c.step()
output_a.backward()
optim_c.step()
</code></pre>
<p>Because I am not sure, when I add them up, as in the first block, the optimizer modifies weights independently (I assume it does). I emphasize this because the output of the critic depends on the actor.</p>
<p>So, I’d appreciate any hint on how to do this or a different way to implement this.</p>
<p>I am aware of <a href="https://github.com/pytorch/examples/blob/master/reinforcement_learning/actor_critic.py" rel="nofollow noopener">actor-critic pytorch tutorial</a>. But they define one neural network with two independent outputs that they use to have as actor and critic, but nevertheless both share some of the weights. Ultimately, if this is the only option, then I’ll implement a shared neural network instead of two independent ones.</p>
<p>Thanks</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="cijerezg" data-post="1" data-topic="77257">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cijerezg/48/14945_2.png" class="avatar"> cijerezg:</div>
<blockquote>
<p>But when I train, the weights are not changing even though the loss is not 0, that is, there is some issue when the backpropagatiion is carried out.</p>
</blockquote>
</aside>
<p>Could you print out some <code>.grad</code> attributes of some parameters of both models after the <code>loss.backward()</code> call?<br>
If they are <code>None</code>, you might be breaking the computation graph somewhere in your forward passes.</p>
<aside class="quote no-group" data-username="cijerezg" data-post="1" data-topic="77257">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cijerezg/48/14945_2.png" class="avatar"> cijerezg:</div>
<blockquote>
<p>Because I am not sure, when I add them up, as in the first block, the optimizer modifies weights independently (I assume it does). I emphasize this because the output of the critic depends on the actor.</p>
</blockquote>
</aside>
<p>I don’t understand this section.<br>
Both approaches (<code>loss.backward()</code> and calling <code>backward</code> of both outputs) should create the same gradients. You could check it again using some dummy data and comparing the <code>.grad</code> attributes.</p>
<p>The difference would be that <code>optimizer.step()</code> is called twice, which would make a difference, e.g. if you are using momentum.</p>

username: cijerezg
cooked: <p>Thanks. I’'l print the attributes to see if I am breaking the training somewhere. If that doesn’t work, I’ll edit my question and word it better to make it more clear.</p>

username: cijerezg
cooked: <p>It seems it was working the entire time. I was just missreading the weights update. Something similar to what happened <a href="https://discuss.pytorch.org/t/gradients-exist-but-weights-not-updating/20484">here</a>.</p>

username: erenovic
cooked: <p>Hello Piotr, I was wondering if PyTorch internally parallelizes the forward pass of given two models here (actor() and critic()). I am trying to build something similar where I have two models which are not capable of using the whole GPU (as far as I noticed it uses 4 GiB of 32 GiB when I checked with nvidia-smi command).<br>
That’s why I want to train two models in single GPU and hopefully have a decent speed compared to sequentially training two models.</p>
<p>To achieve my goal, I have initialized those two models in a nn.Module child class and called them in the forward in seperate lines but the duration was even longer than twice the duration a single training takes. Can you point me in the right direction to achieve my goal?</p>

username: ptrblck
cooked: <p>No, these models won’t run in parallel as the execution is sequential due to the data dependence:</p>
<pre><code class="lang-python">output_c = actor(input)
output_a = critic(output_c, other_inputs)
</code></pre>
<p>I.e. <code>critic</code> cannot start the execution before <code>output_c</code> is calculated.</p>
<p>If models do not depend on each other you could execute them in parallel, if the GPU has free resources. <a href="https://github.com/pytorch/pytorch/issues/59692#issuecomment-859173905">Here</a> is another related post.</p>