Tracking Loss through Parallel Paths

username: Joshua_Clancy
cooked: <p>Hello!</p>
<p>I am a bit of a noob here so forgive me if this has a simple answer. For context, I successively implemented a tied weight VAE a few months ago and now I am testing out some other ideas to do with autoencoders. My current model has three parallel paths going through it in an attempt to make three specialized and separate latent spaces.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/2/42acfdcc3635b169852fdd1dd09d6d6d13fad484.png" data-download-href="https://discuss.pytorch.org/uploads/default/42acfdcc3635b169852fdd1dd09d6d6d13fad484" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/2/42acfdcc3635b169852fdd1dd09d6d6d13fad484_2_690x305.png" alt="image" data-base62-sha1="9vQ90X1DAlU8OfJCFyZR05eigxm" width="690" height="305" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/2/42acfdcc3635b169852fdd1dd09d6d6d13fad484_2_690x305.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/2/42acfdcc3635b169852fdd1dd09d6d6d13fad484_2_1035x457.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/2/42acfdcc3635b169852fdd1dd09d6d6d13fad484_2_1380x610.png 2x" data-dominant-color="F1F3F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1852×821 112 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am experimenting with how to specialize the latent spaces, but that is not my concern here. This question is specifically about the loss. Right now I have a list of losses (one for each path) and I loop through the losses performing backpropagation for each.</p>
<pre><code class="lang-auto">for loss in local_loss_list:
                lossy = loss
                # backpropagation
                optimizer.zero_grad()
                lossy.backward(retain_graph=True)
                # one step of the optmizer (using the gradients from backpropagation)               
                optimizer.step()
                total_loss = total_loss + lossy
</code></pre>
<p>I did this automatically without much thought, but now I am concerned. Will the loss from each parallel path backpropagate properly down the correct path? If so how? It seems to work, but I do not understand how the loss is being associated with the correct path of weights.</p>
<p>Thanks!!!</p>

username: albanD
cooked: <p>Hi,</p>
<p>The autograd computes the gradient for what you give it. So if you backprop for a loss computed with only one branch, it will backprop only that branch.</p>
<p>Note that if you just want to get the sum of the gradients and do a single update, you can do:</p>
<pre><code class="lang-auto">lossy = sum(local_loss_list)
# backpropagation
optimizer.zero_grad()
lossy.backward()
# one step of the optmizer (using the gradients from backpropagation)               
optimizer.step()
total_loss = total_loss + lossy
</code></pre>

username: Joshua_Clancy
cooked: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="64018">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>hat if you just want to get the sum of the gradients and do a single update</p>
</blockquote>
</aside>
<p>Yeah now that you mention it I think I have read that before. Thank you. So to be clear, keeping the parallel path’s losses separate is not actually doing anything because autograd’s computational graph keeps track of which weights are effecting the loss. So I can then just add them up and update them based on that? Is there any scenario where I should be worried about autograd not assigning the gradient to the correct weightings that are effecting the loss? Or does it just do all of that for you?</p>

username: albanD
cooked: <p>The autograd is a tool that just computes gradients. So it will set the gradients only for what is used for each loss.</p>