DCGAN tutorial: Why calling backward twice on the discriminator loss?

rasbt: <p>Was just looking at the DCGAN tutorial (<a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">DCGAN Tutorial — PyTorch Tutorials 2.2.0+cu121 documentation</a>) and noticed that .backward() is called twice:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/5/b58738feaa53b648f5b7ebaa02eed1d297c0a771.png" data-download-href="https://discuss.pytorch.org/uploads/default/b58738feaa53b648f5b7ebaa02eed1d297c0a771" title="Screen Shot 2020-03-01 at 9.30.38 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b58738feaa53b648f5b7ebaa02eed1d297c0a771_2_469x500.png" alt="Screen Shot 2020-03-01 at 9.30.38 PM" data-base62-sha1="pTS8o8Lf2gulzMXSCJEVPNztcZj" width="469" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b58738feaa53b648f5b7ebaa02eed1d297c0a771_2_469x500.png, https://discuss.pytorch.org/uploads/default/original/3X/b/5/b58738feaa53b648f5b7ebaa02eed1d297c0a771.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/5/b58738feaa53b648f5b7ebaa02eed1d297c0a771.png 2x" data-dominant-color="E9E9EC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-03-01 at 9.30.38 PM</span><span class="informations">686×730 94.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Wouldn’t it be more efficient to compute</p>
<pre data-code-wrap="python"><code class="lang-python">errD = errD_fake + errD_real
</code></pre>
<p>first and then call</p>
<pre data-code-wrap="python"><code class="lang-python">errD.backward()
</code></pre>
<p>so that backward has only to be called once (assuming it’s an expensive operation)?</p>

tom: <p>Hi Sebastian,</p>
<p>I think it’s a tradeoff between time and GPU memory.<br>
For the memory: With the setup in the code, you can produce and process the autograd graphs for real and fake separately, so in a very lame sense, you can “double the batch size”.</p>
<p>In terms of performance: In the end, yes, without memory constraints, you would probably go for doing both at the same time (but also for the forward!). However, the advantage of “processing more data in one go” levels off at some point, so it might less general when you manage to efficiently use the GPU with the current setup, you have little to gain by doing real and fake in one go. (Basically just from the gradient accumulation itself, not so much the intermediates in the backward pass.)</p>
<p>Now, if you run on multi-GPU, the scaling probably works in favor of the backward once approach you suggest, at least if you also parallelize the forward passes.</p>
<p>In the end, the situation seems complicated enough that with the right batch size, I’d expect the sequential setup to be competitive in terms of speed. But it’s not optimization until you measure it, so it’d be interesting to hear about your experience.</p>
<p>Best regards</p>
<p>Thomas</p>

rasbt: <p>Thanks for the response, Thomas!</p>
<p>So with “my” setup you would basically call 2-times forward before you call <code>backward()</code>, and since backward clears the graph, you have a larger graph in the background in my case. I fact, now I think that both approaches I listed above may be equivalent (depending on how autograd is implemented) in terms of the number of operations performed by <code>backward</code>. In that case, the approach in the tutorial is probably preferred because you have a smaller graph, keeping memory consumption lower.</p>
<p>Btw when I benchmarked both approaches, both approaches had similar speed. But yeah, it’s a relatively small network and image size.</p>