How to pause discriminator from updating?

Kalle: <p>So I am currently playing around with Super-Resolution Generative Adversarial Network and gray-scale images. I am using the PyTorch implementation <a href="https://github.com/leftthomas/SRGAN" rel="noopener nofollow ugc">here</a>. Since VGG network expects 3 channels as input I did the following to make it work:</p>
<pre><code class="lang-auto">target_images = target_images.repeat(1, 3, 1,1)
out_images = out_images.repeat(1, 3, 1,1)
</code></pre>
<p>However I noticed that the discriminators loss quickly went to 0 and the score of Discriminator and Generator did not oscillate around 0.5 as expected.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/5/55a891c3594b097fb1b7365db2953ea567f6f67d.png" data-download-href="https://discuss.pytorch.org/uploads/default/55a891c3594b097fb1b7365db2953ea567f6f67d" title="srgan_mstar_losses"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55a891c3594b097fb1b7365db2953ea567f6f67d_2_690x244.png" alt="srgan_mstar_losses" data-base62-sha1="cdLK4Zx0K33wmfj5SxBy4X5QC6x" width="690" height="244" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55a891c3594b097fb1b7365db2953ea567f6f67d_2_690x244.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55a891c3594b097fb1b7365db2953ea567f6f67d_2_1035x366.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55a891c3594b097fb1b7365db2953ea567f6f67d_2_1380x488.png 2x" data-dominant-color="F8F8F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">srgan_mstar_losses</span><span class="informations">1478×523 83.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>However when I trained the network with color images I got the following losses which was what I would expect:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/e/5e73eb0f20079334ae8ae1f4f77c7ab04c180c66.png" data-download-href="https://discuss.pytorch.org/uploads/default/5e73eb0f20079334ae8ae1f4f77c7ab04c180c66" title="srgan_losses"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/e/5e73eb0f20079334ae8ae1f4f77c7ab04c180c66_2_690x242.png" alt="srgan_losses" data-base62-sha1="dtzeCulHkYs8SKwyv7jeplYP7qS" width="690" height="242" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/e/5e73eb0f20079334ae8ae1f4f77c7ab04c180c66_2_690x242.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/e/5e73eb0f20079334ae8ae1f4f77c7ab04c180c66_2_1035x363.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/e/5e73eb0f20079334ae8ae1f4f77c7ab04c180c66_2_1380x484.png 2x" data-dominant-color="F8F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">srgan_losses</span><span class="informations">1438×505 142 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I tried multiple things to fix this and none of them worked such as: training generator more often, training discriminator more often and add more layers to both discriminator and generator. The newest Idea was to only train discriminator if the loss was above a specific value. Is this a bad idea? I tried this the following way:</p>
<pre><code class="lang-auto"># Calculate gradients for D in backward pass
if d_loss &gt;D_LOSS_MIN:
    d_loss.backward(retain_graph=True)

# causes the optimizer to take a step based on the gradients of the parameters.
if d_loss &gt;D_LOSS_MIN:
    optimizerD.step()
</code></pre>
<p>Is this the correct way of not updating the discriminator by only doing backward and step if the discriminators loss is above some value? Or am I missing something? When I try with D_LOSS_MIN = 0.9 the discriminators loss never goes above 0.9. So as far as I understand the discriminator never get updated.</p>
<p>And do any of you have some suggestions of how to improve it?<br>
Thanks!</p>