BiGAN implementation

username: zahra97
cooked: <p>Hi,</p>
<p>I am trying to reproduce the results of this <a href="https://github.com/mehak25/BiGAN" rel="noopener nofollow ugc">repo</a> related to BiGAN.</p>
<p>Here is the error that I get:</p>
<pre><code class="lang-auto">one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [5, 1]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

</code></pre>
<p>The part of code related to this error:</p>
<pre><code class="lang-auto"> def run_on_batch(model,discriminator,data,mask,decay,rdecay,args, optimizer,optimizer_d,epoch):
        ret_f,ret = model(data, mask, decay,rdecay,args)
        
        disc = discriminator(ret['originals'], mask, args)
        #print("BATCH LOSS",ret['loss'])
        #print("BATCH LOSS",disc['loss_g'])
        #print("BATCH LOSS",disc['loss_d'])
        #print("one batch done")

        if optimizer is not None:
            #print("OPTIMIZE")
            
            if (epoch%10==0):
                optimizer_d.zero_grad()
                disc['loss_d'].backward(retain_graph=True)
                optimizer_d.step()

            optimizer.zero_grad()
            (ret['loss']+disc['loss_g']).backward()
            optimizer.step()

        return ret_f,ret,disc
</code></pre>
<p>I also do not know the reason for back propagating the loss_d every 10 epochs?</p>

username: ptrblck
cooked: <p>I guess the error is raised by keeping the computation graph alive via:</p>
<pre><code class="lang-python">disc['loss_d'].backward(retain_graph=True)
</code></pre>
<p>Could you explain why <code>retain_graph=True</code> is used as its often used as a workaround masking other (real) issues causing new ones.</p>

username: zahra97
cooked: <p>I am not sure but maybe it is related to the this part of the paper(the repo is the implementation of this paper):</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4aac901074f3d50aadaf9dad70671a96493f0d86.png" data-download-href="https://discuss.pytorch.org/uploads/default/4aac901074f3d50aadaf9dad70671a96493f0d86" title="Screenshot 2023-06-14 at 16.56.11"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4aac901074f3d50aadaf9dad70671a96493f0d86_2_602x499.png" alt="Screenshot 2023-06-14 at 16.56.11" data-base62-sha1="aEB2TUjKni990rBhKD1EE720qbQ" width="602" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4aac901074f3d50aadaf9dad70671a96493f0d86_2_602x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4aac901074f3d50aadaf9dad70671a96493f0d86_2_903x748.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/a/4aac901074f3d50aadaf9dad70671a96493f0d86.png 2x" data-dominant-color="E6E7E4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-06-14 at 16.56.11</span><span class="informations">1010×838 223 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>while we are training the discriminator, we want to maximize the probability of correctly classifying the actual (as real) and the generated values (as fake). and beside that simultaneously the generator wants to minimise the probability that D correctly identifies the fake instances.  so when we are back propagating the discriminator loss we also need to backprobagate the generator loss at same time so we need to keep the computation_graph same between these 2 backpropagations.</p>

username: ptrblck
cooked: <p>Even if you need to keep the computation graph alive to backpropagate multiple times through it you should still make sure the last <code>backward</code> call clears it by using the default <code>retain_graph=False</code> argument.</p>

username: zahra97
cooked: <p>which part of the code should be changed to make sure for it? In this code, the last <code>backward</code> has the default value for <code>retain_graph</code>. Is not it enough?</p>