Detaching a sequence of tensors gives inplace modification error

ZimoNitrome: <p>I previously had some code for a GAN:</p>
<pre><code class="lang-python">generated_samples = G(z)

# Train Discriminator
pred_generated = D(generated_samples.detach())
pred_real = D(real_samples)

loss = BCELoss(pred_generated, 0) + BCELoss(pred_real, 1)
loss.backward()

# Train Generator
pred_generated2 = D(generated_samples)

loss = BCELoss(pred_generated2, 1)
loss.backward()
</code></pre>
<p>This worked fine. I then updated the discriminator to consist of 2 steps (where I want to save the intermediate output for later). This first step returns a sequence of tensors which are called features:</p>
<pre><code class="lang-python">generated_samples = G(z)

# Train Discriminator
generated_features = get_features(generated_samples) # new
real_features = get_features(real_samples)
real_features = [f.detach() for f in real_features]

pred_generated = D([f.detach() for f in generated_features])
pred_real = D(real_features)

loss = BCELoss(pred_generated, 0) + BCELoss(pred_real, 1)
loss.backward()

# Train Generator
pred_generated2 = D(generated_features)

loss = BCELoss(pred_generated2, 1)
loss.backward()
</code></pre>
<p>But I get an error:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [320]] is at version 5; expected version 3 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>Which seems to relate to the line of code:<br>
<code>pred_generated = D([f.detach() for f in generated_features])</code></p>
<p>How should I correctly detach tensors in <code>generated_features</code> as I pass them to the discriminator?</p>
<p>Why does it work for a single tensor and not a sequence?</p>
<p><strong>Edit:</strong> <code>f.clone().detach()</code> nor <code>f.detach().clone()</code> seems to work.</p>
<p><strong>Edit2:</strong> I tried the anomaly detection and it points to the line: <code>generated_features = get_features(generated_samples)</code> as being the “root” of the error.</p>
<p><strong>Edit3:</strong> The following code works, but then I have to extract features 1 time extra… not efficient…:</p>
<pre><code class="lang-python">generated_samples = G(z)

# Train Discriminator
generated_features = get_features(generated_samples.detach()) # new
real_features = get_features(real_samples)
real_features = [f.detach() for f in real_features]

pred_generated = D([f.detach() for f in generated_features])
pred_real = D(real_features)

loss = BCELoss(pred_generated, 0) + BCELoss(pred_real, 1)
loss.backward()

# Train Generator
generated_features = get_features(generated_samples) # new
pred_generated2 = D(generated_features)

loss = BCELoss(pred_generated2, 1)
loss.backward()
</code></pre>

ptrblck: <p>I’m not sure what exactly might be causing the issue as it seems that <code>D</code> gets detached tensors and thus “Train Discriminator” shouldn’t update <code>G</code>.<br>
Could you check, if any gradients are created in the “Train Discriminator” step or if the parameters or <code>G</code> were changed? I don’t know how the optmizer(s) were created but in case you are using an optimizer with running stats and are updating <code>G</code> already in the “Train Discriminator” step (even with zero gradients in <code>G</code>) you might be hitting <a href="https://discuss.pytorch.org/t/struggling-with-a-runtime-error-related-to-in-place-operations/102222/6">this issue</a>.</p>