Backward error caused by batch norm in 1.5.1

Red_Lv: <p>Source code: <a href="https://github.com/agrimgupta92/sgan" rel="nofollow noopener">https://github.com/agrimgupta92/sgan</a><br>
When I set batch_norm to true, and here comes the backward error.<br>
Warning: Error detected in CudnnBatchNormBackward. Traceback of forward call that caused the error:<br>
File “train_dist.py”, line 644, in <br>
main(args)<br>
File “train_dist.py”, line 282, in main<br>
optimizer_d)<br>
File “train_dist.py”, line 433, in discriminator_step<br>
scores_fake = discriminator(traj_fake, traj_fake_rel, seq_start_end)<br>
File “/home/caros/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/caros/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/distributed.py”, line 458, in forward<br>
output = self.module(*inputs[0], **kwargs[0])<br>
File “/home/caros/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/caros/sgan-master/sgan/models.py”, line 619, in forward<br>
scores = self.real_classifier(classifier_input)<br>
File “/home/caros/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/caros/anaconda3/lib/python3.7/site-packages/torch/nn/modules/container.py”, line 100, in forward<br>
input = module(input)<br>
File “/home/caros/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/caros/anaconda3/lib/python3.7/site-packages/torch/nn/modules/batchnorm.py”, line 106, in forward<br>
exponential_average_factor, self.eps)<br>
File “/home/caros/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py”, line 1923, in batch_norm<br>
training, momentum, eps, torch.backends.cudnn.enabled<br>
(print_stack at /pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:60)<br>
Traceback (most recent call last):<br>
File “train_dist.py”, line 644, in <br>
main(args)<br>
File “train_dist.py”, line 282, in main<br>
optimizer_d)<br>
File “train_dist.py”, line 451, in discriminator_step<br>
data_loss.backward()<br>
File “/home/caros/anaconda3/lib/python3.7/site-packages/torch/tensor.py”, line 198, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/home/caros/anaconda3/lib/python3.7/site-packages/torch/autograd/<strong>init</strong>.py”, line 102, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1]] is at version 4; expected version 3 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>

Junochiu: <p>Hi <a class="mention" href="/u/red_lv">@Red_Lv</a>,  is your problem solved? I am running into the same situation as you did. I will really appreciate if you could share how your problem was solved <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Youwei_Liang: <p>I encountered the same error when testing with <code>DistributedDataParallel </code> and <code>convert_sync_batchnorm</code>. I got the same error when testing with <strong>1 GPU</strong> in my local machine. <strong>I found using more GPUs would resolve the error.</strong></p>