Change:
My code runs well for a while , it occurs randomly..... Traceback (most recent call last): File "", line 236, in &lt;moduletrainvgg("",dset='luna16') File "", line 54, in trainvgg lossvalue.backward() File "/home/uih/anaconda2/lib/python2.7/site-", line 169, in backward torch.autograd.backward(self, gradient, retaingraph, creategraph, retainvariables) File "/home/uih/anaconda2/lib/python2.7/site-", line 99, in backward variables, gradvariables, retaingraph) RuntimeError: add: Assertion "pos = 0 &amp;&amp; pos &lt; ()" failed. Any idea what's happend? Thanks in advance~
Yes, it looks like we have a bug somewhere in the code. Can you please try to isolate a small snippet that still triggers the same error so we can look into it? Thank you!
It's hard for me to isolate, the error occurs when I do the train and validation in turn, if I just do either the train or val, everthing runs well.
The problem is that we won’t be able to fix it, until we actually see under what conditions does the error happen
I got the same error in v0.3.0 today. when I trained my model with single GPU, it worked perfectly;with multi-gpu the error occured. RuntimeError: add: Assertion failed.
We've also been struggling with this issues for a long time. Unfortunately, finding which part of our code specifically is triggering the assert (which used to be a segfault before ) has proven extremely hard and, as of now, unsuccessful. In fact, we never submitted a bug report as we couldn't create a self-contained piece of code to trigger it. I'll try to summarize all the information we've gathered: We know for a fact that the issue appeared sometime after this commit: . All our code works perfectly fine on or v0.2. The assert is always triggered during the backward pass, but the exact moment when it happens seems to be completely random: when training over Imagenet we've seen this after 50 iterations as well as after 5000. This makes it super-hard to understand whether a particular piece of code is triggering the assert or not, as you can never be sure how many iterations are enough to exclude that the assert will ever trigger! The issue seems to be somehow related to the use of multiple GPUs (via ): we've never observed it while training with a single GPU. The issue seems also to be related to the use of user-defined classes: we've never seen it when training networks composed of "standard" Pytorch functions / modules only... ... however, simply running forward / backward passes with our custom functions doesn't seem to be enough to trigger the assertion. In our experience, working with "complex" computational graphs increases the likelihood of encountering this issue. Recently we've been able to re-create the issue using and a modified version of , where we replace some of the layers with our custom-made ones. Unfortunately I can't share the code for now (our paper describing the new layers is still not out), but I'll come back to you as soon as I can! Some additional information: all our experiments are on Ubuntu 16.04 using CUDA 8, CUDNN 7 and TITAN X and Xp GPUs.
almost the same condition as us
Encounter the same condition after serveral imagenet epochs on a NasNet after update to 3.0. Pytorch 2.0 is working well.
I am seeing this too -- I have two runs of the same model running with different hyperparameters, one is running smoothly, this one quit with the exact same error avgloss = self.trainepoch(trainingdata,validationdata,batchsize,saveprefix,epoch) File "", line 759, in trainepoch gradients = loss.backward() File "/u/dcontrac/anaconda2/lib/python2.7/site-", line 167, in backward torch.autograd.backward(self, gradient, retaingraph, creategraph, retainvariables) File "/u/dcontrac/anaconda2/lib/python2.7/site-", line 99, in backward variables, gradvariables, retaingraph) RuntimeError: add: Assertion failed. Running PyTorch v0.3 and training on 4 K-80 GPUs. My other run with different parameters is also running on the exact same setup but its still running. Both runs differ only in shuffle of data and a different batch size (the one that failed had a slightly larger batch size)
: I'm not using any user define autograd.Function but I see the error too, so maybe thats not the primary place with a bug. Using nn.DataParallel though.
I'll try again to train a network using standard layers only, maybe I'll also encounter the error in the no-custom-functions case if I let it run a little bit longer. If this is the case, then I guess we have some strong indication that (or multi-GPU computation in general) could indeed be the source of the error!
: I dont think the duration has got anything to do with it --- One model run has been running without a problem for the last 18 hours, the other one had quit last night pretty early on (Crashed after the backward pass of the 8th batch). I have just restarted it and watching it.
I agree that the bug is probably not related to training length. My guess is that each backward pass has a certain probability of randomly failing with this assert, so running for longer just increases the overall chances of encountering the bug.
Thanks for the information. We'd really like to fix this bug, and so there are a few things that would help us out a lot (in order of difficulty): you run your training process with gdb attached with a breakpoint, so that we can get a backtrace when the assertion fails? Also, if you can go one frame up and print the name of the function (run in gdb) that would also be helpful. We basically want to know if it's either a C++ function or a Python function. soon as you can publish your latest layers, uploading a script that reproduces (even if it takes a while to reproduce); we would be happy to run the script to try to reproduce, which would be very helpful for us. We'd love a bisection, but given that it can sometimes take 18hrs+ to repro, it doesn't sound like that would be easy to do.
This is the backtrace: For the function name I'll probably have to recompile pytorch in debug mode, since in gdb returns . Edit - I managed to get the name of the function by catching the exception raised by the assert: . Edit 2 - I did a second run and the name of the function is still the same.
Actually I just realized that what we really need is Can you please try printing that (might require a debug build as well)?
Printing instead of I get , which is the name of our custom function's backward pass. More importantly, our code and paper are now public so I can share this with you: As I mentioned in a previous comment, this code is just the Pytorch ImageNet + ResNet examples, except that standard BN is replaced with our custom layer. When running this on our machines with 4 GPUs we always encounter the assert within the first epoch. Please feel free to ask if you need help running the code.
I also encountered this assert. I have two machines, one with 2, and other one with 4 GPUs. Both machines are Ubuntu 16.04, python 2.7, cuda 9.0, cudnn 7, and pytorch v0.3.0. I never encountered this assert on my 2 GPUs machine. This assert always appeared on my 4 GPUs machine.
same as here. I have python 2.7, cuda 8.0, cudnn 7 and pytorch v0.3.0 installed from binaries. The assertion was triggered when I was training networks on 4 GPUs. It happened at a random time during training.
Same here, with multiple pixel-wise prediction models with just CNNs. Seems to occur only with multiple GPUs, has not happened so far with single GPU. python 2.7, CUDA 8.0, CUDNN 7, PyTorch v0.3.0.
can you take a look at this asap
edit: resolved, I ended up adding a compilation flag for compute capacity 3.5 to pytorchbugreport's building scripts (to work with my gpus) I'm getting the following after trying to run pytorchbugreport. Have you seen this before? !
you have to change to cover your GPU, which is compute35,code=sm_35 i think (if it is K40)
I ran your code for six epochs on a machine with 4 gpus on pytorch 0.3.0 but haven't been able to trigger the assert. I'll try letting it run for longer and see if anything pops up
For us the assert almost always triggers during the first epoch. There might be some important difference in our setups. This is the exact sequence of steps we have used to reproduce the error across several machines: from a clean Ubuntu 16.04 installation with CUDA 8.0, CUDNN 7.0, pip, virtualenv and the compilers. and activate a new python2 virtualenv. necessary python packages . the pytorch repository and checkout and install with our repository and compile our native module with the training as described before. Our machines all have either Titan X or Titan Xp GPUs, so a different architecture from the one you are using, I don't know if this could have an influence on the error.
My machine has 4 GTX 1080 Ti which triggers the error.
we (finally) found a different repro that triggers this issue internally on one of our machines, a fix will be issued soon.
Quick update: we're still looking into this, but so far, I've been able to get this bug to trigger with two machines using python 2.7 but never with python 3. If you're in need of a quick fix using python 3 might help.
I also encountered this propblem, with pytorch 0.3.0, python2.7, CUDA8.0, Titan X GPUs. My session always crashed around 8 epochs. And I degrade pytorch to 0.2.0. now the training seems work well. Someone in emergency can try this.
Thank you , I'll try to switch to python 3 for now!
let me know if the bug comes back in python 3. As far as I can tell the python version shouldn't matter in the triggering of this bug... but it's never happened to me with python 3 yet.
Crashed in Python2.7 RuntimeError: add: Assertion failed Absolutely random crash using multi gpu.
This has been fixed in HEAD but there has not been a release of 0.3 with the fix yet.
Oh, Thanks for your reply. I will try the HEAD version. Thank you .
i have encountered this error , but change the pytorch to the latest version(0.4) will solve it.
Encountered same error when I add dropout layer to a Resnet which I modified. Trained on multi-GPU. Does latest conda release fix the problem?
it is not yet on latest conda release. to get the fix at the moment you will have to install pytorch from source:
The same bug ! !
did you install pytorch from source?
Is there a simple way to block this error without changingthe version of python or pyotrch?
You could disable the python reference cycle . This would probably lead to memory leaks though.
It should be safe if you do this once all the graphs are deleted, and there will be no leaks unless you create cycles in your data structures
After 2 days of struggle, I finally changed back to 0.2.0 and expects the release of binary 0.4.0. Anyway, thank you for your advices. :)
I encountered the same error with version 0.3.0.post4. What was the proper fix here?
upgrade to version 0.3.1 or newer.

Discussion:
self.assertEqual(counter[0], 1, 'bw_hook not called') self.assertEqual(x.grad.data, torch.ones(5, 5) * 2) <del> @unittest.skipIf(sys.version_info[0] == 2, "Python 2 doesn't collect cycles involving __del__") def test_hooks_cycle(self): import gc counter = [0] class GradHook(object): def __init__(self, var): self.var = var def __del__(self): counter[0] += 1 def __call__(self, *args): pass def run_test(): x = Variable(torch.ones(5, 5), requires_grad=True) y = x * 2 x.register_hook(GradHook(x)) y.register_hook(GradHook(y)) y._backward_hooks[1] = GradHook(y) run_test() gc.collect() self.assertEqual(counter[0], 3) </del> def test_hook_none(self): # WARNING: this is a test for autograd internals. # You should never have to use such things in your code.

{ Py_VISIT(self->data); Py_VISIT(self->backward_hooks); <ins> // We don't want to traverse the grad_fn, even if the Variable owns it and the // shared pointer's use count is 1. This is because we would need to treat // the grad_fn as part of the Python state and hold the GIL sometimes when // grad_fn's shared_ptr is copied, otherwise a race condition with the Python // GC could occur. Holding the GIL when the shared_ptr is copied adds // undesirable complexity/overhead. // // When hooks, a Variable, and its grad_fn are involved in a Python reference // cycle, because we're not traversing the grad_fn, the reference cycle will // in fact leak. // // See https://gist.github.com/zou3519/7ac92b84dd7d206dcc6eae55fee8372c // for more details about the race condition involving traversing the grad_fn // and the python GC. </ins> if (self->cdata.defined()) { <del> // Only visit this if we actually own it (no one else use the shared pointer) auto& grad_fn = self->cdata.grad_fn(); if (grad_fn.use_count() == 1) { if (auto fn = dynamic_cast<PyFunction*>(grad_fn.get())) { Py_VISIT(fn->obj); } else { // visit hooks in C++ implemented autograd functions for (auto& hook : grad_fn->pre_hooks) { if (auto pyhook = dynamic_cast<PyFunctionPreHook*>(hook.get())) { Py_VISIT(pyhook->dict); } } for (auto& hook : grad_fn->post_hooks) { if (auto pyhook = dynamic_cast<PyFunctionPostHook*>(hook.get())) { Py_VISIT(pyhook->dict); } } } } </del> for (auto& hook : self->cdata.hooks()) { if (auto pyhook = dynamic_cast<PyFunctionPreHook*>(hook.get())) { Py_VISIT(pyhook->dict);