Code change:
I'm trying to implement distributed adversarial training in PyTorch. Thus, in my program pipeline I need to forward the output of one DDP model to another one. When I run the code in distributed setting, the following error is throwed: frame : c10::Error::Error(c10::SourceLocation, std::string const&) + 0x45 (0x7fa248f906d5 in /export/home/haoran/anaconda3/envs/torch1.2/lib/python3.6/site-) frame : c10d::Reducer::prepareforbackward(std::vector<torch::autograd::Variable, std::allocator<torch::autograd::Variableconst&) + 0x666 (0x7fa26f9d0896 in /export/home/haoran/anaconda3/envs/torch1.2/lib/python3.6/site-) frame : <unknown function+ 0x6df3b8 (0x7fa26f9c43b8 in /export/home/haoran/anaconda3/envs/torch1.2/lib/python3.6/site-) frame : <unknown function+ 0x1fc7f0 (0x7fa26f4e17f0 in /export/home/haoran/anaconda3/envs/torch1.2/lib/python3.6/site-) frame : main + 0x16d (0x400c1d in /export/home/haoran/anaconda3/envs/torch1.2/bin/python) frame : _libcstartmain + 0xf5 (0x7fa285e06c05 in ) frame : /export/home/haoran/anaconda3/envs/torch1.2/bin/python() [0x4009e9] Running the following code with DistributedDataParallel <!-- If you have a code sample, error messages, stack traces, please provide it here as well --The code could run successfully without throwing error. Please copy and paste the output from our (or fill out the checklist below manually). You can get the script and run it with: PyTorch version: 1.2.0.dev20190701 Is debug build: No CUDA used to build PyTorch: 9.0.176 OS: CentOS Linux 7 (Core) GCC version: (GCC) 5.4.0 CMake version: version 3.6.2 Python version: 3.6 Is CUDA available: Yes CUDA runtime version: 9.0.176 GPU models and configuration: GPU 0: Tesla P40 GPU 1: Tesla P40 GPU 2: Tesla P40 GPU 3: Tesla P40 Nvidia driver version: 390.25 cuDNN version: Could not collect Versions of relevant libraries: [pip] numpy==1.16.4 [pip] torch==1.2.0.dev20190701 [conda] blas 1.0 mkl defaults [conda] mkl 2019.4 243 defaults [conda] mklfft 1.0.12 py36ha843d7b0 defaults [conda] mklrandom 1.0.2 py36hd81dba30 defaults [conda] pytorch-nightly 1.2.0.dev20190701 py3.6cuda9.0.176cudnn7.5.1_0 pytorch <!-- Add any other context about the problem here. --
Please, try one of the suggested options: (1) passing the keyword argument findunusedparameters=True to ; (2) making sure all forward function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's forward function. Please include the loss function and the structure of the return value of forward of your module when reporting this issue (e.g. list, dict, iterable). If the issue will still exist, please, reopen this issue.
I double checked your repro to figure out if this is a real issue or not and found that even setting won't help you here. It's a simple example of course, but you're not using the variable in your loss computation. The model's forward function returns it which is an indication for DDP that it will receive a gradient. Then when it doesn't, and you try to run the next iteration, it raises an error, because the previous iteration didn't finish. I recommend you return only the tensor you plan to use in the loss computation.
Is there any way to see which parameters on earth are unused? It is useless to simply give this RuntimeError or set findunusedparameters to True.
I had the same problem with just one GPUI have modified the code according to what you said
I met the same problem and I found this topic. It works for me. (though it's been a long time for you)

Related discussion:
text: - [NVIDIA CUDA](https://developer.nvidia.com/cuda-downloads) 7.5 or above - [NVIDIA CuDNN](https://developer.nvidia.com/cudnn) v5.x <ins> If you want to disable CUDA support, export environment variable `NO_CUDA=1`. </ins> #### Install optional dependencies On Linux