Gradient mismatch for complex networks using nn.DataParallel

Minennick: <p>Dear all,</p>
<p>I am currently building a network that gets complex numbers as input, processes them using complex layers, and gives back again complex numbers. I already recognized, that passing the input to the model, causes the input to automatically be viewed as 2-channeled real values, as is the case for applying torch.view_as_real(). Since I handle data in the complex view the first thing I do in the forward() method of my network is transforming it back to complex view.</p>
<p>When I use only one GPU to train, everything works fine and I have the expected behavior.<br>
Given that I want to train on multiple GPUs I get the following error message in the backward pass.</p>
<pre><code class="lang-auto">Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: Function GatherBackward returned an invalid gradient at index 0 - got [2, 3, 128, 128, 2] but expected shape compatible with [2, 3, 128, 128]
</code></pre>
<p>Now for me, it looks like that as soon as I train on multiple GPUs the gradients are in the shape of the 2-channeled real view, while my inputs and weights are in complex view.</p>
<p>I am currently using DataParallel as:</p>
<pre><code class="lang-auto">if not 'CUDA_VISIBLE_DEVICES' in os.environ:
        os.environ['CUDA_VISIBLE_DEVICES'] = f'{args.gpus}'
model = torch.nn.DataParallel(model)
</code></pre>
<p>and pytorch 1.12.1 py3.9_cuda10.2_cudnn7.6.5_0</p>
<p>Maybe someone can shed some light on why this happens, and how to fix.</p>
<p>Best regards,<br>
Niklas</p>

ptrblck: <p>I’m not sure why the error is raised, but note that <code>nn.DataParallel</code> is in “maintenance mode” as of <a href="https://github.com/pytorch/pytorch/issues/65936">this RFC</a> and should eventually be removed.<br>
We recommend using <code>DistributedDataParallel</code> for a better performance and support.<br>
Could you try to use your code using <code>DDP</code> and see if the same error is raised?</p>

Minennick: <p>Dear Piotr,<br>
thank you very much for the advice. Using DDP instead of DP actually solved the issue.<br>
From my point of view, I sadly did not find out what the actual problem is with using DP. But in case someone else has this problem I want to refer to this <a href="https://medium.com/codex/a-comprehensive-tutorial-to-pytorch-distributeddataparallel-1f4b42bb1b51" rel="noopener nofollow ugc">medium article</a>, which helped in understanding and setting up the DDP along with the official Pytorch documentation.</p>
<p>Best regards,<br>
Niklas</p>

Stanislaw_Raczynski: <p>Hi,<br>
I’m running Pytorch 2.0.1 with CUDA 12.1 and am getting the same problem. I get the following exception on GPU:1, on GPU:0 it runs without issues.</p>
<pre><code class="lang-auto">RuntimeError: Function GatherBackward returned an invalid gradient at index 0 - got [5, 512, 901, 2] but expected shape compatible with [5, 512, 901]
&gt; /work/miniconda3/envs/sayso_dev/lib/python3.9/site-packages/torch/autograd/__init__.py(200)backward()
    198     # some Python versions print out the first line of a multi-line function
    199     # calls in the traceback and some print out the last line
--&gt; 200     Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    201         tensors, grad_tensors_, retain_graph, create_graph, inputs,
    202         allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
</code></pre>

Minennick: <p><a class="mention" href="/u/stanislaw_raczynski">@Stanislaw_Raczynski</a> Do you use DataParallel or Distributed Data Parallel?</p>

Stanislaw_Raczynski: <p>Sorry I wasn’t clear. I was using <code>DataParallel</code>, as in the initial post. I am trying to move to <code>DistributedDataParallel</code> as per suggestions from ptrblck.</p>
<p>The error above points to a complex tensor (result of <code>torch.stft()</code>) within my model.</p>