DistributedDataParallel do not work with custom function in model

username: hadaev8
cooked: <p>Im trying to use this model</p><aside class="onebox githubblob" data-onebox-src="https://github.com/NVIDIA/tacotron2/blob/master/model.py#L487">
  <header class="source">

      <a href="https://github.com/NVIDIA/tacotron2/blob/master/model.py#L487" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/NVIDIA/tacotron2/blob/master/model.py#L487" target="_blank" rel="noopener nofollow ugc">NVIDIA/tacotron2/blob/master/model.py#L487</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="477" style="counter-reset: li-counter 476 ;">
          <li>    input_lengths = to_gpu(input_lengths).long()</li>
          <li>    max_len = torch.max(input_lengths.data).item()</li>
          <li>    mel_padded = to_gpu(mel_padded).float()</li>
          <li>    gate_padded = to_gpu(gate_padded).float()</li>
          <li>    output_lengths = to_gpu(output_lengths).long()</li>
          <li></li>
          <li>    return (</li>
          <li>        (text_padded, input_lengths, mel_padded, max_len, output_lengths),</li>
          <li>        (mel_padded, gate_padded))</li>
          <li></li>
          <li class="selected">def parse_output(self, outputs, output_lengths=None):</li>
          <li>    if self.mask_padding and output_lengths is not None:</li>
          <li>        mask = ~get_mask_from_lengths(output_lengths)</li>
          <li>        mask = mask.expand(self.n_mel_channels, mask.size(0), mask.size(1))</li>
          <li>        mask = mask.permute(1, 0, 2)</li>
          <li></li>
          <li>        outputs[0].data.masked_fill_(mask, 0.0)</li>
          <li>        outputs[1].data.masked_fill_(mask, 0.0)</li>
          <li>        outputs[2].data.masked_fill_(mask[:, 0, :], 1e3)  # gate energies</li>
          <li></li>
          <li>    return outputs</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
But getting this error</p>
<blockquote>
<p>File “/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py”, line 60, in <em>worker<br>
output = module(*input, **kwargs)<br>
File “/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 547, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/ec2-user/SageMaker/tacotron2/model/model.py”, line 480, in forward<br>
output_lengths)<br>
File “/home/ec2-user/SageMaker/tacotron2/model/model.py”, line 452, in parse_output<br>
outputs[0].data.masked_fill</em>(mask, 0.0)<br>
RuntimeError: The expanded size of the tensor (1079) must match the existing size (836) at non-singleton dimension 2.  Target sizes: [4, 80, 1079].  Tensor sizes: [4, 80, 836]</p>
</blockquote>
<p>How to solve it?</p>

username: pritamdamania87
cooked: <p>Could you provide your DDP code to reproduce the issue? Also, does the model work properly without DDP?</p>

username: hadaev8
cooked: <p><a href="https://colab.research.google.com/drive/104LtQ1zIioIOMQEPgVve77m5Rd4Gm0wU" class="onebox" target="_blank" rel="nofollow noopener">https://colab.research.google.com/drive/104LtQ1zIioIOMQEPgVve77m5Rd4Gm0wU</a><br>
Yes, it is works fine, also same code works fine on single gpu colab instance.<br>
Tested on 8 v100 instance from amazon.</p>