Multi-gpu, RuntimeError: Gather got an input of invalid size: got [1, 230], but expected [1, 231]

Vinsent_Paramananth1: <p>RuntimeError: Gather got an input of invalid size: got [1, 230], but expected [1, 231]</p>
<pre><code class="lang-auto">File "main.py", line 517, in train
    fscore, fscore_epoch = ao.train(output_dir=hps.output_dir)
  File "main.py", line 328, in train
    y, _ = nn.DataParallel(self.model(seq, trg), device_ids=gpus, dim=0)  ## TODO  look how they are training the seq2seq model....
  File "/opt/conda/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 153, in forward
    return self.gather(outputs, self.output_device)
  File "/opt/conda/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 165, in gather
    return gather(outputs, output_device, dim=self.dim)
  File "/opt/conda/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 68, in gather
    res = gather_map(outputs)
  File "/opt/conda/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 63, in gather_map
    return type(out)(map(gather_map, zip(*outputs)))
  File "/opt/conda/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 55, in gather_map
    return Gather.apply(target_device, dim, *outputs)
  File "/opt/conda/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/_functions.py", line 68, in forward
    return comm.gather(inputs, ctx.dim, ctx.target_device)
  File "/opt/conda/envs/py36/lib/python3.6/site-packages/torch/cuda/comm.py", line 165, in gather
    return torch._C._gather(tensors, dim, destination)
RuntimeError: Gather got an input of invalid size: got [1, 230], but expected [1, 231] (gather at /opt/conda/conda-bld/pytorch_1579022051443/work/torch/csrc/cuda/comm.cpp:231)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x47 (0x7f32e0f7d627 in /opt/conda/envs/py36/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: torch::cuda::gather(c10::ArrayRef&lt;at::Tensor&gt;, long, c10::optional&lt;int&gt;) + 0x2ad (0x7f32ec0b476d in /opt/conda/envs/py36/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #2: &lt;unknown function&gt; + 0x9f7904 (0x7f3311def904 in /opt/conda/envs/py36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #3: &lt;unknown function&gt; + 0x28c076 (0x7f3311684076 in /opt/conda/envs/py36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
&lt;omitting python frames&gt;
frame #13: THPFunction_apply(_object*, _object*) + 0xa1f (0x7f3311a6de3f in /opt/conda/envs/py36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
</code></pre>
<p>Pointing for more explanation @<br>
</p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/44141" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/44141" target="_blank" rel="nofollow noopener">RuntimeError: Gather got an input of invalid size: got [1, 230], but expected [1, 231] when running with 2 or more gpus</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-09-03" data-time="20:13:49" data-timezone="UTC">08:13PM - 03 Sep 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/VinACE" target="_blank" rel="nofollow noopener">
          <img alt="VinACE" src="https://avatars2.githubusercontent.com/u/7286426?v=4" class="onebox-avatar-inline" width="20" height="20">
          VinACE
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">❓ Questions and Help
Please note that this issue tracker is not a help form and this issue will be closed.
We have...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


ptrblck: <p>Could you post a minimal executable code snippet to reproduce this issue, please?</p>

vinsentds: <p>Hi ,<br>
I will try to come up with some minimal code.<br>
Regards<br>
Vinsent P.</p>