Issue model parallelism

username: Thies1006
cooked: <p>I am quite a pytorch newby, I hope this is the right place to post my issue. I am trying to train a transformer model with model parallelism following closely the megatron example from fairseq (just using complete transformer model instead gpt, same options including --fp16).</p><aside class="onebox githubfolder" data-onebox-src="https://github.com/facebookresearch/fairseq/tree/main/examples/megatron_11b">
  <header class="source">

      <a href="https://github.com/facebookresearch/fairseq/tree/main/examples/megatron_11b" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h3><a href="https://github.com/facebookresearch/fairseq/tree/main/examples/megatron_11b" target="_blank" rel="noopener nofollow ugc"></a></h3>

  <p><a href="https://github.com/facebookresearch/fairseq/tree/main/examples/megatron_11b" target="_blank" rel="noopener nofollow ugc">//github.com/facebookresearch/fairseq/tree/main/examples/megatron_11b</a></p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>My setup is: two nodes with 6 GPUs (Titan RTX) each.<br>
Pytorch 1.6<br>
Cuda 10.1.243<br>
Ubuntu 18.04 LTS</p>
<p>The model trains, however only with 4 GPUs per node. When switching to 6 GPUs per node (plus tweaking the model/dictionary to ensure divisibility by number of GPUs) I get the following error on the second node (right when training should start):</p>
<blockquote>
<p>terminate called after throwing an instance of ‘c10::Error’<br>
what(): CUDA error: misaligned address<br>
Exception raised from create_event_internal at /pytorch/c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first):<br>
frame <span class="hashtag-raw">#0:</span> c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7fc06b9c91e2 in /secondary/thies/.virtualenvs/pytorch-1.6/lib/python3.6/site-packages/torch/lib/libc10.so)<br>
frame [<span class="hashtag-raw">#1</span>]: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xad2 (0x7fc06bc17f92 in /secondary/thies/.virtualenvs/pytorch-1.6/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)<br>
frame <a href="https://github.com/pytorch/fairseq/issues/2" rel="noopener nofollow ugc">#2</a>: c10::TensorImpl::release_resources() + 0x4d (0x7fc06b9b79cd in /secondary/thies/.virtualenvs/pytorch-1.6/lib/python3.6/site-packages/torch/lib/libc10.so)<br>
frame [<span class="hashtag-raw">#3</span>]: std::vector&lt;at::Tensor, std::allocatorat::Tensor &gt;::~vector() + 0x5c (0x7fc0b3262d1c in /secondary/thies/.virtualenvs/pytorch-1.6/lib/python3.6/site-packages/torch/lib/libtorch_python.so)<br>
frame [<span class="hashtag-raw">#4</span>]: torch::autograd::Engine::evaluate_function(std::shared_ptrtorch::autograd::GraphTask&amp;, torch::autograd::Node*, torch::autograd::InputBuffer&amp;, std::shared_ptrtorch::autograd::ReadyQueue const&amp;) + 0x16b2 (0x7fc0a5d8f6b2 in /secondary/thies/.virtualenvs/pytorch-1.6/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)<br>
frame [<span class="hashtag-raw">#5</span>]: torch::autograd::Engine::thread_main(std::shared_ptrtorch::autograd::GraphTask const&amp;) + 0x451 (0x7fc0a5d8ffa1 in /secondary/thies/.virtualenvs/pytorch-1.6/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)<br>
frame [<span class="hashtag-raw">#6</span>]: torch::autograd::Engine::thread_init(int, std::shared_ptrtorch::autograd::ReadyQueue const&amp;, bool) + 0x89 (0x7fc0a5d88119 in /secondary/thies/.virtualenvs/pytorch-1.6/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)<br>
frame [<span class="hashtag-raw">#7</span>]: torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptrtorch::autograd::ReadyQueue const&amp;, bool) + 0x4a (0x7fc0b352834a in /secondary/thies/.virtualenvs/pytorch-1.6/lib/python3.6/site-packages/torch/lib/libtorch_python.so)<br>
frame [<span class="hashtag-raw">#8</span>]: + 0xbd6ef (0x7fc0b46826ef in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)<br>
frame [<span class="hashtag-raw">#9</span>]: + 0x76db (0x7fc0b85746db in /lib/x86_64-linux-gnu/libpthread.so.0)<br>
frame [<span class="hashtag-raw">#10</span>]: clone + 0x3f (0x7fc0b88ad88f in /lib/x86_64-linux-gnu/libc.so.6)</p>
</blockquote>
<p>It seems there is problem with the c10 library, however I get exactly the same error when adding the --ddp-backend=no_c10d option.</p>
<p>When removing the fp16 option the model trains fine on the c10 backend.</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/thies1006">@Thies1006</a>, as developers watching this channel have limited knowledge about fairseq models, it’s hard to tell what went wrong. Have you tried posting this as an issue in fairseq repo?</p>

username: Thies1006
cooked: <p>Hi! Sorry for the delay. Yes, I posted this as well in the fairseq repo. It seems that when compiling pytorch from sources, this error disappears.</p>