Failing to replicate model on MultipleGPUs

username: mujtabaasif
cooked: <p>Previously I raised an issue <a href="https://github.com/pytorch/pytorch/issues/34941" rel="nofollow noopener">#34941</a>. After debugging the issue I found there is a bug in function take_tensors <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/utils/tensor_flatten.cpp#L10" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/csrc/utils/tensor_flatten.cpp#L10</a><br>
The functions lost the track of a buffer in the following scenario.</p>
<p>Function Input:<br>
-&gt; Tensor of size 356, size_limit , fine_grained = false</p>
<p>Function steps:<br>
At line  <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/utils/tensor_flatten.cpp#L55" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/csrc/utils/tensor_flatten.cpp#L55</a><br>
we have 3 groups:<br>
group 1 -&gt; 1 element, size 0<br>
group 2 -&gt; 238 elements<br>
group 3 -&gt; 117 elements</p>
<p>So due to size 0 of group 1, this <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/utils/tensor_flatten.cpp#L57" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/csrc/utils/tensor_flatten.cpp#L57</a> condition becomes True and ignores group 1.</p>
<p>At line <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/utils/tensor_flatten.cpp#L62" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/csrc/utils/tensor_flatten.cpp#L62</a><br>
The result becomes:<br>
group 2 -&gt; 238 elements<br>
group 3 -&gt; 117 elements</p>
<p>so the tensor size is 238 + 117 = 355</p>
<p>Which causes this to assert failure because of (356 != 355)<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/utils/tensor_flatten.cpp#L66" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/utils/tensor_flatten.cpp#L66" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/torch/csrc/utils/tensor_flatten.cpp#L66</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="56" style="counter-reset: li-counter 55 ;">
<li>    auto&amp; group = entry.second;</li>
<li>    if (!fine_grained &amp;&amp; group.size == 0) {</li>
<li>      continue;</li>
<li>    }</li>
<li>    results.emplace_back(std::move(group));</li>
<li>  }</li>
<li>  return results;</li>
<li>}</li>
<li>
</li>
<li>void reorder_tensors_like(std::vector&lt;Tensor&gt;&amp; tensors, TensorList order) {</li>
<li class="selected">  AT_ASSERT(tensors.size() == order.size());</li>
<li>  std::unordered_map&lt;at::DeprecatedTypeProperties*, std::vector&lt;size_t&gt;&gt; type_indices;</li>
<li>  for (size_t i = 0, num_tensors = tensors.size(); i &lt; num_tensors; ++i)</li>
<li>    type_indices[&amp;tensors[i].type()].push_back(i);</li>
<li>
</li>
<li>  std::unordered_map&lt;at::DeprecatedTypeProperties*, size_t&gt; type_used;</li>
<li>  std::vector&lt;Tensor&gt; ordered_tensors;</li>
<li>  ordered_tensors.reserve(tensors.size());</li>
<li>  for (auto &amp; tmpl_tensor : order) {</li>
<li>    auto * type = &amp;tmpl_tensor.type();</li>
<li>    auto &amp; indices = type_indices[type];</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Here is the traceback of a bug</p>
<p><code>"/opt/conda/envs/m2release/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__ result = self.forward(*input, **kwargs) File "/opt/conda/envs/m2release/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 154, in forward replicas = self.replicate(self.module, self.device_ids[:len(inputs)]) File "/opt/conda/envs/m2release/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 159, in replicate return replicate(module, device_ids, not torch.is_grad_enabled()) File "/opt/conda/envs/m2release/lib/python3.6/site-packages/torch/nn/parallel/replicate.py", line 102, in replicate buffer_copies_not_rg = _broadcast_coalesced_reshape(buffers_not_rg, devices, detach=True) File "/opt/conda/envs/m2release/lib/python3.6/site-packages/torch/nn/parallel/replicate.py", line 66, in _broadcast_coalesced_reshape return comm.broadcast_coalesced(tensors, devices) File "/opt/conda/envs/m2release/lib/python3.6/site-packages/torch/cuda/comm.py", line 39, in broadcast_coalesced return torch._C._broadcast_coalesced(tensors, devices, buffer_size) RuntimeError: tensors.size() == order.size() INTERNAL ASSERT FAILED at ../torch/csrc/utils/tensor_flatten.cpp:74, please report a bug to PyTorch.</code></p>