Why do we need "flatten_parameters" when using RNN with DataParallel

username: wwiiiii
cooked: <p>I got the following warning message when I use LSTM with nn.DataParallel.</p>
<pre data-code-wrap="bash"><code class="lang-bash">RuntimeWarning: RNN module weights are not part of single contiguous chunk of memory.
This means they need to be compacted at every call, possibly greatly increasing memory usage.
To compact weights again call flatten_parameters().
</code></pre>
<p>I found the error is gone when I put <code>self.lstm.flatten_parameters()</code> at the top of <code>forward</code> function, but I wonder why do we need it.</p>
<p>Why is the weight of RNN non-contiguous on memory when we use nn.DataParallel?</p>
<p>And also I found the error would be gone if we replace DataParallel with DistributedDataParallel, then why isn’t the weight non-contiguous in latter case?</p>
<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/7092">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/7092" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/7092" target="_blank" rel="noopener nofollow ugc">Multi-GPU autograd error with Pytorch 0.4</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2018-04-30" data-time="09:42:40" data-timezone="UTC">09:42AM - 30 Apr 18 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2020-10-30" data-time="20:18:53" data-timezone="UTC">08:18PM - 30 Oct 20 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/erogol" target="_blank" rel="noopener nofollow ugc">
          <img alt="erogol" src="https://avatars.githubusercontent.com/u/1402048?v=4" class="onebox-avatar-inline" width="20" height="20">
          erogol
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          todo
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">After updating pytorch 0.4 I am getting the following error when I try to train <span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">my model here: https://github.com/mozilla/TTS with multi-gpus. I have no idea about what it means unfortunately. A bug or just a problem that I need some feedback on. Thx. 

```
Traceback (most recent call last):
  File "train.py", line 403, in &lt;module&gt;
    main(args)
  File "train.py", line 393, in main
    model, criterion, train_loader, optimizer, epoch)
  File "train.py", line 111, in train
    model.forward(text_input, mel_spec)
  File "/home/erogol/miniconda3/envs/pytorch4/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 114, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/erogol/miniconda3/envs/pytorch4/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 124, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/erogol/miniconda3/envs/pytorch4/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 65, in parallel_apply
    raise output
  File "/home/erogol/miniconda3/envs/pytorch4/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 41, in _worker
    output = module(*input, **kwargs)
  File "/home/erogol/miniconda3/envs/pytorch4/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/erogol/projects/TTS/models/tacotron.py", line 28, in forward
    encoder_outputs = self.encoder(inputs)
  File "/home/erogol/miniconda3/envs/pytorch4/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/erogol/projects/TTS/layers/tacotron.py", line 205, in forward
    return self.cbhg(inputs)
  File "/home/erogol/miniconda3/envs/pytorch4/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/erogol/projects/TTS/layers/tacotron.py", line 183, in forward
    outputs, _ = self.gru(x)
  File "/home/erogol/miniconda3/envs/pytorch4/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/erogol/miniconda3/envs/pytorch4/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 192, in forward
    output, hidden = func(input, self.all_weights, hx, batch_sizes)
  File "/home/erogol/miniconda3/envs/pytorch4/lib/python3.6/site-packages/torch/nn/_functions/rnn.py", line 323, in forward
    return func(input, *fargs, **fkwargs)
  File "/home/erogol/miniconda3/envs/pytorch4/lib/python3.6/site-packages/torch/nn/_functions/rnn.py", line 287, in forward
    dropout_ts)
RuntimeError: torch/csrc/autograd/variable.cpp:115: get_grad_fn: Assertion `output_nr == 0` failed.
```</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I found some similar questions but none of them had the answer.</p><aside class="quote" data-post="1" data-topic="29186">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/76d3ee/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/what-does-flatten-parameters-do/29186">What does flatten_parameters() do?</a> 
  </div>
  <blockquote>
    I saw many Pytorch examples using flatten_parameters in the forward function of the RNN 
self.rnn.flatten_parameters()

I saw this <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/rnn.html" rel="noopener nofollow ugc">RNNBase</a> and it is written that it 

Resets parameter data pointer so that they can use faster code paths 

What does that mean?
  </blockquote>
</aside>
<aside class="quote" data-post="1" data-topic="18585">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/67e7ee/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/do-we-need-to-call-flatten-parameters-in-lstm-only-if-we-are-using-multi-gpus/18585">Do we need to call flatten_parameters() in LSTM only if we are using Multi-GPUs?</a> <a class="badge-category__wrapper " href="/c/nlp/8"><span data-category-id="8" data-drop-close="true" class="badge-category " title="Topics related to Natural Language Processing"><span class="badge-category__name">nlp</span></span></a>
  </div>
  <blockquote>
    I am trying to understand the use case for flatten_parameters(), when do we use it and what does it do? Is it used only if we are running our model on multiple GPUs?
  </blockquote>
</aside>


username: wwiiiii
cooked: <p>After reading some related codes, I think I almost get it but still have few questions.<br>
So what I understand is</p>
<hr>
<p>Everytime when we make new RNN module instance, it <a href="https://github.com/pytorch/pytorch/blob/c611630b9df4a25007dd299676b2bb88bf692784/torch/nn/modules/rnn.py#L72" rel="nofollow noopener">allocates</a> new <code>w_ih, w_hh, b_ih, b_hh</code> tensors and register them as <code>Parameter</code> for each layer, direction.</p>
<p>But it’s not guranteed that new tensors are contiguous on GPU memory, performance can be dropped due to the fragmentation. So we call <code>flatten_parameters</code> function at the end of constructor to <a href="https://github.com/pytorch/pytorch/blob/c611630b9df4a25007dd299676b2bb88bf692784/torch/nn/modules/rnn.py#L90" rel="nofollow noopener">aggregate all the weight tensors into continuous space of GPU memory</a>.</p>
<p>This task is done as</p>
<ol>
<li>
<a href="https://github.com/pytorch/pytorch/blob/d1623f4cc9334ade7376b3685bb91d3071e3b418/aten/src/ATen/native/cudnn/RNN.cpp#L647" rel="nofollow noopener">Allocate</a> one big buffer tensor called <code>weight_buf</code>
</li>
<li>
<a>Copy</a> values of weight tensor into <code>weight_buf</code>
</li>
<li>Make each weight tensor’s internal data pointer <a href="https://github.com/pytorch/pytorch/blob/d1623f4cc9334ade7376b3685bb91d3071e3b418/aten/src/ATen/native/cudnn/RNN.cpp#L469" rel="nofollow noopener">indicating</a> <code>weight_buf + offset</code>
</li>
</ol>
<p>(The real execution steps are 1-&gt;3-&gt;2 in real code)</p>
<p>But when we use nn.DataParallel, it <a>replicates</a> original module(which is allocated only on certain GPU device) to every GPU it uses, then weight tensors are fragmented again since there’s no gurantee that replicated tensors are still contiguous on memory space.</p>
<p>Therefore we should <code>flatten_parameters</code> again everytime the module is replicated to another GPU, and the best place to put function call would be the head of <code>forward</code> function (of nn.Module), because <code>forward</code> function of <code>nn.Module</code> on each GPU is called only one time when <code>forward</code> of <code>nn.DataParallel</code> is called.</p>
<p>Although I never used <code>nn.DistributedDataParallel</code>, I guess that the reason why it doesn’t need the <code>flatten_parameters</code> call is because when it allocates new instance of RNN module, <code>flatten_parameters</code> are automatically called, then it doesn’t move internal data position on memory unlike <code>nn.DataParallel</code>, but it only copies some values into it.</p>
<hr>
<p>And questions are</p>
<ol>
<li>
<p>Do I understand right? Is there any misunderstood point?</p>
</li>
<li>
<p>When we do the step 3 of aggregation(=Make each weight tensor’s internal data pointer indicating <code>weight_buf + offset</code>), we call the <a>get_parameters</a> function and it</p>
<ul>
<li>calls <a href="https://docs.nvidia.com/deeplearning/sdk/cudnn-developer-guide/index.html#cudnnGetRNNLinLayerMatrixParams" rel="nofollow noopener">cudnnGetRNNLinLayerMatrixParams</a> so that <code>matrix_pointer</code> indicates the GPU memory position of original, un-aggregated weight tensor,</li>
<li>
<a href="https://github.com/pytorch/pytorch/blob/d1623f4cc9334ade7376b3685bb91d3071e3b418/aten/src/ATen/native/cudnn/RNN.cpp#L454" rel="nofollow noopener">sets offset</a> as the difference of <code>matrix_pointer</code> and start of <code>weight_buf</code>,</li>
<li>make internal data pointer of weight tensor <a href="https://github.com/pytorch/pytorch/blob/d1623f4cc9334ade7376b3685bb91d3071e3b418/aten/src/ATen/native/cudnn/RNN.cpp#L469" rel="nofollow noopener">indicating</a> <code>weight_buf + offset</code>
</li>
</ul>
<p>Then isn’t it indicating <code>matrix_pointer</code> again?    Why don’t we replcate</p>
<p><code>Tensor param = at::empty({0}, weight_buf.options()).set_(weight_buf.storage(), offset, size);</code><br>
with</p>
<p><code>Tensor param = at::empty({0}, weight_buf.options()).set_(weight_buf.storage(), cumsum, size); cumsum += size;</code><br>
?<br>
Or does that function calculate expected position of given component with respect to the given (start) data pointer?</p>
</li>
</ol>

username: Valiox
cooked: <aside class="quote no-group" data-username="wwiiiii" data-post="2" data-topic="46506">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wwiiiii/48/6613_2.png" class="avatar"> wwiiiii:</div>
<blockquote>
<p>Therefore we should <code>flatten_parameters</code> again everytime the module is replicated to another GPU, and the best place to put function call would be the head of <code>forward</code> function (of nn.Module), because <code>forward</code> function of <code>nn.Module</code> on each GPU is called only one time when <code>forward</code> of <code>nn.DataParallel</code> is called.</p>
</blockquote>
</aside>
<p>That’s the conclusion I came to as well, except that I actually observe a larger VRAM usage and loss compute time when I put <code>flatten_parameters</code> in the forward pass (and I get no warning)  vs. putting it in the <code>__init__</code> function of the model (and then I get the warning only when using DataParallel).</p>