RNN module weights are not part of single contiguous chunk of memory

Hao_Chu: <p>When I pass input to <code>nn.GRU</code>, it comes across such problem:</p>
<pre><code>UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greately increasing memory usage. To compact weights again call flatten_parameters().
output, h_n = self.gru(concatenated_input.transpose(0, 1), h_0)
...
RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1502006348621/work/torch/lib/THC/generic/THCStorage.cu:66
</code></pre>
<p>What could cause such problem?</p>

Hao_Chu: <p>I solve the OOM problem by setting input <code>Variable</code> to be volatile, while this “not compact” problem remain unsolved.</p>

WendyShang: <p>Hi, did you try</p>
<pre><code>self.gru.flatten_parameters()
output, h_n = self.gru(concatenated_input.transpose(0, 1), h_0)</code></pre>

Hao_Chu: <p>I’ve tried that, but it doesn’t work.</p>

dpernes: <p>This happening not only with GRU but also with (vanilla) RNN and LSTM. I also have no idea why…</p>

Younggun_Lee: <p>I have this problem too.<br>
In my case, this problem occurs only when I wrap the whole model with DataParallel.</p>
<p><a class="mention" href="/u/wendyshang">@WendyShang</a><br>
What is the proper usage of flatten_parameters() and what does it do?<br>
Should I call flatten_parameters at initialization or at every forward step?</p>

WendyShang: <p>Sorry for the late reply.<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/59b139dabd8689163ec71fbcc987c88f3cc7e5ae/torch/legacy/nn/Module.py#L240" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/59b139dabd8689163ec71fbcc987c88f3cc7e5ae/torch/legacy/nn/Module.py#L240" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/59b139dabd8689163ec71fbcc987c88f3cc7e5ae/torch/legacy/nn/Module.py#L240</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="230" style="counter-reset: li-counter 229 ;">
<li>
</li>
<li>    # 7. fix up the parameter tensors to point at the flattened parameters</li>
<li>    for param, meta in zip(parameters, parameterMeta):</li>
<li>        param.set_(flatParameters.storage(),</li>
<li>                   meta['storage_offset'],</li>
<li>                   meta['size'],</li>
<li>                   meta['stride'])</li>
<li>
</li>
<li>    return flatParameters</li>
<li>
</li>
<li class="selected">def flattenParameters(self):</li>
<li>    _params = self.parameters()</li>
<li>    if _params is None:</li>
<li>        return</li>
<li>    parameters, gradParameters = _params</li>
<li>    p, g = self._flatten(parameters), self._flatten(gradParameters)</li>
<li>
</li>
<li>    assert p.nelement() == g.nelement()</li>
<li>    if parameters:</li>
<li>        for param, grad in zip(parameters, gradParameters):</li>
<li>            assert param.storage_offset() == grad.storage_offset()</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>This is the code for flatten paramters. It makes the memory the parameters and its grads occupied contiguous.</p>
<p>However, eventually I solved this issue by deleting all the intermediate variables, such as h_n, c_n etc (for LSTM).</p>

dylanthomas: <p>How should I use flattenParameters to fix the problem?  Where am I supposed to put the code ?</p>

Yu_Li: <p>Hi, did your problem solved? I have the save question and I’m looking forward to the answer.</p>

k_drosos: <p>Hi, any news on that warning? I’m using GRUs and I have the same issue.</p>

norm: <p>Same issue… What changes have to be made in the code?</p>

jef: <p>I also faced same problem. But in my case, the warning was only appeared at first time call of LSTM (<code>forward</code>). And GPU and CPU memory doesn’t increase.<br>
I faced this problem after I changed trainable parameters by myself.</p>

BigeyeDestroyer: <p>I also met this problem when I am trying to stack a few single-layer rnn to constitute a multi-layer one. My code is just like this:</p>
<pre><code class="lang-python">self.rnns = [nn.LSTM(input_size=input_size if l == 0 else 2 * cell_size,
             hidden_size=cell_size, num_layers=1, batch_first=True,
             bidirectional=True) for l in range(num_layers)]
</code></pre>
<p>And I solved the problem by adding following line:</p>
<pre><code class="lang-python">self.rnns = torch.nn.ModuleList(self.rnns)
</code></pre>

KaiyangZhou: <p>I met the same problem and solved it by adding</p>
<blockquote>
<p>self.rnn.flatten_parameters()</p>
</blockquote>
<p>into the</p>
<blockquote>
<p>def forward(self, x)</p>
</blockquote>
<p>at the position before calling</p>
<blockquote>
<p>rnn_output, _ = self.rnn(x, h)</p>
</blockquote>
<p>By doing so, the error never showed up.</p>

klory: <p>This solves the problem, did not know how this func works…</p>

DoubtWang: <aside class="quote no-group" data-username="KaiyangZhou" data-post="14" data-topic="6011">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kaiyangzhou/48/3850_2.png" class="avatar"> KaiyangZhou:</div>
<blockquote>
<p>self.rnn.flatten_parameters()</p>
</blockquote>
</aside>
<p>Hi,<br>
I also use your suggestion, but I get other error message.</p>
<pre><code class="lang-auto">  File "/home/.local/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 65, in parallel_apply
    raise output
  File "/home/.local/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 41, in _worker
    output = module(*input, **kwargs)
  File "/home/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/remote/cnn-daily/models/seq2seq.py", line 170, in forward
    final_dist, sent_dist, word_dist, coverage, target = self.train_model(src_extend_pad, src_len, sent_num_, src_pad_mask, tgt_extend_pad, max_oovs)
  File "/home/remote/cnn-daily/models/seq2seq.py", line 129, in train_model
    sentence_states, sentence_contexts, contexts, features, coverage = self.hierarchy(src_extend_pad, src_len, sent_num)
  File "/home/remote/cnn-daily/models/seq2seq.py", line 108, in hierarchy
    _contexts_, _features_, _states_ = self.encoder(_src_extend_pad_, length_.tolist())
  File "/home/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/remote/cnn-daily/models/rnn.py", line 287, in forward
    hiddens, states = self.rnn(embs)
  File "/home/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/.local/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 192, in forward
    output, hidden = func(input, self.all_weights, hx, batch_sizes)
  File "/home/.local/lib/python3.6/site-packages/torch/nn/_functions/rnn.py", line 323, in forward
    return func(input, *fargs, **fkwargs)
  File "/home/.local/lib/python3.6/site-packages/torch/nn/_functions/rnn.py", line 287, in forward
    dropout_ts)
RuntimeError: torch/csrc/autograd/variable.cpp:115: get_grad_fn: Assertion `output_nr == 0` failed.

</code></pre>
<p>If I remove the <code>self.rnn.flatten_parameters()</code>, this message  will not raise.<br>
Could u give some suggestion ?</p>

Wu_jiang: <aside class="quote no-group quote-modified" data-username="KaiyangZhou" data-post="14" data-topic="6011">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kaiyangzhou/48/3850_2.png" class="avatar"> KaiyangZhou:</div>
<blockquote>
<blockquote>
<p>self.rnn.flatten_parameters()</p>
</blockquote>
</blockquote>
</aside>
<p>Can you explain the effect  of this operation?<br>
should use in forward or <strong>init</strong> ?<br>
THX</p>

Wu_jiang: <p>Can you explain the effect of this operation?<br>
should use in forward or  <strong>init</strong>  ?<br>
THX</p>

guilhermedrud: <p>This works pretty well for me, thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Hiker: <p>Thanks a lot ! I’ve solved my problem with you sugesstion.</p>