[solved] Why we need to detach Variable which contains hidden representation?

wasiahmad: <p>I was going through the pytorch official example - “word_language_model” and found the following line of code in the <code>train()</code> function.</p>
<pre><code># Starting each batch, we detach the hidden state from how it was previously produced.
# If we didn't, the model would try backpropagating all the way to start of the dataset.
hidden = repackage_hidden(hidden)
</code></pre>
<p>I am not understanding why we need to detach hidden variable from the hidden variable associated with the previous batch of input? When gradients are computed and loss is backpropagated, weight matrices are usually affected by the chain of computation.</p>
<p>Why the hidden variables which represents hidden states of a recurrent neural network is at a risk to get affected during backpropagation and should be detached from previous value?</p>
<p>When the hidden variable should be detached from previous value? At the beginning of each batch? or beginning of each training epoch?</p>

chenzhekl: <p>This is for doing truncated BPTT. The dependency graph of RNN can be simply viewed as this.</p>
<pre><code>c1, h1 -&gt; c2, h2 -&gt; c3, h3 -&gt; ... -&gt; cn, hn
    |        |         |                |
  loss1    loss2     loss3            lossn
</code></pre>
<p>If we did not truncate the history of hidden states (c, h), the back-propagated gradients would flow from the loss towards the beginning, which may result in gradient vanishing or exploding. Detailed explanation can be found here.</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <a href="http://www.wildml.com/2015/10/recurrent-neural-networks-tutorial-part-3-backpropagation-through-time-and-vanishing-gradients/" target="_blank" rel="nofollow noopener" title="12:00PM - 08 October 2015">WildML – 8 Oct 15</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/398;"><img src="http://www.wildml.com/wp-content/uploads/2015/10/rnn-bptt1.png" class="thumbnail" width="690" height="398"></div>

<h3><a href="http://www.wildml.com/2015/10/recurrent-neural-networks-tutorial-part-3-backpropagation-through-time-and-vanishing-gradients/" target="_blank" rel="nofollow noopener">Recurrent Neural Networks Tutorial, Part 3 – Backpropagation Through Time and...</a></h3>

<p>This the third&nbsp;part of the Recurrent Neural Network Tutorial. In the previous&nbsp;part of the tutorial we implemented a RNN from scratch, but didn’t go into detail on how Backpropagation Through …</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


elanmart: <blockquote>
<p>Why the hidden variables which represents hidden states of a recurrent neural network is at a risk to get affected during backpropagation and should be detached from previous value?</p>
</blockquote>
<p>As in the diagram below, the error <code>E3</code> depends on hidden state <code>s3</code>, which depends on <code>s2</code>, which depends on <code>s1</code> and so on. At timestep <code>t=100</code>, to compute the gradients we will have to consider last <code>100</code> states.</p>
<p>We don’t want that: we want to assume that for some <code>k</code>, the hidden state <code>s[t-k]</code> is constant and does not depend on anything.</p>
<p>To achieve this we take a batch of <code>k</code> words and simply assume that the initial hidden state for this batch is constant and doesn’t depend on anything.</p>
<p>But for <code>pytorch</code>, if you call <code>hidden = model(batch, hidden)</code>, all of those hidden states are connected. So at the start of each batch you have to manually tell <code>pytorch</code>: “here’s the hidden state from previous batch, but consider it constant”.</p>
<p>I believe you could simply call <code>hidden.detach_()</code> though, no need to create new <code>Variable</code>. (<a class="mention" href="/u/smth">@smth</a>  ?)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/1X/325f00a9303378f974d9b32651d0b626a0591b12.png" data-download-href="https://discuss.pytorch.org/uploads/default/325f00a9303378f974d9b32651d0b626a0591b12" title="rnn-bptt-with-gradients.png"><img src="https://discuss.pytorch.org/uploads/default/original/1X/325f00a9303378f974d9b32651d0b626a0591b12.png" width="690" height="398" data-dominant-color="4B0000"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">rnn-bptt-with-gradients.png</span><span class="informations">953×550 11 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<blockquote>
<p>When the hidden variable should be detached from previous value? At the beginning of each batch?</p>
</blockquote>
<p>Yes.</p>
<p>Image source and a nice blog post about backprop through time: <a href="http://www.wildml.com/2015/10/recurrent-neural-networks-tutorial-part-3-backpropagation-through-time-and-vanishing-gradients/" rel="noopener nofollow ugc">LINK</a></p>
<p><strong>EDIT</strong> Oops, I see <a class="mention" href="/u/chenzhekl">@chenzhekl</a>’s already answered that when I was writing. Sorry for duplicating.</p>

RangoHU: <p>Hello, thanks for the nice answer, it explains a lot! But I still have a question:</p>
<p>Although in the example given by <a class="mention" href="/u/wasiahmad">@wasiahmad</a> the hidden states are detached in train() (using repackeage_hidden() ),  in some examples of nn.lstm(), it’s just simply used as:</p>
<p>def forward(self, input):<br>
outputs, (h_n, c_n) = self.lstm(input)<br>
return outputs</p>
<p>, where input is tensor of sequences (or a packed sequence), and the initial hidden states h_0, c_0 are initialized automatically.</p>
<p>In the example above, the hidden states are not detached manually. I am wondering if the attachment is done automatically here, or BPTT is computed based on all previous hidden states without doing any detachments? Thanks.</p>

apaszke: <p>In your examples new zero-initialized hidden states are initialized at every call to the lstm, and therefore are not connected to the previous sequence in any way. The language modeling example needs to detach them, beacuse it retains the values of hidden states between training states (but doesn’t want to backprop very far back).</p>

RangoHU: <p>Understood. Many thanks.</p>

RangoHU: <p>Hello, I come up with a specific question about the detachment op we discuss here.</p>
<p>For example, we have a seq2seq model with an attention layer between the encoder and the decoder. According to the common implementations of attention models, the last hidden state of the encoder (say, ‘hn’) is used as the first hidden state of the decoder.</p>
<p>My question is: is it necessary to detach hn from the encoder network?</p>

apaszke: <p>It depends what you want to do. If you detach it, the encoder won’t get any gradients from that backward.</p>

ctyuang: <p>Then I suppose detaching the hidden variable from the graph during the evaluation stage is not necessary, or is it?</p>
<pre><code class="lang-auto">def evaluate(data_source):
    # Turn on evaluation mode which disables dropout.
    model.eval()
    total_loss = 0
    ntokens = len(corpus.dictionary)
    hidden = model.init_hidden(eval_batch_size)
    for i in range(0, data_source.size(0) - 1, args.bptt):
        data, targets = get_batch(data_source, i, evaluation=True)
        output, hidden = model(data, hidden)
        output_flat = output.view(-1, ntokens)
        total_loss += len(data) * criterion(output_flat, targets).data
        hidden = repackage_hidden(hidden)
    return total_loss[0] / len(data_source)
</code></pre>
<p>source: <a href="https://github.com/pytorch/examples/blob/master/word_language_model/main.py" rel="nofollow noopener">https://github.com/pytorch/examples/blob/master/word_language_model/main.py</a></p>

adrianjav: <p>During evaluation detaching is not necessary. When you evaluate there is no need to compute the gradients nor backpropagate anything.</p>
<p>So, afaik just put your input variable as volatile and Pytorch won’t hesitate to create the backpropagation graph, it will just do a forward pass.</p>

pp18: <p>hi Adam,</p>
<p>how do we detach the states from the output of GRU layer:</p>
<pre><code>def detach_states(self, states):
	if states is None:
		return states
	return [state.detach() for state in states]
</code></pre>
<p> are output from the GRU layer. When I pass them to</p>
<blockquote>
<p>self.gru(X, initial_states)</p>
</blockquote>
<p>It gives me following error: …</p>
<blockquote>
<p>File “/Users/user_name/anaconda/lib/python3.5/site-packages/torch/nn/modules/rnn.py”, line 162, in check_forward_args<br>
check_hidden_size(hidden, expected_hidden_size)</p>
<p>File “/Users/user_name/anaconda/lib/python3.5/site-packages/torch/nn/modules/rnn.py”, line 153, in check_hidden_size<br>
if tuple(hx.size()) != expected_hidden_size:<br>
AttributeError: ‘list’ object has no attribute ‘size’</p>
</blockquote>
<p>What I am doing wrong. I can put the entire code if this is not clear.</p>

entslscheia: <p>In fact, if you call</p>
<blockquote>
<p>model.init_hidden()</p>
</blockquote>
<p>at the beginning of each batch during training, then there will be no need to call</p>
<blockquote>
<p>hidden.detach()</p>
</blockquote>
<p>.</p>

tastyminerals: <p>I honestly feel a bit disappointed by such ducktaping. Why does a user need to know OR even keep <code>repackage_hidden(h)</code> in his code? This recurrency implementation is specific to pytorch and is just confusing considering that both Tensorflow and even older Torch never expose it.</p>

ptrblck: <p>I would say it depends on your use case and maybe your workflow.<br>
E.g. how would you like to expose the functionality of:</p>
<ul>
<li>backpropagating through <em>all</em> seen data (i.e. in PyTorch just don’t detach the hidden state)</li>
<li>use only the last input batch?</li>
</ul>
<p>New proposals for these use cases (and UX) are always welcome. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

andrei-rusu: <p>Hi there! I would like to know if the first case in your comment is at all possible in PyTorch? I’ve just created a thread on a very similar use case: <a href="https://discuss.pytorch.org/t/how-to-backpropagate-a-loss-through-time-series-rnn/151327" class="inline-onebox">How to backpropagate a loss through time-series RNN?</a>, and I was wondering whether one can actually perform online training with RNNs where the hidden states are never detached (i.e. backpropagating through all seen data) without stumbling upon autograd errors. Thank you!</p>

Messiah: <p>I would also like to know if this can be done, specifically the <a href="https://discuss.pytorch.org/t/how-to-backpropagate-a-loss-through-time-series-rnn/151327">issue</a><br>
mentioned by <a class="mention" href="/u/andrei-rusu">@andrei-rusu</a>.</p>