[solved] Why we need to detach Variable which contains hidden representation?

wasiahmad: <SCODE># Starting each batch, we detach the hidden state from how it was previously produced.
# If we didn't, the model would try backpropagating all the way to start of the dataset.
hidden = repackage_hidden(hidden)
<ECODE> I am not understanding why we need to detach hidden variable from the hidden variable associated with the previous batch of input? When gradients are computed and loss is backpropagated, weight matrices are usually affected by the chain of computation. Why the hidden variables which represents hidden states of a recurrent neural network is at a risk to get affected during backpropagation and should be detached from previous value? When the hidden variable should be detached from previous value? At the beginning of each batch? or beginning of each training epoch?

chenzhekl: This is for doing truncated BPTT. The dependency graph of RNN can be simply viewed as this. <SCODE>c1, h1 -> c2, h2 -> c3, h3 -> ... -> cn, hn
    |        |         |                |
  loss1    loss2     loss3            lossn
<ECODE> If we did not truncate the history of hidden states (c, h), the back-propagated gradients would flow from the loss towards the beginning, which may result in gradient vanishing or exploding. Detailed explanation can be found here.

elanmart: Yes.

RangoHU: Hello, thanks for the nice answer, it explains a lot! But I still have a question: , where input is tensor of sequences (or a packed sequence), and the initial hidden states h_0, c_0 are initialized automatically. In the example above, the hidden states are not detached manually. I am wondering if the attachment is done automatically here, or BPTT is computed based on all previous hidden states without doing any detachments? Thanks.

apaszke: In your examples new zero-initialized hidden states are initialized at every call to the lstm, and therefore are not connected to the previous sequence in any way. The language modeling example needs to detach them, beacuse it retains the values of hidden states between training states (but doesn’t want to backprop very far back).

RangoHU: Understood. Many thanks.

RangoHU: Hello, I come up with a specific question about the detachment op we discuss here. For example, we have a seq2seq model with an attention layer between the encoder and the decoder. According to the common implementations of attention models, the last hidden state of the encoder (say, ‘hn’) is used as the first hidden state of the decoder. My question is: is it necessary to detach hn from the encoder network?

apaszke: It depends what you want to do. If you detach it, the encoder won’t get any gradients from that backward.

ctyuang: Then I suppose detaching the hidden variable from the graph during the evaluation stage is not necessary, or is it? <SCODE>def evaluate(data_source):
    # Turn on evaluation mode which disables dropout.
    model.eval()
    total_loss = 0
    ntokens = len(corpus.dictionary)
    hidden = model.init_hidden(eval_batch_size)
    for i in range(0, data_source.size(0) - 1, args.bptt):
        data, targets = get_batch(data_source, i, evaluation=True)
        output, hidden = model(data, hidden)
        output_flat = output.view(-1, ntokens)
        total_loss += len(data) * criterion(output_flat, targets).data
        hidden = repackage_hidden(hidden)
    return total_loss[0] / len(data_source)
<ECODE>

adrianjav: During evaluation detaching is not necessary. When you evaluate there is no need to compute the gradients nor backpropagate anything. So, afaik just put your input variable as volatile and Pytorch won’t hesitate to create the backpropagation graph, it will just do a forward pass.

pp18: hi Adam, how do we detach the states from the output of GRU layer: <SCODE>def detach_states(self, states):
	if states is None:
		return states
	return [state.detach() for state in states]
<ECODE>  are output from the GRU layer. When I pass them to It gives me following error: … What I am doing wrong. I can put the entire code if this is not clear.

entslscheia: In fact, if you call at the beginning of each batch during training, then there will be no need to call .

tastyminerals: 

ptrblck: backpropagating through all seen data (i.e. in PyTorch just don’t detach the hidden state) use only the last input batch?

andrei-rusu: 

Messiah: