Repository: pytorch/pytorch

Issue:
Documentation: Indexing output from bidirectional RNN (GRU,LSTM)
The documentation for RNNs (including GRU and LSTM) states the dimensionality of hidden state (num_layers * num_directions, batch, hidden_size) and output (seq_len, batch, hidden_size * num_direction), but I cannot figure out how to index the output to get separate vectors for the two directions. Specifically, I'd like to get the output from the last cell on the forward RNN, and the output from the first cell in the backward RNN.

From looking at the source code, you can infer that direction=0 is forward, and direction=1 is backward. I suppose that the "hidden_size * num_direction" dimension contains one forward-vector concatenated with one backward-vector, but I cannot find this anywhere in the documentation. Is this correct, and could this be specified in the documentation?


Discussion:
From what I understand of the CuDNN API, which is the basis of pytorch's one, the output is sorted by timesteps, so `h_n` should be the concatenation of the hidden state of the forward layer for the last item of the sequence and of the hidden state of the backward layer for the first item of the sequence.

I agree that it should be more explicit. Also it is really inconvenient for e.g. bidirectional attention, but I guess there is no way around it if you want to be able to use CuDNN calls.
Hi, thank you. I think this is the most reasonable behavior. I just think it should be clarified in the docs. I think it sounds like what you'd like also for attention. Why would this be inconvenient? 
Because it makes this topology

![image](https://user-images.githubusercontent.com/1656541/32880955-d05c9dc2-caaf-11e7-8fde-e85d21b0bdaa.png)

impossible to get with a bidirectional LSTM. To get per-word (or token, or whatever) hidden states instead of per-timestep, you have to run forward and backward as separate layers and concatenate the outputs afterwards. Add to that that pytorch (as far as I know) supports neither backward-only LSTM nor flipping tensors, this adds some complexity to my code, and I suspect is not so good for performance either.
From what I've understand so far about non `Cell` suffixed CuDNN variants:

1. You need to give them a tensor where the sequences are sorted by length in a decreasing order. If the sequences are guaranteed to always have the same length, you can skip this step. Otherwise, cf. `pack_padded_sequence()`
2. After calling the RNN, you receive a tuple of 2 items: **packed** `Variable` of all hidden states (`hs`) and a normal `Variable` containing the last hidden states (`ht`).
3. For `hs` you unpack it using `pad_packed_sequence()` to get a normal `Variable`.
4. `ht` contains the **correct** forward and backward states for each sequence so you don't have to do something to recover or mask out 0's etc **but** this tensor does not concatenate the forward-backward states although `hs` returns them in a concatenated fashion.

```python
# An input of 5 timesteps and 2 sequences. The shorter one is 0-padded.
# hidden_dim = 3, bidirectional=True, num_layers=1
In [525]: input_
Out[525]:

 1  3
 3  5
 3  2
 2  0
 1  0
[torch.LongTensor of size 5x2]

# hs and ht are the return values of GRU here (for LSTM you'll also have c_t)
In [526]: print(hs[:, 1], ht[:, 1])

Variable containing:
   (( forward states ))                      (( backward states ))
-0.0982  0.0275 -0.3005            0.3609 -0.4958  0.3408
-0.1710 -0.0576 -0.3759            0.2550 -0.3478  0.2796
-0.1935  0.0484 -0.4111            0.2088 -0.2813  0.1440
 0.0000  0.0000  0.0000            0.0000  0.0000  0.0000
 0.0000  0.0000  0.0000            0.0000  0.0000  0.0000
[torch.FloatTensor of size 5x6]

 Variable containing:
-0.1935  0.0484 -0.4111
 0.3609 -0.4958  0.3408
[torch.FloatTensor of size 2x3]
```
Here you can see that the last state for the forward sequence (3->5->2) is the third row's first 3 elements `-0.1935  0.0484 -0.4111` that you also find in the `ht` variable in the first row.

The last state for the backward sequence (2->5->3) is the first row's second part `0.3609 -0.4958  0.3408` that you also find in the `ht`variable in the second row.

So if you want to apply attention the first tensor is the one that you'll need. If you want to just take the last states, second tensor is at your help.

But if `num_layers > 1`, the second tensor becomes a mess :) Overall, I think this part of the PyTorch API really needs more intuitive handling. 

```
ipt = Variable(torch.from_numpy(np.asarray([0., 0, 1, 1.]).reshape(-1, 1, 1).astype(np.float32)))
h0 = Variable(torch.zeros(2, 1,1 ))
rnn = nn.RNN(1,1, 1, bidirectional=True, bias=False, nonlinearity='relu')
for k, v  in rnn.named_parameters():
   setattr(rnn, k, torch.nn.Parameter(torch.ones_like(v.data)))
opt, hn = rnn(ipt, h0)
print list(rnn.named_parameters())
print opt
print ipt
```
the output I got: 
```
[('weight_ih_l0', Parameter containing:
 1
[torch.FloatTensor of size 1x1]
), ('weight_hh_l0', Parameter containing:
 1
[torch.FloatTensor of size 1x1]
), ('weight_ih_l0_reverse', Parameter containing:
 1
[torch.FloatTensor of size 1x1]
), ('weight_hh_l0_reverse', Parameter containing:
 1
[torch.FloatTensor of size 1x1]
)]
Variable containing:
(0 ,.,.) =
  0  2

(1 ,.,.) =
  0  2

(2 ,.,.) =
  1  2

(3 ,.,.) =
  2  1
[torch.FloatTensor of size 4x1x2]
Variable containing:
(0 ,.,.) =
  0

(1 ,.,.) =
  0

(2 ,.,.) =
  1

(3 ,.,.) =
  1
[torch.FloatTensor of size 4x1x1]
```
to be more intuitive, I arrange the result:
```
--forward     0   ->    0   ->   1   ->     2
backward      2   <-     2   <-   2    <-  1
----input      0   --    0    --   1     --  1
wordindex    0   --    1    --   2    --  3
hn:              [forward 2, backward 2]
```
After this experiment:
1. forward and backward are computed independently
2. after running through f/b ward, it returns something like `torch.cat([forward, backward], dim=-1)`
3. the hidden it return is actually  hidden of  word 3 in forward pass, and hidden of word 0 in backward pass.
4. therefore it is not a good idea to use rnn cell and run a for loop by yourself if you set bidirectional=True.

I think these behavior should be clarify somewhere in the official doc, which unfortunately I did not find yet.
@apaszke can you consider about adding it? Or this is something beyond pytorch but go depth to CuDNN?
> Or this is something beyond pytorch but go depth to CuDNN?

That is what I understand from the CuDNN API reference (which is not that clear either…)
Is it really true that "`h_n` should be the concatenation of the hidden state of the forward layer for the last item of the sequence and of the hidden state of the backward layer for the first item of the sequence"? There's an issue in cudnn for torch (soumith/cudnn.torch#357) that says otherwise. @ngimel mentioned there that for an example sequence [x_1, x_2, x_3, x_4, x_5], the output sequence would be:

```
[F(x_1), B(x_1)]
[F(x_2), B(x_2)]
[F(x_3), B(x_3)]
[F(x_4), B(x_4)]
[F(x_5), B(x_5)]
```

The RNN code calling cuDNN in [torch](https://github.com/soumith/cudnn.torch/blob/master/RNN.lua) and [pytorch](https://github.com/pytorch/pytorch/blob/master/torch/backends/cudnn/rnn.py) do not seem to perform any reversing of the output of `cudnnRNNForwardTraining` call so its weird that the order would be different in pytorch and torch.
 @sidharthms 
lstm return two things `output, hn = lstm([x_1, x_2, x_3, x_4, x_5])`
what you point out is `output`
```
[F(x_1), B(x_1)]
[F(x_2), B(x_2)]
[F(x_3), B(x_3)]
[F(x_4), B(x_4)]
[F(x_5), B(x_5)]
```
which is the same as I said, 
As regard to `hn`, it is `[F(x_5), B(x_1)]` which is the last thing computed in forward pass and the last thing computed in backward pass.

Resolving patch:
diff --git a/torch/nn/modules/rnn.py b/torch/nn/modules/rnn.py
--- a/torch/nn/modules/rnn.py
+++ b/torch/nn/modules/rnn.py
@@ -278,13 +278,21 @@ class RNN(RNNBase):
           Defaults to zero if not provided.
 
     Outputs: output, h_n
-        - **output** of shape `(seq_len, batch, hidden_size * num_directions)`: tensor
+        - **output** of shape `(seq_len, batch, num_directions * hidden_size)`: tensor
           containing the output features (`h_k`) from the last layer of the RNN,
           for each `k`.  If a :class:`torch.nn.utils.rnn.PackedSequence` has
           been given as the input, the output will also be a packed sequence.
+
+          For the unpacked case, the directions can be separated
+          using ``output.view(seq_len, batch, num_directions, hidden_size)``,
+          with forward and backward being direction `0` and `1` respectively.
+          Similarly, the directions can be separated in the packed case.
         - **h_n** (num_layers * num_directions, batch, hidden_size): tensor
           containing the hidden state for `k = seq_len`.
 
+          Like *output*, the layers can be separated using
+          ``h_n.view(num_layers, num_directions, batch, hidden_size)``.
+
     Attributes:
         weight_ih_l[k]: the learnable input-hidden weights of the k-th layer,
             of shape `(hidden_size * input_size)` for `k = 0`. Otherwise, the shape is
@@ -377,12 +385,20 @@ class LSTM(RNNBase):
 
 
     Outputs: output, (h_n, c_n)
-        - **output** of shape `(seq_len, batch, hidden_size * num_directions)`: tensor
+        - **output** of shape `(seq_len, batch, num_directions * hidden_size)`: tensor
           containing the output features `(h_t)` from the last layer of the LSTM,
           for each t. If a :class:`torch.nn.utils.rnn.PackedSequence` has been
           given as the input, the output will also be a packed sequence.
+
+          For the unpacked case, the directions can be separated
+          using ``output.view(seq_len, batch, num_directions, hidden_size)``,
+          with forward and backward being direction `0` and `1` respectively.
+          Similarly, the directions can be separated in the packed case.
         - **h_n** of shape `(num_layers * num_directions, batch, hidden_size)`: tensor
-          containing the hidden state for `t = seq_len`
+          containing the hidden state for `t = seq_len`.
+
+          Like *output*, the layers can be separated using
+          ``h_n.view(num_layers, num_directions, batch, hidden_size)`` and similarly for *c_n*.
         - **c_n** (num_layers * num_directions, batch, hidden_size): tensor
           containing the cell state for `t = seq_len`
 
@@ -457,13 +473,21 @@ class GRU(RNNBase):
           Defaults to zero if not provided.
 
     Outputs: output, h_n
-        - **output** of shape `(seq_len, batch, hidden_size * num_directions)`: tensor
+        - **output** of shape `(seq_len, batch, num_directions * hidden_size)`: tensor
           containing the output features h_t from the last layer of the GRU,
           for each t. If a :class:`torch.nn.utils.rnn.PackedSequence` has been
           given as the input, the output will also be a packed sequence.
+          For the unpacked case, the directions can be separated
+          using ``output.view(seq_len, batch, num_directions, hidden_size)``,
+          with forward and backward being direction `0` and `1` respectively.
+
+          Similarly, the directions can be separated in the packed case.
         - **h_n** of shape `(num_layers * num_directions, batch, hidden_size)`: tensor
           containing the hidden state for `t = seq_len`
 
+          Like *output*, the layers can be separated using
+          ``h_n.view(num_layers, num_directions, batch, hidden_size)``.
+
     Attributes:
         weight_ih_l[k] : the learnable input-hidden weights of the :math:`\text{k}^{th}` layer
             (W_ir|W_iz|W_in), of shape `(3*hidden_size x input_size)`


Test patch: