❓ Questions to google/reformer-enwik8, hidden_state is independent to the context?
Author: shipengAlan
Pull request: no

### Background

Based on char-level model reformer, I want to embed a text into embedding as one of downstream feature.
Like Bert, we usually use the hidden state at text embedding at [CLS] position.
So a natural idea is to use the first/last position's hidden_state as the embedding of the sentence.

But I found the hidden_state of each char is independent to the context.
For example, in the two sentence"Hi! ET" and "Hello", the last hidden_state of "H" is same.


```
def encode(list_of_strings, pad_token_id=0):
    max_length = max([len(string) for string in list_of_strings])

    # create emtpy tensors
    attention_masks = torch.zeros((len(list_of_strings), max_length), dtype=torch.long)
    input_ids = torch.full((len(list_of_strings), max_length), pad_token_id, dtype=torch.long)

    for idx, string in enumerate(list_of_strings):
        # make sure string is in byte format
        if not isinstance(string, bytes):
            string = str.encode(string)

        input_ids[idx, :len(string)] = torch.tensor([x + 2 for x in string])
        attention_masks[idx, :len(string)] = 1

    return input_ids, attention_masks

import torch
from transformers import ReformerModelWithLMHead
model = ReformerModelWithLMHead.from_pretrained("google/reformer-enwik8")
encoded, attention_masks = encode(["Hi! ET", "Hello world"])
out = model.forward(encoded, attention_mask=attention_masks, output_hidden_states=True)

out.hidden_states[-1][1, 0, :] == out.hidden_states[-1][0, 0, :]
... tensor([True, True, True,  ..., True, True, True])
# "H"'s hidden_state is all same under different sentences.
```

### Who can help?

@patrickvonplaten, @ArthurZucker and @younesbelkada

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Coder as above

### Expected behavior

Same char's hidden_state under different sentences is different according to attention layer concept.

Comments:
This depends on whether or not the model uses `bidirectional attention` or `causal` attention. 
This is not general and rather model specific. Usually decoders use `causal` attention while encoder `bidirectional`.  Try setting `is_decoder = False` 😉 

Thank you so much, Arthur @ArthurZucker
As your suggestion, the hidden_state of char become different under different context. 👍

```
import torch
from transformers import ReformerModelWithLMHead,AutoTokenizer,ReformerForSequenceClassification,ReformerModel
from transformers import ReformerConfig, ReformerModel
conf = ReformerConfig.from_pretrained('google/reformer-enwik8')
conf.is_decoder=False
model = ReformerModel.from_pretrained("google/reformer-enwik8", config=conf)
model.eval()

encoded, attention_masks = encode(["Hi", "Hello"])
out = model.forward(encoded, attention_mask=attention_masks, output_hidden_states=True)

out.hidden_states[-1][1, 0, :] == out.hidden_states[-1][0, 0, :]

...tensor([False, False, False,  ..., False, False, False])

encoded
...tensor([[ 74, 107,   0,   0,   0],
        [ 74, 103, 110, 110, 113]])

out.hidden_states[-1]
...tensor([[[ -0.4034,   3.6106,  -8.6159,  ...,  -1.6709, -13.2370,   3.6557],
         [ -3.0774,  15.5188,  -6.5163,  ...,   2.5811, -11.8358,   5.5339],
         [  2.9285,  -7.2961, -17.6565,  ...,   1.8124,  -8.6599,   0.6633],
         [ -5.1294,   1.7743, -18.9366,  ...,   1.2802,  -8.8503,   3.6187],
         [ -4.7954,  -5.2946, -16.2219,  ...,   3.6296,  -8.2144,   4.6336]],

        [[-24.0092,  17.1780, -19.1030,  ...,  18.7862,  17.9711,   6.1703],
         [-16.9048,  13.5032, -18.7212,  ...,   3.4969,  19.2100,   1.7013],
         [ -5.7089,  13.5217,   2.0973,  ...,   1.9111,  12.5311, -18.0838],
         [  7.1809,  25.9611, -15.1986,  ...,  -8.1871,  -4.8213,  -0.1738],
         [  5.5061, -15.3236,  -7.9121,  ...,  11.0452,  11.3647, -12.2207]]])
```

For BERT, we use the first special token [CLS] hidden_state to represent the sentence.
But for this char-level reformer, we don't have a "token" during pre-training and use it to embed the sentence. 
I have no idea that I can use the hidden_state of first char to embedding the sentence or use avg. hidden_state of all chars.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.