Gloabl attention not recognised in longformer pretrained MLM model to get sentence vector?
Author: pratikchhapolika
Pull request: no

## Objective:
Fetching Sentence embeddings using **longformer** model sentence by sentence from `
` token. By assigning attention_mask[:, [0,-1]] = 2 that is `
 and 
 token will have values 2`. 

- `transformers` **version:3.0.2**
- Platform:
- Python version: **Python 3.6.12 :: Anaconda, Inc.**
- PyTorch version (GPU?):**1.7.1**
- Tensorflow version (GPU?): **2.3.0**
- Using GPU in script?: **Yes**
- Using distributed or parallel set-up in script?: **parallel**

### Who can help
@patrickvonplaten

##Models:

- longformer, reformer, transfoxl, xlnet: @patrickvonplaten

## Information

Model I am using longformer:

The problem arises when using:
* my own modified scripts: (give details below)

The tasks I am working on is:
* my own task or dataset: (give details below)


## Code:
```
from transformers import LongformerModel, LongformerTokenizer
model = LongformerModel.from_pretrained('allenai/longformer-base-4096',output_hidden_states = True)
tokenizer = LongformerTokenizer.from_pretrained('allenai/longformer-base-4096')

# Put the model in "evaluation" mode, meaning feed-forward operation.
model.eval()
```


```
text=["I like to play cricket"] #For this code I want to fetch embedding.

def sentence_bert():

    list_of_emb=[]

    for i in range(len(all_content)):

        SAMPLE_TEXT = text[i]  # long input document

        print(tokenizer.encode(SAMPLE_TEXT,padding=True,add_special_tokens=True,max_length=20)) #,max_length=10
        print(tokenizer.decode(tokenizer.encode(SAMPLE_TEXT)))

        input_ids = torch.tensor(tokenizer.encode(SAMPLE_TEXT)).unsqueeze(0)  

    
        attention_mask = torch.ones(input_ids.shape, dtype=torch.long, device=input_ids.device) # initialize to local attention
        **attention_mask[:, [0,-1]] = 2**
        
         with torch.no_grad():

            outputs = model(input_ids, attention_mask=attention_mask)
            hidden_states = outputs[2]
            token_embeddings = torch.stack(hidden_states, dim=0)
            # Remove dimension 1, the "batches".
            token_embeddings = torch.squeeze(token_embeddings, dim=1)
            # Swap dimensions 0 and 1.
            token_embeddings = token_embeddings.permute(1,0,2)

            token_vecs_sum = []
            # For each token in the sentence...
            for token in token_embeddings:

            #but preferrable is
               sum_vec=torch.sum(token[-4:],dim=0)

            # Use `sum_vec` to represent `token`.
               token_vecs_sum.append(sum_vec)

           
           h=0
           for i in  range(len(token_vecs_sum)):
              h+=token_vecs_sum[i]
           list_of_emb.append(h)

    return list_of_emb

f=sentence_bert()

```

**Output**

```
length of string:   5
[0, 38, 101, 7, 310, 5630, 2]
`
` I like to play cricket `
`
input_ids:   tensor([[   0,   38,  101,    7,  310, 5630,    2]])

Number of layers: 13   (initial embeddings + 12 BERT layers)
Number of batches: 1
Number of tokens: 512
Number of hidden units: 768

```
## Doubts/Question:

1. When I pass `attention_mask[:, [0,-1]] = 2` for global attention to `
` token. It doesn't seems to work.  Then I pull `0th` token from `512` token from last layer as sentence embedding. Makes sense?
2. Even after passing `max_length=20` I see tensor of size=sentence_length, however ideally it should be padded with max size?
3. Why I see `Number of tokens: 512` ?  I think it should be based on our `sentence length`. When I pass one sentence of length `7 ` to get embedding the token in hidden state should be 7? Based on my sentence what are 512 tokens?? 
4. How can I reduce number of tokens to sentence length instead of 512 ? Every-time I input a new sentence, it should pick up that length. Can we do this for `longformer `?




## Expected behavior

Document1:  Embeddings
Document2: Embeddings



Comments:
Hey @pratikchhapolika,

Instead of setting values in attention_mask to 2 could you try using global_attention_mask instead? 

Also see official docs here: 
https://huggingface.co/transformers/master/model_doc/longformer.html#transformers.LongformerModel.forward

> Hey @pratikchhapolika,
> 
> Instead of setting values in attention_mask to 2 could you try using global_attention_mask instead?
> 
> Also see official docs here:
> https://huggingface.co/transformers/master/model_doc/longformer.html#transformers.LongformerModel.forward

You mean to say I should use this:

```python
global_attention_mask = torch.ones(input_ids.shape, dtype=torch.long, device=input_ids.device) # initialize to local attention
global_attention_mask[:, [0,-1]] = 2
outputs = model(input_ids, global_attention_mask =global_attention_mask)
```


Rather:

```python
global_attention_mask = torch.zeros(input_ids.shape, dtype=torch.long, device=input_ids.device) # initialize to local attention
global_attention_mask[:, [0,-1]] = 1
outputs = model(input_ids, global_attention_mask =global_attention_mask)
```

as shown in the example of [this](https://huggingface.co/transformers/master/model_doc/longformer.html#transformers.LongformerModel.forward) method ;-)


> Rather:
> 
> ```python
> global_attention_mask = torch.zeros(input_ids.shape, dtype=torch.long, device=input_ids.device) # initialize to local attention
> global_attention_mask[:, [0,-1]] = 1
> outputs = model(input_ids, global_attention_mask =global_attention_mask)
> ```
> 
> as shown in the example of [this](https://huggingface.co/transformers/master/model_doc/longformer.html#transformers.LongformerModel.forward) method ;-)

Can you pleas help me wit other questions as well and then I will close this issue? `Question 2, 3 and 4`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.