longformer-large's hidden_states and last_hidden_states have different size in sequence_length
Author: ddf62
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: 
- Python version: 3.7
- PyTorch version (GPU?): GPU
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: @LysandreJik
- T5, BART, Marian, Pegasus, EncoderDecoder: @patrickvonplaten
- Blenderbot, MBART: @patil-suraj
- Longformer, Reformer, TransfoXL, XLNet, FNet, BigBird: @patrickvonplaten
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder, UniSpeech, UniSpeechSAT, SEW, SEW-D, Speech2Text: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:

- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten @narsil
- Tokenizers: @SaulLu
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): Longformer-large

The problem arises when using: student's homework

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [1 ] my own task or dataset: (give details below): NER

## To reproduce

Steps to reproduce the behavior:

1. input the sequence into Longformer-Large
2. oupt the last_states.shape and last_hidden_states[-1].shape
3. find out that them have different length in sequence_length

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
sorry,should be the hidden_states[-1].shape and last_hidden_states.shape

Hi @ddf62 ! Could you post a shot code snippet so we can reproduce this ? Thanks!

```python

class MyModel(tez.Model):
    def __init__(self, freeze_bert=False, model_name='longformer-large', hidden_size=1024, num_classes=2):
        super().__init__()
        config = AutoConfig.from_pretrained(model_name)
        hidden_dropout_prob: float = 0.22
        layer_norm_eps: float = 17589e-7
        config.update(
            {
                "output_hidden_states": True,
                "hidden_dropout_prob": hidden_dropout_prob,
                "layer_norm_eps": layer_norm_eps,
                "add_pooling_layer": False,
            }
        )
        self.automodel = AutoModel.from_config(config)

        self.fc = nn.Sequential(
            nn.Dropout(p=0.5),
            nn.Linear(hidden_size * 4, num_classes, bias=False),
        )

        if freeze_bert:
            for p in self.automodel.parameters():
                p.requires_grad = False

    def forward(self, ids, mask):
        outputs = self.automodel(ids, mask)
        k = ids.shape[1]
        hidden_states = torch.cat(tuple([outputs.hidden_states[i] for i in [-1, -2, -3, -4]]),
                                  dim=-1)  # [bs, seq_len, hidden_dim*4]
        first_hidden_states = hidden_states[:, :k, :]  # [bs, hidden_dim*4]
	print(outputs.hidden_states[-1].shape, outputs.last_hidden_states.shape)
        logits = self.fc(first_hidden_states)
        logits = torch.softmax(logits, dim=-1)
        return logits, 0, {}

```

in forward，’print(outputs.hidden_states[-1].shape, outputs.last_hidden_states.shape)’s results is shown in the picture blow:

The second dim is different in two tensors.

发件人: Suraj Patil
发送时间: 2022年3月7日 18:48
收件人: huggingface/transformers
抄送: ddf62; Mention
主题: Re: [huggingface/transformers] longformer-large's hidden_states andlast_hidden_states have different size in sequence_length (Issue #15962)

Hi @ddf62 ! Could you post a shot code snippet so we can reproduce this ? Thanks!
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android. 
You are receiving this because you were mentioned.Message ID: ***@***.***>



the output is:
 torch.Size([16, 1536, 1024])  torch.Size([16, 1304, 1024])

cc @patrickvonplaten 

@ddf62 - Could you please post a formatted code snippet in:

```python
class MyModel(tez.Model):
```

and one that we can reproduce? What is `tez` ? This issue also doesn't seem to be a core issue of Transformers so maybe the forum https://discuss.huggingface.co/ is the better place?

sorry，you can use this code and their outputs are same：

```py
from transformers import AutoTokenizer, AutoModel, LongformerTokenizerFast
import torch
class MyModel(nn.Module):
    def __init__(self, freeze_bert=False, model_name='longformer-large', hidden_size=1024, num_classes=15):
        super(MyModel, self).__init__()
        self.automodel = AutoModel.from_pretrained(model_name, output_hidden_states=True, return_dict=True)

        self.fc = nn.Sequential(
            nn.Dropout(p=0.5),
            nn.Linear(hidden_size * 4, num_classes, bias=False),
        )

        if freeze_bert:
            for p in self.automodel.parameters():
                p.requires_grad = False

    def forward(self, input_ids, attn_masks):
        outputs = self.automodel(input_ids, attention_mask=attn_masks)

        print(outputs.hidden_states[-1].shape, output.last_hidden_state.shape)
        
        hidden_states = torch.cat(tuple([outputs.hidden_states[i] for i in [-1, -2, -3, -4]]),
                                  dim=-1)  # [bs, seq_len, hidden_dim*4]
        first_hidden_states = hidden_states[:, :, :]  # [bs, hidden_dim*4]
        logits = self.fc(first_hidden_states)
        return logits
```

Gently pinging @ydshieh here since he is becoming our longformer expert. If you have some time it would be amazing if you could take a look here :-)

OK, added to my TODO list :)

Hi @ddf62 

Could you also include the code that you used to create an instance of `MyModel`, prepare the inputs, and pass the inputs to the model in your above code snippet , please? This way, I can run the code to get the output directly and to investigate it.

Thank you.

```
from transformers import AutoTokenizer, AutoModel, LongformerTokenizerFast
import torch
from torch import nn


class MyModel(nn.Module):
    def __init__(self, freeze_bert=False, model_name='longformer-large', hidden_size=1024, num_classes=15):
        super(MyModel, self).__init__()
        self.automodel = AutoModel.from_pretrained(model_name, output_hidden_states=True, return_dict=True)

        self.fc = nn.Sequential(
            nn.Dropout(p=0.5),
            nn.Linear(hidden_size * 4, num_classes, bias=False),
        )

        if freeze_bert:
            for p in self.automodel.parameters():
                p.requires_grad = False

    def forward(self, input_ids, attn_masks):
        outputs = self.automodel(input_ids, attention_mask=attn_masks)

        print(outputs.hidden_states[-1].shape, outputs.last_hidden_state.shape)

        hidden_states = torch.cat(tuple([outputs.hidden_states[i] for i in [-1, -2, -3, -4]]),
                                  dim=-1)  # [bs, seq_len, hidden_dim*4]
        first_hidden_states = hidden_states[:, :, :]  # [bs, hidden_dim*4]
        logits = self.fc(first_hidden_states)
        return logits


model = MyModel(model_name='../model/longformer-large', num_classes=15, freeze_bert=False)
tokenizer = LongformerTokenizerFast.from_pretrained('../model/longformer-large', add_prefix_space=True)
encoding = tokenizer('80% of Americans believe seeking multiple opinions can help them make better choices, and for good reason.'.split(),
                     is_split_into_words=True,
                     # return_offsets_mapping=True,
                     truncation=True)
model(torch.tensor([encoding['input_ids']]), torch.tensor([encoding['attention_mask']]))
```
the output is :torch.Size([1, 512, 1024]) torch.Size([1, 22, 1024])
But i think the shape of hidden_states[-1] should also be [1, 22, 1024].

Hi @ddf62

- Could you run the command `transformers-cli env` and copy-and-paste its output below. We need this information in order to reproduce the issue.
- I could not find any pretrained model with the name `longformer-large`. In your code snippet, you have `'../model/longformer-large'`, which is a local file we don't have. Could you check where does your model comes from?

With `allenai/longformer-large-4096` and a dev version `version: 4.18.0.dev0`, I can't reproduce the issue. The output I got is:

```python
torch.Size([1, 22, 1024]) torch.Size([1, 22, 1024])
```

```
Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.15.0
- Platform: Linux-5.4.0-92-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.11
- PyTorch version (GPU?): 1.10.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: 3090<fill in>
- Using distributed or parallel set-up in script?: <fill in>

```
'../model/longformer-large' is the model i download from [allenai](https://huggingface.co/allenai)
/
[longformer-large-4096](https://huggingface.co/allenai/longformer-large-4096)

> ```
> Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.
> 
> - `transformers` version: 4.15.0
> - Platform: Linux-5.4.0-92-generic-x86_64-with-debian-buster-sid
> - Python version: 3.7.11
> - PyTorch version (GPU?): 1.10.1 (True)
> - Tensorflow version (GPU?): not installed (NA)
> - Flax version (CPU?/GPU?/TPU?): not installed (NA)
> - Jax version: not installed
> - JaxLib version: not installed
> - Using GPU in script?: 3090<fill in>
> - Using distributed or parallel set-up in script?: <fill in>
> ```
> 
> '../model/longformer-large' is the model i download from [allenai](https://huggingface.co/allenai) / [longformer-large-4096](https://huggingface.co/allenai/longformer-large-4096)

Thank you! I am able to reproduce the issue. This issue is already fixed in a pull request #15537 and is included in the latest version [v4.17.0](https://github.com/huggingface/transformers/releases/tag/v4.17.0).

I am going to close this issue, but don't hesitate if you have further question. Thanks!