bert model (bert-base-chinese) consumed too much memory
Author: LiuChiennan
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.6.0
- Platform: Linux-4.19.117.bsk.5-amd64-x86_64-with-debian-10.7
- Python version: 3.7.3
- PyTorch version (GPU?): 1.8.1+cu102 (False)
- Tensorflow version (GPU?): 2.5.0 (False)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
@LysandreJik
Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using Bert:

When I run a code like this: 
```
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained('bert-base-chinese')
bert = AutoModel.from_pretrained('bert-base-chinese')
tokens = tokenizer(query, answer, padding=True, truncation=True, max_length=128, return_tensors="pt")
out = bert(**tokens)
```
where query and answer are both tensors with batch size 128
however, it consumes over 10G memory in this line of code, ```out = bert(**tokens)```, anyone knows why?
and in the next iteration, it consumes 20, 30, 40G memory...


Comments:
A batch size of 128 is a lot! Are you using batch size 128 with sequence length 128?

Most likely you have a large tensor size of 128 * 128 * 768 - and also depends on what type of tensor data you put int32 / float32 / float64? Try to reduce the batch size, even to 2.

> A batch size of 128 is a lot! Are you using batch size 128 with sequence length 128?

yes,  maybe batch size 128 is large, but I don't know why the memory cost becomes larger in each iteration. I mean in the first iteration(with batch size 128) it consumes 10G, and when the process goes to the second iteration(still batch size 128), it consumes 20G, and 30G,40G,.....

> Most likely you have a large tensor size of 128 * 128 * 768 - and also depends on what type of tensor data you put int32 / float32 / float64? Try to reduce the batch size, even to 2.

The input tensor size actually is 128*128? I am just confused why the memory cost rises in each iteration.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.