Insufficient memory occurs during finetune
Author: Henryplay
Pull request: no
Labels: name: Migration

# 📚 Migration

## Information

<!-- Important information -->

Model I am using (Bert,bert-base-chinese):

Language I am using the model on (Chinese ...):

The problem arises when using:
* [x] the official example scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## Details
* Here is my bash
```
export CUDA_VISIBLE_DEVICES=1,2
python run_mlm.py \
    --model_name_or_path bert-base-chinese \
    --train_file /home/xjyu/zhh/workspace/one-stop-ocr/char2char.allchars/quantize-train/data/scripts/wiki-part/wiki-part.split.nosign.txt \
    --validation_file /home/xjyu/zhh/workspace/one-stop-ocr/char2char.allchars/quantize-train/data/scripts/wiki-part/test.txt \
    --do_train \
    --do_eval \
    --output_dir ./test-mlm \
    --line_by_line \
```

When I performed finetune on bert->`run_mlm.py`, it took about 0.6 epoch，the output_dir file "test-mlm" took up 91G RAM，I am really surprised. It stands to reason that the output file cannot be so big，But i don't know where the problem is.
* here is my machine
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64       Driver Version: 440.64       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 00000000:02:00.0 Off |                  N/A |
|  0%   32C    P8     9W / 250W |      0MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 108...  Off  | 00000000:03:00.0 Off |                  N/A |
|  0%   28C    P8     9W / 250W |      0MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX 108...  Off  | 00000000:83:00.0 Off |                  N/A |
| 77%   80C    P2    73W / 250W |   4865MiB / 11177MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX 108...  Off  | 00000000:84:00.0 Off |                  N/A |
|  0%   65C    P2    68W / 250W |   5574MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
<!-- A clear and concise description of the migration issue.
    If you have code snippets, please provide it here as well.
    Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
    Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.
    -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
 
- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

<!-- IMPORTANT: which version of the former library do you use? -->
* `pytorch-transformers` or `pytorch-pretrained-bert` version (or branch):


## Checklist

- [x] I have read the migration guide in the readme.
 ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers);
  [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
- [x] I checked if a related official extension example runs on my machine.

@sgugger 

Comments:
The environment is the same as yours

Please use the [forums](https://discuss.huggingface.co/) for those kinds of questions. You should limit the number of saves of the checkpoints by using the argument `--save_total_limit` for instance, if the defaults get you out of disk memory.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.