RuntimeError: Overflow when unpacking long
Author: manchandasahil
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.0.dev0
- Platform: linux
- Python version: 3.6.10
- PyTorch version (GPU?): 1.7.0a0.     (gpu)
- Using GPU in script?: yes 
- Using distributed or parallel set-up in script?: distributed, 4 nodes with 4 GPU's each


Models:

- albert, bert, xlm: @LysandreJik
running language modelling on a large 335 million token sequences

Library:

- trainer: @sgugger
- Fairscale




Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ x ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. I am running run_mlm with just small changes as my datasets are already tokenized 
2. Getting an error while saving checkpoint


Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self.state.total_flos = distributed_broadcast_scalars([self._total_flos]).sum().item()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 140, in distributed_broadcast_scalars
    self.state.total_flos = distributed_broadcast_scalars([self._total_flos]).sum().item()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 140, in distributed_broadcast_scalars
    self.state.total_flos = distributed_broadcast_scalars([self._total_flos]).sum().item()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 140, in distributed_broadcast_scalars
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self.state.total_flos = distributed_broadcast_scalars([self._total_flos]).sum().item()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 140, in distributed_broadcast_scalars
    self.state.total_flos = distributed_broadcast_scalars([self._total_flos]).sum().item()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 140, in distributed_broadcast_scalars
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self.state.total_flos = distributed_broadcast_scalars([self._total_flos]).sum().item()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 140, in distributed_broadcast_scalars
    self.state.total_flos = distributed_broadcast_scalars([self._total_flos]).sum().item()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 140, in distributed_broadcast_scalars
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self.state.total_flos = distributed_broadcast_scalars([self._total_flos]).sum().item()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 140, in distributed_broadcast_scalars
    self.state.total_flos = distributed_broadcast_scalars([self._total_flos]).sum().item()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 140, in distributed_broadcast_scalars
    self.state.total_flos = distributed_broadcast_scalars([self._total_flos]).sum().item()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 140, in distributed_broadcast_scalars
    self.state.total_flos = distributed_broadcast_scalars([self._total_flos]).sum().item()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 140, in distributed_broadcast_scalars
    tensorized_scalar = torch.tensor(scalars).cuda()
    tensorized_scalar = torch.tensor(scalars).cuda()
    tensorized_scalar = torch.tensor(scalars).cuda()
    tensorized_scalar = torch.tensor(scalars).cuda()
RuntimeError: Overflow when unpacking long
RuntimeError: Overflow when unpacking long
RuntimeError: Overflow when unpacking long
RuntimeError: Overflow when unpacking long


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

Saving the checkpoint normally. It occurs only at some checkpoints randomly!
<!-- A clear and concise description of what you would expect to happen. -->


Comments:
This comes from the `_total_flos` being stored as long and overflowing in a big training. Will fix this by storing them as floats (hoping for a PR by the end of today).