Repository: huggingface/transformers

Issue:
RuntimeError: Overflow when unpacking long
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.0.dev0
- Platform: linux
- Python version: 3.6.10
- PyTorch version (GPU?): 1.7.0a0.     (gpu)
- Using GPU in script?: yes 
- Using distributed or parallel set-up in script?: distributed, 4 nodes with 4 GPU's each


Models:

- albert, bert, xlm: @LysandreJik
running language modelling on a large 335 million token sequences

Library:

- trainer: @sgugger
- Fairscale




Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ x ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. I am running run_mlm with just small changes as my datasets are already tokenized 
2. Getting an error while saving checkpoint


Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    train_result = trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 924, in train
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1026, in _save_checkpoint
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self.state.total_flos = distributed_broadcast_scalars([self._total_flos]).sum().item()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 140, in distributed_broadcast_scalars
    self.state.total_flos = distributed_broadcast_scalars([self._total_flos]).sum().item()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 140, in distributed_broadcast_scalars
    self.state.total_flos = distributed_broadcast_scalars([self._total_flos]).sum().item()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 140, in distributed_broadcast_scalars
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self.state.total_flos = distributed_broadcast_scalars([self._total_flos]).sum().item()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 140, in distributed_broadcast_scalars
    self.state.total_flos = distributed_broadcast_scalars([self._total_flos]).sum().item()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 140, in distributed_broadcast_scalars
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self.state.total_flos = distributed_broadcast_scalars([self._total_flos]).sum().item()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 140, in distributed_broadcast_scalars
    self.state.total_flos = distributed_broadcast_scalars([self._total_flos]).sum().item()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 140, in distributed_broadcast_scalars
    self.store_flos()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1361, in store_flos
    self.state.total_flos = distributed_broadcast_scalars([self._total_flos]).sum().item()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 140, in distributed_broadcast_scalars
    self.state.total_flos = distributed_broadcast_scalars([self._total_flos]).sum().item()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 140, in distributed_broadcast_scalars
    self.state.total_flos = distributed_broadcast_scalars([self._total_flos]).sum().item()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 140, in distributed_broadcast_scalars
    self.state.total_flos = distributed_broadcast_scalars([self._total_flos]).sum().item()
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 140, in distributed_broadcast_scalars
    tensorized_scalar = torch.tensor(scalars).cuda()
    tensorized_scalar = torch.tensor(scalars).cuda()
    tensorized_scalar = torch.tensor(scalars).cuda()
    tensorized_scalar = torch.tensor(scalars).cuda()
RuntimeError: Overflow when unpacking long
RuntimeError: Overflow when unpacking long
RuntimeError: Overflow when unpacking long
RuntimeError: Overflow when unpacking long


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

Saving the checkpoint normally. It occurs only at some checkpoints randomly!
<!-- A clear and concise description of what you would expect to happen. -->



Discussion:
This comes from the `_total_flos` being stored as long and overflowing in a big training. Will fix this by storing them as floats (hoping for a PR by the end of today).

Resolving patch:
diff --git a/src/transformers/trainer.py b/src/transformers/trainer.py
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -959,7 +959,7 @@ def train(
                         tr_loss += self.training_step(model, inputs)
                 else:
                     tr_loss += self.training_step(model, inputs)
-                self._total_flos += self.floating_point_ops(inputs)
+                self._total_flos += float(self.floating_point_ops(inputs))
 
                 if (step + 1) % self.args.gradient_accumulation_steps == 0 or (
                     # last step in epoch but step is always smaller than gradient_accumulation_steps
diff --git a/src/transformers/trainer_callback.py b/src/transformers/trainer_callback.py
--- a/src/transformers/trainer_callback.py
+++ b/src/transformers/trainer_callback.py
@@ -52,8 +52,9 @@ class TrainerState:
             During training, represents the number of update steps completed.
         max_steps (:obj:`int`, `optional`, defaults to 0):
             The number of update steps to do during the current training.
-        total_flos (:obj:`int`, `optional`, defaults to 0):
-            The total number of floating operations done by the model since the beginning of training.
+        total_flos (:obj:`float`, `optional`, defaults to 0):
+            The total number of floating operations done by the model since the beginning of training (stored as floats
+            to avoid overflow).
         log_history (:obj:`List[Dict[str, float]]`, `optional`):
             The list of logs done since the beginning of training.
         best_metric (:obj:`float`, `optional`):
@@ -76,7 +77,7 @@ class TrainerState:
     global_step: int = 0
     max_steps: int = 0
     num_train_epochs: int = 0
-    total_flos: int = 0
+    total_flos: float = 0
     log_history: List[Dict[str, float]] = None
     best_metric: Optional[float] = None
     best_model_checkpoint: Optional[str] = None


Test patch:
diff --git a/tests/test_trainer.py b/tests/test_trainer.py
--- a/tests/test_trainer.py
+++ b/tests/test_trainer.py
@@ -881,6 +881,9 @@ def assert_flos_extraction(trainer, wrapped_model_to_check):
         # with enforced DataParallel
         assert_flos_extraction(trainer, torch.nn.DataParallel(trainer.model))
 
+        trainer.train()
+        self.assertTrue(isinstance(trainer.state.total_flos, float))
+
 
 @require_torch
 @require_optuna