Token Classification OOM
Author: mirfan899
Pull request: no

I am using the Token Classification Example on my dataset. It has around 20k lines for train and around 2k lines for validation and 2k for the test dataset. I have used the following example for my dataset.

https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/token_classification.ipynb

After training the model, when eval is called it eats up all the GPU memory even on batch size 1.

Here is the output.
```shell
Some weights of the model checkpoint at /home/irfan/Downloads/bert-base-uncased were not used when initializing BertForTokenClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']
- This IS expected if you are initializing BertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForTokenClassification were not initialized from the model checkpoint at /home/irfan/Downloads/bert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Reusing dataset gector (/home/irfan/.cache/huggingface/datasets/gector/gector config/1.0.0/410838f652242763258c1912a4316a2d985c51d27bbb94a3301ad15cde38de06)
100%|██████████| 29/29 [00:01<00:00, 24.00ba/s]
100%|██████████| 7/7 [00:00<00:00, 25.27ba/s]
100%|██████████| 6/6 [00:00<00:00, 25.62ba/s]
  2%|▏         | 500/28347 [00:44<42:04, 11.03it/s]{'loss': 1.3763, 'learning_rate': 1.9647228983666704e-05, 'epoch': 0.02}
  4%|▎         | 1000/28347 [01:27<40:33, 11.24it/s]{'loss': 0.8006, 'learning_rate': 1.9294457967333407e-05, 'epoch': 0.04}
  5%|▌         | 1500/28347 [02:10<37:32, 11.92it/s]{'loss': 0.8302, 'learning_rate': 1.8941686951000106e-05, 'epoch': 0.05}
  7%|▋         | 2000/28347 [02:52<35:53, 12.23it/s]{'loss': 0.7262, 'learning_rate': 1.858891593466681e-05, 'epoch': 0.07}
  9%|▉         | 2500/28347 [03:35<36:11, 11.90it/s]{'loss': 0.7766, 'learning_rate': 1.823614491833351e-05, 'epoch': 0.09}
 11%|█         | 3000/28347 [04:19<37:29, 11.27it/s]{'loss': 0.7891, 'learning_rate': 1.7883373902000214e-05, 'epoch': 0.11}
 12%|█▏        | 3500/28347 [05:03<36:17, 11.41it/s]{'loss': 0.7966, 'learning_rate': 1.7530602885666913e-05, 'epoch': 0.12}
 14%|█▍        | 4000/28347 [05:47<35:28, 11.44it/s]{'loss': 0.7518, 'learning_rate': 1.7177831869333615e-05, 'epoch': 0.14}
 16%|█▌        | 4500/28347 [06:31<34:30, 11.52it/s]{'loss': 0.6913, 'learning_rate': 1.6825060853000318e-05, 'epoch': 0.16}
 18%|█▊        | 5000/28347 [07:15<33:57, 11.46it/s]{'loss': 0.7688, 'learning_rate': 1.647228983666702e-05, 'epoch': 0.18}
 19%|█▉        | 5500/28347 [08:01<37:43, 10.09it/s]{'loss': 0.8129, 'learning_rate': 1.6119518820333723e-05, 'epoch': 0.19}
 21%|██        | 6000/28347 [08:48<33:05, 11.25it/s]{'loss': 0.7222, 'learning_rate': 1.5766747804000426e-05, 'epoch': 0.21}
 23%|██▎       | 6500/28347 [09:37<31:53, 11.42it/s]{'loss': 0.7068, 'learning_rate': 1.5413976787667128e-05, 'epoch': 0.23}
 25%|██▍       | 7000/28347 [10:22<31:18, 11.36it/s]{'loss': 0.6917, 'learning_rate': 1.5061205771333829e-05, 'epoch': 0.25}
 26%|██▋       | 7500/28347 [11:07<30:42, 11.31it/s]{'loss': 0.7271, 'learning_rate': 1.4708434755000532e-05, 'epoch': 0.26}
 28%|██▊       | 8000/28347 [11:52<32:11, 10.53it/s]{'loss': 0.6844, 'learning_rate': 1.435566373866723e-05, 'epoch': 0.28}
 30%|██▉       | 8500/28347 [12:40<30:40, 10.78it/s]{'loss': 0.7325, 'learning_rate': 1.4002892722333933e-05, 'epoch': 0.3}
 32%|███▏      | 9000/28347 [13:27<31:27, 10.25it/s]{'loss': 0.7981, 'learning_rate': 1.3650121706000636e-05, 'epoch': 0.32}
 34%|███▎      | 9500/28347 [14:15<28:28, 11.03it/s]{'loss': 0.7432, 'learning_rate': 1.3297350689667338e-05, 'epoch': 0.34}
 35%|███▌      | 10000/28347 [15:00<26:41, 11.45it/s]{'loss': 0.6548, 'learning_rate': 1.294457967333404e-05, 'epoch': 0.35}
 37%|███▋      | 10500/28347 [15:44<26:11, 11.35it/s]{'loss': 0.7042, 'learning_rate': 1.2591808657000742e-05, 'epoch': 0.37}
 39%|███▉      | 11000/28347 [16:27<25:18, 11.42it/s]{'loss': 0.6836, 'learning_rate': 1.2239037640667444e-05, 'epoch': 0.39}
 41%|████      | 11500/28347 [17:11<24:17, 11.56it/s]{'loss': 0.7011, 'learning_rate': 1.1886266624334147e-05, 'epoch': 0.41}
 42%|████▏     | 12000/28347 [17:55<24:45, 11.01it/s]{'loss': 0.6519, 'learning_rate': 1.153349560800085e-05, 'epoch': 0.42}
 44%|████▍     | 12500/28347 [18:38<22:20, 11.83it/s]{'loss': 0.7423, 'learning_rate': 1.1180724591667549e-05, 'epoch': 0.44}
 46%|████▌     | 13000/28347 [19:22<22:19, 11.46it/s]{'loss': 0.7353, 'learning_rate': 1.0827953575334251e-05, 'epoch': 0.46}
 48%|████▊     | 13500/28347 [20:06<21:15, 11.64it/s]{'loss': 0.6847, 'learning_rate': 1.0475182559000954e-05, 'epoch': 0.48}
 49%|████▉     | 14000/28347 [20:48<20:00, 11.95it/s]{'loss': 0.6356, 'learning_rate': 1.0122411542667656e-05, 'epoch': 0.49}
 51%|█████     | 14500/28347 [21:30<19:01, 12.14it/s]{'loss': 0.6993, 'learning_rate': 9.769640526334357e-06, 'epoch': 0.51}
 53%|█████▎    | 15000/28347 [22:11<18:06, 12.29it/s]{'loss': 0.7461, 'learning_rate': 9.416869510001058e-06, 'epoch': 0.53}
 55%|█████▍    | 15500/28347 [22:53<18:01, 11.88it/s]{'loss': 0.723, 'learning_rate': 9.06409849366776e-06, 'epoch': 0.55}
 56%|█████▋    | 16000/28347 [23:35<17:18, 11.89it/s]{'loss': 0.7091, 'learning_rate': 8.711327477334463e-06, 'epoch': 0.56}
 58%|█████▊    | 16500/28347 [24:16<16:33, 11.93it/s]{'loss': 0.7283, 'learning_rate': 8.358556461001164e-06, 'epoch': 0.58}
 60%|█████▉    | 17000/28347 [24:58<15:44, 12.01it/s]{'loss': 0.6658, 'learning_rate': 8.005785444667867e-06, 'epoch': 0.6}
 62%|██████▏   | 17500/28347 [25:39<15:14, 11.87it/s]{'loss': 0.7333, 'learning_rate': 7.65301442833457e-06, 'epoch': 0.62}
 63%|██████▎   | 18000/28347 [26:21<15:00, 11.50it/s]{'loss': 0.6953, 'learning_rate': 7.300243412001271e-06, 'epoch': 0.63}
 65%|██████▌   | 18500/28347 [27:07<14:39, 11.19it/s]{'loss': 0.6792, 'learning_rate': 6.947472395667973e-06, 'epoch': 0.65}
 67%|██████▋   | 19000/28347 [27:53<15:01, 10.37it/s]{'loss': 0.7156, 'learning_rate': 6.594701379334675e-06, 'epoch': 0.67}
 69%|██████▉   | 19500/28347 [28:41<13:53, 10.62it/s]{'loss': 0.6574, 'learning_rate': 6.241930363001376e-06, 'epoch': 0.69}
 71%|███████   | 20000/28347 [29:28<13:19, 10.44it/s]{'loss': 0.724, 'learning_rate': 5.889159346668079e-06, 'epoch': 0.71}
 72%|███████▏  | 20500/28347 [30:13<11:11, 11.69it/s]{'loss': 0.6687, 'learning_rate': 5.5363883303347795e-06, 'epoch': 0.72}
 74%|███████▍  | 21000/28347 [30:56<10:29, 11.67it/s]{'loss': 0.6612, 'learning_rate': 5.183617314001482e-06, 'epoch': 0.74}
 76%|███████▌  | 21500/28347 [31:39<09:29, 12.03it/s]{'loss': 0.6861, 'learning_rate': 4.830846297668184e-06, 'epoch': 0.76}
 78%|███████▊  | 22000/28347 [32:23<08:31, 12.40it/s]{'loss': 0.6709, 'learning_rate': 4.478075281334886e-06, 'epoch': 0.78}
 79%|███████▉  | 22500/28347 [33:04<08:03, 12.08it/s]{'loss': 0.6689, 'learning_rate': 4.125304265001588e-06, 'epoch': 0.79}
 81%|████████  | 23000/28347 [33:46<07:14, 12.31it/s]{'loss': 0.5955, 'learning_rate': 3.7725332486682897e-06, 'epoch': 0.81}
 83%|████████▎ | 23500/28347 [34:27<06:42, 12.05it/s]{'loss': 0.7265, 'learning_rate': 3.4197622323349914e-06, 'epoch': 0.83}
 85%|████████▍ | 24000/28347 [35:09<06:01, 12.03it/s]{'loss': 0.6603, 'learning_rate': 3.0669912160016936e-06, 'epoch': 0.85}
 86%|████████▋ | 24500/28347 [35:50<05:11, 12.36it/s]{'loss': 0.5817, 'learning_rate': 2.7142201996683953e-06, 'epoch': 0.86}
 88%|████████▊ | 25000/28347 [36:32<04:34, 12.20it/s]{'loss': 0.6164, 'learning_rate': 2.3614491833350974e-06, 'epoch': 0.88}
 90%|████████▉ | 25500/28347 [37:13<03:57, 11.98it/s]{'loss': 0.6458, 'learning_rate': 2.0086781670017996e-06, 'epoch': 0.9}
 92%|█████████▏| 26000/28347 [37:55<03:19, 11.77it/s]{'loss': 0.6093, 'learning_rate': 1.6559071506685013e-06, 'epoch': 0.92}
 93%|█████████▎| 26500/28347 [38:36<02:29, 12.33it/s]{'loss': 0.6713, 'learning_rate': 1.3031361343352032e-06, 'epoch': 0.93}
 95%|█████████▌| 27000/28347 [39:18<01:52, 11.97it/s]{'loss': 0.6891, 'learning_rate': 9.503651180019051e-07, 'epoch': 0.95}
 97%|█████████▋| 27500/28347 [39:59<01:09, 12.19it/s]{'loss': 0.7025, 'learning_rate': 5.975941016686069e-07, 'epoch': 0.97}
 99%|█████████▉| 28000/28347 [40:43<00:29, 11.58it/s]{'loss': 0.6524, 'learning_rate': 2.4482308533530886e-07, 'epoch': 0.99}
100%|██████████| 28347/28347 [41:11<00:00, 11.47it/s]
{'train_runtime': 2471.9347, 'train_samples_per_second': 11.468, 'epoch': 1.0}
 35%|███▍      | 2276/6574 [01:49<06:12, 11.55it/s]Traceback (most recent call last):
  File "/home/irfan/PycharmProjects/GecPytorch/token_classification.py", line 62, in <module>
    trainer.evaluate()
  File "/home/irfan/environments/Allen/lib/python3.6/site-packages/transformers/trainer.py", line 1764, in evaluate
    metric_key_prefix=metric_key_prefix,
  File "/home/irfan/environments/Allen/lib/python3.6/site-packages/transformers/trainer.py", line 1900, in prediction_loop
    preds_host = logits if preds_host is None else nested_concat(preds_host, logits, padding_index=-100)
  File "/home/irfan/environments/Allen/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 98, in nested_concat
    return torch_pad_and_concatenate(tensors, new_tensors, padding_index=padding_index)
  File "/home/irfan/environments/Allen/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 66, in torch_pad_and_concatenate
    result = tensor1.new_full(new_shape, padding_index)
RuntimeError: CUDA out of memory. Tried to allocate 2.12 GiB (GPU 0; 7.80 GiB total capacity; 3.81 GiB already allocated; 2.18 GiB free; 3.97 GiB reserved in total by PyTorch)
 35%|███▍      | 2276/6574 [01:49<03:26, 20.77it/s]
```

I'm using Python 3.6, GPU RTX 2060 super, OS Ubuntu 18.04

@sgugger 

Comments:
You should use `eval_accumulation_steps=n` (for instance 20) to have the predictions be moved to the CPU every n steps  during evaluation (n should be lower than the step you get an OOM error)

Thanks, it worked perfectly.

> You should use `eval_accumulation_steps=n` (for instance 20) to have the predictions be moved to the CPU every n steps during evaluation (n should be lower than the step you get an OOM error)

Thank you so much. I have been stuck at the point for several days until I landed here.