finetune_trainer segfault 
Author: rabeehkarimimahabadi
Pull request: no

Hi
I wand to run finetune_trainer on tpu:

export TPU_IB_ADDRESS=10.160.244.2

start=`date +%s`
python xla_spawn.py --num_cores 8 \
    finetune_trainer.py \
    --tokenizer_name t5-base --model_name_or_path t5-base \
    --data_dir /home/rabeeh/ruse/seq2seq/seq2seq/data \
    --output_dir /home/rabeeh/outputs/verify --overwrite_output_dir \
    --learning_rate=3e-4 \
    --warmup_steps 500 \
    --per_device_train_batch_size=256 --per_device_eval_batch_size=256\
    --num_train_epochs=6 \
    --save_steps 500 --eval_steps 500 \
    --logging_steps 200 \
    --max_source_length 128 --max_target_length 128 --val_max_target_length 128 --test_max_target_length 128 \
    --task translation --label_smoothing 0.1 \
    "$@"

end=`date +%s`
runtime=$((end-start))
echo running time $runtime

here are the logs, thank for your help/ 

2020-11-06 09:05:12.060270: E    1335 tensorflow/compiler/xla/xla_client/xla_util.cc:76] 	 [[XRTExecute_G12]]
2020-11-06 09:05:12.060282: E    1335 tensorflow/compiler/xla/xla_client/xla_util.cc:76] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
2020-11-06 09:05:12.060292: E    1335 tensorflow/compiler/xla/xla_client/xla_util.cc:76] 
2020-11-06 09:05:12.060303: E    1335 tensorflow/compiler/xla/xla_client/xla_util.cc:76]   (1) Resource exhausted: Attempting to reserve 7.10G at the bottom of memory. That was not possible. There are 6.88G free, 0B reserved, and 6.88G reservable.
2020-11-06 09:05:12.060314: E    1335 tensorflow/compiler/xla/xla_client/xla_util.cc:76] 	 [[{{node XRTExecute}}]]
2020-11-06 09:05:12.060325: E    1335 tensorflow/compiler/xla/xla_client/xla_util.cc:76] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
2020-11-06 09:05:12.060336: E    1335 tensorflow/compiler/xla/xla_client/xla_util.cc:76] 
2020-11-06 09:05:12.060347: E    1335 tensorflow/compiler/xla/xla_client/xla_util.cc:76] 0 successful operations.
2020-11-06 09:05:12.060358: E    1335 tensorflow/compiler/xla/xla_client/xla_util.cc:76] 0 derived errors ignored.
###### Arguments  (ModelArguments(model_name_or_path='t5-base', config_name=None, tokenizer_name='t5-base', cache_dir=None, freeze_encoder=True, freeze_embeds=True), DataTrainingArguments(data_dir='data/wmt_en_de', task='translation', max_source_length=128, max_target_length=128, val_max_target_length=128, test_max_target_length=128, n_train=-1, n_val=-1, n_test=-1, src_lang=None, tgt_lang=None, eval_beams=None, ignore_pad_token_for_loss=True), Seq2SeqTrainingArguments(output_dir='/home/rabeeh/outputs/verify', overwrite_output_dir=True, do_train=True, do_eval=True, do_predict=False, evaluate_during_training=True, evaluation_strategy=<EvaluationStrategy.STEPS: 'steps'>, prediction_loss_only=True, per_device_train_batch_size=64, per_device_eval_batch_size=64, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, eval_accumulation_steps=None, learning_rate=0.0003, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=6.0, max_steps=-1, warmup_steps=500, logging_dir='runs/Nov06_09-03-01_eee96b6103a2', logging_first_step=True, logging_steps=200, save_steps=500, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, tpu_num_cores=8, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=500, dataloader_num_workers=0, past_index=-1, run_name='/home/rabeeh/outputs/verify', disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=None, label_smoothing=0.1, sortish_sampler=False, predict_with_generate=False, adafactor=False, encoder_layerdrop=None, decoder_layerdrop=None, dropout=None, attention_dropout=None, lr_scheduler='linear'))
######### data_file  data/wmt_en_de/train.source
######### data_file  data/wmt_en_de/val.source
{'loss': 10624.0, 'learning_rate': 6e-07, 'epoch': 0.00011253657438667567}
Exception in device=TPU:0: Resource exhausted: From /job:tpu_worker/replica:0/task:0:
2 root error(s) found.
  (0) Resource exhausted: Attempting to reserve 7.10G at the bottom of memory. That was not possible. There are 6.88G free, 0B reserved, and 6.88G reservable.
	 [[{{node XRTExecute}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

	 [[XRTExecute_G12]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted: Attempting to reserve 7.10G at the bottom of memory. That was not possible. There are 6.88G free, 0B reserved, and 6.88G reservable.
	 [[{{node XRTExecute}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.
Traceback (most recent call last):
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 330, in _mp_start_fn
    _start_fn(index, pf_cfg, fn, args)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 324, in _start_fn
    fn(gindex, *args)
  File "/root/home/rabeeh/seq2seq/finetune_trainer.py", line 330, in _mp_fn
    app.run(main, flags_parser=parse_flags)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "/root/home/rabeeh/seq2seq/finetune_trainer.py", line 279, in main
    model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/trainer.py", line 757, in train
    for step, inputs in enumerate(epoch_iterator):
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/parallel_loader.py", line 31, in __next__
    return self.next()
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/parallel_loader.py", line 37, in next
    xm.mark_step()
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/core/xla_model.py", line 716, in mark_step
    wait=xu.getenv_as('XLA_SYNC_WAIT', bool, False))
RuntimeError: Resource exhausted: From /job:tpu_worker/replica:0/task:0:
2 root error(s) found.
  (0) Resource exhausted: Attempting to reserve 7.10G at the bottom of memory. That was not possible. There are 6.88G free, 0B reserved, and 6.88G reservable.
	 [[{{node XRTExecute}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

	 [[XRTExecute_G12]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted: Attempting to reserve 7.10G at the bottom of memory. That was not possible. There are 6.88G free, 0B reserved, and 6.88G reservable.
	 [[{{node XRTExecute}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.
Traceback (most recent call last):
  File "xla_spawn.py", line 77, in <module>
    app.run(main, flags_parser=parse_args)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "xla_spawn.py", line 71, in main
    xmp.spawn(mod._mp_fn, args=(), nprocs=args.num_cores, start_method='fork')
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 395, in spawn
    start_method=start_method)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
    while not context.join():
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 112, in join
    (error_index, exitcode)
Exception: process 0 terminated with exit code 17
running time 153

Comments:
This seems like a memory error. Can you try reducing the batch size and see if that works?

thanks worked

On Fri, Nov 6, 2020 at 5:28 PM Lysandre Debut <notifications@github.com>
wrote:

> This seems like a memory error. Can you try reducing the batch size and
> see if that works?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/8355#issuecomment-723172299>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ARPXHHYNQGEEYF4TXY4LNQLSOQP3XANCNFSM4TMNPQEQ>
> .
>