Training beyond specified 't_total' steps with schedule 'warmup_linear'. Learning rate set to 0.0. Please set 't_total' of BertAdam correctly.
Author: ZhaofengWu
Pull request: no
Labels: name: wontfix

I am seeing the above error in my training process. Is it a significant issue? Looks like it's related to `t_total`, which should be properly set here:
https://github.com/huggingface/pytorch-pretrained-BERT/blob/b832d5bb8a6dfc5965015b828e577677eace601e/examples/run_classifier.py#L742-L743

What could be potential causes of this issue? I trained exactly `args.num_train_epochs` epochs, and didn't alter the training data in between, so shouldn't this pre-calculated `t_total` work without issue?

My `len(train_examples)` is 49401, `args.num_train_epochs` is 5, using 2 GPUs, and other parameters are left as default. If it matters, my code is based on a version (68a889) before the recent `WarmupLinearSchedule` change.

Comments:
Actually, shouldn't the `int()` be a `ceiling()`? Because let's say `args.gradient_accumulation_steps` is 1, then it is `ceiling(len(train_examples) / args.train_batch_size)` that is the number of batches in an epoch.

I am having the same problem with my finetuned model for gpt2

I am having the same issue in partially changed run_squad.py code.

I have the same issue.

Humm yes, we should probably change `int` to `ceiling` in this example indeed.

Is this a significant issue? If it's only the last few batches in the last epoch that are not being trained on, it shouldn't be a huge problem, right?

Also I find it strange that suddenly a lot of people are running into this bug in this past week (according to the replies to this issue) even though the `int` code was written 3 months ago. Is this also related to some other more recent changes?

Yes there was a huge refactoring of the `BertAdam` optimizer by @lukovnikov (#389, #445, #531)

Hi, this warning is printed to avoid wasted computations with warmup-linear or other surprises with other schedules due to a t_total set too low.
And I think that line should be `int( math.ceil(len(train_examples) / args.train_batch_size) / args.gradient_accumulation_steps) * args.num_train_epochs` (@thomwolf) ?

Hi. I figured out the source of the problem: t_total, aka num_train_optimization_steps 

https://github.com/huggingface/pytorch-pretrained-BERT/blob/3fc63f126ddf883ba9659f13ec046c3639db7b7e/examples/run_squad.py#L903

 is computed over the length of the train examples, while the true number of steps is determined by whatever convert_examples_to_features returns

https://github.com/huggingface/pytorch-pretrained-BERT/blob/3fc63f126ddf883ba9659f13ec046c3639db7b7e/examples/run_squad.py#L970

A print(len(train_examples), len(train_features)) in line 980 returns:

87599 191597 



You could also add the option `drop_last=True` to the `DataLoader`, then the number of samples will be calculated correctly.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I get error below while running the program.. Did I do any mistake?

warmup_linear = WarmupLinearSchedule( warmup=args.warmup_proportion,
t_total=num_train_optimization_steps)

lr_this_step = args.learning_rate * warmup_linear.get_lr(num_train_optimization_steps,
args.warmup_proportion)

WARNING - pytorch_pretrained_bert.optimization - Training beyond specified 't_total'. Learning rate multiplier set to 0.0. Please set 't_total' of WarmupLinearSchedule correctly.