Issue: Adding new tokens to bert tokenizer in QA
Author: andreabac3
Pull request: no

**WARNING**: This issue is a replica of this other [issue](https://github.com/huggingface/notebooks/issues/21) open by me, I  ask you sorry if I have open it in the wrong place.

Hello Huggingface's team (@sgugger , @joeddav, @LysandreJik) 
I have a problem with this code base 
notebooks/examples/question_answering.ipynb - [link](https://github.com/huggingface/notebooks/blob/master/examples/question_answering.ipynb)
`
ENV: Google Colab - 
transformers Version: 4.5.0;
datasets Version: 1.5.0;
torch Version: 1.8.1+cu101;
`
I am trying to add some domain tokens in the bert-base-cased tokenizer
```python3
model_checkpoint = 'bert-base-cased'
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)
list_of_domain_tokens = ["token1", "token2", "token3"]
tokenizer.add_tokens(list_of_domain_tokens)
...
...
model = AutoModelForQuestionAnswering.from_pretrained(model_checkpoint)
print(model.device)  # cpu
model.resize_token_embeddings(len(tokenizer))
trainer = Trainer(...)
```
Then during the trainer.fit() call it report the attached error.
Can you please tell me where I'm wrong?
The tokenizer output is the usual bert inputs expressed in the form of List[List[int]] eg inputs_ids and attention_mask. 
So I can't figure out where the problem is with the device
`Input, output and indices must be on the current device`

Kind Regards,
Andrea

Comments:
I am unable to reproduce: the notebook with your added code works smoothly on my side.

Thank you @sgugger.
I want ask you sorry, I can't figure out on what's going on my side.
Now I have cloned again the notebook and the example works.
In the next days I want test it again and I will tell you more about it.

Thank you again for your help
Kind Regards,
Andrea

Hi @sgugger,
I worked on the notebook and I found the problem.
I have not yet had the opportunity to test it with the original squad dataset but this happens to me both on colab and on my machine.
I warn you it seems an absurd and paradoxical situation, moreover I in no way manage the device.
I can provide you with a video while running the notebook.
As you can see from the screenshot I am forced to keep two versions of the training args, one original from the notebook and one customized by me.

If I perform these operations I get the error
1) I instantiate my training args
2) I instantiate the Trainer
3) I run trainer.fit
I get the error `Input, output and indices must be on the current device`

To solve I have to:
Instantiate the original training args of the notebook, instantiate the trainer, perform the fit to check that it has started and then do it all over again with the training args I customized.

![Screenshot 2021-04-16 at 09 29 34](https://user-images.githubusercontent.com/36055796/114988500-b1311900-9e96-11eb-9043-c67db1924b8e.png)


Kind regards,
Andrea

Hi @sgugger,
I can confirm, the same bug happens in the original notebook with this TrainingArguments (I have tested with squad v2), the temporary fix is to start the train with the original one, stop it and then run with the customized args. 

It looks like a bug in colab (from the screenshots I assume that is what you are using for training?) since I didn't get any error on my side by executing this as a notebook.

Hi @sgugger 
Do you have tested the notebook replacing the trainer args with the following?
```python3
args = TrainingArguments(
    f"my-experiment",
    evaluation_strategy = "epoch",
    learning_rate=2e-5,
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=250,
    num_train_epochs=2,
    weight_decay=0.01,
    fp16=True,
    gradient_accumulation_steps=2,
    eval_accumulation_steps=2,
    fp16_opt_level='O2',
    fp16_full_eval=True,
    save_strategy='epoch',
    metric_for_best_model='eval_loss',
    logging_strategy='epoch'
)
```
Because I encountered the same issue on my machine.
Can you kindly test with it? Please
To test it: remove the old trainer args use the attached one and run the trainer.fit

Kind regards,
Andrea

Ah you're right, I must have made a mistake. This comes from the option `fp16_full_eval=True`.

@stas00 I'm not sure what the best place is for fixing this but if someone uses `fp16_full_eval=True` with training, the model is never sent to the proper device and training fails.

But there is no `do_train` in the args at https://github.com/huggingface/transformers/issues/11200#issuecomment-822566973

The logic is very explicit to not place on the device only for non-train when`fp16_full_eval=True` is used:

```
        if (
            self.is_model_parallel
            or (args.deepspeed and args.do_train)
            or (args.fp16_full_eval and not args.do_train)
            or (self.sharded_ddp in [ShardedDDPOption.ZERO_DP_2, ShardedDDPOption.ZERO_DP_3])
        ):
            self.place_model_on_device = False
```

You need to add `do_train=True` to your `TrainingArguments`, otherwise it defaults to eval only because you have `evaluation_strategy` set.


Hi @stas00 & @sgugger,

> You need to add `do_train=True` to your `TrainingArguments`, otherwise it defaults to eval only because you have `evaluation_strategy` set.

Ok so `do_train=True` is also compatible with `fp16_full_eval=True`? 
My objective is to train the model and pick the best one at the lowest point of eval loss.

Regarding the notebook, can I use the same Trainer object for fit and predict? Because these Booleans are never set in the notebook. I mean when I am doing trainer.predict() is obvious for the trainer to set model.eval() and torch.no_grad()?

Thank you both,
Andrea

> Ok so do_train=True is also compatible with fp16_full_eval=True?

Why did you think it shouldn't be compatible?

The only reason there is a special case for non-training is to avoid placing the full model on device before it was `half()`'ed - as it might not fit in its full size, but might fit in `half()`.

> Regarding the notebook, can I use the same Trainer object for fit and predict? Because these Booleans are never set in the notebook. I mean when I am doing trainer.predict() is obvious for the trainer to set model.eval() and torch.no_grad()?

Of course. It was designed for you to pass all the init args at once and then you can call all its functions.



@stas00 Ok clear, I have just checked and the trainer works perfectly.
What do you think to place a warning to alert the user when call trainer.fit having the trainer.do_train = False?

Because it's clear in the point of view of performance as you said but the documentation don't bring out this things for this reason I have open then issue.

Kind regards,
Andrea

Oh, I see. Until recently `do_train` was sort of optional when using user's custom code and what you're saying we need to then require `do_train=True` if `trainer.train()` is called. But we started relying on `do_train` for more than just knowing to call `train()` from scripts. This makes sense to me.

@sgugger, do you agree if we add this?
```
    def train(...):
[...]
    if not self.args.do_train:
        raise ValueError("To use `train` please make sure you set `do_train=True` when instantiating the Trainer object")
```


I would rather avoid adding this, as users have been used to not have to set that argument to True when not using example scripts. Can we just add the proper line in `train` to put the model on the device if it was not done already?

(Sorry I didn't catch you were using `do_train` in the PR you added that test, I should have caught it and commented there.)

We will probably have to rethink the design then, since it's not a simple "put on device if it wasn't already" - there are multiple cases when it shouldn't happen. For now added a hardcoded workaround: https://github.com/huggingface/transformers/pull/11322