run_ner.py is slower than run_ner_no_trainer.py
Author: h2bit
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.19.3
- Platform: Linux-5.15.0-35-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.2.1
- PyTorch version (GPU?): 1.11.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```


### Who can help?

Models:
BERT @LysandreJik 

Library:
Trainer @sgugger 

Examples:
[token-classification/](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) @sgugger @pat

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I ran the example in [pytorch/token-classification](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) with the two scripts provided in the repo: [run_ner.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/token-classification/run_ner.py) and
[run_ner_no_trainer.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/token-classification/run_ner_no_trainer.py), respectively. And the parameters for the two scripts are set identical as follows:

1. in `run_ner.py`:

```
export TASK_NAME=ner

python3 run_ner.py \
  --model_name_or_path bert-base-uncased \
  --dataset_name conll2003 \
  --task_name $TASK_NAME \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --output_dir /tmp/$TASK_NAME/ \
  --overwrite_output_dir \
  --per_device_eval_batch_size 16 \
  --do_train \
  --do_eval
```

2. in `run_ner_no_trainer.py`:

```
export TASK_NAME=ner

accelerate launch run_ner_no_trainer.py \
  --model_name_or_path bert-base-cased \
  --dataset_name conll2003 \
  --task_name $TASK_NAME \
  --max_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --output_dir /tmp/$TASK_NAME/ \
  --per_device_eval_batch_size 16
```

However, I found `run_no_trainer.sh` is much slower than `run.sh`.
The results are:

1. in `run_ner.py`:
> ***** train metrics *****
>   epoch                    =        3.0
>   train_loss               =     0.1075
>   train_runtime            = **_0:02:02.31_**
>   train_samples            =      14042
>   train_samples_per_second =    344.419
>   train_steps_per_second   =      5.396

2. in `run_ner_no_trainer.py`:

>  33%|████████████████████████████                                                        | 220/660 [00:22<00:44,  9.86it/s]epoch 0: {'precision': 0.8766612641815235, 'recall': 0.904666332162569, 'f1': 0.8904436579142316, 'accuracy': 0.9829800402289959}
>  67%|███████████████████████████████████████████████████████▊                            | 439/660 [00:47<00:22,  9.67it/s]epoch 1: {'precision': 0.9085555373793555, 'recall': 0.9289178792440207, 'f1': 0.9186238835593781, 'accuracy': 0.986712826860591}
> 100%|████████████████████████████████████████████████████████████████████████████████████| 660/660 [**_01:11_**<00:00,  9.75it/s]epoch 2: {'precision': 0.9220243982855258, 'recall': 0.935440709148687, 'f1': 0.928684101286841, 'accuracy': 0.988356800247563}

That is, `run_ner.py` is about half the speed of `run_ner_no_trainer.py`

### Expected behavior

```shell
The two scripts should be at the same throughput.
```


Comments:
How many GPUs do you have? You launch the second script with `accelerate launch`, so will use distributed training if you have several GPUs, while the first script is launched with Python so is not distributed.

> How many GPUs do you have? You launch the second script with `accelerate launch`, so will use distributed training if you have several GPUs, while the first script is launched with Python so is not distributed.

Thanks for your reply.
I used two GPUs on a single machine. But the first script launched with Python also utilizes both of the GPUs when I check through the command `nvidia-smi`. Perhaps the trainer class has already used distributed launch and it is still slower.

Yes, but you're using `DataParallel` vs `DistributedDataParallel`. This is enough to explain the speed difference very likely.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.