~7% drop in performance is noticed for huggingface GPT2 model 
Author: rraminen
Pull request: no

### System Info

platform: ROCm AMD device
python version: 3.7.13

There is a ~7% drop in performance noticed for huggingface GPT2 model after the IFU (https://github.com/ROCmSoftwarePlatform/transformers/pull/15) on https://github.com/ROCmSoftwarePlatform/transformers repository. 

@patil-suraj, @patrickvonplaten, could you please help me in finding the change in transformers that is responsible for the drop in performance?

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Command used to run the model:

python3 -m torch.distributed.launch --nproc_per_node=8 transformers/examples/pytorch/language-modeling/run_clm.py --output_dir output --model_name_or_path gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 --do_train --do_eval --label_smoothing 0.1 --logging_steps 1 --logging_dir log --fp16 --dataloader_num_workers 1 --skip_memory_metrics --per_device_train_batch_size=8 --overwrite_output_dir --max_steps 150

### Expected behavior

I was expecting to see similar or better performance of the model after IFU on Aug 9, 2022. 

I also tried with the recent commits after Aug 9, 2022. Those seem to worsen the performance much more.  

Comments:
Hello @rraminen, could you mention the two versions of `transformers` between which you see the difference in performance?
By performance, do you mean performance in metrics or performance in processing power/speed of iteration?
Thank you.

Hi @LysandreJik, thank you for your response. 

The performance metric I am looking at is **stable_train_samples_per_second**.

The transformers version before performance drop is 4.19.0.dev0.

Performance drop is noticed 4.22.0.dev0 and 4.23.0.dev0 transformers versions.

@rraminen , stable_train_samples_per_second is only in the ROCm fork of HF transformers. This equates to train_samples_per_second with a warmup period. 

So I get the full context: is this happening only on ROCm hardware and using the fork from `ROCmSoftwarePlatform`, or is it happening across the library?

I'm trying to understand if it's link to this repository or to the fork. Thanks!

We observed this on ROCm hardware. @rraminen , can you please test on A100 to confirm if the drop is not limited to MI250 ? 

The perf drop is not happening across the library. Just gpt2. 

The perf drop is not observed on A100. 

@rraminen , I dont think upstream HF can help much here; this is on AMD to root-cause. Please close this ticket. 

Let's get started with figuring out which commit caused the regression on ROCm, and tracking internally. 

I agree @amathews-amd, I don't think we're in a very large capacity to help here. We're happy to follow along however, so please let us know if there's anything we can do to help out.

Thank you @LysandreJik, closing this issue. 

Just seconding what @LysandreJik said: if we can help in any way to improve support or performance of our software on AMD chips, we'd like to help

Just ping us