text: 
  
  TITLE
  Why training time is much more than same task in Fairseq?
  
  COMMENTS
  6
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ### System Info
  
  ```shell
  - `transformers` version: 4.19.0.dev0
  - Platform: Linux-4.18.0-305.28.1.el8_4.x86_64-x86_64-with-glibc2.27
  - Python version: 3.9.12
  - Huggingface_hub version: 0.2.1
  - PyTorch version (GPU?): 1.11.0 (True)
  - Tensorflow version (GPU?): not installed (NA)
  - Flax version (CPU?/GPU?/TPU?): not installed (NA)
  - Jax version: not installed
  - JaxLib version: not installed
  - Using GPU in script?: <fill in>
  - Using distributed or parallel set-up in script?: <fill in>
  ```
  
  
  ### Who can help?
  
  _No response_
  
  ### Information
  
  - [X] The official example scripts
  - [ ] My own modified scripts
  
  ### Tasks
  
  - [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
  - [ ] My own task or dataset (give details below)
  
  ### Reproduction
  
  Device: Tesla V100-PCIE-32GB * 1
  
  ## For HF's Trainer:
  ```
  python run_summarization.py \
      --model_name_or_path facebook/bart-large \
      --do_train \
      --do_eval \
      --do_predict \
      --dataset_name xsum \
      --output_dir /scratch/tw2112/datas/HFPLAY2 \
      --overwrite_output_dir \
      --max_grad_norm 0.1 \
      --label_smoothing_factor 0.1 \
      --fp16 True \
      --learning_rate 3e-05 \
      --lr_scheduler_type polynomial \
      --greater_is_better True \
      --warmup_steps 500 \
      --num_train_epochs 1 \
      --max_source_length 1024 \
      --max_target_length 1024 \
      --val_max_target_length 80 \
      --gradient_accumulation_steps 1 \
      --weight_decay 0.01 \
      --per_device_train_batch_size 8 \
      --per_device_eval_batch_size 24 \
      --num_beams 6 \
      --save_strategy steps \
      --save_steps 2000 \
      --evaluation_strategy steps \
      --eval_steps 2000 \
      --load_best_model_at_end True \
      --metric_for_best_model loss \
      --greater_is_better False
  ```
  The estimated time for 1 epoch is about 3.3h
  
  
  ## For FairSeq:
  
  ```
  WARMUP_UPDATES=500      
  LR=3e-05
  MAX_TOKENS=2048
  BART_PATH=/scratch/tw2112/codes/models/bart.large/model.pt
  
  fairseq-train xsum \
     --restore-file $BART_PATH \
     --task translation \
     --source-lang source --target-lang target \
     --truncate-source \
     --batch-size 8 \
     --layernorm-embedding \
     --share-all-embeddings \
     --share-decoder-input-output-embed \
     --reset-optimizer --reset-dataloader --reset-meters \
     --required-batch-size-multiple 1 \
     --arch bart_large \
     --criterion label_smoothed_cross_entropy  \
     --label-smoothing 0.1 \
     --dropout 0.1 --attention-dropout 0.1 \
     --weight-decay 0.01 --optimizer adam --adam-betas "(0.9, 0.999)" --adam-eps 1e-08 \
     --clip-norm 0.1 \
     --lr-scheduler polynomial_decay --lr $LR  --warmup-updates $WARMUP_UPDATES \
     --fp16 \
     --max-epoch 1\
     --skip-invalid-size-inputs-valid-test \
     --find-unused-parameters;
  
  ```
  The estimated time for 1 epoch is about 1.5h 
   
  
  ### Expected behavior
  
  ```shell
  I checked the config namespace in fairseq's log and changed the settings according to the example in Fairseq repo. I change both 2 commands to train on XSUM, 1 epoch, batch=8, gradient_accumulation_steps=1, and HF's time for 1 epoch is 3 times slower than fairseq's. Did I do anything wrong?
  ```
  