speed becomes slow
Author: taesikna
Pull request: no
Labels: name: Help wanted

name: Need more information

name: wontfix

Hi,
I'm trying to fine-tune Bert-base-uncased model for Squad v1.1 on microsoft azure.
And I'm experiencing slow speed as the training continues.

[logs from first epoch]
Iteration:   3%|▎         | 452/14774 [01:46<57:06,  4.18it/s][A

Iteration:   3%|▎         | 453/14774 [01:46<57:07,  4.18it/s][A

Iteration:   3%|▎         | 454/14774 [01:47<57:02,  4.18it/s][A

Iteration:   3%|▎         | 455/14774 [01:47<57:14,  4.17it/s][A

Iteration:   3%|▎         | 456/14774 [01:47<57:12,  4.17it/s][A

Iteration:   3%|▎         | 457/14774 [01:47<57:23,  4.16it/s][A

Iteration:   3%|▎         | 458/14774 [01:48<57:26,  4.15it/s][A


[logs from 2nd epoch]
Iteration:  29%|██▉       | 4313/14774 [3:51:45<10:33:14,  3.63s/it][A

Iteration:  29%|██▉       | 4314/14774 [3:51:49<10:31:50,  3.62s/it][A

Iteration:  29%|██▉       | 4315/14774 [3:51:52<10:31:40,  3.62s/it][A

Iteration:  29%|██▉       | 4316/14774 [3:51:56<10:28:11,  3.60s/it][A

Iteration:  29%|██▉       | 4317/14774 [3:51:59<10:29:19,  3.61s/it][A

Iteration:  29%|██▉       | 4318/14774 [3:52:03<10:27:00,  3.60s/it][A

I have seen you were also using Microsoft Azure, and I wonder if you could help me to figure out what was wrong with my setting.

[Azure Cluster configuration]
VM size : STANDARD_NC6S_V3 (single Tesla V100)
Operating system : Canonical UbuntuServer 16.04-LTS (latest)
Auto scale : true
Target number of nodes : 1 (Min: 0, Max: 50)

[json file used to submit the job]
    "containerSettings": {
      "imageSourceRegistry": {
        "image": "pytorch/pytorch:latest"
      }
    },
    "jobPreparation": {
      "commandLine": "conda install python==3.7 && pip install requests boto3 tqdm"
    },

And I have used the same setting in the repo except tran_batch_size=6.
Thanks!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I know this is closed, but I'm running into a similar issue. For the first ~100 batches, the script runs with an OK speed (1s/it, batch size 64, 512 tokens, 8x GTX 1080Ti) (specific for `DistilBertForSequenceClassification` in my case).
After that, the speed drops significantly, to about 10s/it, with the GPUs mostly sitting idle. (0% on  `gpustat`.

Any idea on what could be causing this?

Update: Looks like I was accumulating the gradient for too long.

> Update: Looks like I was accumulating the gradient for too long.

@ArthurCamara So, how did you fix the problem in your case? Did you change the `gradient_accumulation_steps` parameter of `Trainer`? What were the initial value and the value which helped to resolve the problem?
And how did you understand that the problem was really in this?

I have the same problem. I noticed that the training speed slows down as GPU temperature goes up... When the temperature goes down (if I wait after terminating the process), the speed becomes okay again.

This issue happens only when I use `Trainer`. When I don't use it (i.e. use PyTorch utilities directly), the training speed is stable and the temperature doesn't go up.

@taesikna Did you fix your issue?

> > Update: Looks like I was accumulating the gradient for too long.
> 
> @ArthurCamara So, how did you fix the problem in your case? Did you change the `gradient_accumulation_steps` parameter of `Trainer`? What were the initial value and the value which helped to resolve the problem?
> And how did you understand that the problem was really in this?

I think the initial setting was 5 or something. I dropped to 1 and it was fine then. 

> > > Update: Looks like I was accumulating the gradient for too long.
> > 
> > @ArthurCamara So, how did you fix the problem in your case? Did you change the `gradient_accumulation_steps` parameter of `Trainer`? What were the initial value and the value which helped to resolve the problem?
> > And how did you understand that the problem was really in this?
> 
> I think the initial setting was 5 or something. I dropped to 1 and it was fine then. 

As I see now the default value is 1, but I still observe the slow down. @ArthurCamara Do you have any idea on that?

> > > > Update: Looks like I was accumulating the gradient for too long.
> > > 
> > > 
> > > @ArthurCamara So, how did you fix the problem in your case? Did you change the `gradient_accumulation_steps` parameter of `Trainer`? What were the initial value and the value which helped to resolve the problem?
> > > And how did you understand that the problem was really in this?
> > 
> > 
> > I think the initial setting was 5 or something. I dropped to 1 and it was fine then.
> 
> As I see now the default value is 1, but I still observe the slow down. @ArthurCamara Do you have any idea on that?

I was not using the Trainer, but my own training loop. dropping the accumulation steps to 1 helped because it was overwhelming the GPUs memory and that makes the GPUs sit idly. If the GPUs on `nvidia-smi` are idle, but their memory is full, it's probably something related to memory usage.  Otherwise, no idea. 



> facing this same issue too, on 2080. will not use Trainer.

```
en ignored: tokens, ner_tags, id.
[INFO|trainer.py:1156] 2021-06-05 16:46:31,375 >> ***** Running training *****
[INFO|trainer.py:1157] 2021-06-05 16:46:31,386 >>   Num examples = 2021
[INFO|trainer.py:1158] 2021-06-05 16:46:31,397 >>   Num Epochs = 1
[INFO|trainer.py:1159] 2021-06-05 16:46:31,407 >>   Instantaneous batch size per device = 10
[INFO|trainer.py:1160] 2021-06-05 16:46:31,418 >>   Total train batch size (w. parallel, distributed & accumulation) = 10
[INFO|trainer.py:1161] 2021-06-05 16:46:31,428 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:1162] 2021-06-05 16:46:31,439 >>   Total optimization steps = 203
  1%|█▍                                                                                                                                           | 2/203 [00:12<16:46,  5.01s/it
  1%|██                                                                                                                                           | 3/203 [00:20<19:53,  5.97s/it
  2%|██▊                                                                                                                                          | 4/203 [00:37<30:55,  9.33s/i
  2%|███▍                                                                                                                                         | 5/203 [00:50<34:44, 10.53s/
  3%|████▏                                                                                                                                        | 6/203 [01:01<34:07, 10.39s
  3%|████▊                                                                                                                                        | 7/203 [01:02<25:16,  7.74s
  4%|█████▌                                                                                                                                       | 8/203 [01:16<31:19,  9.64
  4%|██████▎                                                                                                                                      | 9/203 [01:30<34:47, 10.7
  5%|██████▉                                                                                                                                     | 10/203 [01:43<37:30, 11.6
  5%|███████▌                                                                                                                                    | 11/203 [01:56<38:22, 11.
  6%|████████▎                                                                                                                                   | 12/203 [02:09<38:37, 12
  6%|████████▉                                                                                                                                   | 13/203 [02:23<40:23, 12
  7%|█████████▋                                                                                                                                  | 14/203 [02:33<37:51, 1
  7%|██████████▎                                                                                                                                 | 15/203 [02:44<36:31,
  8%|███████████                                                                                                                                 | 16/203 [02:55<35:39,
  8%|███████████▋                                                                                                                                | 17/203 [03:07<36:25,
  9%|████████████▍                                                                                                                               | 18/203 [03:20<37:19
  9%|█████████████                                                                                                                               | 19/203 [03:34<38:25
 10%|█████████████▊                                                                                                                              | 20/203 [03:47<38:4
 10%|██████████████▍                                                                                                                             | 21/203 [04:00<38:
 11%|███████████████▏                                                                                                                            | 22/203 [04:13<39
 11%|███████████████▊                                                                                                                            | 23/203 [04:27<39
:42, 13.24s/it]
```