Training time increases

Mohammad_Hassan_Soha: <p>Hi</p>
<p>I train a speaker recognition model using ResNet34 model (with little modifications). The code is available at <a href="https://github.com/hajix/VoxSRC-20" rel="noopener nofollow ugc">github</a>. At the beginning epochs, each epoch takes about 15 seconds:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/9/89346e92dab3d43ba67599d232137de211cb1acd.png" data-download-href="https://discuss.pytorch.org/uploads/default/89346e92dab3d43ba67599d232137de211cb1acd" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/89346e92dab3d43ba67599d232137de211cb1acd_2_690x388.png" alt="image" data-base62-sha1="jzLKoCldHN9aoGBCRk7yds0mxxX" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/89346e92dab3d43ba67599d232137de211cb1acd_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/89346e92dab3d43ba67599d232137de211cb1acd_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/89346e92dab3d43ba67599d232137de211cb1acd_2_1380x776.png 2x" data-dominant-color="513247"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×1080 129 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>After a while, time to complete an epoch becomes about 30 seconds:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0f8d88f0cb4b1d24c4271562e82728e3c2829e95.png" data-download-href="https://discuss.pytorch.org/uploads/default/0f8d88f0cb4b1d24c4271562e82728e3c2829e95" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0f8d88f0cb4b1d24c4271562e82728e3c2829e95_2_690x388.png" alt="image" data-base62-sha1="2dAoKEuvNdGiL8yHeONhT13FehT" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0f8d88f0cb4b1d24c4271562e82728e3c2829e95_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0f8d88f0cb4b1d24c4271562e82728e3c2829e95_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0f8d88f0cb4b1d24c4271562e82728e3c2829e95_2_1380x776.png 2x" data-dominant-color="8B7784"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×1080 232 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I really can’t understand what is going wrong! There are huge amount of free ram and the GPU memory footprint does not change during this transition.</p>
<p>P.S. I saw this behavior after adding learning rate scheduler. I’m not sure this may relates to scheduler or not.<br>
Another weird behavior is that killing and running the process (even after rebooting the system!) does not reset the epoch time to 15 seconds! The only solution I found effective is to power off and on my system!</p>
<p>Any help is appreciated.<br>
regards</p>

ptrblck: <p>Could you check the temperature and clocks of your system?<br>
Does the performance recover after you kill the process and wait for some time?<br>
Maybe your system is overheating and thus reducing the clocks.</p>

Mohammad_Hassan_Soha: <p>Thanks a lot ptrblk. I eliminate learning rate scheduler and the problem goes away! So I did not reproduced the situation to look for overheating issue. I’m wondering if I do something wrong with learning rate scheduler?<br>
Currently I replaced <code>scheduler.step()</code> with:</p>
<pre><code class="lang-auto">if (epoch + 1) % 5000 == 0:
    for g in optimizer.param_groups:
        g['lr'] = g['lr'] * 0.75
</code></pre>
<p>and I’m waiting to see whether the problem comes in or not, but tests are a little time consuming…</p>

ptrblck: <p>Let me know, if you can deterministically reproduce the issue using the scheduler and avoid it by removing the scheduler, please.</p>

Mohammad_Hassan_Soha: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I confirm that using scheduler will reproduce the issue (after about 8 hours of training), and removing scheduler will avoid the issue. Currently I’m running the model without the scheduler for 48 hours and I don’t observe such problem.<br>
Here it is all scheduler related lines of my code:</p>
<pre><code class="lang-auto"># optimizer
optimizer = torch.optim.Adam(
    [
        {'params': model.parameters(), 'lr': args.lr},
        {'params': criterion.parameters(), 'lr': args.criterion_lr}
    ]
)

# lr schedule
scheduler = torch.optim.lr_scheduler.StepLR(
    optimizer,
    step_size=args.step_size,
    gamma=args.gamma
)

for epoch in range(scheduler.last_epoch, args.num_epochs):
    for x, target in tqdm(dataloader):
        x = x.to(device)
        target = target.to(device)
        y = model(x)
        scores, loss = criterion(y, target)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    scheduler.step()
    lr = optimizer.state_dict()['param_groups'][0]['lr']
    log.add_scalar('train-lr', lr, epoch + 1)
</code></pre>
<p>Could you please inform me if I used scheduler in a wrong way?!<br>
regards</p>

ptrblck: <p>That’s really weird.<br>
Is the linked code executable or would I need to get some input shapes or other arguments to run it?</p>

Mohammad_Hassan_Soha: <p><code>train.sh</code> provides minimalist arguments needed to run the code, after downloading and converting audio formats. You could eliminate the data loader stuff and feed the model with a NxCxHxT shaped tensor, where: N=batch size (256), C=channels (1), H=num filter banks (40 lmfb/mfsc) and T=time steps (200).</p>
<p>I also see that other repos (like <a href="https://github.com/clovaai/voxceleb_trainer/blob/master/SpeakerNet.py#L256" rel="nofollow noopener">clovai baseline</a> for SR, or <a href="https://github.com/SeanNaren/deepspeech.pytorch/blob/master/train.py#L325" rel="nofollow noopener">deepspeech.pyotrch</a> for ASR) avoid using schedulers. I’m not sure if they have the same experience or not?!</p>

ptrblck: <p>It’s the first time I see this issue, so I’ll try to run the code for some time and report back.<br>
Please ping me in the next two days, if I haven’t posted an update.</p>

Mohammad_Hassan_Soha: <p>sure.</p>
<p>Thanks for your reply and patience.</p>

Mohammad_Hassan_Soha: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Do you succeed to reproduce the issue? I have some free time and I could spend some time to explore the issue. I wonder to write a test to reproduce the issue if it helps?</p>

ptrblck: <p>A minimal code snippet would be great, as I have trouble getting the code to run.<br>
After removing the data loader, I get e.g. this error:</p>
<pre><code class="lang-python">AttributeError: 'Namespace' object has no attribute 'num_spkr'
</code></pre>
<p>and <code>train.sh</code> doesn’t seem to specify this argument.</p>

Mohammad_Hassan_Soha: <p>The trunk network train with classification method, and <code>num_spkr</code> defines the number of the classes. Add:<br>
<code>args.num_spkr = 5994</code><br>
will resolve that error. But it’s not straightforward process to run the code without downloading and pre processing the data, so I’ll write a snippet and inform you soon.</p>

Mohammad_Hassan_Soha: <p>The issue is irrelevant to the pytorch scheduler and the coincidence of occurring the issue with a hardware malfunction, make me mistake.</p>
<p>Indeed, the issue was occurred due to underpowered psu!</p>

ptrblck: <p>Thanks for the follow-up! That’s an unfortunate coincidence, but at least you’ve narrowed it down. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

el_youssfi_azeddine: <p>hello Mohammad,<br>
I think I have same issue, I think you resolved it, can you just tell what is the cause?<br>
also what do you mean by underpower of PSU, how we recognize that since ti the set seems working good,<br>
thanks</p>