akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Restarting with adam

xoul: <p>Hello,</p>
<p>I am training my network with early stopping strategy. I start with a higher learning rate, and based on validation loss, I need to restart training from an earlier snapshot.</p>
<p>I am able to save/load snapshot with model and optimizer state_dicts. No problem with that.</p>
<p>My question is, once I restart training, how do I set the learning rate of adam again? Should I restart adam fresh instead of using a state_dict or should I use<br>
optimizer.param_groups[0][‘lr’] = lr to adjust learning rate with loaded optimizer state_dict?</p>
<p>For example,<br>
I train my network with lr = 1e-6 for 5 epochs, saved model and optimizer state_dict.<br>
I am now restarting from epoch 6, but I need lr = 1e-7 instead. What is the best approach for this?</p>
<p>Thanks!</p>

ptrblck: <p>Are you using a learning rate scheduler at the moment or are you manipulating the lr manually?<br>
I would try to load the <code>state_dict</code> and set the learning rate to the desired value.<br>
An uninitialized Adam optimizer might create loss spikes when the training continues, which might take some training again to lower it to the previous value.</p>

xoul: <p>I’m manipulating it manually as I restart learning when validation loss stops decreasing. I’m just confused what is the right way to manually adjust learning rate, specially when using Adam. I did use ReduceLROnPlateau, but it didn’t result in desired behavior. I didn’t know that behavior about Adam. Thanks for clearing it up!</p>
DistributedDataParallel with sync batchnorm leads to gpu lost

jiang_wenjie: <p>My server is 4xGTX1080TI, when i use DistributedDataParallel with sync batchnorm on 4 gpus, it will lead to gpu lost. But when i use apex sync bathnorm, the program works fine. And  the sync batchborm run on 3 gpus it can works fine also. what’s the problem? how can i deal with it?</p>

ptrblck: <p>Could you explain a bit, what “GPU lost” means?<br>
Is a GPU disappearing from the system, i.e. is it not detectable in <code>nvidia-smi</code> anymore?</p>

jiang_wenjie: <p>Yes. The Server requires to reboot for using GPU. I switch the SyncBatchNorm to BatchNorm, the situation will disappear.</p>