Package: agc-optims
AGC OptimizersA small lib for using adaptive gradient clipping in your optimizer. Currently PyTorch only.NewsIntroductionComparisonTo DoNewsSep 15, 2021Add AGC use independent from optimizer choice in PyTorchSep 14, 2021Add AdamW, Adam, SGD and RMSprop with AGCAdd first comparsion between optimizers with and without AGC based on CIFAR10IntroductionBrock et al.introduced 2021 a new clipping technique in order to increase stability of large batch training and high learning rates in their Normalizer-Free Networks (NFNet), the adaptive gradient clipping. This clipping method is not implemented in leading frameworks, thus I provide optimizers which are capable of AGC.Installationpipinstallagc_optimsUsageTo be consistent with PyTorch all arguments of the optimizer remain the same as in the standard. Only two parameters are added for the AGC:clipping: Hyperparameter for the clipping of the parameter. Default value 1e-2, smaller batch sizes demand a higher clipping parameteragc_eps: Term used in AGC to prevent grads clipped to zero, default value 1e-3Optimizer independentfromtorch.optimimportAdamfromagc_optims.clipperimportAGCnet=Net()# your modeloptimizer=Adam(net.parameters(),lr=0.001)optimizer=AGC(optimizer=optimizer,clipping=0.16)SGDfromagc_optims.optimimportSGD_AGCnet=Net()# your modeloptimizer=SGD_AGC(net.parameters(),lr=0.01,momentum=0.9,clipping=0.16)Adamfromagc_optims.optimimportAdam_AGCnet=Net()# your modeloptimizer=Adam_AGC(net.parameters(),lr=0.001,weight_decay=1e-4,clipping=0.16)AdamWfromagc_optims.optimimportAdamW_AGCnet=Net()# your modeloptimizer=AdamW_AGC(net.parameters(),lr=0.001,weight_decay=1e-4,clipping=0.16)RMSpropfromagc_optims.optimimportRMSprop_AGCnet=Net()# your modeloptimizer=RMSprop_AGC(net.parameters(),lr=0.001,clipping=0.16)Now you can use the optimizer just like their non-AGC counterparts.ComparisonThe following comparison shows that for batch sizes 64 and 128 Adam with AGC performs better than the normal Adam. SGD is unfortunately worse with AGC, but the batch size is also very small compared to the NFNet paper. This requires more comparisons with higher batch sizes and also on other data sets. RMSprop is also better at both batch sizes with AGC than without. The learning rate was left at the default value for all optimizers and the scripts in the performance_tests folder were used as the test environment.Batch Size 64 - SGD Accuracy on Cifar10Batch Size 64 - SGD Loss on Cifar10Batch Size 128 - SGD Accuracy on Cifar10Batch Size 128 - SGD Loss on Cifar10Batch Size 64 - Adam Accuracy on Cifar10Batch Size 64 - Adam Loss on Cifar10Batch Size 128 - Adam Accuracy on Cifar10Batch Size 128 - Adam Loss on Cifar10Batch Size 64 - RMSProp Accuracy on Cifar10Batch Size 64 - RMSProp Loss on Cifar10Batch Size 128 - RMSProp Accuracy on Cifar10Batch Size 128 - RMSProp Loss on Cifar10As a little treat, I have also compared the speed of the optimizer with and without AGC to see whether this greatly increases training times.Batch Size 128 - RMSProp Accuracy on Cifar10Batch Size 128 - RMSProp Loss on Cifar10To DoAdd first comparsion based on CIFAR10 with a small CNNAdd AGC independent from optimizerAdd comparsion with higher batch sizes (256,512,1024)Add tests for each optimizerClipping == 0 no AGCAdd comparsion based on CIFAR100 with a small CNNAdd comparsion based on CIFAR10/100 with ResNetAdd comparsion with ImageNet (I do not have enough GPU-Power currently if someone provides some tests I would be grateful)Add all optimizer included in PyTorchSupport of other frameworks than PyTorchAdd first comparsion based on CIFAR with a small CNN