Package: pytorch_optimizer
pytorch-optimizerBuildQualityPackageStatusLicensepytorch-optimizeris optimizer & lr scheduler collections in PyTorch.
I just re-implemented (speed & memory tweaks, plug-ins) the algorithm while based on the original paper. Also, It includes useful and practical optimization ideas.Currently,60 optimizers (+bitsandbytes),10 lr schedulers, and13 loss functionsare supported!Highly inspired bypytorch-optimizer.Getting StartedFor more, see thedocumentation.Most optimizers are under MIT or Apache 2.0 license, but a few optimizers likeFromage,NerohaveCC BY-NC-SA 4.0 license, which is non-commercial.
So, please double-check the license before using it at your work.Installation$pip3installpytorch-optimizerFrompytorch-optimizer v2.12.0, you can install and importbitsandbytesoptimizers.
please checkthe requirementsbefore installing it.$pipinstall"pytorch-optimizer[bitsandbytes]"Simple Usagefrompytorch_optimizerimportAdamPmodel=YourModel()optimizer=AdamP(model.parameters())# or you can use optimizer loader, simply passing a name of the optimizer.frompytorch_optimizerimportload_optimizeroptimizer=load_optimizer(optimizer='adamp')(model.parameters())# if you install `bitsandbytes` optimizer, you can use `8-bit` optimizers from `pytorch-optimizer`.frompytorch_optimizerimportload_optimizeropt=load_optimizer(optimizer='bnb_adamw8bit')optimizer=opt(model.parameters())Also, you can load the optimizer viatorch.hub.importtorchmodel=YourModel()opt=torch.hub.load('kozistr/pytorch_optimizer','adamp')optimizer=opt(model.parameters())If you want to build the optimizer with parameters & configs, there'screate_optimizer()API.frompytorch_optimizerimportcreate_optimizeroptimizer=create_optimizer(model,'adamp',lr=1e-3,weight_decay=1e-3,use_gc=True,use_lookahead=True,)Supported OptimizersYou can check the supported optimizers with below code.frompytorch_optimizerimportget_supported_optimizerssupported_optimizers=get_supported_optimizers()OptimizerDescriptionOfficial CodePaperCitationAdaBeliefAdapting Step-sizes by the Belief in Observed Gradientsgithubhttps://arxiv.org/abs/2010.07468citeAdaBoundAdaptive Gradient Methods with Dynamic Bound of Learning Rategithubhttps://openreview.net/forum?id=Bkg3g2R9FXciteAdaHessianAn Adaptive Second Order Optimizer for Machine Learninggithubhttps://arxiv.org/abs/2006.00719citeAdamDImproved bias-correction in Adamhttps://arxiv.org/abs/2110.10828citeAdamPSlowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weightsgithubhttps://arxiv.org/abs/2006.08217citediffGradAn Optimization Method for Convolutional Neural Networksgithubhttps://arxiv.org/abs/1909.11015v3citeMADGRADA Momentumized, Adaptive, Dual Averaged Gradient Method for Stochasticgithubhttps://arxiv.org/abs/2101.11075citeRAdamOn the Variance of the Adaptive Learning Rate and Beyondgithubhttps://arxiv.org/abs/1908.03265citeRangera synergistic optimizer combining RAdam and LookAhead, and now GC in one optimizergithubhttps://bit.ly/3zyspC3citeRanger21a synergistic deep learning optimizergithubhttps://arxiv.org/abs/2106.13731citeLambLarge Batch Optimization for Deep Learninggithubhttps://arxiv.org/abs/1904.00962citeShampooPreconditioned Stochastic Tensor Optimizationgithubhttps://arxiv.org/abs/1802.09568citeNeroLearning by Turning: Neural Architecture Aware Optimisationgithubhttps://arxiv.org/abs/2102.07227citeAdanAdaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelsgithubhttps://arxiv.org/abs/2208.06677citeAdaiDisentangling the Effects of Adaptive Learning Rate and Momentumgithubhttps://arxiv.org/abs/2006.15815citeSAMSharpness-Aware Minimizationgithubhttps://arxiv.org/abs/2010.01412citeASAMAdaptive Sharpness-Aware Minimizationgithubhttps://arxiv.org/abs/2102.11600citeGSAMSurrogate Gap Guided Sharpness-Aware Minimizationgithubhttps://openreview.net/pdf?id=edONMAnhLu-citeD-AdaptationLearning-Rate-Free Learning by D-Adaptationgithubhttps://arxiv.org/abs/2301.07733citeAdaFactorAdaptive Learning Rates with Sublinear Memory Costgithubhttps://arxiv.org/abs/1804.04235citeApolloAn Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimizationgithubhttps://arxiv.org/abs/2009.13586citeNovoGradStochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networksgithubhttps://arxiv.org/abs/1905.11286citeLionSymbolic Discovery of Optimization Algorithmsgithubhttps://arxiv.org/abs/2302.06675citeAli-GAdaptive Learning Rates for Interpolation with Gradientsgithubhttps://arxiv.org/abs/1906.05661citeSM3Memory-Efficient Adaptive Optimizationgithubhttps://arxiv.org/abs/1901.11150citeAdaNormAdaptive Gradient Norm Correction based Optimizer for CNNsgithubhttps://arxiv.org/abs/2210.06364citeRotoGradGradient Homogenization in Multitask Learninggithubhttps://openreview.net/pdf?id=T8wHz4rnuGLciteA2GradOptimal Adaptive and Accelerated Stochastic Gradient Descentgithubhttps://arxiv.org/abs/1810.00553citeAccSGDAccelerating Stochastic Gradient Descent For Least Squares Regressiongithubhttps://arxiv.org/abs/1704.08227citeSGDWDecoupled Weight Decay Regularizationgithubhttps://arxiv.org/abs/1711.05101citeASGDAdaptive Gradient Descent without Descentgithubhttps://arxiv.org/abs/1910.09529citeYogiAdaptive Methods for Nonconvex OptimizationNIPS 2018citeSWATSImproving Generalization Performance by Switching from Adam to SGDhttps://arxiv.org/abs/1712.07628citeFromageOn the distance between two neural networks and the stability of learninggithubhttps://arxiv.org/abs/2002.03432citeMSVAGDissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientsgithubhttps://arxiv.org/abs/1705.07774citeAdaModAn Adaptive and Momental Bound Method for Stochastic Learninggithubhttps://arxiv.org/abs/1910.12249citeAggMoAggregated Momentum: Stability Through Passive Dampinggithubhttps://arxiv.org/abs/1804.00325citeQHAdamQuasi-hyperbolic momentum and Adam for deep learninggithubhttps://arxiv.org/abs/1810.06801citePIDA PID Controller Approach for Stochastic Optimization of Deep NetworksgithubCVPR 18citeGravitya Kinematic Approach on Optimization in Deep Learninggithubhttps://arxiv.org/abs/2101.09192citeAdaSmoothAn Adaptive Learning Rate Method based on Effective Ratiohttps://arxiv.org/abs/2204.00825v1citeSRMMStochastic regularized majorization-minimization with weakly convex and multi-convex surrogatesgithubhttps://arxiv.org/abs/2201.01652citeAvaGradDomain-independent Dominance of Adaptive Methodsgithubhttps://arxiv.org/abs/1912.01823citePCGradGradient Surgery for Multi-Task Learninggithubhttps://arxiv.org/abs/2001.06782citeAMSGradOn the Convergence of Adam and Beyondhttps://openreview.net/pdf?id=ryQu7f-RZciteLookaheadk steps forward, 1 step backgithubhttps://arxiv.org/abs/1907.08610citePNMManipulating Stochastic Gradient Noise to Improve Generalizationgithubhttps://arxiv.org/abs/2103.17182citeGCGradient Centralizationgithubhttps://arxiv.org/abs/2004.01461citeAGCAdaptive Gradient Clippinggithubhttps://arxiv.org/abs/2102.06171citeStable WDUnderstanding and Scheduling Weight Decaygithubhttps://arxiv.org/abs/2011.11152citeSoftplus TCalibrating the Adaptive Learning Rate to Improve Convergence of ADAMhttps://arxiv.org/abs/1908.00700citeUn-tuned w/uOn the adequacy of untuned warmup for adaptive optimizationhttps://arxiv.org/abs/1910.04209citeNorm LossAn efficient yet effective regularization method for deep neural networkshttps://arxiv.org/abs/2103.06583citeAdaShiftDecorrelation and Convergence of Adaptive Learning Rate Methodsgithubhttps://arxiv.org/abs/1810.00143v4citeAdaDeltaAn Adaptive Learning Rate Methodhttps://arxiv.org/abs/1212.5701v1citeAmosAn Adam-style Optimizer with Adaptive Weight Decay towards Model-Oriented Scalegithubhttps://arxiv.org/abs/2210.11693citeSignSGDCompressed Optimisation for Non-Convex Problemsgithubhttps://arxiv.org/abs/1802.04434citeSophiaA Scalable Stochastic Second-order Optimizer for Language Model Pre-traininggithubhttps://arxiv.org/abs/2305.14342citeProdigyAn Expeditiously Adaptive Parameter-Free Learnergithubhttps://arxiv.org/abs/2306.06101citePAdamClosing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networksgithubhttps://arxiv.org/abs/1806.06763citeLOMOFull Parameter Fine-tuning for Large Language Models with Limited Resourcesgithubhttps://arxiv.org/abs/2306.09782citeTigerA Tight-fisted Optimizer, an optimizer that is extremely budget-consciousgithubciteCAMEConfidence-guided Adaptive Memory Efficient Optimizationgithubhttps://aclanthology.org/2023.acl-long.243/citeSupported LR SchedulerYou can check the supported learning rate schedulers with below code.frompytorch_optimizerimportget_supported_lr_schedulerssupported_lr_schedulers=get_supported_lr_schedulers()LR SchedulerDescriptionOfficial CodePaperCitationExplore-ExploitWide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedulehttps://arxiv.org/abs/2003.03977citeChebyshevAcceleration via Fractal Learning Rate Scheduleshttps://arxiv.org/abs/2103.01338citeSupported Loss FunctionYou can check the supported loss functions with below code.frompytorch_optimizerimportget_supported_loss_functionssupported_loss_functions=get_supported_loss_functions()Loss FunctionsDescriptionOfficial CodePaperCitationLabel SmoothingRethinking the Inception Architecture for Computer Visionhttps://arxiv.org/abs/1512.00567citeFocalFocal Loss for Dense Object Detectionhttps://arxiv.org/abs/1708.02002citeFocal CosineData-Efficient Deep Learning Method for Image Classification Using Data Augmentation, Focal Cosine Loss, and Ensemblehttps://arxiv.org/abs/2007.07805citeLDAMLearning Imbalanced Datasets with Label-Distribution-Aware Margin Lossgithubhttps://arxiv.org/abs/1906.07413citeJaccard (IOU)IoU Loss for 2D/3D Object Detectionhttps://arxiv.org/abs/1908.03851citeBi-TemperedThe Principle of Unchanged Optimality in Reinforcement Learning Generalizationhttps://arxiv.org/abs/1906.03361citeTverskyTversky loss function for image segmentation using 3D fully convolutional deep networkshttps://arxiv.org/abs/1706.05721citeLovasz HingeA tractable surrogate for the optimization of the intersection-over-union measure in neural networksgithubhttps://arxiv.org/abs/1705.08790citeUseful ResourcesSeveral optimization ideas to regularize & stabilize the training. Most of the ideas are applied inRanger21optimizer.Also, most of the captures are taken fromRanger21paper.Adaptive Gradient ClippingGradient CentralizationSoftplus TransformationGradient NormalizationNorm LossPositive-Negative MomentumLinear learning rate warmupStable weight decayExplore-exploit learning rate scheduleLookaheadChebyshev learning rate schedule(Adaptive) Sharpness-Aware MinimizationOn the Convergence of Adam and BeyondImproved bias-correction in AdamAdaptive Gradient Norm CorrectionAdaptive Gradient ClippingThis idea originally proposed inNFNet (Normalized-Free Network)paper.AGC (Adaptive Gradient Clipping)clips gradients based on theunit-wise ratio of gradient norms to parameter norms.code :githubpaper :arXivGradient CentralizationGradient Centralization (GC)operates directly on gradients by centralizing the gradient to have zero mean.code :githubpaper :arXivSoftplus TransformationBy running the final variance denom through the softplus function, it lifts extremely tiny values to keep them viable.paper :arXivGradient NormalizationNorm Losspaper :arXivPositive-Negative Momentumcode :githubpaper :arXivLinear learning rate warmuppaper :arXivStable weight decaycode :githubpaper :arXivExplore-exploit learning rate schedulecode :githubpaper :arXivLookaheadksteps forward, 1 step back.Lookaheadconsisting of keeping an exponential moving average of the weights that is updated and substituted to the current weights everyklookahead steps (5 by default).Chebyshev learning rate scheduleAcceleration via Fractal Learning Rate Schedules.(Adaptive) Sharpness-Aware MinimizationSharpness-Aware Minimization (SAM) simultaneously minimizes loss value and loss sharpness.In particular, it seeks parameters that lie in neighborhoods having uniformly low loss.On the Convergence of Adam and BeyondConvergence issues can be fixed by endowing such algorithms with 'long-term memory' of past gradients.Improved bias-correction in AdamWith the default bias-correction, Adam may actually make larger than requested gradient updates early in training.Adaptive Gradient Norm CorrectionCorrecting the norm of a gradient in each iteration based on the adaptive training history of gradient norm.Frequently asked questionshereCitationPlease cite the original authors of optimization algorithms. You can easily find it in the above table!
If you use this software, please cite it below. Or you can get it from "cite this repository" button.@software{Kim_pytorch_optimizer_optimizer_2021,
    author = {Kim, Hyeongchan},
    month = jan,
    title = {{pytorch_optimizer: optimizer & lr scheduler & loss function collections in PyTorch}},
    url = {https://github.com/kozistr/pytorch_optimizer},
    version = {2.12.0},
    year = {2021}
}MaintainerHyeongchan Kim /@kozistr