Package: dropgrad
DropGrad: A Simple Method for Regularization and Accelerated Optimization of Neural NetworksDropGrad: A Simple Method for Regularization and Accelerated Optimization of Neural NetworksInstallationRequirementsUsing pipUsing gitUsageBasic UsageUse with Learning Rate SchedulersVaryingdrop_rateperParameterDropGrad is a regularization method for neural networks that works by randomly (and independently) setting gradient values to zero before an optimization step. Similarly to Dropout, it has a single parameter,drop_rate, the probability of setting each parameter gradient to zero. In order to de-bias the remaining gradient values, they are divided by1.0 - drop_rate.To the best of my knowledge DropGrad is an original contribution. However, I have no plans of publishing a paper.
If indeed, it is an original method, please feel free to publish a paper about DropGrad. If you do so, all I ask is
that you mention me in your publication and cite this repository.InstallationThe PyTorch implementation of DropGrad can be installed simply using pip or by cloning the current GitHub repo.RequirementsThe only requirement for DropGrad is PyTorch. (Only versions of PyTorch >= 2.0 have been tested, although DropGrad should be compatible with any version of PyTorch)Using pipTo install using pip:pipinstalldropgradUsing gitgitclonehttps://github.com/dingo-actual/dropgrad.gitcddropgrad
python-mbuild
pipinstalldist/dropgrad-0.1.0-py3-none-any.whlUsageBasic UsageTo use DropGrad in your neural network optimization, simply import theDropGradclass to wrap your optimizer.fromdropgradimportDropGradWrapping an optimizer is similar to using a learning rate scheduler:opt_unwrapped=Adam(net.parameters(),lr=1e-3)opt=DropGrad(opt_unwrapped,drop_rate=0.1)During training, the application of DropGrad is automatically handled by the wrapper. Simply call.step()on
the wrapped optimizer to apply DropGrad then.zero_grad()to reset the gradients.opt.step()opt.zero_grad()Use with Learning Rate SchedulersIf you use a learning rate scheduler as well as DropGrad, simply pass the base optimizer to both the DropGrad
wrapper and the learning rate scheduler:opt_unwrapped=Adam(net.parameters(),lr=1e-3)lr_scheduler=CosineAnnealingLR(opt_unwrapped,T_max=100)opt=DropGrad(opt_unwrapped,drop_rate=0.1)During the training loop, you call.step()on the DropGrad wrapper before calling.step()on the learning rate
scheduler, similarly to using an optimizer without DropGrad:forepoch_ninrange(n_epochs):forx_batch,y_batchindataloader:pred_batch=net(x_batch)loss=loss_fn(pred_batch,y_batch)loss.backward()opt.step()opt.zero_grad()lr_scheduler.step()Varyingdrop_rateperParameterDropGrad allows the user to set a different drop rate for eachParameterunder optimization. To do this, simply
pass a dictionary mappingParametersto drop rates to thedrop_rateargument of the DropGrad wrapper. If a dictionary
is passed to DropGrad during initialization, all optimizedParameters that are not present in that dictionary will have
the drop rate passed to the DropGrad wrapper at initialization (ifdrop_rate=Nonethen drop grad simply won't be applied
toParameters that are not present in the dictionary).The example below will apply adrop_rateof 0.1 to all optimized weights and adrop_rateof 0.01 to all optimized biases,
with no DropGrad applied to any other optimizedParameters:drop_rate_weights=0.1drop_rate_biases=0.01params_weights=[pforname,pinnet.named_parameters()ifp.requires_gradand'weight'inname]params_biases=[pforname,pinnet.named_parameters()ifp.requires_gradand'bias'inname]param_drop_rates={p:drop_rate_weightsforpinparams_weights}param_drop_rates.update({p:drop_rate_biasesforpinparams_biases})opt_unwrapped=Adam(net.parameters(),lr=1e-3)opt=DropGrad(opt_unwrapped,drop_rate=None,params=param_drop_rates)