Package: expectigrad
Expectigrad: Fast Stochastic Optimization with Robust Convergence PropertiesExpectigrad is a first-order stochastic optimization method that fixes theknown divergence issueof Adam, RMSProp, and related adaptive methods while offering better performance on
well-known deep learning benchmarks.Expectigrad introduces two innovations to adaptive gradient methods:Arithmetic RMS:Computes the true RMS instead of an exponential moving average (EMA).
This makes Expectigrad more robust to divergence and, in theory, less susceptible to
gradient noise.Outer momentum:Applies momentumafteradapting the step sizes, not
before.
This reduces bias in the updates by preserving thesuperposition property.Seethe paperfor more details.Pytorch, TensorFlow 1.x, and TensorFlow 2.x are all supported.
Seeinstallationandusagebelow to get started.PseudocodeCitingIf you use this code for published work, please citethe original paper:@article{daley2020expectigrad,
  title={Expectigrad: Fast Stochastic Optimization with Robust Convergence Properties},
  author={Daley, Brett and Amato, Christopher},
  journal={arXiv preprint arXiv:2010.01356},
  year={2020}
}InstallationUse pip to quickly install Expectigrad:pip install expectigradOr you can clone this repository and install manually:git clone https://github.com/brett-daley/expectigrad.git
cd expectigrad
python setup.py -e .UsagePytorch and both versions of TensorFlow are supported.
Refer to the code snippets below to instantiate the optimizer for your deep learning framework.Pytorchimportexpectigradexpectigrad.pytorch.Expectigrad(params,lr=0.001,beta=0.9,eps=1e-8,sparse_counter=True)Argsparams(iterable)Iterable of parameters to optimize or dicts defining parameter groups.lr(float)The learning rate, a scale factor applied to each optimizer step. Default:0.001beta(float)The decay rate for Expectigrad's bias-corrected, "outer" momentum. Must be in the interval [0, 1). Default:0.9eps(float)A small constant added to the denominator for numerical stability. Must be greater than 0. Default:1e-8sparse_counter(bool)If True, Expectigrad's counter increments only where the gradient is nonzero. If False, the counter increments unconditionally. Default:TrueTensorflow 1.ximportexpectigradexpectigrad.tensorflow1.ExpectigradOptimizer(learning_rate=0.001,beta=0.9,epsilon=1e-8,sparse_counter=True,use_locking=False,name='Expectigrad')Argslearning_rateThe learning rate, a scale factor applied to each optimizer step. Can be a float,tf.keras.optimizers.schedules.LearningRateSchedule,Tensor, or callable that takes no arguments and returns the value to use. Default:0.001beta(float)The decay rate for Expectigrad's bias-corrected, "outer" momentum. Must be in the interval [0, 1). Default:0.9epsilon(float)A small constant added to the denominator for numerical stability. Must be greater than 0. Default:1e-8sparse_counter(bool)If True, Expectigrad's counter increments only where the gradient is nonzero. If False, the counter increments unconditionally. Default:Trueuse_locking(bool)If True, apply use locks to prevent concurrent updates to variables. Default:Falsename(str)Optional name for the operations created when applying gradients. Default:'Expectigrad'Tensorflow 2.ximportexpectigradexpectigrad.tensorflow2.Expectigrad(learning_rate=0.001,beta=0.9,epsilon=1e-8,name='Expectigrad',**kwargs)Argslearning_rateThe learning rate, a scale factor applied to each optimizer step. Can be a float,tf.keras.optimizers.schedules.LearningRateSchedule,Tensor, or callable that takes no arguments and returns the value to use. Default:0.001beta(float)The decay rate for Expectigrad's bias-corrected, "outer" momentum. Must be in the interval [0, 1). Default:0.9epsilon(float)A small constant added to the denominator for numerical stability. Must be greater than 0. Default:1e-8sparse_counter(bool)If True, Expectigrad's counter increments only where the gradient is nonzero. If False, the counter increments unconditionally. Default:Truename(str)Optional name for the operations created when applying gradients. Default:'Expectigrad'**kwargsKeyword arguments. Allowed to be {clipnorm,clipvalue,lr,decay}.clipnormis gradient clipping by norm;clipvalueis gradient clipping by value;decayis included for backward compatibility to allow time inverse decay of learning rate;lris included for backward compatibility, recommended to uselearning_rateinstead.