Package: pytorch-minimize
PyTorch MinimizeFor the most up-to-date information on pytorch-minimize, see the docs site:pytorch-minimize.readthedocs.ioPytorch-minimize represents a collection of utilities for minimizing multivariate functions in PyTorch.
It is inspired heavily by SciPy'soptimizemodule and MATLAB'sOptimization Toolbox.
Unlike SciPy and MATLAB, which use numerical approximations of function derivatives, pytorch-minimize usesrealfirst- and second-order derivatives, computed seamlessly behind the scenes with autograd.
Both CPU and CUDA are supported.Author: Reuben FeinmanAt a glance:importtorchfromtorchminimportminimizedefrosen(x):returntorch.sum(100*(x[...,1:]-x[...,:-1]**2)**2+(1-x[...,:-1])**2)# initial pointx0=torch.tensor([1.,8.])# Select from the following methods:#  ['bfgs', 'l-bfgs', 'cg', 'newton-cg', 'newton-exact',#   'trust-ncg', 'trust-krylov', 'trust-exact', 'dogleg']# BFGSresult=minimize(rosen,x0,method='bfgs')# Newton Conjugate Gradientresult=minimize(rosen,x0,method='newton-cg')# Newton Exactresult=minimize(rosen,x0,method='newton-exact')Solvers:BFGS, L-BFGS, Conjugate Gradient (CG), Newton Conjugate Gradient (NCG), Newton Exact, Dogleg, Trust-Region Exact, Trust-Region NCG, Trust-Region GLTR (Krylov)Examples:See theRosenbrock minimization notebookfor a demonstration of function minimization with a handful of different algorithms.Install:git clone https://github.com/rfeinman/pytorch-minimize.git
cd pytorch-minimize
pip install -e .MotivationAlthough PyTorch offers many routines for stochastic optimization, utilities for deterministic optimization are scarce; only L-BFGS is included in theoptimpackage, and it's modified for mini-batch training.MATLAB and SciPy are industry standards for deterministic optimization.
These libraries have a comprehensive set of routines; however, automatic differentiation is not supported.*
Therefore, the user must provide explicit 1st- and 2nd-order gradients (if they are known) or use finite-difference approximations.The motivation for pytorch-minimize is to offer a set of tools for deterministic optimization with automatic gradients and GPU acceleration.__*MATLAB offers minimal autograd support via the Deep Learning Toolbox, but the integration is not seamless: data must be converted to "dlarray" structures, and only asubset of functionsare supported.
Furthermore, derivatives must still be constructed and provided as function handles.
Pytorch-minimize uses autograd to compute derivatives behind the scenes, so all you provide is an objective function.LibraryThe pytorch-minimize library includes solvers for general-purpose function minimization (unconstrained & constrained), as well as for nonlinear least squares problems.1. Unconstrained MinimizersThe following solvers are available forunconstrainedminimization:BFGS/L-BFGS.BFGS is a cannonical quasi-Newton method for unconstrained optimization. I've implemented both the standard BFGS and the "limited memory" L-BFGS. For smaller scale problems where memory is not a concern, BFGS should be significantly faster than L-BFGS (especially on CUDA) since it avoids Python for loops and instead uses pure torch.Conjugate Gradient (CG).The conjugate gradient algorithm is a generalization of linear conjugate gradient to nonlinear optimization problems. Pytorch-minimize includes an implementation of the Polak-Ribiére CG algorithm described in Nocedal & Wright (2006) chapter 5.2.Newton Conjugate Gradient (NCG).The Newton-Raphson method is a staple of unconstrained optimization. Although computing full Hessian matrices with PyTorch's reverse-mode automatic differentiation can be costly, computing Hessian-vector products is cheap, and it also saves a lot of memory. The Conjugate Gradient (CG) variant of Newton's method is an effective solution for unconstrained minimization with Hessian-vector products. I've implemented a lightweight NewtonCG minimizer that uses HVP for the linear inverse sub-problems.Newton Exact.In some cases, we may prefer a more precise variant of the Newton-Raphson method at the cost of additional complexity. I've also implemented an "exact" variant of Newton's method that computes the full Hessian matrix and uses Cholesky factorization for linear inverse sub-problems. When Cholesky fails--i.e. the Hessian is not positive definite--the solver resorts to one of two options as specified by the user: 1) steepest descent direction (default), or 2) solve the inverse hessian with LU factorization.Trust-Region Newton Conjugate Gradient.Description coming soon.Trust-Region Newton Generalized Lanczos (Krylov).Description coming soon.Trust-Region Exact.Description coming soon.Dogleg.Description coming soon.To access the unconstrained minimizer interface, use the following import statement:from torchmin import minimizeUse the argumentmethodto specify which of the afformentioned solvers should be applied.2. Constrained MinimizersThe following solvers are available forconstrainedminimization:Trust-Region Constrained Algorithm.Pytorch-minimize includes a single constrained minimization routine based on SciPy's 'trust-constr' method. The algorithm accepts generalized nonlinear constraints and variable boundries via the "constr" and "bounds" arguments. For equality constrained problems, it is an implementation of the Byrd-Omojokun Trust-Region SQP method. When inequality constraints are imposed, the trust-region interior point method is used. NOTE: The current trust-region constrained minimizer is not a custom implementation, but rather a wrapper for SciPy'soptimize.minimizeroutine. It uses autograd behind the scenes to build jacobian & hessian callables before invoking scipy. Inputs and objectivs should use torch tensors like other pytorch-minimize routines. CUDA is supported but not recommended; data will be moved back-and-forth between GPU/CPU.To access the constrained minimizer interface, use the following import statement:from torchmin import minimize_constr3. Nonlinear Least SquaresThe library also includes specialized solvers for nonlinear least squares problems.
These solvers revolve around the Gauss-Newton method, a modification of Newton's method tailored to the lstsq setting.
The least squares interface can be imported as follows:from torchmin import least_squaresThe least_squares function is heavily motivated by scipy'soptimize.least_squares.
Much of the scipy code was borrowed directly (all rights reserved) and ported from numpy to torch.
Rather than have the user provide a jacobian function, in the new interface, jacobian-vector products are computed behind the scenes with autograd.
At the moment, only the Trust Region Reflective ("trf") method is implemented, and bounds are not yet supported.ExamplesTheRosenbrock minimization tutorialdemonstrates how to use pytorch-minimize to find the minimum of a scalar-valued function of multiple variables using various optimization strategies.In addition, theSciPy benchmarkprovides a comparison of pytorch-minimize solvers to their analogous solvers from thescipy.optimizelibrary.
For those transitioning from scipy, this script will help get a feel for the design of the current library.
Unlike scipy, jacobian and hessian functions need not be provided to pytorch-minimize solvers, and numerical approximations are never used.For constrained optimization, theadversarial examples tutorialdemonstrates how to use the trust-region constrained routine to generate an optimal adversarial perturbation given a constraint on the perturbation norm.Optimizer APIAs an alternative to the functional API, pytorch-minimize also includes an "optimizer" API based on thetorch.optim.Optimizerclass.
To access the optimizer class, import as follows:from torchmin import Minimizer