Package: proxmin
Proximal MinimizationThe methods in this package provide solvers for constrained optimization problems. All of them use proximal operators to deal with non-smooth penalty functions.The algorithms:Proximal Gradient Method (PGM/ISTA): forward-backward splitting with a single smooth function with a Lipschitz-continuous gradient and a single (non-smooth) penalty function. Optional multi-block optimization, Nesterov acceleration (FISTA), and Barzilai-Borwein steps.Proximal Adam and derivatives (AdamX, AMSGrad, PAdam, NAdam): forward-backward splitting with adaptive gradient steps for single- and multi-block optimization.Alternating Direction Method of Multipliers (ADMM): Douglas-Rachford splitting for two potentially non-smooth functions. We use its linearized form to solve for linear mappings in the penalty functions.Simultaneous Direction Method of Multipliers (SDMM): Extension of linearized ADMM for several penalty functions.Block-Simultaneous Direction Method of Multipliers (bSDMM): Extension of SDMM to work with objective functions that are convex in several arguments. It's a proximal version of Block coordinate descent methods.Two-block PGM or bSDMM is used as backend solvers for Non-negative Matrix Factorization (NMF). As the algorithms allow any proxable function as constraint on each of the matrix factors, we prefer the term Constrained Matrix Factorization.Details can be found in thepaper"Block-Simultaneous Direction Method of Multipliers - A proximal primal-dual splitting algorithm for nonconvex problems with multiple constraints"by Fred Moolekamp and Peter Melchior.We ask that any published work that utilizes this package cites:@ARTICLE{proxmin,
    author="{Moolekamp}, Fred and {Melchior}, Peter",
    title="Block-simultaneous direction method of multipliers: a proximal primal-dual splitting algorithm for nonconvex problems with multiple constraints",
    journal="Optimization and Engineering",
    year="2018",
    month="Dec",
    volume=19,
    issue=4,
    pages={871-885},
    doi="10.1007/s11081-018-9380-y",
    url="https://doi.org/10.1007/s11081-018-9380-y"
    archivePrefix="arXiv",
    eprint={1708.09066},
    primaryClass="math.OC"
}Also, let us know (e.g.@peter_melchior), we're curious.Installation and Dependenciespip install proxminFor the latest development version, clone this repository and executepython setup.py install.The code works on python>2.7 and requires numpy and scipy. It is fully compatible with gradient computation byautograd.ApproachThe gradient-based methods PGM and Adam expect two callback function: one to compute the gradients, the other to compute step sizes. In the former case, the step sizes are bound between 0 and 2/L, where L is the Lipschitz constant of the gradient.The penalty functions are given as proximal mappings:X <- prox(X, step).Many proximal operators can be constructed analytically, see e.g.Parikh & Boyd (2014). We provide a number of common ones inproxmin.operators. An important class of constraints are indicator functions of convex sets, for which the proximal operator, given some pointX, returns the closest point toXin the Euclidean norm that is in the set.Example:find the minimum of a shifted parabola on the unit circle in 2DimportnumpyasnpimportproxmindX=np.array([1.,0.5])radius=1deff(X):"""Shifted parabola"""returnnp.sum((X-dX)**2,axis=-1)defgrad_f(X):return2*(X-dX)defstep_f(X,it=0):L=2.# Lipschitz constant of grad freturn1/Ldefprox_circle(X,step):"""Projection onto circle"""center=np.array([0,0])dX=X-center# exclude everything other than perimeter of circlephi=np.arctan2(dX[1],dX[0])returncenter+radius*np.array([np.cos(phi),np.sin(phi)])X=np.array([-1.,-1.])# or whereeverconverged,grad,step=proxmin.pgm(X,grad_f,step_f,prox=prox_circle)Since the objective function is smooth and there is only one constraint, one can simply perform a sequence offorward-backwardsteps: a step in gradient direction, followed by a projection onto the constraint subset. That is, in essence, the proximal gradient method.If both functions are not smooth, one can use ADMM. It therefore operates on two proxed functions. Unlike PGM, feasibility is only achieved at the end of the optimization and only within some error tolerance.Continuing the example above, the smooth function gets turned into a proxed function by performing the gradient step internally and returning the updated position:defprox_gradf(X,step):"""Proximal gradient step"""returnX-step*grad_f(X)converged=proxmin.admm(X,prox_gradf,step_f,prox_g=prox_circle,e_rel=1e-3,e_abs=1e-3)Constrained matrix factorization (CMF)Matrix factorization seeks to approximate a target matrixYas a product ofnp.dot(A,S). If those constraints are only non-negativity, the method is known as NMF.We have extended the capabilities by allowing for an arbitrary number of constraints to be enforced on either matrix factor:# PGM approach for each factorprox_A=...# a single constraint on A, solved by projectionprox_S=...# a single constraint on S, solved by projectionA0,S0=...# initializationproxmin.nmf.nmf(Y,A0,S0,prox_A=prox_A,prox_S=prox_S)# same with AdaProx-AMSGradproxmin.nmf.nmf(Y,A0,S0,prox_A=prox_A,prox_S=prox_S,algorithm=proxmin.algorithms.adaprox,scheme="amsgrad")# for multiple constraints, solved by ADMM-style splitproxs_g=[[...],# list of proxs for A[...]]# list of proxs for SA,S=proxmin.nmf.nmf(Y,A0,S0,algorithm=proxmin.algorithms.bsdmm,proxs_g=proxs_g)# or a combinationA,S=proxmin.nmf.nmf(Y,A0,S0,algorithm=proxmin.algorithms.bsdmm,prox_A=prox_A,prox_S=prox_S,proxs_g=proxs_g)