Automatic and Harmless  Regularization  with Constrained and Lexicographic Optimization: A Dynamic Barrier Approach

1 Introduction . Although machine learning ( ML ) has been typically conceptualized as optimizing a single objective function , most practical ML tasks actually involve trading off two or more objective functions , such as the data fitness function vs. a regularization or auxiliary loss . Let f and g be two objectives functions of interest on Rd . A principled way to trade-off f and g is through constrained optimization : min ✓2Rd f ( ✓ ) s.t . g ( ✓ ) c , ( 1 ) where c is a threshold . When varying c , solutions of ( 1 ) cover the Pareto points of ( f , g ) , providing different trade-offs on f , g. Moreover , ( 1 ) naturally arises when g is an important constraint that should be controlled explicitly , e.g. , in terms of safety , fairness , and other trustworthy measures . However , compared to unconstrained optimization , constrained optimization ( 1 ) has been much less widely used in practical machine learning . In fact , perhaps the most common approach to handling ( 1 ) is to transform it into the unconstrained optimization of the linear combination of f and g , min ✓2Rd f ( ✓ ) + g ( ✓ ) , ( 2 ) where the trade-off between f and g is controlled by a weight coefficient , instead of the threshold parameter c. A folklore argument is that can be viewed as the Lagrange multiplier of ( 1 ) , and hence ( 1 ) can be mapped into ( 2 ) with properly selected . Unfortunately , despite being broadly used , ( 2 ) is insufficient to fully replace ( 1 ) and suffers from a number of disadvantages in both theory and practice : 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . • Interpretability . provides a less intuitive parameter to select than c , since it depends on the relative scale of f and g , whose range is problem-dependent and needs to be optimized as a hyperparameter . In comparison , the threshold c can be specified as a tolerance parameter when g is a metric that users want to control explicitly . • Pareto Coverage . When f , g are non-convex functions , ( 1 ) provides a strictly broader class of problems than ( 2 ) , because for some c , there may exist no 2 R , such that ( 1 ) and ( 2 ) are equivalent . From the multi-objective optimization ( MOO ) perspective , this is related to the fact that ( 2 ) can only capture the convex envelop of the Pareto front while varying , while ( 1 ) provides all Pareto optimal points by varying c. • Invariance . The constrained optimization ( 1 ) is invariant to arbitrary monotonically increasing maps on f and g. That is , let and 0 be two monotonically increasing maps , then we obtain the equivalent problem if we replace f with f , and g with 0 g , and c with 0 ( c ) . In comparison , applying nonlinear transforms on f and g in ( 2 ) yields fundamentally different problems . • Harmless Regularization . In many practical cases , one of the objectives ( say f ) is of secondary importance w.r.t . the other one ( say g ) , in sense that we are interested in minimizing f only when g is fully optimized ; this can be formulated as the following lexicographic optimization problem : min ✓2Rd f ( x ) s.t . g ( ✓ ) g⇤ where g⇤ : = inf ✓2Rd g ( ✓ ) , ( 3 ) where we minimize f inside the optimum set { ✓ : g ( ✓ ) g⇤ } of g. This can be viewed as a special case of ( 1 ) with the minimum threshold c = g⇤ ( which is unknown before hand ) , and can not be captured by ( 2 ) with a finite and fixed . Compared with constrained optimization ( 1 ) , the lexicographic ( or simply lexico ) problem eliminates the need of setting the threshold c , and hence provides a handy , coefficient-free approach for incorporating auxiliary losses without hurting the main loss . As our empirical results show , this finds useful in numerous deep learning tasks , e.g. , fairness ML and semi-supervised learning . Our Contributions We conduct a joint study on the constrained and lexico optimization problems . We propose a simple and general local descent algorithm ( Algorithm 1 ) which offers a unified solution to both problems . By placing a dynamic barrier constraint on the search direction at each iteration , the algorithm finds a trajectory towards the optimal solution by properly balancing f and g with an adaptive combination coefficient coefficient t ; as seen in Eq 4 , the t is decided with a simple formula by the inner prod uct between the objective and constraint gradientsrf and rg . In Section 3.1-3.3 , we study the continuous-time convergence of the method for general non-convex functions and both the constrained and lexico optimization cases . In Section 3.4 , we elaborate the ( often overlooked ) fact that methods based on optimizing the linear combination ( 2 ) is fundamentally unsound as an approach to constrained optimization ( 1 ) with non-convex functions and non-zero duality gap . In Section 4 , we empirically show that our method provides an efficient approach to approximating Pareto sets and incorporating side information in a variety of deep learning tasks . Algorithm 1 Dynamic Barrier Gradient Descent for ( 1 ) and ( 3 ) Choose stepsize { ✏t } and the dynamic barrier function in ( 8 ) ( use ↵ = = 1 by default ; set ĝ = c for constrained optimization ( 1 ) and ĝ to be any value no larger than g⇤ for lexicographic optimization ( 3 ) ) . for iteration t do ✓t+1 ✓t ✏t ( rf ( ✓t ) + trg ( ✓t ) ) , t = max ( ✓t ) rf ( ✓t ) > rg ( ✓t ) krg ( ✓t ) k2 , 0 ! . ( 4 ) end for 2 Main Method . We introduce the main algorithm 1 , a simple gradient-based method for solving both constrained optimization ( 1 ) and lexicographic optimization ( 3 ) in a unified way . The method performs iterative updates of form ✓t+1 ✓t ✏tvt , ( 5 ) where ✏t 0 is a step size and vt 2 Rd is an update direction to be chosen to balance the minimization of f and constraint satisfaction on g. The vt is designed to satisfy the following desiderata : 1 ) When the constraint is not satisfied ( i.e. , g ( ✓t ) > c ) , we should mainly focus on decreasing g to meet the constraint as fast as possible ; meanwhile , f should act as a secondary objective in this phase , meaning that f should be minimized upto the degree that it does not hurt the descent of g. 2 ) When the constraint is met ( i.e. , g ( ✓t ) c ) , we should prioritize to minimize f , which is made possible in general only if we allow g to increase . However , the increasing rate of g should be properly controlled , so that ✓ stays inside or nearby the feasible set while we minimize f . It turns out that both properties can be achieved if we select vt by the following optimization : vt = argmin v2Rd n krf ( ✓t ) vk2 s.t . rg ( ✓t ) > v ( ✓t ) o , ( 6 ) where we want vt to be as close to rf ( ✓t ) as much as possible ( and hence decrease f ) , but subject to a lower bound on the inner product of rg ( ✓t ) and vt to ensure that the change of g is properly controlled by the location of ✓t ; here the lower bound : Rd ! R is a dynamic barrier function which trade-offs loss minimization with constraint satisfaction by controlling the inner product between rg ( ✓t ) and vt. To achieve the desiderata on vt , we should let ( ✓t ) have the same sign as g ( ✓t ) c , so that the constraint { ✓ : g ( ✓ ) c } is equivalent to { ✓ : ( ✓ ) 0 } , that is , sign ( ( ✓ ) ) = sign ( g ( ✓ ) c ) , ( 7 ) where sign ( x ) = x|x| for x 6= 0 and sign ( 0 ) = 0 . In this way , when the step size ✏t is sufficiently small , we have the following properties that will be studied theoretically in Section 3 : 1 ) When ✓t is outside of the feasible set ( g ( ✓t ) > c ) , the constraint is rg ( ✓t ) > vt ( ✓t ) > 0 , which ensures that g decreases strictly outside of the feasible set . 2 ) When ✓t is on the boundary ( g ( ✓t ) = c ) , the constrain reduces to rg ( ✓t ) > vt ( ✓t ) = 0 , meaning that we want to decrease f subject to that g does not increase . 3 ) When ✓t is in the interior of the feasible set ( g ( ✓t ) < c ) , the lower bound on rg ( ✓t ) > vt is negative , allowing g to increase . In this case , the optimal solution vt in ( 6 ) can be shown to have a positive inner product with rf ( ✓t ) unless rf ( ✓t ) = 0 , which means that the algorithm monotonically decreases f until a local optimum is reached . Computing t Since ( 6 ) is a simple quadratic convex programming , it is easy to see that the solution is vt = rf ( ✓t ) + trg ( ✓t ) where t solves the following dual problem of ( 6 ) : t = argmin 0 n krf ( ✓t ) + rg ( ✓t ) k2 ( ✓t ) o = max ( ✓t ) rf ( ✓t ) > rg ( ✓t ) krg ( ✓t ) k2 , 0 ! . Practical Choice of There are a broad range of ’ s that satisfy ( 7 ) . Two particularly simple choices stand out which are suitable for constrained and lexico optimization , respectively : 1 ) ( ✓t ) = ↵ ( g ( ✓t ) c ) , where ↵ > 0 . This puts a strong requirement on descending the constraint g when g ( ✓t ) c is large and positive , and it gives higher flexibility for minimizing f when g ( ✓t ) c is negative as we move towards the interior of the feasible set . In this case , the inner product constraint in ( 6 ) can be viewed as a linearization of the original constraint ( g ( ✓ ) c ) in ( 1 ) : g ( ✓t ✏tvt ) ⇡ g ( ✓t ) ✏trg ( ✓t ) > vt c , with ✏t = 1/↵ . Hence , the algorithm in this case can be viewed as a simple variant of sequential quadratic programming ( SQP ) ( e.g. , Nocedal & Wright , 2006 ) when we use an identity matrix to approximate the Hessian of f . 2 ) ( ✓t ) = krg ( ✓t ) k2 , where > 0 . Assume that rg ( ✓t ) = 0 implies global minimum of g , then this satisfies the sign condition ( 7 ) for the lexico case ( 3 ) , without requiring to estimate the unknown threshold g⇤ . Because ( ✓t ) 0 in this case , it allows us to strictly decrease g until ✓t reaches an optimum of g. Meanwhile , f acts as the secondary objective that is decreased when it does not interfere with the descent of g. In practice , when g has multiple local minima , then the algorithm would minimize f ( ✓ ) subject to g ( ✓ ) g ( ✓̂⇤ ) , where ✓̂⇤ is a local minimum of g that the algorithm encounters . Combining the two cases into a single , we propose to use ( ✓ ) = min ⇣ ↵ ( g ( ✓ ) ĝ ) , krg ( ✓ ) k2 ⌘ , ( 8 ) where we take ĝ = c if the goal is to solve the constrained optimization ( 1 ) with threshold c , and take ĝ to be any lower bound of g⇤ ( i.e. , ĝ g⇤ ) for solving the lexicographic optimization problem ( 3 ) ; for example , if g is a norm or mean square loss , then ĝ = 0 is a natural estimation of the lower bound . If no lower bound can be estimated , we can take ĝ = 1 , and ( 8 ) reduces back to ( ✓ ) = krg ( ✓ ) k2 . The choice of the hyper-parameters ↵ , controls the speed of constraint satisfaction vs. loss minimization . In our experiments , we find that it is sufficient to simply set ↵ = = 1 in most cases so that the algorithm introduces no extra hyper-parameters compared with vanilla gradient descent . See Algorithm 1 for a summary of the algorithm procedure . Profiling Pareto Set The Pareto set of ( f , g ) is the set of points ✓ for which there exists no ✓0 , such that f ( ✓0 ) f ( ✓ ) , g ( ✓0 ) g ( ✓ ) , and ( f ( ✓0 ) , g ( ✓0 ) ) 6= ( f ( ✓ ) , g ( ✓ ) ) . The combination of the constrained and lexico optimization provides an automatic approach to obtain uniformly distributed points from the Pareto set of ( f , g ) . To do so , we start with solving the lexico problem ( 3 ) , which corresponds to one of the endpoints of the Pareto set ; we then walk through the Pareto points by solving the constrained optimization ( 1 ) with c increasing linearly from the g⇤ estimated from lexico optimization . Optionally , we can also solve the mirror version min✓ g ( ✓ ) s.t . f ( ✓ ) f⇤ : = inf✓0 f ( ✓0 ) to obtain the other end point , and then solve ( 1 ) with c on a grid between the two end points .

Summary: In this paper, the authors introduce a unified first-order approach for non-convex constrained optimization with a single constraint, $\min_{\theta \in \mathbb{R}^d} f(\theta) \text{ s.t. } g(\theta) \le c $, and simple bilevel programming (SBP), $\min_{\theta \in \mathbb{R}^d} f(\theta) \text{ s.t. } \theta \in \mathrm{argmin}_{\tilde{\theta} \in \mathbb{R}^d} g(\tilde{\theta}) $.  In a nutshell, the approach relies on the introduction of a *dynamic barrier* $\phi(\theta)$ whose sign coincides with that of the constraint in canonical form, and that is used to bound from below the rate of decrease of $g$. Precisely, the authors propose to use as update direction the vector $v_t$ closest in $L_2$ norm to $\nabla f$ among all directions whose inner product with $\nabla g$ is at least as large as $\phi(\theta)$. The resulting quadratic program can be solved in closed form, leading to $v_t = \nabla f(\theta_t) + \lambda_t \nabla g(\theta_t)$ where $\lambda_t$ can be obtained in closed form as well as the maximizer of the dual. In other words, the resulting algorithm can be informally understood as an adaptive variant of unconstrained minimization over a linear combination of $f$ and $g$, $\min_{\theta \in \mathbb{R}^d} f(\theta) + \lambda g(\theta)$, where the multiplier $\lambda$ is not fixed throughout but adaptively estimated for each $\theta$. Finally, the authors propose a unified choice for the dynamic barrier, $\phi(\theta) = \min(\alpha(g(\theta) - \hat{g}), \beta \vert\vert \nabla g(\theta) \vert\vert^{2})$, where $\hat{g}$ is either set to $c$ in the case of constrained minimization or a lower bound on $g^{*}$ for SBP, and $\alpha, \beta$ are hyperparameters that the authors suggest fixing to $1$ in practice.  The authors provide theoretical guarantees for both setups. For constrained optimization, it is shown that under certain regularity conditions, a continuous-time version of the algorithm with make the penalty function $P_{\mu}(\theta) = f(\theta) + \mu \max(g(\theta) - c, 0)$ decrease monotonically and, at convergence, the iterates $(\theta^{\*}, \lambda^{\*})$ will satisfy the KKT conditions for the problem, with “KKT suboptimality” decreasing at a linear rate. Analogously, the continuous-time version of the algorithm is shown to converge to a local minimum of $g$ with the resulting iterates $\theta_t$ being approximate minimizers of a sequence of constrained optimization problems with constraint $c_t := g(\theta_t)$. Moreover, these theoretical results are extended to the discrete-time setting in the Appendix.  Finally, the authors illustrate their approach on a suite of real-world tasks (sparse representation learning, fairness regularization, $L_2$ regularization, semi-supervised learning and semantic segmentation) as well as a toy 2D example. In these, they primarily compare their method against a linear combination baseline treating $\lambda$ as a (non-adaptive) hyperparameter with encouraging results.