Title: SLM: A Smoothed First-Order Lagrangian Method for Structured Constrained Nonconvex Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Functional constrained optimization

bilevel optimization

primal dual method

Lagrangian method
Abstract: Functional constrained optimization (FCO) has emerged as a powerful tool for solving various machine learning problems. However, with the rapid increase in applications of neural networks in recent years, it has become apparent that both the objective and constraints often involve nonconvex functions, which poses significant challenges in obtaining high-quality solutions. In this work, we focus on a class of nonconvex FCO problems with nonconvex constraints, where the two optimization variables are nonlinearly coupled in the inequality constraint. Leveraging the primal-dual optimization framework, we propose a smoothed first-order Lagrangian method (SLM) for solving this class of problems. We establish the theoretical convergence guarantees of SLM to the Karush-Kuhn-Tucker (KKT) solutions through quantifying dual error bounds. By establishing connections between this structured FCO and equilibrium-constrained nonconvex problems (also known as bilevel optimization), we apply the proposed SLM to tackle bilevel optimization oriented problems where the lower-level problem is nonconvex. Numerical results obtained from both toy examples and hyper-data cleaning problems demonstrate the superiority of SLM compared to benchmark methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a smoothed first-order Lagrange method(SLM) with the help of a primal-dual optimization framework for solving a class of structured nonconvex functional constrained optimization (FCO) problems. It can be used to deal with lower-level (LL) nonconvex bilevel optimization(BO) problems.  Theoretically, the paper proves that the algorithm converges to an $\\epsilon$-KKT points of these nonconvex constrained problems. Under some assumptions, the convergence rate of SLM to an $\\epsilon$-KKT point is $\\mathcal{O}(\\epsilon^{-1})$ for FCO problems. SLM can achieve a convergence rate of $\\mathcal{O}(\\epsilon^{-2})$ to solve nonconvex BO problems when the LL problem satisfies the PŁ condition.\tThe paper compares the proposed method with existing ones on a toy example and a data hyper-cleaning problem. In the toy example, SLM performs better than BOME and V-PBGD. As for the data hyper-cleaning problem, it performs better than the V-PBGD.\n\nSTRENGTHS:\nThis paper proposes a new algorithm to deal with a class of structured FCO problems, and also nonconvex BO problems. \nThe authors give convergence analysis when solving FCO and BO.\n\nWEAKNESSES:\n1. There are no formal definitions for $\\epsilon$-KKT points and $\\epsilon$-stationary points. This makes the paper difficult to follow. Moreover, when $\\delta=0$, I don't think the KKT condition for bilevel optimization is well defined.\n\n2. Assumption A4 needs more explanations.   Why can it hold for the case of $\\delta=0$? It will be beneficial if the authors can give such an application.\n\n1. There are numerous errors in the proof. Some are minor details that need to be carefully checked, but others lead to incorrect conclusions. \n   - In Line 532, inequality (59),  there should be a $p$ in the gradient Lipschitz constant of $K$ with respect to $y$. \n   - I cannot follow the proof of Lemma 6. Do we need a bound for $D(x^{r+1}, z^{r+1}; \\lambda^{r+1}) - D(x^{r}, z^{r}; \\lambda^{r+1})$?\n   - In Line 567, why is $\\lambda_+$  the maximizer in the expression of the objective function? \n\t\t\t\n\t\n4.\tThe author should clarify the theoretical novelty of their compared to reference [1].\n\t\t\n[1] Zhang, J., Xiao, P., Sun, R., & Luo, Z. (2020). A single-loop smoothed gradient descent-ascent algorithm for nonconvex-concave min-max problems. Advances in neural information processing systems, 33, 7377-7389.\n\nQUESTIONS:\n1. Equation (9): is there a missing \\{\\}?\n1.  In Line 617, why does inequality (111) hold?\n1.  In Line 721, E.1 Neural Network on MNIST Data Set. I am wondering why the performance is better when $\\delta$ is larger. You say \"The constraint tolerance parameter $\\delta$ and the dual variable implicitly play the tradeoff between the training and validation losses.\" Is that an explanation for this phenomenon? Could you explain it more clearly? \n\t\t\n1.  In the conditions of Theorem 1, step-sizes are chosen as $\\tau, \\eta, \\alpha = \\mathcal{O}(\\frac{1}{\\lambda^r})$. These step-sizes will be very small when $\\lambda$ is getting large. Will this setting of step-sizes affect the effectiveness of the algorithm?\n1.  In Figure 3,  why does the test accuracy of BOME get down after 400 iterations?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

comments: SUMMARY:
In this paper, a smooth first-order Lagrangian method (SLM) is proposed to solve a class of structured nonconvex functional constrained optimization problems, where the two optimization variable are nonlinearly coupled. The convergence rates are established by the dual error bounds. The proposed method is used to solve bilevel optimization problems with nonconvex lower-level problems. Numerical experiments are used to show that the proposed method outperforms the compared methods.

STRENGTHS:
(1) This method only uses the gradient information and obtains the optimal iteration complexity.
(2) SLM can be used to solve bilevel optimization problems with the lower-level problem satisfying the PL condition.

WEAKNESSES:
These are given in the question section.

QUESTIONS:
(1) This paper is able to handle nonconvexity and non-singleton at the lower level. However, the assumption that the PL condition is quite strong. If there exist multiple isolated minimizers, then such a condition does not hold.
Though [40] give a condition that guarantees the PL condition, overparameterization is not practical. Is there any example
that is practical and satisfies the PL condition?
(2) In the toy example, SLM is notably faster than V-PBGD method. However, both V-PBGD and SLM have the same complexity O(\eps^{-1}). Can the authors explain this?
(3) How to choose the parameters in Algorithm 1? Any suggestions to improve its performance? For many optimization algorithms, their performance highly depends on the choice of parameters. It is important to choose appropriate parameters when compared to other algorithms.

LIMITATIONS:
This paper discussed the limitations of the proposed algorithm, which is to only consider a single constraint problem.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies a class of structured nonconvex functional constrained optimization problems, where the functional constraint satisfies the PL condition w.r.t. one block of variables. The authors propose a smoothed first-order Lagrangian method (SLM) and establish the theoretical convergence rates of SLM to the KKT solutions. The authors demonstrate the application of the proposed SLM to bilevel optimization problems where the lower-level problem is nonconvex. Several experiments are provided.\n\nSTRENGTHS:\n(1) The paper is easy to follow. The structured constrained optimization and then the bilevel optimization has not been well studied. Finding an effective first-order method is both interesting and important. \n\n(2) The proposed SLM algorithm achieves a near-optimal convergence rate for finding the $\\epsilon$-KKT points under certain conditions. The three-layer structure of the propsoed method is interesting. The experiments in Fig. 1 and Fig. 2 support much better performance of SLM than other same-type methods in this area.\n\nWEAKNESSES:\nMajor comment 1: BOME in Table 1 also covers the nonconvexity and non-singleton case, as described in Section 4.3 of their paper. So SLM is not the first work where a first-order method is capable of finding the KKT points of BO problems in the presence of nonconvexity and non-singleton at the lower level, as claimed in the main contributions of this work.\n\nMajor comment 2: For Regularity Condition in Line 201, why does the KKT solution is unique? Is the feasible set convex, and if so, why? Why is the constant $\\underline{\\delta}$ independent of $(x,z)$? Is the Regularity Condition too strong?\n\nQUESTIONS:\nMajor comment 2 above is my first question. The following are others:\n\nQ1: In Line 268, why does the strong error bound immediately come from (14) and (16)?\n\nQ2: In Section 5.2, why do we set $\\delta=1.2$ in Line 308 for the data hyper-cleaning problem? Why not choose a smaller one? Is the performance sensitive to such parameter? \n\nQ3: Could the authors add the figure of the test accuracy and F1 score in the comparison between SLM and V-PBGD on the data hyper-cleaning problem?\n\nMinor comments:\n(1) In Line 124, is the ball open or closed?\n(2) In Equation (4a), why is $g^*(x)$ smooth? It is generally not smooth when non-singleton at the lower level. When $g(x,y)$ is strongly convex w.r.t. $y$, $g^*(x)$ is smooth and its gradient does NOT require the computation of the Jacobian matrix and inverse of Hessian by Danskin’s theorem.  \n(3) In Equation (5), $u_{t}^{r+1}$ should be $u_{t+1}^{r}$.\n(4) In Line 165, does $y^r$ be $y^{r+1}$?\n(5) In Line 1 of Algorithm 1, for $r=0,1,\\cdots,T-1$ do.  \n(6) In Corollary 1, the $\\epsilon$-stationary solution of the bilevel problem (2) does not defined.  \n(7) In Line 233, “result in 1” should be “result in Theorem 1”.\n(8) Correct the expression of $y^*(x^r, \\lambda^{r+1}; z^r)$ between Lines 249 and 250, and in Line 252. Also verify the expression of $y^*$ in other parts of the paper.\n(9) In Line 258, correct the definition of $\\lambda_{+}(z)$.  \n(10) In Line 322, “optimal solution” should be “optimal value”. \n(11) Please update the references.\n\nLIMITATIONS:\nThe authors discussed part of the limitations in the conclusion section of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper studies structured nonconvex functional constrained problems and proposed a smoothed first-order Lagrangian method to solve it. The proposed method decomposes the complexity of the problem by introducing a proximal term. Convergence of the proposed method is also studied. In general this is a well-written paper and the results are interesting.

STRENGTHS:
A new method to solve nonconvex functional constrained problem.

Convergence of the proposed method is analyzed,

WEAKNESSES:
More numerical experiments will make this paper stronger.

QUESTIONS:
Assumption A2 only requires the feasible set X to be compact. However, in the top layer, a projection operator is used. If the the set X is compact but the shape is very irregular, there can be multiple distinct points in X that is closest to x. In this case, how to define the projection operator? Will different choices of projection operator affect the convergence of the proposed method?

For the numerical experiments, if the feasible set X is irregular, say a star shape in R^2, how to apply the proposed method and what is the performance?

If the PL condition is not satisfied, numerically does the proposed method work?

Will the complexity of g or f affect the error and convergence rate theoretically and empirically? I do not see it from Theorem 1.

LIMITATIONS:
Limitations are discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11