Title: Beyond Value Functions: Single-Loop Bilevel Optimization under Flatness Conditions
Venue: NeurIPS 2025
Accepted: yes
Keywords: bilevel optimization

Holder's condition

value function

penalty bilevel
Abstract: Bilevel optimization, a hierarchical optimization paradigm, has gained significant attention in a wide range of practical applications, notably in the fine-tuning of generative models. However, due to the nested problem structure, most existing algorithms require either the Hessian vector calculation or the nested loop updates, which are computationally inefficient in large language model (LLM) fine-tuning. In this paper, building upon the fully first-order penalty-based approach, we propose an efficient value function-free (\textsf{PBGD-Free}) algorithm that eliminates the loop of solving the lower-level problem and admits fully single-loop updates. Inspired by the landscape analysis of representation learning-based LLM fine-tuning problem, we propose a relaxed flatness condition for the upper-level function and prove the convergence of the proposed value-function-free algorithm. We test the performance of the proposed algorithm in various applications and demonstrate its superior computational efficiency over the state-of-the-art bilevel methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes PBGD-Free fully single-loop algorithm for bilevel optimization problems that avoids costly value-function computations.  PBGD-Free simplifies the process by removing the need to solve the inner optimization problem at every iteration. The authors show that while the method may fail to converge under standard Lipschitz assumptions, introducing a relaxed flatness condition—modeled by a H\\o lder-like inequality—ensures convergence to near-optimal solutions. They theoretically establish that under this flatness condition, PBGD-Free achieves favorable iteration complexity and low memory costs. Extensive experiments on parameter-efficient fine-tuning (PEFT) of LLMs demonstrate that PBGD-Free matches or outperforms state-of-the-art baselines in both accuracy and efficiency, validating the proposed theoretical framework.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n1. This work proposes a relaxed assumption for Lipschitz continuity and provides a convergence analysis. \n2. The method empirically performs well.\n\nWeakness: \n1.  The paper is not very clearly written. DPO and its notations are not clearly introduced. For example, it would be clearer if the meanings of $x$ and $y$ in the model were explicitly specified. Additionally, it is unclear what $\\sigma$ represents in equation (13), and how $\\nabla f_{\\text{DPO}}$ is derived.\n2. The assumption on $\\delta$ in the flatness condition is strong. It requires $\\delta$ to be smaller than $\\epsilon$ to allow for a smaller choice of $\\lambda$. More importantly, Theorem 3 requires $\\delta(x_t)$ to be summable over $t$. If $T$ goes to infinity, then $\\delta(x_T)$ must tend to zero.\n4.  The complexity analysis still reveals that the proposed method may not converge. In the main result, Theorem 3, the proposed method includes an extra term in the complexity bound that depends on $\\delta$. Therefore, although the experiments show promising empirical results, the method lacks theoretical convergence guarantees, which makes it less satisfactory.\n\nQUESTIONS:\n1. Line 38, it would be clearer if the authors explained what the smoothness constant refers to. Do they mean the Lipschitz constant of the gradient?\n2. Line 42 $F$ should be $\\tilde F$?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDue to the nested problem structure, most existing algorithms require either the Hessian vector calculation or the nested loop updates, which are computationally inefficient in large language model (LLM) fine-tuning. Building upon the fully first-order penalty-based approach, this paper proposes an efficient value function-free (PBGD-Free) algorithm that eliminates the loop of solving the lower-level problem and admits fully single-loop updates.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**: 1. This paper proposes PBGD-Free, a computationally efficient fully-single-loop, value-function-free, first-order algorithm.\n\n2. This paper introduces a Holder-alike condition to describe the flatness of $f$, which relaxes the standard $l_{f,0}$-Lipschitz continuity assumption when $l_{f,0}$ is small.\n\n3. This paper is well-written.\n\n4. The figures in this paper look very good, for example, Figure 2.\n\n\n**Weaknesses**: 1. Assumption 2 imposes a Lipschitz condition on $\\nabla^2 f$, which appears overly restrictive. Standard requirements in the literature typically do not necessitate such an assumption.\n\n2. Furthermore, the $\\mu$-PL condition imposed on $c f(x,y) + g(x,y)$ in Assumption 2 seems excessively strong, as the PL condition for the sum of functions is rarely encountered in prior work. Additionally, the dynamic nature of $c$ further complicates the satisfaction of this condition. While the authors argue that the PL condition is weaker than strong convexity, they should provide more concrete examples to substantiate this claim, particularly when $g$ lacks strong convexity.\n\n3. The format of references is not consistent.\n\nQUESTIONS:\n1. In Table 1, what do $d_x$ and $d_y$ represent?\n\n2. Has Definition 1 been previously used in the literature? If so, relevant citations should be provided.\n\n3. Observation 2 should explicitly state the smoothness conditions imposed on $f$.\n\n4. In Lemma 3, the notation $x^*$ always denotes the optimal solution, which may cause ambiguity. A distinct notation should be introduced. Moreover, Additionally, the range $\\alpha\\in (1,1.5]$ appears restrictive—does this hold for all functions, or could $\\alpha = 2$ be feasible?\n\n\n5. The algorithm should be reformulated as a single-loop structure, eliminating the dependence on $K$.\n\n6. In line 279, what is $l_F^{-1}$?\n\n\n7. In Equation (17), can $\\delta$ be made arbitrarily small? If not, the algorithm’s output accuracy would remain fixed regardless of $T$, which is undesirable.\n\n\nIf the author can provide sufficient reasons or evidence to prove the effectiveness of the proposed algorithm and the adopted assumptions, I'm willing to increase my score.\n\nLIMITATIONS:\nSee **Weaknesses** and **Questions**.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method called PBGD-Free for bilevel optimization under a flatness condition. Compared with prior methods such as F$^2$SA that needs to solve two lower-level problems of the same condition number, the PBGD-Free only solves one, which make it more efficient in practice. Another benefit of the flatness condition is that it allows using single-loop updates,  while prior works without this condition requires $O(\\log \\epsilon^{-1})$ number of inner loops.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength** Each claim in the paper is well-supported by examples, theorems and experiments. The paper introduces a novel method that is more efficient in practice and provide conditions when the method can converge.\n\n**Weakness** \n\n1. The story of the article is self-consistent, but it doesn't completely convince me. For example, maybe I can come up with another simple explanation: in practice especially neural network training experiments, the network is over-parameterized such that $g^\\ast(x)= 0 $. Consequently, the penalty function $ f(x,y) + \\gamma (g(x,y) - g^\\ast(x))$ is equivalent to  $f(x,y) + \\gamma (g(x,y) $. Therefore, you can also skip the inner loop to approximate $g^\\ast(x)$. I think my story is also self-consistent and can be supported by experiments. \n\n2. I can not understand the necessity of using bilevel optimization in LLM PEFT experiments. Why could the bilevel PEFT be better than the standard PEFT training? I think it is still very uncommon to use bilevel optimization in PEFT.\n\n3. The parameter setting of $\\gamma = \\Omega(\\delta^{-\\frac{2-\\alpha}{\\alpha }}) $ in Line 810 is not correct. For instance, if $\\delta=0$ then such a setting indicates that $\\gamma = +\\infty$. I think the parameters from Line 810 to 817 should be checked, although I believe the final result is still correct. \n\n4. (Minor) The reference [36] by Feihu Huang et al., is flawed, and I think it should be removed from the citations.\n\nQUESTIONS:\nSee the weakness part.\n\nLIMITATIONS:\nSee the weakness part.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the computational inefficiency of bilevel optimization (BLO) algorithms often used in machine learning applications such as fine-tuning large language models (LLMs). Traditional BLO methods typically require expensive Hessian-vector computations or nested loop updates, which are impractical for large-scale problems. To overcome these challenges, the authors propose a computationally efficient, fully single-loop algorithm called PBGD-Free that eliminates the need for lower-level value function evaluations while retaining first-order updates. By leveraging a new flatness (Hölder-alike) condition for the upper-level objective, they provide theoretical guarantees of convergence for their method. Empirical results on parameter-efficient LLM fine-tuning and other bilevel tasks demonstrate that PBGD-Free achieves significant improvements in efficiency while matching or surpassing the accuracy of previous state-of-the-art methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n1. This paper presents a novel, single-loop, value-function-free algorithm for bi-level optimization. The authors provide theoretical guarantees of convergence, and the proposed method empirically outperforms existing approaches.\n\n2. The introduction of a new flatness condition to establish the convergence of the algorithm is notable. This condition may also prove valuable for future research in this area.\n\n**Weaknesses**:\n\n1. The paper does not discuss the empirical computational and memory costs in comparison to existing methods. (Please see my comments in Question 1.)\n\n2. The experimental problem formulation does not reflect a typical setting for LLMs. (Please see my comments in Questions 2 and 3.)\n\nQUESTIONS:\n1. The authors state that the proposed method is more computationally and memory efficient than existing methods. However, no empirical evidence is provided to substantiate this claim. I recommend including results such as (1) convergence versus wall-clock time, and (2) peak memory usage for the proposed method and the baselines.\n\n2. In LLM applications, DPO is typically applied as a post-SFT stage. Employing bi-level optimization in this context is uncommon. Additionally, MT-Bench is often used as a standard testbed under this setting instead of objective value or BLUE scores.\n\n3. Adversarial attack scenarios or mitigating catastrophic forgetting are more common considerations in this area. I strongly encourage the authors to explore and evaluate their approach in these settings.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}