Title: Problem-Parameter-Free Decentralized Bilevel Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Decentralized bilevel optimization

problem-parameter-free

adaptive stepsizes

single-loop
Abstract: Decentralized bilevel optimization has garnered significant attention due to its critical role in solving large-scale machine learning problems. However, existing methods often rely on prior knowledge of problem parameters—such as smoothness, convexity, or communication network topologies—to determine appropriate stepsizes. In practice, these problem parameters are typically unavailable, leading to substantial manual effort for hyperparameter tuning. In this paper, we propose \textbf{AdaSDBO}, a fully problem-parameter-free algorithm for decentralized bilevel optimization with a single-loop structure. AdaSDBO leverages adaptive stepsizes based on cumulative gradient norms to update all variables simultaneously, dynamically adjusting its progress and eliminating the need for problem-specific hyperparameter tuning. Through rigorous theoretical analysis, we establish that AdaSDBO achieves a convergence rate of $\widetilde{\mathcal{O}}\left(\frac{1}{T}\right)$, matching the performance of well-tuned state-of-the-art methods up to polylogarithmic factors. Extensive numerical experiments demonstrate that AdaSDBO delivers competitive performance compared to existing decentralized bilevel optimization methods while exhibiting remarkable robustness across diverse stepsize configurations.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces AdaSDBO, an innovative adaptive single-loop algorithm for decentralized bilevel optimization that eliminates the need for problem-specific hyperparameter tuning. By dynamically adjusting stepsizes based on accumulated gradient norms and incorporating a hierarchical stepsize design, AdaSDBO efficiently balances updates across primal, dual, and auxiliary variables. Theoretically, the algorithm achieves a convergence rate of $\\widetilde{\\mathcal{O}}(\\log^4(T)/T)$, matching state-of-the-art methods while remaining robust to stepsize choices. Extensive experiments on synthetic and real-world datasets, including hyperparameter optimization and meta-learning tasks, demonstrate AdaSDBO’s competitive performance and superior adaptability compared to existing approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: The paper studied the decentralized bilevel problem and proposed a novel algorithm of which the hyper-parameter selection does not rely on problem-specific factors. The rigorous theoretical analysis is provided in the paper with clear format. Numerical experiments are provided to verify the results. Overall, the paper is well-written and with clear novelty. \n\n\nWeakness: \n- How the algorithm become problem-parameter-free is unclear. It may be better to highlight what algorithm improvement achieve this.\n- It is better to verify in numerical study that the algorithm is truly problem-parameter-free and outperforms the existing algorithms.\n\nQUESTIONS:\n1. The algorithm has 6 hyper-parameters, which is much more than the other decentralized bilevel algorithms. Would this make algorithm hard to tune? How sensitive is performance to their choice, and do you have guidelines for setting them?\n\n2. In algorithm step 6, the communication cost seems to be 6x of the model parameters. Is the proposed algorithm communication-costly than other decentralized bilevel algorithms?\n\n3. Although the hyper-parameters doesn't reply on the problem-parameters, how about the final convergence rate? If the final convergence rate is also affected by these problem-parameters, does it mean the algorithm still need hyper-parameters tuning based on the problem-parameters to achieve similar convergence speed?\n\n4. The experiments use relatively small networks. How does the algorithm scale to hundreds or thousands of nodes?\n\nLIMITATIONS:\nThe paper didn't discuss the limitation. It may worth to discussing the algorithm communication complexity, memory cost and its applicable in real world tasks.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces AdaSDBO, a novel algorithm designed for decentralized bilevel optimization that does not require prior knowledge of problem parameters. The proposed method addresses the issue of hyperparameter tuning by employing adaptive step sizes based on cumulative gradient norms. This enables the algorithm to update variables simultaneously while adjusting its progress dynamically. The paper provides a thorough theoretical analysis demonstrating that AdaSDBO achieves competitive convergence rates. Furthermore, the experimental results confirm that AdaSDBO is robust and performs well across a range of step size configurations.\n\nSTRENGTHS AND WEAKNESSES:\nStrength：\n1. The approach of eliminating the need for problem-specific hyperparameter tuning is a significant contribution to decentralized bilevel optimization. \n2. The paper includes a detailed theoretical analysis of AdaSDBO, demonstrating its convergence rate and the conditions under which the algorithm performs comparably to state-of-the-art methods. \n3. By removing the dependence on problem-specific hyperparameters, AdaSDBO could be particularly valuable in real-world applications, where such parameters are often unknown or costly to determine.\nWeakness:\n1. Although the experiments are comprehensive, it would be beneficial to see more experiments on large datasets. The experiments on MNIST, FashionMNIST and Cifar10 are not enough.\n2. In decentralized algorithms, communication overhead is a critical concern. The paper does not provide a detailed analysis of the communication complexity of AdaSDBO, which could be important for real-world applications involving distributed systems. A comparison of communication cost with other methods could be helpful.\n3. The proposed method relies on the strongly convex assumption of the lower level, which is restrictive in the real world.\n\nQUESTIONS:\nSee weakness.\n\nOther questions:\n1. In Figure 2, which variable's step size is being depicted? Since the proposed method involves three distinct step sizes, how sensitive is the method to the choice of initial values for these three step sizes?\n2. The proposed method is parameter-free, but from Figure 2, it appears that the step size still has a significant impact. Could you explain why this is the case?\n\nLIMITATIONS:\nThis paper focus on the algorith and analysis and does not have societal impact .", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes AdaSDBO, a fully problem-parameter-free algorithm for decentralized bilevel optimization with a single-loop structure. AdaSDBO employs adaptive stepsizes based on cumulative gradient norms to simultaneously update all variables, enabling dynamic adjustment without requiring problem-specific hyperparameter tuning. Theoretical analysis in the nonconvex–strongly-convex bilevel setting shows that AdaSDBO achieves a convergence rate matching that of well-tuned state-of-the-art methods up to polylogarithmic factors. Extensive experiments validate the effectiveness and robustness of AdaSDBO across various tasks and settings.\n\nSTRENGTHS AND WEAKNESSES:\nAdaSDBO leverages accumulated gradient norms to dynamically adjust stepsizes at each iteration, thereby eliminating the need for hyperparameter tuning. It introduces two key mechanisms: (1) a hierarchical stepsize design that accounts for the interdependence of variables while preserving the autonomy of adaptive updates; and (2) a stepsize tracking scheme that synchronizes gradient-norm accumulators across agents, effectively mitigating stepsize discrepancies in the decentralized setting. The effectiveness of AdaSDBO is supported by both rigorous theoretical analysis and comprehensive experimental results.\n\n**Weaknesses:**\n1. The stepsize tracking scheme introduces significant communication overhead due to the additional transmission of gradient norms.\n2. AdaSDBO appears to be designed for deterministic bilevel optimization and relies on full gradient information, limiting its applicability in stochastic settings.\n3. While often considered negligible, the convergence rate includes an extra $\\log^4(T)$ factor, which creates a gap compared to the optimal $O(1/T)$ rate.\n4. Each iteration requires Jacobian and Hessian computations, which can be computationally expensive for large-scale problems and may hinder scalability.\n5. The experimental evaluation is limited in scale, involving no more than 12 agents and relatively simple datasets and models.\n6. The runtime performance of the algorithm is not reported, making it difficult to assess its practical efficiency.\n7. The code is not provided, which hinders the reproducibility of the experimental results.\n\nQUESTIONS:\n1. What is the rationale behind using the accumulators in Lines 178 and 181? Transmitting gradient norms can significantly increase communication overhead. Are there alternative accumulator designs that could reduce this cost?\n2. What motivated the design of the primal variable update in Line 196? Could the authors provide more explanation for the choice of the proportional terms used in the update?\n\nLIMITATIONS:\nPlease refer to weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
The authors propose AdaSDBO -- a parameter-free single-loop algorithm for decentralized bilevel optimization. Leveraging accumulators and a hierarchical stepsize schedule, the algorithm eliminates the need for knowledge of smoothness and strong-convexity constants. Authors provide theoretical analysis showing that AdaSDBO achieves an $O\left(\frac{\log^4{T}}{T}\right)$ stationarity rate. Experiments on synthetic and real-world problems, and CIFAR-10 meta-learning demonstrate competitive performance compared to baselines.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The authors guide the reader through the paper step by step, explaining design choices from a high-level perspective.
2. The algorithm is parameter-free, so practitioners can deploy the algorithm without tuning hyperparameters.
3. The authors establish the SOTA $O(\log^{4}T/T)$ convergence rate, although I have not checked the proofs.

Weaknesses:
1. The experiments cover only a few graph shapes, and they do not disclose how the topology of the network affects the convergence.
2. Many deep-learning tasks are neither smooth nor strongly convex, which limits the direct relevance of the theory.

QUESTIONS:
1. The second equation on line 174 is not informative — removing it might improve readability.
2. What is the set $\mathcal{V}$?
3. Figure 2 (c & d) — in the legend description, the formula $(n = x)$ is not horizontally centered.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-06