Title: Block Coordinate Descent for Neural Networks Provably Finds Global Minima
Venue: NeurIPS 2025
Accepted: yes
Keywords: Deep learning theory

block coordinate descent

global convergence

generalization error analysis
Abstract: In this paper, we consider a block coordinate descent (BCD) algorithm for training deep neural networks and provide a new global convergence guarantee under strictly monotonically increasing activation functions. While existing works demonstrate convergence to stationary points for BCD in neural networks, our contribution is the first to prove convergence to global minima, ensuring arbitrarily small loss. We show that the loss with respect to the output layer decreases exponentially while the loss with respect to the hidden layers remains well-controlled. Additionally, we derive generalization bounds using the Rademacher complexity framework, demonstrating that BCD not only achieves strong optimization guarantees but also provides favorable generalization performance. Moreover, we propose a modified BCD algorithm with skip connections and non-negative projection, extending our convergence guarantees to ReLU activation, which are not strictly monotonic. Empirical experiments confirm our theoretical findings, showing that the BCD algorithm achieves a small loss for strictly monotonic and ReLU activations.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical justification for the global convergence of a custom Block Coordinate Descent (BCD) algorithm applied to deep neural networks with more than two layers. In their formulation, the algorithm introduces auxiliary variables $V_{j,i}$, which approximate the outputs of each hidden layer and allows the optimization problem to be decomposed into layer-wise subproblems. Under assumptions including i.i.d. data, full-rank input, and strictly monotonic activation functions, the authors prove global convergence to zero training loss. They also extend their framework to accommodate ReLU activations by introducing a residual network structure with skip connections and non-negative projection operators. Additionally, they show justification for a generalization error bound, showing that the difference between training and test error of their approach remains small with high probability.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper provides a theoretical analysis of a custom Block Coordinate Descent (BCD) algorithm for training deep neural networks with more than two layers. It justifies a global convergence guarantee outside of frameworks like RKHS and NTK, where previous work often allows minimal parameter movement. The authors show global convergence under a more dynamic optimization regime, using auxiliary variables and assuming monotonically increasing activation functions. However, there is not much experimental validation, as they rely on synthetic data generated from a teacher network and do not empirically verify the claimed generalization bounds. Additionally, while the theoretical results are interesting, the proposed algorithm does not seem very practical for larger applications due to its reliance on multiple nested updates and restrictive assumptions on data structure. So while this paper provides an interesting contribution to the theoretical understanding of optimizing models, its practicality is limited.\n\nQUESTIONS:\nCould this approach potentially be extended to handle other non-strictly monotonic activation functions beyond ReLU? \n\nWhat would be the practical applications of this approach be?\n\nCould you provide a more explicit discussion of the time and memory complexity of the proposed BCD approach?\n\nLIMITATIONS:\nYes, as they mention the assumptions needed in order for their justifications to be true.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the convergence property of block-coordinate descent on deep multi-layer perceptrons. The primary focus of the paper is on activation functions with derivatives bounded from above and below by positive numbers. By rewriting the objective into layer-wise components, the paper shows a linear convergence of the training error. Moreover, the paper also provides a generalization bound based on the Rademacher complexity. The paper also extended the theory to ReLU activation by adding skip connections. The paper provided experimental result to demonstrate the layer-wise loss dynamic under both the strictly monotonic activation and the ReLU activation, which shows that on the re-written objective, the ReLU network without skip connection does not achieve a favorable loss.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\n1. The theoretical contribution of this paper is quite comprehensive: the paper covers both the training convergence and generalization property. The analysis is on standard MLP with strict monotonic activation, but is also extended to ReLU activations by adding skip connection. Lastly, it seems that the paper also has an extension to the case of output dimension greater than 1.\n\n2. The convergence guarantee is strong in the sense that it is a linear convergence to arbitrarily small training loss. The assumptions on the neural network structure is fairly relaxed since it does not require a significant over-parameterization.\n\n**Weaknesses**\n\n1. The primary concern I have about the paper is that it deviates from the standard way of training neural networks. Traditional block coordinate descent should be simply update layer weights separately. By re-writing the objective, the algorithm requires the optimization of both the hidden-layer outputs and the neural network weights, to enforce alignment between the two. It is not clear whether this algorithm has practical applicability. On the other hand, the paper also lacks experimental support to show that the behavior of the algorithm considered in this paper is similar to the behavior of traditional coordinate descent. Therefore, there might be possibility that the paper offers limited insight into algorithms people use in practice.\n\n2. The generalization bound seems to be quite large given that it scales as $(2r)^\\frac{L}{2}$. Also since rank$(X) = n$, it is necessary that $d\\_{in} \\geq n$. Moreover, based on the standard matrix concentration result, for $X$ with I.I.D. entries, we have $\\\\|X\\\\| = O(\\sqrt{n})$. Therefore, the first term in the generalization gap's bound may not decrease with $n$ under a lot of input data.\n\n3. The experiments are conducted only on artificial data so it remains doubt whether the same behavior can be carried along to real-world applications.\n\nQUESTIONS:\nNone.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a block coordinate descent algorithm for training deep neural networks by reformulating the standard training objective with auxiliary variables that represent the activations at each layer for each training sample. This lifted formulation allows the optimization problem to be decomposed into simpler subproblems over weights and activations, enabling alternating updates. The authors prove that, under several assumptions on data rank and network width, their method converges to global minima of the objective function, with guarantees provided for both bijective and certain non-bijective activation functions.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The paper introduces a lifted optimization formulation that decouples layer-wise computations using auxiliary variables, enabling a principled application of block coordinate descent to neural network training.\n\n2. The convergence analysis is rigorous, showing that the proposed method reaches global minima under several assumptions, and accommodates both bijective and certain non-bijective activation functions.\n\n3. The authors derive a generalization bound based on Rademacher complexity, showing that the learned model exhibits provable generalization ability under norm and width constraints.\n\n4. The experimental results on a synthetic dataset support the theoretical claims for both bijective and certain non-bijective activation functions, and the ablation studies highlight the necessity of key assumptions in ensuring convergence. Experiments are reproducible with the provided codes.\n\n**Weaknesses**\n1. The paper focuses on achieving zero training loss through a carefully constructed optimization framework under several restrictive assumptions, such as the invertibility of the hidden layer matrices $W_2, \\cdots, W_{L-1}$ and wide first layer $W_1$ (The authors assume $n\\leq d_{in}$). However, these conditions may be unrealistic in practical settings, where perfect fitting is neither expected nor necessary for good generalization.\n\n2. The update of $V_{j-1,i}$ deviates from the standard BCD principle by considering only one of the two terms in the objective involving this variable. Specifically, the update minimizes $\\|\\| \\sigma(W_j V_{j-1,i}) - V_{j,i} \\|\\|^2$ but ignores the term $\\|\\| \\sigma(W_{j-1} V_{j-2,i}) - V_{j-1,i} \\|\\|^2$, which also depends on $V_{j-1,i}$. This partial update may simplify computation but is not a full block minimization step, and the paper does not provide a clear justification.\n\n3. The introduction of auxiliary variables $V_{j,i}$ incurs non-negligible memory overhead, as it requires storing separate variables for each layer and each data sample. Even if one can leverage GPU parallelism to reduce memory usage, this extra cost remains significant in large-scale deep learning applications such as training LLMs.\n\n4. The paper discusses how the proposed framework can be adapted to handle non-bijective activation functions, such as ReLU. This extension is valuable, but it would be better if the approach were formalized in a more general way without relying on a specific example.\n\n5. The authors critique prior work for being limited to shallow architectures, typically with *two* or *three* layers, and emphasizes that the proposed method can in principle handle networks of *arbitrary depth*. However, the experiments are only conducted on a network with *four* hidden layers. It would be helpful to evaluate deeper architectures to better demonstrate the method’s practical scalability.\n\n6. Minor issues:\n- Please use vector graphics for Figure 1.\n- The contributions are summarized too long, please make them concise.\n- Line 178 tells that the algorithm applies multiple iterations per update for V, is it possible that Lines 3 and 4 in Algorithm 1 miss an inner loop?\n\nQUESTIONS:\n1. Why do the authors aim to achieve zero training loss under relatively strong assumptions, given that perfect fitting is often unnecessary in practice?\n2. Although some BCD variants modify the block-wise objective for simplified updates in practice, could the authors explain why the update for $V_{j-1,i}$ considers only one of the two relevant terms in the original objective?\n3. How can we handle the extra memory overhead of $V_{j,i}$'s? While GPU parallelism may alleviate the issue, could the authors provide extra analytical or experimental evidence to support its practicality?\n4. The authors explicitly mention the use of skip-connections as in ResNet in Section 6, but the corresponding experiment is still simple (see weakness 5). Could the authors consider adding experiments on real-world tasks with deeper networks?\n5. Please address the minor issues to improve paper quality.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies block coordinate descent (BCD) applied to deep neural networks with strictly monotonically increasing activation functions. It claims to be the first theoretical results proving a convergence to global minima with arbitrarily many layers without NTK. It provides a theoretical generalization bound analysis assuming iid samples. The paper further modifies (BCD) (by incorporating skip connections and non-negative projection steps) so that the corresponding analysis extends to networks using ReLU activation.\nBCD is formalized so that each layer of the neural network is treated as one block, and additional auxiliary variables are introduced corresponding to each pair of (input, layer output). A small numerical experiment is performed.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- the paper is very well written\n- global convergence of any neural network training is a valuable analysis, and this is corroborated by numerical experiments\n\nWeakness:\n- the paper assumes the number of data points $n \\leq d$ the number of dimensions, which is non-standard\n- the re-formulation of the objective leads to a blowup in the number of variables. And in fact, it is not clear to me why on page 7, line 218, you have that the total number of gradient computations is $\\tilde{O}(\\log^2(1/\\epsilon))$ without a dependence on $n$.\n- On page 5, lines 172-176, you discuss the role of SVB as a heuristic but not in sufficient detail. Is there a formal discussion of why the singular values of $W$ should remain bounded throughout training? Based on the numerical experiment, it looks like loss plateaus above $10^{-4}$ without SVB, suggesting that this is in fact a crucial step for ‘fast’ convergence. Could you add a longer discussion or additional experiments?\n\nQUESTIONS:\nsee strength and weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}