Title: Irreducible Loss Floors in Gradient Descent Convergence and Energy Footprint
Venue: NeurIPS 2025
Accepted: no
Keywords: gradient descent

convergence

loss bounds

optimization

training dynamics

sustainability

efficiency

feasibility

computational cost

irreducible loss

non-convex optimization

lower bounds
Abstract: Despite their central role, convergence analyses of the dynamics of loss functions
during training require strong assumptions (e.g convexity and smoothness) which
are non-trivial to prove. In this work, we introduce a framework for deriving
necessary convergence conditions that hold without restrictive assumptions on
the dataset or the model architecture. By linking microscopic properties such as
individual sample losses and their gradient to macroscopic training dynamics, we
derive tight lower bounds for loss functions, applicable to both full-batch and mini-
batch gradient systems. These bounds reveal the presence of irreducible floors
that optimizers cannot surpass and beyond theoretical guarantees, this framework offers a practical tool for anticipating convergence speed, and estimating
minimum training time and energy requirements. Thus, this framework can be
used to ensure the sustainability and feasibility of large-scale training regimes.


Peer reviews:
comments: SUMMARY:
The paper examines the connection between gradient flow and the instantaneous rate of change for commonly used loss functions in machine learning. Across four studied cases, a structural relationship is identified, which allows for the derivation of a lower bound on the convergence behavior of the loss function. This bound can be used to estimate worst-case training times and to provide worst-case estimates for the energy consumption of the method.

STRENGTHS AND WEAKNESSES:
**Strengths**
- Very well-written and easy-to-understand paper.
- Using simple mathematics, it reveals a connection that wasn't previously known in this way. That's why I think it's a very interesting approach that should be pursued further.
- No assumptions about smoothness or convexity are needed to derive the lower bound on training time.
Seems useful for estimating the effort required to train large models.

**Weaknesses**
- The theory is limited to the standard (stochastic) gradient method; regularization can be incorporated. However, in the toy example presented, the approach does not work for common variations like RMSProp. An extension of the approach is briefly discussed, but its correctness cannot be verified without further details.
- The tightness of the derived bounds is not discussed. It is unclear how, or whether, the concept can be generalized to other loss functions beyond the four considered here

QUESTIONS:
- Can the approach be extended to additional loss functions?
- How tight are the derived bounds?
- Is it possible to compare them to other lower bounds, such as the convergence rate of $\theta$?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose lower bounds for the value of the loss function during gradient descent training of parametric models in supervised learning. The bounds factor into two terms, one depending on the model and the dataset, while the other is determined by the loss function. The aim of the bounds is to estimate worst-case training time.\n\nSTRENGTHS AND WEAKNESSES:\n1. Quality: The abstract claims that the bounds are tight, but Figure 5 demonstrates that they are in fact loose. Necessary convergence conditions refer to the proposed bound reaching a given threshold. The extension to adaptive optimizers in Section 8 seems to be wrong, as it assumes that the parameter update is a linear function of the gradient. The derivations seem to be correct modulo some typos. The claims are not collected into propositions. The last section clearly states that more work is needed for practical significance.\n\n2. Clarity: Assumptions, claims and proofs are not separated, making the paper more difficult to follow. Notation is often unclear. It is not stated how the loss floor plotted in Figure 5 is calculated.\n\n3. Significance: The proposed bounds are too loose to be significant. The idea that loss lower bounds can provide a way to estimate compute requirements could prove to be valuable if tight bounds are obtained.\n\n4. Originality: Other works tend to focus on loss upper bounds, so that exploiting lower bounds seems to be novel. On the other hand, the proof techniques are less innovative.\n\nQUESTIONS:\nIt would benefit readability if the theoretical part is separated into propositions and their proofs, with the assumptions clearly stated.\n\nEquation (29) does not describe adaptive optimizers, as the parameter update is in general not a linear function of the gradient.\n\nHow exactly is the minibatch loss floor plotted in Figure 5 calculated?\n\nThe argument of the function $g$ is inconsistent, being either $\\mathcal{L}$, $\\ell$ or $x$. Around equation (2), considering the data distribution over $\\ell_i$ is a bit misleading. The learning rate is denoted $lr$ in equations, it would read better if it is only one letter. These are three of the number of ways notation could be improved.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes general lower bounds for the time derivative of the loss under gradient flow for several different kinds of losses (MAE, MSE, categorical and multi-label binary cross-entropy). Extensions involving multi-component losses, regularization and SGD are also discussed. It is argued that the derived bounds can help estimate energy consumption during model training. A section with experiments shows learning curves of several optimization algorithms compared with the respective theoretical bounds.\n\nSTRENGTHS AND WEAKNESSES:\nThe merit of the paper is that it attempts to develop a general theory of loss lower bounds applicable to various losses and general problems. Unfortunately, I cannot  call this attempt successful.\n\nWhile the paper is theoretical, it is written in a \"stream of consciousness\" style, as a chain of arguments and calculations without any theorem statements. To understand what the results are and under which assumptions they are obtained the reader basically needs to read attentively through the whole paper. Some assumptions are buried deep in the arguments (e.g., the bound for the categorical cross-entropy in Section 4.2 seems to be obtained under the assumption $Var(l)\\\\le 2^{-m} L^2$).  \n\nThe derived bounds depend on the \"instantaneous decay rate\" $K=\\\\sqrt{\\\\tfrac{1}{n}\\sum_{i=1}^n \\\\|\\\\nabla f(x_i,\\\\theta)\\\\|^2} $ which is unknown in general. As a result, the respective differential inequalities for the loss are not fully specified and hence do not generally produce concrete loss bounds. This significantly limits the usefulness of the obtained results.\n\nAs a way to overcome this difficulty, the paper considers linear models, since $K$ is known and constant in this case: $K=\\\\sqrt{\\\\tfrac{1}{n}\\sum_{i=1}^n \\\\|x_i\\\\|^2}$. This is a very special case, and this is the only case in which we get a specific lower bounds for $L(t)$ (table 3). However, these bounds strongly underestimate the actual loss. In the MAE case, the bound $L(t)\\\\ge L_0-K^2t$ becomes negative and hence vacuous for sufficiently large $t$. In the MSE case, an explicit exact form of the loss is available: \n$$L(t)=\\sum_{m}c_me^{-2h_mt},$$ \nwhere $h_m\\\\ge 0$ are the eigenvaliues of the Hessian $H=\\\\tfrac{1}{n}\\sum_{i=1}^n x_i x_i^T$ and $c_m\\\\ge 0$ are the respective initial eigencomponents. We can compare this exact solution with the bound $L(t)\\\\ge L_0e^{-2K^2 t}$ from table 3. We have $L_0=\\sum_m c_m$ and $K^2=\\\\tfrac{1}{n}\\sum_{i=1}^n \\\\|x_i\\\\|^2=Tr(H)=\\\\sum_m h_m,$ so that the bound $L(t)\\\\ge L_0e^{-2K^2 t}$ can be equivalently written as   \n$$L(t)\\\\ge\\\\Big(\\sum_m c_m\\\\Big)e^{-2\\\\sum_m h_m t}.$$ \nWe see that the proposed bound can substantially underestimate the actual loss.\n\nDiscussion of the energy footprint in section 7 is unspecific; no examples of actual numbers for the energy consumption are provided. Given the lack of tightness of the bounds, it is unclear how accurate and useful such results could be.   \n\nSection 8 with experiments is confusing. First, it provides results not only for SGD, but also for other algorithms (AdamW, etc.), not theoretically analyzed up to this point. The section contains a very brief sketch of ideas related to these extensions, but this discussion is too short and obscure. For example, it is not clear why the generalized equation (29) is first-order, since the natural continuous-time analog of dynamics with momentum is second-order. It is accordingly not clear what are the theoretical bounds shown in Figure 5, and why they hold. Next, it is puzzling that for RMSProp the bound is violated in the experiments. This suggests that the proposed bounds are neither tight (as both Figure 5 and my comments above show) nor reliable.\n\nQUESTIONS:\nThe paper needs a major revision. Key statements should be properly formulated, preferably as theorems with clear hypotheses and conclusions. Technical details of the proofs can be moved to the appendix. (Lack of) tightness of the bounds should be properly addressed. If optimization algorithms other than SGD are considered, they deserve separate careful analysis and discussion. The usefulness of the obtained bounds (e.g., for energy footprint estimation) should be demonstrated more convincingly.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}