Data-Aware Training Quality Monitoring and Certification for Reliable Deep Learning

## Abstract

Abstract Deep learning models excel at capturing complex representations through sequential layers of linear and non-linear transformations, yet their inherent black-box nature and multi-modal training landscape raise critical concerns about reliability, robustness, and safety, particularly in high-stakes applications. To address these challenges, we introduce YES training bounds, a novel framework for real-time, data-aware certification and monitoring of neural network training. The YES bounds evaluate the efficiency of data utilization and optimization dynamics, providing an effective tool for assessing progress and detecting suboptimal behavior during training. Our experiments show that the YES bounds offer insights beyond conventional local optimization perspectives, such as identifying when training losses plateau in suboptimal regions. Validated on both synthetic and real data, including image denoising tasks, the bounds prove effective in certifying training quality and guiding adjustments to enhance model performance. By integrating these bounds into a color-coded cloud-based monitoring system, we offer a powerful tool for real-time evaluation, setting a new standard for training quality assurance in deep learning.

## 1 Introduction

Deep learning models have become crucial for tackling complex computational problems, owing to the rich representations they develop through their multi-layered structures and non-linear transformations [, ]. Despite their remarkable effectiveness, these models are often perceived as black boxes, raising concerns related to their robustness, reliability, and safety. As neural networks become increasingly integral to critical applications, ensuring that they are properly trained and perform as intended is paramount.

To evaluate the training performance and a network’s ability to store a model after training (i.e., achieve zero loss), one approach is to statistically analyze neural networks under certain assumptions. This has been done for networks with thresholding activation functions like ReLU, where researchers have determined the number of parameters needed to achieve full memory capacity []. It is well-known that for ReLU-based neural networks (NNs), once a sufficient number of weights is
reached, the network can achieve full memory capacity or even zero loss in some cases. In [], the authors theoretically demonstrate that in the over-parameterization
regime, the stochastic gradient descent (SGD) algorithm can converge to the global minimum. However, these methods are statistical in nature and rely on specific assumptions about the input data and the model, which may limit their applicability.

The most widely encountered scenario in mathematical optimization, including for training deep neural networks, is that the true optimum value of the objective function is not known
*a priori*—i.e., it is unclear to what extent the minimization objective can be reduced. Therefore, finding a way to evaluate the effectiveness of optimizers in minimizing the training objectives is an intriguing pursuit. In this paper, we introduce a novel framework for *data-aware certification and monitoring* of deep neural network training, which we refer to as the *YES training bounds*. These bounds aim to provide a qualified answer to the question: *Is the neural network being properly trained by the data and the optimizer (YES or NO)?* The certificates are *data-aware* in the sense that they take into account the specific structure and properties of the training data, allowing for more precise and tailored bounds on the training performance. This means that rather than relying on generic or overly conservative estimates, the guarantees reflect the actual dataset being used, providing more relevant insights into the training process. They are also *real-time*, fulfilling a long-standing desire in the deep learning community for tools that can evaluate and certify the progress made in training a neural network as it unfolds. By offering concrete, data-specific evaluations during training, these guarantees allow practitioners to confidently assess when sufficient learning has taken place and whether the model is converging toward optimal training, all without the need for extensive post-hoc analysis or probabilistic approximations.

To facilitate the practical application of the YES bounds, we offer an intuitive *cloud-based monitoring system* that visually represents the training progress. This system uses a color-coded scheme—red (ineffective training), yellow (caution, training non-optimal), and green (effective training in progress)—to certify the training status in real time; see Fig. 3.
As the field of artificial intelligence (AI) continues to evolve, there is a growing emphasis on *AI safety* and *regulation* to ensure that AI systems meet the highest standards of reliability and effectiveness [, , , , , , , , , , , , ]. The proposed YES training bounds and the associated training cloud system present a promising pathway toward establishing a benchmark for the AI industry, regulators, and users alike. We envision a future where these bounds are widely adopted, serving as a standardized measure to evaluate and ensure the optimal training performance of AI systems across diverse applications. This standardization could play a crucial role in fostering trust and accountability within the AI ecosystem.

The YES bounds and their associated cloud system primarily serve as a *sanity check for the optimizer*—the mechanism driving the training process. This system continuously challenges the optimizer with intelligently crafted examples grounded in mathematically rigorous heuristics and tailored to the layer-wise architecture of neural networks. By doing so, the bounds lay the proper groundwork for standardization of training practices in deep learning. Two key observations support this assertion:

- 1.
*The Bounds Go Beyond Local Optimization Perspectives:* Traditional optimization approaches in neural networks predominantly focus on local information—examining the immediate vicinity of the optimization landscape, such as attraction domains and local optima. However, numerical examples demonstrate that training losses can often plateau in regions where the YES bounds and the associated cloud unequivocally indicate suboptimality. This phenomenon underscores the bounds’ ability to transcend local optimization insights, providing a more global perspective on training efficacy.
- 2.
*Deterministic Certification Without Randomization:* Unlike methods that rely on randomization around the current training state to certify local or neighborhood optimality, the YES bounds operate deterministically. They do not produce varying certification results across different training realizations, even when initialized identically or following similar optimization paths. This determinism is particularly advantageous for standardization, as it ensures consistent and reproducible determinations of training quality. By eliminating the variability introduced by randomization, the YES bounds present a robust and reliable candidate for establishing standardized training benchmarks in deep learning.

The rest of the paper is organized as follows: In Section 2, we present the preliminaries of our work and outline the architecture of the model to which we apply the YES bounds. Section 3 draws attention to a fundamental bound for single-layer NNs that will later aid in the development of the YES bounds. In Section 4, we extend this idea to deep neural networks, accounting for multiple layers and the nonlinearity of activation functions. Section 5 details our numerical analysis, evaluating the training performance on both synthetic and real data across various applications, such as phase retrieval and image denoising, demonstrating the capability of the proposed bound in assessing AI reliability.

*Notation:* Throughout this paper, we use bold lowercase and bold uppercase letters for vectors and matrices, respectively. We represent a vector $\mathbf{x}$ and a matrix $\mathbf{B}$ in terms of their elements as $\mathbf{x}=[x_{i}]$ and $\mathbf{B}=[B_{i,j}]$, respectively. $(\cdot)^{\top}$ is the vector/matrix transpose.
The Frobenius norm of a matrix $\mathbf{B}\in\mathbb{C}^{M\times N}$ is defined as $\|\mathbf{B}\|_{\mathrm{F}}=\sqrt{\sum^{M}_{r=1}\sum^{N}_{s=1}\left|b_{rs}
\right|^{2}}$, where $b_{rs}$ is the $(r,s)$-th entry of $\mathbf{B}$. Given a matrix $\mathbf{B}$, we define the operator $[\mathbf{B}]_{+}$ as $\max\left\{B_{i,j},0\right\}$. $\mathbf{B}^{\dagger}$ is the Moore-Penrose pseudoinverse of $\mathbf{B}$.

## 2 Preliminaries

Let $\mathbf{A}_{k}$ and $\mathbf{b}_{k}$ denote the weight matrix and bias vector for layer $k\in[K]$ of the network, respectively. Suppose the input and output data are represented by matrices $\mathbf{X}\in\mathbb{R}^{n\times d}$ and $\mathbf{Y}\in\mathbb{R}^{m\times d}$, respectively, such that

$$ $\mathbf{X}=\left[\begin{array}[]{c|c|c}\mathbf{x}_{1}&\cdots&\mathbf{x}_{d} \end{array}\right],\quad\mathbf{Y}=\left[\begin{array}[]{c|c|c}\mathbf{y}_{1}& \cdots&\mathbf{y}_{d}\end{array}\right],$ (1) $$

where $\{\mathbf{x}_{i}\in\mathbb{R}^{n}\}_{i=1}^{d}$ and $\{\mathbf{y}_{i}\in\mathbb{R}^{m}\}_{i=1}^{d}$ denote $d$ observations of $n$-dimensional and $m$-dimensional input and output features, respectively. Define the matrices $\{\mathbf{B}_{k}\}_{k=1}^{K}$ as

$$ $\mathbf{B}_{k}=\left[\begin{array}[]{c|c|c}\mathbf{b}_{k}&\cdots&\mathbf{b}_{k }\end{array}\right]\in\mathbb{R}^{m\times d},\quad k\in[K].$ (2) $$

We consider two closely-related training losses for such a DNN employing a nonlinear activation function $\Omega(.)$, namely

$$ $\mathcal{L}_{0}\left(\{\mathbf{A}_{k}\}_{k=1}^{K},\mathbf{X},\mathbf{Y}\right) \triangleq\|\Omega(\mathbf{A}_{K}\Omega(\mathbf{A}_{K-1}\Omega(\cdots\Omega( \mathbf{A}_{1}\mathbf{X}+\mathbf{B}_{1})\cdots+\mathbf{B}_{K-1})+\mathbf{B}_{K })-\mathbf{Y}\|_{\mathrm{F}}^{2},$ (3) $$

and

$$ $\mathcal{L}_{K}\left(\{\mathbf{A}_{k}\}_{k=1}^{K},\{\mathbf{Y}_{k}\}_{k=2}^{K} \right)\triangleq\sum_{k=1}^{K}\|\Omega(\mathbf{A}_{k}\mathbf{Y}_{k}+\mathbf{B }_{k})-\mathbf{Y}_{k+1}\|_{\mathrm{F}}^{2},$ (4) $$

which is to be minimized with respect to $\{\mathbf{A}_{k}\}$ and $\{\mathbf{Y}_{k}\}$ by setting $(\mathbf{Y}_{1},\mathbf{Y}_{K+1})=(\mathbf{X},\mathbf{Y})$. For our purpose, we will use the following notations:

$$ $\displaystyle\tilde{\mathbf{A}}_{k}$ $\displaystyle=\left[\begin{array}[]{c|c}\mathbf{A}_{k}&\mathbf{b}_{k}\end{ array}\right],$ (5) $\displaystyle\tilde{\mathbf{Y}}_{k}$ $\displaystyle=\left[\begin{array}[]{c|c}\mathbf{Y}^{\top}_{k}&\mathbf{1}\end{ array}\right]^{\top}.$ $$

Using these notations, the training loss expression in (4) can be reformulated as follows:

$$ $\mathcal{L}_{K}\left(\{\tilde{\mathbf{A}}_{k}\}_{k=1}^{K},\{\tilde{\mathbf{Y}} _{k}\}_{k=2}^{K}\right)\triangleq\sum_{k=1}^{K}\|\Omega(\tilde{\mathbf{A}}_{k} \tilde{\mathbf{Y}}_{k})-\mathbf{Y}_{k+1}\|_{\mathrm{F}}^{2}.$ (6) $$

Note that any nonlinearity in DNNs can be dealt with by imposing the output structure of the activation function on $\{\mathbf{Y}_{k}\}$. For instance, for a ReLU activation function, we only need $\mathbf{Y}_{k}\geq\mathbf{0}$, $k\in\{2,\cdots,K\}$. With this in mind, one may consider the alternative constrained quadratic program:

$$ $\displaystyle~{}~{}\quad\underset{\{\tilde{\mathbf{A}}_{k}\}_{k=1}^{K},~{}\{ \tilde{\mathbf{Y}}_{k}\}_{k=2}^{K}}{\textrm{minimize}}~{}\displaystyle\sum_{k= 1}^{K}\|\tilde{\mathbf{A}}_{k}\tilde{\mathbf{Y}}_{k}-\mathbf{Y}_{k+1}\|_{ \mathrm{F}}^{2}$ (7) $\displaystyle\mbox{subject to}\qquad\qquad\tilde{\mathbf{Y}}_{k}\in H_{\Omega},$ $$

where $H_{\Omega}$ denotes the matrix space created by applying $\Omega$. This representation, which disregards the activation function and imposes a space constraint on the solution, is useful in establishing the foundation for our bounds in subsequent sections. For the rest of the paper, we present our formulations without considering the effects of bias terms. However, these formulations can be easily extended to include bias terms by substituting $\{\mathbf{A}_{k}\}$ and $\{\mathbf{Y}_{k}\}$ with $\{\tilde{\mathbf{A}}_{k}\}$ and $\{\tilde{\mathbf{Y}}_{k}\}$, respectively.

## 3 An Optimality Bound for Single-Layer Neural Networks

We begin by considering a one-layer neural network where the goal is to approximate the function $f:\mathbb{R}^{n}\to\mathbb{R}^{m}_{+}$. Let $\mathbf{A}$ be the weight matrix that we aim to optimize, such that the objective $\|\mathbf{Y}-\Omega(\mathbf{AX})\|_{\mathrm{F}}^{2}$ is minimized. Assume $\mathbf{Y}$ is in the feasible set, i.e. the range of a non-linear activation function $\Omega(.)$ of the layer. The weight matrix $\mathbf{A}$ that minimizes the alternative objective $\|\mathbf{Y}-\mathbf{AX}\|_{\mathrm{F}}^{2}$ can then be expressed as:

$$ $\mathbf{A}=\mathbf{Y}\mathbf{X}^{\dagger}.$ (8) $$

The minimal achievable loss of training for a one-layer neural network is thus bounded as

$$ $\|\mathbf{Y}-\Omega(\mathbf{A}^{\text{OPT}}\mathbf{X})\|_{\mathrm{F}}^{2}\leq \|\mathbf{Y}-\Omega(\mathbf{YX}^{\dagger}\mathbf{X})\|_{\mathrm{F}}^{2},$ (9) $$

where $\mathbf{A}^{\text{OPT}}$ is the optimal weight matrix that minimizes the objective $\|\mathbf{Y}-\Omega(\mathbf{AX})\|_{\mathrm{F}}^{2}$. For instance, if $\Omega(.)$ is the ReLU function, then we simply have

$$ $\|\mathbf{Y}-[\mathbf{A}^{\text{OPT}}\mathbf{X}]_{+}\|_{\mathrm{F}}^{2}\leq\| \mathbf{Y}-[\mathbf{YX}^{\dagger}\mathbf{X}]_{+}\|_{\mathrm{F}}^{2}.$ (10) $$

Since the solution in (8) is feasible (not necessarily optimal, although meaningful) for the optimizer of $\|\mathbf{Y}-\Omega(\mathbf{AX})\|_{\mathrm{F}}^{2}$, a well-designed training stage is generally expected to satisfy the bound in (9).

## 4 The Training Performance Bounds for Multi-Layer Networks

So, what is the case for *depth* and *nonlinearity*? In scenarios where a single-layer is insufficient, neural learning models employ multiple layers with nonlinear activation functions such as ReLU to progressively refine input mappings (leading to *deep* learning). The necessity for multiple layers can be attributed to the complexity of the mapping task in question. The necessity for nonlinear transformations in multi-layer mappings, however, comes from
the fact that multiple consecutive linear mappings are in effect equivalent to a single-layer mapping (with a weight matrix equal to the product of linear mapping matrices).

Inspired by our observation in the single-layer case, in the following we introduce the YES training bounds for multi-layer NNs. These bounds aim to provide a qualified answer to the question as to whether a neural network is being properly trained by the data: YES or NO?

Figure: Figure 1: Schematic of the iterative mapping from $\mathbf{X}$ to $\mathbf{Y}$ through intermediate steps $\mathbf{Y}_{2},\mathbf{Y}_{3},\ldots,\mathbf{Y}_{K}$.

### 4.1 The YES-0 Bound

Assuming an initial value $\mathbf{Y}_{1}=\mathbf{X}$, the network aims to transform $\mathbf{Y}_{1}$ through intermediate states $\mathbf{Y}_{2},\mathbf{Y}_{3},\ldots,\mathbf{Y}_{K}$, finally achieving $\mathbf{Y}_{K+1}=\mathbf{Y}$; as illustrated in Fig. 1.
A sensible but sub-optimal approach will be to assume at each layer that we aim to project directly to $\mathbf{Y}$, instead of other useful intermediate points $\{\mathbf{Y}_{k}\}$. Considering our one-layer bound, this no-intermediate approach will be equivalent to setting

$$ $\mathbf{A}_{k}=\mathbf{Y}\mathbf{Y}_{k}^{\dagger},\quad k\in[K].$ (11) $$

In particular, when there are no intermediate mappings we are dealing with an order-0 (referred to as YES-0) bound:

$$ $\mathcal{L}_{0}\left(\{\mathbf{A}^{\text{OPT}}_{k}\}_{k=1}^{K},\mathbf{X}, \mathbf{Y}\right)\leq\text{B}_{\text{YES-0}}\triangleq\|\mathbf{Y}-\mathbf{Y}_ {K+1}\|_{\mathrm{F}}^{2},$ (12) $$

where

$$ $\mathbf{Y}_{k+1}=\Omega\left(\mathbf{Y}\mathbf{Y}_{k}^{\dagger}\mathbf{Y}_{k} \right),\quad k\in[K].$ (13) $$

For instance, we have

$$ $\mathbf{Y}_{k+1}=\left[\mathbf{Y}\mathbf{Y}_{k}^{\dagger}\mathbf{Y}_{k}\right] _{+},\quad k\in[K],$ (14) $$

for the ReLU activation function.

Since the central idea behind the creation of the YES-0 bound, in essence, stems from sequential projections, one may readily expect a decreasing behavior from the bound as the number of layers grows large. The following theorem provides concrete conditions under which such a monotonic behavior will be observed.

###### Theorem 1 .

Let $\Omega$ be an activation function in a deep neural network. If $\Omega$ is applied in an element-wise manner and satisfies the following conditions:

- •
*1-Lipschitz Condition*:
$\|\Omega(\mathbf{x}_{1})-\Omega(\mathbf{x}_{2})\|\leq\|\mathbf{x}_{1}-\mathbf{
x}_{2}\|,\quad\forall\,\mathbf{x}_{1},\mathbf{x}_{2}\in\mathbb{R}^{n},$
(15)
- •
*Projection Property*:
$\Omega(\mathbf{Y})=\mathbf{Y},\quad\mathrm{if}\quad\mathbf{Y}\in H_{\Omega},$
(16)

then the YES-0 bound is monotonically decreasing with respect to the depth of the network. That is, for each layer $k$:

$$ $\|\mathbf{Y}-\mathbf{Y}_{k+1}\|_{\mathrm{F}}^{2}\leq\|\mathbf{Y}-\mathbf{Y}_{k }\|_{\mathrm{F}}^{2},$ (17) $$

where $\mathbf{Y}_{k}$ is generated following (13).

###### Proof.

Following our formulations, the error at layer $(k-1)$ is

$$ $\mathbf{E}_{k-1}=\mathbf{Y}-\mathbf{Y}_{k},$ (18) $$

where $\mathbf{Y}$ is the target output, and $\mathbf{Y}_{k}$ is the network output after $(k-1)$ layers. At each layer, the network updates its output via:

$$ $\mathbf{Y}_{k+1}=\Omega(\mathbf{A}_{k}\mathbf{Y}_{k}),$ (19) $$

with $\mathbf{A}_{k}$ representing the weight matrix associated with the YES-0 bound at layer $k$. The error at layer $k$ is thus:

$$ $\mathbf{E}_{k}=\mathbf{Y}-\Omega(\mathbf{A}_{k}\mathbf{Y}_{k}).$ (20) $$

By considering the 1-Lipschitz and projection properties of the activation function $\Omega$, we have:

$$ $\displaystyle\|\mathbf{E}_{k}\|_{\mathrm{F}}^{2}$ $\displaystyle=\|\mathbf{Y}-\Omega(\mathbf{A}_{k}\mathbf{Y}_{k})\|_{\mathrm{F}} ^{2}$ (21) $\displaystyle=\|\Omega(\mathbf{Y})-\Omega(\mathbf{A}_{k}\mathbf{Y}_{k})\|_{ \mathrm{F}}^{2}$ $\displaystyle\leq\|\mathbf{Y}-\mathbf{A}_{k}\mathbf{Y}_{k}\|_{\mathrm{F}}^{2}.$ $$

Since $\mathbf{A}_{k}$ is the minimizer of the quadratic criterion $\|\mathbf{Y}-\mathbf{A}_{k}\mathbf{Y}_{k}\|_{\mathrm{F}}^{2}$, we have:

$$ $\|\mathbf{Y}-\mathbf{A}_{k}\mathbf{Y}_{k}\|_{\mathrm{F}}^{2}\leq\|\mathbf{Y}- \mathbf{Y}_{k}\|_{\mathrm{F}}^{2}=\|\mathbf{E}_{k-1}\|_{\mathrm{F}}^{2}.$ (22) $$

Combining (22) with (21) completes the proof.
∎

Obtaining the YES-0 bound, which is evaluated for each layer and projects the input of each layer to the final output, can be viewed as a layer-wise optimization with a linear closed-form operator. The YES-0 bound can serve as an immediate benchmark for the assessment of training in deep learning: One can verify whether the training has a YES-0 bound *certificate*, meaning that they are achieving a training loss lower than YES-0. Otherwise, they can attest that the training is not *proper*.

###### Theorem 1 .

###### Proof.

### 4.2 Beyond the YES-0 Bound

Note that the YES-0 bound is an easily calculated bound that may be used to immediately detect if proper (not necessarily optimal) training has been carried out, in the
sense that the network weights have been *meaningfully impacted by the training data*. The answer (YES or NO) will provide immediate relief as to whether training has been meaningful at all. More sophisticated bounds, such as the YES bounds of higher degrees may be used to further assess the quality of training, as discussed in the following.

We begin the construction of the promised bounds by examining whether one can enhance the bounds by considering a more conducive route to the output $\mathbf{Y}$.

#### 4.2.1 Is direct path the best path?

Figure: Figure 2: The illustration of a non-monotonic per-layer error (in dB) observed across both training and test stages for a DUN.
Refer to caption: x1.png

Consider a local optimizer tasked with navigating a highly non-convex loss landscape riddled with numerous local minima. Optimizers like SGD may become trapped in suboptimal regions. Interestingly, allowing the optimizer to momentarily increase the loss can enable it to escape these attraction domains and ultimately reach a better minimum. This fact highlights that sometimes, a non-direct path—including temporary increases in the loss function—can be more effective in minimizing the ultimate objective.

This behavior has been widely observed in Deep Unfolding Networks (DUNs) [], which are typically formulated based on first-order optimization methods exhibiting monotonic convergence properties. When trained with sufficient data, DUNs learn to behave non-monotonically, allowing temporary increases in loss to escape attraction domains associated with poor local optima. As a result, they can achieve lower overall objectives compared to their original first-order counterparts. Figure 2 illustrates this behavior, showing that the loss at each layer’s output does not decrease monotonically.

We will leverage this observation and explore projecting the input from each layer onto a meaningfully selected sequence of intermediate points in lieu of immediately projecting them onto $\mathbf{Y}$. Inspired by the indirect optimization path, this adjustment introduces the notion of the YES bounds of higher degrees, which will be discussed in detail in the following.

#### 4.2.2 Exploring Structured Transition through Nonlinearity-Compliant Spaces

Enhanced bounds can be established by defining a sequence of useful intermediate points $\{\mathbf{Y}_{k}\}$ that conform to the nonlinear activation function of the network, i.e. $\mathbf{Y}_{k}\in H_{\Omega}$. Let us assume we have such a useful sequence as the outcomes of layers $t_{2},\cdots,t_{\Sigma}$ (where $t_{\Sigma}=K+1$) as $\mathbf{Y}_{t_{2}}^{\star},\cdots,\mathbf{Y}_{t_{\Sigma}}^{\star}$ (where $\mathbf{Y}_{t_{\Sigma}}^{\star}=\mathbf{Y}$). This gives rise to the YES-$\Sigma$ bound, i.e.,

$$ $\mathcal{L}_{0}\left(\{\mathbf{A}^{\text{OPT}}_{k}\}_{k=1}^{K},\mathbf{X}, \mathbf{Y}\right)\leq\text{B}_{\text{YES-}\Sigma}\triangleq\|\mathbf{Y}- \mathbf{Y}_{K+1}\|_{\mathrm{F}}^{2},$ (23) $$

where

$$ $\mathbf{Y}_{k+1}=\Omega\left(\mathbf{Y}_{t_{\sigma}}^{\star}\mathbf{Y}_{k}^{ \dagger}\mathbf{Y}_{k}\right),\quad t_{\sigma-1}\leq k\leq t_{\sigma},\quad 2 \leq\sigma\leq\Sigma,$ (24) $$

with $\mathbf{Y}_{1}=\mathbf{X}$. Given a judicious selection of $\mathbf{Y}_{t_{2}}^{\star},\cdots,\mathbf{Y}_{t_{\Sigma}}^{\star}$, the latter should provide a tighter error bound compared to the YES-0 bounding approach.

In addition to the local optimization perspective discussed in Sec. 4.2.1, a domain-aware viewpoint is also helpful. While it is fair to say that we hope that by iterative mapping, we get closer and closer to the output of interest, it has also been observed in various machine learning problems that after extensive training (resembling what we can describe as optimal training), the output of some inner layers become something meaningful to domain experts (see, e.g., the literature on object detection in image processing, where the primary task of certain layers is well-understood, such as detecting edges, and certain features). This observation closely associates with the vision put forth above on tightening the YES-0 bound by considering useful and meaningful intermediate mapping points.

The key question in deriving the enhanced YES bounds is thus the judicious designation of intermediate mapping points.
One may suggest these two ways:

- 1.
*Problem-specific construction-based sets of intermediate mapping points:* As for the image processing example above, in specific applications, we might have prior knowledge of the structure of the intermediate projection points. This also presents very interesting points of tangency with the literature on model-based deep learning, and in particular, DUNs. In this context, the network layers and the input-ouput relations are strongly connected to a well-established iterative procedure, which can help with identifying suitable intermediate mappings discussed herein.
- 2.
*Training data-driven generation of the mapping points:* This approach takes advantage of the training data to enhance the YES bounds along with the training loss. This is going to be the focus of our proposed YES bounds of “higher degree" in the following.

#### 4.2.3 The YES- k 𝑘 k italic_k Training Bounds ( k ≥ 1 𝑘 1 k\geq 1 italic_k ≥ 1 )

Leveraging the training data, we utilize the output from a subset of layers acquired during the training process as the intermediate mapping points and incorporate these points into the YES bounds by projecting the input of the layer onto these intermediate outputs rather than relying solely on $\mathbf{Y}$. Specifically, one can utilize $k\in[K-1]$ intermediate mapping points in a $K$ layer network to create associated YES bounds. To better illustrate this idea, we present the example of the higher-degree YES bounds framework for the case of $k=1$:

- 1.
We set $\mathbf{Y}_{1}=\mathbf{X}$.
- 2.
We choose $\mathbf{Y}^{\star}_{2}$ as the output of the first layer during the training stage.
- 3.
Following Section 3, we aim to optimize the weight matrix of the first layer $\mathbf{A}_{1}$, such that the alternative objective $\|\mathbf{Y}^{\star}_{2}-\mathbf{A}_{1}\mathbf{Y}_{1}\|_{\mathrm{F}}^{2}$ is minimized. This is achieved by
$\mathbf{A}_{1}=\mathbf{Y}^{\star}_{2}\mathbf{Y}^{\dagger}_{1}.$
(25)
- 4.
We then obtain the output of the first layer as $\mathbf{Y}_{2}=\Omega\left(\mathbf{Y}^{\star}_{2}\mathbf{Y}^{\dagger}_{1}
\mathbf{Y}_{1}\right)$.
- 5.
Since we only chose $k=1$ intermediate point, we then progressively project the input of each layer to the output $\mathbf{Y}$ as
$\mathbf{Y}_{k+1}=\Omega\left(\mathbf{Y}\mathbf{Y}_{k}^{\dagger}\mathbf{Y}_{k}
\right),\quad k\in\{2,\cdots,K\}.$
(26)
- 6.
We compute the resulting error of this process as $\|\mathbf{Y}-\mathbf{Y}_{K+1}\|_{\mathrm{F}}^{2}$.
- 7.
We repeat steps (3-6) by choosing other intermediate points $\mathbf{Y}^{\star}_{3},\cdots,\mathbf{Y}^{\star}_{K}$ in step 2.
- 8.
Take the minimum of all errors obtained in Step 6, leading to the creation of the YES-1 training bound.

The name YES-1 bound takes into account the fact that we have only considered $k=1$ intermediate point in the above process. Note that the above formulation can be easily extended to $k\in\{2,\cdots,K-1\}$ intermediate points, generating higher degree YES bounds, namely YES-$k$ bounds for $k\in\{2,\cdots,K-1\}$.
In contrast to the YES-0 training bound, the YES bounds of higher degree are
*real-time*, i.e., they evolve alongside the training loss.
It is important to note that YES-$k$ bounds for larger $k$ are not necessarily smaller than those for smaller $k$, particularly at the initial epochs where the training may not suggest excellent intermediate points. Higher degree bounds are, however, highly likely to perform better when the training is in good condition. We will numerically validate this phenomenon in the numerical results section. In Algorithm 1, we reformulate the YES-$k$ bounds for $k\geq 1$, incorporating a monotonic modification through the inclusion of YES-$k$ subsets to ensure the bounds remain monotonic.

Figure: Figure 3: YES training cloud-system for quality monitoring: A training loss that remains above the YES training cloud (red area) indicates ineffective training. When the loss penetrates the cloud (yellow area), it suggests that meaningful training has occurred or is in progress—network weights have been significantly influenced by the data. However, caution is advised, as the training is certainly not optimal. Dropping below the cloud (green area) signals effective training in in progress and suggests potential for optimality. It may also indicate diminishing returns in the training process, where further gains could be incremental.
Refer to caption: extracted/5926115/cloud.jpeg

Figure: Algorithm 1 Generation of the YES Training Bounds ($k\geq 1$).

#### 4.2.4 The YES Training Cloud-System for Quality Monitoring

We now illustrate the integration of the proposed YES bounds with the training process, culminating in the creation of an intuitive training cloud to monitor progress in real time (see Fig. 3). As the YES bounds evolve over epochs, similar to the training loss, they enable users to visually observe the interaction between training performance and the YES bounds. This visualization allows practitioners to track key moments in the training process, such as when the training loss surpasses the YES-0 bound, when it improves beyond the best YES-$k$ bounds, the epochs at which these transitions occur, and how the bounds and training results interact throughout the process.

To help users navigate key transitions during the training process, the YES training cloud system employs a color-coding scheme. A training loss that remains above the YES training cloud—depicted as the red area—indicates ineffective training. When the loss reaches and enters the cloud—the yellow area—it signifies that meaningful training is taking place, with the network weights being substantially influenced by the data. However, caution is still advised at this stage, as the training has not yet achieved optimality. A loss that descends below the cloud into the green area denotes effective training, suggesting a potential for achieving optimal performance.

Figure: (a)
Refer to caption: extracted/5926115/pr_1_1.png

Figure: (a)
Refer to caption: extracted/5926115/signal_denoising_1_1.png

Figure: (a)
Refer to caption: extracted/5926115/pr_1_1.png

## 5 Numerical Examples and Discussion

In this section, we numerically assess the effectiveness of our proposed YES bounds in evaluating the performance of the training process for both synthetic and real-world data recovery tasks. For the synthetic data, we generated datasets based on two models: phase retrieval and one-dimensional signal denoising. For the real data example, we applied our bounds to real-world image recovery from noisy quadratic measurements and the MNIST classification task. All codes are available at [https://github.com/WaveOpt-Lab/YES-bound](https://github.com/WaveOpt-Lab/YES-bound).

### 5.1 Synthetic Data Recovery Examples

To generate the dataset, we employ the following models and configurations:

- •
*Phase Retrieval:* The data is generated by the following model:
$\mathbf{b}_{i}=\left|\mathbf{A}\mathbf{x}_{i}\right|,\quad i\in[d],$
(27)
where $\mathbf{A}\in\mathbb{R}^{20\times 20}$ is a Gaussian sensing matrix with entries drawn from $\mathcal{N}(0,1/20)$, the signal $\mathbf{x}\in\mathbb{R}^{20}$ is generated as $\mathcal{N}(0,1)$, and $d=1000$ samples are generated.
- •
*One-Dimensional Signal Denoising:*
The dataset is constructed using the model:
$\mathbf{b}_{i}=\mathbf{x}_{i}+\mathbf{n}_{i},\quad i\in[d],$
(28)
where the signal $\mathbf{x}\in\mathbb{R}^{20}$ is drawn from $\mathcal{N}(0,1)$, and the noise $\mathbf{n}\in\mathbb{R}^{20}$ follows $\mathcal{N}(0,0.2)$. For this model, we generate $50$ fixed signals, each perturbed by $20$ noise vectors sampled from $\mathcal{N}(0,0.2)$. This process is repeated to generate a dataset of $d=1000$ samples.

We train five-layer fully connected networks to approximate models under different parameter configurations, utilizing the ADAM algorithm for optimization. In all experiments, the learning rate is initialized at $\eta=\eta_{0}$ and reduced by a decay factor of $0.9$ every $50$ epochs. It is important to note that the bias term is excluded in the phase retrieval model, whereas it is included in the one-dimensional signal denoising task. We give the following criterion for stopping the training:
the rate of change in network weights is sufficiently low.

In Fig. 4, we present color-coded clouds illustrating the training process and corresponding bounds for the phase retrieval model under various conditions and parameters. Figs. 4(a)-(c) show the clouds for different batch sizes, while Figs. 4(d)-(f) illustrate the clouds for varying learning rates. Specifically, Fig. 4(a) corresponds to a batch size of $20$, Fig. 4(b) to a batch size of $100$, and Fig. 4(c) to a batch size of $500$, all with a fixed learning rate of $1e-3$. Similarly, Fig. 4(d) uses a learning rate of $1e-3$, Fig. 4(e) uses $1e-2$, and Fig. 4(f) uses $1e-4$, all with a fixed batch size of $20$.

As seen in Figs. 4(a)-(b), training enters the green region after approximately $100$ epochs, signaling effective training is in progress. Notably, the convergence rate for the batch size of $20$ is slightly faster than that of $100$, as indicated by the earlier entry into the green region. In Fig. 4(c), the model struggles to reach the green region, only doing so after $1250$ epochs, suggesting that batch sizes of $20$ or $100$ are more suitable for this task. Notably, by our real-time bounds, one can notice the ineffectiveness of the chosen parameters in a real-time manner instead of running the model for various parameter settings.

In terms of learning rate, Fig. 4(d) shows that a rate of $1e-3$ leads to effective training, with the loss entering the green region after $100$ epochs. In Fig. 4(e), increasing the learning rate to $1e-2$ does not alter the entry point into the green region, but the training loss plateaus closer to the YES bound, implying that the solution aligns with a linear projection—potentially suboptimal in this context. This pattern is consistent in Fig. 4(f) with a learning rate of $1e-4$, where the model enters the green region after $1250$ epochs and similarly plateaus near the YES bound. Overall, these observations suggest that $1e-3$ is a better learning rate for this task compared to $1e-2$ and $1e-4$.

Figure: Figure 7: YES-0 decay with respect to the number of layers.
Refer to caption: extracted/5926115/YES_0_layer.png

In Fig. 5, we present color-coded clouds illustrating the training process and corresponding bounds for the signal denoising model under various conditions and parameters, following a similar approach as in the phase retrieval case. Figs. 5(a)-(c) highlight the effects of varying batch sizes, while Figs. 5(d)-(f) show the impact of different learning rates. Specifically, Fig. 5(a) corresponds to a batch size of $20$, Fig. 5(b) to a batch size of $100$, and Fig. 5(c) to a batch size of $500$, all with a fixed learning rate of $1e-3$. Figs. 5(d)-(f) reflect learning rates of $1e-3$, $5e-3$, and $1e-4$, respectively, using a fixed batch size of $20$.

Comparing Fig. 5(a) and Fig. 5(b), we see that the training loss with a batch size of $20$ converges faster than with a batch size of $100$, as evidenced by the earlier entry into the green region. Although the results of both batch sizes ultimately reach similar error levels, the YES bound for the batch size of $100$ is closer to the training loss, a characteristic of our real-time bounds, which are constructed from specific intermediate instances along the optimization path. In Fig. 5(c), the training loss plateaus in the yellow region, indicating that the training is influenced by the data but remains suboptimal, failing to enter the green region.

Turning to the impact of learning rates, Fig. 5(d) shows that increasing the rate to $1e-2$ accelerates convergence compared to $1e-3$. However, in Fig. 5(f), with a learning rate of $1e-4$, the training loss again plateaus in the yellow region, failing to reach the green region and signaling suboptimal performance. These insights into training effectiveness are achieved in real time, without requiring multiple parameter comparisons retrospectively, providing an immediate and clear assessment of the training process. It is important to observe that the bound is more capable than just a local optimality check. For example, you can generate a number of random points around the current weights and plot the minimum loss curve associated with them, noting that for local optimality we should be on that curve. However, one can easily see that the YES bound goes further, as observed in Figs. 5(c),(f), the training losses converge in the yellow.

To check the performance of the YES bounds across different layers, we train fully connected networks for the phase retrieval model, with results shown in Fig. 6. Specifically, Fig. 6(a) illustrates the cloud for a fully connected network with $5$ layers, Fig. 6(b) for $6$ layers, and Fig. 6(c) for $7$ layers. As observed, the convergence rate of $7$-layer network appears to slow down under the same parameter settings. For example, in Fig. 6(b), the training takes nearly $100$ epochs to reach the green region, whereas in Fig. 6(c), the loss enters the green region after more than $400$ epochs. This suggests that real-time monitoring using YES bounds can quickly indicate when the chosen parameters may be suboptimal for the given model architecture.

In Fig. 7, we validate Theorem 1 by demonstrating the decay of the YES-0 bound across layers. This result is based on the phase retrieval model.
YES training bounds with different degrees, without imposing monotonicity, are shown in Fig. 8 for the phase retrieval model. An interesting observation from this figure is that increasing the degree does not necessarily improve the YES bounds. In fact, all the bounds remain relatively close to each other.
Even for the YES bounds with monotonicity, as illustrated in Fig. 9 with various initializations, it is evident that the YES bounds are closely grouped. We investigated this observation using fully-connected networks with both 5-layer and 7-layer architectures, conducted this experiment $1000$ times, and consistently observed similar results. This observation suggests that we may leverage the advantages of higher-degree YES bounds by calculating only the first few YES-$k$ bounds, which could be beneficial from a computational standpoint.

Figure: Figure 8: YES training bounds with varying degrees, without imposing monotonicity, are presented. As can be observed, increasing the degree of the bound does not necessarily improve it, as the bounds remain closely aligned with each other.
Refer to caption: extracted/5926115/YES_bounds_comparison.png

Figure: (a)
Refer to caption: extracted/5926115/YES_bounds_close_1.png

Figure: Figure 10: The YES bounds cloud for the training process is presented alongside the test stage for the same training results monitored by the YES training bounds.
Refer to caption: extracted/5926115/test_train.jpeg

It is often a challenge for both users and AI professionals to determine the optimal point to stop training (cost vs performance trade-off). A common approach is to monitor the rate of change in the loss function, waiting for the loss to plateau as a sign of potential convergence. However, as shown in Fig. 10, the training objective of neural networks (and our training procedures) can result in the loss appearing to plateau multiple times before quality training is achieved. This is where the YES clouds come to the rescue: *To know when the smaller rate of change in the loss or the weights of the neural network does not equate optimality, and when to act on it—i.e., when we are in the green.*

Beyond the training process, it is insightful to investigate how test results behave as the training progresses through different regions of the color-coded clouds. To explore this, we present both training and test outcomes for the phase retrieval model in Fig. 10. As observed, when the training loss decreases in the red region, the test loss similarly declines. When training plateaus in the yellow region, the test loss also plateaus. Interestingly, upon entering the green region, the training loss initially exhibits fluctuations, likely due to the learning rate, before plateauing—a pattern mirrored in the test loss. However, after approximately $2000$ epochs, the test loss increases.

One scenario in which the YES bounds and the associated training cloud are particularly instrumental is in the detection of lingering errors during the training process. Specifically, when inaccuracies arise in the gradient computations—perhaps due to small batch sizes leading to high variance in stochastic gradient estimates, or the use of a non-vanishing learning rate—the optimizer may fail to converge precisely to the optimum. Instead, the training dynamics may induce oscillations around the optimal point. This phenomenon introduces a steady-state error that maintains a non-negligible distance between the empirical loss $\mathcal{L}$ and its minimal attainable value $\mathcal{L}_{\text{opt}}$.
In cases where this steady-state error is significant, the training loss may remain above the lower boundary of the YES training cloud, residing within the intermediate (yellow) region. This occurrence indicates that the residual error $\mathcal{L}-\mathcal{L}_{\text{opt}}$ exceeds the gap between the YES-K bound $B_{\text{YES-K}}$ and the optimal loss value $\mathcal{L}_{\text{opt}}$. Consequently, the YES bounds serve as a diagnostic tool, revealing that the training process is impeded by persistent errors that prevent convergence to optimality. Detecting such discrepancies prompts practitioners to investigate and rectify underlying issues in the training regimen, such as improving the accuracy of gradient estimates or adjusting hyperparameters, like the learning rate, to facilitate convergence to the desired optimum.

### 5.2 Image Recovery From Corrupted Quadratic Model

To elucidate the practical significance of the YES bounds and their associated cloud system, we further examine an image recovery task for an image degradation process characterized by the model:

$$ $\mathbf{b}_{i}=\left|\mathbf{A}\mathbf{x}_{i}\right|^{2}+\mathbf{n}_{i},\quad i \in[d],$ (29) $$

where each $\mathbf{x}_{i}$ represents a distinct patch of the original image undergoing recovery. This model presents a complex degradation process that involves three primary challenges:

*1. Blurring Operation ($\mathbf{A}\mathbf{x}_{i}$):* The matrix $\mathbf{A}$ applies a blurring operator to the image patch $\mathbf{x}_{i}$, necessitating deblurring techniques to counteract the smoothing effects.

*2. Phase Loss ($\left|\cdot\right|^{2}$):* The absolute value squared operation results in phase loss, requiring phase retrieval methods to restore essential phase information for accurate reconstruction.

*3. Additive Noise ($\mathbf{n}_{i}$):* The term $\mathbf{n}_{i}$ introduces additive noise, demanding denoising strategies to mitigate its adverse effects on image quality.

The performance of the proposed YES bounds and cloud system is illustrated in Fig. 11 and Fig. 12, which present the model’s recovery performance under four distinct conditions:

Figure: (a)
Refer to caption: extracted/5926115/original_cman.png

Figure: (a)
Refer to caption: extracted/5926115/original_boat.png

- •
*Training Loss at Initial Value:* At the outset, the training loss is significantly higher than the YES bounds, indicating suboptimal performance. The recovery results at this stage exhibit pronounced blurring, substantial phase distortions, and noticeable noise artifacts, reflecting the model’s nascent state.
- •
*Training Loss at YES-0 (Top of the Cloud):* As training progresses, the loss approaches the YES-0 bound—the top of the cloud. At this juncture, the model achieves a moderate level of recovery, with reduced blurring and phase errors, alongside diminished noise. However, the performance remains below optimal, as indicated by the fact that the training loss has not yet breached the lower bounds of the cloud.
- •
*Training Loss at YES-$(K-1)$ (Bottom of the Cloud):* Further training brings the loss down to the YES-$(K-1)$ bound—the bottom of the cloud. The recovery results at this stage demonstrate significant improvements, with minimal blurring, accurate phase reconstruction, and negligible noise. This indicates that the model is nearing the performance limits as prescribed by the YES bounds.
- •
*Optimized Convergence:* Upon convergence, the training loss reaches its optimal value, falling within the YES bounds. The recovery results are exemplary, showcasing precise deblurring, flawless phase retrieval, and excellent noise suppression. This final stage confirms that the model has achieved a state of optimal performance, as validated by the YES bounds.

To numerically validate the training results monitored by the YES bounds cloud, we apply these bounds to a corrupted phase retrieval model using two different images: the $128\times 128$ cameraman and the boat, where we consider the patch size of $8\times 8$ of these images for the model in (29).
These images help assess the effectiveness of the YES cloud across various regions and determine whether the training loss entering the green region can indeed lead to practical solutions for real-world tasks, such as image denoising. Figs. 11 and 12(a) show the original image of the cameraman and the boat, respectively, Figs. 11 and 12(b) present the YES cloud used for tracking the training loss with the YES bounds, Figs. 11 and 12(c) illustrate the initial results when the training loss is in the red region, Figs. 11 and 12(d) show the output as the loss enters the cautionary yellow region, Figs. 11 and 12(e) depict the outcomes when the training loss reaches the green region, indicating effective training according to the YES bounds, and finally, Figs. 11 and 12(f) shows the point at which the training loss converges. As illustrated in Fig. 11(c) and Fig. 12(c), the quality of the reconstructed images at the initial stage of training is poor, with noticeable noise, blurring, and patch artifacts. However, in Fig. 11(d) and Fig. 12(d), these imperfections are reduced compared to the earlier stage. Interestingly, by the time the training loss enters the green region, as shown in Fig. 11(e) and Fig. 12(e), the background of the images appears much clearer, with a significant reduction in noise and blurring artifacts. Finally, once the training fully converges in the green region, we observe a well-reconstructed input version, with most distortions effectively removed.

Note that while dropping below the YES cloud signals entering into the effective training regime, it is certainly not recommended to stop the training once this occurs. In fact, it would be reasonable to continue the training, e.g., as long as the rate of decrease in the loss is satisfactorily large. However, we show in this example that one may expect satisfactory performance even if they stop the training prematurely in the green. Based on this, we recommend the following criteria for stopping the training: 1) in the green,
2) the rate of change in network weights is sufficiently low.

### 5.3 MNIST Classification

To further assess the performance of our YES bounds in practical scenarios, we conducted experiments using the MNIST dataset, which was designed for classification tasks. We worked with $5000$ training and $5000$ test samples. Each image, representing a digit $i\in\{0,\cdots,9\}$, was encoded by generating a zero matrix with the same dimension as the input image with a single $1$ placed at $(i+1,i+1)$. A $5$-layer fully connected network was trained with SGD, using an initial learning rate $\eta_{0}$ and a decay factor of $0.7$ every $50$ epochs. The classification was performed by minimizing the MSE between model outputs and encoded images, with the success rate determined by accurate classifications over the entire dataset.

As shown in Fig. 13(a), with a learning rate of $1e-4$, the training loss struggles to move beyond the caution region and remains close to the bottom of the YES clouds. In terms of success rates, Fig. 13(b) displays the training process, while Fig. 13(c) presents the test stage. Although the performance appears satisfactory, the YES cloud suggests that the model’s solution is akin to a linear projection, indicating suboptimal training parameters. Adjusting these parameters could lead to improved model performance.

In Fig. 13(d), we apply a learning rate of
$5e-4$ for the solver. In this case, the training loss reaches the green region after approximately $30$ epochs. The success rate for the training results, shown in Fig. 13(e), indicates that when the training loss enters the yellow region, the success rate is $85$ percent. Once it enters the green region, the success rate increases to $95$ percent, and at the convergence point, we achieve $100$ percent performance. For the test results depicted in Fig. 13(f), the loss reaches the yellow region at $83$ percent, and upon entering the green region, the success rate becomes $92$ percent. At the convergence rate, the test results reach $95$ percent.
As discussed earlier, when the training loss plateaus in the green region, the model’s solution can be a strong candidate for optimality. Figs. 13(e) and (f) illustrate the model’s effectiveness, achieving a $100\%$ success rate in training and $95\%$ in testing. This demonstrates the model’s high accuracy and generalization, indicating that it is well-tuned to the task at hand.

Figure: (a)
Refer to caption: extracted/5926115/MNIST_cloud_newEnDecoder_1.png

## 6 Beyond Certification: Can We Guide the Training Process?

It is our understanding that the YES bounds and their associated clouds provide not only a mathematical framework for certifying AI training but may also aid practitioners as a method to guide the training process. To harness this dual utility without compromising the integrity of the certification process, it is imperative to maintain a clear information barrier between training and certification. This separation ensures that the training algorithm does not gain access to the YES bound data or the certification network weights. Allowing such access would undermine the certification’s purpose by enabling the training process to exploit the bound information, leading to several adverse consequences:

- •
*Loss of Randomization Benefits:* Randomization, particularly during initialization and throughout training, plays a crucial role in escaping local minima and ensuring robust convergence. If the training process can access YES bound data and network weights, it may inadvertently eliminate these randomization benefits, resulting in deterministic and potentially suboptimal training trajectories.
- •
*Faulty Optimization Directions:* The training algorithm might adopt step directions that do not align with the true optimization landscape. Since there is no guarantee that the certification network weights resemble the optimum, leveraging these weights could steer the training process in misleading directions, ultimately degrading the quality of the trained model.
- •
*Obsolescence of the Certification Test:* The primary purpose of the certification test is to provide a reliable bound on the network’s training performance. If the training process can consistently operate at or below this bound by utilizing certification data, the test will be rendered ineffective.

To mitigate the risks associated with direct access to certification data, it is essential to devise mechanisms that allow the training process to benefit from the YES bounds without exposing the certification test itself. One effective strategy is to share only the *test results*, such as those visualized through the YES cloud, rather than the underlying certification data or network weights. This approach provides the training algorithm with a lower bound on the distance between the current loss and the optimal loss without revealing any specific information about the certification criteria. Specifically, the distance of the current loss from the bottom of the YES cloud serves as a valuable indicator for adjusting the learning rate:

$$ $\displaystyle d_{k}=\max\{\mathcal{L}_{k}-\mathcal{L}_{\text{YES}},0\},$ (30) $$

where $\mathcal{L}_{k}$ is the current loss at epoch $k$, and $\mathcal{L}_{\text{YES}}$ represents the lower bound provided by the YES cloud. This distance $d_{k}$ may inform the training process on how large of a step size could be chosen to make meaningful progress, ensuring that the learning rate adapts dynamically based on the proximity to the optimal loss.
A natural implementation of this guidance mechanism involves defining an adaptive learning rate that incorporates both the traditional vanishing component and an additional term based on the distance $d_{k}$ to the YES bounds.

## 7 Conclusion

One may wonder why the YES bounds work as they do. The reason is simple: heuristics outperform random, and optimal beats heuristic. In mathematics, many effective bounds emerge
from insightful heuristics. The YES training bounds are similarly grounded in solid mathematical principles—specifically, that neural networks, with their non-linearities, should outperform linear projections. These bounds provide a useful heuristic that separates the wheat from the chaff, the random from the meaningful. This is particularly evident with the YES-0 bound. But even with the more sophisticated YES bounds and the associated training cloud, the principle holds: they expose the randomness (in the sense of not sufficiently taking the
shape of the optimal) in the solution by offering a better real-time heuristic. Ultimately, as the process nears optimality, we witness the final convergence—optimal surpasses heuristic. This transition is marked by entering the green zone beneath the training cloud.

It is worthwhile to note that the YES bounds and their certification should be regarded as a necessary condition for optimality, not a sufficient one. They provide a certificate of potential optimality—a certificate one must certainly have to assert quality in training. They are, however, a concrete step toward bridge the gap between the opaque nature of neural networks and the pressing need for transparency and reliability in high-stakes applications.

Future work may explore the extension of the YES bounds to various network architectures and training paradigms, as well as their integration into automated training systems. Developing more sophisticated bounds and understanding their impact on the AI training ecosystem are promising avenues for research.

## Acknowledgment

This work relates to Department of Navy Award N00014-22-1-2666 issued by the Office of Naval Research, Mathematical and Resource Optimization Program. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the Office of Naval Research.

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 3
presentation: 2
questions: - Can you clarify how the YES training bounds directly contribute to improvements in model robustness and safety?
soundness: 1
strengths: The proposed system's clarity is enhanced by the color-coded cloud-based monitoring system, which makes it intuitive for practitioners to interpret training status visually.
summary: This paper introduces YES training bounds, a framework for real-time, data-aware certification and monitoring of neural network training. The framework evaluates data utilization efficiency and optimization dynamics, providing insights into training progress and detecting suboptimal behavior. The paper validates the YES bounds using synthetic and real data experiments, offering a tool for certifying training quality and guiding performance enhancements.
weaknesses: This paper has over-claimed its applicability, especially in model robustness and safety. None of the experiments discuss model safety and robustness. Accepting this paper unchallenged may send the wrong signal that the proposed method for enhancing model safety or robustness has been vetted, which it has not due to the omissions of related experiments.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: Q1: Could the authors please clarify L090-L092: "They do not produce varying certification results across different training realizations, even when initialized identically or following similar optimization paths."

Q2: Why would randomness (L088-L095) be such an issue?

Q3: $\mathbf{Y}_{k}$ is never defined. What does it denote?

Q4: Could the authors clarify L245-L253. How does one obtain the YES-SIGMA bound precisely?

Q6: How will the YES cloud help regulators?

Q7: Is there a formalism to justify why projecting from each layer to $Y$ is reasonable?
soundness: 1
strengths: S1: Convergence diagnostics is a worthwhile problem to study. The criticism of local, curvature based convergence diagnostics is justified and hence, providing a empriical, yet strategic "benchmark" framework is an interesting approach.

S2: The proposed method is simple and readily accessible, even to a non-specialist audience, that is likely to benefit most from "training support".
summary: The paper presents a heuristic for convergence diagnostics in multi-layer perceptrons that is data-aware. The authors propose to use solve a OLS-like linear problem for each layer to determine what a reasonable, but suboptimal weight matrix for each layer is. They then propose a traffic light system that compares training loss to this heuristic.
weaknesses: **Motivation**

W1: I am sceptical of the utility of a "standardization of training practices in deep learning" (L078). The author's more detailed account that "The proposed YES [is] a promising pathway toward establishing a benchmark for the AI industry, regulators, and users alike" (L070) is vague and does not provide concrete details on _how_ this is valuable. For instance, the authors could provide examples or consult recent legislation on this issue.

W2: Prior work is not properly cited. In fact, only a single paper (Oymak et al.) on neural network optimisation / convergence diagnostics is cited.

**Presentation**

W3: The Appendix is missing.

W4: The language and notation for the main results is not always clear. Examples:

- Eq 3 uses $\mathbf{Y}_k$ without ever defining it before. I suggest the authors provide a clear definition first.
- First paragraph in Section 4.1 is not clear (L183 - 189). Same for L245-L253.
- The paper describes a theoretical optimal model, the actual model, and the "bounding" model obtained by setting weights through the pseudoinverse. It is not always clear which of these models weight matrices or activations belong to.
- It is not clear what "intermediate states", "intermediate mappings", "intermediate points" are.

W5: The authors are inconsistent in their claims and tend to overstating their contributions. While the authors state at times that their method is a "sanity check", they later state:

- "[...] can attest that the training is not proper." (L205)
- "The answer (YES or NO) will provide immediate relief as to whether training has been meaningful at all" (L211)
- "The reason is simple: heuristics outperform random, and optimal beats heuristic" (L521) or L534 to L536.
- "These bounds aim to provide a qualified answer to the question as to whether a neural network is being properly trained by the data: YES or NO?" (L179).
- "cloud unequivocally indicate suboptimality" (L085). This is a very strong statement, for which there is no supporting evidence.
A more precise account of the contributions and limitations would be appropriate as well as fewer absolute statements without justification.

W6: The proposed method is a _heuristic_ not a certificate, which typically describes provable statements that can be asserted about a model.

W7: The authors are overstating the impact to the safety of models. This work predominantly cites literature on ML safety, but does not set out a clear path how their work impacts safety, how it can establish trust and how it will help regulators. The statement "This standardization could play a crucial role in fostering trust and accountability within the AI ecosystem." lacks proper justification.

**Method**

W8: Using a linear model as baseline during training as bechnmark is common practice (when appropriate). The main insight seems to be the prediction of the target from each intermediate layer. I disagree with the authors statement: "A sensible but sub-optimal approach" (L185) and do not see sufficient justification for this statement. The authors state: "it has also been observed in various machine learning problems that after extensive training (resembling what we can describe as optimal training), the output of some inner layers become something meaningful to domain experts" (L255-258), for which they do not provide sources.

W9: layerwise OLS solutions are a very basic heuristic that do not mark significant contributions to the ML community. One can trivially, see that this bound becomes vacuous, even for single layer models when replacing ReLU with sigmoid (i.e. regard data-generating model $\sigma(AX+e)$ where $A$ has large values.)

W10: At times, the authors do not provide sufficient proof or citation when making non-trivial statements.

- "Given a judicious selection of ... the latter should provide a tighter error bound compared to the YES-0 bounding approach" (L252). What is a judicious selection? Such claim should be supported by a theorem or a more detailed analysis.
- L255-258 as cited above.

W11: The authors solely focus on the train loss, when in practice the test loss is most relevant in learning problems.

**Experiments**

W12: The paper describes experiments on 2 small toy data. This is not enough to extrapolate to real impact. There are various regression and classification datasets that seem like suitable tests for this method. In particular, Boston / California Housing Prices, SVHN, CIFAR-10(0) or TinyImageNet. Convergence diagnostics will become more relevant the more complicated the loss landscape and the more non-linear the problem gets. At the same time the proposed bound will become more vacuous in these settings. A detailed discussion of this would be of interest. Larger, more complex, and high-dimensional datasets are important to judge the potential impact of this method.

W13: The authors do not compare their method against diagnostics baselines. For instance, fitting a simple one-layer linear model, or discussing other strategies for convergence diagnostics.

W14: Figure 3 stops before models are converged.

**Minor**

MW1: It is community standard to have pdf links between in-text citation and the bibliography. This would be appreciated. Citations should be in round parantheses when not not part of the sentence's grammatical structure, e.g. (Goodfellow et al., 2016).

MW2: Figure 3: should share X and Y axis.

MW3: The lack of algorithm boxes, makes it difficult to follow the exact procedures described.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Please discuss the relevant work in the field of optimization and clarify the novelty of this paper.
2. Please clarify the contribution and explain the application of YES bounds in the field of model robustness and reliability.
3. Please explain why the enhanced bounds in Section 4.2.2 can tackle the issues discussed in Section 4.2.1.
4. Please provide more evaluation on different CV/NLP tasks to highlight the generalization of the proposed bounds.
soundness: 2
strengths: 1. The discussed topic is interesting, which focuses on monitoring the training quality and progress. With the proposed bounds, users could better control the optimization, which could benefit the community.
2. By considering the specific structure and properties of training data, the proposed YES bounds could provide tailored and precise evaluations of training performance.
3. Some experiments on image denoising task demonstrate the effectiveness of the proposed bounds.
summary: This paper introduces YES training bounds, a framework for real-time, data-aware certification of training, which aims at assessing the quality of model training. Specifically, these bounds evaluate the efficiency of data utilization and optimization dynamics. The depth and non-linear activation functions of models are taken into consideration. Experimental validation on synthetic and real data demonstrates the effectiveness of the bounds in certifying training quality.
weaknesses: 1. No detailed discussion of relevant works, which makes it difficult to situate this paper. Some discussions of relevant optimization works are missing, such as [a, b].
2. The contribution of this paper could be overclaimed. YES bounds are introduced to indicate the training quality. Although the authors claim that the proposed training bounds aim at improving the reliability, robustness, and safety of models, it is difficult to see how the YES bound can be utilized for such a purpose.
3. This paper discusses the scenario of non-direct paths in Section 4.2.1, however, it is difficult to see how the enhanced bounds that involve intermediate points can tackle this issue.
4. Another concern lies in the evaluation, it is unclear whether the proposed training bounds can be generalized to different CV and NLP applications, such as image generation via diffusion, VQA via LLaVa, etc.
5. It is also unclear how the proposed training bounds can motivate new research works or provide insights into this field.

[a]. Generalization Bounds for Stochastic Gradient Descent via Localized ε-Covers. NeurIPS 2022.

[b]. Closing the convergence gap of SGD without replacement. ICML 2020.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Please address my comments in the Weaknesses section.
2. The experiments show that sometimes the training gets stuck in loss plateaus while in the yellow region. Is there any action that one might take to avoid or overcome this obstacle, other than waiting for the loss to drop? 
3. It is not clear how Deep Unfolding Networks are relevant in the context of this paper, as they are only briefly mentioned in section 4.2.1, without sufficient explanation, and then never mentioned again. Can you give a more detailed explanation?
soundness: 2
strengths: The idea of using linear projections as a weak model to compute upper bounds on the training loss is interesting and original. 

The method is well-described and easy to understand. Mathematical proofs are easy to follow.

Getting insights on the training process beyond the local optimization perspective might be a promising and relevant direction for the future of ML optimization.
summary: The paper explores the challenge of monitoring the quality of training for deep learning models. The paper proposes a method based on upper bounds for the training loss, estimated using linear projections from different layers during training. The authors show that comparing the training loss with the estimated bounds can provide real-time insights into the quality of the training procedure, facilitating the intervention in case of ineffective or sub-optimal training.
weaknesses: * The paper only analyzes small networks with FeedForward architecture and only Linear layers. It is not clear how the procedure could be applied to practical architectures such as ResNet or Transformers. Additionally, there are no concrete details about the architectures used. I assume all hidden layers have the same dimensionality as the input and output, but this should be clearly stated. I suggest the authors extend their experiments to more complex architectures and to include a detailed description of the architectures used (e.g. a table with the number of layers, number of hidden neurons per layer, activation functions).
* The tasks used for training the networks are not clearly explained. It is not clear what is the input and output of the models, what is the loss function and whether the trained models can achieve any significant generalization power. In particular, for the 1D Signal Denoising task, it is unclear how the random signal drawn from $N(0,1)$ could be recovered after applying noise drawn from $N(0,0.2)$. I would like to see more detailed explanations regarding the tasks and the training setup.
* The practical image recovery experiment in the Appendix does not seem to have real practical applications. My understanding is that the model is specifically trained to recover one image after it has been corrupted, which would require full access to the uncorrupted image during training. If this is correct, then this example has no more practical applicability than the synthetic data tasks. It is also unclear how the network is constructed and what are its inputs and outputs. I suggest adding some more detailed explanations about the training and the model architecture.
* The paper presents no analysis of the running time of the proposed method. It is unclear how the method will impact the training time of the model under real-time conditions. It would be interesting to see a comparison of training times with and without the YES bounds computation and an analysis of how the computational overhead scales with model size (I would expect the costs of estimating higher order YES bounds to significantly increase for very deep models).
* The utility of the proposed method concerning train-test generalization is only presented in the Appendix. I consider that this experiment should be presented in the main body of the paper, as ultimately the purpose of training models is generalization to unseen test data. However, it is unclear how well the model is able to generalize in this case: Figure 4 clearly shows a correlation between the evolution of the training and testing losses, but the minimum value of the test loss is similar to the starting value of the training loss, hinting towards very poor generalization capabilities. This is very likely caused by the low correlation between train and test data, so repeating the experiment on real-world data might show more favorable results. Additionally, this experiment does not present any training details (learning rate, batch size, train/test split ratio etc.).

**Minor points**
* Algorithm 1 should have the model as input along with the training data.

**Formatting**
* The Appendix should be part of the main pdf, not as a separate document
* References to Sections, Figures, Tables, Equations and Citations should have hyper-ref links for better navigability. 
* Multiple citations are not well integrated into the text, for example: Line 031 should read “... transformations (LeCun et al.2015, Goodfellow et al. 2016).” which can be achieved by using the \citep{} command. Line 041 should read “Oymak & Soltanolkotabi (2019) theoretically demonstrate …” which can be achieved by using the \citet{} command.
* Line 050 has an unmatched parenthesis after YES.
* Figures 3,4 should have a clear ordering for the learning rates (instead of 1e-3, 1e-2, 1e-4). Subfigure labels for Figures 3,4,5 would also facilitate understanding.