(Almost) Smooth Sailing: Towards Numerical Stability of Neural Networks
Through Differentiable Regularization of the Condition Number

Rossen Nenov * 1 2 Daniel Haider * 1 Peter Balazs 1

Abstract
Maintaining numerical stability in machine learning models is crucial for their reliability and performance. One approach to maintain stability of a
network layer is to integrate the condition numberof the weight matrix as a regularizing term into
the optimization algorithm. However, due to its
discontinuous nature and lack of differentiability
the condition number is not suitable for a gradientdescent approach. This paper introduces a novel
regularizer that is provably differentiable almost
everywhere and promotes matrices with low condition numbers. In particular, we derive a formulafor the gradient of this regularizer which can beeasily implemented and integrated into existing
optimization algorithms. We show the advantagesof this approach for noisy classification and denoising of MNIST images.

Figure 1. Results of MNIST denoising with autoencoders. Top:MNIST images with added Gaussian noise. Mid: No regularization. Bottom: Proposed regularization. While the vanilla autoencoder struggles significantly, the regularized one performs well.

here as it enables the incorporation of such methods into
gradient-based optimization algorithms, hence allowing togradually regularize the numerical stability during training.
In this paper, we focus on the numerical stability of a neural network by means of their weight matrices, and how
to maintain it through regularization. In numerical linear
algebra, a way to measure the numerical stability of a matrix S is via its condition number. For invertible matrices
S it is defined as ∥S∥2 S −1 2 , where ∥·∥2 is the matrix
operator norm induced by the Euclidean norm [5]. If S
is non-invertible or non-square, alternative definitions of acondition number have been proposed. In the context ofleast squares minimization a particularly common one is

1. Introduction
Numerical stability in neural networks refers to the sensitivity of model predictions and training dynamics to small
perturbations in the input data, model parameters, or othercomputational operations. Good stability properties offer
significant benefits, such as consistency enhancement androbustness in predictions, thereby improving generalizationand interpretability [1]. Instabilities can arise due to several
factors, such as the choice of activation functions, initialization, optimization hyperparameters, and quantization effects
during training and inference. Several regularization methods have been developed to intercept this, e.g. dropout,lasso, randomness, etc. [2]–[4]. Differentiability is essential

κ(S) := ∥S∥2 S † 2 ,(1.1)

where S † is the Moore-Penrose inverse, or pseudoinverse [6].
In this sense, a matrix is considered optimally conditioned
if κ(S) = 1. Existing approaches to promote this kindof stability are based on adaptive optimization techniques[7], iterative projections [8], or adding κ directly to thelearning objective as a regularizing term [9]. In the latterapproach, minimizing κ via gradient-based optimization has
been shown to work well. However, it should be noted thatthe function κ : Rn×m → R is discontinuous and, therefore
not differentiable.

*

Equal contribution 1 Acoustics Research Institute, Austrian
Academy of Sciences, Vienna, Austria 2 Department of Mathematics, University of Vienna, Vienna, Austria. Correspondence
to: Rossen Nenov <rossen.nenov@oeaw.ac.at>, Daniel Haider<daniel.haider@oeaw.ac.at>.
Published at the 2 nd Differentiable Almost Everything Workshop
at the 41 st International Conference on Machine Learning, Vienna, Austria. July 2024. Copyright 2024 by the author(s).

1(Almost) Smooth Sailing

Theorem 2.1. For any S ∈ Rn×m the regularizer r(S)defined in Eq. 2.1 is non-negative. If S ̸= 0, then r(S) = 0
if and only if S has full rank and κ(S) = 1.

To address this issue, this paper introduces an alternative
quantity as regularizer that achieves an optimal condition
number, guarantees full rank, and is differentiable almost
everywhere.

2

Proof. Since ∥S∥F =This manuscript is structured as follows. After this introduction we introduce the proposed regularizer and prove that its
minimization is equivalent to minimizing the condition number κ. Subsequently, we prove that the regularizer is differentiable almost everywhere and derive the (sub-)differentialof it in Section 3. Finally, in Section 4 we demonstrate thebenefits in numerical experiments.

2∥S∥2 −Let S ∈ Rn×m be a matrix and let ν = min{n, m}. Defineσ(S) : Rn×m → Rν as the function that maps S to its
singular values in decreasing order. Let σmax (S) := σ1 (S)and σmin>0 (S) := mini∈{1,...,ν} {σi (S) | σi (S) > 0} denote the largest and smallest non-zero singular values ofS, respectively. The rank of a matrix is the number of its
non-zero singular values.
We will denote the singular value decomposition of amatrix S ∈ Rn×m as S = U (Diag(σ(S)))V T , whereU ∈ Rn×n and V ∈ Rm×m are unitary matrices andDiag(σ(S)) ∈ Rn×m denotes a rectangular diagonal matrix
with Diag(σ(S))i,i = (σ(S))i .
Throughout this paper we will use the notion of the condition
number κ(S) of a matrix S as defined in (1.1). This quantity
determines the numerical stability of S by indicating how
much the output Sx can change in response to small changes
in the input vector x. It is known that ∥S∥2 = σmax (S) and−1
that S † 2 = σmin>0 (S) [5]. This means minimizing

ν
1X 2
1
2
2
∥S∥F = σmax(S) −σ (S) ≥ 0, (2.2)νν i=1 iAs noticed before, the condition number is discontinuousand approaches +∞ if a singular value is approaching zero.
Thus, finding a tight connection between the regularizer
r and the condition number κ becomes challenging. The
following theorem provides a relationship between the regularizer and the condition number, capturing the divergent
behavior of the condition number, whenever σmin>0 (S) vanishes. A proof can be found in Appendix C.
Theorem 2.2. For S ∈ Rn×m it holds that
κ(S) ≤ eνσmin>0 (S)

−2

r(S)

.

This theorem shows that for small r(S), the condition number remains small as long as σmin>0 (S) is bounded awayfrom zero. This is important as in practical scenarios, achieving the absolute minimum of regularizers is rarely possible.

max (S)
κ(S) corresponds to minimizing the ratio σσmin>0(S) ≥ 1.
While this quantity is perfectly well-defined for arbitrary
non-zero matrices, it has drawbacks.
Clearly, κ(S) = 1 is equivalent to the situation where allnon-zero singular values of S are equal. However, there is
no guarantee or control over the amount of non-zero singular values. For example, matrices with only one non-zerosingular value trivially attain the minimal condition number κ. Such matrices are not very useful as one is usuallyinterested in full-rank solutions. Furthermore, the mappingκ : Rn×m → R is discontinuous whenever a singular value
approaches zero (Appendix A), and therefore does not allow a proper definition of a gradient. To circumvent both
mentioned issues, we propose a different quantity instead:1
1
2
2∥S∥2 −∥S∥F ,22ν

2i=1 σi (S) [5], we observe thatas σmax (S) is the largest singular value and, hence, the2(S) and the average value of thedifference between σmax
squared singular values of S is always non-negative, thusr(S) ≥ 0. It is straightforward to see that (2.2) attains 0 if
and only if S has ν singular values that are all equal. Since
for full rank matrices S all ν singular values are non-zero,κ(S) = 1 is equivalent to σmax (S) ≡ σi (S) > 0 for alli ∈ {1, ..., ν}, which concludes the result.

2. Matrix Regularizationr(S) :=Pν

We note the close relation to the well-known Tikhonov regularization [10], also known as L2 -regularization or Ridge
regression [11], where ∥S∥2F is used as a regularizer. While
it is effective in improving stability of the problem formulation [12], it does not directly address the issue of maintaining a low condition number of the solution, which is crucial
for numerical stability.

3. Differential Calculus
In this chapter we derive the (sub)-differential properties
and formula of the proposed regularizer r in Eq. (2.1).
First, note that the function r : Rn×m → R is the differenceof two convex functions and therefore non-convex. We willshow that it is differentiable almost everywhere and its subdifferential exists everywhere. For a function f : Rn → R
the subdifferential at a point x̄ ∈ Rn is simply denoted by∂f (x̄). In this work, we use the well-known Mordukhovich
subdifferential ∂M , also known as the limiting subdifferential (Appendix B).

(2.1)

where ∥·∥F denotes the Frobenius-norm for matrices. In
the following theorem we show that matrices that minimizer also minimize κ and additionally have full rank, bothdesirable features for numerically stable matrices.
2

(Almost) Smooth Sailing

Intuitively, this subdifferential generalizes the concept ofsubgradients to non-convex non-smooth functions, capturing the behavior at points where they are not differentiable by considering limits of gradients of nearby smooth
points. The Mordukhovich subdifferential serves as a unifying framework for subdifferentials, accommodating bothsmooth and non-smooth, convex and non-convex functions.
For convex functions it coincides with the convex subdifferential, and for smooth functions it contains only the gradient(Theorems B.6 and B.7). For this reason and to enhanceclarity throughout the main part of the paper, we will use
the notation ∂ for all subdifferentials at points, where thegradient is not unique.

Figure 2. Results of least-squares minimization of (4.1) after 105
iterations for different regularization parameter λ values

In the following theorems we derive the (sub-)differential
for the regularizer r, using classical results from convex
analysis and the work of A.S. Lewis on the differentiabilityof univariate matrix functions [13]. The detailed definitions
and theorems used are listed in the Appendix B.
Theorem 3.1. Let S ∈ Rn×m with a singular value decomposition given by S = U (Diag(σ(S))V T . Let ui and vi
denote the respective column vectors of U and V .

which are intentionally kept basic to illustrate the core concepts. A comparative analysis with Tikhonov regularizationof the results for the experiments is provided in Appendix D.
A Python implementation can be accessed via the following
link: github.com/danedane-haider/Almost-Smooth-Sailing.
4.1. Basic Functionality

If σ1 (S) > σi (S) for i > 1, then r is differentiable at S:

Introducing regularization inevitably leads to a trade-off between the main optimization goal and achieving the desired
regularization. To illustrate the impact of our regularizer, let
us consider the matrix least-squares problem

1∇r(S) = σmax (S)u1 v1T − S.

νOtherwise, the subdifferential of r at S is given by:∂r(S) = conv{σi (S)ui viT | i : σi (S) = σmax (S)} −

1
S.
ν

min

W ∈Rn×m

The following results underscores the practicality of our regularizer. It shows that applying gradient descent steps withrespect to our regularizer decreases the condition number ofthe updated matrix. A proof is found in Appendix C.
Theorem 3.2. Assume that for S ∈ Rn×m the largest singular value is unique, i.e. σ1 (S) > σi (S) for i ∈ {2, . . . , ν}.
Then there exists a L ∈ (0, 1], s.t. for S ′ = S − λ∇r(S)with λ ∈ (0, L] it holds

2∥W X − Y ∥F + λ · r(W ),(4.1)

where X ∈ Rm×d and Y ∈ Rn×d are fixed. The parameterλ ≥ 0 is our regularization parameter, which controls thetrade-off between the objective and the regularization.
For the experiment we choose n = 20, m = 50 and
d = 100, and let X and Y be Gaussian random matrices. We employ gradient descent with the gradient formulas
provided in Theorem 3.1 for 105 steps, compare the resultsfor different values of λ and denote the resulting matrices by Wλ . We repeat this experiment 10 times for eachλ ∈ {0, 10, ..., 100} value.

κ(S ′ ) < κ(S).
Remark 3.1. For a weight matrix at random initialization,r is differentiable almost surely. This follows from the factthat the space of real symmetric matrices with at least one
repeated eigenvalue has codimension 2 in the space of all
real symmetric matrices [14]. Consequently, for a matrix S
whose entries are i.i.d. samples from a typical continuous
distribution, such as Gaussian or uniform, the matrix S T S
has distinct eigenvalues with probability one. Therefore, S
almost surely has distinct singular values.

In Figure 2 we plot the ratio of the (approximation) errors∥Wλ X−Y ∥F∥W0 X−Y ∥F and the condition number κ(Wλ ). We observethat increasing λ leads to an slight increase in the error
and a sharp decrease in the condition number, as expected.
For large values of λ the condition number becomes nearly
optimal, while the error increases by less than 6% comparedto the non-regularized case.
In the following experiments we integrate the regularizer
into the training of a neural network on MNIST [15], demonstrating the benefits of a well-conditioned model in the presence of noise.

4. Numerical Experiments
We demonstrate the functionality and the benefits of the proposed regularizer through a series of numerical experiments,3

(Almost) Smooth Sailingλκ(W1 )

SNR ∞

SNR 1

SNR 0.5

λ1

λ2

κ(E1 )

κ(E2 )

κ(D2 )

κ(D1 )

0
10−3
10−2
10−1
1

43.12
9.43
4.62
1.45
1.53

98.42 %
98.38 %
98.11 %
96.77 %
96.50 %

93.80 %
91.72 %
91.95 %
93.68 %
92.61 %

71.91 %
62.51 %
63.61 %
74.27 %
84.25 %

0
0.1

0
0.005

604.43
23.39

59.58
1.08

30.76
1.07

102.82
12.35

Table 2. Condition numbers of all weight matrices of the trained
autoencoder for image denoising, with and without regularization.
It is noticeable that the weight matrices of the outer layers (E1 , D1 )are absurdly high if not regularized.

Table 1. The table presents the condition numbers of the first network layer (κ(W1 )) and the classification accuracy on the test set
with different SNRs: ∞ (no noise), 1, and 0.5. These results arecompared for different values of the hyperparameter λ. The morenoise is present (low SNR), the more effective the regularizer is.

where x̂ are noisy versions of x with a SNR of 1. The
optimizer is Adam with a learning rate of 0.05 for 50 epochs.

y[i] − log(Φ(x)[i]) + λ · r(W1 ), (4.2)

Table 2 shows the condition numbers of all weight matrices of the naively trained autoencoder (λ1 = λ2 = 0) andthe regularizer one (λ1 = 0.05, λ2 = 0.01). Upon initialobservation, it becomes immediately apparent that the condition numbers of the outer weight matrices are absurdly
high (≈ 600!), indicating severe numerical instabilities. Figure 1 illustrates the denoising performance, qualitatively.
It is noteworthy that the naively trained model appears toencounter difficulties in reconstructing the images at all,whereas the regularized model demonstrates remarkable
performance, even in the presence of substantial noise.

where y[i] and Φ(x)[i] are i-th components of the target andpredicted label vector, respectively. Optimization is donefor 50 epochs using Adam with a learning rate of 0.0001.

We are aware that there exist more optimal architectures
for both classification [16] and denoising [17], however, theobjective of this experiments is to illustrate the advantagesof the proposed regularizer in a fundamental setting.

4.2. Noisy MNIST Classification
For a proof-of-concept approach, we choose the classifiermodel Φ to be a small neural network with two dense layers,
the first one with sigmoid activation, the second one with
soft-max. To stabilize the weight matrix of the first layer,
W1 ∈ R2048×784 , we perform empirical risk minimization(ERM) using a regularized cross-entropy loss
L(x; Φ) = −

10
X
i=1

Table 1 shows the performances of the same model, trainedwith different values of λ, and tested on unseen data withadded Gaussian noise for different signal-to-noise ratios:∞, 1, and 0.5. In the absence of noise (SNR ∞) one canclearly see the influence of the regularizer on the classification performance in terms of the mentioned trade-off. Whenincreasing the noise level the better conditioned models
start outperforming the others due to the induced robustness
properties. At initialization the condition number is 4.28.

5. Conclusion
Traditional approaches to regularization, such as Tikhonov
regularization, often fail to directly address the condition
number, which is crucial for maintaining numerical stability.
This paper introduces a novel almost everywhere differentiable regularizer that enhances numerical stability in neural networks by promoting low condition numbers in theirweight matrices. Through a theoretical analysis and a seriesof numerical experiments, we proved and demonstrated theproperties of this regularizer. In a noisy classification anda denoising task using the MNIST dataset, models that aretrained with the proposed regularizer exhibit significantly
lower condition numbers of the weight matrices and showrobustness against noise successfully compared to models
that are trained without regularization.

Clearly, the final goal is to achieve both, best performancefor all noise levels and optimal stability. However, it needs
further research to determine if and when this is possible.
4.3. Denoising MNIST
For denoising we use a basic autoencoder architecturewith two dense layers in the encoder and decoder, respectively. Let the weight matrices in order of application be
denoted by E1 , E2 , D2 , D1 . We set E1 , D1⊤ ∈ R256×784
and E2 , D2⊤ ∈ R32×256 and use ReLU activation for all
except the last layer, which uses a sigmoid activation. For
training, we perform ERM with respect to the regularizedℓ2 loss

Acknowledgment
The authors thank the reviewers for their time reviewing
and their feedback. The work of R. Nenov and P. Balazs
was supported by the FWF project NoMASP (P 34922). D.
Haider is recipient of a DOC Fellowship of the Austrian
Academy of Sciences at the Acoustics Research Institute (A26355).

L(x̂; Φ) = ∥x − Φ(x̂)∥2 + λ1 · (r(E1 ) + r(D1 ))(4.3)+ λ2 · (r(E2 ) + r(E2 )),4

(Almost) Smooth Sailing

References
[1]

[2]
[3]

[4]

[5]
[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]
[14]

[15]

[16]

[17]

[18]

[19]

[20]

C. Santos and J. Papa, “Avoiding overfitting: A survey on
regularization methods for convolutional neural networks,”
ACM Computing Surveys, vol. 54, 2022.
I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning.
MIT Press, 2016.
H. Zou and T. Hastie, “Regularization and variable selection via the elastic net,” Journal of the Royal StatisticalSociety Series B: Statistical Methodology, vol. 67, no. 2,
pp. 301–320, 2005.
P. Hagemann and S. Neumayer, “Stabilizing invertible neural networks using mixture models,” Inverse Problems,
vol. 37, 2020.
G. H. Golub and C. F. Van Loan, Matrix Computations,Third. The Johns Hopkins University Press, 1996.
L. N. Trefethen and D. Bau III, Numerical Linear Algebra. Philadelphia, PA: Society for Industrial and Applied
Mathematics, 1997.
M. Hasannasab, J. Hertrich, S. Neumayer, G. Plonka, S.
Setzer, and G. Steidl, “Parseval proximal neural networks,”
Journal of Fourier Analysis and Applications, vol. 26,
pp. 1–31, 2020.
M. Cisse, P. Bojanowski, E. Grave, Y. Dauphin, and N.
Usunier, “Parseval networks: Improving robustness to adversarial examples,” Proceedings of the 34th International
Conference on Machine Learning, 2017.
P. Balazs, D. Haider, F. Perfler, and V. Lostanlen, “Trainable signal encoders that are robust against noise,” in Proceedings of the Internoise Conference, 2024.
A. Tikhonov, “Solution of incorrectly formulated problems
and the regularization method,” Soviet Math. Dokl., vol. 4,
no. 2, pp. 1035–1038, 1963.
A. E. Hoerl and R. W. Kennard, “Ridge regression: Biased
estimation for nonorthogonal problems,” Technometrics,
vol. 42, no. 1, pp. 80–86, 2000.
C. Budd, M. Freitag, and N. Nichols, “Regularization techniques for ill-posed inverse problems in data assimilation,”
Computers & Fluids, vol. 46, pp. 168–173, 2011.
A. Lewis, “The convex analysis of unitarily invariant matrix
functions,” Journal of Convex Analysis, vol. 2, 1995.
B. Schlein, “Terence Tao: “Topics in random matrix
theory”,” Jahresbericht der Deutschen MathematikerVereinigung, vol. 115, pp. 57–59, 2013.
Y. LeCun, C. Cortes, and C. Burges, “MNIST handwritten digit database,” ATT Labs [Online]. Available:
http://yann.lecun.com/exdb/mnist, vol. 2, 2010.
W. Hua, C. Li, and X. Wang, “Review of convolutional
neural network models and image classification,” Academic
Journal of Science and Technology, vol. 10, pp. 178–184,
2024.
Q. Zhang, J. Xiao, C. Tian, J. Lin, and S. Zhang, “A robust
deformed convolutional neural network (CNN) for image
denoising,” CAAI Transactions on Intelligence Technology,
vol. 8, 2022.
V. Rakočević, “On continuity of the Moore-Penrose andDrazin inverses.,” Matematichki Vesnik, vol. 49, no. 3-4,
pp. 163–172, 1997.
R. Rockafellar, Convex Analysis (Princeton Landmarks
in Mathematics and Physics). Princeton University Press,
1997.

[21]

[22]

5

B. Mordukhovich, Variational Analysis and Applications(Springer Monographs in Mathematics). Springer International Publishing, 2018.
D. Calvetti and L. Reichel, “Tikhonov regularizationof large linear problems,” BIT Numerical Mathematics,
vol. 43, no. 2, pp. 263–283, 2003.
M. P. Deisenroth, A. A. Faisal, and C. S. Ong, Mathematics
for Machine Learning. Cambridge University Press, 2020.

(Almost) Smooth Sailing

Appendix
A. On the Discontinuity of the Condition Number
Example A.1. This example shows that the condition number κ(S) := ∥S∥2 S † 2 is not a continuous mapping [18]. Let




1 0
0 0
A=
and E =.
0 0
0 1
For each 1 > ε > 0 we have†

(A + εE) =



† 
1 0
1
=
0 ε
0

0
ε−1



.

Therefore κ(A + εE) = ∥A + εE∥2 (A + εE)† 2 = ε−1 . Hence A + εE → A as ε → 0, but limε→0 κ(A + εE) does
not exist, even though κ(A) = 1.

B. Essentials from Subdifferential Calculus
In this part of the appendix, we include all definitions and results used in the paper to make the document self-contained andthe derivation of the Theorems precise and unambiguous.
B.1. Convex Subdifferential [19]Definition B.1 (Proper function). An extended value function f : Rn → R ∪ {+∞} is called proper, if its domaindom(f ) := {x ∈ Rn : f (x) < +∞} is not empty.
Definition B.2 (Convex Subdifferential). Let f : Rn → R∪{+∞} be a proper convex function. The (convex) subdifferentialof f at x ∈ dom(f ) is defined as∂f (x) = {v ∈ Rn | f (y) ≥ f (x) + ⟨v, y − x⟩ ∀y ∈ Rn }.
The elements of ∂f (x) are called subgradients of f at x.
Remark B.3. If x ∈/ dom(f ), then ∂f (x) = ∅.
B.2. Mordukhovich Subdifferential [20]Definition B.4 (Mordukhovich Subdifferential). Let f : Rn → R ∪ {+∞} be a lower semicontinuous function. The
Mordukhovich (or limiting) subdifferential of f at x ∈ dom(f ) is defined asnoˆ (xk ) and f (xk ) → f (x) ,∂M f (x) = v ∈ Rn | ∃xk → x, v k → v with v k ∈ ∂f
ˆ (x) denotes the Fréchet subdifferential of f at x, defined bywhere ∂f

f (y) − f (x) − ⟨v, y − x⟩nˆ∂f (x) = v ∈ R | lim inf≥0 .
y→x∥y − x∥Remark B.5. The Mordukhovich subdifferential generalizes the concept of subgradients to non-convex functions and isparticularly useful in variational analysis and optimization.
B.2.1. C OINCIDENCE OF S UBDIFFERENTIALS [20]Theorem B.6. Let f : Rn → R ∪ {+∞} be a proper, lower semicontinuous, and convex function. Then, for any x ∈ Rn ,∂f (x) = ∂M f (x).
Theorem B.7. Let f : Rn → R be a differentiable function at x ∈ Rn . Then,∂f (x) = ∂M f (x) = {∇f (x)}.
nTheorem B.8. Let f : R → R ∪ {+∞} be a proper, lower semicontinuous, and convex function. If the convex
subdifferential ∂f (x) at x ∈ Rn is a singleton, say ∂f (x) = {v}, then f is differentiable at x and ∇f (x) = v.
6

(Almost) Smooth SailingB.2.2. M ORDUKHOVICH S UBDIFFERENTIAL OF THE S UM OF F UNCTIONS [20]Theorem B.9. Let f : Rn → R ∪ {+∞} be a proper, lower semicontinuous, and convex function, and let g : Rn → R be adifferentiable function. Then, for any x ∈ dom f (x):∂M (f + g)(x) = ∂f (x) + ∇g(x).
B.3. Rules of Differentiation [20]Definition B.10 (Convex Hull). Let S be a subset of Rn . The convex hull of S, denoted by conv(S), is the smallest convex
set containing S. It can be defined as:
)
( k
k
X
X
λi = 1, k ∈ N .
λi xi xi ∈ S, λi ≥ 0,conv(S) =i=1

i=1

In other words, the convex hull of S is the set of all convex combinations of points in S.
Theorem B.11. Let {fi }i∈I be a finite family of proper, lower semicontinuous, convex, and differentiable functions from
Rn → R. Define f : Rn → R by
f (x) = max fi (x).
i∈I

Then, for any x ∈ Rn ,∂M f (x) = conv {∇fi (x) | i ∈ Ix } ,where Ix = {i ∈ I | fi (x) = maxi∈I fi (x)} denotes the set of indices, for which fi attains the largest value at x.
B.4. Results on unitarily invariant matrix functions [13]Definition B.12 (Absolutely Symmetric Function). A function f : Rq → R is said to be absolutely symmetric if
f (γ) = f (γ s ) for any permutation s of the components of γ and for any γ ∈ Rq . Equivalently, f is absolutely symmetric if
f (Qγ) = f (γ)

for all γ ∈ Rq and Q ∈ Λq ,where Λq denotes the set of generalized permutation matrices (matrices with exactly one non-zero entry in each row andeach column, that entry being ±1).
This means that the function value at x of absolutely symmetric functions is independent of the ordering of the entries of x.
Recall that σ maps a matrix onto its singular values in nonincreasing order.
Theorem B.13 (Characterization of Convexity). Suppose that the function f : Rq → (−∞, +∞] is absolutely symmetric.
Then the corresponding unitarily invariant function f ◦ σ is convex and lower semicontinuous on Cm×n if and only if f isconvex and lower semicontinuous.
Theorem B.14 (Characterization of Subgradients). Suppose that the function f : Rq → (−∞, +∞] is absolutely symmetric,
and that the m × n matrix X has σ(X) in dom(f ). Then∂(f ◦ σ)(X) = U (Diag µ)V T µ ∈ ∂f (σ(X)), X = U (Diag σ(X))V T .
Theorem B.15 (Gradient Formula). If a function f : Rq → (−∞, +∞] is convex and absolutely symmetric then thecorresponding convex, unitarily invariant function f ◦σ is differentiable at the m×n matrix X if and only if f is differentiableat σ(X). In this case,∇(f ◦ σ)(X) = U (Diag ∇f (σ(X)))V T ,for X = U (Diag σ(X))V T .

7

(Almost) Smooth Sailing

C. Proofs
In this section we proof Theorems 3.1, 2.2, and 3.2 formulated in the main body of the paper.
Proof of Theorem 3.1
Proof. We define f : Rν → R and g : Rν → R as
ν

1 2
xi ,
i∈1...ν 2

g(x) =

f (x) = max

1 X 2
x .
2ν i=1 i

One can see that by Definition B.12 both f and g are absolutely symmetric and r(S) = f (σ(S)) + g(σ(S)). The function f
is convex by the rule for pointwise maxima Theorem B.11 and its subdifferential is given by∂f (x) = conv{xi ei | i : x2i = x2max },where ei denotes the i − th standard unit vector and max denotes the index, for which the largest value is attained. By the
results of Theorem B.13 we deduce that f (σ(S)) is convex and by Theorem B.14∂(f ◦ σ)(S) = conv{σi (S)ui viT | i : σi (S) = σmax (S)}.
The function g is differentiable everywhere and its gradient is given by ∇g(x) = ν1 x. By Theorem B.15 we deduce∇(g ◦ σ)(S) = U (Diag(∇g(σ(S)))V T
11= − U (Diag(σ(S))V T = − S.
ννThus, by Theorem B.9, we derive1S.
νIf the largest singular value of S is unique, ∂r(S) is a singleton and by Theorem B.8 therefore differentiable with∂r(S) = conv{σi (S)ui viT | i : σi (S) = σmax (S)} −∇r(S) = σmax (S)u1 v1T −1
S.
N

Proof of Theorem 2.2
Proof. Using the Mean Value Theorem for the logarithm, we derive for c ∈ (σk (S)2 , σ1 (S)2 )2 ln (κ(S)) = ln (κ(S)2 )= ln (σ1 (S)2 ) − ln (σk (S)2 ),1
σ1 (S)2 − σk (S)2 ,=c
1(σ1 (S)2 − σk (S)2 ).
≤σk (S)2
Furthermore estimating the regularizer yieldsk2r(S) = σ1 (S)2 −1X
σi (S)2 ,
ν i=1


1
(k − 1)σ1 (S)2 + σk (S)2 ,ν1(ν − 1)σ1 (S)2 + σk (S)2 ,≥ σ1 (S)2 −ν1= (σ1 (S)2 − σk (S)2 ).
ν≥ σ1 (S)2 −8

(Almost) Smooth Sailing

Combining those two results we conclude:ln (κ(S)) ≤νr(S),σk (S)2νκ(S) ≤ e σk (S)2

r(S)

.

Proof of Theorem 3.2
Pν
Proof. Let S = U Diag(σ(S))V T = i=1 σi (S)ui viT be the singular value decomposition of S. Let k be the index of thesmallest non-zero singular value, i.e. k = argmini {σi (S) | σi (S) > 0 for k ∈ {1, . . . , ν}}. Then it holds that:S ′ = S − λ∇r(S)λS − λσ1 (S)u1 v1T
= 1+
ν

 ν


λ
λ X
T
= 1 + − λ σ1 (S)u1 v1 + 1 +σi (S)ui viTνν i=2= U Diag(σ ′ )V T ,
where σ ′ = 1 + λν − λ σ1 (S), 1 + λν σ2 (S), . . . , 1 + λν σν (S) . Given this decomposition of S ′ , we notice that σ ′are the singular values of S ′ , but not necessarily in the right order. Therefore we are going to distinct two cases.

Since S is assumed to have a unique largest singular value, there exists an α > 1 s.t. σ1 (S) = ασ2 (S). Notice that1
κ(S) ≥ α holds. Choose L ∈ (0, 1], s.t. for all λ ∈ (0, L] : 1 − 1+λ λ ≥ κ(S).
ν

1. Case: α1 < 1 − 1+λ λ .
νBy the case distinction we see:


1+




λ
1
λ
1+
σ1 (S),− λ σ1 (S) >
ν
α
ν


λ
σ2 (S),= 1+νand thus σ1′ > σ2′ . Since σi (S) ≥ σj (S) for i < j, we deduce that σ ′ are exactly the singular values of S ′ in non-increasingorder. Furthermore the amount of non-zero singular values stays the same. Therefore
1 + λν − λ σ1 (S)σ1′′κ(S ) = ′ =σkσk (S)
1 + λν
!
λ
= 1−
κ(S).
1 + λν
1.
2. Case: α1 ≥ 1 − 1+λ λ ≥ κ(S)ν

Similarly to the previous case, simple arithmetics show that in this case σ2′ becomes the largest singular value of S ′ and σk′remains the smallest. Therefore:κ(S ′ ) =σ2 (S)σ2′
1 σ1 (S)==σk′σk (S)
α σk (S)1= κ(S).
αIn both cases we see that κ(S ′ ) < κ(S), since λ ∈ (0, L] and α > 1.

9

(Almost) Smooth Sailing

Figure 3. MNIST denoising results with (bottom) and without regularization (mid) with three different SNRs, from left to right: 10, 1, 0.5.
Already with reconstructing the images from almost no noise, the non-regularized autoencoder struggles. Due to the high condition
numbers in the network, the output is very sensitive to perturbations in the input, resulting in the network being unable to learn properly.

Figure 4. Left: Results of MNIST denoising with SNR 1 with Tikhonov regularization for two different sets of parameters. Mid:λ1 = 0.01, λ2 = 0.0001. Bottom: λ1 = 1, λ2 = 0.01. Right: Results of least-squares minimization of (4.1) after 105 iterations with
Tikhonov Regularizer for different regularization parameter λ values

D. Discussion on Tikhonov Regularization
Tikhonov Regularization is a well-known method used to stabilize the solutions of ill-posed problems and to prevent
overfitting in machine learning models [21], [22]. This method adds a regularization term to the loss function, which
penalizes large coefficients in the solution, and helps in balancing between fitting the data and simplicity in the solution.
We include this discussion on Tikhonov regularization due to its widespread recognition and frequent application as acommon regularizer, although its direct impact on the condition number of the weight matrices remains, to the best of theauthors knowledge, ambiguous in the existing literature.
Mathematically, Tikhonov regularization adds to the standard least squares problem a penalty term proportional to the squareof the norm of the coefficients, in our case:min

W ∈Rn×m

2

2∥W X − Y ∥F + λ · ∥W ∥F ,where λ is the regularization parameter controlling the trade-off between model fit and the magnitude of coefficients [10].
D.1. Impact on Condition Number
While Tikhonov regularization is effective in promoting stability in poorly behaved optimization problems and preventing
overfitting, it does not directly address the condition number of the weight matrices. In the context of neural networks, using
the Frobenius norm as a regularizer can help in reducing the overall magnitude of the weights but might not sufficiently
control the condition number, as we will see in the following simulation.
2

We repeat the simulations from Section 4.1 for Tikhonov Regularization, where r(W ) = ∥W ∥F in (4.1). The results areshown in Figure 4. We observe that Tikhonov regularization slightly reduces the condition number, but not significantly,
before the (approximation) error ∥Wλ X − Y ∥F becomes too large compared to the non-regularized error ∥W0 X − Y ∥F .
10

(Almost) Smooth Sailingλκ(W1 )

SNR ∞

SNR 1

SNR 0.5

0
10−3
10−2
10−1
1

43.29
56.43
242.394044.34
4020.47

98.42 %
98.31 %
98.25 %
98.15 %
97.68 %

93.80 %
94.34 %
94.83 %
95.10 %
95.02 %

71.91 %
74.24 %
76.94 %
78.67 %
82.79 %

Table 3. Results when using Tikhonov regularization for MNISTclassification.

λ1

λ2

κ(E1 )

κ(E2 )

κ(D2 )

κ(D1 )

1
0.1

0.01
0.001

12.80
1277.211.4331641.66

2.6520358.70

6061.535347.09

Table 4. Condition numbers of the weight matrices when using
Tikhonov regularization for denoising in two parameter settings.

D.2. Numerical Experiments with Tikhonov Regularization
We conducted numerical experiments on the same problems as in Section 4 to compare the performance of Tikhonov
regularization with the proposed method. The results are summarized below:
D.2.1. MNIST C LASSIFICATION
Table 3 shows the classification accuracy and condition numbers for different levels of noise (SNR) using Tikhonov
regularization. As λ increases, the condition numbers of the weight matrices increase dramatically, indicating potential
numerical instability. Yet, we see that it performs well at a medium noise level (SNR 1) and outperforms the baseline, as
well as our proposed regularizer in all settings of λ. For the high noise level (SNR 0.5), however, it falls back again.
D.2.2. MNIST D ENOISING
Figure 4 illustrates the denoising performance with low (mid) and high (bottom) influence by means of the values λ1 , λ2 .
Table 4 shows the corresponding condition numbers. We see that Tikhonov regularization fails to maintain low condition
numbers consistently across all layers, which explains the worse denoising performance compared to the proposed regularizer.
D.3. Conclusion
While Tikhonov regularization offers benefits in terms of regularizing the magnitude of weights and robustness to a medium
level of noise, it does not effectively control the condition number of weight matrices, leading to potential numericalinstability. The proposed regularizer in this paper addresses this limitation by specifically targeting the condition number,thus enhancing the robustness and stability of neural networks, especially in very noisy environments.

11