Training More Robust Classification Model via Discriminative Loss and Gaussian Noise Injection
Authors: Hai-Vy Nguyen
Fabrice Gamboa
Sixin Zhang
Reda CHHAIBI
Serge Gratton
Thierry Giaccone
Venue: Accepted by TMLR ()

Abstract: Robustness of deep neural networks to input noise remains a critical challenge, as naive noise injection often degrades accuracy on clean (uncorrupted) data. We propose a novel training framework that addresses this trade-off through two complementary objectives. First, we introduce a loss function applied at the penultimate layer that explicitly enforces intra-class compactness and increases the margin to analytically defined decision boundaries. This enhances feature discriminativeness and class separability for clean data. Second, we propose a class-wise feature alignment mechanism that brings noisy data clusters closer to their clean counterparts. Furthermore, we provide a theoretical analysis demonstrating that improving feature stability under additive Gaussian noise implicitly reduces the curvature of the softmax loss landscape in input space, as measured by Hessian eigenvalues.This thus naturally enhances robustness without explicit curvature penalties.  Conversely, we also theoretically show that lower curvatures lead to more robust models. We validate the effectiveness of our method on standard benchmarks and our custom dataset. Our approach significantly reinforces model robustness to various perturbations while maintaining high accuracy on clean data, advancing the understanding and practice of noise-robust deep learning.

Published in Transactions on Machine Learning Research (12/2025)
Training More Robust Classification Model via Discrimina- tive Loss and Gaussian Noise Injection
Hai-Vy Nguyen
Ampere Softwore Technology - Institut de mathématiques de Toulouse
Institul de Recherche en Informatique de Toulouse
Fabrice Gamboa
fabrice, Jabrice.gambooeeeee... @mmtto wwiv-toulouse
Instifut de mathématiques de Toulouse.
Sixin Zhang
sizin. Surin.hhang@rirk
Institut de Recherche en Informatique de Toulouse
Reda Chhaibi
reda.chhatbrrrmmivvcoceeeoo fr
Laboraloire Jean Alezandre Dicudonné, Université Côte d' d'Azur Serge Gratton
scrg.gguulonn@Boulluusscchh.pppp
Instifut de Recherche en Informatique de Toulouse
Thierry Giaccone
thierv.jiacconeeSapper.cccr
Ampere Software Technology
Reviewed on OpenReview: https: // openreview. net/ formm? id- f
Abstract
Robustness of deep neural networks to input noise remains critical challenge, as naive noise injection often degrades accuracy on clean (uncorrupted) data. We propose a novel training framework that addresses this trade-off through two complementary objectives. First, we introduce a loss function applied at the penultimate layer that explicitly enforces intra-class compactness and increases the margin to analytically defined decision boundaries. This enhances feature discriminativeness and class separability for clean data. Second, we propose a class-wise feature alignment mechanism that brings noisy data clusters closer to their clean counterparts. Furthermore, we provide a thcoretical analysis demonstrating that improving feature stability under additive Gaussian noise implicitly reduces the curvature of the softmax loss landscape in input space, measured by Hessian eigenvalues. This thus naturally enhances robustness without explicit curvature penalties. Conversely, we also theoretically show that lower curvatures lead to more robust models. We validate the effectiveness of our method on standard benchmarks and our custom dataset. Our approach significantly reinforces model robustness to various perturbations while maintaining high accuracy on clean data, advancing the understanding and practice of noise-robust deep learning.
1 Introduction
Deep neural networks have achieved remarkable success across a wide range of tasks. However, their in- stability to input perturbations, including noise and adversarial atlacks, remains major issue (Szegedy) ett al., 2014 Goodfellow et al. 2015). Training models to be robust against such perturbations is crucial for deploying reliable machine learning systems in real-world noisy environments. For instance, when a classifi- cation model is trained only with uncorrupted data, it tends to produce features that are not well separated between classes when facing noisy data (Fig. Ta
A common and straightforward approach to improve robustness to augment the training data by adding some noise (Bishop) 1995). While noise injection often improves model robustness, can also degrade the accuracy the uncorrupted (clean) data. This trade-off arises because naively forcing the model to fit

---

Published in Transactions on Machine Learning Research (12/2025)
noisy data generally reduces the discriminativeness of the learned features. This results in blurred decision boundaries and ambiguous feature representations (Fig. Tb
Therefore, naively injecting noise into the training data is inadequate and may adversely affect the model's performance on the original data distribution. This observation motivates the need for principled methods
that explicitly balance robustness to noisy inputs with the preservation of discriminative feature representa- tions for original data (features produced by our method are depicted in Fig. Ic
data
Naisy data
data
Noisy data
(a) Training with clean training data.
(b) Training with both clean and noisy data. Naisy data
Clean data
(c) Training with our method.
Figure 1: I-SNE feature visualization for test set of CIFAR10 of original (clean) and noisy data (additive Gaussian noise), produced by models trained with different methods. When training the model with the clean data, it produces features that are not well separated between the classes for noisy data (right plot of Fig. Tap. Training model with both clean and noisy data helps to produce more discrininative features for the noisy data (Fig. 1bp but also makes features on the clean data less discriminative. Our method helps the model to produce more discriminative features both clean and noisy data (Fig. Ic
In this paper, we propose a novel framework that explicitly addresses these challenges by focusing on two complementary objectives (illustrated in Fig. 2
1. Boosting the discriminativeness of features for clean data. We introduce new loss operating at the penultimate layer of the network, which encourages intra-class compactness and simultane- ously increases the margin between features and decision boundaries. Crucially, in this penultimate fcature space, decision boundaries can be analytically characterized as hyperplancs. This enables an explicit and tractable formulation for the loss. Consequently, by directly shaping the geometry of the feature space, our loss strengthens class separability, mitigating the blurring effect that noise injection may induce.
2. Aligning noisy data features with clean data clusters. Unlike classical approaches that enforce individual noisy samples to be close to their clean counterparts (c.g., stability training (Zheng et al. 2016) or Lipechitz-constraaiedd networks (Tsuzuku et al. 2018)), our method performs class-wise alignment. Specifically, we encourage the entire cluster of noisy samples for a given class to be close to the corresponding cluster of clean samples from the same class. This cluster-level alignment is less restrictive and allows the model to maintain expressiveness, avoiding the performance degradation often observed when forcing strict pointwise matching between noisy and clean samples.
2

---

Published in Transactions on Machine Learning Research (12/2025)
Feature Space
Uncorrupted (clean) data
Noise
Model
injection
Noisy data
Figure 2: Illustration of our method applicd on the features of the penultimate layer. the feature space, each color represents the data cluster of each class, where darker and lighter colors represent clean and noisy data, respectively. Our method focuses on boosting discriminativeness of features on clean data (by enforcing intra-class compactness and inter-class separability) and aligning noisy data clusters with those of clean data from the same class (in feature space).
Theoretical insights on noise injection and curvature. Beyond the novel loss function design, we provide a theoretical analysis revealing that training with Gaussian noise implicitly acts as loss curvature regularizer in the input space (that we call inpul loss curvature). More precisely, we show that noise injection reduces the eigenvalues of the Hessian of the loss with respect to the input, effectively smoothing the local loss landscape. This curvature reduction serves as a natural regularization mechanism that enhances robustness against imput perturbations without explicitly adding curvature penalty terms. Conversely, we also show that a lower curvature in the input space leads to a model producing more stable features under Gaussian perturbations. This insight bridges the gap between noise-based robustness methods, which traditionally rely on data augmentation or smoothing, and curvature-base of the loss surface (Moooavi-Dezzooli al.) (0019). Our work is, to the best of knowledge, the first to formally link Gaussian noise training, curvature reduction in input space, and improved robustness in unified framework.
approaches that explicitly constrain the geometry
Contributions. The main contributions of this paper are summarized as follows:
We propose a novel loss function applied at the penultimate layer that simultancously enforces intra-class compactness and maximizes inter-class margins with respect to analytically defined decision boundaries (hyperplanes). This encourages more discriminative feature representations for clean data. We also provide theoretical insights showing that enforcing only intra-class compactness or only inter-class sep- arability lead to trivial solutions that fail to improve model performance (Proposition 5.3 This justifies our design choice of combining both constraints.
In addition to the loss applied on clean data, we introduce simple yet effective class-wise feature alignment strategy that encourages clean and noisy data clusters to align in the feature space. This contrasts with classical point-wise stability methods and helps preserve model expressiveness, avoiding the performance degradation commonly seen with naive noise augmentation (see our method in Fig. Ic
We theoretically demonstratt that training with additive Gaussian noise reduces the eigenvalues of the input-space Hessian of the standard cross-entropy loss, effectively regularizing curvature. Conversely, we also prove that reduced curvature in the input space leads to more stable features under Gaussian noise, from probabilistic slandpoint (Theorem 6.1
We provide further thcoretical insights into the proposed method. This includes a generalization behavior analysis for the compactness constraint (Theorem 6.2), and an investigation of how applying the loss solely at the penultimate layer impacts intermediate representations (Theorem 6.3
3

---

Published in Transactions on Machine Learning Research (12/2025)
We demonstrate the theoretical insights and validate the effectiveness of our method through experiments
on both standard benchmarks and our road image dataset (Section 7 Notably, models trained exclusively on Gaussian-noised data using our approach show substantial performance gains over baseline methods when evaluated on other perturbations, such as random occlusion and resolution degradation (see Fig. 7 for examples).
Paper Organization. Section 2 reviews related works. Section g introduces the standard softmax clas- sification framework and the notations used throughout the paper. It also discusses thcoretical decision boundaries and the limitations of the standard softmax loss. Building on this, Section g presents our pro- posed method, followed by a discussion of its design properties in Section 5 Theoretical insights into our method are provided in Section
[ including the impact of our method on input-space curvature under Gaus- sian noise (Section 6.1], generalization analysis of the compactness constraint (Section 6.2 and the effect of our method on intermediate representations (Section (.3). Experimental results on standard and custom datasets are presented in Section 7 Our code is publicly available at (code-robust-trainnng
2 Related works
Adversarial perturbations and Lipschitz models. Training models that are more stable against input perturbations is an important subject in deep learning. A well-studied direction is to include examples with adversarial perturbations in training &Dong et al. 2020 Miyato et al. 2016 Qin et al., 2019). While this approach has proven its effectiveness, it is very time-consuming to generate adversarial perturbations. Another direction is to construct models with Lipschitz property so that features of the corrupted input stay close to those of clean counterpart (Fazlyab et al. 2023 Zhang et al.) 2022 2021). This approach requires to modify the model architecture (to ensure Lipschilz property by construction), so it is not applicable on existing base model.
Noise Injection. Training neural networks with noise injected into inputs or weights is a classical technique for improving generalization (e.g.: Bishop (1995); Zheng el al. (2016); He et al. (201g)) Bishop established that input noise training is approximately equivalent to Tikhonov regularization in the parameter space, of- fectively smoothing the learned function. However, this equivalence does not explicitly connect noise injection to changes in the curvature of the loss landscape in the input space. More recent research has leveraged noise during inference to improve robustness against perturbations such the method of randomized smoothing
CCohen et al. 2019 Levine & Feizi, 2020 Scholten et al., 2023). This method is more computationally expensive, it requires multiple inference passes per data point to average the model outpuls over Gaussian noise added to the inputs. Moreover, it does not directly analyze or control the curvature of the loss function with respect to the inputs.
Loss curvature in Deep Learning. Understanding the geomelry and curvature of the loss landscape pe in deep learning has been an important area of research for understanding the model stability. These studies mainly focus on the model parameter space and reveal that sharp or flat minima in the parameter space significantly affect model's generalization performance (see, e.g., Foret et al. (2021); Dinh et al. 20117 Li et al. (0188) Andriushchenko & Flammarion (2022)). However, the relationship between the model stability and the loss curvature in the input space seems to be less studied. Moosavi-Dezfooll et al. Moosavi-Deztooli
et al. 2019 demonstrate that the Hessian of the loss function with respect to the input captures local curvature information that is essential for understanding a model's sensitivity to perturbations. Several works have explored explicit regularization strategics targeting curvature metrics, such as Hcssian norms of the model output loss with respect. to the inputs (Mustafa al. 2020 Moosavv-Dezzoolli et al.) 2019). Most of these approaches requires the explicit computation or approximation of the Hessian, which can be computationally expensive. Other related methods focus on Jacobian regularization (Rifai et al. 2011 or smoothing techniques for boosting local linearity (Qin et al. 2019) that implicitly reduce sensitivity to input perturbations but do not necessarily analyze the sigenvalues directly.
Discriminativeness in feature spaces. To enhance the discriminativeness of learned representations, Tang (0013) proposed a margin-bascer approach that integrates multi-class support vector machines (SVMs) into deep networks. Classical SVMs and the deep SVM formulation of Tang 2013) essentially maximize
4

---

Published in Transactions on Machine Learning Research (12/2025)
inter-class margins via objectives defined over separate linear classifiers This is done by replacing the softmax layer with C' one-vs-rest. SVMs, introducing additional classifier parameters and slack variables. This setup enforces the margin at the classifier output level but does not directly constrain the geometry of the features. In contrast, our approach retains the standard softmax layer and instead defines, analytically, the decision boundaries induced by this softmax classifier in the feature space. This enables us to impose margin-based constraints and intra-class compactness directly on the learned features without introducing additional classifiers or slack parameters. [Elaaydd al. (2018) similarly advocate maximizing the margin to decision boundaries in feature space via quadratic approximation. However, their formulation focuses solely on the approximated Euclidean margin and does not impose constraints on intra-class compactness. In contrast, our loss explicitly incorporates intra-class compactness, which plays a crucial role in encouraging smaller curvature, as analyzed in our theoretical section. In another line of work. Wen el al. (2016) introduced the center loss, which explicitly encourages intra-class compactness by minimizing the distance between each feature and its corresponding class centroid. However, this formulation does not explicitly enforee inter-class separability. The advanlages of large-margin learning in deep neural networks were further emphasized by Liu et al. (2016) and et al. [2017]. More recently, Papyan et al. (2020) observed that within-class variation tends to vanish if one continues training a softmax- based classitier beyond the zero-error regime. While this finding is theoretically appealing, achieving the zero-error regime in practice is challenging. Therefore, it is often desirable to introduce supplementary discriminative constraints, as in the aforementioned works. Tn our method, we propose to simultaneously enforce intra-class compactness and inter-class separability to enhance feature discriminativeness when training with noise-injected inputs.
Our contribution compared to related literature. In contrast to prior work, our method theoreti- cally and empirically demonstrates that training with Gaussian noise inherently lcads to reduction in the eigenvalues of the Hessian of the loss function with respect to the inputs. This mechanism acts as im- plicit curvature regularizer, naturally flattening the local loss landscape without the need for explicit penalty terms. Conversely, we also demonstrate that the curvature reduction correlates with robustness to Gaus- sian noise perturbations at inference time. This provides a principled explanation for why noise injection enhances oobustness beyond classical generalization arguments. To the best of our knowledge, this explicit link between noise injection during training, input loss curvature reduction, and improved noise robustness has not been previously characterized in the literature. Morcover, our method also emphasizes enhancing the discriminative power of clean data features to ensure strong performance on clean inputs. This is in contrast with standard approaches, where one typically focuses only on improving the accuracy on noisy (corrupted) data. Our paper thus offers both practical and theoretical insights that complement and extend existing approaches.
3 Background and framework
3.1 Preliminaries: classification with softmax model
Let us consider a classification problem with C' classes (C2 2). The input space is denoted by X, which can be a space of images, time scries or vectors. The neural network (backbone) transforms an input into fixed-dimension vector. Formally we model the network by a function: So J C Rd, where 0 is the set of parameters of the neural network, d is the dimension of the so-called feature space F. That is, F is the representation space induced from X' by the transformation fe. Given an input E X, let q = fo(x). In order to perform a classification task, is then passed through a softmax layer consisting of an affine (lincar) transformation (Eq. (1) and a softmar function (Eq. 2 Expressed in formal equations, we have
2 = Wq + b, W E Cxd and be C
(1)
d(2);
(2)
Ej=1 e²
Here, for = 1,... C, 2 (resp. (z)) is the 8th component of column-vector (resp. (2)).. The components 2, 's of are called logits (and so is called logit vector). The predicted class is then the class with maximum
5

---

Published in Transactions on Machine Learning Research (12/2025)
value for (2), i..
πxx) arg max (z))
summary, the whole softmax model can summarized as the following sequence of transformations in Fig.
fo
=(1) RC softmax function Affine (W,b)
X
((x) E Rd
a(=) E 6 R
No
Figure 3: Pipeline from input x to softmax output a(2) E AC-1, which is the simplex in RC of probability measures. The whole softmax model is denoted by Ne, composed of fa and the softmax layer, where O is the concatenation of all the parameters of (0, W.
Remark 3.1 We nole that arg max; (2); arg max; zi. So thal, arg max; Zi-
In the standard approach, to train the neural network to predict maximal score for the right class, the softmax loss is used. This loss written as
Ls(O;B) = [B]
Σ
(xx,)),
(3)
(x.0)EB
where
1(x log a (W (((c))+b)))) b),
(4)
Here, à is the concatenation of all the parameters of (0, W, b). B is the current mini-batch, æ being a training example associated with its ground-truth label E (1.2..,,)) By minimizing this loss function w.r.t. O the model learns to assign maximal score to the right class.
Throughout this paper, q and ((x) refer generally to the same object. Writing g(x) highlights its dependence By analogy, depending the context, we may write 2(g) in place of 2. We also denote the whole softmax model by Me, composed of fo and the sofumax layer. This notation will be used in Section 5.2 for our theoretical insights.
3.2 Decision boundaries and the drawback of softmax loss
As presented in the last section, feature vector q F induces logit vector 2(g) Wq+be RC, Consider the pair of classes {i,}}, we have:
=(()) 2z(1)= (Wq + 0); (Wq+b); (W, Wg,4) + (b)-
(5)
21(4) : and W
WT
Here, z(g)
Note that for j j= 1..... W2 E Rd. Set
=0(1))
WT
Púj (q F (W: Wg.g) (b. by) 0)
(6)
Notice that pq F: 2y(9)} and {q F 2(g) 2)(4)} are the two half-spaces separated by Piz Hence, the decision boundary for the pair {i.}} is the hyperplane Pij. and for E Pij, the scores assigned to the classes and j are the same (=+(g) = 2)()). Using Eq. (3, for an input of class i, we see that the softmax loss pushes 2 to be larger than all other 2y's (j # i), Le., (W,-Wy W,. WW,-W,ss the softmax loss enforces the features to be in the right side w.r.t. decision hyper-planes. Notice that the softmax loss has contraction effect. In fact, inputs from the same class tend to produce probability vectors that are close to each other and lie near extremal point of C-1 the simplex of probability measures
+ (b, - 6,) > 0. Hence,

---

Published in Transactions on Machine Learning Research (12/2025)
in R R However, this does not guarantee that features of the same class form compact cluster in feature space. To make this precise, introduce the following definition.
Definition 3.1 (Class dispersion) We define the dispersion of a given class the marimal Euclidean distance (in feature space F) between two samples belonging to that class:
dispersion(c) == max (1((2)) (((2))
where CO = fx: y) E D, y c and D denotes the training dataset.
A key reason is the translation invariance of the softmax function. Specifically, for any e E R, we have (z) = +2 +1), where the all-ones vector. Thus, for a reference feature q E J with logits z = Wq+b. the of features that yield the same softmax output is
S(g) = ( u Wo= EER)
This implics that even (=((1)) (z((2))), the corresponding features qi and Ø can be arbitrarily far apart (unless the classifier matrix W has very specific structure). Hence, although two inputs æ1 and æ2 from the same class may produce probability vectors close to each other (and close to extremal point of AC-1), there is guarantee that g(x1) and ((x2) are close in feature space. This is because entire subspaces of features can map to the same probability vector (illustrated in Fig. 4.. This suggests the need of an explicit constraint that enforces such compactness (i.e., small class dispersion for all the classes).
8(q)
A C-
q
Figure 4: Motivating remark. Ideally, inputs from the same class produce probability vectors close to each other and near corner of C-11 However, for given feature F, there exists an entire subspace mapping to the same probability vector. Consequently, using the softmax loss alone does not enforce small class dispersion.
4 Proposed method
As presented in the introduction, our method includes two complementary objectives: boosting the discrim- inativeness of features for clean data (Fq. 77) and aligning noisy data features with clean data clusters (Eq. 1. This will be introduced in Sections 4.1 and 4.2 subscquently. Note that our method, these proposed constraints are applied alongside the standard softmax loss on clean data, as this helps produce more separable features (as discussed in Section 3). These objectives are optimized with two simultaneous forward passes corresponding to clean and noisy inputs. Although the proposed constraints general and applicable to arbitrary noise distributions, in this work restrict our focus to data perturbed with Gaussian noise during training.
Formally, C- ((p...... Sene.....ol pc) R
De Pi= =1, 20%).

---

Published in Transactions on Machine Learning Research (12/2025)
4.1 Proposed loss function on clean data
To have beller classification, we rely on the following two factors: intra-class compactness and inter- class separability. To obtain these properties, we work in the feature space F. In many classification problems, the intra-class variance is very large. So, by forcing the model to map various samples of the same class in a compact representation, the model learns the representative features each class and ignores irrelevant details. Moreover, it may happen that samples in different classes are very similar. This leads to misclassification. Hence, we also aim to learn representation having large margins the decision boundaries between classes. To achieve all these objectives, we propose the following loss function on uncorrupted (clean) data:
Lelean Leompact + 3 Emargin + "rreg" freg
(7)
Here, Ceompact; Luarrin and freg enforce the constraints for class compactness, inter-class separability and regularization, respectively. We now discuss in detail these three terms. Let us consider the current mini- batch B - D, where D is the given training dataset. Let Cв be the set of classes present in B, i.e., Cs = (y E (1,2.... C): (z. v) B). Let eº be the examples of class e in B, i.e., C. = (x (I. y) E B. = c).
To ensure intra-class compactness, we use the discriminative loss proposed in De Brabandere et al. (2017). Note that in the latter article, this loss used in the different context of image segmentation. This loss is written
Ceompact (0:
Σ ZECT
11 - g(2))l -
(8)
|CB
C
ecCs
where (xx) fo( (recall from Section 3.1 we work in the feature space F) and mc is the centroid of the class c. We will discuss how to compute these centroids in Section 4.4 Here, II-II is the Euclidean norm, [4]+ = max(0, 9). This function is zero when ||m, - all < 6g. Hence, this function enforces that the distance of each point to its centroid is smaller than 6, (here, the subscript v in 8, denotes the variance of the class). Notice that this function only pushes the distance to be smaller than 8, and not to be zero. Hence, we avoid the phenomenon of feature collapse.
To have a better inter-class separability, we build loss function enforcing large margin between classes and the decision boundaries. A naive strategy would be to maximize distance of each sample to all the decision boundaries. However, this is very costly and not really necessary. Instead, we propose to maximize the distance of each centroids to the decision boundarics. Indeed, we will give in Proposition 5.2 a lower bound for the class margins. The margin loss is defined as follows,
marrgin (a;argin(::B)
Σ cECs
max (ōa d(me; Pai) sign(z(m.)) - 2(ma)c)]+ isc
(9)
|CBl
where we recall that z mm,) Wm,, + DE RC and Pas is defined in Eq.66) This function is inspired by 2 Intuitively, when the centroid m. is on the right side of the decision boundary, z(me);; - z(mo)e < O. Hence, in this case we minimize Iōa - d(me, Pos)[+ and consequently d(me: Pa)) encouraged to be larger than 5s (here, the subscript in 80 denotes the distance from the centroid to the boundaries). In contrast, if m. is the wrong side of the decision boundary, then we minimize [8 + ddme, P.:)++. This enforces m. to pass to the right side. Hence, this loss is only deactivated if the centroid is on the right side w.r.t all the decision boundaries and its distance to the decision boundaries are larger than 6g. Moreover, notice that we opt for the aggregation operation max instead of mean
the work of Elsayed et al. (2018)
Note that the work of [Ebayed boundaries and does not impose constraints intra-class compactness. As result, their method maximizes the distance of each individual sample the decision boundarics. In contrast, our approach maximizes the distance betweeu class centroids and the decision boundaries, which significantly reduces computational cost. This because have already the intra-class compactness, which ensures that individual samples remain close to their corresponding class centroids.
2018 focuses on maximizing an approximated Euclidean margin to the decision
8

---

Published in Transactions on Machine Learning Research (12/2025)
Indeed, it may happen that some pairs of class are casier to separate than others. With meon aggregation, loss can be minimized by focusing only on easy pairs and ignoring difficult pairs. In contrast, with aggregation max, we enforce the neural networks to focus on difficult pairs. As such, it can learn more useful features to increase discriminative power. Notice that the distance of m. to the hyperplane Pes: the decision boundary of class pair (c, i), can be computed explicitly as,
d(me, Pa) KW. W1, me) + (be b.)l
(10)
IW. Will
Our loss function encourages each centroids to be far away from the decision boundaries. However, there are no guarantee that the decision boundaries lead to closed cells 3 The resulted centroids could be pushed far away. Hence, to address this problem, we add regularization term as proposed in De Brabandere et al. 22017,,
Lreg (0:
Σ
|mell
(11)
|CB
CECB
Note that this regularization term is optional, meaning that we can set "reg 0.
We also note that we use the squared loss for Lcompact, while a non-squared version is used for Lmargin- The rationale behind this design choice provided in Appendix A.1
4.2 Aligning noisy data with clean data
In the section above, we have introduced the constraint to boost the discriminativeness of the features on clean data. Now, we need to introduce an additional constraint to ensure that the model produces more stable features under input perturbations. For this, let noised(x) denote the noisy version of (after being injected with noise). We then introduce the following loss function,
Luoisy (hoolyy(9::0) ();
x
Σ
[llme [(x)] - -
(12)
|CBl
C
cEC
ZECE
where q is the corresponding feature of the perturbed input noised i.e., 7(x) = f (nniseee((z)
We notice that the final constraint similar to the intra-class compactness loss applied to the clean data. This constraint encourages the features of noisy samples to lie on the same hypersphere as those of clean samples from the same class. Unlike standard methods that align each noisy sample directly with its clean counterpart, approach performs feature alignment only at the class level. This more flexible design helps maintain the expressiveness of the learned features.
4.3
Our loss
With all the components presented above, we come up with the total constraint as follows
Lours Lelean + >L
(13)
where we fix for the sake of simplicity.
4.4 Computing class centroid: partial momentum strategy
In our loss presented above, the class centroids appear in different constraints (Eqs. We now discuss how to compute and update such class centroids during training.
(
99
11 and 12
Let us now consider a
By closed cell, all others. In the idcal case, these boundarics enclose finite region correspouding to the domain of that class. practice, however, depending on the parameter configuration of the softmax classifier, the decision boundaries may fail form fully closed region.
the region in feature space bounded by the decision boundaries separating given cluss from
9

---

Published in Transactions on Machine Learning Research (12/2025)
class To compute the centroid of this class, use only the clean data. This is because during training, the features of corrupted data can be very far away from those of clean data, or even fall in the wrong class. Consequently, this can make the centroid updated in the wrong direction. There are 2 straightforward ways:
Naive way. Using all the sample of the considered class in the current mini-batchi m := m. [eड Exece (xx)
murrent (θ; (0;B)=
Using momentum. m. m momentum = -7-m. 1p.mmp11p (1 -7) i mourentt where y is chosen to be close to 1, such as 0.9. Note that m from the last batch is used as fixed quantity here (no gradient).
One major advantage of using momentum is stability. Recall that in modern machine learning, very small mini-batches are now common to achieve speed-ups at the cost of noisy gradient steps. Thus it can happen that the centroid of each class fluctuates too much from one batch another. In such case, we do not have stable direction to that centroid. As Leompact aims push each point to its corresponding centroid, the optimization becomes less effective. Hence, the use of momentun allows to avoid this problem. However, using momentun makes the gradient much smaller when updating the model parameters. More precisely, we have following proposition:
Proposition 4.1 V margin = (1 71 V Velnargn Here, f and Emoment are the margin losses computed using the centroids updated based on naive way and momentum way, respectively.
Proof 4.1 See Appendix C
This proposition shows that using centroid with or without momentuum gives the same gradient direction. Nevertheless, with momentum the very small shrinking scaling factor 1-7 appears.
Further, this small gradient is multiplied by small learning rate (typically in the range [10-5, 10,0,,105 100-1)) So, the one hand, the parameter updating in the momentum method is extremely small (or even get completely canceled out by the computer rounding limit or machine epsilon). On the other hand, as discussed previously, using momentum allows more stability. To overcome the gradient drawback but to conserve the stability benefit, we combine the naive and momentum ways. We come up with a strategy named partial momentum. This strategy uses momentum for the compactness loss and naive way for the margin loss, respectively. Doing so, we have stable centroids. So that, each point pushed in a stable direction. But at the same time, the centroids are kept consistent (consistent here means that centroids receive sufficiently large gradients to be back-propagated to the set of parameters 0 of fe, preventing them from becoming outdated during training).
5
Some properties of our loss function
5.1 Properties of intra-class compactness and inter-class separability
In this section, we investigate the properties of compactness and separability of the loss function. Fur- thermore, we discuss the impact of the syper-parameters 6, in Leompact and da Emargin This gives us guideline on the choice of these ayper-parameters. We recall that D denotes the training dataset and Cº {x: (x. y) E D, y c).
Proposition 5.1 Tf Ccompact (0: (O:D) then the dispersion of all classes (see Defimition 3.1 is at most 28..
Proof 5.1 See Appendir B.1
This last proposition shows that the hinge center loss ensures the antra-compactnesss property of each class. Definition 5.1 (Class margin) Let us define the margin of given class as the smallest Euclidean distance of samples in this class to its closest decision boundary, i.c.
margin(c) == min
min ((in))) Pat)
#EC
iff
10

---

Published in Transactions on Machine Learning Research (12/2025)
Proposition 5.2 Assume that Crompact (O; (a: =0. Then,
1. 1f δa > 6,, then the margin of all classes is at least 8ạ - -6.
2. If 8a 26.; then in the feature space F, the distances between any points in the same class are smaller than the distances between any points from different classes.
Proof 5.2 See Appendiz B.
Hence, if we aim to obtain class margin at least E, then we can set 6y de t. Furthermore, this proposition provides guideline for the choice of 6, and 6d- We are aiming for representation with not only a large inter-class margin, but also one in which the distances between points in the same class are smaller than the distances between points in different. classes. This is particularly useful for problems where samples in each class too diverse whereas samples from different classes are too similar.
5.2 Intra-class compactness or inter-class separability constraint alone does suffice
In order to boost class compactness, one can minimize the distance of each point to the corresponding class centroid. As presented in Section 2 this approach is proposed in Wen et al (2016) using a center loss. Boosting the class margin is proposed by Elsaaed et al. (2018). However, we shall prove that applying only one of these two approaches, the model can be encouraged to evolve in a direction that does not change the model prediction.
Indeed, they can lead to solutions that allow for enforcing class compactness or class margin but where the predictions of the model remain unchanged and so does the generalization capability. In such cases, the model is not encouraged to learn more discriminative featurcs. This also suggosts that the use of both compactness and margin is necessary. More formally, let us recall that N (with the parameters θ) denotes the softmax model, as defined in Section 3.1 We suppose that the layer prior to the feature space F is a standard image convolutional layer or fully-connected one, that can be followed or not followed by the non-lineate loneriiyar ReLU. Suppose that the current parameters of N is θ (denoted by Ne).
Proposition 5.3 For v 0, there erists map, depending on v, Te, such that θ = (e) satisfies Ne(x)) (z(z) for all x E X. Furthermore,
1. If No is such that min, margin(c) dimariin 0 (see Definition 5.1 then, for the new model No. mine margin(c) vdlmargin-
If Ne is such that < max, dispersion(c) ddispersion (see Definition 3.1 then, for the new model Ng. max, dispersion vddispersion-
Tx(0) - θ = (v-1)u where u is a vector depending only on e
Proof 5.3 See Appendir D
Corollary 5.1 V V1, > (9) and (0) are vectors of opposite directions. In particular, as Vy or V2 increase, the movement of paromelers happens in opposile directions (illustrated in Fig. 5 That is, using this family of transformations, the class margin expansion and the class dispersion shrinkage cannot happen at the same time.
Suppose that we are given a network Ne with certain class margin and dispersion and that was trained with softmax loss for example. We note that the statements (1) and (2) in Proposition 5.3 hold for all v > 0. This indicates that, by choosing v or 1, we can explicitly perform transformation to obtain new parameters of A to increase the class margin or decrease the class dispersion to satisfy the constraints imposed by either center loss margin loss alone, respectively- without changing the predictions of N.
11

---

Published in Transactions on Machine Learning Research (12/2025)
Direction for increasing both class dispersion and class margin
J (0)
Current parameters
Direction for decreasing both class dispersion and class margin
V2,,22 > 1, (0) (0) O and T1/v; (9) (0) O are vectors of opposite directions.
T1/v2 (0)
Figure 5: For any V1,22 > 1, the vectors Tz, - and (0) - point in opposite directions. Moving along cither direction preserves model prodictions but increases intra-class dispersion or decrcases the inter- class margin, thereby violating the joint optimization enforced by our constraints. Therefore, our loss avoids such solutions Tr, which do not improve generalization.
That is, the model is not really encouragod to learn more discriminative features and the generalization of the model is not improved. However, Corollary 5.1 tells that we cannot both expand the class margin and shrink the class dispersion simultaneously. This is because θ is moved in two exactly opposite directions for these two objectives. In other words, moving along either of the two directions defined by T, would increase the intra-class dispersion or decrease the inter-class margin, thereby violating the joint optimization enforced by our constraints. Consequently, our loss avoids such solutions Te, which do not lead improved generalization.
6 Theoretical insights on our method
In this section, we present theoretical insights into our method. Section 6.1 we analyze how our approach affects the curvature of the softmax loss the input space. Since our method also enforces that the inputs are projected onto the hypersphere corresponding to their class, Section 6.2 investigates the generalization behavior of this constraint, based on how well it is empirically satisfied on the training set. Lastly, method is applied only al the penultimate layer (i.e., in the feature space F), we analyze in Section 6.3 its potential impact on intermediate layers.
6.1 Impact of our method on the curvature of the softmax loss in the input space
When analyzing model stability with respect to input perturbations, a common approach is to examine the curvalure of a standard loss function in the input space. Typical loss functions include the mean squared error for regression and the softmax loss for multiclass classification. Following this line of reasoning, in this section we analyze how our method effectively reduces the curvature of the softmax loss in the input space (that we called input loss curvature). Conversely, we also demonstrate that reduction in this curvature leads to improved model stability under Gaussian input perturbation. Note that prior works (e.g.. Moosavi-
12

---

Published in Transactions on Machine Learning Research (12/2025)
Dezfooli al. (2019) have also shown experimentally that reducing the input loss curvature leads to models that are more robust to input perturbations.
Background: input loss curvature and its relation with the Hessian matrix. We first recall some background for studying the curvature a loss function at an input point x E R via its Hessian matrix (assume that the input domain X' _ IR*) Consider the softmax loss function (Eq. (4):
(xx) (z, log (W fo(x) 6)
Notice that, we aim at studying the curvature with respect to the input so we drop in the annotation for brovity, and we write ((x). Assume that I is twice continuously differentiable (with respect to z). The Hessian matrix H(x) is defined as the matrix the second-order partial derivatives:
(a)()) dxf
(())
= VZ((z)
((
Bag 8m1
axe
This can be interpreted as the Jacobian the gradient vector: H(x) da VI(x). If we move a direction U. R& we consider the path s()) = x tv. We notice that
Vi(x + tw) vi(x) Vl(I + tv) lim
H(x)v.
dt
1==0
Hence, if v is unit vector ([|=l = 1), (x)eell describes the change in the gradient per unit length (or gradient change rate) at the point x, as we move direction v. This can be regarded as the curvature along the direction v. Consider an eigenvector v of ([]=] = 1), with corresponding eigenvalue X, then we have (x)x|] = | [사]. Therefore, the set absolute eigenvalue magnitudes ([>I) of Hxx) provides representation of the geometric (unsigned) curvature of point æ. For instance, way to quantify the overall curvature at is computing E, X2 [Al-
Theoretical insights in the impact of our method on the softmax loss curvature. Equipped with the background above, we now introduce our theoretical results. For a perturbation around x E X, we assume the following approximation
1(x +e) 1(x)+ vi(x) (())) + fe (()ss
(14)
Consider an input x from class y, with feature representation q(x) fo(x) in the fcature space F. Wo use fo(x) instead of q(x:) emphasize its dependence the model. For ô > O, let C(Je(v), 6) denote the hypersphere of radius 8 centered at fo().. Under the compactness loss Lcompact (Fq. (10 for clean inputs and the alignment loss Lalign (Eq. (12)) for noisy inputs, well-trained model ensures that both clean and
noisy features of the same class lie within hypersphere of radius 6, around the class centroid. Hence, their distance at most 26,- Therefore, under a Gaussian perturbation E, we expect
fo(x + E ((fo(x)))) with 25,;
with high probability. This observation is the key link between input loss curvature and model stability under Gaussian perturbations. More formally, we define stability under Gaussian perturbations as follows.
Definition 6.1 (Stability under Gaussian perturbations) Consider Gaussian noise ~ o1) For a given 8 > 0, let n = Pr (fo(x + E C(fo(z). lout Eg [(x + +) fo(a + E) & C(fo(x),
Feature stability. We say that So produces stable features w.r.l. (at x)) is large for small 8.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Loss stability. We say that fo produces stable loss w.r.t. (at x)) if Vouu - [xx) is small. That is, even when features fall outside C(fo(r),0), the resulling loss does not deviate much from the clean loss Ux)) (in expectation).
In the extreme case of feature stability, where 17 = for = O, fo produces features that are almost surely invariant under the noise E. In this case, we will show that the input loss curvature is zero. Conversely, we will also show that a low input loss curvature encourages least feature stability or loss stability.
Assuming that the logit values are bounded within [- Kmax] for all imputs, the following theorem formalizes the connection between input loss curvature and model stability under additive Gaussian noise.
Thcorem 6.1 (Upper-bound and lower-bound of the imput loss curvature) Assume that follows the distribution (O, I²I), and let n = P. (fo( +) E C(f(x)e. 8)) (we assume that 1j > 0). Then,
1. Denoling the set of eigenvalues of H(x) by (Mi) and || = max, [ we have
Σ x 04 1 8² + +4(1 - 1) .K) K2ax)
(15)
2. Let lout = Ee [1= + +E) fo(x +E) $ C(fo(x), we have
2 σo IAl))
4 (1 1) (lout - -(x)) -
2
Σ x +
(16)
Proof 6.1 See Appendix
b
Remark 6.1 Theorem 6.1 contains two statements that address complementory regimes. Stotement (1) considers a setting in which feature stability holds, and we aim to show that improved feature stability leads to smaller impul loss curvature. In contrast, Statement (2) focuses on another regime where feature or loss stability has not yet been achieved (i.e., when (1 (1-1) 17) (lout [(x))² is sufficiently large). We aim to understand what happens if we con somehow reduce the input loss curvature. in this regime.
Statement (1). In Eq. 15 the left-hand side (LHS) Σi represents the input loss curvature. In the right- hand side (RHS), if 7 increases to then the second term decreases to An obvious way increase y is by increasing 8 (there is higher chance that fo( e) C(fo(z), 5)). However, by doing so, also increase the first term of the RHS (as it depends
8). Therefore, one way to mininize the upper bound on the curvature is to train the model such that fo(x +)) C(fo(r),) with high probability n,, for a small 6. This also suggests that choosing 8 close to zero is not a good choice. Indeed, in this case it is difficult to train a model such that 1j is large. With our intra-compactness constraint, we push the features of the clean and noisy inputs (Jo(x) and fo(x +)) to lie within the hypersphere C(my: 5,) of the same class. Hence, if the model well trained and taking 28,, with high probability we have fo(x + E) C(fa(z).)), So, this effectively helps to decrease the input loss curvature. This will be demonstrated in experiments (sce Section 7.4
Statement (2). As mentioned in Remark 6.1 we focus on the regime in which feature (or loss) stability has not been achieved, namely when (1 - v) (Tous I(x))² is sufficiently large. Under this regime, consider Eq. 166. If the gradient of the loss at x is sufficiently small and noting that a is typically small, we have [I() 2 Consequently, this term can be neglected compared to (1 7) (lout - 1(x))2 In this case,
4(1 4(1-n) 2
2
>
x Σ
a (lout 1(x))2
This inequality suggests that, for fixed 8, reducing the input loss curvature (i.c., the Hessian eigenvalues) either increases 17 or encourages smaller value of Bont [(x)]. This indicates that reducing the curvature
14

---

Published in Transactions on Machine Learning Research (12/2025)
of the loss with respect to the input encourages at least feature stability loss stability. Besides, recall that our loss function includes a constraint that explicitly increases the margin between clean sample features and the classification decision boundaries. As a result, when noisy features remain close to their clean counterparts, the classifier more likely to make consistent predictions under perturbations. This highlights the complementary role of the margin-based constraint. promoting classífication stability.
As concluding remark, this last theorem reveals strong connection between the input loss curvature and the model stability.
6.2 Generalization behavior of the compactness constraint for mapping features onto hypersphere
Recall that our method enforces imputs to be projected onto the hypersphere associated with their class via the intra-class compactness constraint Lcompact in Eq. (8) In this section, we fix particular class and focus on the compactness constraint. We assume that Compacc vanishes on the training set; that is, for all training samples z, we have [z) ml| S dg. We now examine whether compactness is preserved on the test data. More concretely, for given r > 0, let C(m,r) denote the hypersphere centered at m with radius r. We then study the probability that a test point æ is projected inside C(m, r). In short, we investigate how well the compactness observed on the training set generalizes to the test set.
To this end, we first recall the notion of margin loss introduced in Mohri 2018
Definition 6.2 (Margin loss function) For any p > O, the p-margin loss function is defined as
1
if S 0, if S SS
(T( = 1 T-
(17)
0
if pp.
Now, for given >>> 0 and 0 x < x2, define :x):=;² 11(()) - ml| Then,
if [[f((x) ml| 2 r, - (r2 1-5011 - (om))m//3)/ if - S [1(()) mm| Sr. (p(x)z))
(18)
il [f) mm| S vr² p p.
Remark 6.2 When p 0, the function in Eq. 18 penalizes inputs x that are projected outside C(m, r) (i.e., [I((x) mm]] 2 r). This allows us to quantify the projection error. For larger p, the function also penalizes points lying inside C(m,r) but within a margin p of the boundary (i.e., v/2 - S (1) - mml| S r). Thus, the parameter p can be interpreted as a confidence margin.
In our method, by enforcing the model to satisfy the compactness constraint on the training set, we expect the compactness property to generalize to the test set. Is this a reasonable objective? To address this question, we quantify the mapping error, i.e. the probability that point is projected outside C(m,r) for a given 0.
Recall that F denotes the feature space, and let M(X,F) be the set of all measurable functions from X to F. For parameters R r, A > 0, we consider the following function class:
H H= { M()) , (15() ml -Im]| s R, E M(2,F), sup [[(())) S A
sEx
This class allows both the feature mapping ff and the hypersphere center to vary, subject to boundedness assumptions. Such assumptions are natural in light of the regularization term in Eq. 11 The next theorem provides an empirical generalization bound on the mapping error. Let S denote a sample consisting of N > 0 i.i.d. copies of X.
"The boundeducss parameters essentially refer to the class centroids and the outputs of f. Specifically, the regularization term in Eq. [1] encourages the learned features remain within bounded range. Intuitively, the feature space were unbounded. generalization would he severely impaired, features could accupy arhitrarily large regions of the space.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Theorem 6.2 For any 8 >> O, with probability at least 1 5 over the draw of an i.i.d. sample S of size N, the following holds for all h E H:
2 (A3
R²
log 3
P(((X) 0) - R' ( A²
2RA
31
(19)
VN
2N
Here, = (p(h(z;)) denoles the empirical error on S.
Proof 6.2 See Appendia F.1
Remarks. If an input is mapped inside the hypersphere in the feature space, then h(æ) 0. Thus, P(((X) < 0) measures the mapping error, ie., the probability that inputs are mapped outside the hyper- sphere.
From Eq. 18 and Remark 6.2 we note that (1) penalizes training examples æ such that - - [I ml]- In particular, 6, S V12-p, p. then (h) = Besides, observe that the upper bound in Eq. 19 decreases as the number of training examples N increases. This natural, since a larger training set provides better coverage of the underlying input distribution. Consequently, if the model is well trained (i.e., the empirical loss on the training set is small), it is more likely to generalize well to unseen data.
6.3 Impact on the features of intermediate layers
Our loss is applied only to the features at the penultimate layer. This naturally raises important ques- tion: does it also inffuence the representations learned in the earlier, intermediate layers? In this section, we provide theoretical insights into this question. In short, Theorem 6.3 shows that enforcing inter-class separability at the penultimate layer allows us to derive a lower bound on the inter-class separability of the Teatures in intermediate layers.
Let X E X and Y y denote the random variables (I.v.), modeling respectively the input and the label. Let Dx and Dy be the distributions of X and Y, respectively. Assume that fo is composed of L layers. For given layer / E 21, LJ, let Q' denote the feature r.v. of this layer. That is, the distribution Do of Q' is an induced distribution of Dx. Given E V, let Do be the induced distribution from the conditional input distribution D(X = v).
We recall that our loss operates at the outputs of fa; i.e., the feature space of layer This aims to enforce features of different classes to be contained in different hyper-spheres. Hence, in some sense, pushes away Po from Do for any (,,v') E y2 (y v'). In this way, the features of different classes can be more separable.
Now, let us consider an arbitrary intermediate layer 1 < L before the penultimate layer. The question is: For all (y,v') E y² (y + y'), does our loss Leleon make Darl, more separated from Do ? To measure the divergence between two distributions D1 and D2, we use the Jensen--nneeeeeeeeeeeee Shannon divergence (JSD), denoted by Das (Di || D2). For completeness, we refer the reader to Appendix G for formal definitions.
Now, let us consider a trained model with our method. Recall that the compactness loss encourages the features in F. of each class y to be contained in hyper-sphere of radius 8, and centered at my denoted by C(m,, 6.). Thus, it is natural to make the following margin assumption
(H) There exists a (1/2 < 1) such that for all E V/ we have
D Do (C 8.)) == PQ E C(m, 6.)l% =) 2T
Under this assumption, the probability that each point is projected in the right hyper-sphere is al least 72 1/2. This is a reasonable assumption if the model is well trained. Under the assumption (H), we have the following theorem.
Thcorem 6.3 Let y, yf E y such that y # v'. For any intermediate layer S before the penultimate layer, we have
Das I1 Dow)) 2 (2r 1)2/2.
(20)
16

---

Published in Transactions on Machine Learning Research (12/2025)
Proof 6.3 See Appendir G
Remark 6.3 Note that our goal is to derive a model-agnostic lower bound that holds universally, without rebying on architectural details. Nevertheless, under additional assumptions on the layer type or network struclure, il may be possible to oblain sharper, layer-dependenl bounds.
This theorem gives a lower bound for the JSD between the conditional feature distributions of intermediate layers. the model well trained, T gets larger, so does Dus ()s(P || Doyg:) That is, the intermediate layers, features of different classes become more separated from each other. The intuition behind this rather simple. Indeed, let us argue using contradiction reasoning in an informal manner. Suppose for example that Dofly is very similar to Dotly' and that they share large part of their supports. Then, the features of this layer are propagated until the feature space F by the same mapping which is the set of layers bet ween layer I and the penultimate layer. Hence, the two conditional distributions Dotty and Dotly' in F should be also similar. This contradicts the condition imposed by our loss, where features of different classes should be contained in distinct hyper-spheres. Hence, Da should be distinct from Datly' so that these constraints on the penultimate layer are satisfied. Theorem 6.3 formalizes this idea. Indeed, this matches the results
observed in our experiments (see Section 7.5
7 Experiments
In this section, we conduct experiments to evaluate the robustness our method in comparison with baseline approaches (Section 7.2 In addition, we aim to verify whether our method can preserve model performance on clean data (relative to training without noise injection), while simultaneously improving robustness against input perturbations (Section 7.2 We further investigate in Section 7.3 the relationship between robustness
and the curvature of the input loss surface, providing empirical evidence that supports our theorelical result in Theorem 6.. Moreover, we demonstrate that our method effectively reduces input loss curvature (Section 7.4. Finally, we present qualitative results showing that, when our method is applied at the penultimate layer, the intermediate feature representations also become more discriminative (Section 7.5 thereby supporting the theoretical findings Theorem 6.3
7.1 General experimental details
Here, we provide some general experimental details; all additional information can be found in Appendix H In our experiments, we use 3 different datasets.
CIFAR-10 dataset. CIFAR-10 (Krizhevsky et al. 2009 consists of 50,000 training images and 10,000 test. images across 10 classes, including both vehicles and animals. These are 32 x 32 color images.
Street View House Numbers (SVHN) dataset. SVHN (Netzrr el al. 2011 contains 73,257 training images and 26,032 test images of house numbers captured from Google Street View. These are also 32 x 32 RGB images.
Road Condition Image Dataset. We construct a custom dataset tailored to our task, comprising 1,897 road images collected from both publicly available sources and our own road image recording campaign. All the images are resized to 400 x 500 x 3. The dataset is divided into three classes: dry (460 images), wet/un /wattereeedd (460 images), and icy/ has/sowy (977 images). To ensure diversity, we deliberately seloct images from wide range of scenes within each class. Some examples from our dataset are depicted in Fig.
Neural network models. For the two datasets CIFARIO and SVHN, we use ResNet18 (H el al. 2016 as backbone, followed by a fully connected layers prior to softmax layer 5 For our own dataset, we test on the model MobileNetV3 (Howard al.) 2019), as this model is sufficiently lightweight for real-time application on vehicles. Note that the entire model is trained end-to-end, with no layers frozen.
More precisely, the backbone is followed by fully connected layer with output dimension 128 preceding the softmax layer, which has 10 output classes. See Appendix H for details.
17

---

Published in Transactions on Machine Learning Research (12/2025)
dry
snowylicy
wet/watered
Figure 6: Some examples from our custom road image dataset, including 3 categories.
Noisy test accuracy. For each type of input perturbation, we perform 10 independent runs and report the average and standard deviation (std) of the test accuracy under noisc.
Baseline methods. We compare our approach with the following baseline methods.
Normal training. We train models with the standard softmax loss clean training data (with standard data augmentation, see Appendix H.
Training only on noisy data. We apply Gaussian noise (on top of data augmentation) to clean data for training model with the standard softmax loss
Training on both clean and noisy data. Besides noisy data, we also keep the clean data. Then the model trained both clean and noisy data with the standard softmax loss.
Stability training. This method was first proposed in Zheng et al. (0066). It introduces a stability regularization term in addition to the standard softmax loss on the clean data. The stability regularization encourages the feature representations of clean and noisy versions of the same input be close, by minimizing their distance. For fair comparison, we apply this constraint the same feature space F used in method, our experiments.
minimize []fo(z) fo(x + +)]l during training, where is Gaussian noise in
Note that in our method, the proposed constraints applied alongside the standard softmax loss clean data, as this helps produce more separable features (as discussed in Section 3). For simplicity, we fix a = 3 1 and Tre = 10-3 for all the experiments.
Calculation of the input loss curvature. We notice that the curvature of the softmax loss function at a point I can be calculated as Σ, X² = E, [=''(=))=]] = [=)))=||?, where E ~ I). Moreover, for any s, by definition of the Hessian, H(r) de
+ + lim,-o Vi(a+te) Hence, we can estimate the input loss curvature without calculating the second derivatives w.r.t. inputs follows:
A(x) = Σ
Vi(x + ts,) v((x))|?
(21)
K
where Ej are i.i.d. samples from (0, I). In our experiments, we fix - and K 20. Preliminary results show that K 20 alrcady provides stable estimates, with small variance across different approximations.
7.2 Model performance under input perturbations
Experiments on standard datascts CIFAR-10 and SVHN. For CIFAR-10, we inject additive Gaussian noise with std 0.06 during training for all methods. After training, the models evaluated under varying
18

---

Published in Transactions on Machine Learning Research (12/2025)
levels of additive Gaussian noise, shown in Table 1 higher noise level of std = 0.15 during training, since SVHN appears more robust to small perturbations compared to CIFAR-10. The results for CIFAR-10 and SVHN are reported in Tables and 2 respectively.
We follow the same procedure for SVHN, but use
Table 1: Accuracy on noisy test set at different Gaussian noise level for CIFARIO. Note that for training, we set the standard deviation (std) of the Gaussian noise to 0.06.
Additive Gaussian noise level (sid)
Clean
2/255
4/255
8/255
16/255
20/255
Normal training
92.73
92.63
0.09
91.95
0.09
87.73 1 0.19
64.93
0.18
50.32 :: 0.29
Training only on noisy images
89.64
89.65 1 0.06
89.70
1 0.07
89.99 1 0.08
89.81
1 0.14
87.918
1 0.16
Training on both clean and nuisy imnages
91.48
91.45 1 0.04
91.41 1 0.07
91.15 1 0.11
90.06
1 0.14
89.04
1 0.21
Stability training
91.13
91.16 1 0.04
91.24
0.09
91.26
1 0.08
89.12
0.15
84.62
0.21
Ours
92.67
92.70
0.05
92.68
0.07
92.61 22.61.1
0.07
91.15
0.20
89.42
0.11
Table 2: Accuracy on noisy test set at different Gaussian noise level for SVHN. Note that for training, we set the standard deviation (std) of the Gaussian noise to 0.15.
Additive Gaussian noise level (std)
Clean
5/255
10/255
20/255
36/255
42/255
Normal training
95.69
95.41 1 0.06
94.32 + 0.08
86.85 - 0.13
66.36 + 0:21
58.87 1 0.16
Training only on noisy images
93.42
93.39 1 0.03
93.37 1 0.04
93.02 = 0.07
91.64 = 0.15
90.39 1 0.11
Training on both clean and noisy images
95.34
95.25
0.02
94.96 0.07
93.94 = 0.07
91.62 = 0.09
90.43 1 0.08
Stability training
96.01
95.93
0.03
95.63 = 0.07
94.50 = 0.06
91.45 0.14
89.71 + 0.07
Ours
96.13
96.06 = 0.03
95.91 = 0.05
95.17 1 0.07
92.47 1 0.09
90.86 = 0.08
From Tables 1 and 2 we observe that training solely on noisy data, or on both clean and noisy data, tends to degrade performance on uncorrupted (clean) mildly corrupted test samples (i.e., perturbations smaller than 4/255 for CIFAR10 and 10/255 for SVHN). In contrast, when the noise level is sufficiently high (i.e., 2 8/255 for CIFAR10 and 2 20/255 for SVHN), training on noisy data can actually outperform standard training. This suggests that noisy training helps the model fit to the noisy distribution, but does not necessarily promote learning of well-generalized features that perform well on clean inputs.
On CIFAR-10, stability training notably reduces dlean-data accuracy, whereas our method maintains perfor- mance comparable to normal training. On SVHN, our method even improves accuracy on clean data. With small perturbations (2/255 for CIFAR-10 and 5/255 for SVHN), the performance gain over normal training is limited. However, the noise level increases, our method consistently oulperforss both normal training and alternative approaches. Specifically, it yields improvements of up to nearly 30% over the normal training bascline at the highest noise level on both datasets. Furthermore, our method systematically outperforms the stability training counterpart, demonstrating its effectiveness.
The model extrapolates stability beyond the noise level seen during training. For each dataset (Tables 1 and 2 we additionally evaluate the trained models under noise levels higher than those used during training. namely 20/255 for CIFAR-10 and 42/255 for SVHN. At these magnitudes, the injected noise already significantly alters the semantic content of the inputs, making evaluation at substantially higher noise levels less meaningful. Despite this, we observe that the model trained with our method retains strong performance when tested beyond the training noise regime. This behavior indicates that the learned stability properties generalize beyond the specific noise levels encountered during training, providing evidence of enhanced robustness.
Qualitative results. We use the t-SNE technique to visualize the learned feature representations of CIFAR- 10 in two-dimensional space, shown in Fig. [ The visualization reveals that under normal training, features are not well scparated when evaluated on noisy data (right plot of Fig. Lap. When trained on both clean and noisy data (Fig. [b], feature separation improves on noisy inputs, but the structure of clean data becomes less distinct compared to normal training. This trade-off leads to a performance drop on clean inputs. In contrast, our method enhances feature soparability on both clean and noisy data (Fig. Icp. These qualitative observations are consistent with the quantitative results, further demonstrating the effectiveness of our approach.
19

---

Published in Transactions on Machine Learning Research (12/2025)
Experiments our custom road image dataset. We randomly split the dataset into 80% for training and 20% for testing. During training, all methods are exposed to additive Gaussian noise with std = 40/255.
Additional training details are provided in Appendix H Once trained, the models are evaluated under various types of input perturbations to assess their robustness.
We evaluate model robustness under the following perturbations (illustrated Fig.
Additive Gaussian noise. As in the CIFAR-10 and SVHN experiments, we apply varying noise levels (standard deviation), cach evaluated over 10 independent runs to ensure statistical reliability.
Random occlusion. We randomly mask 20 patches of size 70 70 per image. Results are averaged over 10 independent runs.
Downsampling- lownssanpligg-Upaaning (DU sampling). In this method, we first downsample each image by factor of 1/3 in both dimensions, then upsample it by factor of 3 to restore the original size. This operation is deterministic, so report results from single run.
Random Stripe Masking. We randomly mask 10 vertical stripes of thickness equal to
Combination. We also evaluate the combined effects of (i) random occlusion with Gaussian noise, (ii) random occlusion with DU sampling and (iii) DU sampling with random stripe masking.
Results on additive Gaussian noise. The results at different noise levels are reported in Table 3 Our method consistently outperforms the other methods. Notably, it slightly improves performance even on clean data. At the noise level 40/255, our approach yields an improvement of more than 30% compared to the baseline of standard training. At a noise level of 60/255, which exceeds the training noise level (40/255), the model trained with our method retains strong performance. When using For the other methods, we observe the same phenomenon as CIFAR10 and SVHN.
Clean
Down/Up Sampling (ratio=3)
Additive Gaussian (std=40/255)
Patch Occlusion
Stripe Masking
Figure 7: Examples of perturbations applied on the inpuls at inference, including Gaussian noise, random occlusion, down/up up sampling and stripe masking. In occlusion, we randomly mask 20 patches of size 70 x 70. In stripe masking, we randomly mask 10 stripes of thicknoss equal to
Results for other perturbations (Table 4h. We further evaluate the models under additional perturba- lions, including random occlusion, DU sampling and stripe masking, as previously described. For random occlusion, our method significantly outperforms the other methods. Notably, while stability training per- forms well under additive Gaussian noise, it yields poor results under random occlusion. When combining random occlusion with additive Gaussian noise, our method again surpasses the others. On DU sampling, our method is slightly outperformed by stability training. However, when DU sampling is combined with
20

---

Published in Transactions on Machine Learning Research (12/2025)
Table 3: Accuracy on noisy test set at different Gaussian noise level for road data. During training, we set the standard deviation (std) of the Gaussian noise to 40/255.
Additive Gaussian noise level (std) normal training (clean data)
clean
10/255
30/255
40/255
60/255
96.84
95.57
0.28
75.42 + 1.08
59.34
1.28
43.81
1.02
training with noisy data
91.58
92.66 + 0.41
95.71 + 0.24
94.95 + 0.62
92.05 + 0.84
training with noisy and clean data
96.31
96.36 + 0.35
95.97 + 0.58
95.39 + 0.73
92.92 + 0.46
stability training
96.05
96.31 + 0.23
94.63 + 0.58
92.92 = 0.80
90.87 + 0.69
ours
97.37
97.29 0.17
96.71 = 0.49
95.52 = 0.64
93.65 0.49
occlusion, our method clearly demonstrates superior performance over all other methods. This also the case with random stripe masking. These experiments strongly support the conclusion that training with additive Gaussian noise alone with our method can enhance the model's robustness to broader range of perturbations beyond Gaussian noise.
Table 4: Accuracy on noisy test set for road data under input perturbations. The standard deviation (std) of Gaussian noise 40/255. For random occlusion, we randomly mask 20 patches of size 70 70. stripe masking, we randomly mask 10 stripes of thickness equal DU Sampling stands for Downsampling: Upsampling, where we down-sample image resolution by 1/3 and then up-sample by the factor of 3 recover the initial size.
Perturbation type
Normal training
Training only noisy images
Training on both
Stability training
Ours
clean and noisy images
Occlusion
89.89 + 0.49
80.21 + 1.02
87.92 0.68
80.99 + 1.19
94.05 0.75
Occlusion +
27.68 + 0.30
74.00 + 1.23
78.60 1.48
54.07 + 1.32
82.73 + 1.45
Gaussian noise
DU Sampling
92.36
86.58
91.84
96.05
95.79
DU Sampling +
77.58 + 1.09
78.86 + 1.35
81.34 0.71
74.47 + 1.02
88.05 0.55
Occlusion
Stripe masking
84.39 + 1.30
76.21 + 1.23
88.00 + 0.65
86.63 + 1.16
90.42 0.62
DU Sampling
69.52 + 1.62
70. + 1.33
77.78 + 1.06
82.44 + 0.99
87.99 + 1.53
Stripe masking
7.3 Correlation between input loss curvature and model stability under input perturbations
The results of Theorem 6.1 suggest strong correlation between the input loss curvature with the model performance under input perturbations. In this section, perform experiments to observe this curvature- stability relation in practice, where we use uniform and Gaussian noises.
Noisy test accuracy as a function of retained low-curvature samples. Consider the clean test set Drest = {(,,, ))) For each sample 24, we estimate the loss curvature A(z;) using the procedure {A(x:) We then evaluate noisy test accuracy by retaining different proportions of low-curvature samples. Specifically, for [0, 1]. we compute the p-th quantile ile of
Eq. (21 and collect the results into
A, denoted by Quantile(A,p), p). We define the subset
Drest (p) { V1) E Drest A(x;) S Quantilc(A.p))
The average noisy accuracy on this subset is computed as
acc(Prest(P)) = DDoss (p)
1
Σ
1 ((()))))=)))
(#..
where 7(z + Ex) denotes the model prediction under noisy input I; + Ess and we consider both uniform and Gaussian noise.
21

---

Published in Transactions on Machine Learning Research (12/2025)
By varying p. obtain the results in Fig. For both standard training and our proposed method, the noisy test accuracy decreases larger portion of samples is retained. Equivalently, restricting evaluation to fewer (lower-curvatury samples yields higher robustness to noise. This behavior indicates that robustness is strongly correlated with low-curvature inputs, consistent with the theoretical result in Theorem
"est accurory 196 on naloy images
Narmal training- Gaussian nose
Ours Caussian noise
(%)
training uniform noise
Ours uniferm naise
Normal Geussian noise
Ours Gaussian noise
uniffomm
Ours uniform noise
Retained Samples LLowest Rest)
Portion Samples Curvature
(a) CIFAR1O.
(b) SVHN.
Figure 8: Test accuracy on noisy images for different retained portion of test samples with lowest curvature. We conduct experiments using both models trained with our method and normal training. For CIFAR-10, set the standard deviation of the Gaussian noise and the amplitude of the uniform noise to 0.06. For SVHN, we fix both values to 0.15.
Further analysis of the robustness-ani curvature relationship. To further investigate the connection between robustness and input loss curvature, we conduct the following experiment. For each test sample (z,, 91) E Duet: we record how many times the model produces correct prediction over 10 independent random noise perturbations:
10 Σ
count
(()))
where [z; + Ez) denotes the model prediction under noisy imput 2; + Ez-
Based on this count, partition the test set into groups according to the number of correct predictions. Specifically, for each k E {0, l..., 10), we define
G(k) (i count(() k.
We then compute the average input loss curvalure (on clean inputs) for each group:
A(k) = |G(K) iEG(R)
Σ A(r.).
Finally, we plot i versus A(k) analyze how robustness correlates with input loss curvature. The results for both uniform and Gaussian noise on CIFAR-10 and SVHN are shown in Fig. g We observe a strong negative correlation between the number of correct predictions and the average input loss curvature, indicated by high Pearson correlation coelficient. This provides additional evidence that lower input loss curvature is associated with greater robustncss to input noise.
7.4 Impact of our method the input loss curvature
Theoretical results from Theorem 0.1 indicate that training a model to be robust to Gaussian noise should lead to a reduction in input loss curvature. Our method enforces that features from clean and noisy inputs li on the same hypersphere for each class, promoting feature stability under Gaussian perturbations. As a result, the input loss curvature expected to decrease.
Given that the noisy test accuracy of the best-performing method is around 90%, we select the 90% of test samples with the lowest input loss curvatures for each method and compute their curvature values. Fig. 10
22

---

Published in Transactions on Machine Learning Research (12/2025)
Gaues an Notee Pearson coreation:
Gauss an Noice. Peamson correation:
Uritorm No se Pearson correiation: 0.37
Unitom Nose Pearson correaatoo: 0.32
Correct predictions 10
predictions 10
10 10
22
34
Average curveture each group
Average curvature each group
(a) CIFAR1O.
(b) SVHN.
Figure Average input loss curvature for groups of test samples, partitioned by the number of correct prodictions over 10 random noise injections. For CIFAR-10, we set the standard deviation of the Gaussian noise and the amplitude of the uniform noise to 0.06. For SVHN, we both values to 0.15.
shows boxplots of the curvature distributions for each training method. Our method significantly reduces input loss curvature compared to standard training. While training on both clean and noisy data also lowers curvalure, it less effective than our approach.
0.00100
Estimated Curvature
Estimated Curvature
0.000
Ours
Stabilitt training
Stability treining
0.000
0.00
Normal trairing
Training on both clean and noisy data
Ours
Stability training
Normal
Training on both clean and noisy data
Ours
Stability training
training
(a) CIFAR10.
(b) SVHN.
Figure 10: Boxplot of the top 90% lowest curvature values on the test set for each method. Since the accuracy of the best-performing method is around 90%, we visualize whether better performing method has lower curvatures among the top 90% lowest-curvature samples.
We also qualitatively examine the curvature of the input-space loss surface. For a fixed test example, we sample a grid of points the input space along two directions: the gradient of the loss with respect to the input, and a random direction orthogonal 2D grid, as shown in Fig. [I The results show that our method significantly smooths the loss landscape compared to the basclines. This qualitative obscrvation is consistent with the quantitative results in Fig. 10 where our method notably reduces the input-space curvature of the loss.
We then visualize the negative of the loss function over this
7.5 Impact of method intermediate layers
In the experiments above, use ResNet18 HH al. 2016) layer prior to the softmax layer. Recall that our loss function is applied to the features of the penultimate layer (i.c., the layer immediately before the softmax). Thus, it is natural to ask whether the proposed loss influences only the final fully comnected layers, or whether also aflects the backbone feature extractor. Figure [ visualizes the features at the penultimate layer and shows improved class separability under our method. However, one could argue that this improvement limited to the penultimate layer and does not reflect changes the backbone representations. While Section 6.3 provides theoretical insights into how our constraint alfects the intermediate layers of the network, we also investigate this question experimentally. To
the backbone, followed by fully connected
23

---

Published in Transactions on Machine Learning Research (12/2025)
Gradhet Disction
Cradand scection
(a) Normal training.
(b) Ours.
Figure 11: Illustration on 2D surface of the negalive of the standard softmax loss al a given input of SVHN. Original input is marked by the black dot.
do so, we apply the same t-SNE visualization technique to the intermediate feature maps the ResNet18 backbone. Specifically, ResNet18 consists of four main residual blocks, each producing outputs of dimension H; W; C. for 1, . 4, where H4 and W; are the spatial dimensions and C, is the number of output channels for block i. For each block output, obtain feature vector of size C. We then apply t-SNE these pooled features, done for the penultimate layer. The resulting visualizations for the clean and noisy CIFAR-10 test set are shown in Figures and respectively.
apply Global Average Pooling over the spatial dimensions to
Block
Block
Block
Block
(a) Normal training.
Block
Block 2
Block
Block
(b) Our method.
Figure 12: (-SNE visualization for features clean CIFAR1O test set of different intermediate layers of the models trained with normal approach and our approach.
We observe that the features extracted from the first two blocks do not form distinctive clusters, which is expected since these early layers primarily capture low-level patterns rather than semantic class information. Starting from block 3, however, clusters compared to standard training, shown in Figures 12 and 13 This effect becomes especially pronounced in block 4 for both clean and noisy inputs. In particular, the t-SNE plots for the noisy data (Figure 13 show a striking difference between our method and the slandard baseline. While slandard training fails to produce well-separated class clusters under noise, our method significantly enhances feature discriminability, even the presence of perturbations. These findings indicate that our loss function not only improves representations the penultimate layer but also promotes the learning of more structured and discriminative features in intermediate layers of the backbone. This empirical evidence aligns with
loss function begins to encourage the formation of more clearly separated
24

---

Published in Transactions on Machine Learning Research (12/2025)
Block
Block 2
Block
Block
(a) Normal training.
Block
Block 2
Block
Block 4
(b) Our method.
Figure 13: 1-SNE visualization for features noisy CIFAR10 Lest set of different intermediate layers of the models trained with normal approach and our approach (Gaussian noise with std equal to 0.06).
theoretical results presented in Theorem 6.3 further supporting the claim that our approach affects the modol's internal representations in a meaningful and robust manner.
7.6 Ablation study: complementarity of the constraints in our method
To investigate the complementarity of the proposed constraints, we conduct an ablation study. Note that the constraint imposed noisy data plays role similar enforcing intra-class compactness clean data. Therefore, to isolate the individual contributions of inter-class separability and intra-class compactness, we restrict our ablation study to constraints applied on clean data only. This avoids a potential form of implicit enforcement of compactness through noisy-data constraints, which could otherwise obscure the interpretation of the results.
We first fix the regularization weight to 'reg 10 and consider three configurations: (i) = 3 1, (ii) 0, = and (iii) a 1, = 0, where and control the strengths of the intra-class compactness and inter-class separability constraints, respectively. This set of experiments aims to assess the individual and joint effects of the two constraints. Next, to evaluate the role of regularization, we fix = 8 1 and compare three settings: "er 0, Treg 10-3 and Treg 1.
The results are summarized in Table 5 Several observations can be drawn from these experiments. First, combining both compactness and scparability constraints consistently outperforms using cither constraint alone, highlighting their complementary roles. Second, when applied in isolation, the compactness constraint yields better performance than the separability constraint. This aligns with our earlier discussion: while the softmax loss inherently encourages inter-class separability, it does not explicitly enforce intra-class compact- ness, making the compactness constraint particularly beneficial. Finally, when a 3 = 1, incorporating regularization leads to improved performance compared to the unregularized case (Treg = 0).
From a practical perspective, extensive tuning undesirable, small regularization value such as Treg 10-3 provides a robust choice, as performs well across different ablation settings.
Table 5: Ablation Study
Hyper parans
0.001
0,
Ttes
0.001
0.001
Trog
CIFARIO
93.43
02.25
94.01
93.66
93.81
SVHN
96.15
95.98
96.64
96.55
96.07
25

---

Published in Transactions on Machine Learning Research (12/2025)
8 Conclusion
We presented simple and general training framework that improves the robustness of deep neural networks to input perturbations while maintaining strong accuracy on clean data. Our method introduces a loss at the penultimate layer that is easy to integrate into existing models and training routines without requiring architectural changes or complex optimization procedures. This loss promotes intra-class compactness and inter-class separability of clean features, while also aligning noisy features the class level encouraging robustness without sacrificing expressiveness. From theoretical perspective, demonstrated that training with additive Gaussian noise implicitly regularizes the input-space curvature of the loss function by reducing the cigenvalues of its Hessian. This provides a principled explanation for the improved stability and ro- bustness observed in practice. Importantly we also show that this curvature reduction leads to more stable feature representations under noise, establishing a bidirectional connection between noise injection and input- space geometry. Empirically, our approach consistently improves performance on both clean and noisy data. Moreover, models trained with our method on Gaussian noise exhibit strong generalization other types of corruptions, including occlusion and resolution degradation. This demonstrates robustness beyond the specific perturbation seen during training. Overall, our work offers theoretically grounded, easy-to-adopy and empirically validated approach for enhancing robustness in modern deep learning systems.
References
Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-aware minimization. In International conference on machine learning, pp. 639-668. PMLR, 2022.
Christopher M Bishop. Training with noise is equivalent to tikhonov regularization. Neural computation, 7 (1):108-116, 116, 1995.
Jeremy M Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning, 2019.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory 2nd Edition (Wiley Series in Telecommunientions and Signal Processing). Wiley-Interscience,, July 2006. ISBN 0471241954.
Bert De Brabandere, Davy Neven, and Lue Van Gool. Semantic instance segmentation with discriminative loss function. arXiv preprint Xrxi::1788.00 02551, 2017.
Laurent Dinh, Razvan Pascanu, Yoshua Bengio, and Samy Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, 2017.
Yinpeng Dong, Zhijie Deng, Tianyu Pang, Jun Zhu, and Hang Su. Adversarial distributional training for robust deep learning. NeurIPS, 33:8270 8283, 2020.
Gamaleldin Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy Bengio. Large margin deep networks for classification. NewrIPS, 31, 2018.
Mahyar Fazlyab, Taha Entesari, Aniket Roy, and Rama Chellappa. Certified robustness via dynamic margin maximization and improved lipschitz regularization. NewrIPS, 36:34451 34464, 2023.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021.
Tan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. International Conference on Learning Representations, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In
Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 770 778, 2016.
Zhezhi He, Adnan Siraj Rakin, and Deliang Fan. Parametric noise injection: Trainable randomness improve deep neural network robustness against adversarial attack. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp. 588 597, 2019.
26

---

Published in Transactions on Machine Learning Research (12/2025)
Andrew Howard, Mark Sandler, Grace Chu, Liang- Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 1314 1324, 2019.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report, 2009. URL http://www.c tttp///twww..t.oooooooooooooooo. wttpa/tt..eddu/kk/////aaaarr html
Alexander Levine and Soheil Feizi. (de) randomized smoothing for certifiable defense against patch attacks. NewrIPS, 33:6463-6475, 2020.
Hao Li, Zhengdao Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In NewrIPS, 2018.
Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolutional neural networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learnäng-Vokie Volume 48, pp. 507-516, 2016.
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 212-220, 2017.
Takeru Miyato, Shin Ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. Distributional smoothing with virtual adversarial training. In 4th International Conference on Learning Representations, ICLR 2016, 2016.
Mehryar Mohri. Foundations of machine learning. Adaptive computation and machine learning series. MIT press, 2nd edition edition, 2018. ISBN 9780262039406.
Seyed-Mohsen Moosavi-Dezzoolli, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Robustness via curvature regularization, and vice versa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 9078 9086, 2019.
Walced Mustafa, Robert A Vandermeulen, and Marius Kloft. Input hessian regularization of neural networks. arXiv preprint ar *=Xim:200900677) 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 4. Granada, 2011.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24652 24663, 2020.
Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhusscin Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. Adversarial robustness through local linearization. NewrIPS, 32, 2019.
Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto-encoders: Explicit invariance during feature extraction. In ICML, 2011.
Yan Scholten, Jan Schuchardt, Aleksandar Bojchevski, and Stephan Günnemann. Hierarchical randomized smoothing, NewrIPS, 36:49783 49813, 2023.
Christian Szegedy, Wojciech Zaremba, Πla Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
Yichuan Tang. Deep learning using support vector machines. CoRR, abs/1306. 0239 2(1), 2013.
Yusuke Tsuzuku, Issci Sato, and Masashi Sugiyama. mosschtzz-marin training: Scalable certification of perturbation invariance for deep neural networks. In NewrIPS, 2018.
27

---

Published in Transactions on Machine Learning Research (12/2025)
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In Compuler Aesion-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11 14, 2016, Proceedings, Part VII 14, pp. 499 515. Springer, 2016.
Bohang Zhang, Tianle Cai, Zhou Lu, Di He, and Liwei Wang- Towards certifying linfinity robustness using neural networks with l-inf-dist neurons, In International Conference Machine Learning, pp. 12368- 12379. PMLR, 2021.
Bohang Zhang, Du Jiang, Di He, and Liwei Wang. Rethinking lipschitz neural networks and certified robustness: A boolean function perspective. NewrIPS, 11119998-194199 2022.
Stephan Zheng, Yang Song, Thomas Leung, and Ian Goodfellow. Improving the robust of deep neural networks via stability training. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 4480-4488, 2016.
28

---

Published in Transactions on Machine Learning Research (12/2025)
Appendix A More insights about our loss function
A.1 To square the loss or not to square the loss
We can notice that each term under the sum operation in Ccompact is squared, whereas this is not the case for Lmargin- this section, we will discuss the rationale behind this design choice. We notice that terms under the sum operation of Loompact and Lmargii can be written a general form as
d(u) = [s||u - urel b/+,
if non-squared version ([s]|u Wreell || + 114 if squared version
where e equal to or -1 (the sign indicator) and user is the reference point. In the case of Crompact: ||u urelll the distance from a generic point to its corresponding centroid denoted here by uref. In the case of Emargiu ||u Wrell is the distance of centroid to its projection on the closest boundary. For sake of simplicity, we ignore the sign indicator before |/u utef as here this does not matter. Consider the case where @(u) is still activated, i.e.,
||u- Mret |I + b > 0. In this case, we can show that:
n-Me I lu-wa uoot II 2 u Urot || + b) x
non squared version
Beet u
squared version
which implies
1, 12 x ||u usetll + b). if squared version
if non squared version
Consider the regime where u approaches Uref- If not squared, the gradient remains of constant magnitude as long it is still activated. In contrast, if the term is squared, then, for states close to be deactivated, |||u - Mfeel + bl is close to 0. Hence, the gradient is very small. Thus, the magnitude of the update dircction for u becomes minimal when is close to the deactivated state.
Following this discussion, on the one hand, squaring in Lcompact gives a smoother and less stringent loss. For example, if there are very abnormal points, then this condition does not enforce completely the points to be in the hyper-sphere around their centroid. On the other hand, by not squaring for the margin loss, enforce stronger conditions on the centroid until it attains the deactivated state. That is, its distance to the closest boundary is at least larger than da, This is important as the position of each centroid impact the distribution of the whole class, This insight justifies our proposed loss functions.
Appendix B Proofs of intra-class compactness and inter-class separability
B.1 Proof of Proposition 5.1
Proof B.1 Let @i and d2 be arbitrary points (in feature space F) in any class By triangle inequality, < || - mell + [| 92ll S 8, 6, 28..
we have: ||q 22ll
B.2 Proof of Proposition 5.2
Proof of Proposition 5.2 1
Proof B.2 Consider an arbitrary class c. Let p be a point in this class (in feature space F) and let g be ony point on the hyperplane Pas for any # Then, by triangle inequality, we have:
|m.- 411 I ||me pll 11p gll-
As d(me, Pa)) 8az have |lme all 2 5g. Aq E Per- At the same time, we also have I|me pll dg Consequently, obtain:
5a ||me //l < |mm pll +lp all s 6. + +lpp all-
29

---

Published in Transactions on Machine Learning Research (12/2025)
Hence, 1pp- 2 5a - de, Vy Pa. So, by definition, d(p. Pa) 88 88. This holds vp Ce and Vi & Hence, margin(c) = minpeC. (minize d(p, Pa)) 2 da 8u. As we choose an arbilrary class e, this holds for all classes. The proposition is proved.
Proof of Proposition 5.2
Proof B.3 Let q1 and 92 be 2 arbitrary points in any class same By Proposition 5.1 1 we have: ll 92ll s 25.. Now, let and by 2 arbitrary points in any two different classes i and j, respectively. It suffices to show that [pp - q1 > 26, with 54 > 26.. Again, by triangle inequality, we have:
Im; m,ll - I|m; - pll + 1p myll S I| - pll + (lp all + 1lq-my)l))
So, 1|p 2 I|m myl (I|m pll 11g - mzll) |i mmj--mpl 28.- Now, let consider ||m myll- By (cuchy-Schwaar inequality, we have:
(m, my, IW W/1)/
W: -w-WW W,
$ <1lm; mmyl- IW; Will
1|m; myll.
IW: Wyll
Hence,
(W, w,,m)) (W, - W,, my)
m4 m,]1 2
IW; - -Will I|: Will
(W, Ws)m))) + (b. - bbp) (W,
W3, m + (b: b,)
IIW, - WW,l
[ Wyll
As m, and m, are on 2 different sides of the hyperplane Pig (which is the decision boundary), (W; - W mi)
(be - by) is of opposile sign of (W; - m,) + (b. - b,)) Hence,
(W.- W, m2) + (b. - b,) Will
(W; - W,, + (b. - b,) I[W Will
( Wg.m,) + (b, - -b,)
+
W.
IW: -Will
(W: Wy.m,) + (b: bj)
I W,ll
Consequently,
(W; Wj, + (b: - -b))
(W; W3, m. + (b; bi) IW, W,ll
mm mjll 2
+
I, Will
=d(m;, Pij) + d(m,, Pis)) > 26g.
So, [|p 4/1 [| - mmil- 26,, 2 28 28,, With da 28,, we have l|p 11--9/ 91
28.. So, 1lp 1p-9l gll
|| -qall-
Appendix C Proof of Gradients in Proposition 4.1
Proof C.1 If Lmargin = 0, then the grudient w.r.l. is 0, hence proposition is proved. Let us now consider the case where Emargin is still larger than O. We can easily see that
VoLmargin = sign((me)) (mele) Vod(me, Pey).
Hence, if we show Vod(m (1 - 7) Vod(m Pei)) the proposition is proved. First, recall that for two differentiable functions: 91 1 Idd and g2 RRda + Rdd, and if g denotes the composed function = 92 91, then for I4 we have the following chain rule for computing the Jacobion matria:
Jz(x) Jop (x) Jgy (2))
30

---

Published in Transactions on Machine Learning Research (12/2025)
In the case where d3 1, then have:
Vag Jg(x)² = -2)(2) (z) x J (01(x)) J (2) x V (1()))2
Now, recall that d(me; Pai) W.b.-..,m,++b---- For suke of brevily, consider the case where (We -Wll
W, me) + (bc- by) 2 and the orgument the same for the case (W Wa, me) + (b.- 6,) < O. We have, Using the choin rule from Eq. (.11 with 92(-) = 1W.. +.b) bi)l we
d(me, Pa) (We- We will
I Wil
get:
Vad(m
Tud(meurrent current .Pa) - X Mo.22.
On the one hand, we can easily show that Vm,92 is independent of me and equals to We-W, [W.-
So,
We W
Vod(ma, Pa) - (0) X
I Will
On the other hand, we have that m moment
-.mm, 7 m - + (1 7) m
ourrent Consequently,
(1-11-)) J Ingurrent (0). J m
momoen
Hence, Vod(m moment Pas) = 7) V Vod(m .Pci). So,
V
margin
= = - 7) Vo V
guaive margin
Appendix D Proof of Proposition 5.3
Recall that we work in the feature space where each point is denoted by g. Each q itself is the output of its prior layer. As assumed in the theorem, this is convolutional layer fully connected layer, with without non-linearity ReLU, Convolutional or fully-connected layer can be written in a general matrix-operation form as,
W +b, ReLU(W q + bb), if ReLU.
no non-linearity.
==
Here, W- and b- are parameters of the prior layer, q is the input of this layer. We denote this layer by hyw-a b- We recall that ReLU(a) max(0, a), Va R. When ReLU is applied to vector, understand that ReLU is applied to each element of the vector.
Remark D.1 It is obvious that (y(w-,->)9-)) vhiw--->lq VV R.
We also recall that the softmax layer has parameters (W, b) (see Eq. For case of notation, we refer to (W-,h) b,, as flattened colunm-vector concatenating W-, b- and b. Furthermore, we denote θ = 9, (W-,b~,b)) i.e., O is the vector containing all the parameters N except (W-,b=, bb,"). For an v > 0,
define To
0 (WV-) To: (W-. (WW-bb-.)) .,)
6 (wW-.bb..)) ( = ((W-,b-b))) (wW-b--b))
θ
In words, by applying To θ, (W-,b- and other parameters of @ stay unchanged (θ, O,).
bb_ b) is multiplied by that is to say (WV- b.. = -(W NoW--b--00 ,D)
1. First, we prove that after applying Tr, the predictions of model do not change. Consider an arbitrary input. By Remark D.1 we know that q is transformed into q = vg. Morcover, from Eq. we have
== b= == Way vb (W b)
31

---

Published in Transactions on Machine Learning Research (12/2025)
Now, by Remark 3.1 the outputs of the new model and the old model arg max, ve, and arg max; 2 respectively. As 0, arg max; vz; arg max; So, the new model has exactly the same prediction the old one.
2. Now, let us consider the class margin. Recall that the distance from to the decision boundary Piy (between class i and j) is d(g, Pıj) = I(W, W. ( (b)= So, the distance after the transformation is |W; W
d(q,Pi) Piy) IW; - W,. ②) + (b: - bi)l
(as W stays unchanged)
IW, Will
IWW; wg. - -v(b; -bj]l
- KW, - -W + +(b,- -by)l vd(a, Pis).
I[; Will
So, the distance of q to an arbitrary decision boundary is multiplied by v. As we consider an arbitrary input, by the definition class margin, the class margin all classes multiplied by v.
3. Now, lct us consider the class dispersion. Let consider two arbitrary points q1 92 E F. The (euclidean) distance between these 2 points is ll q2ll- By Remark D.I we know that qi and 92 are turned into @1 = ugi and Q2 := u42, respectively. Hence, the distance between these 2 points is now ll 02ll = 11/q1-1 g2ll. So, the distance between @i and q2 is multiplied by v. As chose 2 arbitrary points, this holds for any 2 points. By the definition of class dispersion, it is obvious that the class dispersion of any class multiplied by v as well.
4. Prove that Te))) = (V-1)op where u is a vector depending only on By definition Tu,
we have that
Tz(O) θ v(W b bb) .b) - (IV-)) O, (W-, b-, )))
9,
(v- 1)(W ,,, (,,)) (v 11) ( (W-,b-, NoW-,b_b, .))
0
So, we have that Te( - θ = (v 1)u, where u = (W--8-,1 ,,, depending only on θ
Appendix E Proof of Theorem 6.1
In this section, provide complete proof for the bounds of the curvatures. For clarity, introduce
separately some technical lemmas that are useful for the main proof (Lemmas E.1 E.2
E.3
and
E.4
in
Subscction
E.3
E.1 Proof of the upper-bound
Proof E.1 Using the properly tr(ff²)= = |1 = E, X in Lemma E. 1 it suffices to upper bound tr(H)).
Using the assumption on quadratic approrimation of 1(x +E) around x, we have:
1(*+=) vi(=) J HE.
Step 1. Showing that tr(H²) < Ex|((x + - ((x)1
We analyze the squared loss change:
-(xx)) = (Vi(z) & 5 THe 2
32

---

Published in Transactions on Machine Learning Research (12/2025)
Let's denote A = V, B = TeT... He. Take expectation:
- [(x))) = =[A²] + 2E[AB] + +
For the cross-term, we have:
E[AB] [[([V] e) H&)] 1 Vi Else T Ha] 2
2
By Lemma E.2 we know that Else He] 0. Hence,
E[AB] = 0
For the quadratic term, by using Lemma E.3 we have
1 BI(E H=) 1 -0 [2
So we get:
+ x + + (tTH) 2 0²
[ (c) -())))
2 |
(22)
Therefore,
-
|1l o. () e)) ((x))
Step 2. Partitioning the expectation of the squared loss change into 2 regions.
Let A be the event "fo(x +e) 8) So, P(A) 7)- We split the expectation into two regions A and A:
[(x = [1(x e)) A] n + +e) -))x)) [ (1 --m))
(23)
Step 9. Showing that [[xx +e) 1(x) < 2 max, AZzي where Azj = 2,(0 + 8) - 2x)x)) is the logit shift.
Recall that the cross-entropy loss w.r.t. the true class is
eFu(m)
1(x) = log Σ, E = - -y(x)) + log M e>((0)
Hence, let P; o(2), 6 E, et (so E, Pj = 1), have
E, (3((+))
((x +e 1(x) = -Ay + log
-Azy + log Σ Drede.
Σ, 0z)(z)
Therefore,
M< + +) - -(xx) < A
log Σ Epeess
33

---

Published in Transactions on Machine Learning Research (12/2025)
As E 2 0, the sum Ej pseAss is conver combination of exponentiols. Hence, the log-sum-erp function satisfies
min Azj S log Σ pre I max Azj.
Hence, log E, p S max, Azyl. Obviously, we also have | I max, / So,
[1x - 8) - -((x)] s 2 max |
Step 4. Upper-boundivg max; |Azl in the case where A happens.
In this case, fo(x + E) E (x)(x), 8). It is easy to see that llfo(x + -o fo(x)) S Recall that the logits for imputt x defined by
=(x) = W +bE
The logits are
= fo(x) + bj.
Let v so(x +0) So(x), so after perturbation,
2,(x + = W3 fola + +)) 2z(x)) + W,. v.
Thus the logit shift Azy bounded by
| = (2 -2 < || ||||| - W|, 8.
Let
| == max [W,ll.
Then for all j,
| S W 6.
This means that max; |Azyl S || 8.
Step 5. Upper-bounding max; |azو in the case where A does not happen. In this case, as assume that the absolute volue of logit is bounded by Kmax, it is easy to see that max, |azy I 2K
Step 6. Upper-bounding E [1(x + - (x))²] to upper-bound tr( (r(
From Eq.
23 and using the results of Steps 3, 4 and 5, we have that
4n (()))) 8² + 1661 17) K2
-
max
So
8 01 (nllW]2..cc 8² + 441 11) K2nx)) tr(H12) <
This completes the proof.
E.2 Proof of the lower-bound
- E, de in Lemma E. we first establish relation between the terms relating to (H) and the expectation of the squared loss change.
Proof E.2 Step 1. Using the property tr(H2) |
E, X3 =
and tr(H)=
34

---

Published in Transactions on Machine Learning Research (12/2025)
From the result of Eq. 22 in previous part, we know that:
=) - -(=)) - 1 +
where A ViT.. It is easy to that E[A?] =
= 1
So,
2 2| + (trH)² ot (B[(((r +) 1(x)) oF [V
We note that | = Ei X and tr(Hi) = | A S > Ail which leads to
2
2 >xae+ + Σ (all
ス 04
4
[B[((z ( [(=))( o² )2I
Step 2. Partitioning the expectation of the squared loss change into 2 regions to find a lower bound.
Let A be the event "fe(x + E C(f(z),))' So, P(A) 77 We split the expectation into two regions and A:
= E (1( A n7 + E (())) { (1 -1-10 1)
E
2 (1 n) E (Mxx+)) + - -(xx))² 지
We notice that by Lemmaa E.4 we have:
E [1(x + e) 1(x))² A = Var(1(x + 8) | [A) + (e [[x E) A - -1xx))
2 [B] [ A - ((x))² ²
(loue-(=))) (E [(= + E) | A Lout)
Thus,
E
(() ((=])[()]]]]] 2 (1-n) (lout ((x))²
Step 3. Conclusion.
Using the results of step 1 and step leads us to the conclusion
2 Σ 22 +
Σ |ul)
4 σd (a- 11) (low 1(x)))= -021|
This completes the proof.
E.3 Some useful technical lemmas
We recall some classical lemmas (with complete proof) that are useful for the proof of Theorem 6.1
Lemma E. Let H E RRan be a symmetric matris with eigenwalues Xι, A2, A2,.... An Then,
tr(I)) = Di d and tr(1f2) X
Proof E.3 Since H is symmetric, by the spectral theorem, there exists on orthogonal matrix Q (i.e., Q I) and a diagonal matria A diag(A1, A2.... An) such that
H = QAQ..
35

---

Published in Transactions on Machine Learning Research (12/2025)
Then,
a
tr(I)) (r(QAQ") ( Σ X
Moreover,
H² = ==== =
Q1²Q²
where A diag(A}, .g...., A Using the cyclic properly of the trace, have
=(#))==(Q["")== -rr())) = Σ x2.
2
This completes the proof.
Lemma E.2 Let E ~ NN(0,² and let H E RRxx be a deterministic matris. Then:
Elee He] =0
Proof E.4 We consider the th component of the vector Elee" He] E R". It is given by:
E 88 Hel); = R
ese (HE)(
Bleje,(He)]l
Note that
n (He)
Haakk
80 we can urile:
n
(B(es) Hel),
Σ j=1k=1
He
Now, since e ~ NO0, ((0,02In) I2),, the random variables e, are jointly Gaussian with zero meon. Therefore, any odd-order moment (i.e., the product of an odd number of Gaussian variables) has expectation zero:
E[exefe]] = O for all i,j,k,
Hence, every term in the above double sum vanishes, and we conclude:
(EleeT HEl), = for all i.
Therefore,
Elss T He] = 0.
Lemma E.3 Let ~ I², and H E Rax be symmetric matrix. Then,
Bi(s T He) = 20 | (
Proof E.5 Let o2 and H nxn be symmetric matria. We want compute
El( He)
Since H is symmelric,, il be diagonalized
H = QAQ
36

---

Published in Transactions on Machine Learning Research (12/2025)
where Q is orthogonal matriz and A diag (ang(A1,,... An) contains the eigenvalues of H.
Define É == QTe. E. Then ~ o and we have
T = = =TQAA - = =E Σ
1=1
Define x=
= Ei=1 7157 Then:
12
E[X²] E
Σ X1E
Σ
M Blefag]
Now recall:
[lE =30 for all
E[ = =|le) = 0², for i #j..
Therefore,
B[X²] =
X -30 + Σ X1dy..0 0d
1=1
#F
n
30
X2 + 04
Σndd - Σ x2
(/)
i=1
=30 =30*1H a
((1rH)2-111))
20
H + (trH))
So, E[(E" T HE) = 20 + (trH))
Lemma E.4 Let X E R be a real-valued random variable, and let be an event (or more generally, sub-o-algebrra. Let p TR be fized constant. Then:
E [[[X p)² [A] Var(X AA) + [E [A] 1 2p)
-p2²
Proof E. 6
Let = X A). Then can write:
X- (X - 1)) + -Jr- ((--)). p),
and hence:
(X p)- (X - 112 + 2(X - 10(H p) + (p--00 p)?
Taking conditional expectation with respect A, and using linearity of expectation:
E((X- - 0)² A] [x[(X- 14)2 A] + 2(u- p) 1 A] + (1r - 2)
Since fa E[ | A], we have E X - 44 A = 0, and thus:
EE(( 0)2 A] E((X 11)² A) (n p)² 102=Vark Var(X A) (E[X A] -
37

---

Published in Transactions on Machine Learning Research (12/2025)
Appendix F Proofs of generalization errors
In this section, we provide the complete proofs concerning the generalization errors discussed in Section 6.2 Our proofs are inspired by the methodology developed Mohri (2018). To begin with, we introduce some notations and results given in Mohri (2018).
Definition F.1 (Empirical Rademacher complexity, p. 30 in Mohri (2018)) Let be family of functions mapping from 2 (a, bl and S (21,..,,2) 2N) fixed sample of size N with elements in Z. Then, the empirical Rademacker complexily of H with respect to the sample S is defined
N Eor((z))
R&(H) (H) E. sup KEH
(24)
where a = (N), with o; 's independent uniform random variables taking values in f-1,+1}. The random variables o, are called Rademacher variables.
Theorem F.1 (Theorem 3.3, p. 31 in Mohri (2018) Let G be family of functions mapping from z []. Then, for any 0, with probability least 1 5 over the draw of an i.i.d. sample S of size N, the sollowing holds for all g G:
1 B[g(Z)) S N Σ 2R + +3
log 2 8 2N
Lemma F.1 (Talagrand's lemma, p.93 in Mohri (2018)) Lel + R be 1-Läpschilz ( 0).
Then, for any hypothesis set H of real-valued functions, the following inequality holds:
Rs H) $ (H)
Using Theorem and Lemia we can derive following theorem:
Theorem F.2 Let H be set of real-valued functions and let Px be the distrilrution over the impul space X. Fix p O, then, for aryy O, with probability at least 8 over sample. S of size N drawn according to Px, the following holds for all h E H:
2 R(h) S (h) + Rs(HH) +3
log 3
(25)
2N
where R(h) = [[1 (yeneralization error), ( - + (p(h(r:)) is the empirical maryin loss on of size N and Rs(H) is the empirical Rademacher complerity of H with respect to the sample S
Proof F.1 Let g = o oh: H}. By theorem
F.1 for all g E EG
1
N g(2,) + (2s(9)) +
log
Elg(X)] <
N
2NN
and consequently, for all h E H,
1 N
log a
[[o <
Σ x (h(z)) ( H) + +3
N
2NV
1=1
Since (150) S ()),,, we have R(h) E[ [mx) S E(0,((((X)),
log 2
R(h) S Rs, + ( o H) + 3/
2N
38

---

Published in Transactions on Machine Learning Research (12/2025)
On the other hand, since pp is Lipschtt, by lemma F.1 we have Rs(0, H) - RH) So, R(h)
(h))(h) (H) 3/ log 2N
With all these notions and theorems, we are well-equipped to prove Theorem 6.2
F.1 Proof of Theorem 6.2
By using Theorem F.2 to prove Theorem 6.2 it suffices to prove the following result:
Theorem F.3 The empirical Rademacher complexity of H can be bounded as follows:
R VN
Rs(H) S A² 2RA +
(26)
Proof F.2 Let G2 = {f: susuex f(x)]l S A}. Let o = (o1,....oy) with 0, 's independent uniform random
wariables loking values in +1] By definition of the empirical Rademacher complerity, we have:
(H)
N
Eg
sup
0, [f(xz) -mll3)
N
|m]| R E92
i=1
1
E
sup
f()))((x;)) ml|
|| REG2
N
N
Eg
sup
ff - mm|
as Eg
Σ
o1p2
= ,2
Eg [o.] = =0
N
||m R.IIIRR./EGA
E.
sup
Σ (-orll1( -a
m||m|||? + 20, (f(2s)
N
<
E,
sup
o.llf(>>)11 + sup
Σ os m/||m||
sup
Σ
20, 20, (f(s)
N
FEG2
[m]]
mmIlERR/EE) E092 i=1
Considering each. term inside the expectation operation, we get,
N
Eg
sup fE
Σ
12
s E,
sup
o (1+))))
5 E.
sup
Σ
1o
lo
fEGz
JEG2 im1
N
NV
E,
sup
Σ
(x;))1
< E
Σ A2
NA²
fEGa i=1
39

---

Published in Transactions on Machine Learning Research (12/2025)
Consider now the second term. We have,
N
N
N
Eg
sup S|
Σ
|m|
< E
sup
o |m|
= Eg
sup
|mml
Σ
5
||
||m||
i=1
i=1
N
N
27
R
Σ
T
S
R2
Ex
Σ
0i
1=1
i=1
N
R²
Eug
Jiaj
W==
N
R²
E
of
2
( as
[o = 0 if x and 1 otherwise)
R2VN
Consider now the final term inside the expectation operation.
N
F
sup mIIIRR
Σ 1=1
20 (Is)),
S EF
sup
Σ
20 (1:)), m)
R.
1=1
N
< Err
sup
20 || [xx)
|| f2
1=1
< E
2R sup /
liil [1
EE,
2R sup 8E92
Σ
[[f(z.)]|
S Eg
2R A
2N
Hence, RM) < (A + RVN + 2NRA) = A² 2RA + R²
Appendix G Some Definitions and Proof of Theorem 6.3
We first recall some definitions and lemmas needed in the proof.
Definition G.1 (Kullback Leibler divergence) Let Dj and D2 probability mensures on a measurable space Q. The Kuliback-Letie divergence (KL divergence) between D2 and D: defined as (Kz(Di) Jeen log
dDi 2D (w) Di (dw). and too if the Radon-Nikodym derivative d'd does not crist. aDz
We note that D is not symmetric. To overcome such drawbacks, we can use the Jensen-Shannon divergence with the following definition.
Definition G.2 JensennShannnn divergence (JSD)) Let D2 and Di2 be probability measures on a mea- surable space
Q. The Seanen-Shhnnen divergence. (JSD) between D2 and DD is defined DDs(Di])D)) } (Pi((P) } D where D 2 (D, D2) is mixture distribution of D2 and Dz.
Definition G.3 (Total variation distance) Let D2 and D2 be probability mensures a mensurable space. Q. The total variation (TV) distance between Dj and D2 is defined as
drv(Dr.D.) sup (D(S)) (S)) sca
40

---

Published in Transactions on Machine Learning Research (12/2025)
Lemma G.1 (Pinsker's inequality) For every D1, D2 ft,
drv D2) 1 V (DL((i))))
Lemma G.2 (Relation between JSD and mutual information) Let B be the equiprobable binary r.v. taking value in {0.1}. Given two distributions D1 and D2, let X1 DD let X2 D2. Assume that X1 and X2 independent of B, let X BX1 (1 (1-B) X2. Then, we have
DJs(Dil(s) = B),
(27)
where I(X, B) is the mustual information of and B (see Cover Thomas (2006)).
Remark. We see that we choose the value of X according to D if B = 0 and according to D2 if B = 1. That is, B = 0) D, and D B = 1) D2. Honce, B can be considered as "flipping variable'. Morcover, notice that in Eq. 27.. the entropy (used in the mutual information) is with respect to a reference measure and that implicitly Dj and D2 are absolutely continuous with respect this reference measure.
Now, we are ready for the proof.
Proof G.1 We consider any class pair (y 4y) y2, Let us first consider the conditional distributions on F. For brevity, write instead of Q². We have that
Dol + +Dow
(Ql DQn((S) + Dow (S)
dTv DQly
sup SC.F
2
2
(Qlx DDir (S)
sup SCF
2
We consider = C(my. 5z), the hyper-sphere of radius 5 centered at my- Recall that we have assumption DQly(C(m, ([((myy,,)) 6.)) 2 r, Vy E (1/2 < T 1). Hence, we have (Qiy(S) 2
Moreover, as the Spper-spheres of each class are disjoint, 5 = C(mmy: 80) S C(myy 86). So, DQ (S) S Dow. (F N C(mmy 5.)) = DQw (C(my 8n)) S -1-
Therefore, have that (S) Dy (S) -- (1 27 1. Note that this term is positive 2 1/2. Hence, (S) (5)) (y) (S) (25)220 11
So, drv (Dqlu) Denu+Daix
supsEF (S) Den (5)1/2 (2r 1)/2.
Using Lemma G.1 we have that
DQu + Danx')
Dol + DDql' (2r 1)2
DKL.
(Dll ||
2 drv
(Delu
2
As this hold for any (y # y') E V, we also have that
Daw + Dajy'
(2r 1)2
DKL (Dew
Therefore, by summing these terms, using the definition of JSD, we have
(2r - 1)2
Dus (Dqls || DQw: >
(28)
Let B be the equiprobable binary r.v. taking value in {0, 1}. Consider now the conditional distributions Da'ly and D Do in the feature space F of lager before the penultimate layer. Let Q2 be the random variable
41

---

Published in Transactions on Machine Learning Research (12/2025)
associated to mixture distribution of Dosly and Dolle with B as flipping variable in the way defined in Lemma G.2. So, we have
DJS (Darlu DQl'v) I(B
(29)
Let ft denote the mapping representing the set of layers between layer I and the penultimate layer. So, we have that the distributions Do y and Dols' are induced by fi from Do and Dally' respectively. Let Q'B QB. The data processing inequality
QB - f)( So we have the Markovian relationship: Cover & Thomas (2066) sollows that
T(B, 2 T(B,QB)
(30)
Moreover, note that the distribution Dos is induced by St from Da's" Given =0, 0, D(Q')) = =0) = 1o So, the distribution D B = 0) is induced from Da by ft. That is, D(Q B = 0) DDqw Similarly, we also have DiQn = 1) = DQ y-- Hence, by Lemma G. we have
DJS (Dely Dawr I(B,QB)
(31)
From 29 have that
30 and
31 we have Das (Dolv || (Qtlr)) 2 Das (Daiv || Dewx)) Combining with 28 we
DJS (Dor Dollr) (2r - 1)2
This completes the proof.
Appendix H
Experiment details
In this section, we provide the details for conducting our experiments. All of our experiments are performed using Pytorch. When the models are trained, we fix the models and test with different noise levels and perturbations. We run independent, perturbations to report the results.
H.1 Training ResNet18
For ResNet18, the features of the final convolutional layer are subjected to Global Average Pooloing (GAP) over the spatial dimensions to obtain vector of dimension 512. Then this vector is passed through a ReLU non-lineariy, followed by fully connected layer (output dim=128). The output space of this layer is the feature space F in our framework. Finally, this is passed through the softmax layer (with 10 output classes), which is composed an affine transformation and softmax function.
We intentionally aimed keep a very minimal setup. Thus, we use the same training scheme for all the experiments with ResNet18, with softmax loss and loss, both CIFAR10 and SVHN. We use SGD optimizer with nesterov momentum, with initial learning rate of 10-3 (decayed by factor of 5 at epochs 60,120 and 160), momentum 0.9, weight decay 5 10-4, Model is trained for 200 epochs, with mini-batch size equal to 64.
For the training sets of CIFARIO and SVHN, we apply data augmentation techniques including Random Cropping (padding=4) and Random Horizontal Flipping before data normalization. This referred as normal training in our main text. For the other methods using Gaussian noise, we simply add Gaussian noise top of these data augmentations.
For our method, we set 5, 0.5 and da = 5.0 in our constraints. The total loss is Liotal = Lours + Ls, where Cs is the standard softmax loss (see Eq. 33)
H.2 Training MobileNetV3 on our road image dataset
We use models pretrained on ImageNet as initialization as our dataset is quite small train the model from scratch.
42

---

Published in Transactions on Machine Learning Research (12/2025)
During training, apply data augmentation techniques including: ColorJiter(rrighhne==-03.3 con- 3.st==00,,sttuaaioo=033333 RandomPoiation((1) Random Perspective (with pytorch default parametersjand Random Horizontal Flipping, before the data normalization. This is referred to as normal training in our main text. For the other methods using Gaussian noise, we simply add Gaussian noise on top of these data augmentations.
We use Adam optimizer with learning rate 10-4 (decayed by factor of 10 at epochs 50). We train models for 70 epochs, with and keep the best models based the validation accuracy.
For our method, we set 5, 0.5 and 9ạ = 3.0 in our constraints. The total loss is Leotal = Lours + Lss, where S is the standard softmax loss (see Eq. 3
43


Reviews:
[{"review_id": "8xGdfBDbYP", "reviewer": "Reviewer_KYC7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose to append several complementary \"loss\" terms to the training objective of deep neural network classifiers in order to improve their predictive robustness (resilience to noise). Interestingly, all these terms rely on the centroid of each class representation as given by the penultimate layer of the neural network (denoted $m_c$ for $c = 1,...,C$). \n- The \"intra-class compactness\" term, borrowed from De Brabandere et al. (2017), enforce every representation of the sample sharing a same class $c$ to be close to $m_c$\n- An \"inter-class separability\" term enforcing a large margin between class centroids, \n- An optional \"regularization\"  term penalizing the norms centroid vectors $m_1,...m_C$.\n- A \"noisy data feature alignment\" term ensuring that the representations of perturbed inputs of samples of class $c$ are close to the centroid $m_c$ (computed solely on clean data).\n\nThe manuscript clearly explains how the authors have achieved combining these terms to keep the accuracy of neural network classifiers on clean data while making them robust to noisy samples. On the downside, the complementariness of each term could be better explained, by clarifying some passages and providing ablation studies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The rationale of each design choice is well explained. The notation is clean. In general, the paper provides both intuitive explanations and theoretical insights, as well as a convincing empirical assessment of the method. However, there are some aspects that could be improved.\n\n**1. Discussion on margin-based methods**\n\nEnforcing the margin between classes is an historic principle in machine learning, and few are said about this in the paper. The SVM is only mentioned when citing Tang (2013) in the related work section, even if the expression \"hinge loss\" appears before Definition 5.1. I would like to see a short discussion about the relation of the proposed method with the SVM objective.\n\nAlso, it is quickly mentioned after Eq. (9) that the proposed  \"inter-class separability\" margin loss is inspired by the work of Elsayed et al. (2018). It would be important to highlight the key differences. Note that Elsayed et al. (2018) discussed the relation of their approach with the  SVM margin. The current manuscript only claims that \"[maximizing the] distance from each sample to all the decision boundaries\" would be \"very costly and not really necessary\" (Section 4.1), which is elusive and deserves explanations.\n\n**2. On the complementarity of intra-class compactness and inter-class separability**\n\nI think Section 5.2 is an important justification for the proposed method, but is difficult to read. The first sentence (\"Recall that the Huygens Inertia Theorem decomposes total inertia into\"...) deserves a reference and further explanations. The implications of Proposition 5.3 and Corrollary 5.1 could be explained more gently. I suggest explaining $\\mathcal{T}_\\alpha$ before the proposition (is it a *linear* map? Also, it might be better to rename $\\alpha$ to avoid confusion with Eq. (7)'s hyperparameter). A figure could possibly help the reader here!\n\n**3. Empirical role and complementariness of all loss terms**\n\nThe manuscript lacks training details, particularly about the values of hyperparameters $\\alpha$, $\\beta$ and $\\lambda$ multiplying the training objective loss components. Ideally, I would like to see illustrations of the classification accuracy when varying these hyperparameters, which would also lead to ablation studies for the cases $\\alpha=0$, $\\beta=0$ and $\\lambda=0$.", "audience_yn": "Yes", "audience_explanation": "Despite not being very knowledgeable of the relevant literature, the proposed method seems new. I believe it can be added to the list of tools one can implement to improve the robustness of their neural network.", "requested_changes": "I consider the points numbered **1** to **3** critical.\n\nAlso, here are minor comments:\n- Section 4: Please state from the beginning that the proposed \"loss\" terms are intended to be added to a typical empirical loss term in the overall training objective\n- For the reader's benefit, I suggest recalling close to Eq. (9) that $\\mathcal{P}_{ci}$ is defined by Eq. (6)\n- Section 7: There are some incorrect usages of the citation commands. e.g.:\n    - CIFAR-10 (Krizhevsky et al. (2009))  --> CIFAR-10 (Krizhevsky et al., 2009)\n    - ResNet18 He et al. (2016) --> ResNet18 (He et al., 2016) \n- Section 7.1; paragraph \"Neural network models': It is unclear if the \"backbone networks\" (ResNet18 and MobileNetV3) are frozen or fine-tuned during the experiments. The sentence segment \"followed by some fully connected layers\" could be replaced with an exact statement.\n\nLast but not least:\n- The references section needs to be proofread. Some published peer-reviewed works are cited as arXiv preprints (e.g., Foret et al. is an ICLR 2021). Conferences are sometimes referred to by their full names and sometimes by their abbreviations (e.g., NeurIPS and Advance in neural information processing systems).", "broader_impact_concerns": "-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "14JbTQuZrB", "reviewer": "Reviewer_nSWX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work studies the robustness of a classification model.\n\nFirst, the authors propose a loss function that operates in the feature space. This loss function encourages features belonging to the same class to be compact, while pushing features from different classes farther apart. The authors also propose injecting Gaussian noise into the data (similar to adversarial training) to further enhance the model’s robustness. This method consistently improves over baseline methods.\n\nSecond, the authors investigate the relationship between robustness under Gaussian noise and the curvature of the input loss. Roughly speaking, they show that robustness is equivalent to having a flat input loss landscape. The authors also conduct numerical experiments to verify this theoretical insight.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I have read the main paper but not the appendix. Most of the claims made are well supported by clear evidence, either theoretically or empirically.\n\nBelow are a few claims that I found unconvincing:\n\n- **Discussion in Section 5.2:** The authors theoretically justify that using one of their loss functions, either intra-class compactness or inter-class separability, is insufficient to ensure robustness. Hence, both loss functions are necessary. At the end of the section, it is stated that \"*Thus, by maximizing the class margin and minimizing the class dispersion simultaneous, our loss guarantees that transformations along opposite directions cannot occur. That is, our loss can avoid a solution that does not improve the generalization error of the model.*\"\n\n    I think this claim is overstated. Corollary 5.1 only shows that both loss functions must be used at the same time, but it does not imply that using them together will always prevent solutions that fail to improve the model’s generalization error. I may have overlooked something, so please correct me if I am mistaken. \n\n- **Discussion on Theorem 6.1 (Upper-bound and lower-bound of the input loss curvature):** The authors establish a connection between the input loss curvature and the model’s stability. It is stated that \"*In Eq. (16), if we assume that the gradient of the loss at $x$ is sufficiently small, combining with the fact that $\\sigma$ is generally small, we then have $\\sigma^2\\lVert \\nabla l(x)\\rVert^2\\approx 0$. In this case,  $ 2\\sum_i\\lambda_i^2 + \\left( \\sum_i |\\lambda_i| \\right)^2 \\gtrsim \\frac{4(1-\\eta)}{\\sigma^4}(l_{\\text{out}})-l(x))^2$.*\"\n\n   In fact, to obtain this last inequality, what we need is $(1-\\eta)(l_{\\text{out}}-l(x))^2 \\gtrsim \\sigma^2\\lVert \\nabla l(x)\\rVert^2$. Should we expect this condition to hold? As far as I understand, in the idealized setting, the three terms $1-\\eta$, $l_{\\text{out}}-l(x)$ and $\\lVert \\nabla l(x)\\rVert$ are all small, so it’s unclear to me whether the inequality $(1-\\eta)(l_{\\text{out}}-l(x))^2 \\gtrsim \\sigma^2\\lVert \\nabla l(x)\\rVert^2$ is actually satisfied.\n\n- **Theorem 6.3:** I am surprised that the result does not depend on the layer index $l$. Intuitively, I would expect the lower bound to increase with $l$. Is such a dependence hidden somewhere? \n\nMinor comments on the experiments:\n- In the experiments in Section 7.2, the highest noise level used for testing (in Table 1, Table 2, and Table 3) is roughly equal to the noise level used during training. I would be interested in seeing what happens when the test noise level exceeds the train noise level, as this could further demonstrate the model’s robustness under noise misspecification.", "audience_yn": "Yes", "audience_explanation": "Yes. Robustness is an important aspect of neural network models, and this paper provides valuable techniques and insights.", "requested_changes": "I would appreciate it if the authors could clarify the claims that I found unconvincing above.", "broader_impact_concerns": "There are no ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "M0uUF0hLew", "reviewer": "Reviewer_hc69", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "With the intent of improving feature discriminativeness and robustness against input noise, the authors of \"Training More Robust Classification Model via Discriminative Loss and Gaussian Noise Injection\" discuss a loss function that combines several existing loss formulations and apply it to the penultimate layer of a neural network. Their loss function specifically addresses the issues of **class dispersion, class margins and model robustness** against input feature noise. Fundamentally, it encourages the network to map embeddings of a class into the vicinity of its corresponding class centroid in the embedding space. Assuming optimal conditions (e.g., a loss being zero), the authors analyze theoretical properties of the neural network induced by their loss functions. Thereby, they derive an upper bound for class dispersions in the embedding space and a lower bound for class margins, both expressable in terms of loss hyperparameters. The authors further argue that optimizing solely for either minimal class dispersions or maximal class margins is a suboptimal, and they formalize a link between input loss curvature and  feature- as well as loss stability under Gaussian input noise. The theoretical part of this work concludes with a discussion about generalization properties and the impact of the loss on feature representations prior to the penultimate layer. The experimental part empirically validates the theoretical findings: 3 datasets (CIFAR-10, Street View House Numbers (SVHN) and a self-created dataset referred to as \"Road Condition Image Dataset\") are used to demonstrate performance improvements towards 4 baseline methods, to experimentally show the relation of model performance to input loss curvature and finally to visualize the effects on intermediate feature representations.\n\n- **Strengths:**\n\t- *S1:* The proposed loss function allows for in-depth theoretical research on lower- and upper bounds for embedding space projections in terms of loss hyperparameters, and for establishing theoretical connections between model performance and input loss curvature.\n\t- *S2:* The structure of the theoretical part of this work allows for targeted experiment designs that precisely analyze theoretical claims.\n\t- *S3:* The experiment results validate claims anticipated by the theory.\n\t- *S4:* Experiment code has been provided.\n- **Weaknesses**:\n\t- *W1:* The overall readability of the paper requires improvement. Detailed points are provided in the additional comments.\n\t- *W2:* The following references do not exist:\n\t\t- Rong Jin, Chiyuan Zhang, Cho-Jui Zhang, Qiang Tang, and Le Song. Input hessian regularization for adversarial robustness. In NeurIPS Workshop on Robust Machine Learning, 2019.\n\t\t- Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. A second-order approach to adversarial attacks and defenses. arXiv preprint arXiv:1902.02864, 2019a.\n\t- *W3:* Proposition 5.3 claims more than the paper proves:\n\t\t- Proposition 5.3 claims to be applicable to fully-connected layers or to arbitrary convolution layers.\n\t\t- The proof only considers fully connected layers or image convolutions. In case the term \"convolution\" should be kept, it needs to be explicitly specified what sort of convolutions are considered. E.g., the definition of an image convolution is different from that of a graph-convolution.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors pose three core claims:\n- The defined loss function shall enforce **intra-class compactness**.\n    - The authors derive theoretical evidence such as Proposition 5.1 or Theorem 6.2.  Additionally, t-SNE visualizations (Figures 11 and 12) illustrate compact clusters obtained using the proposed method.\n- The loss function shall increase the **margin of embeddings to decision boundaries**.\n    - To show this, the authors theoretically derive a lower bound for the margin between class centroids (Proposition 5.2). Experimentally, Section 7.2 shows that classifiers typically achieve higher test accuracies when trained with their loss function compared to other baseline methods, validating improved decision boundaries.\n- The loss function shall enhance **model robustness**.\n    - In Section 4.4 the authors discuss how the computation of class centroids can be stabilized across batches, that may have fluctuating noise ratios in practice, by introducing momentum terms.\n    - Furthermore, Section 6.1 examines feature- and loss stability, relating these properties to upper and lower bounds for input loss curvature (Theorem 6.1), which in turn influences to classifier performance. Experimentally, the authors provide empirical evidence for this relation in Figures 7 and 8.\n\nOverall, the author provide ample convincing theoretical and empiricial evidence in support of their core claims.", "audience_yn": "Yes", "audience_explanation": "The fundamental concepts underlying contrastive learning frameworks, i.e., learning class prototypes, maximizing intra-class compactness and inter-class margins, do not represent new ideas for training classifiers [4, 5, 6]. Likewise, the visual or theoretical analysis of how model robustness relates to hessian matrices has been addressed before [1, 2, 3]. However, this manuscript relates the three topics: (a) contrastive learning, (b) training with Gaussian input noise and (c) analysis of input loss curvature into one coherent framework. This unified perspective enables the authors to derive - to the best of the reviewer's knowledge - new theoretical insights, such as lower and upper bounds for loss curvature in terms of formerly defined hyperparameters for the contrastive learning framework.\n\nGiven the comprehensive nature of this work and its theoretical contributions, the work has the potential to spark interest in TMLR's audience.\n\n\n\n[1]: Li, Hao, et al. \"Visualizing the loss landscape of neural nets.\" _Advances in neural information processing systems_ 31 (2018).\n\n[2]: Mustafa, Waleed, Robert A. Vandermeulen, and Marius Kloft. \"Input hessian regularization of neural networks.\" _arXiv preprint arXiv:2009.06571_ (2020).\n\n[3]: Dinh, Laurent, et al. \"Sharp minima can generalize for deep nets.\" _International Conference on Machine Learning_. PMLR, 2017\n\n[4]: Yang, Hong-Ming, et al. \"Robust classification with convolutional prototype learning.\" _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2018.\n\n[5]: Wen, Yandong, et al. \"A discriminative feature learning approach for deep face recognition.\" _European conference on computer vision_. Cham: Springer International Publishing, 2016.\n\n[6]: Gupta, Rohit, et al. \"Class prototypes based contrastive learning for classifying multi-label and fine-grained educational videos.\" _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 2023.", "requested_changes": "My requested changes mainly evolve around the mentioned weaknesses:\n1. Please address the concern regarding the references that cannot be found.\n2. Please improve the readability of the paper by clarifying the aspects noted in the additional comments.\n3. Please either revise the proof or the claims made by Proposition 5.3.", "broader_impact_concerns": "None.", "additional_comments": "**Aspects in the paper that require clarification:**\n1. The work makes three core claims: the proposed loss function shall (a) enforce intra-class compactness (b) increase class margins and (c) increase model robustness. While the manuscript investigates these claims in great detail, it includes a few digressions. E.g., Sections 5.2 or 6.2 could be moved to the appendix.\n2. About Equation 13:\n\t- This equation describes the overall loss proposed by the authors. It should be pointed out that this loss requires at least two forward passes, given that $L_{\\text{clean}}$ requires clean- and $L_{\\text{noisy}}$ requires noisy samples.\n3. About Proposition 5.3: \n\t- $T_\\alpha$ is **defined** as a linear shift and thus assumed to be present. Conversely, the change of network parameters is not necessarily something that has been observed as a training dynamic. This should be emphasized in Section 5.2.\n\t- It remains questionable whether the addressed methods in the second paragraph of Section 5.2 really cause a linear shift $T_\\alpha$ of the model parameters as assumed by the proof.\n4. About Theorem 6.3:\n\t- The authors write: \"That is, in the intermediate layers, features of different classes become more separated from each other.\". This phrasing suggests that Theorem 6.3 implies an increasing inter-class distance across layers as experimentally visualized in Section 7.5. However, it merely describes the existence of a lower bound for the JSD between two class distributions at layer $l$. In particular, it does not guarantee increasing margins between class centroids across layers.\n5. About Section 5.2: \n\t- The analogy to Huygens Inertia Theorem seems to be far fetched. The authors should either choose an analogy more common to the ML-domain or introduce the Inertia Theorem sufficiently well.\n6. Equations 8 and 9:\n\t- These equations introduce the hyperparameter $\\delta_v$ and $\\delta_d$, which are repeatedly used throughout the manuscript. Their naming should more clearly convey their respective meanings.\n7. Below Equation 10 it reads: \"However, there are no guarantee that the decision boundaries lead to closed cells.\". \n\t- The manuscript should clarify what is understood as a \"closed cell\".\n8. Below Proposition 4.1 it reads: \"But at the same time, the centroids are kept consistent. That is, the parameters of the neural network evolve along training with sufficiently large gradients.\". \n\t- It is unclear why \"consistency\" describes the magnitude of gradients.\n9. At the end of Section 5.2 it reads: \"However, Corollary 5.1 tells that we cannot both expand the class margin and shrink the class dispersion simultaneously. This is because $\\Theta$ is moved in two exactly opposite directions for these two objectives. Thus, by maximizing the class margin and minimizing the class dispersion simultaneously, our loss guarantees that transformations along opposite directions cannot occur.\".\n\t- This phrasing makes the two sentences read like a contradiction and should be rephrased for clarity.\n10. The function class $H$ in Section 6.2 introduces a variety of of new parameters that introduce boundedness conditions. The paper should explain why these boundedness conditions are necessary.\n11. The paragraph before Definition 6.3 references $L_{\\text{clean}}$.  A reference to Equation 13 should be included for clarity.\n12. Section 6.2 starts to denote inputs to the model as capital X, while before it was written in lower case. The notation should be held consistent throughout the manuscript.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)