Label Smoothing is a Pragmatic Information Bottleneck
Authors: Sota Kudo
Venue: Accepted by TMLR ()

Abstract: This study revisits label smoothing via a form of information bottleneck. Under the assumption of sufficient model flexibility and no conflicting labels for the same input, we theoretically and experimentally demonstrate that the model output obtained through label smoothing explores the optimal solution of the information bottleneck. Based on this, label smoothing can be interpreted as a practical approach to the information bottleneck, enabling simple implementation. As an information bottleneck method, we experimentally show that label smoothing also exhibits the property of being insensitive to factors that do not contain information about the target, or to factors that provide no additional information about it when conditioned on another variable.

Published in Transactions on Machine Learning Research (07 (07/2025)
Label Smoothing is Pragmatic Information Bottleneck
Sota Kudo
Bolakudo9&@gmaaie com
Nara Institute of Science and Technology
Reviewed on OpenReview: http:://paaaaerre...aaa///ttttaa
Abstract
This study revisits label smoothing via form of information bottleneck. Under the assumption of sulficient. model flexibility and conflicting labels for the same input, we theoretically and experimentally demonstrate that the model output obtained through label smoothing explores the optimal solution of the information bottleneck. Based on this, label smoothing can be interpreted as practical approach to the information bottleneck, enabling simple implementation. As an information bottleneck method, we experimentally show that label smoothing also exhibits the property of being insensitive to factors that do not contain information about the target, or to factors that provide no additional information about it when conditioned on another variable.
HY
IB
Cross entropy
Label smoothing
A(M)
i(X;T)
Figure 1: Empirical IB curve and solutions for cross entropy and label smoothing loss under practically reasonable assumptions. Here, X is the input, Y is the target, and T is the model output. Theoretical results are shown in Section 42
1 Introduction
Label smoothing (LS) SZzeedyy al. 2015 is operation that softens hard one-hot vector by taking weighted average with another distribution, such a uniform onc. Despite its simplicity, it has been empirically shown to improve the performance of deep learning models and used to train SOTA models in various tasks such as vision (Zoph et al. 2018 Huang et al. 2018 Real et al. 2019 Wortsman et al. 2022 Liu et al.) 2022), speech recognition (Chorowski & Jaitly 20166) machine translation (Vaswani al. 2017
NLLB Team al. 2022), and multi-modal models (Yu al. (022). The specific effects of label smoothing have become active of research. It has been pointed that it reduces robustness against. adversarial

---

Published in Transactions on Machine Learning Research (07/2025)
attacks (Zantedeschi et al. 2017 improves calibration Miiller al. 2019 mitigates label noise
Lukasik
et al. 2020 Chen et al. 2020 Liu 2021 does not allow for sparse output distributions (Meister et al. 2020
accelerates the convergence of stochastic gradient descent (Xu et al. 2020), degrades selective classification
(Xia et al. 2024), and learns low-variance features [Chidambaram & Ge 2024]. h. In addition to these points,
label smoothing has also been shown to be a regularivation limiting model outpuls or representations Tt. can
be seen as penalizing overconfident (i.e., low-entropy) predictions (Pereyra al. 2017 Meister et al. 2020
Furthermore, it has been experimentally demonstrated that it compresses the internal representations of the model, such as the penultimate layer or the logits (Mäller al. 2019). These insights suggest connection between label smoothing and information-theorele regularization techniques for representations.
Parallel to these developments, the machine learning community has also been exploring the Information Bottleneck (IB) framework (Tishby et al., 2000), a method for supervised representation learning based on information theory. The objective of TB is compress the information about the input while preserving as much information as possible about the target labels. From the perspective of statistical learning theory, the compression known to function
form of regularization (Shamir et. al. 2010 Vera et al. 2018 2020
Kawaguchi et al. 2023). Furthermore, by compressing the input, the model is encouraged to ignore factors that do not contain information about the target (Achille & Soatto 2017), factors that do not provide any additional information about the target when conditioned on another variable (Ahuja et al. 2021 p, which do not directly cause the target, thereby enabling the construction of more reliable models. In addition to these theoretical discussions, several practical benefits applying IB methods to deep learning models have been reported, including robustness against adversarial attacks (Alemi et al.] 2016)) oft-of-distribution detection (Alemi et al., 2018: Pan al. 2020), domain generalization (Ahuja et al. 2021 Li et al. 2022)
and calibration (Alemi et al. 2018).
Considering these accumulated research findings on IB, studying the relationship between label smoothing and IB can significantly advance our understanding of the behavior of label smoothing SOTA models that utilize it, and open up new dircctions for its application. Motivated by this insight, this work revisits label smoothing through the lens of the Information Bottleneck principle. Our contributions are threefold:
We clarify the correspondence, similarities, and differences between label smoothing and the Vari- ational Information Bottleneck (Alemi et al.j 2016) (Section 4.1). Based on this correspondence, where the representation corresponds to the model output, we interpret label smoothing as a form of IB. Under the practically reasonable assumptions of model flexibility and the absence of multiple different labels for the same input, we theoretically show that label smoothing can explore the optimal solutions of IB with arbitrary compression levels within the range of interest (sce Figure I (Section
4.2
This suggests that label smoothing enables simple implementations and avoids the practical issues associated with IB (Section 4.3) by limiting the scenarios where its strictness as an IB holds to typical cases. Thus, we argue that "label smoothing pragmatic information bottleneck." "
We experimentally validate the theoretical findings and show that, conversely, in siluation where different labels exist for the same imput, label smoothing is not necessarily optimal from the perspective of IB (Section 5.2 This demonstrates not only the effectiveness of label smoothing, but also its limitations as an IB method in certain situations.
We experimentally demonstrate the effectiveness of label smoothing as IB, In particular, we consider muisance factors, which do not contain information about the target, and redundant factors, which do not provide additional information about the target when conditioned on another factor. We show experimentally that label smoothing makes the model less sensitive to these factors (Section 5.3). These results show cases where label smoothing enhances independence from factors that do not have a direct causal relationship with the target, which is an important property for building reliable models.
2

---

Published in Transactions on Machine Learning Research (07 (07/2025)
2
Related work
2.1 Similarities between Label Smoothing and the Information Bottleneck
Several studies have presented fragmentary findings suggesting a potential connection between label smoothing and the information bottleneck. Müller et al. (2019) experimentally demonstrated that applying label smoothing reduces the mutual information between the input data and output logits, and mentioned its potential relationship to the information bottleneck. Furthermore, their visualization of penultimate layer representations showed that label smoothing causes data points cluster around the centers of their respective classes. This behavior is consistent with the fact that the deterministic IB (Strouse & Schwab, 2016) favors hard clustering (Kolchinsky et al.) 2018 Such intra-class information compression explains the degradation in transferability (Kornblith et al. 2020) caused by label smoothing. Thcorctically, connections between label smoothing and confidence penalty have also been pointed out (Pereyra et al. 2017 Meister el al. 2020). These
regularization constrain the expressive capacity of the model's output, implicitly suggesting a relationship with the information bottleneck. Based on this connection, the Variational Information Bottlenock paper Alemi et al., 2016) introduces the confidence penalty as a related work, although it does not offer a detailed discussion on the correspondence, similarities, far as know, there has been no research that investigates the optimality of label smoothing as an information bottleneck method or attempts to understand its characteristies from the perspective of the information bottleneck.
differences between them. While these results suggest.
potential connection,
2.2 Variants of Label Smoothing
As mentioned above, uniform label smoothing (Szegedy et al.) 2015) is used in SOTA models due to its simplicity and ease of handling, and it remains an imporlant technique in practical applications Based on this, in this study, as a first step in investigating the relationship between label smoothing and IB, uniform label smoothing is selected as the subject of analysis. In particular, in the theoretical analysis, the smoothing distribution and smoothing strength are considered fixed across classes, data samples, and the training phase. In the experiments, the smoothing distribution is set as uniform distribution. Meanwhile, in recent. years, label smoothing methods that allow for more flexibility in these fixed settings have been proposed. For example, Online Label Smoothing (Zhang et al.) 2021 modifies the smoothing distribution during training based on the model's predictions. Structural Label Smoothing (Li et al 2020) applies adaptive smoothing strengths depending on regions in the feature space. Low-rank Adaptive Label Smoothing [Ghoshal let al. 2020) jointly learns the smoothing distribution and the model parameters. In natural language tasks, Adaptive Label Smoothing (Wang et al. 2021a) dynamically sets the smoothing distribution at each time step depending on the context. Furthermore, Yuan et al. (2019) show that Knowledge Distillation (Hinton ct al. 2015) can be interpreted as type of label smoothing. where the smoothing distribution defined per data sample based on the teacher model's output. The extension of this study to include such methods is left. for future work.
3 Preliminaries
3.1 Notations
Let X X and Y E y be random variables with joint distribution p(X,Y). Here, X and Y correspond to input variable and target variable, respectively. In this study, we consider classification scttings; thus, we set y = (1, ...,, This assumption is sometimes adopted and justified when using digital computers (Kawauuchi et al. 2023). We denote an i.i.d. sample of N instances as ((11,9)), (I2, U2),..., ., (æ,, 9N)), and its empirical distribution as ((X.Y). We use the hat symbol to indicate quantities calculated using p(X,Y) instead of p(X. Y), e.g., empirical mutual information f and empirical entropy A.
KY. K}, where K is the number of classes. For simplicity, X' is also restricted to be a finile set.
3

---

Published in Transactions on Machine Learning Research (07 (025)
3.2 Label Smoothing
We consider the parameter 0 E ④ to be optimized for modeling p(YIX) using po(T)X); where T = y represents the prediction. This notation is intended to seamlessly connect to the information bottleneck later. With cross-entropy defined as H(p(T).g(T)) == -EET p(t) log ((t,, the Cross Entropy loss (CE), which performs maximum likelihood estimation, can be written as
N
(CE == Σ Po
(1)
where 1y= is an indicator function, meaning that T equals ys with probability 1.
Label smoothing modifies this by softening the hard target 1r= More formally, label smoothing replaces it with a weighted average of the hard target and a smoothing distribution r(T).
(T) = (1 a)lr= + ar(T),
(2)
where a E [0, 1]. Generally, a uniform distribution is used as the smoothing distribution, and r(7) = * The loss function of label smoothing is
(1s(0;a) = N
Σ H (T))po(T(T))
(3)
In particular, (zs(0:0)= (C)(
3.3 Information Bottleneck
The Information Bottleneck (Tishby et al. 2000) aims transform X into a representation T E T that retains much information about Y as possible while compressing the information about X. Formally, under the assumption that p(X, Y) is known and using mutual information [(X;T) ELEX,ET p(x,t) (e,,)loo, log p(a.) this can be written as
((r)(())
marimize I(T:Y) s.t. T(X:T) Sr TY+=NT
(4)
where 2 0. Here, the constraint Y ++ X + T restricts T to be generated from X. We refer to Equation 4
as the IB objective. It is known that the solutions of the IB objective form generalization of the minimal
sufficient statistic, where Y(T;Y) controls sufficiency and [(X;T) controls minimality (Shamir et al. 2010 In the information bottleneck literature, the information plane, defined by the two axes I(X;7) and ((;;Y) is often considered. On the information plane, we can plot (I(X;T), [T;; Y)) = (r, ITT;; Y)) for cach r, where T, a solution of the IB objective for r. This plot forms a curve called the IB curve, which separates the feasible and infeasible regions.
In practice, usually do not know p(X, Y), so it is replaced with the empirical distribution p(X, Y). We can then draw the empirical IB curve on the information plane defined by the empirical mutual information. In this case, the constraint on i(; can be seen as a form of regularization. The generalization gap between the true [(T; Y) and the empirical Í(T; is bounded by a term related to I(X;T) (Shamir et al. 2010 The generalization gap for cross-entropy also bounded by I(X:T) (Vera et al. 2018 2020). IB-based statistical learning theory for deep learning has also been investigated (Kawaguchi et al. 2023
Here, we introduce two specific effects of the information bottleneck. These effects of label smoothing will be addressed in the experimental section. One of the benefits of compression is the removal of muisance. A nuisance is random variable that affects the input X but is independent of the target Y. The representation should be invariant to, or at least. less informative about, the nuisance, Achille & Soatto (2017), Proposition 3.1, shows that when the representation sufficient, the mutual information between the representation and the nuisance upper bounded by the mutual information between the representation and the input X.
Additionally, redundancy of representation can also arise from factors that become independent of the target when conditioned on another factor. We refer to these redundant factors and informative factors, respectively.
4

---

Published in Transactions on Machine Learning Research (07/2025)
For example, if a representation includes both an informative factor and redundant factor, removing the redundant factor allows for a more compressed representation without reducing the information about the target. Formally, X), where R and F are the redundant and informative factor respoctively. This relates to the benefit of IB the fully informative invariant features (FIIF) scenario discussed by [Auuja et al. (202T). Theorem 4, the field of domain generalization. The following is an example of case where the removal of redundant factors is effective. Beerr et al. (2018) trained a convolutional neural network to classify camels and cows, but it was found that the model relied on background colors (e.g., green for cows, brown for camels) rather than essential features such as the shape of the animals. As discussed in Ahuja et al. (202T), 1 labels are assigned based on the shape of the animal rather than the background, so while the background correlates with the labels, it becomes independent of the labels when conditioned on the shape of the animal. As in this example, it is possible to focus only on factors that are causally directly related through compression, which is an important property for building reliable models (c.g., models for unbiased diagnosis).
FY)) = Y(); + ([RYY))) = Y(F and (RR, FR F) = [(F + I (() >
One of the representative implementations of IB in deep learning is Variational Information Bottleneck (Variational IB) (Alemi et al., 2016), which will be introduced in the next section. This paper deals with
VIB, but note that there are also other methods for IB or IB-related objectives (Strouse & Schwab 2016
Kolchinsky et al. 2019 Pan et al. 2020 Piran et al. 2020
Fischer 2020
Yu et al.
2021
Wang et al. 2021b
Kudo et al. 2024a
3.3.1
Variational IB
Here we introduce Variational IB, which will later be connected to label smoothing. Instead of directly optimizing the constrained IB objective, the following loss function is minimized.
(
(5)
where 3 2 O. This is a Lagrangian relaxation (Lemaréchal 2001] of the IB objective, known as the IB Lagrangian. The IB Lagrangian is casier to optimize; however, there still exists a problem in that mutual information contains integrals that are intractable or difficult to compute. VIB enables the learning of the IB Lagrangian in general settings by providing representation T is obtained via the feature extractor px)T First, let us consider sufficiency, i.e., ((;Y), By using another model qs (Y(T), which call classifier, variational approximation of (T)) the following variational lower bound is obtained.
upper bound through variational approximation. Here, the
I(T;Y) 2 Ep(r, y( (T (o HYY x.y
(6)
Here, H(Y) represents the entropy of Y. Since this value remains constant throughout the learning process, it can be ignored during training (Poole et al., 2019). Next, for the minimality term I(X;T), an upper bound can be obtained by using r(T) as variational approximation of PT))
T(X:T)= <
Σ p(x) (RLIIo((()))I((T)).
(7)
where DEL is the Kullback-Leiin (KL) divergence. In practice, a fixed distribution is used for r(T). By combining these elements, an upper bound on the IB Lagrangian can be obtained. Using the empirical distribution for p(X,Y), the loss function of the VIB is dcrived.
1 N N
Lvin 0) =
E Epo go + 3D (lo
(8)
In particular, when the task reconstruction, the objective function above becomes that of the Variational Autoencoder (Kingma & Welling 2013).
The inequality is derived from the nen-nigativity of the Kullback-Leibler divergence between (Y(T) and (Y
2The inequality is derived from the non-negativity of the Bellback-Teeiblee divergence between (T)) and r(T).
5

---

Published in Transactions on Machine Learning Research (07 (07/2025)
4 Label Smoothing through the IB lens
In this section, theorctically revisit label smoothing from the perspective of the IB. In Section 4.1 clarily the correspondence, similarities, and differences between label smoothing and Variational IB. Based this correspondence, we interpret label smoothing an IB method. In particular, the model output corresponds to the IB representation. Next, in Section 4.2 we discuss the IB optimality of label smoothing. We consider the assumptions that the model sufficiently flexible and that there are no confficting labels for deep learning models, and the latter generally
the same input. The former is a reasonable assumption satisfied in benchmark datascts. Under these pragmatic settings, we show that label smoothing can fully explore the interesting range of the IB curve. Finally, in Section 4.3 we discuss the properties of label smoothing as an instance of the IB.
4.1 Label Smoothing and Variational IB
Here, we present the correspondence between label smoothing and the Variational IB, clearly illustrating their similarities and differences. In terms of related work, the relationship between label smoothing and the confidence penalty has already been clarified (Pereyra et al. 2017 Meister et al. 2020h. As far as we know, although an implicit relationship between the confidence penalty and Variational IB has been suggested AAlemi et al., 2016),, it has not been demonstrated that they are interchangeable through single, simple modification. In this section, aim to elucidate this relationship, thereby clarify the connection between label smoothing and Variational IB. As result, it becomes possible to recognize label smoothing IB method.
Label Smoothing
Confidence Penalty
Variational IB
Reversing DKL
Swapping log and E
Figure 2: Relationship between label smoothing and Variational IB
For the label smoothing model, let (Y = = be a fixed distribution, which connected Variational IB later Using this, we can rewrite the label smoothing loss as
(r
Σ (1 a)) - aDDr (xxx)))))))))(()))<><><))))))))) (aH(()))) N
im
(1 (og
o (DDrlll(T)) (po((())) const.
We use (2(2)[ = p(=) for the second equality. Note that Hr((T)) is constant and can be ignored.
Here, we introduce another well-known regularization method, the confidence penalty (Pereyra al. 2017 The confidence penalty is empirical method that penalizes the entropy of the output to avoid overconfidence of models. With the same model setting in label smoothing (Section 3.2). the loss function for the coufidence penalty is represented as
1 Σ - 3H 31
(o
(10)
Note that from the Variational IB perspective, (Y()) ly=r implies using the model's prediction, which is consistent with the practical usage lahel smoothing.

---

Published in Transactions on Machine Learning Research (07/2025)
where 3 2 0. As in label smoothing, by setting the distributions (Y ly=rr and r(T) K the
uniform distribution, it is represented as
N Σ
(cp(0:8) = N 2=1
Eo (Taaollooo))) (aDaaa/loool)))))xxxxxxxxxoool +
+ const.
(11)
Comparod to Equation 9 it is evident that reversing the direction of the KL divergence transforms the objective from label smoothing to the confidence penalty (Pereyra el al. 2017 Similarly, in comparison with the Variational IB objective (Fquation 8), swapping the order of the logarithm and expectation in the first term changes the objective from confidence penalty to Variational IB. These relations are summarized in Figure 2 These three methods can be interpreted qualitatively in the same way, despite their subtle differences. In other words, each method balances a term that increases 4o(y () for T following po(T) maa), and a term that brings po (x) closer to r(2).
Up to this point, we have seen that label smoothing is similar to the Variational IB when the IB representation is the model output and the classifier is fixed as an indicator function. Based on this similarity, we recognize label smoothing as an IB method through this correspondence.
4.2 On IB-optimality of Label Smoothing
In the following, we demonstrate that, regardless of the differences in objective functions and simplified model settings (i.e., 1-dimensional representation and fixed classifier), label smoothing can explore the optimal solution of the information bottleneck in practical settings. The results of this section are summarized in Figure ! which should be referred to as appropriate. Please refer to the Appendix for the proofs of the following propositions.
We consider the case where there are different labels for the same input. Formally, we state this assumption below.
Assumption 4.1 (No contradicting labels). For any pair of indices i,j E 21. N). if 2, Fj, then y: Vj-
If there is no overlap in the data samples of X, this assumption is always satisfied. This is typically the case in standard applications where |X| is sufficiently large. For example, general benchmark datasets such as the CIFAR datasets (Krizhevsky et al. 2009) or ImageNet (Deng et al. 2009) satisfy this assumption. On the other hand, note that some specific applications do not meet this assumption, c.g., cases where multiple doctors provide diagnoses for the same subjects. Under this assumption, we can draw the IB curve as discussed in Kolchinsky et al. (2018).
Proposition 4.2 (Empirical 1B curve (Kolchinsky et al. 2018 ). Under Assumption 4.1 the empirical IB curve given by ((;YY) T():T) for I(X;T) E [0, (Y }}, and î(T;Y) = (YY for I(X;T) fIYY).
We usually not interested in the region where f(X:T) > HY). For example, from the perspective of generalization, if iT; Y) is the same, smaller ((X:T) is preferable.
Below, we derive the optimal solutions for cross-entropy loss or label smoothing loss in the information plane and compare them with the IB curve. Here, we assume the model is flexible enough to represent any probability mass function for all unique data points. When the model consists of deep neural networks, this assumption is accepted either empirically or theorctically (Hornik et al. 1989). Formally, this is stated as follows.
Assumption 4.3 (On model flexibility). Let I (a, 1, 2,,,. N) be a set of unique input data points. We assume that the model is flexible enough to represent
(p)(TT | θ) A
(12)
where 47 Tp:T [0, EieT =1] denotes the probability simplex over T4
'(-)Zz denotes tuple of length [Z], where each component is defined by the corresponding E To define it, implicitly assume a fixed ordering of
7

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Note that T= (1, for the cross entropy or label smoothing model, set in Section 3.2 With this assumption, the optimal solution for the cross entropy loss shows empirical minimal sufficiency, i.e, it retains minimal information about the input while containing maximal information about the target in the empirical distribution.
Proposition 4.4 (Cross entropy results in empirical minimal sufficiency.). Under Assumption 4.1 and
Assumplion 4.3 all represenlations T oblained by oplämizing the cross entropy loss (ox(() salisfy i (T;Y)= I(X;T)= HY).
Note that empirical minimal sufficiency does not necessarily lead to the best outcome. As shown in Shamir et al. (2110), compressing TT) can reduce the generalization gap on (T"; Y); thus, it is often possible to achieve a higher Y) with a smaller ((;; T). This explains the necessity of sweeping the IB curve. Next, we demonstrate that the optimal solution of label smoothing can explore the IB curve.
Proposition 4.5 (Label smoothing sweeps empirical IB curve.). Under Assumplions representation Ta obtained by optimizing the label smoothing loss (I)(0;a) for O S S sweeps the line defined by İ(T;Y) = [(X;T), where [(x;T) E [0, (Y)].
4.1 and 4.3 the
4.3 Properties of Label Smoothing as an IB
It has been demonstrated that label smoothing can explore the optimal solution of IB in practical settings. Below, we discuss the following three properties of label smoothing as an IB method.
Label smoothing can be implemented with simple operation that only transforms the labels, without requiring any changes to the model or the loss function. Due to its simplicity and compatibility with other methods, it is a practical and easy-to-use Information Bottleneck method.
Label smoothing adopts a one- dieeeninnii discrete variable for the representation. While this may seem like an excessive simplification at first glance, it still enables the optimization of IB in practical scenarios. Thanks to this simplification, it avoids the challenging problem of estimating mutual information in high dimensions (Poole et al. 2019). On the other hand, unlike other IB methods, it does not have the flexibility to compress arbilrary layers.
Under conditions such as Assumption 4.1 is known that trivial solutions exist for the IB objective with any level of compression (Kolchinsky et al. 2018 On the other hand, label smoothing explicitly specifies how to perform the compressior thereby avoiding trivial solutions.
In summary, label smoothing can be considered a pragmatic approach to IB, as it achieves IB optimality under practical settings while offering a simple implementation and avoiding the practical issues associated with IB. While demonstrating implementation advantages and addressing potential concerns, this study does not claim that label smoothing outperforms other IB methods in terms of performance. For instance, there are cases where VIB has been shown to outperform label smoothing in terms of generalization performance
(Alemi et al. 2016).
5 Experiments
In this section, we first describe the experimental setup, then discuss the IB optimality of label smoothing, and finally show that the concrete effects of compression can also be observed in label smoothing.
Note that this study does not claim that label smoothing outperforms other IB methods in terms of performance. In fact, there is result showing that VIB achieves better generalization than label smoothing (Alemi et al. 2016). Therefore, instead of comparing performance with other IB methods, we focus here on investigating the qualitative behavior observed in label smoothing as an IB method.
*Its example is the likely to output 0 due to compression. Whether the solution is trivial can be determined, for example, by whether it is effective for generalization. Since this example completely loscs information with certain probability, it expected to negatively affect generalization. On the other hand, the effectiveness of label smoothing has been empirically validated. 6The representation T is explicitly defined by the designed (T)-
Ta in the proof of Proposition 4.2 variable representing either Y or 0, which becomes more
8

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
5.1 Experimental setup
We conduct experiments with four datasets: CIFAR-10 Krizhevsky et al. 2009,, Flowers- (Nilsback & Zisserman 2008), Occluded CIFAR (Achille & Soatto 2018). and a variant of Cluttered MNIST (Mnih et al. 2014). Our learning setup is based on established standard settings. However, to investigate the effects of label smoothing, we have not employed certain other regularization techniques that could potentially interfere with this objective. For the CTFAR-10 and Occluded CTFAR datasets, we adopt a standard training setup with ResNet (He al. 2015), while weight decay is removed. The training lasts for 160 epochs, with initial learning rate of 0.1, which is multiplied by 0.1 at epochs 80 and 120. The model architecture is ResNet-56, and the optimizer is SGD with momentum 0.9. The setup for Cluttered MNIST is the same above, but the architecture used is ResNet-20. For the Flowers- datasct, we adopt the training settings used in Hassani et al. (2021), while removing auto-augmentation, mixup, and cutmix. The model used is the Compact Convolutional Transformer (CCT-7/7x2) (Hassani et al. 2021), which is a hybrid CNN and Transformer. Also, note that throughout all experiments this study, the smoothing distribution is set to the most basic uniform distribution. In addition, note that since a onc-dimensional discrete variable used T, the empirical mutual information value calculated as exact. value rather than estimate.
5.2 On IB-optimality of Label Smoothing
Here, we experimentally examine whether label smoothing results in IB-optimal solution in two situation; the case with or without contradicting labels.
5.2.1 The case without contradicting labels
First, we consider the case without contradicting labels. Tn this case, have theoretically proven the solution sweep the IB curve. While considering the assumption on model flexibility is obviously satisfied with deep learning, the thcoretical result will be obviously obtained by experiments, however, experimental confirmation with practical setup still important. For this objective, we train models with CIFAR-10 and high resolution, more practical Flowers- dataset, changing {0.0, 0.1..., 0.9) in label smoothing. The model is CNN-based ResNet-56 for CIFAR-10 and CNNN-Transformer-yybbr CCT-7/7SS for Flowers-102. Figure g shows the empirical information plane of training set for each dataset. confirm the theoretical results, where all solutions are in line with the IB curve and cross entropy results in the minimal sufficiency.
CIFAR-10
Flowers-102
2.3
4.6
1(T: Y
curve
LS
CE
0.0
0.0
0.0
2.3
0.0
4.6
T(::T)
i(x: T)
Figure 3: Empirical IB curve and models trained with cross entropy label smoothing loss. The training set is used for the plot.
5.2.2 The case with contradicting labels
In the case without contradicting labels, we have theoretically and experimentally shown that label smoothing sweeps the IB curve. Here, we consider the case with contradicting labels, where multiple different labels exist for the same input. In this scenario, we show that there solutions that are better in terms of IB than the
9

---

Published in Transactions on Machine Learning Research (07 (07/2025)
optimal solution of label smoothing. We use the CIFAR-1OH dataset (Peterson et al. 2019) which provides multiple labels for each of the 10,000 images in the CIFAR-10 test set. We train a 10000 x 10 dimensional matrix followed by softmax function to optimize the 1B Lagrangian with 8 E {0.0, 0.1,... ....,........... 1.0). Here, the (1,j) component of this matrix corresponds to p(T = X = Iz), where 2; represents the index of a unique image. This optimization seeks the IB-optimal solution of the empirical distribution under the assumption of model flexibility (Assumption 4.3). The training conducted using the Adam optimizer. Figure f compares the obtained solutions with the theoretical optimal solutions of label smoothing. Note that the empirical mutual information values represented in the figure exact values, not estimated values. The results show that there exist feasible solutions better than label smoothing, indicating that label smoothing with uniform smoothing distribution is not necessarily IB-optimal in this scenario. Intuitively, this is likely due to the fact that the uniform smoothing distribution does not take inter-class similarity into account. The analysis of other smoothing distributions and the design of a smoothing distribution that is IB-optimal in this setting are left for future rescarch.
2.0
DPI bound
Lagrangian
LS (theory)
i(T: Y)
1.6
1.6
2.0
i(::T)
Figure 4: Theoretical solutions of label smoothing and solutions obtained by optimizing the IB Lagrangian. Each axis is calculated using the entire CIFAR-1OH dataset, which is interpreted as the training set. The DPI bound is given by the data processing incquality, and it is impossible to exceed this bound under any data distribution.
5.3
On effects of compression in label smoothing
So far, we have shown that label smoothing serves as an information bottleneck method for the model's output. Additionally, Müller et al. (2019) visually and quantitatively demonstrated through experiments that label smoothing causes compression in the model's penultimate layer and logits. Summarizing these findings, label smoothing applies IB-optimal compression the model's output while also compressing the internal representations to some extent. Here, we investigate the information bottleneck effect of label smoothing by examining the internal representations, which better capture the model's functionality. Specifically, we investigate two major effects of compression: the removal of nuisance or redundant factors, which is introduced in Section 33
5.3.1 Insensitivity to nuisance factors
Achille & Soatto (2088) uses the Occluded CIFAR dataset to demonstrate the nuisance removal effect of their model, which is equivalent to VIB. We show that label smoothing has a similar effect the same manner. As shown Figure 5 Occluded CIFAR contains CIFAR-10 images occluded by randomly selected MNIST images. In this case, the MNIST image is obviously the nuisance factor. We train models with or without label smoothing to classify the CIFAR-10 labels. We investigate how much information about the MNIST image is contained in the activations of the penultimate layer" of these models. Figure D shows the accuracy CIFAR-10 in the original model and on MNIST when new models are trained to classify MNIST
7hhis corresponds the input of the final fully connected layer.
10

---

Published in Transactions on Machine Learning Research (07 (07/2025)
labels from the activations of the penultimate layers. By applying label smoothing, the information about MNIST images is significantly removed, indicating its effect the insensitivity to nuisance factors. Tn this experiment, no significant accuracy improvement is observed from label smoothing on CIFAR-10. This may be because, for all a values, the learning setting optimized for non-label-smoothing models is consistently applied, masking its effectiveness.
CIFAR-14 10
88
Accuracy
86
84
0.00
0.05
0.10
0.15
0.20
MNIST
Accuracy
12.0
11.5
0.00
0.05
0.10
0.15
0.20
a
Figure 5: Occluded CIFAR Datasel
Figure 6: The lest accuracy for CIFAR-10 labels, and that for MNIST labels when another model is trained to classify MNIST labels from the penultimate layer of the models. The standard deviation over trials is also shown.
5.3.2 Insensitivity redundant factors
Since the redundant factor correlates with the target, it is difficult to quantitatively evaluate through the same type of experiment as above. Instead, we demonstrate this effect by visualizing the attention regions of the trained model. We create a variant of Cluttered MNIST. In this dataset, an original MNIST image and its randomly cropped versions "cluttered image." Note that in this study, the cluttered image is created from the original image used, whereas the original dataset creates it from other MNIST images rather than the one used as the original image. Consequently, our cluttered image contains information about the target. Given that the cluttered image is generated only from the original image in our Cluttered MNIST, Y ++ + is satisfied, where and are the original image and the cluttered image, respectively. Thus, the cluttered image (redundant. factor) is independent of the target Y when conditioned on the original image (informative factor). The new original image and 12 of its cropped versions. Figure 7 shows three
composed into new image. We refer to the overlapping cropped images
images of size 60 60 and contain randomly selected input images in test set and their saliency maps produced by Grad-CAM (Selvaraju et al. 2017) for models trained with cross entropy or label smoothing (a = 0.1). While the cross entropy model shows allention on the cluttered image, the label smoothing model is sensitive only to the original image. This indicates label smoothing's ability to remove redundant factors. Saliency maps for 20 randomly selected images each of the three experimental trials are shown Figure which discussod the Appendix [A.4 and lead the same conclusion. The accuracy of the cross entropy model 97. 97.99%, and that of the label smoothing model is 98.05%.
11

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Input
Cross Entropy
Label Smoothing
g:4
4 1 +
Figure 7: Images of the variant of the Cluttered MNTST dataset and the corresponding saliency maps by Grad-CAM for the cross entropy label smoothing model.
6 Conclusion and future work
In this study, we theoretically demonstrated that label smoothing explores IB optimal solution in practical settings. Furthermore, we discussed that is a method that offers simple implementation while avoiding the practical challenges of IB. As result, label smoothing can be regarded as practical approach IB. Additionally, experiments showed that label smoothing exhibits specific effects of IB, such removing nuisance factors and redundant factors.
Finally, we present several interesting topics for future research:
Insensitivity redundant factors may enable applications toward unbiased models and domain generalization. Further research on these applications of label smoothing is an intriguing direction.
In this study, we theoretically demonstrated that label smoothing leads to IB-optimal compression of the model' s output. The experimental results so far indicate that the model compresses its internal representations. Here, in principle, it seems possible for the model to retain redundancy in internal layers while compressing the output. This suggests a theoretical gap between output compression and internal representation compression. Further discussion considering the behavior of optimization needed to address this point.
We show that when labels are not unique for some inputs, uniform label smoothing is not necessarily optimal for the IB objective. This situation arises when multiple annotators label the same (especially ambiguous) input, and it also exists in certain practical scenarios. Analysis of other smoothing distributions and the design label smoothing mothods that achieve IB-optimality in such cases are left for future work.
Techniques that modify labels to improve accuracy, such as various label smoothing methods and knowledge distillation, have become indispensable in modern deep learning. While this study focused
12

---

Published in Transactions on Machine Learning Research (07/2025)
on basic label smoothing, it would be interesting to examine more sophisticated methods from the viewpoint of information theory and IB.
While label smoothing leads to IB optimality and demonstrates similar effects in several aspects, including our experimental results, it shows different outcomes from other IB methods regarding robustness to adversarial attacks. This aspect also leaves room for further discussion.
References
A Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. International Conference on Internet Technology and Applications, 19(50):1-9, June 2017.
Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal reprosentations through noisy computation. IEEE Trans. Pattern Anal. Mach. Intell., (0(12))28772200) December 2018.
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Jean- Mean-Christopha Gagnon-Audet, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of-distribution generalization. Adv. Neural Inf. Process. Syst., 14:3438-3450, 2021.
Alexander A Alemi, lan Fischer, Joshua V Dillon, and Kevin Murphy, Deep variational information bottleneck. arXiv LCs, December 2016.
Alexander A Alemi, Tan Fischer, and Joshua V Dillon. Uncertainty in the variational information bottleneck. arXiv Lcs.LG], July 2018.
Sara Meghan Beery, Grant Van Horn, and Perona. Recognition in terra incognita. ECCV, abss/1007.049f 456 473, July 2018.
Blair Chen, Liu Ziyin, Zihao Wang, and Paul Pu Liang. An investigation of how label smoothing affects generalization. arXio [cs, October 2020.
Muthu Chidambaram and Rong Ge. For better or for worse? learning minimum variance features with label augmentation. In The Thirteenth International Conference on Learning Representations, October 2024.
Jan Chorowski and Navdeep Jaitly. Towards better decoding and language model integration sequence to sequence models. arXiv NEJ, December 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 288-255. IEEE, June 2009.
lan Fischer. The conditional entropy bottleneck. arXiv [cs.LG], February 2020.
Asish Ghoshal, Xilun Chen, Sonal Gupta, Luke Zettlemoyer, and Yashar Mehdad. Learning better struc- tured representations using low-rank adaptive label smoothing. In International Conference on Learning Representations, October 2020.
Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, and Humphrey Shi. Escaping the big data paradigm with compact transformers. 2021. URL httpp://arrdv..or//a///000
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv fes. CVJ, December 2015.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXio Astat.MI, March 2015.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural Netw., 2(5):359 366, January 1989.
13

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Yanping Huang, Yonglong Cheng, Dehao Chen, Hyoukjoong Lee, Jiquan Ngiam, Quoc V Le, and Z Chen. GPipe: Efficient training of giant neural networks using pipeline parallelism. Neural Inf Process Syst, 32: 103 112, November 2018.
Kenji Kawaguchi, Zhun Deng, Xu Ji, and Jiaoyang Huang. How does information bottleneck help deep learning? arXie Lcs, pp.
May 2023.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv [stat.ML) December 2013.
Artemy Kolchinsky, Brendan D Tracey, and Steven Van Kuyk. Caveats for information bottleneck deterministic scenarios. arXiv (stat.MJ/: August 2018.
Artemy Kolchinsky, Brendan D Tracey, and David H Wolpert. Nonlinear information bottleneck. Entropy, 21 (12):1181, November 2019.
Simon Kornblith, Ting Chen, Honglak Lee, and Mohammad Norouzi. Why do better loss functions lead to less transferable features? arXiv [cs.CV], October 2020.
Alex Krizhevsky, Geoffrey Hinton, and Others. Learning multiple layers of features from tiny images. 2009.
Sota Kudo, N Ono, Shigehiko Kanaya, and Ming Huang. Flexible variational information bottleneck: Achieving diverse compression with a single training. ArXiv, abs/2402.01233 February 2024a.
Sota Kudo, Naoaki Ono, Shigehiko Kanaya, and Ming Huang. Exploring the trade-off in the variational information bottleneck for regression with a single training run. Entropy (Basel), 26(12):1043, November 2024b.
Claude Lemaréchal. Lagrangian relaxation. In Lecture Notes in Computer Science, Lecture notes in computer science, pp. 112-156. Springer Berlin Heidelberg, Berlin, Heidelberg, 2001.
Bo Li, Yifei Shen, Yezhen Wang, Wenzhen Zhu, Colorado Reed, Dongsheng Li, Kurt Keutzer, and Han Zhao. Invariant information bottleneck for domain generalization. Proc. Conf. AAAI Artif. Intell., 36(7): 7399-7407, June 2022.
Weizhi Li, Gautam Dasarathy, and Visar Berisha. Regularization via structural label smoothing. AISTATS, 108:1453-1463, January 2020.
Yang Liu. Understanding instance-lovel label noise: Disparate impacts and treatments. ICML, 139:6725-6735, February 2021.
Zhuang Liu, Hanzi Mao, Chao Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. arXiv fes.C CVJ, January 2022.
M Lukasik, Srinadh Bhojanapalli, Menon, and Surinder Kumar. Does label smoothing mitigate label noise? ICML, 0ts./2003.00219.64455 6458, March 2020.
Clara Meister, Elizabeth Salesky, and Ryan Cotterell. Generalized entropy regularization or: There's nothing special about label smoothing. arXiv fcs. CLI, May 2020.
Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukeuoghu. Recurrent models of visual attention. arXiv LCs, June 2014.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Neural Inf Process Syst, abs/ 029//9996.0622,, June 2019.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classos.
In 2008 Sizth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, December 2008
14

---

Published in Transactions on Machine Learning Research (07/ (025))
NLLB Team, Marta R Costa-jusså, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Flahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrott, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Kochn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. No language left behind: Scaling human-centered machine translation. arXiv fes. CL), July 2022.
Ziqi Pan, Li Niu, Jianfu Zhang, and Liqing Zhang. Disentangled information bottleneck. ar les. LGJ, December 2020.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv [es.NEJ, January 2017.
Joshua C Peterson, Ruairidh M Battleday, Thomas T. Griffiths, and Olga. Russakovsky. Human uncertainty makes classification more robust. arXiv [cs.CV], August 2019.
Zoe Piran, Ravid Shwartz-Ziv, and Naftali Tishby. The dual information bottleneck. arXiv (es.IT], June 2020.
Ben Poole, Sherjil Ozair, Aäron van den Oord, Alexander A Alemi, and G Tucker. On variational bounds of mutual information. JCML,
May 2019.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier archilecture search. Proc. Comf. AAAT Arlif. Intell., 33(01):4780 97800-9978-4778 July 2019.
Ramprasaath Selvaraju, Michael Cogswell, Abhishck Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Ohad Shamir, Sivan Sabato, and Naftali Tishby, Learning and generalization with the information bottleneck. Theor. Comput. Sci., (2)()))2666-227))) June 2010.
D J Strouse and David J Schwab. The deterministic information bottleneck. arXiv [q-bio.NN), April 2016.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. arXiv fes.CV) December 2015.
Thomas M. Cover and Joy A Thomas. Elements of Information Theory. John Wiley & Sons, Nashville, TN, 2 edition, June 2006.
Naftali Tishby, Fernando C Percira, and William Bialek. The information bottleneck method. arXiv [physics. [physics.data-anj, daja-an/,
April 2000.
Ashish Vaswani, Noam M Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Neural Inf Process Syst, 30:5998 6008, June 2017.
Matias Vera, Pablo Piantanida, and Leonardo Rey Vega. The role of the information bottleneck represen- tation learning. In 2018 IEEF, International Symposium on Information Theory (ISIT), pp. 1580-1584. IEEE, June 2018
Matias Vera, Leonardo Rey Vega, and Pablo Piantanida. The role of mutual information in variational classifiers. arXiv [stat. October 2020.
Yida Wang, Yinhe Zheng, Yong Jiang, and Minlie Huang. Diversifying dialog generation via adaptive label smoothing, arXiv fes. CLJ, May 2021a.
Zifeng Wang, Shao-Lun Huang, Ercan E Kuruoglu, Jimeng Sun, Xi Chen, and Yefeng Zheng. PAC-bayes information bottleneck. Xixie [es. September 2021b.
15

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Mitchell Wortsman, Gabriel Ilharco, S Gadre, R Roclofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongsook Namkoong, Ali Farhadi, Y Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. ICML, abs 2203.05822 18963-239999 March 2022.
Guoxuan Xia, Olivier Laurent, Gianni Franchi, and Christos-Savvar Bouganis. Towards understanding why label smoothing degrades selective classification and how to fix it. arXiv [cs.LG/, March 2024.
Yi Xu, Yuanhong Xu, Qi Qian, Hao Li, and Rong Jin. Towards understanding label smoothing. arXiv [es.LG], June 2020.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosscini, and Yonghui Wu. CoCa: Contrastive captioners are image-text foundation models. arv fes.CV], May 2022.
Xi Yu, Shujian Yu, and Jose Principe. Deep deterministic information bottlneck with matrix-based entropy functional. arXiv [C), January 2021.
Li Yuan, Francis E H Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. arviv fcs. CVJ, September 2019.
Valentina Zantedeschi, Maria-Irimg Nicolac, and Ambrish Rawat. Efficient defenses against adversarial attacks. In Proceedings of the 10th ACM Workshop on Arlificial Intelligence and Securily, New York, NY, USA, November 2017. ACM.
Chang-Bin Zhang, Pong-Tap Tao Jiang, Qibin Hou, Yunchao Wei, Qi Han, Zhen Li, and Ming-Ming Cheng. Delving deep into label smoothing. IEEE Trans. Image Process., 30:5984 5996, June 2021.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In 2018 IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 8697 8710. IEEE, June 2018.
A Appendix
A.1 Proof of Proposition 4.2
Proof. The following discussion entirely based on the empirical distribution p(X, Y), not the true distribution. From the data processing inequality (DPT) (Thomas M. Cover & Thomas] 2006). for the Markov chain Y + X ++ T, have i(T;Y) < T(X;T). Under Assumption 4.1 the empirical distribution corrosponds to deterministic scenario (Kolchinsky et al. 2018), where there exists a deterministic function / such that Y = =(X). Below, the DPI is shown to be saturated through an example. Consider the representation Ta = -Ba . f(X), where Ba is a Bernoulli random variable that takes the value with probability a. This can be written as Tg = Ba Y, thus the Markov chain ++ ++ Ta holds. This is the condition for the equality in the DPI, thus i(Ta; Y)- = T(X;Ta) for any a. When a 1, we have T= Y and f(T; = f(X;T) AYY), while when varying a from 0 to 1, monolonically increasing by definition. Additionally, i(T;Y) s fi(Y)) follows from the properties of mutual information. Thus, i(T;Y)= = =x( for i(XT) > (YY)..
0, T 0 and I(T;Y) = I(X;T) 0. From the continuity of mutual information, by sweep f(X:T) E [0, HY)] while satisfying î(T;Y) f(X;T). The IB curve is
A.2 Proof of Proposition 4.4
Proof. The following discussion entirely based on the empirical distribution p(X, Y), not the true distribution.
Cross-entropy can be written as H(p.g) Hp) (xr,pllg))- Thus, H(p, q) is minimized with respect to q if
and only if q p. Under Assumptions 4.1 and 4.3 (CE(O) is minimized if and only if (T 22)) 1r=
for all i. This means T Y. which implics I(T:Y) (Y(Y). Furthermore, under Assumption
f(X:T) fY(Y- f( X) = AYY..
16

---

Published in Transactions on Machine Learning Research (07 (07/2025)
A.3 Proof of Proposition 4.5
Proof. The following discussion is entirely based the empirical distribution p(X, Y), not the true distribution. T is generated from X; thus, the Markov chain Y ++ X + T holds. By the data processing inequality, Y(T;Y)= S T): Given that the cross entropy (pp, q) is minimized with respect to q if and only if q q= p. under Assumptions 4.1 and 4.3 LLs(0: a) is minimized if and only if po(T)) xx) = (a.((T) for all i. We denote the optimal representation by Ta- Since (T) is determined only by Y, we have the Markov chain X Y ++ T. This is the condition for equality in the data processing incquality, and thus [(T,; Y) = [(;;T,). When a = O, we have (Ls(0; a:) = (cE(O), and from Proposition 4.4 ((Ta;Y) = [():Ta) = HYY.. When a = 1, Ta independent of both and Y, and thus iTT:: Y) T.) = O. By the continuity of mutual information, as moves from I, we can sweep [(X;T) [0, (Y)] while satisfying [(T;Y) ((x;T). ]
A.4 Insensitivity to redundant factors
Figure Showw the results of cluttered MNIST for 20 randomly selected images in each the three experimental trials. Consistent with Figure 7 label smoothing makes models insensitive cluttered images for classes other than digit 1. On the other hand, for the digit 1 class, the saliency maps for cross entropy tend to focus the entire image, including areas without characters. This tendency becomes even more pronounced when label smoothing is applied. We will discuss this point in detail below. The idea that the model focuses only on the original image due to compression is based on the assumption that it is identifiable whether a line belongs to the original image or the cluttered image based on its shape. However, for the digit class, it is fundamentally difficult determine whether a straight line the image belongs the original image of digit 1 or cluttered image of digits
7. Therefore, the model cannot focus solely on the original image; instead, it appears to use the absence of curves across the entire image as decision criterion. As a result, the class representing the digit 1 requires information from the entire image, and this is reflected in the saliency map.
17

---

Published in Transactions on Machine Learning Research (07 (275))
g
qp
qp'
x
x
18
i
4
1 24
14
24
99
3
3
17
) 12
) 13
3
3 2'
3 2'
S
S
Figure 8: Randomll selected 20 images from the variant of the Cluttered MNIST dataset and the corresponding saliency maps. Three results with different random seeds are shown.
18


Reviews:
[{"review_id": "1urmAJecte", "reviewer": "Reviewer_eNKm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe paper is clear and well presented, with its contributions well laid out and not overstated. I believe the contributions themselves are solid and suitable for the publication venue.\n\n**Weaknesses**\n\n- They paper is missing a couple of references [1,2]. In particular, some discussion around [1] would be especially relevant, as this work demonstrates that label smoothing hurts the transferability of imagenet features to other classification tasks, which I think fits quite nicely with the potential *negative* effects of an information bottleneck removing non-label-related information.\n\n[1] Kornblith et al. Why do better loss functions lead to less transferable features? *NeurIPS 2021*\n\n[2] Xia et al. Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It *ICLR 2025*\n\n**Questions**\n\nI'm mainly familiar with label smoothing, not the information bottleneck, so I would like to additionally check some intuitions for my own benefit. \n\n1. Would mixing softmax outputs with a uniform distribution at *test* time trivially create an information bottleneck according to the empirical mutual information measures?\n2. Given that label smoothing is a one-to-one transformation on the training labels, my intuition tells me that it does not in fact reduce \"information\" (unlike label noise). Can you reconcile my intuition with the ideas presented in the paper?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The authors clarify theoretically the connection between label smoothing and the (variational) information bottleneck loss functions, showing that they are in fact quite closely related.\n- They theoretically demonstrate that under mild assumptions label smoothing can achieve the optimal solution of the information bottleneck.\n- They validate their theory on small scale image-classification data.\n- They further demonstrate information bottleneck-like behaviour, by showing experimentally that label smoothing leads to insensitivity to redundant and nuisance factors.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Add the missing references and suggested discussion.\n\nI would be grateful if the authors could also clarify my two questions related to understanding of the information bottleneck (this will not affect my acceptance recommendation).", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GMinklZ3H9", "reviewer": "Reviewer_LZ5r", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper presents some nice theoretical connections that do not seem to be present in the literature.\n2. There is a possibility that the simple insights provided might lead to further development of practical methods, e.g., development of different smoothing distributions.\n\nWeaknesses:\n1. The practical import of the findings are not clear, with it being likely that other information bottleneck approaches are more effective.\n2. The clarity of the derivations could be improved, with a bit more detail included for non-expert readers.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper notes a connection between the regularization approaches of label smoothing and Variational IB, and show that label smoothing represents a way to optimally explore the tradeoff between minimality and sufficiency in that setting. Their experiments show that this holds empirically as well with some simple experiments, and also explores the representation compression effects of label smoothing.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I do not have any major changes to require, but I did struggle a bit to follow some of the derivations as a non-expert on the information bottleneck perspective. Perhaps it would be useful to add some further lines or comments on the derivations of equations (6)-(11) to make the work a bit more self-contained. One could of course read the Alemi and Pereyra references, but a bit more self-containedness is appreciated.\n\nOne small additional note: In S3.3, in paragraph 4, I think it's poor form to use I for both mutual information and the informative factor. Please use something unique for the informative factor.", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gJpGbBtWz2", "reviewer": "Reviewer_HfJy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n* The paper is nicely written and easy to read. I can learn authors’ insights on this topic through the manuscript. The scope and claims are mostly supported by theory and experiments.\n* The key idea and derivation of drawing the analogy/correspondence between label smoothing and variational information bottleneck is reasonable with clearly stated assumptions and restrictions.\n* The experiment designs (especially the section 5.3 on demonstrating the insensitivity of the trained model to nuisance and redundant factors) are interesting and reasonably convey the key ideas.\n\n\n### Weaknesses\nI enjoy reading most of this paper, but I have four questions left with some confusion:\n* In the proof of Proposition 4.2 (Empirical IB curve; in Appendix A.1), the authors consider that $T_\\alpha = B_\\alpha \\cdot f(X)$ where $Y = f(X)$. However, this appears to be a reverse causality with other sections in this paper, where $T$ is defined more as an intermediate representation between $X$ and $Y$ (e.g., like $T=h(X)$, $Y=g(T)$). If this is an assumption to derive Proposition 4.2, could the authors make it clear in the main content?\n* Section 5.2.2 demonstrates that label smoothing is not IB-optimal when X->Y have contradicting labels by showing that the empirical IB Lagrangian results are better than the theoretical label smoothing results. This discussion itself is interesting. However, with the observation that the IB-Lagrangian results are not significantly different from the label smoothing, it would be more insightful to discuss their curve difference and a practical way to mitigate this level of gap. While the authors have mentioned one possible mitigation method (consider other smoothing distributions) and left it for future work, discussing the issue itself more deeply would be helpful in the scope of this paper.\n* The differences among the various $\\alpha$ values in Figure 6 are very small, especially for the results on CIFAR-10, which have limited statistical significance. Can the authors explain the limited difference in Section 5.3.1?\n* Section 5.3.2 shows plots with different attention regions of models trained using label smoothing or simply cross-entropy and shows their different characteristics. However, since model training can introduce different saliency maps when using different random seeds, I’m interested in whether the authors have tried using multiple random runs and tested if the saliency maps show the same trends across the runs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper theoretically links the methods of label smoothing with the methods of information bottleneck. While they are two different research areas in machine learning, they share a similar fundamental idea and mechanism. The paper also empirically demonstrates the properties of label smoothing as an information bottleneck method, in terms of the effects impacted by label characteristics, compression that introduces insensitivity to nuisance and redundant factors.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Here are some minor suggestions for editing:\n### Typos:\n* Section 1, Paragraph 2, “...the information about the inuput while preserving…” => input\n* Page 6 footnote, “$q\\phi(Y|T)$” => “$q_\\phi(Y|T)$”\n* Just after Proposition 4.2, “Below, we plot the optimal solutions…” => I guess the “Below” here should be replaced with “In Figure 1”, as there is no figure on the page.\n* Section 5.2.1, 2nd sentence, “...we have theoretically prooven the solution…” => proven\n### May need additional information:\n* Section 2.2 first sentence, “... is used in SOTA models due to its simplicity…” => can the authors specify what kinds of SOTA models and works with citations?", "broader_impact_concerns": "I do not see a requirement for this work to include a Broader Impact Statement, as the paper is about a general algorithm without ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)