Cauchy-Schwarz Divergence Information Bottleneck for Regression
Authors: Shujian Yu
Xi Yu
Sigurd Løkse
Robert Jenssen
Jose C Principe
Venue: ICLR 2024 poster (2024)
Keywords: Information Bottleneck
Cauchy-Schwarz Divergence
Regression
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We study the IB principle for the regression problem and develop a new way to parameterize the IB with deep neural networks by exploiting favorable properties of the Cauchy-Scwarz (CS) divergence.

Abstract: The information bottleneck (IB) approach is popular to improve the generalization, robustness and explainability of deep neural networks. Essentially, it aims to find a minimum sufficient representation $\mathbf{t}$ by striking a trade-off between a compression term $I(\mathbf{x};\mathbf{t})$ and a prediction term $I(y;\mathbf{t})$, where $I(\cdot;\cdot)$ refers to the mutual information (MI). MI is for the IB for the most part expressed in terms of the Kullback-Leibler (KL) divergence, which in the regression case corresponds to prediction based on mean squared error (MSE) loss with Gaussian assumption and compression approximated by variational inference. 
In this paper, we study the IB principle for the regression problem and develop a new way to parameterize the IB with deep neural networks by exploiting favorable properties of the Cauchy-Schwarz (CS) divergence. By doing so, we move away from MSE-based regression and ease estimation by avoiding variational approximations or distributional assumptions. We investigate the improved generalization ability of our proposed CS-IB and demonstrate strong adversarial robustness guarantees. We demonstrate its superior performance on six real-world regression tasks over other popular deep IB approaches. We additionally observe that the solutions discovered by CS-IB always achieve the best trade-off between prediction accuracy and compression ratio in the information plane. The code is available at \url{https://github.com/SJYuCNEL/Cauchy-Schwarz-Information-Bottleneck}.

Published as a conference paper at ICLR 2024
CAUCHY-SCHWARY DIVERGENCE INFORMATION BOT- TLENECK FOR REGRESSION
Shujian Yu!.3 Yu Sigurd Løkse
Robert Jenssen Jose C. Principe
Vrije Universiteit Amsterdam BBroohhaven National Laboratory UIT The Arctic University of Norway "NORCE Norwegian Research Centre
University of Florida University of Copenhagen
s. yu3 xyu1@bn1..... gov; siglenor
robert. jenssen@uit no; princiicipeeeeelll .ul aufl.edu edu
ABSTRACT
The information bottleneck (IB) approach is popular to improve the generaliza- tion, robustness and explainability of deep neural networks. Essentially, it aims to find minimum sufficient representation by striking a trade-off between a compression term [xx; t) and a prediction term I (3: t). where ((::-) refers to the mutual information (MI). MI is for the IB for the most part expressed in terms of the Kullback-Leibler (KL) divergence, which the regression case corresponds to prediction based on mean squared error (MSE) loss with Gaussian assumption and compression approximated by variational inference. this paper, we study the IB principle for the regression problem and develop new way to parameterize the IB with deep neural networks by exploiting favorable properties of the Cauchy- Schwarz (CS) divergence. By doing so, move away from MSE- regression and ease estimation by avoiding variational approximations or distributional as- sumptions. We investigate the improved generalization ability of proposed CS-IB and demonstrate strong adversarial robustness guarantees. We demonstrate its superior performance on six real-world regression tasks over other popular deep IB approaches. We additionally observe that the solutions discovered by CS-IB always achieve the best trade-off between prediction accuracy and compression ratio in the information plane. The code is available at /tppttp:://gthhiee com orma in Bott.
INTRODUCTION
The information bottleneck (IB) principle was proposed by (Tishby et al. 1999) an information- theoretic framework for representation learning. considers extracting information about target variable through correlated variable The extracted information is characterized by another variable t, which is (a possibly randomized) function of x. Formally, the IB objective is to learn representation t that maximizes its predictive power to subject some constraints the amount of information that carries about x:
max Tyy; s.t. T(x;t) R
p(t(x)
where I(::)) denotes the mutual information. By introducing a Lagrange multiplier 3 0. t found by optimizing a so-called IB Lagrangian (Gilad-Bachha Bachrach et al. 2003; Shamir et al. 2010):
min -T t) (x; p(t[x)
Maximizing (3:; t) ensures the sufficiency of t to predict y, whereas minimizing [(x; t) encourages the minimality (or complexity) of and prevents from encoding irrelevant bits. The parameter / controls the fundamental tradeoff between these two information terms. In this sense, the IB principle also provides a natural approximation of minimal sufficient statistic (Gilad Bacahraccchcchhhho et al. 2003).
Traditionally, the IB principle and its variants (e.g., (Strouse & Schwabl 2017 Creutzig al.) 2009)) have found applications in document clustering (Slonim Tishby 2000), image segmen- tation (Bardera al. 2009), biomolecular modeling (Wang al. 2019), etc. Recent studies have

---

Published as a conference paper at ICLR 2024
established close connections between IB and DNNs, especially in supervised learning scenario. In this context, x denotes input feature vectors, y denotes desired response such as class labels, and t refers to intermediate latent representations activations of hidden layers. Theoretically, it was observed that the layer representation t undergoes two separate training phases: a fitting or memorization phase in which both [(x; t.) and T(y;)) increase, and a compression phase in which (x:tt) decreases while (1; t) continues to increase or remains consistent (see (Shwartz-Ziv &
Tishby, 2017 [axe et al. 2018 Chelombiev et al. 2019 Yu et al. 2020 Lorenzen al. 2022) for series of work in this direction; although the argument itself is still under debate). The existence of compression also provides new insights to the generalization behavior of DNNs (Wang et al. 2022
Kawaguchi et al., 2023). Practically, the intermediate representations learned with the IB objective have been demonstrated to be more robust to adversarial attacks (Wang et al. 2021 and distributional shift (Ahuja et al. (02TT) In a parallel line of research (Bang et al. 2021 Kim et al.,
Pan et al., 2021)
202I), the IB approach has been leveraged to identify the most informative features (to a certain
decision) by learning a differentiable mask on the input, i.c., t = x ④ m, in which @ refers to element-wise product.
Unfortunately, optimizing the IB Lagrangian remains challenge due to its computational intractabil- ity. Although scalable methods of IB are feasible thanks to variational bounds of mutual informa- tion (Alemi al., 2017: Kolchinsky al. 2019b Poole et al. 2019) well as Gaussian discrete data assumptions (Chechik et al. [2003] Tishby et al. 1999). the choice of such bounds, the imposed data distributional assumptions, as well as specific details on their implementations may introduce strong inductive bias that competes with the original objective (Ngampructikorn & Schwab 2023).
In this paper, we propose new method for performing nonlinear IB on arbitrarily distributed p( y). by exploiting favorable properties of the Cauchy-Schward (CS) divergence (Principe et al. 2000 Yu et al.) 2023). We focus our attention on the regression setup, which is far less investigated than classification, except for (Ngampruetikom & Schwab) 2022) that analyzes nigh-dimmnsional linear regression from an IB perspective. For (v: t), we demonstrate that the commonly used mean squared error (MSE) loss is not an ideal approximation. Rather, the CS divergence offers new prediction term which does not take any distributional assumptions on the decoder. Similarly, for [xx; t). we demonstrate that it can be explicitly estimated from samples without variational non-parametric upper bound approximations, by making use of the Cauchy-Schward quadratic mutual information (MI-QMI). Both terms can be optimized with gradient-based approaches. To summarize, we make the following major contributions:
We show that nonlinear IB for regression on arbitrarily distributed p(x, yy can be carried out, with least dependence on variational approximations and no distributional assumptions with the aid of CS divergence estimated in non- parametric way. The resulting prediction term is no longer a simple MSE loss, as in the Kulbbeck-eeiblrr (KL) case; the compression term measures the true mutual information value, rather than its upper bound.
We demonstrate that the CS divergence induced prediction and compression terms can be estimated elegantly from observations with closed-form expressions. We also establish the close connection between CS divergence with respect to maximum mean discrepancy (MMD) (Gretton et al. 2012) and KL divergence.
We provide in-depth analysis on the generalization error and adversarial robustness of our developed CS-IB. We demonstrate the superior performance of CS-IB on six bench- mark regression datasets against five popular deep IB approaches, such as nonlinear IB (NIB) (Kolchinsky et al., 2019b) and Hilbert-Schmidt Independence Criterion Bottleneck (HSIC-bottleneck) (Wang et al. (02T)), in terms of generalization error, adversarial robust- ness, and the trade-off between prediction accuracy and compression ratio in the information plane (Shwartz-Ziv & Tishby 2017).
2 BACKGROUND KNOWLEDGE
2.1 PROBLEM FORMULATION AND VARIANTS OF IB LAGRANGIAN
In supervised learning, we have training set D fx,, y4 N of input feature x and described response y. We assume x; and y are sampled i.i.d. from a true data distribution p(x, y) = p(y)x)p(z). The usual high-level goal of supervised learning is to use the dataset D to learn a particular conditional
2

---

Published as a conference paper at ICLR 2024
distribution (0) of the task outputs given the input features parameterized by 0 which is good approximation of p(y)x), x). in which refers to the predicted output.
If we measure the closeness between p(y|x) and (0(ū))) x) with KL divergence, the learning objective becomes:
min DKL (plu) x); go (0)x)) = min E (- (og - H(yxx) + min E log((o(( (3)
where (y)x) only depends D that is independent parameters 8.
For classification, qo x) is characterized by a discrete distribution which could be the output of a neural network (2), and Eq. 3) is exactly the cross-entropy loss. For regression, suppose go (o(g)xg distributed normally N (ho(x), o²,), and the network hx(x) gives the prediction of the mean of the Gaussian, the objective reduces to IE (lly (1))) which amounts to the mean squared error (MSE) loss and empirically estimated by N/ EE= 9)².
Most supervised learning methods aim to learn an intermediate representation before making predic- tions about y. Examples include activations of hidden layers in neural networks and transformations in feature space through the kernel trick in SVM. Suppose t is possibly stochastic function of input x, determined by some parameterized conditional distribution (o(t)x), the overall estimation of the conditional probability p(y(x) is given by the marginalization over the representations:
(ū) (((x)) f qe (t))))) or go (
Σ go (o(otttgo
(4)
There are multiple ways to learn to obtain (0(())). From an information-theoreti perspective, particular appealing framework is the IB Lagrangian in Eq. ( which, however, may involve intractable integrals. Recently, scalable methods IB on continuous and non-Gaussian data using DNNs became possible thanks to different mutual information approximation techniques. In the following, we briefly introduce existing approaches on approximating [3: t) and t(x;t).
2.1.1 APPROXIMATION TO tyy
In regression, maximizing Le(y: t) is commonly approximated by minimizing MSE loss.
Proposition 1. (Rodriguez Gulvez 2019) With Gaussian assumption qo(ū)(,, maximizing To(y; t) essentially minimizes DxL. y x) ; go (o) x)), both of which could be approximated by mini- mizing MSE loss.
Proof. All proofs can be found in Appendix A
2.1.2 APPROXIMATION TO [(x; t)
The approximation to I (x: t) differs for each method. For variational IB (VIB) (Alemi et al. 2017 and similar works (Achille & Soatto 2018). [(x; is upper bounded by:
T(x;t) Eppo..) log p(t -Fppee) log p(t) s (p,,)) log log e() (p(()())) v()).
(5)
where some prior distribution such as Gaussian. On the other hand, the nonlinear informa- tion bottleneck (NIB) (Kolchinsky et al.] 2019b) uses non-parametric upper bound of mutual information (Kolchinsky & Tracey] (017):
1
I(x; t) < N
log N
exp e (mDDa...tttta))::dttlll).
(6)
Recently, Kolchinsky et al. of 3 cannot explore the IB curve when y is deterministic function of Therefore, the authors of (Kolchinsky et al.) (219a) propose simple modification to IB Lagrangian, which is also called the squared-IB Lagrangian:
2019a showed that optimizing the IB Lagrangian for different values
min -1 -1(; t) + t)(
Note that, log Vira exp (- 1 1
log log(22) Iu-fo (x]]] 20"
3

---

Published as a conference paper at ICLR 2024
The convex-IB (Rodriguez Gálvez al., 2020) further showed that applying any monotonically increasing and strictly convex function и on [x: t) can explore the IB curve. For example, by instantiating u with shifted exponential function, the objective of convex-IB is expressed as:
min - (y; t) + cxp (n: - r")) 17 > 0,r* E [0, o0).
(8)
In both squared-IB and convex-IB, [xx; t) is evaluated with Eq. 6p. There are other approaches that do not estimate (x; t). The conditional entropy bottleneck (Fischer 2020) and the deterministic information bottleneck (Strouse & Schwabl 2017) replace [x; t), respectively, with / [x; t/y) and (t)). The decodable information bottleneck (Dubois et al., (020) makes use of the V-information (Xu not being able distinguish between et al., 2020) and defines "minimality" in classification
examples with the same labels. One can also replace (xx; t) with other nonlinear dependence measures like the Hilbert-Schmidt independence criterion (HSIC) (Gretton et al., 2007).
2.2 CAUCHY-SCHWARY DIVERGENCE AND ITS INDUCED MEASURES
Motivated by the famed
(CS) inequality for square integrable functions:
( [[xx)(x))dx S f p(x)2xx / ((x)²
$
(9)
with equality iff p(x) and (x) are linearly dependent, a measure of the "distance" between p(x) and (xx) be defined, which named CS divergence (Principe al. 2000 Yu et al. 2023 with:
(S ((x))(x))xx
Des(p5 = log Tx(x)2xx x(x)2dx
(10)
The CS divergence is symmetric for any two probability density functions (PDFs) p and q, such that 0 < Des < 00, where the minimum is obtained iff p(x) (x).. Given samples {x, and {x() v 12 drawn i.i.d. from respectively p(x) and (x). CS divergence can be empirically estimated
with the kernel density estimator (KDE) (Parzen 1962) as:
m
m
n
Des(p:g) g) log
K (x x
+log
Σ 1J=1
-2 log Ks (x,,x9)
<xxx,,
m2
2
n²
mi
1-=1
j=1
(11)
where K is a kernel function such as Gaussian Ki (x, x')
x -x 2022
Remark 1. The CS divergence is also special case of the generalized divergence defined by (Lutwak al. 2005) which relies on a modification of the Hölder inequality) when a 2:
(L ( (
(ʃ ((x))))
(u = =log
(12)
((x)))) oi1 dx
The KL divergence DxL (pllg) = p (2) is obtained when + 1.
Remark 2. The CS divergence is closely related to the maximum mean discrepancy (MMD) (Gret ton et al. (2012). In fact, given a characteristic kernel s(x,x) = (Ø(x), o(x') 16 let us de- note the (empirical) mean embedding for 3 and {x] in as Up 1 E o(x²) and
mn
ug = 1 Σn o(x²), the empirical estimators of CS divergence and MMD can be expressed as:
(p)p |14411leelle
Des(r:g) = -22og
-2 log cos Mg),
(13)
2 (p: = (1ppr Mgy = 1| 11 + |1 11 - -2(1 Myy
MMD
1
m
2
(14)
x(x x(x",x"))
x(X xx)),x9)
<(x,,,
x9)).
1-=1
1J=1
1=1
That is, CS divergence measures the cosine similarity between μt, and Ml in a Reproducing kernel Hilbert space (RKHS) H. whereas MMD uses Euclidean distance.
Eq. 10 can be extended to measure the independence between two random variables x and y.
"Se Appendix B.1 for in-depth discussion on the relationship hetween CS & KL. divergencees and MMD.
4

---

Published as a conference paper at ICLR 2024
Cauchy-Schwarz Quadratic Mutual Information (CS-QMI) The independence between and y can be measured by any (valid) distance or divergence measure over the joint distribution p(x. y) with respect to the product of marginal distributions p(x)p(y). If we substitute p(x) and g(x) in
Eq. (10) with p(x. y) and p(x pty) we obtain the Cauchy-Schwan quadratic mutual information
(CS-QMI) (Principe al. 2000
| plx, ()((()))((())))e) J [x w)xxdy
(xs(x. y) Dcs(p(x.v):
-log
(15)
Distinct to KI. divergence that is notoriously hard to estimate, we will show that both CS-QMI and the divergence between p(y|x) ())x) and x()x can be elegantly estimated in a non- non-parametric way with closed-form expressions, enabling efficient implementation of deep IB without approximations.
3 THE CAUCHY-SCHWARZ DIVERGENCE INFORMATION BOTTLENECK
existing deep IB approaches rely on Shannon's definition of information and are essentially minimizing the following objective:
As has been discussed Section 2.1
min DKL (p((xx) go (ū)x)) + (x;;
(16)
p(t x)
where [xx; t) is defined also in a KL divergence sense. Both terms in Eq. (16) are hard estimate. In case of regression, min DRr. (p(y)x); qo (0 x)) gets back to the MSE loss under a parametric Gaussian assumption. However, Laplacian assumption. On the other hand, the use of an upper bound to [(x; t) also makes the solution of IB sub-optimal. Moreover, the choices of bounds assumptions are hard decide for practitioners. We refer interested readers to Appendix B.2 for a detailed discussion on different approximation biases of previous literature. We also demonstrate the advantage of CS divergence over variational KL divergence in terms of optimization via toy example in Appendix
could also uncover the mean absolute error (MAE) loss if take
In this paper, we consider a new formulation of IB that entirely based on the CS divergence:
min Dcs(p(u)xe)) qo x)) + (x; t). p(t(x)
(17)
The first term of Eq. 17 measures the conditional CS divergence between p(y x) and qo (0)x))
whereas the second term is the CS-QMI between x and t. We will show in next subsections how the favorable properties of CS divergence possibly affect the IB's computation and performance. compared to its KL. divergence counterpart.
3.1 ESTIMATION OF CS DIVERGENCE INDUCED TERMS
Both terms in Eq. 17 (i.e., Des(p(y)x): ((()))) and (xs(x; t)) can be efficiently and non- parametrically estimated from given samples (Ur, X4, t4, Dt } with the kernel density estimator (KDE) a closed-form expression. When parameterizing IB with a DNN, refers to the latent representation of one hidden layer (i.e., f(x)), denotes mini-batch size.
Proposition (Empirical Estimator of Des Dcs(p(y)x): qo (ilx))) Given observations ((x,, Uss 2 where E EP denotes p-dimensional input variable, y is the desired response, and ÿ is the predicted output generated by a model fe. Let K, L and L denote, respectively, the Gram matrices for the variable x, y, and ÿ (i.e., Kij n(x;, X3)) Lly K(y;, U5) and L2 =K(p 0ÿ). in which Gaussian kernel function). Further, let L denote the Gram matrix between i and (i.e., L = k(Dr,))) Mi)). The empirical estimation of Des(plylxx qo (0(()x)) is given by:
=cxp
20
K
Des(plu)x): go) = llog
Σ
x=
i=
(18)
N
Kj
N Σ
K K5i)²
+ +log
2 log
N Ky²)²
j=l
5

---

Published as a conference paper at ICLR 2024
Remark 3. The empirical estimator of Des(p(y) (0(xxx)) in Eq. 18 is non-negative. Compared to MSE, it assumes and encourages that if two points x; and x, are sufficiently close, their predictions (ūs and Uy) will be close well, which also enhances the numerical stability (Nguyen & Raff 2019) of the trained model. Moreover, when the kernel width a reduces to O, it gets back to MSE.
Remark 4. A precise estimation on the divergence between p(y]x) and qo ()]]x) is non-trivial task. The use of KL divergence combined with Gaussian assumption is likely to introduce in- ductive bias. Another alternative is the conditional MMD by (Ren al. 2016) with the expres- sion (MMM(P(y)x)); qo (()x))) tr(KKK
Tr(KK-II' K²
Str(KK-11) (-)))),
in which K = K + MI. Obviously, CS divergence avoids introducing an additional hyperparametr X and the necessity of matrix inverse. which improves computational efficiency and stability. Moreover, Eq. (78) does not rely on any parametric distributional assumption on qo (x)x.).
We provide detailed justifications regarding Remark and Remark 4 iin Appendix B.3
Proposition 3 (Empirical Estimator of CS- CS-QMI). Given N pairs of observations {(x,, ) sample contains two different types of measurements x E X and realization. Let K and Q denote, respectively, the Gram matrices for variable x and variable t, which are also symmetric. The empirical estimator of CS- CS-QMI is given by:
fi=l, each T obtained from the same
Ics(x;t) t) = logg
KipQus
+ log
1
Σ
KipQar)
22lgg
KuQia
N
N4
N
(19)
1
= log
N2
(r(KQ)
log
N
2oog
N
where 1. is a N x 1 vector of ones. The second line of Eq. 19) reduces the complexity to O(N2).
Remark Our empirical estimator of CS-QMI in Eq. (19) is closely related to the most widely used biased (or V-statistics) estimator of HSIC, which can be expressed as (Gretton et al. 2007):
N
N
N
HSIC&( tx:
KG Qis
1 + N
Σ 6..ir
K.yQur
2
Σ 6,119
K.jQin = N2 tr( tr(KHQH), (20)
N2
NA
where 1 111 is a N N centering matrix. Comparing Eq. 19 with Eq. 20 it is easy to observe that CS-QMI adds a logarithm operator on each term of HSTC. This is not surprising. CS- QMI equals the CS divergence between p(x, t) and p(t); whereas HSIC equivalent to MMD between p(x,t, t) and p(x)p(t). According to Remark similar rule could be expected. However, different to HSIC, the CS-QMI is a rigorous definition of mutual information. measures the mutual dependence between variables and y in units like bit (with logg) or nat (with In).
3.2
THE RATIONALITY OF THE REGULARIZATION TERM Ics (x;;
The advantages of replacing MSE (or MAE) loss with Des(p(y)x): go(()x)) is elaborated in Re- marks 3 and 4 Hence, we justify the rationality of the regularization term (cs(x:t).
3.2.1 EFFECTS OF (xs(x; t) ON GENERALIZZATION
Theorem T suggests that CS divergence is upper bounded by the smaller value between forward and reverse KT. divergences, which improves the stability of training. Additionally, Corollary [ implies that (x; t.) encourages smaller value on the dependence between x and t, i.e., a heavier penalty on the information compression. These results can be extended to arbitrary square-integral densities as shown in Appendix B.5 Note that, this is distinct to the mainstream IB optimization idea that just minimizes an upper bound of Shannon's mutual information [(x; t) due to the difficulty of estimation. Similar to Eq. (18). Eq. (19) makes the estimation of Tcs t) straightforward without any approximation or parametric assumptions on the underlying data distribution.
Theorem 1. For arbitrary d-variate Gaussian distributions p ~ N En) and ~ N ( Ez),
Dcs(p;g) min DKi(p;)) q), Dka(
(21)
Corollary 1. For two random vectors x and t which follow a joint Gaussian distribution
Σx Ext
Σt )
) the CS-QMI is no greater than the Shannon's 's mutual information:
N
Ena
Tes(x;t) S T(x;t).
(22)

---

Published as a conference paper at ICLR 2024
Ics(x:t) t)
[cs(x; tly)
[cs(x; t)
Ics (x:(xx)y))
0.30
0.31
T
0.40
0.44
MIC
0.38
0.47
MIC
0.46
0.54
Table 1: The dependence between (x; t) (or (x; tly)) and generalization gap (measured by Kendall's 7 and MIC) on synthetic data (left) and real-world California housing data (right).
Minimizing unnecessary information (by minimizing the dependence between x and t) to control generalization error has inspired lots of deep learning algorithms. In classification setup, recent study states that given m training samples, with probability 8, the generalization gap could be upper
bounded by 22(xx)) +1og(1/6) (Shwartz-Ziv et al. 2019 Galloway et al. 2023), which has been 2m
rigorously justified in (Kawaguchi et al. 2023) by replacing (x(x)) with [(x; tly) It is natural to ask if similar observations hold for regression. We hypothesize that the compression on the dependence between x and also plays fundamental role to predict the generalization of a regression model, and (x; t) Ics (x; tly) correlates well with the generalization performance of a trained network.
To test this, we train nearly one hundred fully-connected neural networks with varying hyperparam- eters (depth, width, batch size, dimensionality of t) on both a synthetic nonlinear regression data with 30 dimensional input and a real-world California housing data, and retain models that reach a stable convergence. For all models, we measure the dependence between [cs(x: t) (or Ics (x; tly)) and the generalization gap (i.e., the performance difference in training and test sets in terms of rooted mean squared error) with both Kendall's 7 and maximal information coefficient (MIC) (Reshef et al., 2011), as shown in Table [ For Kendall's т. values of close to indicate strong agreement of two rankings for samples in variables x and y. that that is, if x; > Z. then y; > y/y- Kendall's T matches motivation well, since would like to evaluate if small value of Ics (x; t) (or Ics (x; tly)) is likely to indicate a smaller generalization gap. This result is in line with that in (Kawaguchi et al. 2023) and corroborates our hypothesis. See Appendix C.T for details and additional results.
Finally, we would like emphasize that when and q are sufficiently small, Theorem D could be extended without Gaussian assumption, which may enable us derive tighter generalization error bound in certain learning scenarios. We refer interested readers to Appendix B.5 for more discussions.
3.2.2 ADVERSARIAL ROBUSTNESS GUARANTEE
Given a network hg = (x)). where f Rdx + Rd maps the input to an intermediate layer representation t. and g dT + R maps this intermediate representation t to the final layer, assume all functions hg and we consider are uniformly bounded by Mxx and M2, respectively. Let us denote F and & the induced RKHSs for kernels xx and kz, and assume all functions in F and G are uniformly bounded by My and Mg. Based on Remark aad the result by (Wang et al. [021], let ((Px)) and p(Px x & Fr) denote, respectively, the (empirical) kernel mean embedding Fx & Pr in the RHKS F & G. CS- QMI bounds the power of an arbitrary adversary in S, when vN is sufficiently large, in which S, is l ball of radius r, i.., S, = {ō dxdx 800 I r).
Fx and
Proposition 4. Denote 7-
a M Ma
(E + 6) he (x)) o(r)), ifx N(0,021)
and (|((xx))l = [Ix r logo M2
Pr) ||H|]), when VN > I ()))C( then:
- gHSIC((x [ (gS(CIS t))
P (Tes(x:t) 2 9(7)) 1 1
(23)
in which = 2l0g==22lg(122221111 is monotonically increasing function, & is the cumulative distribution function of standard Gaussian, and (HSSC, HSIC(x;;)) (0.,
4 EXPERIMENTS
We perform experiments on four benchmark regression datasets: California Housing, Appliance Energy, Beijing PM2.5, and Bike Sharing from the UCI repository. To showcase the scalability of CS-IB to high-dimensional data (e.g., images), we additionally report its performance rotation
7

---

Published as a conference paper at ICLR 2024
MNIST and UTKFace (Zhang et al. 2017). in which the tasks are respectively predicting the rotation angle of MNIST digits and estimating the age of persons by their face images. We compare CS-IB to popular deep IB approaches that could be used for regression tasks. These include VIB (Alemi et al. 2017), NIB (Kolchinsky et al., 2019b), squared-NIB (Kolchinsky et al.) 2019a), convex-NII with exponential function (Rodríguez Gálvez al. (020) and HSIC-bottleneck (aagg et al. (02T).
Similar to (Kolchinsky et al.J 2019b a Rodríguez Gálvez et al.) 2020), we optimize all competing methods for different values of B. producing series of models that explore the trade-off' between compression and prediction.
The network hy = g(f())) is consists of two parts: a stochastic encoder fenc(x) + w where w is zero-centered Gaussian noise with covariance di diag(a)) and p(t/x) = [N (x), diag(o))), and a deterministic decoder y = (te=(t). For benchmark regression datasets, the encoder fene a 3-layer fully-connected network or LST'M For rotation MNIST and UTKFace, we use VGG-16 (Simonyan Zisserman 2015) as the backbone architecture. Detail on experimental setup is in Appendix C.2
4.1 BEHAVIORS IN THE INFORMATION PLANE
For each IB approach, we traverse different values of 3 2 0. Specifically, we vary 8 E [10- 10] for CS-IB, 3 E [10-2. 1] for NIB, square-NIB, and exp-NIB, & E [10-6, 10-3 for VIB, and / E [1, 10) for the HSIC(x; t.) term in HSIC-bottleneck. These ranges were chosen empirically ing models fully explore the IB curve. To fairly compare the capability of prediction under different compression levels, we define the compression ratio at B 8 as 1 [(x; t))=3. (x(; t) Hence, r equals 0 when 3 = 0 (i.e., no compression term in the IB objective). Intuitively, large / would result in small value of [xx: t). and hence large r. Here, [(x: t) calculated by each approach's own estimator, whereas the true value of I [y; t) is approximated with }log( (ar(())rrss)))) sky et al. 2019b). The results are summarized in Fig. and Table Interestingly, when r= 0, our CS-IB have already demonstrated an obvious performance gain, which implies that our prediction term (i.e.. Eq. (18)) alone is more helpful than MSE to extract more usable information from input x to predict y. For each data, additionally report the best performance achieved when * (see Table in Appendix C.4). Again, our CS-IB outperforms others. We also perform an ablation study, showing that Eq. (18) could also improve the performances of NIB, etc. (by replacing their MSE counterpart), although they are still inferior to CS-IB. See Appendix C.4 for additional results.
that the result-
(Kolchin
Table 2: RMSE for different deep IB approaches with compression ratio r = 0 and r = 0.5. When whereas others use MSE.
0, CS-IB uses prediction term Des (p(y x); go (0)x)) in Eq. I8
Housing
Energy
PM2.5
Bike
Rotation MNIST
UTKFace
Model
0
0.3
0.3
0.5
0
0.5
0.3
0
0.5
VIB
0.258
0.347
0.059
0.071
0.025
0.038
0.428
0.523
4.351
5.358
8.870
9.258
NIB
0.258
0.267
0.059
0.060
0.025
0.034
0.428
0.435
4.351
4.102
8.870
8.756
Square-NIB
0.258
0.293
0.059
0.063
0.025
0.028
0.428
0.447
4.351
4.257
8.870
8.712
Exp-NIB
0.258
0.287
0.059
0.061
0.025
0.030
0.428
0.458
4.351
4.285
8.870
8.917
IISIC-bottlenck
0.258
0.371
0.059
0.065
0.025
0.031
0.428
0.451
4.351
4.573
8,870
8.852
CS-IB
0.251
0.245
0.056
0.058
0.022
0.027
0.404
0.412
4.165
3,930
8.702
8.653
4.2
ADVERSARIAL ROBUSTNESS
We then evaluate the adversarial robustness of the model trained with our CS-IB objective. Different types of adversarial attacks have been proposed to "fool" models by adding small carefully designed perturbations on the input. Despite extensive studies on adversarial robustness of classification networks. the adversarial robustness in regression setting is scarcely investigated but of crucial importance (Nguyen & Raff 2019 Gupta et al. 2022I.. In this section, we use the most basic way to evaluate adversarial robustness, that the regression performance on adversarially perturbed versions of the test set, also called the adversarial examples.
There no formal definitions adversarial attacks in the regression setting, we follow Nguyen & Raff (019) and consider adversarial attack as a potential symptom of numerical instability in the learned function. That is, we aim learn a numerically stable function so such that the output of two points that are near each other should be similar:
[fo(x) fo Ax) 8 S.t. ||Ax|p <c.
(24)
8

---

Published as a conference paper at ICLR 2024
1.4
2,00
12
1.75
1.0
1.50
0.8
1.25
0.6
*
VIB
1.00
NIB
0.75
NIB
0.4
*
square NIB
square NIB
exp
0.50
exp-
b5lc-bkttleneck
HSiC-bottleneck
0.25
CB-IB
IB
00.0
8.0
Compression on Ratio
Compression
(a) California Housing
(b) Beijing PM2.5
Figure I: Information plane diagrams on California Housing and Beijing PM2.5 datasets.
Different to classification, there are no natural margins in regression tasks. Hence, we just consider untargeted attack and define adversarial example x that causes the learned function to produce an output with the largest deviation:
point within Ep ball with radius around
X
arg max L(fo(x',,))
(25)
||x-x'||<e
where C is a loss function such as mean squared error.
We apply commonly used ways to solve Eq. 25 the Fast Gradient Sign Attack (FGSM) (Good fellow et al. 2014) and the Projected Gradient Descent (PGD) (Madry et 2018). In our experiments, evaluate performance against both white-box FGSM attack with perturbation image datasets (i.e, Rotation MNIST and UTKFace), and = 0.1 for the remaining four benchmark regression datasets and white-box 5-step PGD (f attack with perturbation p = 0.3 and step size = 0.1. The RMSE in the test set is shown in Table As can be seen, our CS-IB outperforms other IB approaches for both types of attacks. This result corroborates our analysis Section 3.2.2
1/255 for two
Table 3: White-box robustness (in terms of RMSE) with FGSM and PGD attacks. The best perfor- mance highlighted.
Housing
Energy
PM2.5
Bike
Rotation MNIST
UTKFaae
Method
FGSM
PGD
FGSM
PGD
FGSM
PGD
FGSM
PGD
FGSM
PGD
FGSM
PGD*
VIB
0.706
0.917
0.502
0.543
0.432
0.464
1.633
2.072
6.754
7.109
12.151
13.172
NIB
0.641
0.732
0.394
0.471
0.381
0.415
1.487
1.840
5.577
6.898
11.375
12.654
Square-NIB
0.649
0.789
0.441
0.463
0.383
0.423
1.592
1.937
5.413
6.813
11.056
12.785
Exp-NIB
0.661
0.784
0.434
0.367
0.435
1.532
6.845
12.895
HSIC- hottlennek
0.651
0.765
0.385
0.485
0.311
0.349
1.519
2.011
5.785
6,923
11.457
12.776
Ours
0.635
0.755
0.290
0.402
0.278
0.324
1.478
1.786
5.023
6.543
10.824
12.058
CONCLUSION
We discuss the implementation of deep information bottleneck (IB) for the regression setup on arbitrarily distributed p(x, (xx,)) y). By making use of the Cauchy-Schward (CS) divergence, we obtain a new prediction term that enhances numerical stability of the trained model and also avoids Gaussian assumption on the decoder. We also obtain a new compression term that estimates the true mutual information values (rather than an upper bound) and has theoretical guarantee on adversarial robust- ness. Besides, we show that CS divergence is always smaller than the popular Kullback-Leible (KL) divergence, thus enabling tighter generalization error bound. Experiments on four benchmark datasets and two high-dimensional image datasets against other five deep IB approaches over variety of deep architectures (e.g., LSTM and VGG-16) suggest that our prediction term is scalable and helpful for extracting more usable information from input predict y; the compression term also improves generalization. Moreover, our model always achieves the best trade-off terms of prediction accuracy and compression ratio. Limitations and future work discussed in Appendix

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The authors would like thank the anonymous reviewers for constructive comments. The authors would also like to thank Dr. Yicong Lin from the Vrije Universiteit Amsterdam for helpful discussions on proofs in Appendix and Mr. Kaizhong Zheng from the Xi' a' Jiaotong University for performing initial study on extending CS-IB to predict the age of patients with a graph neural network in Appendix [ This work was funded in part by the Research Council Norway (RCN) under grant 309439, and the U.S. ONR under grants N00014-18. 18-1-66,,,,,, N00014-21-1-3324, N00014-21-1-2295, the DARPA under grant
Reproducibility Statement. To ensure reproducibility, we include complete proofs to our theoretical results in Appendix A thorough justifications on the properties or advantages of our method in Appendix B detailed explanations of our experimental setup in Appendix C Our code is available at
https: https://githuse /tgpttttuuuuoooo http://c.....o.occcccccca//aaaaa
REFERENCES
Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE transactions on pattern analysis and machine intelligence, 40 (12):2897-2905, 2018.
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, -hai-hhrit Gagnon-Audet, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of- distribution generalization. Advances in Neural Information Processing Systems, 34:3438-3455 2021.
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy, Deep variational information bottleneck. In International Conference Learning Representations, 2017.
Seojin Bang, Pengtao Xie, Heewook Lee, Wei Wu, and Eric Xing. Explaining black-box by using deep variational information bottleneck approach. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11396-11 1404, 2021.
Anton Bardera, Jaume Rigau, Imma Boada, Miquel Feixas, and Mateu Sbert. Image segmentation using information bottleneck method. IEEE Transactions on Image Processing, 18(7):1001-161,, 2009.
Jan Beirlant, Edward J Dudewicz, László Györfi, Edward C Van der Meulen, et al. Nonparametric entropy estimation: An overview. International Journal of Mathematical and Statistical Sciences, 6(1):17-39, 1997.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference on machine learming, pp. 531-540. PMLR, 2018.
Gal Chechik, Amir Globerson, Naftali Tishby, and Yair Weiss. Information bottleneck for gaussian variables. Advances Neural Information Processing Systems, 16, 2003.
Ivan Chelombiev, Conor Houghton, and Cian O'Doneel. Adaptive estimators show information compression in deep neural networks. In International Conference Learning Representations, 2019.
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered alignment. The Journal of Machine Learning Research, 13(1):795-828, 2012.
Thomas M Cover. Elements of information theory. John Wiley Sons, 1999.
Felix Creutzig, Amir Globerson, and Naftali Tishby. Past-future information bottleneck in dynamical systems. Physical Review E, 79(4)0441225, 2009.
Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time. iv. Commuicattinn on pure and applied mathematics, 36(2):183-212 1983.
10

---

Published as a conference paper at ICLR 2024
Sever S Dragomir, CEM Pearce, and J Pevcari'e c. Interpolations of jensen's inequality. Tamkang Journal of Mathematics, 34(2):175-188, 2003.
Yann Dubois, Douwe Kiela, David J Schwab, and Ramakrishna Vedantam. Learning optimal representations with the decodable information bottleneck. Advances in Neural Information Processing Systems, 33:18674-18690 2020.
Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. In International Conference on Learning Representations, 2020.
Thomas Ferguson. A course in large sample theory. Routledge, 2017.
Ian Fischer. The conditional entropy bottleneck. Entropy, 22(9):999, 2020.
Angus Galloway, Anna Golubeva, Mahmoud Salem, Mihai Nica, Yani Ioannou, and Graham W. Taylor. Bounding generalization error with input compression: An empirical study with infinite- width networks. Transactions on Machine Learning Research, 2023.
Ran Baahdaaa Amir Navot, and Naftali Tishby. An information theoretic tradeoff between complexity and accuracy. In Learning Theory and Kernel Machines: 16th Annual Conference on Learning Theorr and 7th Kernel Workshop, pp. 395-609, Springer, 2003.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2014.
Daniel Greenfeld and Uri Shalit. Robust learning with the hilbert-schmidt independence criterion. In International Conference Machine Learning, pp. 3759-3768. PMLR, 2020.
Arthur Gretton, Kenji Fukumizu, Choon Teo, Le Song, Bernhard Schölkopf, and Alex Smola. A kernel statistical test of independence. Advances in neural information processing systems, 20, 2007.
Arthur Gretton, Karsten M Borgwardd, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
Kavya Gupta, Chan-Christophe Pesquet, Beatrice Pesquet-Popescu, Fragkiskos D Malliaros, and Fateh Kaakai. An adversarial attacker for neural networks in regression problems. In IJCAI Workshop on Artificial Intelligence Safety (Al Safety), 2021.
Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
Robert Jenssen. MAP IT to visualize representations. In The Twelfth International Conference on Learning Representations, 2024.
Robert Jenssen, Jose C Principe, Deniz Erdogmus, and Torbjørn Eltoft. The cauchy-schwarz divergence and parzen windowing; Connections to graph theory and mercer kernels. Journal of the Franklin Institute, 343(6):614-629) 2006.
Kittipat Kampa, Erion Hasanbelliu, and Jose C Principe. Closed-form muuchy-schware pdf divergence for mixture of gaussians. In The 2011 International Joint Conference on Neural Networks, pp. 2578 2585. IEEE, 2011.
Kenji Kawaguchi, Zhun Deng, Xu Ji, and Jiaoyang Huang. How does information bottleneck help deep learning? In International Conference on Machine Learning, 2023.
Jackyeom Kim, Minjung Kim, Dongyeon Woo, and Gunhee Kim. Drop-bottleneck Learning discrete compressed representation for noise-robust exploration. International Conference on Learning Representations, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International
Conference on Learning Representations, 2015.
Artemy Kolchinsky and Brendan D Tracey. Estimating mixture entropy with pairwise distances. Entroy, 2017. 2017.
11

---

Published as a conference paper at ICLR 2024
Artemy Kolchinsky, Brendan D. Tracey, and Steven Van Kuyk. Caveats for information bottleneck in deterministic scenarios. In International Conference on Learning Representations, 2019a.
Artemy Kolchinsky, Brendan D Tracey, and David H Wolpert. Nonlinear information bottleneck. Entropy, 21(12):1181, 2019b.
Hongzhi Liu, Kaizhong Zheng, Shujian Yu, and Badong Chen. Towards more stable and general subgraph information bottleneck. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 2023.
Stephan Sloth Lorenzen, Christian Igel, and Mads Nielsen. Information bottleneck: Exact analysis of
(quantized) neural networks. In International Conference on Learning Representatioss, 2022.
Erwin Lutwak, Deane Yang. and Gaoyong Zhang. Cramér-rao and moment-entropy inequalities for renyi entropy and generalized fisher information. IEEE Transactions Information Theory, 51 (2):473 27:8473-778, 2005.
Wan-Duo Kurt Ma, JP Lewis, and W Bastiaan Kleijn. The hsic bottleneck: Deep learning without back-propagation. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 5085-5092, 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference Learning Representations, 2018.
Rabeeh Karimi Mahabadi, Yonatan Belinkov, and James Henderson. Variational information bottle- neck for effective low-resource fine-tuning. In International Conference Learning Representa-
fions, 2021.
Vudtiwat Ngampruetikorn and David J. Schwab. Information bottleneck theory of sigh-dimensional regression: relevancy, efficiency and optimality. In Advances in Neural Information Processing Systems, 2022.
Vudtiwat Ngampruetikorn and David J Schwab. Generalized information bottleneck for gaussian variables. arXiv preprint S3Ki::3033.176:3 2023.
A. Tuan Nguyen, Toan Tran, Yarin Gal, Philip Torr, and Atilim Gunes Baydin. KL guided domain adaptation. In International Conference on Learning Representations, 2022.
Andre T Nguyen and Edward Raff. Adversarial attacks, regression, and numerical stability regular- ization. In AAA/ Workshop Engineering Dependable and Secure Machine Learning Systems (EDSMLS), 2019.
Ziqi Pan, Li Niu, Jianfu Zhang, and Liqing Zhang. Disentangled information bottleneck. In Proceedings of the AAAI Conference on Arlificial Intelligence, volume 35, pp. 9285-9293, 2021.
Emanuel Parzen. On estimation of probability density function and mode. The annals of mathemat- ical statistics, 3363)1106 1962.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference Machine Learning, pp. 5171-5180.
PMLR, 2019.
Jose C Principe, Dongxin Xu, John Fisher, and Simon Haykin. Information theoretic learning. Unsupervised adaptive filtering, 1:265-319, 2000.
Yong Ren, Jun Zhu, Jialian Li, and Yucen Luo. Conditional generative moment-matching networks. Advances in Neural Information Processing Systems, 29, 2016.
David N Reshef, Yakir A Reshef, Hilary K Finucane, Sharon R Grossman, Gilean McVean, Pe- ter J Turnbaugh, Eric S Lander, Michael Mitzenmacher, and Pardis C Sabeti. Detecting novel associations in large data sets. science, 334(6662):1518-1555 2011.
Borja Rodriguez Galvez. The information bottleneck: Connections to other problems, learning and exploration of the ib curve, 2019.
12

---

Published as a conference paper at ICLR 2024
Borja Rodríguez Gálvez, Ragnar Thobaben, and Mikael Skoglund. The convex information bottleneck lagrangian. Entropy, 22(1);98, 2020.
Andrew Michael Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Bren- dan Daniel Tracey, and David Daniel Cox. On the information bottleneck theory of deep learning. In International Conference on Learning Representations, 2018.
Ohad Shamir, Sivan Sabato, and Naftali Tishby. Learning and generalization with the information bottleneck. Theoretical Computer Science, 41 (0):3000-2666777 2010.
Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjin Wang, and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classification. In International Joint Conference on Artificial Intelligence, 2021.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. 2017.
arXiv preprint
Ravid Shwartz-Zin -Zit, Amichai Painsky, and Naftali Tishby, Representation compression and general- ization in deep neural networks, 2019. httpp:///ppeeeeeeeeeeeee////rrrr
CqK7]
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Noam Slonim and Naftali Tishby. Document clustering using word elusters via the information bottleneck method. In ACM SIGIR, pp. 208 215, 2000.
DJ Strouse and David J Schwab. The deterministic information bottleneck. Neural computation, 29 (6):1611-1630, 2017.
Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. In Proc. of the 37-th Annual Allerton Conference Communication, Control and Computing, pp. 368-377, 1999.
Linh Tran, Maja Pantic, and Marc Peter Deisenroth. Cauchy-schwarz regularized autoencoder. Journal of Machine Learning Research, 23, 2022.
Matias Vera, Leonardo Rey Vega, and Pablo Piantanida. The role of mutual information in variational classifiers. Machine Learning, pp. 1-46, 2023.
Yihang Wang, João Marcelo Lamim Ribeiro, and Pratyush Tiwary. Past-future information bottleneck for sampling molecular reaction coordinate simultaneously with thermodynamics and kinetics. Nature communications, 10(1):3573, 2019.
Zifeng Wang, Tong Jian, Aria Masoomi, Stratis loannidis, and Jennifer Dy. Revisiting hilbert-schmidt information bottleneck for adversarial robustness. Advances Neural Information Processing Systems, 34:586-597, 1021.
Zifeng Wang, Shao-Lun Huang, Erean Engin Kuruoglu, Jimeng Sun, Xi Chen, and Yefeng Zheng. PAC-bayes information bottleneck. In International Conference Learning Representations, 2022.
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable infor- mation under computational constraints. In International Conference on Learning Representations, 2020.
Shujian Yu, Kristoffer Wickstrøm, Robert Jenssen, and Jose C Principe. Understanding convolutional neural networks with information theory: An initial exploration. IEEE transactions on neural networks and learning systems, 32(1))4335444 442, 2020.
Shujian Yu, Hongming Li, Sigurd Løkse, Robert Jenssen, and José C Príncipe. The conditional cauchy-schwazz divergence with applications to time-series data and sequential decision making. arXiv preprint arXiv:2301.00 300.099700 2023.
Zhifei Zhang, Yang Song, and Hairong Qi. Age progression/reeression by conditional adversarial autoencoder. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5810-5818, 2017.
13

---

Published as a conference paper at ICLR 2024
The appendix is organized into the following topies and sections:
TABLE OF CONTENTS
A Proofs
A.1
Proof of Proposition
A.2
Proof of Proposition
A.3
Proof of Proposition
A.4
Proof of Proposition
A.5
Proof of Theorem 1
A.6
Proof of Corollary I
Properties and Analysis
B.1
The Relationship between CS Divergence, KL Divergence and MMD
B.2
The Bias Analysis
B.3
Advantages of the Prediction Term (cs(p(y)x)::gp011
B.4
Asymptotic Property of CS Divergence Estimator
B.5
Extension Theorem and Implication
C
Experimental Details and Additional Results
C.1
Effects of Ics (x; on Generalization
C.2
Experimental Setup in Real- World Regression Datasets
C.3
Adversarial Robustness
C.4
Additional Results of Section |41 and an Ablation Study
C.5
Comparing CS Divergence with Variational KL Divergence
D Limitations and Future Work
A PROOFS
A. PROOF TO PROPOSITION 1
Proposition 1. (Rodriguez Galvez 2019 With Gaussian assumption on qo(ğtt), maximizing (o t) essentially minimizes DxL x): qe (i))), both of which could be approximated by mini- mizing MSE loss.
Proof. A complete proof is in (Rodriguez vez network he g. where f is the encoder and g the decoder, minimizing DKL (p(y)x); go x))
2019). Intuitively, given feed-forward neural
can be approximated by minimizing a MSE loss LMSE (0) = E( he (x))² ) = El - (t)
(under Gaussian assumption) (Vera et al. 2023). whereas minimizing LCr (0) or LMSE( maximizes
T(y; t.) (Proposition 3.2 in (Rodriguez Galvez 2019)).
L
A.2 PROOF TO PROPOSITION 2
Given observations {(x, Ye dt NN where x E R denotes p-dimensional input variable, y is the desired response, and ů is the predicted output generated by a model fo. Let K, Lh and L2 denote, respectively, the Gram matrice for the variable y, and (i.e., Kig x;), Lij k(ys, U5) and L2 k0 Dj), in which is
Proposition (Empirical Estimator of (cs(pp))))
i=1
Gaussian kernel and takes the form of K. = exp
1-112 20 ). Further, let 1 denote the Gram matrix
II kernel learning, the Gram or kemel matrix is symmetric matrix where each entry is the inner product of the corresponding data points reproducing kernel Hilbert space (RKHS), defined by kernel function
14

---

Published as a conference paper at ICLR 2024
between 9 and y (i.e.. Lij k(ū.,y)) The empirical estimation of Des(p(y (x))) is given by:
Σ Di=1
K
9cs(plulx): log
Σ
(Ei= Kzi)²
(26)
+ log
Σ
N
أ=ز Kjilis
2 2og
N
2i=1
KiiLjt
i=1 K3;)²
(E Kjt)²
Proof. The derivation is largely inspired by Yu et al.
2023). By definition, we have
( p" (vlx)
+ log (ʃ (00(0)))xxd
(es()()))): log
(ʃ p(
2 log
(27)
(
(xx, v) dxedty + log (f
I ( 1)
log
dxdy
p²(x)
p²(xx
(fole w)oo(x (x. D) dxdy
2 log
(x(x)
[Estimation of the conditional quadratic terms J
p² (x.w) dxdy and I x²(x)
a (x.0) dxdy] (x())
The empirical estimation of f
2² (x..)) dxdy can be expressed as: (P(x)
p² (xx, v) (²(x
p(x, y) p²(x)
p(xj,85) Hb) (x(x))
dxdy Ep(x,u)
(28)
N
By kernel density estimator (KDE), we have:
Uz) xx(x))
xe N En (x, - x; (i) - 11²)
(29)
Em No (x, xz)
Therefore.
1x, v) dxdy (x)
Ei No (x x1, xo) (11 y1)
Σ K
Σ
Σ
(30)
E Nio (x, x1)
E N
Kz:)²
Similarly, the empirical estimation of 12(x.00 dxdy is given by: (xx)
f (x(
20 (x 1) dxdy
ΣN Ni (x, x,, (x) (1)
ΣN K Kyrbi
(31)
EN Nio (x, xi)
E Ky,)²
j=1
[Estimation of the cross term f p(x.,)990xx.0) dxdy] (x()
The empirical estimation of f ()x dxdy can be expressed as: px(x)
p(x, u)go (x, 1)
qa(xx, D)
N
go go Us)
(x) dxdy Exp(x.)))
(32)
px(x)
p²(x;)
"p(x, v) ()))))((() and go(xx 9) (x)))))((x).
15

---

Published as a conference paper at ICLR 2024
By KDE, we further have:
go jx Ds)
20 N Eim Nio (x, - xxi) (0g - 3i)
(33)
px(x))
Eis Noo (x; xxi)
Therefore,
J (o(x,))P((xx9)) dxdy Rz (x(x))p(xx)
E (x( - x:) (ūl) - Vi)
Nv
E K 3i
E Kzi)²
E/ (x - xi)
(34)
Combine Eqs. given by:
300. 31 and
34
with Eq.
27
empirical estimation to Des (x(v)x);; 40 (m)xx) is
E Kز
Pcs(p(u/x): = log Σ
E
(35)
N
أ=ر Kjilij
KiiLii
+ logg
2 log
E K3t)²
(En=) Kzt) ?
/=1
j=1
A.3 PROOF TO PROPOSITION 3
Proposition (Empirical Estimator of CS-QMI). Given N pairs of observations {(xi, (.)) f=l each sample contains two different types of measurements x E X and E TT obtained from the same realization. Let K and Q denote, respectively, the Gram matrices for variable and variable The empirical estimator of CS- CS-QMI is given by:
N
N
N
1
[cs(x;t) = log
Σ
Kij
+ +
Σ
Kiy KirOar - -2oog
Σ 2.g
KinQin
K
N
N
N3
=log
1 (r(KQ)
+ log
(Na NTITKII) - log
[N
N2
Na
(36)
Proof. By definition, we have:
J p(x, t)p(x))(()dxatt
(xs(x, t) = Des(p(x, t):p(x)p(t)) = = -log
TDP t)dxdt I (
(37)
= log (fDitx
(ddxtt) ()xxdt)
+ log (ʃ (2((x)22(t)xxxl (t))xcd) - 2 log (f pfx ()xx())((((()s)s
Again, all three terms inside the "log" can be estimated by KDE as follows,
f f (x²
ΣΣ x(x; xx)x(tt) (oti--tt) t,) N2
1
Σ Kig
(38)
N2
16

---

Published as a conference paper at ICLR 2024
f p(x, t)p(())((tddxct = Ep(x,)) [p(x)p(())
1
N
n(x; xx)))
(# -t,)
N
j=1
(39)
N
N
Σ
<(x;- xx, tyg)
N
{=1
f= q=1
Σ
Kij Qig:
N
N Σ
N
N N
J 1
xxx; - x;)
ΣΣ Σ
k(tt - t,)
=
N2
N2
jml
q
(40)
N
N
N N
N Σ
-(x;-xx))ee - xx)x(tt - tr) =
Kig
Qgr.
N
N4
1.3-9.
By plugging Eqs.
38
40
into Eq.
37p we obtain:
N
[cs(x:t) = log
K Quj
+ log
1er
2 -oggg
Σ
Kip
N2
N
N3
1.
(41)
An exact and naïve computation of Eq. form which needs O(n²) operations can be formulated as (both and Q are symmetric):
[1 would require O(N4) operations. However, an equivalent
(x: log
1 N2
(((K))
log 1 KIIK
22oo N33
KQ1).
(42)
where Il is vector of 1s of relevant dimension.
A.4 PROOF TO PROPOSITION 4
Given a network hg = g (x)). where f Rdx + Rd maps the input to an intermediate layer representation t, and g d + E maps this intermediate representation assume all functions ho and we consider are uniformly bounded by Mxx and Mz, respectively. Let us denote F and & the induced RKHSs for kernels xx and kz, and assume all functions in F and g are uniformly bounded by Mf and Mg. Based on Remark and the result by (Wang et | (02T), let ((Px)) and pPPx & Pr) denote, respectively, the (empirical) kernel mean embedding Fx & Pr in the RHKS F G. CS-QMI bounds the power of an arbitrary adversary in S, when VN sufficiently large, in which S, is -bll of radius r, ie.., S, = {ō Tdx 500 Sr).
the final layer, we
Pxr and
Proposition 4. Denote 7
aM
olgo (Bilho(x + 5) hx(x)) o(r)) x N(O,)
and [|] = [|x(Px rv
& Pr) = [| then:
(N(g(T) - g(SIC((x:
PP (Tcs(x; 2 u(7)) = =1
(43)
(H(S
which g(x) = 2log is monotonically increasing function, + is the cumulative distribution function of standard Gaussion, and (x) HSIC(x;;)) 2, (0)
17

---

Published as a conference paper at ICLR 2024
Proof.
We first provide the following two Lemmas.
Lemma 1 (asymptotic distribution of HXIC;(x;t) when Pxt o Pxx Pr Gretton al. 2007 ) Let
1 id.g
Rajgr 4!
Σ
KunQtu KiwQvw n
(44)
t,0,,1,00
where the sum represents all ordered quadruples (t, u. v, w) drawn without replacement from (i.).g.r), and assume B(h2) I 00. Under H4 that is Pxr + Fx @ P HSIC,(x; t) converges in distribution as N + oc to a Gaussian according to:
VN(SICG,(x: t) HSIC(x;t)) 22 N(0,o)))
(45)
where o 16 (E, H jzrr H))
Lemma 2 (adversarial robustness guarantee HSIC(x; t) (Wang et al. 2021 Assume x ~ NO, o²1). Also assume all functions he and we consider are uniformly bounded respectively by
Mx and Mz, and all functions in and are uniformly bounded respectively by My and M then:
-2loog (1dxx22 HSIC(x; + o(r) 2 [ + 8) - hx(1)]]
(46)
MM
for all 5 E S,.
By Remark [] for samples {x and {x{) drawn i.id from respectively any two square- integral distributions p and q,
Des(p: = -2log (Mp: My)
log cos H4))
(47)
and
MMD (p: = (pp M,) |1 + 1 11,113 2(p My)
(48)
in which up and M, refer to, respectively, the (empirical) kernel mean embeddings of (x?) and (x))= in the Reproducing kernel Hilbert space (RKHS) H.
Px @ Pr (the product of marginal distributions). Further, let (( and u(Pxx < IPr) denote, respectively, the (empirical) kernel mean embedding of Pxx and Px @ PT in RKHS F @ G, we have:
Let = Pxr (the joint distribution) and
(PPx Fr))
= = Des( & Pr) = -2 log
(49)
[I((Px))] |p((Px [A(Px @ Pr))
and
HSIČ(x;t) P -11(Px))) (Px @ Pr) 2(p(Pxx). @ Fr))
(Px
(50)
[[[x = (Px @ Pr)l = ||l (i.e., the norm of the (empirical) kernel mean embeddings for Pxr and Px Pr is the same, please also refer to Fig, for an geometrical interpretation), then:
[xs(x;t) = (HSIC((:: t)) = - log(1 HSIC(x t) ).
(51)
2|/
Here, g(x) is a monotonically increasing function.
'In application, and will be completely independent. Otherwise, the latent representation learns no meaningful information from x and training fails.
WWe refer interested readers to Assumptions and 2 in (Wang al. 2021 for the mathematical formulations regarding these assumptions.
18

---

Published as a conference paper at ICLR 2024
P XY
HSIC, - - PY)
IP x Py
xPP
0
Ics = -ogog 0)
Figure 2: Geometrical interpretation of CS-QMI and HSIC, in which H = F & g. When
xx) = (PPx @ Pr)llH = ||인|] CS-QMI and HSIC has a monotonic relationship.
By applying the delta method to Lemmal I we obtain:
(T((fc((xtt g(HSIC(x;t)) 2s No, ig' (H)
(52)
Here, (x) = 2-2/2 exists and always non-zero if ||A(Px))ll=o and [Ix(Px 8 PT)lsa are not orthogonal
Then, by applying Lemma
o MM
HSIC(x;t) 2
hx + +6) - -ho(x)] o(r)) = 7
(53)
TV log o0 M2
Hence,
g(HSIC(x;t) 2 9(7)-
(54)
Then,
PP (cc(xxt)22()) = (PV/N(Tcsxx:)) - - (HSIC((X Z VN(a(>) - (HSIC((x;))
VN(((1)) (HSIIC(
(55)
~ 1 - +
By applying Eq.
52
Ig' ('HSIC((x; t))]oи
By Lemma
g(>) g(HSIC(x; t)) s O.
(56)
Hence, TP (fcs(x; 2 9(7)) least 0.5.
On the other hand, when VN= > Ig ([))C(( we have:
(Tes(x;:) 2 o(7)) 1.
(57)
In fact, even when the condition (P(P)))ll [| PP |||||| does not hold, the following Lemma provides supplement.
Lemma 3. Minimizing the empirical estimator of CS CS-CCM SM- (i.e, Eq. 19 implies the minimization of that of HSIC.
19

---

Published as a conference paper at ICLR 2024
Proof. First observe that the functional relationship between the estimator of CS-QC and HSIC is equivalent to the functional relationship between the estimator of the Cauchy-Schree Schwarz divergence and squared MMD. This is due to HSIC being defined as MMD between two specific distributions, and CS-QM being defined as the CS-Divergence between the same two distributions. Without loss of generality, we base this analysis the Cauchy- Schharz divergence and squared MMD.
Given samples x(() {x x( = {x; and X x((p) U x(
" we have:
Dps(p: 9) = log (1 lit) + log (11 2 2oo ((Hpp Mg)
(ppp Myy
(58)
-22 |114p1111
log cos(Mpt M4))
and
2 MMD (p: 11 M 11 11 p 111l 2pp M,y
(59)
in which , + E. i=l (xx)) and ueg Eim1 o(x²) refer to, respectively, the (empirical) mean embeddings of ((()) and X in H.
Therefore, we have:
aDes 1
a Pllle
1 all
2
a(pe: Hy)
+ |1
(60)
0x,
p1pl 0x,
Ox,
(Mpp My)
Ox3
and
0MMD
a al1mp1114
a allp
11.
20( My)
(61)
0x
0xj
0x,
Thus, for each term
0MMD" 8x; the "log" on CS just scales the gradient contribution, but does not change the direction. That taking one step with gradient descent of Des will decrease MMD"
To corroborate our analysis, we implement a CS Divergence optimization simulation. The setup is as follows: initializing two datasets, one fixed (denote by X,), and one for optimization (denoted by X2). The goal is to minimize the CS divergence between the probability density function (PDF) of X2 and the PDF of X, by performing updates on X₂ using gradient descent.
X, was initialized as a 2d, from mixture of two Gaussian distributions, i.e. (P1, Ej) and N2(u2, Σ2), where μ4 = [-4,- -4] lr2 [4, 4) T and Σ; = Σg diag(o?). where a = 1. The number of samples N₁ = 400.
X₂ was initialized as 2d, from single Gaussian distribution, i.e., (N,,,, E), where f4 = [o, o] and Σ = diag(o²), where
1. The number of samples N2 200. The kernel width was set to 1, and
the learning rate was set to 10.
We compute MMD for each step in the optimization. The results as shown in Fig. 3 are in line with our Lemma, as MMD is decreasing while performing gradient descent on Des-
A.5 PROOF TO THEOREM
Theorem 1. For two arbitrary d-variate Gaussian distributions ~ N N(11 E1) and q ~ NM2,, E2).
Des(p: q) & Diu(Dxu( Dx)(q:p))
(62)
Proof. Given two d-dimensional Gaussian distributions ~ N (MI. Ei) and q ~ Nu2,, Ez), the KL divergence for and 9 is given by:
Σ2
DKL q)
(1(2) - d (w2 u1) M3 (w2 m1) In
(63)
Σ
in which | is the determinant of matrix.
20

---

Published as a conference paper at ICLR 2024
Refor optimization
optimirat CS divergence
-00
Findd inpoutttties
Food reavasttatinn Ditinined mnprrrttionn
7.
Osteroad reprorctics
50
5.-
255
80-
0.5
-25
-25
-40
-0.
-75-
-15
-30 60.0
10.5
-13
(a) X3 and X₂ before optimization.
(b) X and X2 after optimization.
1000
3003
1000
3000
+005
500
Ierttio
haodcc
Des each step in the optimization.
(d) MMD in each step the optimization.
Figure 3: Lemma supporting simulation.
The CS divergence for p and q given by (Kmmaa al. 2011
Dcs( - log(z12) 2
log(=n) + log(z22);
(64)
where
212 NW1luuu (E1 + E2)) exp(-H(41 (2))) )2) +22 -1 (M1 p2)
211 (2 2 (2
(65)
(2x)d1250g)
Therefore.
Dcs(p: = _ (11) (E) + -1
(w2 - -1) + log(
(2 21 + Σzl)
2 log(1 (2.))125,1) log( (2x7)
1
(66)
E1 +21
= =2 -1002
(122 - m1) (En + - - m1) +
2
2d E
We first consider the difference between Des(p: q) and DRI q) results from vector discrep- ancy, i.e., (4] 112
Lemma 4. (Horn & Johnson, 2012 For any two positive semi-definite Hermitian matrices A and B of size x n, then A - B is positive semi-definite if and only if B-1 A-1 is also positive semi-definite.
Note that, apply constant 1/2 on the definition the main manuscript.
21

---

Published as a conference paper at ICLR 2024
Applying Lemma 4 to (E1 + Σ2) Σg (which is positive semi-definite), obtain that Σz"
(Σ1
Σ2)
also positive
from which we obtain:
(02 - m1) " 2ig (u2 - nI) (u2 - 21) p1)² + -2) (u2 - -11
(67)
(#2 m1) [>g] (E) + Σ2) (12 m1) 20.
We then consider the difference between Des(p: q) and (p) results from covariance matrix discrepancy, i.c. 2i1 2...
We have,
[ + Σ2
Σ2
2(Dcs(p34) - (Ka(P: = log
- log
- tx(222 En) + d.
2d E> [22
Σ
=dl + log (EE) + Σ2()
(log Σ1 + log Σ2()
log Σn| + log |Σ - 1r(22 + d
[ + Σzl
Σ
= -dlog + log
+ 2
log
- tr( Er(Ezg 1m,) +
|E2
-loog 2 + log (222 21 1|)
log (|>: |
tr(21g -1, + d
2
(68)
denote the eigenvalues of 2, which are non-negative, since Eng En also positive semi-definite.
Let {Ar]d
We have:
1/8
d
(2 2
12
A
(69)
in which we use the property that geometric mean is no greater than the arithmetic mean. Similarly, we have:
d
1 / f (r(25) d
ES E +
11
II (1 X.) <
(70)
By plugging Eqs.
169 and 770 into Eq.
68). we obtain:
2(Des(p:4) - (p: = ld + log ( (I + II) + log | ur( +d
I dlog 2 + dlog(1 + tr(Ez En)) (x()) + log( tr(E, Ex)) (E)) - tr[ En) +
2
(71)
Let us denote (r(EE -E d; 2 0, then
2(Des(p;g) - DKL(p: 9)) = f(x) = - -loog 2 + dlog(1 + T. + log
d x
_ - x + d. (72)
2
Let f'(x) = 0, we have d. Since f"(x - = d) < 0, we have,
2(Des(p: (RI 9) - f(x) - f(x d)= 0.
(73)
Combining Eq. 67 with Eq. 773. we obtain:
Des(pig) - DRI (p: 4) 50
(74)
The above analysis also applies to DxL (g:pp)
22

---

Published as a conference paper at ICLR 2024
Specifically, we have:
2 2(DKa(g:P) Des(p: =), = (12 m1)² [x] - (E) + E2) (12 -11 2 0, (75)
[Σ + E2l
2( 2(Dcs(p; 9) - DKI. (KI((:)))) = =log
log
- Ex( + d.
2d / [Σ Σ2
=dloog 2 + log (|Σ) + Σ2()
(log ΣΣ + log Σ2|]
- log Σ + log |Σa| - -r(2i Σ2)
-1((i)12)) +
E + Σzl
ΣΣ
-dlog + log
+ 2
log
tr(Ei' Σ2) +
|1
2 + log (5T + 11) + log (5 - E2l) ir EE)) + d
< -loog 2 + dllog(1 + tr( E2)) + d
log(
tr(Ei Σ2)) tr(x; E2) +
< 0.
(76)
Hence,
Dcs(p: - DRa(a: S O.
(77)
Combining Eq. 74 and Eq. (77).
we obtain:
Des(p; & (p: (g:(p:
(78)
A.6 PROOF TO COROLLARY
Corollary 1. For two random vectors x and t which follow a joint Gaussian distribution
N
Pi
Σx
Exe
E₁ ) the CS-QMI is no greater than the Shannon's 's mutual information:
Ere
Lcs(x;t) S (xx:)).
(79)
Proof.
Ics(x: t) Des ()s p(x)p(())
(80)
(x: DKL ()(xxxx)
(81)
When x and are jointly Gaussian,
p(x, t) ~ ~N N ((1 [
Ex
Eggt
))
(Me)
(82)
Ei
Mn
p(x)p(t)~L ~N ((ta)
Enp
Σn 2)))
(pc)
(83)
Applying Theorem N (1zz)
Eer Enge
((Hz) (30
0 ) we obtain:
(me)
Erx Σ,
and N
Ics(x;t) t) S [(x;t).
(84)
23

---

Published as a conference paper at ICLR 2024
B PROPERTIES AND ANALYSIS
B.1 THE RELATIONSHIP BETWEEN CS DIVERGENCE, KL DIVERGENCE AND MMD
Integral probability metrics
f ddivergeence
(xI !) 80 p log
P
Dx (P: sup Ex (X) Ey- Wet
pxx) ((x))
P8
P-Q
dx
Q
(MMD)
divergence
(Sq²-1p) 1- (Sa²)²
= =1
(Ps Q) = 2 -2lo((cos((p Ho))
MMD² Q) = ||Up - H0ll
2 = 2: Des (P;
DxL (P:Q)
Des (P: Q) = log [ (p((x)x { (PGD)(()d))
f PQ
Figure 4: Connection between CS divergence with respect to MMD and KI. divergence
There are two popular ways to define a valid divergence measure for two densities p and q: the integral probability metries (IPMs) aim to measure "p 9"; whereas the Rényi's divergences with order aim to model the ratio of (see Fig. 4). The general idea of CS divergence is totally different: neither horizontally computes p nor vertically evaluates 2. Rather, quantifies the "distance" of two distributions by quantify the tightness (or gap) of inequality associated with the integral of densities. That is, it is hard to say the CS divergence belongs to either IRM or the traditional Rényi's divergence family.
One of the incqualities one can consider here is the CS inequality:
if plz)g(z)dZl s
1/2
(f((co)a))
1/2
(85)
which directly motivates our CS divergence.
In fact, the CS inequality can be generalized L,p space by making use of the Hölder inequality:
/ p(z))(r)dd) (fD*co)t)) 1/a
(ʃ (((())
1/b
1 = 1, (86)
Va,b> 1 and
b
from which we can obtain so-called Hölder divergence:
(U ((r())))) (S (Ur (x)()))))""))
(p(z)g(z)dd))
DH -
(87)
although DH introduces one more hyperparameter.
One immediate advantage to define divergence in such a way (i.e., Eq. 10) and Eq. (87) is that, compared to the basic KI. divergence (or majority of divergences by measuring the ratio 2) the CS divergence is much more stable in the sense that relaxes the constraint on the distribution supports, which makes it hard to reach a value of infinity.
In fact, for any two square- inttre--ntegral densities p and q, the CS divergence goes to infinity if and only there is no overlap on the supports of and i..., supp(p) = 0, since log 0 is undefined. For p(x)
DRt. (p; q). it has finite values only if supp(p) C supp(g) (note that, p(x) log
- 0o Cover
0
24

---

Published as a conference paper at ICLR 2024
p
q
p
q
p
q
DKc. (p; 9) + 00
DKL. (p; q) + co
DKI. (p; q) + co
DKL. (q: p) 00
DxL (q:p) oo
DxL (q:p) finite
Dos (p: q)
Dcs (p:q) finite
Des (p: q) finite
Figure 5: KL divergence infinite even though there is an overlap between sppp(p) and supp(?), but neither is subset of the other. CS divergence does not has such support constraint.
1999)); whereas for DRI (q: p), it has finite values only if supp(a) C supp(p). Please see Fig. 5 for an illustration.
Finally, would like emphasize here that, although CS divergence does not belong to either IRMs and f-divergence family. It has close connections to both these two. On the one hand, if we rely on another inequality associated with densities:
(fz)) 2
(f
(88)
can be shown that when À = 2, we obtain CS divergence; whereas when 1, we obtain KL divergence.
the other hand, from Remark 2 the empirical estimator of CS divergence measures the cosine similarity between (empirical) kernel mean embeddings of two densities; whereas MMD uses Euclidean distance.
B.2 THE BIAS ANALYSIS
We discuss in this section the bias induced by variational approximations in previous literature. Let the conditional probability po (t|xx denote parameterized encoding map with parameter 0. Also let the conditional probability Po (y(tt denote a parameterized decoding map with parameter Φ
B.2.1 THE UPPER BOUND oF [(x:t)
To (x;;; t) can be expressed as:
To(x:t)= / po (xx
(t(()))
dxdd
(t)
/ po (xxx
po(t|x))
dxdt. / po(() log
(po(t)
dt r(t)
(89)
(t)
/ po (x(x t) log
(t(()
dxdt.
r(t)
which r(t) is a variational approximation to the marginal distribution (t( = ((x))o(t|x)dx
As can seen, the approximation error exactly (xL())())rr(t) = po log
poit) r(t)
dt.
In VIB (Alemi al. 2017) and lots of its downstream applications such (Mahabadi et al. 2021). r(t) is set as simple as a Gaussian. However, by our visualization in Fig. 6 (t(t) deviates lot from Gaussian.
Regarding the upper bound of I (xx t) used in other existing methods, such as NIB (Kolchinsky et al. 2019b) and its extensions like exp-NIB (Rodríguez Gálvez et al. 2020), our argument is that these
25

---

Published as a conference paper at ICLR 2024
methods may have similar bias than con-aaamettic estimator [x; t). This is because their estimators can also be interpreted from a kernel density estimation (KDE) perspective.
In fact, in NIB and exp-IB, (x; t) is upper bounded by (Kolchinsky & Tracey 2017
1x(x;t)
log
exp (-Dxxuu(((()))
(90)
N
which, in practice, is estimated as (aaxe et al. 2018):
N
N
h
h
Tx(x:t)
log
exp
(91)
N
N
2
==1
in with h is the hidden activity corresponding to each input sample or the Gaussian center, i.e., t = h + e, and *O o²1).
We argue that Eq. (91) is closely related to the KDE on the Shannon entropy of t. Note that, the Shannon's 's differential entropy can be expressed as:
((t) - J log)) (logp))))
1
log ptt)).
(92)
N
By KDE with a Gaussian kernel of kernel size a, we have:
1
1
||t t
p(t;))
exp
(93)
N
2nT
20
Hence,
1
1
N
||t; 1
H(t) N
log
exp
(94)
N
2
20²
j=1
Comparing Eq. 91 with Eq. 94
replaced with + e, and there difference on
constant V2Ka"
From perspective, using t rather than h seems to be better choice. This is because,
[(x; t) = H(t) (t))))
(95)
(t) He))) H(t) d(log V2ma + )
Hence, we keep a unchanged (during training) or take a value of amounts to minimize [(x; t). The only bias comes from the way to estimate (t).
minimizing H(t)
To summarize, previous methods either have much more obvious bias than ours (severe mismatch between r(t) and (t)) or have similar bias (due to the same KDE in essence). But some improper choices (c.g., using h or t, or varying noise variance) may incur other bias
B.2.2 THE LOWER BOUND OF Ily:t)
To(y; t) can be expressed as:
To(y:t) = H((lv)) - (po(ult))
H(p(v)) H(po(ylt) D
(96)
= Hp(vv) + EEpola,) (po()))))
where (b(tt) is a variational approximation to ()))tt). The smaller the KL divergence DRL. (po (yl t): (o(y)t))) the smaller the approximation error.
Further, we assume Po (ylt) follows a Gaussian, then we obtain MSE loss; whereas we assume Po(y)t) follows a Laplacian, we obtain MAE loss.
26

---

Published as a conference paper at ICLR 2024
~000
30
-20
-10
(a) MSE
(b) VIB
(c) NIB
Figure 6: PCA projection of bottleneck layer activations for models trained with (a) MSE loss only,
(b) VIB, and (c) NIB on California housing.
Hence, the bias on estimating [y;; t) simply by MSE comes from two sources: the closeness between Po(ytt) and pe (y]t), and the suitability of parametric Gaussian assumption.
Unfortunately, it is hard to justify or investigate if the bias from the first source is small or not. This because (Alemi al. 2017):
(o()lx))o(z()))pp dx, (z(
po (o(y
(97)
which is intractable. On the other hand, although Gaussian assumption on Po(u(t) the most popular choice, it is very likely that MAE (or losses induced by other parametric assumptions) has better performance.
To summarize, the bias term in estimating [z: t) by variational approximation is a bit hard to control. Moreover, there is no theoretical guarantee on properties like consistency. Fortunately, CS divergence with KDE does not have such limitations. Interested readers refer to Section
B.3 ADVANTAGES OF THE PREDICTION TERM Des(p(v)x):
Given observations {(x,, Vi, D.) Fi= where E I2 denotes p-dimensional input variable, y is the desired response, and ÿ is the predicted output generated by a model fgg Let K, L and L denote, respectively, the Gram matrices for the variable x, y, and ý (i.e., K,ş <(x;, xj). L!j n(y;, vj) and L3 x(ss. úy)). Further, let 2 denote the Gram matrix between & and y (i.c., LTj <0 8) The prediction term Des(p(y)x xx;; (ū)) is given by:
Kyi
Des(p(r)x): = log
Σ
K₁t)²
(98)
Σ - Kys Ky
Σm K5rLgh
+ log
Σ
2lo
Σ
E Kjt)²
N
(Ein Kji)²
j=1
We focus our analysis on the last term, as it directly evaluates the difference between y and ÿ, which should governs the quality of prediction.
By the second order Taylor expansion, we have:
(ūi - y;)
1
?2 = ==(1,,))) = exp
~
(0) - -0, y,)
(99)
20
2
27

---

Published as a conference paper at ICLR 2024
For fixed j, we have:
1=
N
K
Kj1
Di +
2
L22 +
KJN
LiN
Kji)²
(2i=)
Kji)²
(2i=1 Kji)²
(2im) Kzi)²
K₁
Ky²
(0j 81
(9) 1/2)
(100)
Di=1
N
Kji
20
(Eiz) N
Kzx)²
(E Kj;)²
Kjj E Kj,)²
KGN
(Ø) y3)) 4...4
(ūs yN
E Kji)²
That is, compared with the na've mean squared error (MSE) loss that only minimizes (Új y;)², prediction term additionally considers the squared distance between Ūs and yı(E + j). The weight on (ū, y1)² determined by K i.e., the kernel distance between X, and x;: if ||x; - x, small, there is heavy weight on (Új y:) otherwise, the weight on (ύ₁ y4)² is very light or negligible. Hence, our prediction term encourages that if two points and x, are sufficiently close, their predictions (úi and ÿy) will be close as well.
To empirically support this, we trained neural network the California Housing data with respectively the MSE loss and our conditional divergence loss (i.e., Eq. (88). ). In test set, we measure all the pairwise distances between the j-th sample and the j-th sample (i.e., ||x; x,112) and their predictions (i.c., 1 - y D). From Fig. 7 we observe that our conditional CS divergence loss encourages much higher correlation between X Xyl12 and 1 ūyl than MSE, which corroborates
our analysis. That is, close points have closer predictions.
1.75
divergence
1.50
1.25
vil
1.00
0.75
0.50
0.25
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
||x - x|1
Figure 7: ||x; Xy ll2 with respect 10 0,1-
On the other hand, when + O, we have Kjj and KJ; + 0(i * j). In this case, our prediction term reduces to the MSE.
If we measure the divergence between p(y[x) and go (ū(x)) with conditional MMD by (Ren et al. 2016). The empirical estimator given by
(MMD(p(y/>x)) go (0)x)) = tr(Kk - -,- (L- (101)
which K K + AI.
By comparing conditional MMD in Eq. (101) with conditional CS in Eq. (98). observe that conditional CS avoids introducing an additional hyperparametr and the necessity of matrix inverses, which is computationally efficient and more stable.
28

---

Published as a conference paper at ICLR 2024
To justify argument, additionally compared conditional MMD with conditional CS as loss function to train neural network also on California Housing data. The result is Fig. which clearly corroborates our analysis.
MSE
0.45
Conditional MMD
Conditional CS divergence
0.40
RMSE
0.35
0.30
0.25
0
20
40
60
80
100
Epoch
Figure 8: Comparison learning curves of conditional MMD, MSE, and conditional CS divergences on the California Housing test data. The learning curve of conditional CS is much smoother than that that of conditional MMD (almost no jitter).
Finally, one can also non-parametrically measure the closeness between p(yl x) and qo (ú) x) with DRL (p(vlx); (o(lx))) By the same KDE technique, we obtain:
1
p(,,, Vi)
(xvll() (ūlx)) DKL w) (x, 0))
log
N
Go (X:, 1/:)
1=
N
x;
X
1|2/ 18//202
(102)
N
exp
Ud
log
Σj
exp
||
x;
0
Xj
22/22
Unfortunately, we found that the above formula is hard to converge or achieve a promising result. One possible reason is perhaps the support constraint on the KL divergence to obtain a finite value: our case, there no guarantee that supp(p(x, v)) C supp(go 0)).
B.4 ASYMPTOTIC PROPERTY OUR CS DIVERGENCE ESTIMATOR
For simplicity, we analyze the asymptotic property of our KDE estimator on the basic CS divergence between two distributions p and g. which has the following form:
Des(p; -log (f ((ddu) log (f
(f (mddd)
2log
(103)
loo (E, (p)) log (E)(g)) - log (Ep(g)) - log (p))
Note that, the term 2 log (ʃ pqdu) can be expressed as:
wi log (T)) wa log (F,(P)) Vin W2 2, wi 2 0, 22 > 0.
(104)
29

---

Published as a conference paper at ICLR 2024
In the following, consider two distributions with equal number of samples and hence set wi We = 1. Given observations X := {I1 Z2. ... .2N] and := {41. 92. UN drawn i.idd from p and 9, respectively, the empirical estimator of Des (p: q) is given by:
N
N
N
N
1
Des(p(z): log
B(ui))
+ +log
((vi
-
log
((.))
log
Σ
p(yn)
NV
N
N
N
i=1
i=1
i=1
N
1
= log
Na Sal xi)
+ log
Kio (M3 y44
N2
N2
i=1
i=1
NN
N
N
-log
Ri (o555 - 24)
- -log
Ki (or vi)
N
N2
i=
==
i=
1=1
(105)
where the second equation is by KDE with kernel function of width
Note that, our CS divergence estimator Eq. (105) is different to that derived in (Jenssen et al. 2006).
In fact, quantities like [p2 and J pqdu can be estimated in a couple of ways (Beirlant et al. 1997 Our estimator in Eq. (TO5) is also called the resubstitution estimator; whereas the authors (Jenssen et al. 2006) use the plug-in estimator that simply inserting KDE of the density into the formula, i.e.,
I ppddu ~
/ ((o)dd
/
Σ
Ko (2; 2) dr
i=1
(106)
N
/
Ka(x;- 2)) x (2, r)dr.
N2
Authors of (Jnssen al. 2006) then assume a Gaussian kernel and rely on the property that the integral of the product of two Gaussians is exactly evaluated as the value of the Gaussian computed the difference of the arguments and whose variance is the sum of the variances of the two original Gaussian functions. Hence,
I dd
1 N
ΣΣ $=1==1
Ka(a: - x)) x Ko - r)drr
Σ
N
Σ
N
2 - 33). (107)
To knowledge, other kernel functions, however, do result in such convenient evaluation of the integral because the Gaussian maintains the functional form under convolution.
By contrast, we estimate f dd as:
( pd Ep(p)
1 N
Σ
p(x.)
(x; 2z)) dr
N
(z+ (z)-I;).
N
N
N2
(108)
only differs from Eq. 107 by replacing 22 with estimator offers two immediately advantages over that in (Jenssen et al. [006]: 1) our estimator is generalizable to all valid kernel functions; 2) the asymptotic property of estimator be guaranteed, whereas such analysis in (Jenssen et al. 2006) is missing.
Although Eq. 108
Having explained the difference between resubstitution estimator and plug- in estimator. we analyze the asymptotic bias and variance of our estimator. As an example, we focus our analysis on the cross-term f pqdp= (g)
30

---

Published as a conference paper at ICLR 2024
B.4.1 BIAS OF pqdp Ep(1)
The estimator of (g) is given by:
N N2
Ep(g)
Σ
Ko (ol xz).
(109)
Let us denote (p) and S
Ei=1
Σ Kir [yg - xi). the bias of 3 can obtained by
N2
expectation:
E(S)
E
Kig (Uا Iia))
N2
j=1
(110)
N
12
[Nirr (Uل - xz)] (1o - x;:]]
2
j=1
Further,
/f
NNo (yi - -x:]]
Ki (o - -(:))((;))))
/ Kalus (ouj-xiddlun xx)a(y;)dy) dr:
p(za))
/ (o(())(( as)d((ss) dx: Let us denote
p(aaa)
8 M/ -
/ p(a.))
/ ((s)a(=;+ + de: Note: ko(s)o
1 expl-52/2) kis)
V2
f p(())
/ [x()]((x)+ + ts) 1 o00 )2)))( + o(o2))ds dr Taylor expansion
= =sm(n) p(x) g(x)) ((s))s+od sk(s)ds foop/dss0000/00 f (n(s)ds+00o0 dx
1 = =sn(w)a(e)+ + oz (r) o(03),
2
(111)
where μxx f (k(s)dss (for Gaussian kernel, Mx = 1).
Namely, the bias of S is:
bias(S) =E(S)
padu
2 0² ux / pq o(02)
(112)
2 o 2 14x + o(02).
We see that the bias of Š increases proportionally to the square of the kernel size multiplied by the expected value of the second derivative of (under distribution p). This result also reveals an interesting factor: the bias is caused by the curvature (second derivative) of the density function, which coincides with KDE.
31

---

Published as a conference paper at ICLR 2024
B.4.2 VARIANCE OF f pqdy Ep(g)
For the analysis of variance, can obtain an upper bound straightforwardly:
N
N
var 5) var
Ko (U3
- -x;)
N
j=l
N
N
M/j Iz.
var
K
N
0
1=1
j=1
N
Σ
Mj
22
N
var
R
U/
var
R
independence assumption
220
1
22
0
1
2
Vj I2
pla, dr,dyd
N202
1
1 12 y- y-x x a(u)dy| dr
p(x)
N202
/ p(x)
(3))((2 + + de Let us denote
1/ -
No0
N26 / p(x)
/ (>(()(a(>) t o(o)]ds dr
N / p(x) g(z) / (s) ood(x)
sk (s) +o(a)
dx We assume symmetric kernel.
/ p(z)g(=)02 + of N2
N2o
(113)
where o: = {(s)ds (for Gaussian kernel, of = 2/=
So, from the above analysis, we conclude that the variance of S will decrease inversely proportional to N2, which a comfortable result for estimation.
The asymptotic mean integrated square error (AMISE) of $ is therefore:
AMISE($) = E
(S-
(114)
0² 4 ( + Noo (g)o
in which 1s f (xs(sssss and of = f(())ds.
To summarize, AMISE will tend to zero when the kernel size goes to zero and the number of samples goes to infinity with 2 + 0, that is, S a consistent estimator of S.
Finally, one should note that, the log operator does not influence the convergence of S:
N - S) = O,(1)
(115)
N (o(log( log(S)) = N No (1+
NVOIS 5) N NYOS
=O,(1)
(116)
32

---

Published as a conference paper at ICLR 2024
Additionally, this result can also be obtained by the delta method Ferguson 2017)
The bias and variance of other terms such as log (p() and log (g) in CS divergence can be quantified similarly.
The same result also applies for CS-QMI, i... (x: This is because we can construct new concatenated variable yx in the joint space of and y, and let p(z) p(x, y) and g(z) = p(x)p(y), then:
Ics(x:y) Dcs(((zz)
(117)
B.5 EXTENSION OF THEOREM AND ITS IMPLICATION
Theorem D suggests that, under Gaussian assumptions, (es(p:) q) < (pt q). In this section, we show that such conclusion can be generalized to two arbitrary square square-interral density functions. We also provide its implication.
B.5.1 CS DIVERGENCE IS USUALLY SMALLER THAN KI. DIVERGENCE
Proposition 5. For any density functions p and q that are square interali let [K] denote the length of the integral's integration range K with K >
have:
C1 [Des(p; q) log K| + 2 2og C2] S DRL(p:
(118)
in which C1 = p(x)dxx and C2 (Jx (x))xx P (x))xx)
Si p(x)dx
(Jx (xx)xx fxx 4 (x))xx)
Proof. The following results hold for multivariate density functions. For straightforward illustration, prove the results for the univariate case.
We first present Lemma which is also called the Jensen weighted integral inequality.
Lemma 5. (Dragomir al. 2003) Assume convex function f II - R and g, h [x1, IR are measurable functions such that g(x) E and h(x) 2 0 Vo E [2], Also suppose that h, gh, and (f o g): are all integrable functions on [21, 22] and 2 h(x)dz 0, then
(
1 g())h(x)ddz
(f (z))((z))
(119)
ST hx))d
ST h(r)dr
Let us set ((x) = b(x) and g(x) and f log(x) which is convex function, by applying Lemma 5 obtain:
a(æ) b(a)
(
12 alz)dz) log
JT a(z)do J b(x)dx
a(x) dr. b(z)
a(x) log
(120)
The inequality above holds for any integration range, provided the Riemann integrals exist. Moreover, this inequality can be easily extended general ranges, including possibly disconnected sets, using Lebesgue integrals. In fact, Eq. (120) can be understood as a continuous extension of the well-known
log sum inequaity. For simplicity, we denote fT2 a(x)dr = -Sx a(x)dar in which K refers the length of the integral's interval.
22 31
Now, suppose we are given two distributions p(x) and g(x), let us construct the following two functions:
a(x) p(x) p(x)
C2 Sxx plz)dr (fola)d / ((x))dx)
1/4
(121)
bxx) = vl(c)
Clearly,
p(x) a(x)
Cz.
(122)
((x)
b(x)
33

---

Published as a conference paper at ICLR 2024
We have,
/ p(x:) log p(z)
(K
dr
K ((x))
2 / p(x) log V
((x)
dx
K
((x))
a(x=)
2 K
alx) log
C2
dr
b(x)
(123)
Urr aa(x) log
(a(x)
dr + log C2 / a(x)dxx)
2Cc
b(xx)
[[x a(z)dx)
Lx a(r)dr
+ log O2 f( a(x)do)
202
log
SK b(x)dx
2C2 / a(z)dx
JK a(x)dr
log
log C2
Sxx b(z)dx)
in which the fifth line is due Eq. (20).
Note that,
[ a(z)do= = 1x p(x) dx - C2 C2 1 p(z)de (1 (())dd [ 2 (())dd 1/4
(124)
and, using the Cauchy-Schward inequality.
( (a))az) (
2 - - -1 vp(z)g(z)) 1dx
2 & (f ((plt)als)))) ( 1dx = =(1s p(o))((())d)) ) |K).
(125)
By plugging Eqs. 124 and 125) into Eq. 123) we have:
(x: 2 20, / a(x)dx
LK a(x)dxx
log
log C2
Jx b(x)dx
J p(a)da
Sx a(z))dx
log
2og C2
Jx b()dde)
(Jx fxx 92 (2())d 1/2
-Se p(z)dd
log
+ 2 log C2
(Jx (z))d))
(126)
(Jx fx g² (x:)dx) 1/2 (Jx (x(((x)(x))) [K]
S
p(a)ddx log
log O2
K
/ p(x))dx (Ds((rgg) log |K| + log C2]
Ci [Des(p: q) - log [K] + log C2]
in which Ci fK p(x)da 1.
B.5.2 EMPIRICAL JUSTIFICATION
We also provide an empirical justification, showing that in general cases, the following relationship largely holds:
Dcs S DRL.:
(127)
in which p and g need not be Gaussian, and the symbol & denotes "less than or similar to".
34

---

Published as a conference paper at ICLR 2024
We focus our justification discrete and q for simplicity. This is because, unlike the CS divergence. KL divergence does not have closed-form expression for mixture-ou of Gaussians (MoG) (Kampa et al. 2011 Hence, it becomes hard to perform Monte Carlo simulation the continuous regime.
For two discrete distributions and q on the finite set A' {01 different discrete states), let denote p(æ;) p(a 2z), we have:
Øx (i.c., there are K
(p(zi)
DRL 9) =
Σ i=1
p(z;) log
(((24)
(128)
K
K
s.t.
p(x;)
Σ ((z;)
=
==1
1=1
Σp(x:)g(x))
Des(p: q) -llog
(129)
To empirically justify our analysis, for each value of K, we randomly generate 100 pairs of distributions p and q. Fig. ddmmonstratss the values of DxI. with respect Des when K K and K = 10, respectively.
0 cumgag
C5 doegennn
25 doggees
(a) =
(b) K =
(c) K = 10
Figure 9: Values DKL with respect Des for 1, 000 pairs of randomly generated p and q when K == 2, K = 3 and K = 10, respectively. The diagonal indicates DKL.
Dcs-
B.5.3 TIGIITER GENERALIZATION ERROR BOUND UNSUPERVISED DOMAIN ADAPTATION
In the problem of unsupervised domain adaptation, aim to learn classifier from samples in source distribution Ps that is generalizable to a related and different target distribution pt. Suppose we learn latent representation t such that log p( () is bounded by constant M
From (Nguyen et al. 2022), the loss hest in the target domain can be upper bounded by the loss hrain in the source domain as:
M / ()) /2
hest Iuaai +
(130)
Eq. (130) implies that the generalization gap from source target domain is upper bounded by the mismatch on the joint distributions Pa (t, y) and p. (t, y). The exact management of the KL divergence is usually hard. This drawback motivates possibility replace KL divergence with CS divergence, which, by Theorem [ may enable tighter generalization error bound. We leave systematic evaluation of this proposal as future work.
'In classification, we can enforce this condition easily by augmenting the output softmax of the classifier so that each class probability is always least M) (gguyen et al. 2022). For example, if we choose M 4, then exp(- M) ))=)))))) 0.012.
35

---

Published as a conference paper at ICLR 2024
C EXPERIMENTAL DETAILS AND ADDITIONAL RESULTS
C.I EFFECTS OF (cs(x; t) ON GENERALIZATION
For the correlation experiment in Section 3.2.1 we generate a nonlinear regression data with 30-dimensional input, in which the input variable x is generated i.i.d. from an isotropic multi- variate Gaussian distribution, i.e, x No Ian). The corresponding output y generated as y sin (w" x) + +logg (w" x), where w ~ (0, (30). We generate 5, 000 samples and use 4, 000 for training and remain the rest I, 000 for test. We also select the real-world California housing dataset, and randomly split into 70% training samples, 10% validation samples, and 20% testing samples.
For the first data, we use fully-connected networks and sweep over the following hyperparameters: (i) the depth (1 hidden layer or 2 hidden layers); (ii) the width (first hidden layer with number of neurons (256, 224, 192, 164, 128, 96, 64); second hidden layer with number of neurons {64, 48, 32, 16)); (iti) batch size ((128, 96, )4 (iv) learning rate (0.1, 0.05). We train every model with SGD with hard stop at 200 epochs. We only retain models that have a stable convergence.
For the second data, We also use fully-connected networks and sweep over the following hyper- parameters: (i) the depth (2, 4, 6 or 8 hidden layers); (ii) the width (16, 32, 64, or 128) and keep the number of neurons the same for all the hidden layers: (iii) batch size (64, 128, 256, or 512); (iv) learning rate (0.001, 0.0005, 0.0001). We train 200 epochs for each model with Adam and only retain converged models. In total, we have nearly 100 models on two NVIDIA V100 GPUs.
For each resulting model, we compute three quantities: 1) the CS-QMI (i.e., (xs(x; t) between input x and hidden layer representation t); the conditional mutual information between input x and hidden layer representation t given response variable (i.c., Ics (x; t(y)); 3) the generalization gap (i.e., the performance difference training and test sets in terms of rooted mean squared error).
To evaluate (x; t/y), we follow the chain rule in (Federici et al. 2020): (x;; t) Ics (x; tly) + [y:(y: t). in which (xs(x; t tly) is also called the superfiuous information, and Ics (y: t) the predictive information.
We evaluate the dependence between (x;(x;t) t) and the generalization gap, and the dependence between (xs(x; tly) and the generalization gap. Two kinds of dependence measures are used: Kendall's 7 and maximal information coefficient (MIC) (Reshef et al.] 201T). For Kendall's т. values of close to 1 indicate strong agreement of two rankings for samples in variables and y, that that is, d; > Ty. then y: > Vy. Kendall's T matches our motivation well, since we would like to evaluate small value of (xs t) (or (x: tly)) is likely to indicate a smaller generalization gap. Compared to Kendall's Is T, MIC is able to capture more complex and nonlinear dependence relationships.
According to Table and Fig. 10 can conclude that both Ics (x; t) and Ics (xx: (xy have a positive correlation with empirical generalization error gap. This result is in line with (Kawaguchi et al. 2023 Galloway et al. 2023). although these two works focus on classification setup.
C.2 EXPERIMENTAL SETUP IN REAL- WORLD REGRESSION DATASETS
We first provide more details on the used datasets in the main paper.
California Housing This dataset contains 20, 640 samples of 8 real number input variables like the longitude and latitude of the house. The output is the house price. A log-transfoo trgntraasformed house price was used as the target variable, and those 992 samples with a house price greater than $500, 000 were dropped. The data were normalized with zero mean and unit variance and randomly split into 70% training samples, 10% validation samples, and 20% test samples.
Appliance Energy [o] This dataset contains 12, 630 samples of appliance energy use in low-energy building. Energy data was logged every 10 minutes for about 4.5 months. In the dataset, each record has 14 features, such air pressure, outside temperature and humidity, wind speed, visibility, dew point, energy use of light, and kitchen, laundry, and living room temperature and humidity. We select 80% for training and 20% for testing, normalize the data between 0 and with MInMaxscaler.
learn http:////eaattttererr//////////e
fet
ch
ca 1 fornio hhwaahhossso htm
https::/ h/op:://www.5 kagg com
36

---

Published as a conference paper at ICLR 2024
0.35
0.35
0.3
0.3
Generalization Gap
Generalization Gap
0.25
0.25
0.2
0.2
0.15
0.15
0.1
0.1
2
5
3
4
CS-QMI
Conditional S-MMI
0.08
0.08
Generalization Gap
Deneralization Gap
0.06
0.06
0.05
0.05
0.04
0.04
X
0.03
0.03
0.02
0.4
0.5
0.6
0.7
0.8
0.9
0.0
0.2
0.4
0.6
0.8
Conditional
Figure 10: Scatter plot of Ics (x; t) (left) and (xs(x; tly) (right) with respect the generalization gap in synthetic data (first row) and California housing (second row).
Beijing PM2.5 This dataset collected PM 2.5 between Jan 1st, 2010 Dec 31st, 2014 in Beijing. Each record in the dataset has 7 features, including dew point, temperature, pressure, combined wind direction, wind speed, hours of snow and hours of rain. The dataset contains 41, 757 samples. We selected 80% for training and 20% for test.
Bike Sharing 12 This dataset includes the hourly and daily count of rental bikes between 2011 and 2012 in the Capital Bikeshare system, along with weather and seasonal information. In this paper, we utilize the hourly count data and each record in the dataset includes the following features: holiday, weekday, workingday, weathersite, temperature, feeling temperature, wind speed, humidity count of casual users, and count of registered users. Consisting of 17, 379 samples, the data was collected over years, and be partitioned by year and season. We use the first three seasons samples as training data and the forth season samples as test data.
The rotation MNIST contains synthetic images of handwritten digits together with the corresponding angles (in degrees) by which each image is rotated. The input image is of size 28 28, the output is the range [-x/4, /4, ///4///// m/4). The dataset consists of 10, 000 samples. We 5, 000 for training and 5, 000 for test.
Rotation MNIST
UTKFaae 14 For UTKFace, the input is grayscale face images with the size of 91 91, and the output is the corresponding age of each face. The original dataset includes samples of people with an age ranging from 0 to 116 years old. It also includes additional personal information such gender and ethnicity. In our study, we use 7, 715 samples for training and 1, 159 samples for test, the used dataset includes people aged in a range of 80 with no additional information.
For fair comparison, we consider the same architecture in (Kolchinsky et al. 2019b). Specifically, for California Housing and Bike Sharing datasets, the encoder fenc layer fully-connected
hhttps://ww.kaaee
set
eural eoraa__
37

---

Published as a conference paper at ICLR 2024
encoder with 128 ReLU hidden units, and the decoder Gdee is fully- fully-connected layer with 128 ReLU units followed by an output layer with 1 linear unit. For the Appliance Energy and Beijing PM2.5 datasets, utilize the past 4 days of data to predict the data of the next day. The decoder remains the same, while the encoder a 3-layer LSTM with 32 hidden units followed by fully-connected layer with 128 units. The IB regularization is added to the output of the encoder. The backhone architecture for both rotation MNIST and UTKFace is VGG-16, rather than the basic fully-connected network or LSTM with only few layers.
All datasets are normalized between [0, I] with MinMaxscaler, and we set the kernel width о 1 for CS-IB and HSIC-bottlenedd which is actually common choice for HSIC literature (Greenfeld & Shalit 2020). In all experiments, we train networks with the Adam (Kingma & Ba 2015) optimizer for 100 epochs and set the batch size to 128.
In our implementation, we use the normalized CS- CS-QMI motivated by the formulation of centered kernel alignment (CKA) (Cortes et al. 2012) guarantee the dependence value between x and bounded between [0, 1]:
Ics(x;t)
[xs(x:t)
(131)
v [xs((x:x) Ics(t;t)
This strategy is also used in HSIC-bottleneck (Ma et al. 2020 Wang et al. 2021
For each competing method, the hyperparameters are selected as follows. We first choose the default values of 3 (the balance parameter to adjust the importance of the compression term) and the learning rate Ir mentioned in its original paper. Then, we select hyperparameters within a certain range of these default values that can achieve the best performance in terms of RMSE. For VIB, search within the range of 3 E [10-3 10 10-51 For NIB, square-NIB, and exp-NIB, we search within the [10 10 $1 For our CS-IB, we
range of 3 E [10 found that the best performance was always achieved with 3 between 10-2 to 10 Based on this, we sweep the 3 within this range and select the best one. The learning rate range for all methods was set as [10 3 10 10.41. We train all methods for 100 epochs on all datasets except for PM2.5, which requires around 200 epochs of training until converge. All hyperparameter tuning experiments are we record all hyperparameters for each deep IB approach that achieve the best RMSE. We train models with such hyperparameter setting to evaluate their adversarial robustness performances, as shown in Table
10 $1. For HSIC-bottleneck, we search
conducted on the validation set. In Table &
Table 4: Hyperparameters for different IB approaches that achieve best RMSE on six real-world regression datasets.
Dataset
param.
VIB
NIB
Square-NIB
exp-NIB
HSIC
CS-IB
lcarning rate epuchs
1 10
1 10
10-3
1 1 10
1 x 10-3 100
x 10 100
Housing
100
100
100
/
1 1 10
3 x 10-2
1 10-2
1 x 10-2
x 1007 x 10m>
1 x 10
learning rate epuchs B
1 10-3
1 10
1 x 10-4
1 x 10-4
1 x 10-4
1 10
Energy
100
100
100
100
100
100
10
x 10-4
10-4
2 10-4
x 10 7/4 10-2
5 10-2
learning rate epochs /
x 10
x 10
1 x 10
1 x 10
1 x 10
x 10
PM2.5
200
200
200
200
200
200
5 10
1 10-5
1 x 10-5
3 10
1s 10-3
5 x 10
learning rate epuchs B
1 1 10
1 10-4
1 10-4
1 10-4
1 x 10-4
x 10-3
Bike
100
100
100
100
100
100
3 10
5 10
6 x 10-3
2 x 10-3
1 414 10-3
x 10
learning rate epochs B
1 -3
1 10 3
1 x 10-3
1 10
10-3
1 10
Rotation MNIST
100
100
100
100
x 10-5
10-2
1 x 10-2
1 x 10-2
x 10 x 10-2
x 10-3
learning rate epuchs /
1 10
1 10-3
1 x 10-3
1 x 10-3
1 10-4
x 10-3
UTKFaace
100
100
100
100
100
100
1 1
1 10
1 x 1
1 1
1 10 x 10-
1 10
38

---

Published as a conference paper at ICLR 2024
2.25
VIB
0.7
NIB
square-NiB
0.6
1.75
ex
HSic-bottleneck
1.50
CS-IB
RMSS
RMSE
VIB
1.00
0.3
0.75
exp-NIB
0.50
HSIC bottleneck
0.25
CS IB
0.2
0.4
0.6
0.0
0.1
0.2
0.3
0.4
0.5
0.6
(a) California Housing
(b) Appliance Energy
0.7
0.5
RASE
0.4
0.3
VIB
VIB
NIB
NIB
0.2
square-NIB
square-NIB
EXp
exp NIB
exp-NIB
HSIC bottleneck
HSIC-bettleneck
CS IB
CS-IB
0.1
0.2
0.3
0.4
0.5
0.6
(c) Beijing PM2.5
(d) Bike Sharing
VIB
22
NIB
square-NIB
20
exp NIB
battleneck
RMSE
12
RMSF
16
14
12
exp-NIB
10
HSIC bottleneck
CS-18
0.0
1/255
2/255
3/255
4/255
5/255
6/255
1/255
2/255
3/255
4/255
5/255
6/255
(c) Rotation MNIST
(f) UTKFace
Figure 11: Test RMSE under FGSM attack with different six regression datasets. Our CS-IB is consistently better that other competing approaches.
C.3 ADVERSARIAL ROBUSTNESS
We further evaluate the adversarial robustness by comparing the behaviors of different IB approaches under FGSM attack with different perturbations e = [0.1, 0.3, 0.4, 0.5, 0.6]. We use the same parameter configuration in Table a and only add adversarial attack the test set. Our CS-IB outperforms other IB approaches with different perturbation strengths on all datasets as shown in Fig. 11
39

---

Published as a conference paper at ICLR 2024
C.4 ADDITIONAL RESULTS OF SECTION 4.1 AND AN ABLATION STUDY
In Tables and we complete Table 2 and provide the standard deviation (std) for each method over independent runs with different network initialization. In Table 7 we additionally provide the best performance achieved with r * 0 (i.e., B > 0) for different deep IB approaches.
We also conduct an ablation study on California Housing dataset to investigate the effectiveness of combining our prediction loss (the conditional CS divergence Des (p(y) x); go (f/x)) in Eq. (8)) with [x; t) regularization that calculated by mutual information estimators in other IB approaches. For instance, "CS-div+VIDD refers to the combination of conditional CS divergence with [x; t) calculated by variational lower bound. T we observe that replacing MSE with our conditional CS divergence Des (p(p) (y() x); qo (ū)x)) improves the performance for all competing IB approaches. This result indicates that our conditional CS divergence is more helpful than MSE in extracting useful information from predict y-
C.5 COMPARING CS DIVERGENCE WITH VARIATIONAL KL DIVERGENCE
We aim verify the advantage of CS divergence over variational KL divergence when we want to approximate an unknown distribution p(x) with another distribution q(x).
Specifically, we assume that p(x) as the true distribution consists of two Gaussians with mean vectors [4, 4)" and 1-4.-4]² and covariance matrix I2. Suppose we approximate p(x) with single Gaussian go (x) by minimizing the KL divergence Dxr. (x);p(x)) (the reverse KL divergence also in different variational IB approaches) with respect to q's variational parameters &. We initialize the variational parameters as:
6n [0, 0T Φr I2.
(132)
and optimize the following objective with gradient descent:
(x)()((x::((x))
(gx(x)) Ens (log ((o)).))).
(133)
As can be seen in Fig. b2), the minimization forces (x) to be zero where p(x) is zero and hence makes it concentrate one of the modes. This phenomenon is called mode-seeking.
Note that, single Gaussian is a common choice for variational inference with KL divergence, this is because KL divergence does not have closed-form expression for oftttr----Gaussiann (MoG). We would like to point out here that the CS divergence has foosed-fomm expression for MoG (Kampal al. 201T), which may further strength its utility. Although this property is not used in paper.
Now, we consider using CS divergence to approximate p(x) with (xx). To do this, we sample N1 = 400 samples from p(x) and initialize q(x) by sampling N₂ 200 samples from standard Gaussian. Then, we optimize samples in q(x) by minimizing the CS divergence. Because CS divergence is estimated in non-noramann -on-paraett way that does not make any distributional assumptions, can fit two Gaussians perfectly as shown in Fig. 12 d).
D LIMITATIONS AND FUTURE WORK
Given the promising results demonstrating the usefulness of CS-IB, it also has limitations.
First, we still would like to mention limitation of CS divergence with respect to the KL divergence. That is, we did not find a dual representation for the CS divergence. The dual representation of KL divergence, such as the well-known Donsker- Varadhan representation (Donsker & Varadhan 1983). enables the use of neural networks to estimate the divergence itself, which significantly improves estimation accuracy in high-dimensional space. A notable example is the mutual information neural estimator (MINE) (Belghazi et al. 2018).
Second, although kernel density estimator (KDE) offers elegant expressions for both [(x; t) and D(p(y) x); go (6) (9]x), (x). . its performance depends heavily on a proper choice of kernel width a. this paper, we normalized our data and observed that a 1 is always a reliable heuristic. We do observe that our estimators provide consistent performance gain in a reasonable range of kernel size (e.g..
1 3), illustrated in Fig. 15
40

---

Published as a conference paper at ICLR 2024
oprimization divegnnce
160
Before sptimization
100
15
58-
-21
-15-
-68
-15
-15-
10.0
-200
100
00.0
-25
-0.0
0..0
2.3
3.1
7.5
10.0
(b) After optimization with variational KL. diver- gence.
(a) Before optimization.
Belor optiiization
Afier opimication with CS divergmoo
35.0
Fland
Thed reprrtetziionn
Opcimizal
-100
-7.5
0.0
2.5
10
75
10.0
-300
-75
-5.0
-25
2.5
0.0
160
(c) Before Optimization.
(d) After optimization with CS divergence.
Figure 12: Comparison between CS CS-divereence and variational KL divergence on the simulated dataset.
There are multiple avenues for future work:
1) We would like to mention another two intriguing mathematical properties of CS divergence that both KL. divergence and traditional Rényi's divergence do not have.
CS divergence has closed-form expression for mixture-of Gaausssias (MoG) (Kampa et al. 2011
CS divergence invariant to scaling. That is, given Å A2 > we always have:
Des(p; 4) (es(A
(134)
Although both properties not used in this paper. They should be beneficial to other statistical or machine learning tasks. For example, the first property has been leveraged in (Tran et al. 2022) to train variational autoencoders replacing the single Gaussian prior distribution with MoG; whereas the second property has recently been used in (Jenssen 2024) to project and visualize high-dimensional data.
2) We would like to extend CS-IB framework to other types of data, such as graphs. Some initial works have been done.
Here, we present our initial study to predict the age of patients based on their brain functional MRI (fMRI) data with a graph neural network (GNN), but leave comprehensive and in-depth investigation as future work.
To this end, rely the brain information bottleneck (BrainIB) framework (Liu al. 2023)
framework as shown in Fig. 13 We use the original code from its author but replace the graph encoder with graph transformer network (Shi et al. 2021). We add the information bottleneck regularization on the extracted graph representations, I.c., g and luub and train the whole network by
41

---

Published as a conference paper at ICLR 2024
the following objective:
min Desce(((VV) + flcs (gi Bubb).
(135)
For other deep IB approaches, the objective is simply:
min (lyly ú133 + (g gaah).
(136)
We train all models the Autism Brain Imaging Data Exchange (ABIDE) datase which contains the fMRI data from 1, 028 patients (ages 7-64 years, median 14.7). We split the data with 60% training, 20% validation, 20% test, and normalize the age between 0 and 1. For all IB approaches, choose 3 from four values: 0.0001, 0.001, 0.01, and 0.1, and select the one that achieves the best prediction accuracy. Fig. [4] demonstratess the predicted age with respect true age for MSE loss only, NIB, HSIC-bottleneck, and our CS-IB. Table S records the quantitative evaluation results, in which the "prediction loss" indicates using (ps(p(y g): p(ūlg) only (as surrogate of MSE).
G
9
Subgraph generator
Graph Transformer
MI Estimator 1
((g, Isub)
Ssub
Ssub
Figure 13: Subgraph information bottleneck framework for age prediction. G the original brain network, G is the identified brain sub-network that reduces irrelevant edge information for pre- diction. g and Bub refers to graph vector representations corresponding G and Grub, respectively, which are learned from joint graph encoder with shared parameters. Isub is used for prediction.
Table 5: The mean absolute error (MAE) and pearson correlation coefficient (PCC) between predicted age and true age.
Method
MAE
PCC
MSE
0.079
0.564
Prediction Loss
0.067
0.571
NIB
0.072
0.573
HSIC-bottleneck
0.079
0.615
CS-IB
0.067
0.631
htttee//feoounuooo.ooeeeeeeeeeee /abi de/abi aaaaa_idee111 htm1
42

---

Published as a conference paper at ICLR 2024
0.45
035
0.90
025
0.99
62
0.15
0.7
30
02
03
11.5
135
118
127
(a) MSE
(b) NIB
3.5
66
35
64
M
833
0.25
0.10
305
02
(c) HSIC-bottleneck
(d) CS-IB
Figure 14: Scatter plot of predicted age (y-axis) with respect to true age (x-age) for each patient in the test set.
0.275
0.270
RMSE
0.265
0.260
0.255
0.250
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
0
Figure 15: Performance of CS-IB with different kernel width Californi Housing dataset.
43

---

Published as a conference paper at ICLR 2024
Table 6: Ablation study of conditional CS divergence California Housing dataset. The best performance is highlighted. By replacing MSE with our conditional CS divergence in Eq. (18) all models have a performance gain.
Method
Test RMSE
VIB
0.257
CS-div+VIB
0.255
NIB
0.251
CS-div+NIB
0.248
Square-NIB
0.255
CS-div-Square-NIB
0.251
Exp-NIB
0.253
CS-div+Exp-NII
0.250
HSIC-bottleneck
0.254
CS-div+HSIC
0.252
CS-IB
0.244
Table 7: The best performance achieved with r + 4 (i.e., 3 > 0) for different deep IB approaches.
Method
Housing
Energy
PM2.5
Bike
Rotation MNIST
UTKFaace
VIB
0.257
0.063
0.024
0.421
5.213
9.274
NIB
0,251
0.058
0.021
0.414
3,926
8.712
Square-NIB
0,255
0.060
0.023
0.420
4.102
8.673
Exp-NIB
0.253
0.058
0.024
0.423
4.325
8.927
DSSC-bootleeneck
0,254
0.062
0.029
0.417
4.685
8.793
CS-IB
0.244
0.054
0.020
0.392
3.765
8.552
Table 8: RMSE for different deep IB approaches California Housing, Appliance Energy, and Beijing PM2.5 with compression ratio r 0 and r = 0.5. When r = 0, CS-IB uses prediction term Des (p(y x); go x))) in Eq. (18 whereas others use MSE
Housing
Energy
PM2.5
Model
0.5
0
0.5
0
0.5
VIB
0.258=0.002
0.347=0.015
0.059:10.004
0.071=10.010
0.025=10.003
0.038-10.008
NIB
0.258=00.02
0.267=00010
0.059-+-0.04
0.060-10.008
0.025++0.003
0.03/1+0.005
Square-NIB
0.258-00.002
0.293+00.008
0.059++0.004
0.0633+0.005
0.0255+0.003
0.0204000004
Exp-NIB
0.258=0.002
0.287=0.010
0.059-10.004
0.061110.06
0.02511.003
0.030-10.006
HSIC-hottenck
0.258-00.002
0.371-00.006
0.059++0.004
0.065
0.025-+0.003
0.031+0.000
CS-IB
0.251=0.001
0.245-00010
0.056-10.002
0.058-00005
0.022-00.002
0.027+0.000
Table 9: RMSE for different deep IB approaches on Bike Sharing, Rotation MNIST, and UTK-
Face with compression ratio r and r = 0.5. When r 0, CS-IB uses prediction term
Des x); ())))) in Eq. 18 whereas others use MSE
Bike
Rotation MNIST
UTKFaae
Model
0.5
0
0.5
0.5
VIB
0.428=00005
0.523=0.010
4.351=:0.025
5.358-10.030
8.870=t0.050
9.258-t0.080
NIB
0.008
0.435=00.088
1.3511+0.025
1.102-10.020
8.870=10.00
8.756:10.060
Square-NIB
0..005
0.117=40.006
1.351++0.025
4.257+40.010
8.870++0.000
8.712+00050
Exp-NIB
0.028=
0.45810.000
4.351-10.05
4.285-1.0015
8.870-1.050
8.917-10050
HSiC-hottenck
0.428 +0.005
0.4511+0.000
4.351+0.025
4.573+00.05
8.870++0.050
8.852++0.055
CS-IB
0.004
0.412+-0.005
4.165+00.01
3.930-0..00
8.702-00.035
8.655+0..03
44


Reviews:
[{"review_id": "UOm6zcezpJ", "reviewer": "Reviewer_SV68", "summary": "The authors of this paper introduce a novel approach to regression using the IB principle, leveraging the Cauchy-Schwarz divergence for parameterization. This departure from MSE-based regression eliminates the need for variational approximations or distributional assumptions, leading to improved generalization and strong adversarial robustness. Their CS-IB method outperforms other deep IB approaches across six real-world regression tasks, achieving an optimal balance between prediction accuracy and compression ratio in the information plane.", "questions": "The complexity of CS-divergence estimation approach is O(N^2). How is it compared to that of the traditional KL divergence based IB method?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The authors show the connections between CS divergence to MMD and HSIC. \n- The effect of CS divergence to generalization and adversarial robustness is well quantified. \n- Thorough discussions are provided for most of remarks or theoretic findings.", "weaknesses": "- The CS divergence estimation is based on the Gaussian kernel assumption, which will depend on the parameter $\\sigma$. What is the effect of $\\sigma$ to the IB performance is not shown clearly. \n- The KL IB using variational approach is friendly to optimization based methods (gradient-based approaches). On the other hand, CS IB method is based on Gaussian kernel assumption, which may require the tuning of $\\sigma$. \n- I think it’s better to have a section of identifying what are some potential disadvantages of CS-IB method compared to KL-IB method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k7CLMIAISe", "reviewer": "Reviewer_zqzB", "summary": "The authors study information bottleneck (IB) methods for regression tasks using a new divergence (Cauchy-Schwarz). The authors showed how to algorithmically design an IB approach based on this new divergence, analyzed the theoretical properties of the new divergence, and numerically demonstrated a visible advantage over existing approaches.", "questions": "- Could the authors provide a high-level idea/motivation in the Introduction to describe why one should consider the Cauchy-Schwarz divergence instead of the more traditional KL divergence? Why is it better to avoid variational approximations?\n\n- Are there toy problems where Cauchy-Schwarz divergence performs significantly better than other traditional approaches?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The use of Cauchy-Schwarz divergence in information bottleneck approaches is reasonable and novel to my understanding.\n\nThe authors derived an efficient algorithm for training IB approaches based on the Cauchy-Schwarz divergence.\n\nThe authors demonstrate visible improvement over existing approaches.", "weaknesses": "The improvement over existing approaches is fairly limited. In many tasks, the improvement is as small as 0.1 RMSE (where the relative improvement is close to 0.01~0.04). I am not sure if the limited improvement on these datasets is particularly meaningful.\n\nIt remains conceptually unclear to me why we want to use the Cauchy-Schwarz divergence. It is shown to be always <= KL divergence, but it is not clear how much smaller would this be (maybe only minimally).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iNMZJ4AF4W", "reviewer": "Reviewer_MhfL", "summary": "- The authors considered using the Couchy--Schwarz (CS) divergence to parameterize the information-bottleneck (IB) with deep neural networks (DNNs), which is beyond the MSE (or MAE) loss based on Gaussian (or Laplace) variational distribution.\n- Furthermore, they proposed using the non-parametric estimation by plugging in the output of DNNs through the kernel density estimator (KDE) for the following two terms in their objective: the prediction term (CS divergence) $D_{\\mathrm{CS}}(p(y|x); q_{\\theta}(\\widehat{y}|x))$ and the compression term (CS-QMI) $I_{\\mathrm{CS}}$, which allows us directly estimate the MI instead of its upper bound as in existing approaches.\n- They also provided a discussion regarding the relationship between CS and MMD-based estimators.\n- They offered theoretical and empirical analyses for generalization error and adversarial robustness based on the work of [Kawaguchi et al. in 2023] and [Wang et al. in 2021].\n- They empirically confirmed the predictive and robustness performance of their CS-IB method.", "questions": "In connection with the weaknesses mentioned above, I would like to pose several questions related to the concerns raised. I would appreciate your responses.\n\n- It seems that this nonparametric approach could also be applicable to the conventional objective function based on KL divergence. Why was it not discussed, proposed, or included as a subject for comparative experiments?\n- As mentioned in Remark 1, CS divergence is a special case of Rényi divergence. Under this premise, it could have been considered as a generalization of the conventional KL-based methods using Rényi divergence, especially from the perspective that it converges to KL divergence as α approaches. Why was the emphasis placed on this particular special case?\n- The uniqueness of the KDE estimator in CS divergence has already been provided by [3]. I understand the need to construct an estimator consisting of three elements: predictions, input data, and label data, in the context of the IB method. However, it seems that there might not be such a fundamental difference between the basic nonparametric estimation and the proposed method. If these aspects are positioned as variations based on the ideas from related research, it would be beneficial to explicitly state this and discuss the differences, if any. What are your thoughts on this?\n- Can you guarantee the asymptotic properties of the two estimation methods in the proposed approach? For instance, is asymptotic unbiasedness or consistency ensured?\n- Why do the performance values for existing methods in Table 2 match perfectly? \n- Why was it not considered to conduct repeated experiments and report the mean and standard deviation?\n- How were the hyperparameters selected?\n\n## Citation\n(Note: I am not the author of the following papers)\n\n[1]: Q. Wang, S. R. Kulkarni, and S. Verdú. Divergence estimation for multidimensional densities via k-nearestneighbor distances. IEEE Transactions on Information Theory, 55(5), 2009.\nhttps://www.princeton.edu/~kulkarni/Papers/Journals/j068_2009_WangKulVer_TransIT.pdf\n\n[2]: F. Perez-Cruz, Kullback-Leibler divergence estimation of continuous distributions. 2008 IEEE International Symposium on Information Theory, Toronto, ON, Canada, 2008, pp. 1666-1670, doi: 10.1109/ISIT.2008.4595271.\nhttps://www.tsc.uc3m.es/~fernando/bare_conf3.pdf\n\n[3]: R. Jenssen, J. C. Principe, D. Erdogmus, T. Eltoft, The Cauchy–Schwarz divergence and parzen windowing: Connections to graph theory and mercer kernels, Journal of the Franklin Institute 343 (6) (2006) 614–629.\nhttps://www.sciencedirect.com/science/article/abs/pii/S0016003206000767\n\n\n================ AFTER REBUTTAL & DISCUSSION ================\n\nThe authors diligently worked to enhance their paper, successfully addressing certain concerns, such as the comparison of a non-parametric estimator.\nConsidering the substantial revisions and additional analysis, I am of the opinion that the manuscript should undergo another round of peer review to validate these changes.\nI raised my score, but with the perspective that it is just below the acceptance borderline.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- They introduced a new choice of loss function based on CS divergence for the IB method, which was often used with MSE based on Gaussian settings or MAE loss based on Laplace distribution.\n- They pointed out the challenges in estimating MI and the adoption of indirect methods, such as estimating upper bounds. They proposed an IB framework based on direct estimation by performing non-parametric estimation using KDE.\n- They pointed out that existing methods adopt indirect methods for MI estimation, such as estimating upper bounds, and they proposed an IB framework based on direct estimation by performing non-parametric estimation using KDE.\n- They attempted empirical verification regarding the correlation between CS-QMI and generalization error, providing insights into the generalization analysis of the IB method based on CS divergence.\n- They conducted performance evaluation experiments on a wide range of real-world data, including California Housing, MNIST, Beijing PM2.5, and UTKFace.", "weaknesses": "I would like to express my sincere respect for all the efforts the authors have invested in this paper. Unfortunately, however, I cannot strongly recommend this paper as an ICLR 2024 accepted paper for the following reasons: (1) a misalignment between the claims of contribution, the assumptions of theoretical analysis, and the content of theoretical analysis; (2) a lack of theoretical guarantees on the properties of the proposed estimations, and the unclear discussion of the pros and cons between direct estimation and upper bound estimation approaches, or the absence of sufficient comparative experiments to complement it; and (3) concerns regarding the reliability of the experimental results. The details are provided below. If there are any misunderstandings, I apologize, and I would appreciate it if you could explain them to me.\n\n## A misalignment between the claims of contribution, the assumptions of theoretical analysis, and the content of theoretical analysis\n- The authors claim that by leveraging the CS divergence, they can perform non-linear IB regression for any distribution $p(x, y)$ without heavy reliance on variational approximations and without making distribution assumptions. How accurate is this claim? The independence from the variational distribution seems to be achieved not so much due to the properties of the CS divergence but rather through non-parametric estimation using Kernel Density Estimation (KDE) on the outputs from neural networks, treating them as samples from the training data. Therefore, the motivation for introducing the CS divergence may lack a clear and solid rationale, leaving room for discussion regarding its validity in the context of this presentation.\n- Furthermore, while the authors claim that their contribution lies in the ability to estimate without making distribution assumptions, the actual theoretical analysis deals with a limited setting where both the model distribution and variational distribution are assumed to be normal distributions. This limitation is acknowledged as a one of limitations in Appendix D. However, it can be considered a significant weakness that the proposed method's theoretical guarantees are limited, given the broad claim of removing the necessity of distribution assumptions. This disparity should ideally be addressed in the main part of the paper.\n- For one of the contributions mentioned, I believe the explanation of the adversarial robustness aspect is rather too concise. Besides Lemma 1, there should be a proper proof for [Wang et al., 2021] Theorem 2's Corollary as well. In the current presentation, while one may vaguely understand the content being discussed, the explanations are too informal, making it difficult to accurately determine whether the validity is established. The theoretical part of the proof should ideally be self-contained within the main paper, rather than relying on related literature.\n- The assumptions underlying generalization analysis and adversarial robustness have not been well-organized. It is recommended to consolidate these assumptions.\n\n## A lack of theoretical guarantees on the properties of the proposed estimations, and the unclear discussion of the pros and cons between direct estimation and upper bound estimation approaches, or the absence of sufficient comparative experiments to complement it\n- When constructing nonparametric estimators, a critical concern always revolves around whether (asymptotic) unbiasedness or (asymptotic) consistency is guaranteed. This theoretical assurance holds true for nonparametric estimations like Kernel Density Estimation (KDE) and k-Nearest Neighbors (k-NN) based on KL divergence [1,2], among others. In the context of the IB method under focus here, I presume that unbiasedness and consistency play a significant role in determining its performance. The importance of analyzing this aspect lies in the bias introduced by parameter estimation in variational approaches dependent on the variational distribution, as well as the bias in the estimators of the proposed method. Deciding which of these biases is smaller - the one induced by parameter estimation or the one introduced by the proposed method - is a crucial perspective in determining whether a direct estimation approach, like the proposed method, or a variational estimation approach is more useful. Unfortunately, this paper lacks comprehensive discussions on this aspect. The lack of convincing arguments on why a direct estimation approach like the proposed method is effective. This issue holds immense significance in the comparison between the proposed method and conventional methods, and it remains one of the unresolved problems in this paper, marked as a limitation. Relevant questions pertaining to this matter are summarized in the Question section.\n- When comparing the bias introduced by the variational approach with that introduced by nonparametric estimations such as the proposed method, it would be beneficial to investigate a toy example with an increased sample size to determine which approach exhibits higher estimation accuracy. Unfortunately, this paper did not provide such experimental confirmations.\n- The authors, I believe, are understood, but the theory in Section 3.2.1 assumes that the generalization error upper bound is derived in a manner that includes the objective function of the proposed method, which alone does not guarantee generalization performance, as is evident. As mentioned in the Strength section, the attempt to experimentally verify this aspect is a fascinating endeavor. However, the empirical correlations have only been validated on toy data. To increase the persuasiveness of the empirical evidence regarding the correlation with generalization performance, it would have been necessary to confirm the experimental results on benchmark data adopted in Section 4.\n\n## Concerns regarding the reliability of the experimental results.\n- Due to the absence of reports on measures of dispersion, such as standard deviations, for all the experimental results in this paper, it is difficult to determine whether the proposed method consistently achieves superior performance compared to other methods. This raises concerns about the validity of the claim made by the authors as one of their contributions, that CS-IB outperforms other methods. Since the proposed method is built on nonparametric estimation, it is conceivable that the variance of the estimates can become large when there are not enough samples, leading to increased variability in performance. In essence, to support the claim that the proposed method is superior, it would be challenging to agree without considering factors such as the randomness of model initialization, mini-batch datasets, and adversarial perturbations. It is advisable to repeat the experiments several times, report the average and standard error of the obtained prediction accuracy, and evaluate performance while taking variability into account.\n- I find it puzzling that some of the reported numerical values in the experiments are identical in every aspect. For instance, in Table 2, the RMSE for the existing methods when $r=0$ is a complete match across all experimental settings. This further emphasizes the need for reporting both the average prediction accuracy and the standard error. While the average prediction accuracy may be the same, the standard error might differ between methods. If they do indeed match perfectly, it is advisable to check for any implementation bugs. At the very least, there are concerns regarding the credibility of the currently reported numerical values.\n- I apologize if I missed any details earlier, but it does appear that there is a lack of explanation regarding hyperparameter settings, such as in Table 4. This omission can give the impression that there might be arbitrariness in the experimental results. If you have indeed eliminated arbitrariness, it would be beneficial to provide more detailed explanations about how you selected the reported hyperparameters. This would help enhance the transparency and credibility of your findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m7yTwT4qm9", "reviewer": "Reviewer_uyPi", "summary": "I’m unfamiliar with entropy-based methods. So my reviews are purely based on reading the paper without comparison to the related literature.\n\nThis paper uses the CS divergence of IB for regression estimation. It can be regarded as empirical CS divergence minimization on the conditional distribution $p(y|x)$ with a regularization term on $I(x,t)$. This problem can be efficiently estimated via kernel density estimator. The effectiveness of this metric is demonstrated on four benchmark datasets and two image datasets.", "questions": "See the above", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- the CS divergence is smaller than KL divergence under Gaussian distribution as well as some relatively general setting (e.g., $p$ and $q$ are sufficiently small)\n- The rationality of the regularization term is verified on generalization and adversarial robustness", "weaknesses": "Since I’m not an expert in this community, it appears difficult for me to evaluate the contribution. I only have few questions on this work:\n\n- The CS divergence is based on the Cauchy-Schwartz inequality, a special case of Holder inequality. Does this metric can be extended in the general $L_p$ space?\n- Theorem 1 is restricted to Gaussian distribution. Though this assumption can be relaxed if $p$ and $q$ are sufficiently small, it would be possible to extend to the sub-Gaussian, sub-exponential case?\n- For the generalization bound, the authors obtain a tighter bound but $I_{CS}(x;t)$ and $I(x;t)$ are still in the same constant order?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2HpTqwnZc7", "reviewer": "Reviewer_aubm", "summary": "The paper discusses an information bottleneck (IB) approach for neural network-based regression, using Cauchy-Schwarz divergence in place of Kullback-Leibler divergences and estimating these using kernel methods. The authors connect their approach to several methods from the literature (MMD, HSIC, etc.) and illustrate in their results that their approach is able to outperform IB approaches based on other approximations of (KL-based) mutual information.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper belongs to an interesting branch of literature, namely deep learning with information-theoretic cost functions. While the use of kernel methods in this regard is not new, the usage of Cauchy-Schwarz divergence is novel as far as I know. The paper is written quite accessibly, and important connections are made between the proposed approach and the existing related work. The results seem to indicate that the proposed methods can outperform existing approaches based on IB and its approximations.", "weaknesses": "There are a few aspects of the work that prevent me from recommending publication at this stage. I look forward to reading the authors replies to my concerns, upon which I may improve my score.\n* In Remark 3, the authors clearly state that a kernel width of $\\sigma=0$ reduces the CS divergence between the true posterior and the decoder distribution to classical MSE. With this in light, it is not clear in what respect CS divergence is fundamentally different from estimating $I(y;t)$ using a Gaussian variational distribution (limits novelty). Also, it is not clear why the CS-IB outperforms other methods also for $r=0$.\n* In Remark 4, the authors claim that the use of KL divergence combined with a Gaussian assumption is \"likely to induce inductive bias\". The same can be said about using CS divergence with Gaussian kernels (which resonates with Remark 3, for which CS divergence reduces to MSE for $\\sigma=0$). Specifically, at a very superficial glance it appears that the inductive biases of KL+Gauss and CS+Gauss are not too different. I would appreciate a paragraph explaining the fundamental differences (from the perspective of inductive bias).\n* While I agree that CS-QMI is a rigorous definition, I do not agree that the proposed estimator measures mutual dependence in bits. As has been known for some time (see, e.g., Th. 1 in Amjad and Geiger or the work of Goldfeld et al.), for deterministic networks true MI between $x$ and $t$ is infinite. I doubt that $D_{CS}$ remains finite in such a setting (e.g., Renyi divergence is infinite if the joint and product of marginal distributions are singular w.r.t. each other, cf. Th. 24 in van Erven and Harremoes). From that perspective, I do not see how your estimators based on CS divergence have substantial mathematical advantages over existing estimators (although I acknowledge that the estimation may still be useful from a practical perspective; also, I understand that your networks are stochastic, because $t$ is obtained by adding Gaussian noise to a decoder output).\n* One of my main concerns is the comparison of methods in Table 2 and Fig. 2. As far as I can see, $r$ is defined via the respective estimators (i.e., CS divergences for CS-IB, variational costs for VIB, etc.). This makes a comparison difficult, as the same $r$ indeed may lead to different compression ratios when compression is measured in a common currency (e.g., CS divergence for all methods). While this is problematic only for Table 2, in Fig. 2 the same holds for $I(y,t)$. Specifically, CS-IB is the only method that does not use the MSE, for the estimation of $I(y;t)$ which explains the difference at $r=0$. Essentially, the red line may be just above all others because $I(y;t)$ is measured differently (namely, using CS divergence). Also here a common currency would be better, such as MSE or RMSE. (Further, using the RMSE would allow for a better comparison with Table 2.)\n\n_EDIT:_ Improved my score after rebuttal phase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)