Sufficient and Disentangled Representation Learning
Authors: Jian Huang
Yuling Jiao
Xu Liao
Jin Liu
Zhou Yu
Venue:  (2021)
Keywords: Conditional independence
f-divergence
rotation invariant
neural network
statistical guarantee

Abstract: We propose a novel approach to representation learning called sufficient and disentangled representation learning (SDRL). With SDRL, we seek a data representation that maps the input data to a lower-dimensional space with two properties: sufficiency and disentanglement. First, the representation is sufficient in the sense that the original input data is conditionally independent of the response or label given the representation. Second, the representation is maximally disentangled with mutually independent components and rotation invariant in distribution. We show that such a representation always exists under mild conditions on the input data distribution based on optimal transport theory. We formulate an objective function characterizing conditional independence and disentanglement. This objective function is then used to train a sufficient and disentangled representation with deep neural networks. We provide strong statistical guarantees for the learned representation by establishing an upper bound on the excess error of the objective function and show that it reaches the nonparametric minimax rate under mild conditions. We also validate the proposed method via numerical experiments and real data analysis.

Under review as a conference paper at ICLR 2021
SUFFICIENT AND DISENTANGLED REPRESENTATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We propose a novel representation learning approach called sufficient and disen- tangled representation learning (SDRL). With SDRL, we seek data representa- tion that maps the input data to lower-dimensional space with two properties: sufficiency and disentanglement. First, the representation is sufficient in the sense that the original input data is conditionally independent of the response or label given the representation. Second, the representation is maximally disentangled with mutually independent components and is rotation invariant in distribution. We show that such a representation always exists under mild conditions on the input data distribution based on optimal transport theory. We formulate an objec- tive function characterizing conditional independence and disentanglement. This objective function is then used to train a sufficient and disentangled representa- tion with deep neural networks. We provide strong statistical guarantees for the learned representation by establishing an upper bound on the excess error of the objective function and show that reaches the nonparametric minimax rate under mild conditions. We also validate the proposed method via numerical experiments and real data analysis.
INTRODUCTION
Representation learning is a fundamental problem in machine learning and artificial intelligence (Bengio et al., 2013).. Certain deep neural networks are capable of learning effective data represen- lation automatically and achieve impressive prediction results. For example, convolutional neural networks, which can encode the basic characteristics of visual observations directly into the net- work architecture, is able to learn effective representations of image data (LeCun et al., 1989). Such representations in turn can subsequently used for constructing classifiers with outstanding perfor- mance. Convolutional neural networks learn data representation with simple structure that captures the essential information through the convolution operator. However, other application domain- optimizing the standard cross-entropy and least squares loss functions do not guarantee that the learned representations enjoy any desired properties (Alain & Bengio 2016). Therefore, it is im- perative to develop general principles and approaches for constructing effective representations for supervised learning.
There is growing literature representation learning in the context deep neural network modeling. Several authors studied the internal mechanism of supervised deep learning from the perspective of information theory (Tishby & Zaslavsky 2015 Shwartz-Ziv & Tishby 2017 Saxe al.) 2019). where they showed that training deep neural network that optimizes the information bottleneck (Tihbby et al., 20000) is trade-off between the representation and prediction at each layer. To make the information bottleneck idea more practical. deep variational approximation of information bottleneck (VIB) is considered in Alemi al. (2016). Information theoretic objectives describ- ing conditional independence such as mutual information are utilized loss functions to train representaiion-learingg function, i.c., an encoder in the unsupervised setting (Hjelm et al., 2018 Oord et al.) 2018 Tschannen et al.] [2019] Locatello et al., 2019: Srinivas et al.) 2020). There are several interesting extensions of variational autoencoder (VAE) (Kingma & Welling (013) in the form of VAE plus regularizer, including beta-VAE (Higgins et 2017), Annealed- VAE (Burgess
et al. 2018), factor- VAE (Kim Mnih (018), beta-TC- (Chen et al. 2018), DIP-VAE (Kumar
al.
2018 The idea of using latent variable model has also been used in adversarial auto-

---

Under review as a conference paper at ICLR 2021
encoders (AAE) (Mkkhzani et al.) 2016) and Wasserstein auto-encoders (WAE) (Tolstikhin et al. 2018 However, these existing works focus on the unsupervised representation learning.
A challenge of supervised representation learning that distinguishes it from standard supervised learning is the difficulty in formulating a clear and simple objective function. In classification, the objective is clear, which is minimize the number of misclassifications; in regression, least squares criterion for model fitting error is usually used. In representation learning, the objective is different from the ultimate objective, which is typically learning classifier regression function for prediction. How to establish simple criterion for supervised presentation learning has remained an open question (Bengio al. 2013).
We propose sufficient and disentangled representation learning (SDRL) approach in the context of supervised learning. With SDRL, we seek data representation with two characteristies: suffi- ciency and disentanglement. In the context of representation learning, sufficient means that good representation should preserve all the information This is basic requirement and long-standing principle in statistics. This is closely related to the fundamental concept of sufficient statistics in parametric statistical models (Fisher [922). A sufficient representation can be naturally characterized by the conditional independence principle, which stipulates that, given the representation, the original input data does not contain any additional
the data about the supervised learning task.
information about the response variable.
In addition to the basic sufficiency property, the representation should have simple statistical struc- ture. Disentangling is based the general notion that some latent causes underlie data generation process: although the observed data are typically high-dimensional, complex and noisy, the un- derlying factors are low-dimensional, independent and have relatively simple statistical structure. There is a range of definitions of disentangling (Higgins et al. 2018 Eastwood & Williams) 2018
Ridgeway Mozer 2018: Do & Tran 2020). Several metries have been proposed for the evaluation of disentangling. However, none of these definitions and metrics have been turned into empirical criterions and algorithms for learning disentangled representations. We adopt simple definition of disentangling which defines representation to be disentangled if its components are independent (Achille & Soatto (018). This definition requires the representation to be maximally disentangled in the sense that the total correlation is zero, where the total correlation is defined as the KL di- vergence between the joint distribution of gxx) and the product of the marginal distributions of its components (Watanabe 1960).
In the rest of the paper, we first discuss the motivation and the theoretical framework for learning sufficient and disentangled representation map (SDRM). This framework leads to the formulation of an objective function based on the conditional independence principle and a metric for disentan- glement and invariance adopted in this work. We estimate the target SDRM based the sample version of the objective function using deep neural networks and develop an efficient algorithm for training the SDRM. We establish an upper error bound on the measure of conditional independence and disentanglement and show that it reaches the nonparametric minimax rate under mild regularity conditions. This result provides strong statistical guarantees for the proposed method. We validate the proposed SDRL via numerical experiments and real data examples.
2 SUFFICIENT AND DISENTANGLEI REPRESENTATION
Consider pair of random vectors (x, y) E TRP R&, where x is a vector of input variables and is vector of response variables or labels. Our goal find a sufficient and disentangled representation of x.
Sufficiency We say that a measurable map g : RP - Rdd with d S p is a sufficient representation of x if
(1) that is, y and x are conditionally independent given gxx). This condition holds if and only the conditional distribution of y given and that of given gxx) are equal. Therefore, the information in x about y is completely encoded by gx)). Such a g always exists, since if we simply take g(x) = x, then (I) holds trivially. This formulation is nonparametric generalization of the basic condition in sufficient dimension reduction (Li 1991 Cook 1998), where it is assumed g(x) = B a with B TRpxd belonging to the Stiefel manifold, i.e., B B la-
y x
2

---

Under review as a conference paper at ICLR 2021
Denote the class of sufficient representations satisfying ① by
F [g IR + I measurable and satisfies y 11 xlg(x)).
We refer to F as a Fisher class because of its close connection with the concept of sufficient statistics
Fisher 1922. Cook 2007). For an injective measurable transformation T I2 + R and g E F, T' Toog g x is also sufficient by the basic property of conditional probability. Therefore, the Fisher
class F is invariant in the sense that
F=F=F, provided T is injective,
where ToF = = 9 transformations, T Tog- Ag b. where A is a d x d nonsingular matrix and b RR
9 E FJ. An important class of transformations is the class of affine
Disentanglement We focus on the disentangled representations among those that are sufficient. Therefore, we start from the functions of the input data that are sufficient representations in the Fisher class F. For any sufficient and disentangled representation gxx), let Eg = Var(g(x)). Since the components of g(x) are disentangled in the sense that they are independent, Mg a diagonal ma- 1/2 trix, thus -g gxx) also has independent components, Therefore, we can always rescale g(x) such that it has identity covariance matrix. To further simplify the statistical structure of representation g. we also require it to be rotation invariant in distribution, that is, Qg(x) g(x) in distribution for any orthogonal matrix Q E Rdddd The Fisher class F is rotation invariant in terms of condi- tional independence, but not all its members are rotation invariant in distribution. By the Maxwell characterization of the Gaussian distributions (Maxwell, 1860 Bartlett 1934 Bryc 1995 Gyenis 2017). random vector of dimension two more with independent components is rotation invari- ant in distribution if and only if it is Gaussian with zero mean and a spherical covariance matrix. Therefore, after absorbing the scaling factor, for sufficient representation map to be disentangled and rotation invariant, is necessarily distributed as Na(0, Ia)- Let M be the Maxwell class of functions g : Rd Rd, where g(x) is disentangled and rotation invariant in distribution. By the Maxwell characterization, we can write
M [g R'
RR (10,)~N(Oaaa)))
Now our problem becomes that of finding a representation in F n,, the intersection of the Fisher class and the Maxwell class.
The first question to ask is whether such representation exists. The following result from optimal transport theory provides an affirmative answer and guarantees that F FMM is nonempty under mild conditions (Brenier 1991 McCann] 1995 Villani 2008).
Lemma 2.1. Let μ be probability measure Rd. Suppose it has finite second moment and is absolutely continuous with respect to the standard Gaussian measure, denoted by Tia Then it admits unique optimal transportation map T Rdd - Bd such that Tæj =aa (0, Ia), where T4и denotes the pushforward distribution of μ under Moreover, T is injective y-almost everywhere.
Denote the law of random vector z by Mz- Lemma 2.1 implies that, for any 9 F with (x) < co and Mg(x) absolutely continuous with respect to Tas there exists a map T trans- forming the distribution of gxx) to (0, Ia). Therefore, H² := T og E F,M, that is,
x 1l Rx and R R*(x) N Lz).
(3)
ie., RR is sufficient and disentangled representation map (SDRM).
3 OBJECTIVE FUNCTION FOR SDRL
The above discussions lay the theoretical foundation for formulating objective function that can be used for constructing a SDRM R' satisfying 3 or cquivalently, R FFM.
Let V be a measure of dependence between random variables and with the following properties:
(a) V[x, y] 2 with V(x, yl = if and only if x L y; (b) V(x.yl y] 2 ([ y] for all measurable function R; (c) V y = ( (x). if and only if R E F. The properties (a)-(c) imply that
Rd E F
R E arg max V R(x). y]
arg min - V [[())
R
R
3

---

Under review as a conference paper at ICLR 2021
We use a divergence measure DD quantify the difference between (a(()) and Ya, as long this mea- sure satisfies the condition D((R(x) lIra) 2 0 for all measurable function R and (()[(()])aa)) = if and only if E M.
Then the problem of finding an R E F M can be expressed as constrained minimization problem:
arg R
min - VIR(x), subject to D(uR(w)l O.
Its Lagrangian form is
((R) = v] + (D(un())laa)
(4)
where À 2 O is tuning parameter. This parameter provides balance between the sufficiency property and the disentanglement constraint. small X leads to representation with more emphasis on sufficiency, while a large À yields representation with more emphasis on disentanglement. We show in Lemma 4. below that any R satisfying 3 is a minimizer of (Rt... Therefore, we can train SDRM by minimizing the empirical version of L(R).
There are several options for V with the properties (a)-(c) described above. For example, we can take V to be the mutual information V[[((x. y] I I(R(x):y). However, in addition to the estimation of the SDRM R. this choice requires the estimation of the density ratio between Rxx)) and p(y )p( R(x)), which is not easy task. We can also use the conditional covariance operators on reproducing kernel Hilbert spaces (Fukumizu et al.) 2009). To be specific, in this work we use the distance covariance (Székely et al. 2007) of y and R(x), which has an elegant U-statistic expression, does not involve additional unknown quantities and is easy to compute. For the divergnce measure of two distributions, use the f-divergence (Ali & Silvey 1966), which includes the KL- divergence special case.
4 LEARNING SUFFICIENT AND DISENTANGLED REPRESENTATION
We first describe some essentials about distance covariance and f-divergence.
Distance covariance We first recall the concept of distance covariance (Székely et al. 2007 which characterizes the dependence of two random variables.
Let i be the imaginary unit (-1)1/2. For any t E Idd and s E R", let (t) = E[exp it" "1, (s) = E Eleeppe VI, and y (t, s) = E[exp z+a" y)] be the characteristic functions of random vectors E Rd, y R2, and the pair (z,y). respectively. The squared distance covariance V/z, y] is defined as
V/z,y] ffc
Je,,y(t, (t, 8) 2/2 (t)) dtds, where ca I[(d + 1)/2)
[d+11//2
Ratm caCm || d| ||||||000000 |+
Given n i.dd copies { of y), an unbiased estimator of is the empirical distance covariance Dm, which can be elegantly expressed U-statistic (Huo & Székely 2016)
V.lzyll
1
Σ
h ((Zn) (Zu. Via),
(5)
Cl
ish shsciiinn
where h is the kernel defined by à ((#1, ...., (zg, y4)) = =
1515544 (F
||z| - -21|ll31 yill
- Et Σ 15554 ||z >,11Σ
15154
Ily: - -,,l)) + सी Σ
155554 ||| - 2z 2 1srus4 1n4
13 - y,
193
f-divergence Let j and be two probability measures on Rd The fddivergence Ali & Silvey
1966) between and 7 with < 7 is defined as (u(lr) Sge f( where f R I - R
differentiable convex function satisfying f11 = 0. Let f be the Fenchel conjugate of
[Rockafellar] 1970), defined as (*) sup EE (tx - f(r)]. E R. The f-divergence admits the
following variational formulation (Keziou 2003 Nguyen al. 2010 Nowozin et al. 2016).
Lemma 4.1.
Dy(ph)
max >dom())
Ezp [(()] If=
(6)
where the maximum is attained at D(2) r' (2)).
4

---

Under review as a conference paper at ICLR 2021
Commonly used divergence measures include the Kullback-Leibler (KL) divergence, the Jensen- Shanon (JS) divergence and the -iveieeree
Learning SDRM We now ready to formulate empirical objective function for learning SDR-
M Rr. Let R E M, where M is the Maxwell class defined in (2). By the variational formulation
6 write the population version of the objective function 44
[(R) V(R(x),
max D:Rd (dom(()
[
DI ())) E (D(w)))
This expression is convenient since we can simply replace the expectations by the corresponding empirical averages.
Theorem 4.2. We have R² E arg minneM ((R) provided (3)
holds.
According to Theorem 4.2 it is natural to estimate I2 based on the empirical version of the objective
function (7) when a random sample (xx,,))) is available. We estimate R using deep neural networks. We employ two networks as follows:
Representer network Rg: This network used for training R Let R be the set of such neural networks Rg IRP TRd.
Discriminator network Dæ: This network is used as the witness function for checking whether the distribution of the estimator of R is approximately the same as W Ia).
Similarly, denote D as the set of such neural networks D& R² - R.
Let {w;}?=1 be i.i.d random vectors drawn from 7dd The estimated SDRM defined by
Ro E arg min
((Ro) = - Ro (x). + (M
(8)
ROER
where Ro (x), (x), y] is an unbiased and consistent estimator of V) y] defined in based on (Ro(x,).)))) 1, .n...,n] m) and
in (y(mRo(x) 17a)
max ( D&ED n
Statistical guarantee Since SDRM R is only identifiable up to orthogonal transforms under the constraint that R (xx (O, I.). no consistency results for Re itself can obtained. But this not a flaw of the proposed method. Indeed, the most important statistical guarantee of the learned R" that the objective conditional independence and disentanglement is achieved. Therefore, we establish an upper bound on the excess risk L - C(R") of the deep nonparametric estimator Re in (8). We make the following assumptions.
(A1) For any
there is constant Bi such that px Lipschitz continuous on [-B1, B. with Lipschitz constant L1.
and Rt is
(A2) For R M, assume r(z) dun(s)
drd (z) is Lipschitz continuous on [-B1, B,} with Lipsehitz constant L2, and 0 < c1 < r(z) S c2.
Denote B₂ max( B3 = 1ist B, (s))
The specifications of the network parameters, including depth, width, size and the supremum norm over the domains of the representer Re and the discriminator D& are given in Appendix B.
Theorem 4.3. Suppose X > O and X O(1). Suppose conditions (A1)-(A2 hold and set the network parameters according to (i)- Then
E wes.,, where C' is a constant that depends B1, B2 and B3 but not n, 9, and d.
L(R)) I +((L)+2)) I + L2 ziz).
The proof of this theorem is given in Appendix B. The result established in Theorem provides strong statistical guarantees for the proposed method. The rate -2((+t) matches the minimax non- parametric estimation rate for Lipschitz class contained in R of functions (Stone) 1982 Tsybakov 2008). Up to a log n factor, the rate (log n n)n -2/(2++) 2/(2+ matches the minimax rate nonparametric estimation of Lipschitz densities via GANs (Singh al. 2018
Liang 2018).
5

---

Under review as a conference paper at ICLR 2021
5 COMPUTATION
We can update 0 and & alternately as in training GANs (Goodfellow et al. 2014). However, this ap- proach suffers from the instability issues. In our implementation, we utilize the more stable particle method based on gradient flow (Gao et al. 2019 2020). The key idea is to find a sequence of nonlin- + sv(z,, pushing the samples from (Ra(x) to the target distribution Na along velocity fields v(z)
ear but simpler residual maps, say T(z)
(r(z)) that most decreases the f-divergence Dy (-Ihaa at A (x)) The residual maps can be estimated via deep density-ratio estimators, which take the form T(z)= z +z)),, 2 E Rd, where is step size and ((z) VF()) Here f(z) ) is an estimated density ratio of the density of Rx(x) at the current value of 0 over the density of the reference distribution. We use T to transform z, (x)x,), I...., n into Gaussian samples.
Once this is done, we update via minimizing the loss Dn, (x). = | z, 11 /n. We describe the algorithm below.
Input (x,, yi 1= Tuning parameters: X, d. Sample {wi) Tdd.
Outer loop for O
-Inner loop (particle method)
*
Let z; R2(x)))))))))))))
..., 1.,....,
Solve Do E arg minq, E 1 (log + exp exp Po(ni)) + log(1 + exp (o(w))))
*
Define the residual map T(z) sf= with f(z) = exp Bx(2)
*
*
Update the particles 2; = TT(,,), i 100==1,.. 1,2...,
End inner loop
Update via minimizing - - + + [ 2:ll//n using SGD.
End outer loop
6 EXPERIMENTS
We evaluate the proposed SDRL with the KL--dieegeecce using both simulated and real data. The goal of our experiments is to demonstrate that the representations trained based the proposed method perform well. Our proposed method is not trying to learn a classifier or regression function directly, but rather to learn representation that preserve all the information. So our experiments are designed to evaluate the performance of simple classification and regression methods using the representations we learned as input. The results demonstrate that a simple classification or regression model using the representations we trained performs better than or comparably with the best classification or regression method using deep neural networks.
Details on the network structures and hyperparameters are included in Appendix A. Our experiments were conducted on Nvidia DGX Station workstation using single Tesla V100 GPU unit. The PyTorch code of SDRL is available at http://///.tttttoooooo/////o////
com wtt.////nooommu/sssi...i
6.1 SIMULATED DATA
In this subsection, we evaluate SDRL on simulated regression and classification problems.
Regression We generate 5, 000 data points from two models. Model A: y x1 x1 + (xg + (x²+
x1)x (1 + oE. where x N (0, I,); Model B:y sin (πx1 + 1) + de, where x ~ Uniform((0, 1)²). In both models, ε ~ N (0, I4). We use 3-layer network with ReLU activation for Ro and single hidden layer ReI.U network for Dz- We compare SDRL with two prominent sufficient dimension reduction methods: sliced inverse regression (SIR) (Li 1991) and sliced average variance estimation (SAVE) (Cook& & Weisberg 1991). We fit a linear model with the learned features and the response variable, and report the prediction errors in Table We see that SDRL outperforms SIR and SAVE in terms of prediction error.
Classification We visualize the learned features of SDRL on two simulated datasets. We first generate (1) 2-dimensional concentric circles from two classes as in Figure((aa) (2) 2-dimensional moons data from two classes in Figure T (e); (3) 3-dimensional Gaussian data from six classes

---

Under review as a conference paper at ICLR 2021
Table 1: Averaged prediction errors and their standard errors (based 5-fold validation).
Model A
Model B
Method
0 = 0.1
a = 0.4
= 0.8
0 0.1
a = 0.2
0 = 0.3
SDRL.
1.101 + .199
1.179 + .117
1.401 +
0.149
.050
0.231
.025
0.325 = .026
SIR
1.521 .133
1.614 +: 223
1.704 + .095
0.266 + .003
0,319
.004
0.391
- .010
SAVE
1.521 .134
1.614 1: .211
1.702 1 .098
0.266 1: .003
0.319 1 .004
0.391 - .010
as in Figure ()). In each dataset, we generate 5,000 data points for each class. We next map the data into I00-dimensional space using matrices with entries i.i.d Unifrom 1]). Finally, we apply SDRL to these 100-dimensional datasets to learn 2-dimensional features. We use a 10-layer dense convolutional network (DenseNet) (Huang et al. 2017) as Rg and a 4-layer network as Dø- We display the evolutions of the learned 2-dimensional features by SDRL in Figure [I For case of visualization, we push all the distributions onto the uniform distribution on the unit circle, which done by normalizing the standard Gaussian random vectors to length one. Clearly, the learned features for different classes in the examples are well disentangled.
(a) Epoch 0
(b) Epoch = 10
(c) Epoch = 30
(d) Epoch =500
(c) Epoch =
(f) Epoch = 10
(g) Epoch = 30
(h) Epoch 500
(i Epoch 0
(i) Epoch 10
(k) Epoch = 30
(1) Epoch = 500
Figure 1: Evolving learned features. The first, second and third rows show concentric circles, moons and 3D Gaussian datasets, respectively.
6.2 REAL DATASETS
Regression We use a benchmark YearPredictionMSDD dataset to demonstrate the prediction per- formance of SDRL lattps://archive ics uci. /tt.:ett//////daatttsseeeaeeeeee This dataset has 515,345 observations with 90 predictors. The problem is to predict the year of song release. We randomly split the data into five parts for cross validated evaluation of the predic- tion performance. We employ 3-layer network for both D& and Rş. A lincar regression model fitted using the learned representations and the response. The mean prediction errors and their standard errors based SDRL, principal component analysis (PCA), sparse principal component analysis (SPCA) and ordinary least squares (OLS) regression with original data are reported Table 2 SDRL outperforms PCA. SPCA and OLS in terms prediction accuracy.
Classification We benchmark the classification performance of SDRL using MNIST (LeCun et al.) 2010), FaShionMNIST (Xiao et al. 2017), and CIFAR-10 (Krizhevsky al. 2009 against altema-

---

Under review as a conference paper at ICLR 2021
Table 2: Prediction error + standard error: YearPredictionMSSD dataset
Methods
d = 10
d = 20
d = 30
d = 40
SDRL
8.8 + 0.1
9.2 - 0.8
9.2 0.8
8.8 + 0.1
SPCA
10.6 - 0.1
10.4 0.1
9.6 2 0.1
10.2 + 0.1
PCA
10.6 0.1
10.4 0.1
10.3 10.30..1 0.1
10.2 0.1
OLS
9.6 +0.1
tive methods including convolutional networks (CN) and distance correlation autoencoder (dCorAE) [Wang et al., 2018). With CN, we use the feature extractor by dropping the cross entropy layer of the DenseNet trained for classification. The MNIST and FashionMNIST datasets consist of 60k and 10k grayscale images with 28 x 28 pixels for training and testing, respectively, while the CIFAR-10 dataset contains 50k and 10k colored images with 32 32 pixels for training and testing, respec- tively. For the learning from scratch strategy. the representer network Ro has 20 layers for MNIST data and 100 layers for CIFAR- 10 data. We apply the transfer learning technique to the combination of SDRL and CN on CIFAR- 10 (Krizhevsky et al.) 2009). The pretrained WideResnet- 101 model Zagoruyko & Komodakis 2016) on the Imagenet dataset with Spinal FC (Kabir et al. 2020) is adopt for Rg. The discriminator network D& is 4-layer network. The the architecture of Rg and most hyperparameters are shared across all four methods SDRL, CN, SDRL+CN and dCorAE. Finally, we use the k-nearest neighbor (A 5) classifier on the learned features for all methods.
The classification accuracies are reported in Tables 3 and We can see that the classifica- tion accuracies of SDRL comparable with those of CN and dCorAE. As shown in Table 4 the classification accuracies CN leveraging SDRL outperforms those CN. We also cal- culate the estimated distance correlation (DC) between the learned features and their labels as P2yy ylz /V/223 x V[y]?), where V(z] and V/yl are the distance variances such that Vizl = V(z, z). V[y] Vly.yl, For more details, please see SSzkklly et al. 2007). Figure shows the DC values MNIST, FashionMNIST and CIFAR-10 data. SDRL and SDRL+CN achieves higher DC values.
Table 3: Classification accuracy for MNIST and FashionMNIST
MNIST
FashionMNIST
d
SDRL
dCorAE
CN
SDRL
dCorAE
CN
d 16
99.41
99.58
99.39
94.44
94.18
94.21
d 32
99.61
99.54
99.45
94.18
93.89
94.41
64
99.56
99.53
99.49
94.13
94.24
94.38
Table 4: Classification accuracy for CIFAR-10 data
Learning from scratch
Transfer learning
d
SDRL.
dCorAE
CN
SDRI.
CN
SDRL+CN
= 16
94.29
94.15
94.21
97.52
97.4
97.68
d 32
94.58
94.18
94.92
97.33
97.79
97.96
64
94.46
94.66
95.09
97.49
97.90
97.91
7 CONCLUSION AND FUTURE WORK
In this work, we formulate framework for sufficient and disentangled representation learning and construct an objective function characterizing conditional independence and disentanglement, This enables learn representation with the desired properties empirically. We provide statisti- cal guarantees for the learned representation by deriving an upper bound on the excess risk of the objective function.
8

---

Under review as a conference paper at ICLR 2021
0.90
Datano Combetto
6:90
Diuune cuneldion
0.55
0..00
0.75
0.75
0.70
6.70
0.80
cCasAE
SERIL
OCCrAE
0.60
0.50
50
150
200
300
100
250
Epooh
Epocc
(a) MNIST, d = 16
(b) Fashion MNIST, 16
0.8
0.50
Diganee Condare Condatte
0.7
0.70
64
oCor/E
52RE
0.68
SDRL-CN
100
250
200
230
300
10
20
90
Epocc
Epoch
(c) CIFAR-10 (from scratch), = 16
(d) CIFAR-IO (transfer learning), d = 16
Figure 2: The distance correlations of labels with learned features based on SDRL, CN, SDRL+CN and dCorAE for FashionMNIST and CIFAR-10 data.
There are several questions that deserve further study. First, we can adopt different measures of conditional independence including mutual information and conditional covariance operators producing kernel Hilbert spaces (Fukumizu et al. (2099). We can also use other divergence measures such the Wasserstein distance in the objective function. Finally, Lemma suggests that the tersection of the Fisher class F and the Maxwell class M can still be large, and there can be many statistically equivalent representations in F n M. We can make further reduction of F E M by imposing additional constraints, for example, certain minimal properties, sparsity, and robustness against noise perturbation.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep rep- resentations. The Journal of Machine Learning Research, 19(1):1947-113 2018.
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint 16Xiv: 0.44,4 2016.
Alexander A Alemi, Jan Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410 2016.
Syed Mumtaz Ali and Samuel Silvey. A general class coefficients of divergence of one distri- bution from another. Journal of the Royal Statistical Society: Series (Methodological), 28(1): 131-142, 1966.
Brandon Amos and J. Zico Kolter. A PyTorch Implementation of DenseNet. htt.t::://///....... htt.:::://g Com/ hann/bamss/Geee densenet pyterch Accessed: [20 Feb 2020].
Martin Anthony and Peter I. Bartletr. Neural Network Learning: Theoretical Foundations, cam- bridge University Press, 2009.
M. S. Bartlett. The vector representation of sample. Mathematiccal Proceedings of the Cambridge Philosophical Sociery, 30(3):327-340, 1934. doi: 10.1007/0003550441000025222
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3:463-482, 2002.
Peter I. Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight ve-dimension and pscudodimension bounds for piecewise linear neural networks. Journal of Machine Learning Research, 20:1-17, 2019.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions Pattern Analysis and Machine Intelligence, (ttelligennee3358))117998--- 3588):1798-1888. 2013.
Yann Brenier. Polar factorization and monotone rearrangement of vector-valued functions. Commu- nications on Pure and Applied Mathematics, 44(4):375-417, 1991.
Wlodzimierz Bryc The Normal Distribution: Characterizations with Applications. Lecture Notes in Statistics. Springer New York, 1995.
C. Burgess, I. Higgins. A. Pal, Loic Matthey, Nick Watters, G. Desjardins, and Alexander Lerchner. Understanding disentangling in beta-vae, arXiv: Machine Learning, 2018.
Ricky T. Q. Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of dis- entanglement in variational autoencoders. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Dasa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 2610-2620 Curran Associates, Inc., 2018
Dennis R. Cook. Regression Graphics: Ideas for Studying Regressions Through Graphics. Wiley Series Probability and Statisties. Wiley, 1998. ISBN 9780471193654
Dennis R. Cook and S. Weisberg. Sliced inverse regression for dimension reduction: comment.
Journal of the American Statistical Association. 86(414):328-333 1991.
Dennis Cook. Fisher lecture: Dimension reduction in regression. Statistical Science, 22(1):1-26, 2007.
Victor De la Pena and Evarist Giné. Decoupling: from Dependence to Independence. Springer Science & Business Media, 2012.
Kien Do and T. Tran. Theory and evaluation metrics for learning disentangled representations. ArXiv, abs/ 901990888.99..1... 2020.
10

---

Under review as a conference paper at ICLR 2021
Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018.
Ronald A. Fisher. On the mathematical foundations of theoretical statisties. Philosophical Transac- tions of the Royal Society of London, A, 222:309-368, 1922.
Kenji Fukumizu, Francis R Bach, Michael I Jordan, al. Kernel dimension reduction in regression. The Annals of Statistics, 37(4):1871-1905 2009.
Yuan Gao, Yuling Jiao, Yang Wang. Yao Wang, Can Yang, and Shunkang Zhang. Deep generative learning via variational gradient flow. In International Conference on Machine Learning, pp- 2093-2101, 2019.
Yuan Gao, Jian Huang, Yuling Jiao, and Jin Liu. Learning implicit generative models with theoretical guarantees. arXiv preprint (iX.ii:0002.08662, 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems, volume 27, pp. (680-2680. 2014.
Balázs Gyenis. Maxwell and the normal distribution: A colored story of probability, independence, and tendency toward equilibrium. Studies in History and Philosophy of Science Part B: Studies in History and Philosophy of Modern Physics, 57:53-65, 2017. doi: 10. 10.1016//.sii.. 1001//..hpps.20177 1.00..
Trina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vac: Learning basic visual concepts with a constrained variational framework. In 5th International Conference on Learning Representations, 2017.
Irina Higgins, D. Amos, D. Pfau, Sébastien Racanière, Loic Matthey, Danilo Jimenez Rezende, and Alexander Lerchner. Towards a definition disentangled representations. ArXiv, ab- 1/1812.0 2018.
Alex Hjelm, Devon R.and Fedorov, Samuel Marchilnar,,,, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information esti- mation and maximization. arXiv preprint arXiv: 06670, 2018.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700-4708, 2017.
Xiaoming Huo and Gábor J Székely. Fast computing for distance covariance. Technometrics, 58(4): 435 447, 2016.
HM Kabir, Moloud Abdar, Seyed Mohammad Jafar Jalali, Abbas Khosravi, Amir F Atiya, Saeid Nahavandi, and Dipti Srinivasan. Spinalnet: Deep neural network with gradual input. arXiv preprint arXiv:2007.033. 00340007.0337,,0 2020.
Amor Keziou. Dual representation of i-divergences and applications. Mathématique, 336(10):857-888 2003.
Comptes Rendus
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pp. 1849-2658, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arX- iv: 13:161166144 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical Report TR-2009, University of Toronto, Toronto., 2009.

---

Under review as a conference paper at ICLR 2021
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disen- tangled latent concepts from unlabled observations. In International Conference on Learning Representations, 2018.
Yann LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1:541-551, 1989.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. 2010. URL http://yamm. lecun. com/exdb/unist, 7:23, 2010.
Ker-Chau Li. Sliced inverse regression for dimension reduction. Journal of the American Statistical Association, 86(414):316322) 1991.
Tengyuan Liang. On how well generative adversarial networks learn densities: nonparametric and parametric results. arXiv: Statistics Theory, 2018.
Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variation using few labels. arXiv preprint arX- iv:1905.01288, 2019.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, and Ian Goodfellow. Adversarial autoencoders. In International Conference on Learning Representations, 2016.
James C. Maxwell. Illustrations of the dynamical theory of gases. part i. on the motions and colli- sions of perfectly elastic spheres. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, (9(114):19-33 1860.
Robert J. McCann. Existence and uniqueness of monotone measure- presurvi maps. Duke Mathe- matical Journal, 80(2):309-324, 1995.
XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions Information Theory, 2010.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: using variational divergence minimization. In Advances in neural information processing systems, pp. 271-279, 2016.
Training generative neural samplers
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 2018.
Guido Philippis. Regularity of optimal transport maps and applications, volume 17. Springer Science & Business Media, 2013.
Karl Ridgeway and Michael C. Mozer. Learning deep disentangled embeddings with the f-statistic loss. ArXiv, Abbs/880.0053 2018.
Tyrrell R. Rockafellar. Convex analysis. Princeton University Press, 1970.
Andrew M. Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment,
2019.
Zuowei Shen, Haizhao Yang, and Shijun Zhang. Deep network approximation characterized by 2019.
number of neurons. arXiv preprint arXiv:
Ravid Shwartz-Zr Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprint arXiv: 1003. 00000000000.0..0 2017.
12

---

Under review as a conference paper at ICLR 2021
Shashank Singh, Ananya Uppal. Boyue Li, Chun-Liang Li, Manzil Zaheer, and Barnabás Póczos. Nonparametric density estimation under adversarial losses. In Advances in Neural Information Processing Systems, pp. 10225-102333 2018.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. arXiv preprint arXiv:2004.041336 2020.
Charles J. Stone. Optimal global rates of convergence for nonparametric regression. The Annals of Statistics, pp. 1040-1053, 1982.
Gáhor J. Székely and Maria I. Rizzo. Brownian distance covariance. The Annals of Applied Statis- fics, 14))1136-1 2009.
Gábor Székely, Maria L.. Rizzo, Nail K. Bakirov, et al. Measuring and testing dependence by correlation of distances. The Annals of Statistics, 35(6))2769-2277 299944 2007.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pp- 1-5. IEEE, 2015.
Naftali Tishby, Fernando C Pereira, and W. Bialek. The information bottleneck method. ArXiv, physics/0004057. 2000.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. encoders. International Conference on Learning Representations, 2018.
Wasserstein auto-
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint
13625, 2019.
Alexandre Tsybakov. Introduction to Nonparametric Estimation. Springer Science Business Media, 2008
Cédric Villani. Optimal Transport: Old and New, volume 338. Springer, 2008
Rick Wang, Amir-Hossein Karimi, and Ali Ghodsi. Distance correlation autoencoder. In 2018 International Joint Conference on Newral Networks (IJCNN), pp. 1-8. IEEE, 2018.
Satosi Watanabe. Information theoretical analysis of multivariate correlation. IBM Journal of Re- search and Development,
82, 1960.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint /rXiv: 2017.
Sergey Zagoruyko and Nikos Komodakis. iv:1605.07140 2016.
Wide residual networks.
arXiv preprint arX-
A APPENDIX: EXPERIMENTAL DETAILS
A. SIMULATION STUDIES
The values of the aaper-parameters for the simulated experiments are given in Table A1 where is the penalty parameter, d is the dimension of the SDRM, n is the mini-batch size in SGD, Ti is the number of inner loops to push forward partieles 2,, T2 is the number of outer loops for training Ro. and is the step size update particles. For the regression models, the neural network architectures are shown Table A2
As shown in Table A3 multilayer perceptron (MLP) is utilized for the neural structure D in the classification problem. The detailed architecture of 10-layer dense convolutional network (DenseNet) (Huang al. settings, we adopted the Adam (Kingma & Bal 2014) optimizer with an initial learning rate of 0.001 and weight decay of 0.0001.
& Kolter) deployed for Ro is shown in Table A4 For all the
13

---

Under review as a conference paper at ICLR 2021
Table Al: Hyper-parameters for simulated examples, where varies according to epoch
8
Task
X
d
T
Tg
0-150
151-225
226-500
Regression
1.0
2
64
500
3.0
2.0
1.0
Classification
1.0
500
2.0
1.5
1.0
Table A2: MLP architecturrs for D& and Ro in regression
DΦ
Ro
Layers
Details
Output size
Details
Output size
Layer
Linear, LeakyReLU
16
Linear, LeakyReLU
16
Layer Linear
Layer
1
Linear, LeakyReLU
8
Linear
d
Table A3: MLP architecture for D& of simulated classification examples and the benchmark clas- sification datasets
Layers
Details
Output size
Layer
Linear, Leaky ReLU
64
Layer3 Linear, LeakyRela ReLU
128
Layer Linear, LeakyReLU
64
Layer Linear
1
Table A4: DenseNet architecture for Rg in the simulated classification examples
Layers
Details
Output size
Convolution
3 x Conv
24 x 20 20
BN, 1 Conv
Dense Block
36 20 20
BN, 3 x 3 Conv
Transition Layer
BN, ReLU, 2 x 2 Average Pool,1 x Conv
30 x 10 x 10
BN, x Conv
Dense Block
BN, 3
Conv
x
18 10 10
Transition Layer 2
BN, ReLU, 2 x 2 Average Pool, 1 x 1 Conv
15 X 5 x
BN, 1 X Conv
Dense Block
x 1
27 x 5 x 5
BN, 3 Conv
Pooling
BN, ReLU, 5 x 5 Average Pool, Reshape
27 2
Fully connected
Linear
14

---

Under review as a conference paper at ICLR 2021
A.2 REAL DATASETS
Regression: In the regression problems, hyper-parameters are presented Table A5 optimizer with an initial learning rate of 0.001 and weight decay of 0.0001 is adopted. architectures of D& and Rg for the YearPredictionMSD data are shown Table A6
The Adam The MLP
Table A5: Haper-parameters for YearPredictionMSSD data
Dataset
d
π
Ti
T2
YearPredictioMMSD
1.0 10, 20, 30, 40
64
500
1.0
Table A6: MLP architectures for D& and Rg for YearPredictionMIID data
DΦ
Rg
Layers
Details
Output size
Details
Output size
Layer
Linear, LeakyReLU
32
Linear, LeakyReLU
32 8
Layer
Linear, LeakyReLU
8
Linear, LeakyReLU
Layer
Linear
1
Linear
d
Classification: For the classification problems, tlyer-parameters are shown in Table A7 We again use Adam as the SGD optimizers for both D and Rg. Specifically, learning rate of 0,001 and weight decay 0.0001 are used for D in all datasets and for Rg on MNIST (LeCun et al.) 2010). We customized the SGD optimizers with momentum at 0.9, weight decay at 0.0001, and learning rate in Table A8 for FashionMNIST (Xiao et al., 2017) and CIFAR-10 (Krizhevsky et al. 2012). For the customized SGD optimizer with initial learning rate of 0.001
transfer learning of CIFAR-10, and momentum 0.9 for Rg- MLP architectures of the discriminator network Dø for MNIST, FaShionMNIST and CIFAR-10
given in Table A3 The 20-layer DenseNet networks shown in Table were utlized for Rg on the MNIST dataset, while the 100-layer DenseNet networks shown Table A10 and All are fitted for Ro on FashionMNIST and CIFAR-10.
Table A7: Hyper-parameters for the classification benchmark datasets
Dataset
A
d
Ti
T2
MNIST FashionMNIST
1.0
16, 32, 64
64
300
0.1
1.0
16,
32, 64
64
300
1.0
CIFAR-10
1.0
16,
32, 64
64
300
1.0
CIFAR-10 (transfer learning)
0.01
16, 32, 64
64
50
1.0
B APPENDIX: PROOFS
In this appendix, we prove Lemmas 2.1 and 4.1 and Theorems 4.2 and 4.3
B. PROOF OF LEMMA
Proof. By assumption μ and Na are both absolutely continuous with respect to the Lebesgue mea- sure. The desired result holds since it is a spacial case the well known results on existence of optimal transport (Brenier 1991 McCann 1995). see, Theorem 1.28 page 24 of (Philippis) 2013) for details.
15

---

Under review as a conference paper at ICLR 2021
Table A8: Learning rate p varies during training.
Epoch
0-150
151-225
226-300
p 0.1
0.01
0.001
Table A9: Architecture for MNIST, reduced feature size is
Layers
Details
Output size
Convolution
3 X 3 Conv
24 x 28 x 28
BN, 1 X Conv
Dense Block
BN, 3 Conv
X
48 x 28 28
Transition Layer
BN, ReLU, 2 2 Average Pool,1 1 Conv
24 14 x 14
BN, 1 Conv
Dense Block 2
x
48 X 14 14
BN, 3 x Conv
Transition Layer
BN, ReLU, x 2 Average Pool, Conv
24 X 7
BN, 1
Conv
Dense Block 3
x
48
BN, 3 X 3 Conv
Pooling
BN, ReLU, 7 x 7 Average Pool, Reshape
48 d
Fully connected
Linear
Table A10: Architecture for FashionMNIST, reduced feature size is d
Layers
Details
Output size
Convolution
3 x 3 Conv
24 x 28 28
BN, 1 X 1 Conv
Dense Block
X 16
216 x 28 x 28
BN, 3 X 3 Conv
Transition Layer
BN, ReLU, 2 Average Pool,
Conv
108 x 14
BN, X Conv
Dense Block
BN, 3 x 3 Conv 16
300 X 14 x 14
Transition Layer
BN, ReLU, x 2 Average Pool, x 1 Conv
150 x x 7
BN, x Conv
Dense Block
16
342 7
BN, 3 x 3 Conv
Pooling
BN, ReLU, 7 x 7 Average Pool, Reshape
342 d
Fully connected
Linear
Table A11: Architecture for CIFAR-10, reduced feature size is d
Layers
Details
Output size
Convolution
3 x Conv
24 32 32
BN, x Cony
Dense Block
16
216 x 32 x 32
BN, 3 x 3 Conv
Transition Layer
BN, ReLU, 2 x Average Pool, x Conv
108 x 16 x 16
BN, X Conv
Dense Block
16
300 x 16 16
BN, 3 x 3 Conv
Transition Layer
BN, ReL.,, x 2 Average Pool, x Conv
150 x 8 x 8
BN, 1 x Conv
Dense Block
x 16
342 x x 8
BN, x Cony
Pooling
BN, ReLU, 8 Average Pool, Reshape
342 d
Fully connected
Linear
16

---

Under review as a conference paper at ICLR 2021
B.2 PROOF OF LEMMA 4.1
Proof.
Our proof follows Keziou 2003). Since f(() is convex, then AA E R, we have S(1) = (m)
where
(*) = sup (
sER
the Fenchel conjugate of fr. By Fermat's 's rule, the maximizer s* satisfies
t Of"
i.e.
E a)(()
Plugging the above display with / duz (x) into the definition of f-divergence, we derive 66
B.3 PROOF OF THEOREM
1. For R satisfying (3) and any R E R, we have R = P(n.R-)R R + ER, where P(n, is the correlation coefficient between and R ER = R P(R.R.) R². It is easy to see that ER IL R and thus Y ER. As (P(R,R-) R Y) is independent of (ER, 0), then by Theorem 3 of Székely & Rizzo (2009)
Proof. Without loss of generality,
assume
yl RV) + ER.y] S ) -V(P(R,R-) Rt, .y] (((R VIR", SVIRC, .y].
R .yl V(ER.0)
As R(x) ~ NO 1) and (x(x) (O, 1). then Dy(uR(x) (x) 0, and ((R L(R") = VIR"", RR 2 O.
The proof is completed.
B.4 PROOF OF THEOREM
Denote B2 (ax() B3 v B, [s*(s)). We set the network
parameters of the representer Ro and the discriminator D3g as follows.
(i) Representer network Ro.ws.... parameters: depth D 9 9lgg + 12, width
logg)) n) + 4p, 12n llog n + 14), size S = i log (npd), W = d max /lg
B = (2B3L1 log
(ii) Discriminator network M M. parameters: depth D 9 12, width W logn)) n) + 4d, 12n * logn + 14], size S + /log rpdd,, B 2L2vdlogn B2.
max = /log
Before getting into the details of the proof of Theorem 4.3 we first give an outline of the basic structure of the proof.
Without loss of generality, assume that À 1 and scenario that y is bounded almost surely, say ly| S C1. We also assume B1 < co. We can utilize the truncation technique to transfer the unbounded cases the bounded under some common tail assumptions. Consequently, an additional log multiplicative term will appear in the final results. For any R ND...S,s. have,
i.e. y R. First consider the
L( L(R*) ((Rs) [(RO) + ((R)) [(R) ((R) -()) + L(R) C(R))
< 2 sup [C(R) - Z(RR) + inf
(C(R) - (CRR)
(10)
REND.w.s.s
RENo.w.s.8
where the definition of Rs in (8) and the feasibility of R. Next we bound the
terms in (10 i.c., the approximation error inf ReNo. (C( - L(R*) and the statistical
error SuPREND..S ..s..... (CR - [(R) separately. Then Theorem 4.3 follows after bounding these
two error terms.
17

---

Under review as a conference paper at ICLR 2021
B.4.1
THE APPROXIMATION ERROR
Lemma B.I.
inf
[C(R) [(R)) S 2600C| B1 LI V pdn şीz
(II)
Proof. By 3 and 6 and the definition of we have
inf REND.w.s.
[CR ((R*) s [B)(uR (x) VXRg(x),
(12)
where Rg E EN is specified in Lemma B.2 below. We finish the proof by (14) in Lemma B.3
and (15 in Lemma B.4 which will be proved below.
Lemma B.2. Define ie (a) min [tS (a), log There exist Rg E ND. with depth
= 9lgg n + 12, width W = Emax((d(wv /logn) n) + 4d, 12n 240 /log n + 14), and size
dn =-3 /log" npd), B (21) logn) Vd, such that
Rg R (( S 160L, pdn 7
(13)
Proof. Let Ri (z) be the i-th entry of (2) Rd Rd By the assumption on R easy check that R (x)) Lipschitz continuous on | B1. Bi with the Lipschitz constant L, and || 1| I logn. By Theorem 4.3 in Shen et al. 2019). there exists ReLU network Rg, with
with depth logn 12, width max (dd(1 /4p log n) 4d, 12n ズhp log n + 14), [Rg 1l 2B LI + log n, such that
|R |0= 2B logn,
and
|R Rc S 80L, B1 /pn 272,
80L, B1 vpm न
(x( < 2B, Li vp + log
Define Rg (
...,, Ro.] ND..ws.B. The above three display implies
Rg R² R2 S 1601, B1 pdn ,ंं
Lemma B.3.
| (x),
2580C, /pdn A
(14)
Proof. Recall that Székely et al. 2007)
V(z.y] =I 21
2:2 ly -22ll 2F [|z] -2 ysll
E [||z] - 22 121 E [ly: - -2ll
where (z;, yi), 1,2,, are i.i.d. copies of y). We have
[x[)(()), y] V ())g
$ E [[[] (x1) - R² (x²))l
(xx)xx Rx(x2) (x(x))))lx y2l]
+ 2| [(]]R* (x1) R (x2) - | (x1) - Rg x2))l)ly1 - ysll
+ E [xi) - (x2)) - 1 [x)( - [x2)] IF [[lyy] - ->llll
- E [[]] - R² Rxxxx))
[ - Ro (x2))l|
< 16C E [[]][((x)]] Ra (x)
16C, |1 (xx (x)
(x)]
+ [ ( (x)
18

---

Under review as a conference paper at ICLR 2021
where in the first and third inequalities use the triangle inequality, and second follows from the boundedness of y. By (1)). the first term in the last line bounded by 2560C; B1 L1 ypdn ptz. Some direct calculation shows that
n)
[| (x)) ||] I C2
n
(log (logm)
We finish the proof by comparing the order of the above two terms, i.., C2 20C; B1 s.LIvpe ppn pizz Lemma B.4.
D B (( S 20C, Bi Li v pdn +ी..
(15)
Proof. By Lemma B.2 Rg approximate R arbitrary well, the desired result follows from the fact that Dy (JR (x) I4) = 0 and the continuity of ID (PR()))))))) on R. We present the sketch of
the proof and omit the details here. Let r" (2) have
denr (z) and F(z)
(z). By definition
dya
daa
T3 (MR (laa) = B (((r
Ew (W"())) (eBall(O. + E (EBa))e
(We represent Dy (, Iaa similarly. Then
D (M = [D (M - cxllaa
- Ew
f(F(W)) ((((N)))eermanooeom))
+ Ew (If (W)) - f(r" (weaarroooo)))
[f())()) (z) r(=)ldna(2)
[f(((2)))))> (2) F(=))dna(2)
llllslg
||
- (()))>(=)]ddaal + C4 /
SCa |
-+(2)-((2))
|=||>|ogn
The first term in the above display small due to Rg can approximate S² well. The second term is small due to the boundedness of and the exponential decay of the Gaussian tails.
B.4.2
THE STATISTICAL ERROR
Lemma B.5.
[C(R) [(RR) s Cis(BI(L + L2) pd)n + + (L2Vd+ B2 + Bs) Bog nn 사ीय)
sup REND.
(16)
Proof.
By the definition and the triangle inequality we have
[C(R) - -Z(R))
sup
w.s.D.
< E
sup
REND.w.s.B
+ E
sup
(H - (h(l
ReAD,w.s.s
We finish the proof based below.
17 in Lemma B.6 and (22 in Lemma B.7 which will be proved
Lemma B.
E
sup REND.w.s.B
yl -V((()),)1))) 4C,CCCroBB21 Vpda 222.
(17)
19

---

Under review as a conference paper at ICLR 2021
Proof. We first fix some notation for simplicity. Denote 0 = yx,y) E R x R' and O (x). = ....n are i.d copy of O, and denote Mx, and p& P and P", respectively. VR E W.., let Ô (R(x). y) and Ô (()x:),ys): ..... i.i.d copy of Ó Define centered kernel hg: (R R + R
R 02 03. 04) 1 Σ 1sigs4. [R(x;) R(Xx))(lx Ysl
123 Eii [(x,)) R(xx)1I
[[((x,) R(xx))1 Σ isssa,
ly: - -35) ly, y,ا V[(((),,,
(18)
(4)
Then, the centered U-statisties (((x). y) V[R(x),)) y) can represented as
Un(hn) C₁
Σ
hn(ō,,, O O..).
Our goal is to bound the supremum of the centered U-process U, (k))) with the nondegenerate kernel hR. By the symmetrization randomization Theorem 3.5.3 De Pena & Giné (2012), we have
E sup w...s
(hn)] S Cg
sup RENo.w.s.r
Σ
hn 0 Ow))],
CA
(19)
where, v I, i.idd Rademacher variables that also independent with O 1,...,n. We finish the proof by upper bounding the above Rademacher process with the matric entropy N,.. To this end we need the following lemma.
n.
Lemma B.7. If i 1,.. are m finite linear combinations of Rademacher variables ej, 1,..J. Then
E Ee..
max Isism
|&d S Co(log m2
max
(EE?) 1/2
(20)
1515m
Proof. This result follows directly from Corollary 3.2.6 and inequality (4.3.1) in De la Pena & Giné 2012 with (xx) = exp(x?).
By the boundedness assumption on y and the boundedness of R E ND.w.S,B. we have that the kernel hgg is also bounded, say
PRRII S LC
(21)
VR, i No.w.s.. define a random empirical measure (depends on O,, I,...,n)
(R, RR it) = Ee,, .1
Σ
(h)
Condition on 0,, 1,..., n, let C(N, 1, 6)) be the covering number ND...s.B with respect to the empirical distance 6n, at scale of 5 > Denote Ng as the covering set of ND., with
20

---

Under review as a conference paper at ICLR 2021
cardinality of C(N, en,1, 6)). Then,
Bei sup
Σ
en hn O1g O.)l
R.ND.W.S.E
C4 R 1gh <im
$ 8 + Be,, sup RENss
Σ
4, (,, O Oz.)ll
28+ + Co
1
(log ew, 6))
max REN,
Σ
Σ
(hR(Or,, Org
C1
< 304-0C((BBLLL5-+-grryyyyllgggg 8)) 1 n(n))² 1/2
C ((n 3)1)²
344-000h(22144777-gggnn22vhh2h22 /))
2eBn
2 33++6GC((2BL11x1/-0g))//v//vv//0 8VCN
1/2 1/2/Vnn
Bn log 8DS log 5) )1/2 ///
where the first inequality follows from the triangle inequality, the second inequality uses (20). the third and fourth inequalities follow after some algebra, and the fifth inequality holds due to C(,V,,,,11 1, v C(V,en, en,s,, 5) and the relationship between the metric entropy and the VC- dimension of the ReLU networks ND.w.s.B (Anthony & Bartlet 2009). i...
2eIn
log C(V, en,o0, 6)) S VCw log 8VCN*
and the last inequality holds due to the upper bound of VC-dimension for the ReLU network WD.wssss satisfying
C&DS S VCN Cog
see Bartlett et al. (2019). Then 17 holds by the selection of the network parameters and set 8 and some algebra.
Lemma B.8.
s Cia(LaVd+ B Ba)((n 2ी. log nn = (22)
E sup R....
(z), 9r((z) = f(((2)). By assumption 9n(z) : Rd is Lipsehitz continuous with the Lipschitz constant To2 and [[gnll.> S B2. With- log
Proof. VR E ND..Ws.B. let r(z) damis)
dra
out loss of generality, assume supp(gR)
lolg log Then, similar to the proof of
Lemma B.2
show that there exists Do MD.W.s.8 with depth D 9lg + 12, width W max(8d(n log n)) + 4d, 12n z#a logn + 14), and size S n # (log" npd),
B 2T 2d + B2 such that for z ~ Yd and (R(x)
Ez -x(2)) - 160L: पोअ.
(23)
Ag Rd + R, define
E(g) = Ex [o([[())]] Ew
E(g) = Elg. R)
Σ
(o(R(x)) (o(W)))
By 6 we have
E(gR) = (Hmmsll =
sup
E(D).
(24)
measureable DRRR-RR
21

---

Under review as a conference paper at ICLR 2021
Then,
D (P(mm))))))) Dy (Hm))))aaa
=1E(gn)
E(Dx)
max D.eMa,w.s.
< (E(gR)) D.EMo,w.s.
sup
E(Da)l+ +1
sup
E(Ds)
max
E(Da)
D.EM
D.EMo.
S (E(gg)) E(Dz))
(((D)) E(D))l
sup
D.
$ Be-prco [lon [z) + Sw (Dx))( +
(E(Dg) - {(D))
sup D.EMB.W...
& 160(1 + B3) nn म sup ((D))-E(D)) D.EMp.W.s.
where we use the triangle inequality in the first inequality, and we use E(gn) > supD&EMp.w.. 5,5. {(Db) followed from (24) and the triangle inequality in the second inequality, the third inequality follows from the triangle inequality. and the last incquality follows from (23 and the mean value theorem. We finish the proof via bounding the empirical process
U(D, E
sup RENo w.w.s.D............. w.....
(E(D) E(D)I..
Let S = (x,z) Px X Va and S;, l..... i.d i.dd copy of S. Denote
b(D, 5) - D(R(x)) - (
Then
E(D, R) = Es(b(D, R)
and
E(D,R) = 1 n
MD, R;
Let
G(M N) = -BS 71
NEND.......EM...... bDD R(
sup
be the Rademacher complexity of MD,w,s.8 X ND..w...B (Bartlett & Mendelson 2002). Let C(M en,1, 8)) be the covering number of M..w.s. x NND.w.S.B with respect to the em- pirical distance (depends on 8t)
d,, ((D, R), (D, Rt)
-F.. E.
(b(D,, b(,,
scale of 8 > 0. Let Ms x Ns be such a converging set MB...s.B x No Then, U(D, R) 2g N\
2E E i=
I9N x (S S.)l
2 Es
< 28 + n
Sn E bl El...
sup (D,N)EMsXNG
Σ S) S.)l
1
9n [log C(M N., 28 C12 Es
-
bD R: S22
n
(D.R)EM;X.,
1
< 26 + C12 -12-Es (, (MM X en, ( + B2 + B3)
n
28 + C12
(2 Vdlonn + B3 + (log ((M, ,,, + log C(N, dn 1 6)) 1/2
Vn
L2 Le log + B2 + B3 (DS log S Slgg 8DSlog + DS log $ log 8D$ log S
Bn
Bn
< 28 + C13
1/2
Vn
22

---

Under review as a conference paper at ICLR 2021
where the first equality follows from the standard symmetrization technique, the second equality holds due to the iteration law of conditional expectation, the first inequality follows from the tri- angle inequality, and the second inequality uses equation 20, the third inequality uses the fact that b(D, R: S) is bounded, i.e., []ō (D, R; S> 2Lg follows from some algebra, and the fifth inequality follows from CC,, S C(N, Cn.oo: 6) 2e8n (similar result for MM) and log CN, en,oo, 6)) < VC log 8VCx* 7 and ND.w.ss. satisfying
B2 + Bs, and the fourth inequality
C&DS log S < VCN < CgDS log S, see Bartlett et al. (2019). Then (2) follows from the above display with the selection of the network parameters of
and with 8 1. O
Finally, Theorem 4.3 direct consequence of in Lemma B.1 and 16 Lemma B.5 This
completes the proof of Theorem 4.3
23


Reviews:
[{"review_id": "y-6j1ib1WU9", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new representation learning framework for supervised learning \nIn order to achieve good prediction accuracies while maintaining some desired properties. \nThe sufficiency and disentangled properties of the representation are formalized in\nthis paper to achieve this goal. Most results in the literature have focused on\nthe unsupervised setting, therefore this framework is novel to me.\nBelow I have a few comments to revise the paper. \n\nTo achieve sufficiency, this paper proposes to use the distance covariance defined in Section 4. \nThis seems to be related with statistical correlation,\ncan this be written out in the paper? As when I read the proof of \ntheorem 4.2, I do not know what is the \\rho(R,R*). It seems not \nto be defined in the paper. \nAlso is there any relation between V[z,y] with the sufficient statistics (Fisher 1922), as mentioned in the introduction? \nFrom (5) I then understand that this covariance V replies on the metric on X and Y. \nFor multi-class classification problems, it is not always clear how to choose\na metric on Y. I recommend the authors to clarify this choice\nat least in the numerical experiments. \n\nIn the statement of Theorem 4.3, should \\lambda = O(1) be strictly positive? The objective (7) is formalized as a bi-level optimization problem, and the computation problem is addressed using recently proposed methods based on gradient flows. There is a typo 2 lines before Section 5, should the rate be (log n) n^{-2 / (2+d)} ? \n\nSection 6 presents numerical results. I recommend to clarify what is Y in the classification problem \nand how the metric is chosen. Figure 1 should be significantly improved. Figure 1 illustrates well that the learnt features are sufficient, but it is unclear whether the features are disentangled. To make the results more convincing, I suggest to plot the learned representation directly (maybe \nin Appendix) to check whether it looks like an isotropic 2d Gaussian. Any statistical test of Gaussianity would be more convincing. \n\nWhat is the distance correlation computed in Figure 2? \nV[z,y] is called the distance covariance, so I guess it is something \nelse. Is the Table 3 showing the training or test accuracy? \n\nTypo: \n- in the definition of V[z,y] on page 4, the constant c_m should be c_q? \n- 3 lines before (2), N_d (0, I_d) -> N (0, I_d)?\n- The (a) and (e) in Figure 1 are not the learned features, but the original data.\n- page 19: matric entropy -> metric entropy\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "vdmlGM5kSmy", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present a new representation learning algorithm that trades off between a sufficiency condition (that is, the label should be independent of the input conditioned on the representation) and what they call a \"disentangling\" condition - that the representation vectors should be independent of one another and rotationally invariant. While the first condition has been used to define disentangled representations, the second is not standard. From the condition of rotational invariance, they require that the distribution over representations is isomorphic to a uniform Gaussian. They arrive at a loss with two terms, the first is a distance correlation between labels and representation, and the second is a divergence between the representation and a uniform Gaussian. In this sense, the regularization term looks quite similar to a VAE while the loss term looks quite similar to standard classification losses. The regularization is represented as a maximum over another loss, leading to a GAN-like coupled optimization problem.\n\nThe paper is written with quite a lot of complicated math. While I did not find anything wrong with the math, it did seem like at times it was meant more to obfuscate and impress and was often not really necessary. The experiments were almost entirely either comparisons against classic statistical methods (section 6.1 and 6.2, Table 1) or experiments on MNIST, FashionMNIST and CIFAR-10 (6.2, Table 3). Given the explosion of different deep learning methods for classification, representation learning, and disentangling in the last decade, it seems like the comparisons against classic statistical methods is missing the point - it seems highly likely that other deep learning methods could work just as well. In the experiments on MNIST, FashionMNIST and CIFAR-10, the authors set quite a high bar for themselves. These are 3 of the most over-studied datasets in all of machine learning - literally thousands, maybe tens of thousands of papers have been written on various deep learning algorithms applied to these datasets in the last several years. The experiments presented here do not seem to be a proper apples-to-apples comparison, however. A standard MLP is used for the SDRL (the method developed in this paper) while a DenseNet architecture trained with a classification loss is used for the \"convolutional network\" method. I was not able to find details on the dCorAE architecture. To properly reduce the number of possible confounding factors, the same architecture should be used with both the SDRL and the baseline methods, with different objective functions. Also, the classification numbers presented on CIFAR-10 do not seem to be state-of-the-art. According to Papers With Code, the DenseNet is able to achieve 96% accuracy on CIFAR-10, and even newer methods are able to reach 99% accuracy without additional data.\n\nMy biggest objection to the paper, however, is that it seems completely unrelated to disentangling. The only experiments presented are on visualization and classification - no results on standard disentangling tasks are presented. The rotational invariance condition seems completely against the grain of disentangling research. Despite disagreements over the exact definition of disentangling, there is at least broad agreement that a disentangled representation is one in which certain directions in latent space are privileged over others, and align with \"true\" latent factors in the world. While I disagree with the probabilistic definition of disentangling given in Locatello et al (2018), and prefer the geometric definition of Higgins et al (2018), I do agree with the point made in Locatello that, if different unique directions in latent space are not identifiable, then disentangling is not possible. Yet in this paper, the non-identifiability of different directions is given as a *necessary condition* for disentangling. This is the completely contrary to how the term is usually used. Even in the beta-VAE, which does have a rotationally-invariant loss function, the invariance is broken by requiring that the approximate posterior from the encoder has a diagonal covariance. Given this, I would recommend that the authors remove any reference to disentangling, and rewrite this purely as an alternative approach to supervised learning.", "extra_scores": {}, "extra_text": {}}, {"review_id": "vUZXZ64nI1H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a method, SDRL, for learning sufficient and disentangled representations, with the additional property that the representations should also be rotation-invariant. Together with the disengtangled property, the ration-invariant property specifies the distribution of representations to be isotropic Gaussians. On the other hand, the repressentations are required to be sufficient for predicting the target labels. These two goals motivate the Lagrangian formulation of the objective function, based on which the authors apply two different estimators for these two goals. Experiments are extensive, with a good mixture of synthetic and real-world data. I particularly like the visualization of the learned representations on the synthetic data, which well corroborates the theoretical claims in the first part of the paper. \n\nOverall, I think the paper has a sound logic and clear presentation. The only downside is that the novelty is a bit limited: both the sufficiency criterion (see Tishby et al. 2015) and the disentangled criterion (mutual independency) have well been studied in the literature, despite using other divergence measures, e.g., mutual information and KL divergence. That being said, I haven't seen similar work using exactly the same divergences as the ones in this paper, e.g., distance correlation and f-divergence in this context. \n\nA few more minor comments and questions:\n-   I could understand representations that are rotation-invariant are desirable in vision tasks, but I am wondering is this requirement too strong for other applications, e.g., in language and speech? Other than the technical convenience brought by this property, is there any other reason that could motivate this criterion? \n\n-   In Eq. (2), shouldn't the domain of g be R^p instead of R^d? Otherwise the intersection of M and F would be empty. \n\n-   If my understanding about Lemma 2.1 is correct, it is the pushforward of \\mu under T to be an isotropic Gaussian distribution, right? The original Z = g(X) is only sufficient, but not disintangled nor rotation-invariant. If this is the case, then I am not sure I totally agree with the second comment after Theorem 4.2, where the authors remark that the discriminator network is the pushforward function T. My understanding is that the representer network R is used to approximate the composite of T and g, so that R_\\sharp \\mu_X = R^* in Eq. (3), while the discriminator network D is only served as the witness function in the definition of f-divergence in Eq. (6). Please clarify this point, thanks!\n\n-   The line above Eq. (9), typo: no \\lambda is needed before the V term.\n\n-   IMO the second assumption of Theorem 4.3 is quite strong: it requires the density to be lower bounded by a positive constant c_1, which does not even hold for Gaussians. On the other hand, the compactness of the domain helps to reconcile this assumption, and I understand that this is a standard assumption in the analysis of nonparametric density estimation/regression.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "B8tzfP4tVcJ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces SDRL, a representation learning algorithm for supervised learning.\nSDRL aims to enforce three constraints to the learned representation g(x) :\n1) That all the information from X about Y is preserved: Y independent to x given g(x)\n2) That it's distributionally rotation invariant: Law(O g(x)) = Law (g(x))\n3) That all it's coordinates are independent: Law(g(x)_i) = Law(g(x)_j) for all i, j\n\nSince 2 and 3 are equivalent to asking that g(x) is Gaussian, they look for a pushforward map g such that g(x) is Gaussian.\nIn particular, they pick the optimal transport one given by Brenier's theorem.\n\nI think the paper needs more work in the following on 4 very important aspects:\n\nA) The paper is not very well motivated in my opinion. Why do we want to learn a representation with statistically independent and rotation invariant coordinates? I know many works already try to achieve this goal, but this paper doesn't do a good job at motivating it. It merely cites a bunch of other papers that do so, but reading the paper I am left with the impression of \"we want to come up with a representation with properties 1, 2, 3. Why? Ask other people\". It doesn't feel like I'm given a sufficient explanation of why I should be reading this paper and the reality is that if I wasn't asked to review it I would have left it in the pile of papers in my desk half way through the introduction. This is very concretely seen in the sentence \"We also require that the representation is rotation invariant in distribution. Such an invariance property is an important characteristic in many visual and morphological tasks \\cite{}\". This is a third of the problem you're trying to solve, it is a bit crazy that all the motivation it has is a citation to two papers that are not mentioned anywhere else in the paper.\n\nB) The method is very complicated. Why use the distance covariance rather than the empirical risk to assure that the information from X about Y is preserved in G(X)? I know the authors have several choices for V, but the choice they pick is quite particular and a complicated one at it. Without much motivation or experiments ablating this choice it's hard to justify it. It's hard to convince the reader to bother like this.\n\nC) The learning bounds are pretty irrelevant. the n^-{2/(2+d)} bound, while minimax optimal, is useless in high dimensions (high meaning d > 10, which is usually what we are interested in representation learning). While minimax optimal ***in the distributionally worst case***, it is irrelevant in practice (and it is not an advance in any theoretical field). It's simply taking extra space in the paper, it really brings nothing to the table in my opinion.\n\nD) The experiments are extremely underwhelming. For such a complicated method does not bring any significant benefit in the real classification datasets, and a very minor one in the regression ones. In regression, furthermore, an important baseline is missing: the last layer of an NN trained with a regression and / or ordinal regression loss. It seems unfair to compete with the NN inductive bias in the author's algorithm vs linear algorithms.\n\nOverall, the authors take us through an interesting but overly complicated mathematical journey with a poor motivation, irrelevant learning bounds, and no empirical evidence that the journey was useful whatsoever.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims to present a new representation learning framework for supervised learning based on finding a representation such that the input is conditionally independent given the representation, the components of the representations are independent and the representation is rotation-invariant. While there were both positive and negative assessments of this paper by the reviewers, there are 3 major concerns that lead me to recommend rejecting this paper:
1. Most importantly, experiments do not seem to be conclusive as they do not properly ablate the specific aspects of this method. More specifically, the authors compare their deep learning based approach with non-deep learning approaches but do not compare against deep learning baselines. This makes it impossible to assess the merit of the proposed approach (which also appears to be complicated) over much simpler baselines.
2. The required properties of the representations do not seem to be properly motivated. 
3. The paper refers to their produced representations as disentangled representations. As pointed out by AnonReviewer4, this appears not to be consistent with prior uses of that word in the community.

Decision: Reject