Year: 2020

Abstract:
We propose a novel representation learning approach called sufficient and disentangled representation learning (SDRL). With SDRL, we seek a data representation that maps the input data to a lower-dimensional space with two properties: sufficiency and disentanglement. First, the representation is sufficient in the sense that the original input data is conditionally independent of the response or label given the representation. Second, the representation is maximally disentangled with mutually independent components and is rotation invariant in distribution. We show that such a representation always exists under mild conditions on the input data distribution based on optimal transport theory. We formulate an objective function characterizing conditional independence and disentanglement. This objective function is then used to train a sufficient and disentangled representation with deep neural networks. We provide strong statistical guarantees for the learned representation by establishing an upper bound on the excess error of the objective function and show that it reaches the nonparametric minimax rate under mild conditions. We also validate the proposed method via numerical experiments and real data analysis.

heading: 1 INTRODUCTION
text: Representation learning is a fundamental problem in machine learning and artificial intelligence (Bengio et al., 2013). Certain deep neural networks are capable of learning effective data representation automatically and achieve impressive prediction results. For example, convolutional neural networks, which can encode the basic characteristics of visual observations directly into the network architecture, is able to learn effective representations of image data (LeCun et al., 1989). Such representations in turn can be subsequently used for constructing classifiers with outstanding performance. Convolutional neural networks learn data representation with a simple structure that captures the essential information through the convolution operator. However, in other application domains, optimizing the standard cross-entropy and least squares loss functions do not guarantee that the learned representations enjoy any desired properties (Alain & Bengio, 2016). Therefore, it is imperative to develop general principles and approaches for constructing effective representations for supervised learning.
There is a growing literature on representation learning in the context deep neural network modeling. Several authors studied the internal mechanism of supervised deep learning from the perspective of information theory (Tishby & Zaslavsky, 2015; Shwartz-Ziv & Tishby, 2017; Saxe et al., 2019), where they showed that training a deep neural network that optimizes the information bottleneck (Tishby et al., 2000) is a trade-off between the representation and prediction at each layer. To make the information bottleneck idea more practical, deep variational approximation of information bottleneck (VIB) is considered in Alemi et al. (2016). Information theoretic objectives describing conditional independence such as mutual information are utilized as loss functions to train a representation-learning function, i.e., an encoder in the unsupervised setting (Hjelm et al., 2018; Oord et al., 2018; Tschannen et al., 2019; Locatello et al., 2019; Srinivas et al., 2020). There are several interesting extensions of variational autoencoder (VAE) (Kingma & Welling, 2013) in the form of VAE plus a regularizer, including beta-VAE (Higgins et al., 2017), Annealed-VAE (Burgess et al., 2018), factor-VAE (Kim & Mnih, 2018), beta-TC-VAE (Chen et al., 2018), DIP-VAE (Kumar et al., 2018). The idea of using a latent variable model has also been used in adversarial auto-
encoders (AAE) (Makhzani et al., 2016) and Wasserstein auto-encoders (WAE) (Tolstikhin et al., 2018). However, these existing works focus on the unsupervised representation learning.
A challenge of supervised representation learning that distinguishes it from standard supervised learning is the difficulty in formulating a clear and simple objective function. In classification, the objective is clear, which is to minimize the number of misclassifications; in regression, a least squares criterion for model fitting error is usually used. In representation learning, the objective is different from the ultimate objective, which is typically learning a classifier or a regression function for prediction. How to establish a simple criterion for supervised presentation learning has remained an open question (Bengio et al., 2013).
We propose a sufficient and disentangled representation learning (SDRL) approach in the context of supervised learning. With SDRL, we seek a data representation with two characteristics: sufficiency and disentanglement. In the context of representation learning, sufficient means that a good representation should preserve all the information in the data about the supervised learning task. This is a basic requirement and a long-standing principle in statistics. This is closely related to the fundamental concept of sufficient statistics in parametric statistical models (Fisher, 1922). A sufficient representation can be naturally characterized by the conditional independence principle, which stipulates that, given the representation, the original input data does not contain any additional information about the response variable.
In addition to the basic sufficiency property, the representation should have a simple statistical structure. Disentangling is based on the general notion that some latent causes underlie data generation process: although the observed data are typically high-dimensional, complex and noisy, the underlying factors are low-dimensional, independent and have a relatively simple statistical structure. There is a range of definitions of disentangling (Higgins et al., 2018; Eastwood & Williams, 2018; Ridgeway & Mozer, 2018; Do & Tran, 2020). Several metrics have been proposed for the evaluation of disentangling. However, none of these definitions and metrics have been turned into empirical criterions and algorithms for learning disentangled representations. We adopt a simple definition of disentangling which defines a representation to be disentangled if its components are independent (Achille & Soatto, 2018). This definition requires the representation to be maximally disentangled in the sense that the total correlation is zero, where the total correlation is defined as the KL divergence between the joint distribution of g(x) and the product of the marginal distributions of its components (Watanabe, 1960).
In the rest of the paper, we first discuss the motivation and the theoretical framework for learning a sufficient and disentangled representation map (SDRM). This framework leads to the formulation of an objective function based on the conditional independence principle and a metric for disentanglement and invariance adopted in this work. We estimate the target SDRM based on the sample version of the objective function using deep neural networks and develop an efficient algorithm for training the SDRM. We establish an upper error bound on the measure of conditional independence and disentanglement and show that it reaches the nonparametric minimax rate under mild regularity conditions. This result provides strong statistical guarantees for the proposed method. We validate the proposed SDRL via numerical experiments and real data examples.

heading: 2 SUFFICIENT AND DISENTANGLED REPRESENTATION
text: Consider a pair of random vectors (x,y) ∈ Rp×Rq , where x is a vector of input variables and y is a vector of response variables or labels. Our goal is to find a sufficient and disentangled representation of x.
Sufficiency We say that a measurable map g : Rp → Rd with d ≤ p is a sufficient representation of x if y x|g(x), (1) that is, y and x are conditionally independent given g(x). This condition holds if and only if the conditional distribution of y given x and that of y given g(x) are equal. Therefore, the information in x about y is completely encoded by g(x). Such a g always exists, since if we simply take g(x) = x, then (1) holds trivially. This formulation is a nonparametric generalization of the basic condition in sufficient dimension reduction (Li, 1991; Cook, 1998), where it is assumed g(x) = BTx with B ∈ Rp×d belonging to the Stiefel manifold, i.e.,BTB = Id.
Denote the class of sufficient representations satisfying (1) by
F = {g : Rp → Rd, g measurable and satisfies y x|g(x)}.
We refer toF as a Fisher class because of its close connection with the concept of sufficient statistics (Fisher, 1922; Cook, 2007). For an injective measurable transformation T : Rd → Rd and g ∈ F , T ◦ g(x) is also sufficient by the basic property of conditional probability. Therefore, the Fisher class F is invariant in the sense that
T ◦ F = F , provided T is injective,
where T ◦ F = {T ◦ g : g ∈ F}. An important class of transformations is the class of affine transformations, T ◦ g = Ag + b, whereA is a d× d nonsingular matrix and b ∈ Rd.
Disentanglement We focus on the disentangled representations among those that are sufficient. Therefore, we start from the functions of the input data that are sufficient representations in the Fisher class F . For any sufficient and disentangled representation g(x), let Σg = Var(g(x)). Since the components of g(x) are disentangled in the sense that they are independent, Σg is a diagonal matrix, thus Σ−1/2g g(x) also has independent components. Therefore, we can always rescale g(x) such that it has identity covariance matrix. To further simplify the statistical structure of a representation g, we also require it to be rotation invariant in distribution, that is, Qg(x) = g(x) in distribution for any orthogonal matrix Q ∈ Rd×d. The Fisher class F is rotation invariant in terms of conditional independence, but not all its members are rotation invariant in distribution. By the Maxwell characterization of the Gaussian distributions (Maxwell, 1860; Bartlett, 1934; Bryc, 1995; Gyenis, 2017), a random vector of dimension two or more with independent components is rotation invariant in distribution if and only if it is Gaussian with zero mean and a spherical covariance matrix. Therefore, after absorbing the scaling factor, for a sufficient representation map to be disentangled and rotation invariant, it is necessarily distributed as Nd(0, Id). Let M be the Maxwell class of functions g : Rd → Rd, where g(x) is disentangled and rotation invariant in distribution. By the Maxwell characterization, we can write
M = {g : Rp → Rd, g(x) ∼ N (0, Id)}. (2)
Now our problem becomes that of finding a representation in F ∩M, the intersection of the Fisher class and the Maxwell class.
The first question to ask is whether such a representation exists. The following result from optimal transport theory provides an affirmative answer and guarantees that F ∩M is nonempty under mild conditions (Brenier, 1991; McCann, 1995; Villani, 2008). Lemma 2.1. Let µ be a probability measure on Rd. Suppose it has finite second moment and is absolutely continuous with respect to the standard Gaussian measure, denoted by γd. Then it admits a unique optimal transportation map T : Rd → Rd such that T#µ = γd ≡ N (0, Id), where T#µ denotes the pushforward distribution of µ under T . Moreover, T is injective µ-almost everywhere.
Denote the law of a random vector z by µz. Lemma 2.1 implies that, for any g ∈ F with E‖g(x)‖2 < ∞ and µg(x) absolutely continuous with respect to γd, there exists a map T ∗ transforming the distribution of g(x) to N (0, Id). Therefore, R∗ := T ∗ ◦ g ∈ F ∩M, that is,
x y|R∗(x) and R∗(x) ∼ N (0, Id), (3)
i.e., R∗ is a sufficient and disentangled representation map (SDRM).

heading: 3 OBJECTIVE FUNCTION FOR SDRL
text: The above discussions lay the theoretical foundation for formulating an objective function that can be used for constructing a SDRM R∗ satisfying (3), or equivalently, R∗ ∈ F ∩M. Let V be a measure of dependence between random variables x and y with the following properties: (a) V[x,y] ≥ 0 with V[x,y] = 0 if and only if x y; (b) V[x,y] ≥ V[R(x),y] for all measurable function R; (c) V[x,y] = V[R∗(x),y] if and only if R∗ ∈ F . The properties (a)-(c) imply that
R∗ ∈ F ⇔ R∗ ∈ arg max R V[R(x),y] = arg min R {−V[R(x),y]}.
We use a divergence measure D to quantify the difference between µR(x) and γd, as long as this measure satisfies the condition D(µR(x)‖γd) ≥ 0 for all measurable function R and D(µR(x)‖γd) = 0 if and only if R ∈M. Then the problem of finding an R∗ ∈ F ∩ M can be expressed as a constrained minimization problem:
arg min R
−V[R(x),y] subject to D(µR(x)‖γd) = 0.
Its Lagrangian form is L(R) = −V[R(x),y] + λD(µR(x)‖γd), (4) where λ ≥ 0 is a tuning parameter. This parameter provides a balance between the sufficiency property and the disentanglement constraint. A small λ leads to a representation with more emphasis on sufficiency, while a large λ yields a representation with more emphasis on disentanglement. We show in Lemma 4.1 below that any R∗ satisfying (3) is a minimizer of L(R). Therefore, we can train a SDRM by minimizing the empirical version of L(R). There are several options for V with the properties (a)-(c) described above. For example, we can take V to be the mutual information V[R(x),y] = I(R(x);y). However, in addition to the estimation of the SDRM R, this choice requires the estimation of the density ratio between p(y, R(x)) and p(y)p(R(x)), which is not an easy task. We can also use the conditional covariance operators on reproducing kernel Hilbert spaces (Fukumizu et al., 2009). To be specific, in this work we use the distance covariance (Székely et al., 2007) of y and R(x), which has an elegant U -statistic expression, does not involve additional unknown quantities and is easy to compute. For the divergnce measure of two distributions, we use the f -divergence (Ali & Silvey, 1966), which includes the KLdivergence as a special case.

heading: 4 LEARNING SUFFICIENT AND DISENTANGLED REPRESENTATION
text: We first describe some essentials about distance covariance and f -divergence.
Distance covariance We first recall the concept of distance covariance (Székely et al., 2007), which characterizes the dependence of two random variables.
Let i be the imaginary unit (−1)1/2. For any t ∈ Rd and s ∈ Rm, let ψz(t) = E[expit T z], ψy(s) = E[expisTy], and ψz,y(t, s) = E[expi(t T z+sTy)] be the characteristic functions of random vectors z ∈ Rd,y ∈ Rq, and the pair (z,y), respectively. The squared distance covariance V[z,y] is defined as
V[z,y] = ∫ Rd+m |ψz,y(t, s)− ψz(t)ψy(s)|2 cdcm‖t‖d+1‖s‖q+1 dtds, where cd = π(d+1)/2 Γ((d+ 1)/2) .
Given n i.i.d copies {zi,yi}ni=1 of (z,y), an unbiased estimator of V is the empirical distance covariance V̂n, which can be elegantly expressed as a U -statistic (Huo & Székely, 2016)
V̂n[z,y] = 1
C4n ∑ 1≤i1<i2<i3<i4≤n h ((zi1 ,yi1) , · · · , (zi4 ,yi4)) , (5)
where h is the kernel defined by
h ((z1,y1) , . . . , (z4,y4)) = 1 4 ∑ 1≤i,j≤4 i6=j ‖zi − zj‖‖yi − yj‖
− 14 ∑4 i=1 (∑ 1≤j≤4 j 6=i ‖zi − zj‖ ∑ 1≤j≤4 i6=j ‖yi − yj‖ ) + 124 ∑ 1≤i,j≤4 i6=j ‖zi − zj‖ ∑ 1≤i,j≤4 i6=j ‖yi − yj‖.
f-divergence Let µ and γ be two probability measures on Rd. The f -divergence (Ali & Silvey, 1966) between µ and γ with µ γ is defined as Df (µ‖γ) = ∫ Rd f( dµ dγ )dγ, where f : R
+ → R is a differentiable convex function satisfying f(1) = 0. Let f∗ be the Fenchel conjugate of f (Rockafellar, 1970), defined as f∗(t) = supx∈R{tx − f(x)}, t ∈ R. The f -divergence admits the following variational formulation (Keziou, 2003; Nguyen et al., 2010; Nowozin et al., 2016). Lemma 4.1.
Df (µ‖γ) = max D:Rd→dom(f∗) Ez∼µ[D(z)]− Ew∼γ [f∗(D(w))], (6)
where the maximum is attained at D(z) = f ′(dµdγ (z)).
Commonly used divergence measures include the Kullback-Leibler (KL) divergence, the JensenShanon (JS) divergence and the χ2-divergence.
Learning SDRM We are now ready to formulate an empirical objective function for learning SDRM R∗. Let R ∈ M, whereM is the Maxwell class defined in (2). By the variational formulation (6), we can write the population version of the objective function (4) as
L(R) = −V[R(x),y] + λ max D:Rd→dom(f∗) {Ex∼µx [D(R(x))]− Ew∼γd [f∗(D(w))]}. (7)
This expression is convenient since we can simply replace the expectations by the corresponding empirical averages. Theorem 4.2. We have R∗ ∈ arg minR∈M L(R) provided (3) holds.
According to Theorem 4.2, it is natural to estimateR∗ based on the empirical version of the objective function (7) when a random sample {(xi,yi)}ni=1 is available. We estimate R∗ using deep neural networks. We employ two networks as follows:
• Representer network Rθ: This network is used for training R∗. Let R be the set of such neural networks Rθ : Rp → Rd.
• Discriminator network Dφ: This network is used as the witness function for checking whether the distribution of the estimator of R∗ is approximately the same as N (0, Id). Similarly, denote D as the set of such neural networks Dφ : Rd → R.
Let {wi}ni=1 be n i.i.d random vectors drawn from γd. The estimated SDRM is defined by R̂θ ∈ arg min
Rθ∈R L̂(Rθ) = −V̂n[Rθ(x),y] + λD̂f (µRθ(x)‖γd), (8)
where V̂n[Rθ(x),y] is an unbiased and consistent estimator of V[Rθ(x),y] as defined in (5) based on {(Rθ(xi),yi), i = 1, . . . , n} and
D̂f (µRθ(x)‖γd) = max Dφ∈D
1
n n∑ i=1 [Dφ(Rθ(xi))− f∗(Dφ(wi))]. (9)
Statistical guarantee Since a SDRM R∗ is only identifiable up to orthogonal transforms under the constraint that R∗(x) ∼ N (0, Id), no consistency results for R̂θ itself can be obtained. But this is not a flaw of the proposed method. Indeed, the most important statistical guarantee of the learned R∗ is that the objective of conditional independence and disentanglement is achieved. Therefore, we establish an upper bound on the excess risk L(R̂θ)−L(R∗) of the deep nonparametric estimator R̂θ in (8). We make the following assumptions.
(A1) For any ε > 0, there is a constant B1 > 0 such that µx([−B1, B1]p) > 1 − ε, and R∗ is Lipschitz continuous on [−B1, B1]p with Lipschitz constant L1.
(A2) For R ∈ M, we assume r(z) = dµR(x)dγd (z) is Lipschitz continuous on [−B1, B1] p with
Lipschitz constant L2, and 0 < c1 ≤ r(z) ≤ c2.
Denote B2 = max{|f ′(c1)|, |f ′(c2)|}, B3 = max|s|≤2L2 √ d logn+B2 |f∗(s)|.
The specifications of the network parameters, including depth, width, size and the supremum norm over the domains of the representer Rθ and the discriminator Dφ are given in Appendix B. Theorem 4.3. Suppose λ > 0 and λ = O(1). Suppose conditions (A1)-(A2) hold and set the network parameters according to (i)-(ii). Then
E{xi,yi,wi}ni=1 [L(R̂θ)− L(R ∗)] ≤ C((L1 + L2)
√ dpn− 2 2+p + L2 √ d(log n)n− 2 2+d ),
where C is a constant that depends on B1, B2 and B3 but not on n, q, p and d.
The proof of this theorem is given in Appendix B. The result established in Theorem 4.3 provides strong statistical guarantees for the proposed method. The rate n−2/(2+p) matches the minimax nonparametric estimation rate for Lipschitz class contained in Rp of functions (Stone, 1982; Tsybakov, 2008). Up to a log n factor, the rate (log n)n−2/(2+d) matches the minimax rate of nonparametric estimation of Lipschitz densities via GANs (Singh et al., 2018; Liang, 2018).

heading: 5 COMPUTATION
text: We can update θ and φ alternately as in training GANs (Goodfellow et al., 2014). However, this approach suffers from the instability issues. In our implementation, we utilize the more stable particle method based on gradient flow (Gao et al., 2019; 2020). The key idea is to find a sequence of nonlinear but simpler residual maps, say T(z) = z+sv(z), pushing the samples from µRθ(x) to the target distribution γd along a velocity fields v(z) = −∇f ′(r(z)) that most decreases the f -divergence Df (·||γd) at µRθ(x). The residual maps can be estimated via deep density-ratio estimators, which take the form T(z) = z+sv̂(z), z ∈ Rd,where s is a step size and v̂(z) = −f ′′(r̂(z))∇r̂(z).Here r̂(z) is an estimated density ratio of the density ofRθ(x) at the current value of θ over the density of the reference distribution. We use T to transform zi = Rθ(xi), i = 1, . . . , n into Gaussian samples. Once this is done, we update θ via minimizing the loss−V̂n[Rθ(x),y]+λ ∑n i=1 ‖Rθ(xi)−zi‖2/n. We describe the algorithm below.
• Input {xi,yi}ni=1. Tuning parameters: s, λ, d. Sample {wi}ni=1 ∼ γd. • Outer loop for θ
– Inner loop (particle method) ∗ Let zi = Rθ(xi), i = 1, 2, ..., n. ∗ Solve D̂φ ∈ arg minQφ ∑n i=1 1 n ( log(1 + expDφ(zi)) + log(1 + exp−Dφ(wi)) ) .
∗ Define the residual map T(z) = z − sf ′′(r̂(z))∇r̂(z) with r̂(z) = exp−D̂φ(z) . ∗ Update the particles zi = T(zi), i = 1, 2, ..., n.
– End inner loop – Update θ via minimizing −V̂n[Rθ(x),y] + λ ∑n i=1 ‖Rθ(xi)− zi‖2/n using SGD.
• End outer loop

heading: 6 EXPERIMENTS
text: We evaluate the proposed SDRL with the KL-divergence using both simulated and real data. The goal of our experiments is to demonstrate that the representations trained based the proposed method perform well. Our proposed method is not trying to learn a classifier or a regression function directly, but rather to learn representation that preserve all the information. So our experiments are designed to evaluate the performance of simple classification and regression methods using the representations we learned as input. The results demonstrate that a simple classification or regression model using the representations we trained performs better than or comparably with the best classification or regression method using deep neural networks.
Details on the network structures and hyperparameters are included in Appendix A. Our experiments were conducted on Nvidia DGX Station workstation using a single Tesla V100 GPU unit. The PyTorch code of SDRL is available at https://github.com/anonymous/SDRL.

heading: 6.1 SIMULATED DATA
text: In this subsection, we evaluate SDRL on simulated regression and classification problems.
Regression We generate 5, 000 data points from two models. Model A: y = x1[0.5 + (x2 + 1.5) 2 ]−1 + (1 + x2) 2 + σε, where x ∼ N (0, I4); Model B: y = sin2 (πx1 + 1) + σε, where x ∼ Uniform([0, 1]4). In both models, ε ∼ N (0, I4). We use a 3-layer network with ReLU activation for Rθ and a single hidden layer ReLU network for Dφ. We compare SDRL with two prominent sufficient dimension reduction methods: sliced inverse regression (SIR) (Li, 1991) and sliced average variance estimation (SAVE) (Cook & Weisberg, 1991). We fit a linear model with the learned features and the response variable, and report the prediction errors in Table 1. We see that SDRL outperforms SIR and SAVE in terms of prediction error.
Classification We visualize the learned features of SDRL on two simulated datasets. We first generate (1) 2-dimensional concentric circles from two classes as in Figure 1 (a); (2) 2-dimensional moons data from two classes as in Figure 1 (e); (3) 3-dimensional Gaussian data from six classes
as in Figure 1 (i). In each dataset, we generate 5,000 data points for each class. We next map the data into 100-dimensional space using matrices with entries i.i.d Unifrom([0, 1]). Finally, we apply SDRL to these 100-dimensional datasets to learn 2-dimensional features. We use a 10-layer dense convolutional network (DenseNet) (Huang et al., 2017) as Rθ and a 4-layer network as Dφ. We display the evolutions of the learned 2-dimensional features by SDRL in Figure 1. For ease of visualization, we push all the distributions onto the uniform distribution on the unit circle, which is done by normalizing the standard Gaussian random vectors to length one. Clearly, the learned features for different classes in the examples are well disentangled.

heading: 6.2 REAL DATASETS
text: Regression We use a benchmark YearPredictionMSD dataset to demonstrate the prediction performance of SDRL (https://archive.ics.uci.edu/ml/datasets/YearPredictionMSD). This dataset has 515,345 observations with 90 predictors. The problem is to predict the year of song release. We randomly split the data into five parts for cross validated evaluation of the prediction performance. We employ a 3-layer network for both Dφ and Rθ. A linear regression model is fitted using the learned representations and the response. The mean prediction errors and their standard errors based on SDRL, principal component analysis (PCA), sparse principal component analysis (SPCA) and ordinary least squares (OLS) regression with original data are reported in Table 2. SDRL outperforms PCA, SPCA and OLS in terms prediction accuracy.
Classification We benchmark the classification performance of SDRL using MNIST (LeCun et al., 2010), FashionMNIST (Xiao et al., 2017), and CIFAR-10 (Krizhevsky et al., 2009) against alterna-
tive methods including convolutional networks (CN) and distance correlation autoencoder (dCorAE) (Wang et al., 2018). With CN, we use the feature extractor by dropping the cross entropy layer of the DenseNet trained for classification. The MNIST and FashionMNIST datasets consist of 60k and 10k grayscale images with 28× 28 pixels for training and testing, respectively, while the CIFAR-10 dataset contains 50k and 10k colored images with 32 × 32 pixels for training and testing, respectively. For the learning from scratch strategy, the representer network Rθ has 20 layers for MNIST data and 100 layers for CIFAR-10 data. We apply the transfer learning technique to the combination of SDRL and CN on CIFAR-10 (Krizhevsky et al., 2009). The pretrained WideResnet-101 model (Zagoruyko & Komodakis, 2016) on the Imagenet dataset with Spinal FC (Kabir et al., 2020) is adopt for Rθ. The discriminator network Dφ is a 4-layer network. The the architecture of Rθ and most hyperparameters are shared across all four methods - SDRL, CN, SDRL+CN and dCorAE. Finally, we use the k-nearest neighbor (k = 5) classifier on the learned features for all methods.
The classification accuracies are reported in Tables 3 and 4. We can see that the classification accuracies of SDRL are comparable with those of CN and dCorAE. As shown in Table 4, the classification accuracies of CN leveraging SDRL outperforms those of CN. We also calculate the estimated distance correlation (DC) between the learned features and their labels as ρ2z,y = V[z,y]2/ √ (V[z]2 × V[y]2), where V[z] and V[y] are the distance variances such that V[z] = V[z, z], V[y] = V[y,y]. For more details, please see Székely et al. (2007). Figure 2 shows the DC values MNIST, FashionMNIST and CIFAR-10 data. SDRL and SDRL+CN achieves higher DC values.

heading: 7 CONCLUSION AND FUTURE WORK
text: In this work, we formulate a framework for sufficient and disentangled representation learning and construct an objective function characterizing conditional independence and disentanglement. This enables us to learn a representation with the desired properties empirically. We provide statistical guarantees for the learned representation by deriving an upper bound on the excess risk of the objective function.
There are several questions that deserve further study. First, we can adopt different measures of conditional independence including mutual information and conditional covariance operators on reproducing kernel Hilbert spaces (Fukumizu et al., 2009). We can also use other divergence measures such as the Wasserstein distance in the objective function. Finally, Lemma 2.1 suggests that the intersection of the Fisher class F and the Maxwell classM can still be large, and there can be many statistically equivalent representations in F ∩ M. We can make further reduction of F ∩ M by imposing additional constraints, for example, certain minimal properties, sparsity, and robustness against noise perturbation.

heading: A APPENDIX: EXPERIMENTAL DETAILS

heading: A.1 SIMULATION STUDIES
text: The values of the hyper-parameters for the simulated experiments are given in Table A1, where λ is the penalty parameter, d is the dimension of the SDRM, n is the mini-batch size in SGD, T1 is the number of inner loops to push forward particles zi, T2 is the number of outer loops for training Rθ, and s is the step size to update particles. For the regression models, the neural network architectures are shown in Table A2
As shown in Table A3, a multilayer perceptron (MLP) is utilized for the neural structure Dφ in the classification problem. The detailed architecture of 10-layer dense convolutional network (DenseNet) (Huang et al., 2017; Amos & Kolter) deployed for Rθ is shown in Table A4. For all the settings, we adopted the Adam (Kingma & Ba, 2014) optimizer with an initial learning rate of 0.001 and weight decay of 0.0001.
Table A1: Hyper-parameters for simulated examples, where s varies according to epoch
s
Task λ d n T1 T2 0-150 151-225 226-500
Regression 1.0 2 or 1 64 1 500 3.0 2.0 1.0 Classification 1.0 2 64 1 500 2.0 1.5 1.0
Table A2: MLP architectures for Dφ and Rθ in regression
Dφ Rθ
Layers Details Output size Details Output size
Layer 1 Linear, LeakyReLU 16 Linear, LeakyReLU 16 Layer 2 Linear 1 Linear, LeakyReLU 8 Layer 3 Linear d
Table A3: MLP architecture for Dφ of simulated classification examples and the benchmark classification datasets
Layers Details Output size
Layer 1 Linear, LeakyReLU 64 Layer 2 Linear, LeakyReLU 128 Layer 3 Linear, LeakyReLU 64 Layer 4 Linear 1
Table A4: DenseNet architecture for Rθ in the simulated classification examples
Layers Details Output size
Convolution 3× 3 Conv 24× 20× 20 Dense Block 1 [
BN, 1× 1 Conv BN, 3× 3 Conv
] × 1 36× 20× 20
Transition Layer 1 BN, ReLU, 2× 2 Average Pool,1× 1 Conv 30× 10× 10 Dense Block 2 [
BN, 1× 1 Conv BN, 3× 3 Conv
] × 1 18× 10× 10
Transition Layer 2 BN, ReLU, 2× 2 Average Pool, 1× 1 Conv 15× 5× 5 Dense Block 3 [
BN, 1× 1 Conv BN, 3× 3 Conv
] × 1 27× 5× 5
Pooling BN, ReLU, 5× 5 Average Pool, Reshape 27 Fully connected Linear 2

heading: A.2 REAL DATASETS
text: Regression: In the regression problems, hyper-parameters are presented in Table A5. The Adam optimizer with an initial learning rate of 0.001 and weight decay of 0.0001 is adopted. The MLP architectures of Dφ and Rθ for the YearPredictionMSD data are shown in Table A6.
Table A5: Hyper-parameters for YearPredictionMSD data
Dataset λ d n T1 T2 s
YearPredictionMSD 1.0 10, 20, 30, 40 64 1 500 1.0
Table A6: MLP architectures for Dφ and Rθ for YearPredictionMSD data
Dφ Rθ
Layers Details Output size Details Output size
Layer 1 Linear, LeakyReLU 32 Linear, LeakyReLU 32 Layer 2 Linear, LeakyReLU 8 Linear, LeakyReLU 8 Layer 3 Linear 1 Linear d
Classification: For the classification problems, hyper-parameters are shown in Table A7. We again use Adam as the SGD optimizers for bothDφ andRθ. Specifically, learning rate of 0.001 and weight decay of 0.0001 are used for Dφ in all datasets and for Rθ on MNIST (LeCun et al., 2010). We customized the SGD optimizers with momentum at 0.9, weight decay at 0.0001, and learning rate ρ in Table A8 for FashionMNIST (Xiao et al., 2017) and CIFAR-10 (Krizhevsky et al., 2012). For the transfer learning of CIFAR-10, we use customized SGD optimizer with initial learning rate of 0.001 and momentum of 0.9 for Rθ. MLP architectures of the discriminator network Dφ for MNIST, FashionMNIST and CIFAR-10 are given in Table A3. The 20-layer DenseNet networks shown in Table A9 were utlized for Rθ on the MNIST dataset, while the 100-layer DenseNet networks shown in Table A10 and A11 are fitted for Rθ on FashionMNIST and CIFAR-10.
Table A7: Hyper-parameters for the classification benchmark datasets
Dataset λ d n T1 T2 s
MNIST 1.0 16, 32, 64 64 1 300 0.1 FashionMNIST 1.0 16, 32, 64 64 1 300 1.0 CIFAR-10 1.0 16, 32, 64 64 1 300 1.0 CIFAR-10 (transfer learning) 0.01 16, 32, 64 64 1 50 1.0

heading: B APPENDIX: PROOFS
text: In this appendix, we prove Lemmas 2.1 and 4.1, and Theorems 4.2 and 4.3.

heading: B.1 PROOF OF LEMMA 2.1
text: Proof. By assumption µ and γd are both absolutely continuous with respect to the Lebesgue measure. The desired result holds since it is a spacial case of the well known results on the existence of optimal transport (Brenier, 1991; McCann, 1995), see, Theorem 1.28 on page 24 of (Philippis, 2013) for details.
Layers Details Output size
Convolution 3× 3 Conv 24× 32× 32 Dense Block 1 [
BN, 1× 1 Conv BN, 3× 3 Conv
] × 16 216× 32× 32
Transition Layer 1 BN, ReLU, 2× 2 Average Pool,1× 1 Conv 108× 16× 16 Dense Block 2 [
BN, 1× 1 Conv BN, 3× 3 Conv
] × 16 300× 16× 16
Transition Layer 2 BN, ReLU, 2× 2 Average Pool, 1× 1 Conv 150× 8× 8 Dense Block 3 [
BN, 1× 1 Conv BN, 3× 3 Conv
] × 16 342× 8× 8
Pooling BN, ReLU, 8× 8 Average Pool, Reshape 342 Fully connected Linear d

heading: B.2 PROOF OF LEMMA 4.1
text: Proof. Our proof follows Keziou (2003). Since f(t) is convex, then ∀t ∈ R, we have f(t) = f∗∗(t), where
f∗∗(t) = sup s∈R {st− f∗(s)}
is the Fenchel conjugate of f∗. By Fermat’s rule, the maximizer s∗ satisfies
t ∈ ∂f∗(s∗), i.e., s∗ ∈ ∂f(t) Plugging the above display with t = dµZdγ (x) into the definition of f -divergence, we derive (6).

heading: B.3 PROOF OF THEOREM 4.2
text: Proof. Without loss of generality, we assume d = 1. For R∗ satisfying (3) and any R ∈ R, we have R = ρ(R,R∗)R∗ + εR, where ρ(R,R∗) is the correlation coefficient between R and R∗, εR = R − ρ(R,R∗)R∗. It is easy to see that εR R∗ and thus Y εR. As (ρ(R,R∗)R∗, Y ) is independent of (εR, 0), then by Theorem 3 of Székely & Rizzo (2009)
V[R,y] =V[ρ(R,R∗)R∗ + εR,y] ≤ V[ρ(R,R∗)R∗,y] + V(εR, 0) =V[ρ(R,R∗)R∗,y] = |ρ(R,R∗)|V[R∗,y] ≤V[R∗,y].
As R(x) ∼ N (0, 1) and R∗(x) ∼ N (0, 1), then Df (µR(x)‖γd) = Df (µR∗(x)‖γd) = 0, and L(R)− L(R∗) = V[R∗,y]− V[R,y] ≥ 0.
The proof is completed.

heading: B.4 PROOF OF THEOREM 4.3
text: Denote B2 = max{|f ′(c1)|, |f ′(c2)|}, B3 = max|s|≤2L2 √ d logn+B2 |f∗(s)|. We set the network parameters of the representer Rθ and the discriminator Dφ as follows.
(i) Representer networkRD,W,S,B parameters: depth D = 9 log n+ 12, width W = dmax{8d(n p 2+p / log n) 1 p + 4p, 12n p 2+p / log n+ 14}, size S = dn p−2 p+2 / log4(npd),
B = (2B3L1 √ p+ log n) √ d,
(ii) Discriminator network MD̃,W̃,S̃,B̃ parameters: depth D̃ = 9 log n + 12, width W̃ = max{8d(n d 2+d / log n) 1 d + 4d, 12n d 2+d / log n + 14}, size S̃ = n d−2 d+2 /(log4 npd), B̃ =
2L2 √ d log n+B2.
Before getting into the details of the proof of Theorem 4.3, we first give an outline of the basic structure of the proof.
Without loss of generality, we assume that λ = 1 and m = 1, i.e. y ∈ R. First we consider the scenario that y is bounded almost surely, say |y| ≤ C1. We also assumeB1 <∞. We can utilize the truncation technique to transfer the unbounded cases to the bounded ones under some common tail assumptions. Consequently, an additional log n multiplicative term will appear in the final results. For any R̄ ∈ ND,W,S,B, we have,
L(R̂θ)− L(R∗) = L(R̂θ)− L̂(R̂θ) + L̂(R̂θ)− L̂(R̄) + L̂(R̄)− L(R̄) + L(R̄)− L(R∗) ≤ 2 sup
R∈ND,W,S,B |L(R)− L̂(R)|+ inf R̄∈ND,W,S,B |L(R̄)− L(R∗)|, (10)
where we use the definition of R̂θ in (8) and the feasibility of R̄. Next we bound the two error terms in (10), i.e., the approximation error infR̄∈ND,W,S,B |L(R̄) − L(R
∗)| and the statistical error supR∈ND,W,S,B |L(R) − L̂(R)| separately. Then Theorem 4.3 follows after bounding these two error terms.

heading: B.4.1 THE APPROXIMATION ERROR

heading: Lemma B.1.
text: inf
R̄∈ND,W,S,B |L(R̄)− L(R∗)| ≤ 2600C1B1L1
√ pdn− 2 p+2 . (11)
Proof. By (3) and (6) and the definition of L, we have
inf R̄∈ND,W,S,B
|L(R̄)− L(R∗)| ≤ |Df (µR̄θ̄(x)‖γd)|+ |V[R ∗(x),y]− V[R̄θ̄(x),y]|, (12)
where R̄θ̄ ∈ ND,W,S,B is specified in Lemma B.2 below. We finish the proof by (14) in Lemma B.3 and (15) in Lemma B.4, which will be proved below.
Lemma B.2. Define R̃∗(x) = min{R∗(x), log n}. There exist a R̄θ̄ ∈ ND,W,S,B with depth D = 9 log n + 12, width W = dmax{8d(n p 2+p / log n) 1 p + 4d, 12n p 2+p / log n + 14}, and size S = dn p−2 p+2 /(log4 npd), B = (2B1L1 √ p+ log n) √ d, such that
‖R̄θ̄ − R̃∗‖L2(µx) ≤ 160L1B1 √ pdn− 2 p+2 . (13)
Proof. Let R̃∗i (x) be the i-th entry of R̃ ∗(x) : Rd → Rd. By the assumption on R∗, it is easy to check that R̃∗i (x) is Lipschitz continuous on [−B1, B1]d with the Lipschitz constant L1 and ‖R̃∗i ‖L∞ ≤ log n. By Theorem 4.3 in Shen et al. (2019), there exists a ReLU network R̄θ̄i with with depth 9 log n + 12, width max{8d(n p 2+p / log n) 1 p + 4d, 12n
p 2+p / log n + 14}, ‖R̄θ̄i‖L∞ =
2B1L1 √ p+ log n, such that
‖R̄θ̄i‖L∞ ≤ 2B1L1 √ p+ log n,
and ‖R̃∗i − R̄θ̄i‖L∞([−B1,B1]p\H) ≤ 80L1B1 √ pn− 2 p+2 ,
µx(H) ≤ 80L1B1
√ pn− 2 p+2
2B1L1 √ p+ log n .
Define R̄θ̄ = [R̄θ̄1 , . . . , R̄θ̄d ] ∈ ND,W,S,B. The above three display implies ‖R̄θ̄ − R̃∗‖L2(µx) ≤ 160L1B1 √ pdn− 2 p+2 .

heading: Lemma B.3.
text: |V[R∗(x),y]− V[R̄θ̄(x),y]| ≤ 2580C1B1L1 √ pdn− 2 p+2 . (14)
Proof. Recall that Székely et al. (2007)
V[z,y] =E [‖z1 − z2‖|y1 − y2|]− 2E [‖z1 − z2‖|y1 − y3|] + E [‖z1 − z2‖]E [|y1 − y2|] ,
where (zi,yi), i = 1, 2, 3 are i.i.d. copies of (z,y). We have
|V[R∗(x),y]− V[R̄θ̄(x),y]| ≤ |E [ (‖R∗(x1)−R∗(x2)‖ − ‖R̄θ̄(x1)− R̄θ̄(x2)‖)|y1 − y2| ] |
+ 2|E [ (‖R∗(x1)−R∗(x2)‖ − ‖R̄θ̄(x1)− R̄θ̄(x2)‖)|y1 − y3| ] |
+ |E [ ‖R∗(x1)−R∗(x2)‖ − ‖R̄θ̄(x1)− R̄θ̄(x2) ] E [‖y1 − y2‖] |
≤ 8C1E [ |‖R∗(x1)−R∗(x2)‖ − ‖R̄θ̄(x1)− R̄θ̄(x2)‖| ] ≤ 16C1E [ |‖R∗(x)− R̄θ̄(x)‖
] ≤ 16C1(E [ ‖R̃∗(x)− R̄θ̄(x)‖ ] + E [ ‖R∗(x)1R∗(x)∈Ballc(0,logn)‖ ] ),
where in the first and third inequalities we use the triangle inequality, and second one follows from the boundedness of y. By (13), the first term in the last line is bounded by 2560C1B1L1 √ pdn− 1 p+2 . Some direct calculation shows that
E [ ‖R∗(x)1R∗(x)∈Ballc(0,logn)‖ ] ≤ C2 (log n)d
n .
We finish the proof by comparing the order of the above two terms, i.e., C2 (logn)d
n ≤ 20C1B1L1 √ pdn− 2 p+2 .
Lemma B.4. |Df (µR̄θ̄(x)‖γd)| ≤ 20C1B1L1 √ pdn− 2 p+2 . (15)
Proof. By Lemma B.2 R̄θ̄ can approximate R ∗ arbitrary well, the desired result follows from the fact that Df (µR∗(x)‖γd) = 0 and the continuity of Df (µR(x)‖γd) on R. We present the sketch of the proof and omit the details here. Let r∗(z) = dµR∗(x)dγd (z) and r̄(z) = dµR̄θ̄(x) dγd (z). By definition we have
Df (µR∗(x)‖γd) = EW∼γd [f(r∗(W ))] = EW∼γd [f(r∗(W ))1W∈Ball(0,logn)] + EW∼γd [f(r∗(W ))1W∈Ballc(0,logn)].
(We can represent Df (µR̄θ̄‖γd) similarly. ) Then
|Df (µR̄θ̄(x)‖γd)| = |Df (µR̄θ̄(x)‖γd)− Df (µR∗(x)‖γd)| ≤ EW∼γd [|f(r∗(W ))− f(r̄(W ))|1W∈Ball(0,logn)] + EW∼γd [|f(r∗(W ))− f(r∗(W ))|1W∈Ballc(0,logn)]
≤ ∫ ‖z‖≤logn |f ′(r̃(z))||r∗(z)− r̄(z)|dγd(z) + ∫ ‖z‖>logn |f ′(r̃(z))||r∗(z)− r̄(z)|dγd(z)
≤ C3 ∫ ‖z‖≤logn |r∗(z)− r̄(z)|dγd(z) + C4 ∫ ‖z‖>logn |r∗(z)− r̄(z)|
The first term in the above display is small due to R̄θ̄ can approximate R ∗ well. The second term is small due to the boundedness of r̄ and the exponential decay of the Gaussian tails.

heading: B.4.2 THE STATISTICAL ERROR

heading: Lemma B.5.
text: sup R∈ND,W,S,B
|L(R)− L̂(R)| ≤ C15(B1(L1 + L2) √ pd)n− 2 2+p + (L2 √ d+B2 +B3) log nn − 22+d )
(16)
Proof. By the definition and the triangle inequality we have
E[ sup R∈ND,W,S,B |L(R)− L̂(R)|]
≤ E[ sup R∈ND,W,S,B |V̂n[R(x),y]− V[(R(x),y)|]
+ E[ sup R∈ND,W,S,B |D̂f (µR(x)||γd)− Df (µR(x)||γd)|].
We finish the proof based on (17) in Lemma B.6 and (22) in Lemma B.7, which will be proved below.

heading: Lemma B.6.
text: E[ sup R∈ND,W,S,B
|V̂n[R(x),y]− V[R(x),y]|] ≤ 4C6C7C10B1L1 √ pdn− 2 p+2 . (17)
Proof. We first fix some notation for simplicity. Denote O = (x,y) ∈ Rp × R1 and Oi = (xi,yi), i = 1, ...n are i.i.d copy of O, and denote µx,y and P ⊗ n as P and Pn, respectively. ∀R ∈ ND,W,S,B, let Õ = (R(x),y) and Õi = (R(xi),yi), i = 1, ...n are i.i.d copy of Õ. Define centered kernel h̄R : (Rp × R1) ⊗ 4 → R as
h̄R(Õ1, Õ2, Õ3, Õ4) = 1 4 ∑ 1≤i,j≤4, i6=j ‖R(xi)−R(xj)‖|yi − yj |
− 14 ∑4 i=1 (∑ 1≤j≤4, j 6=i ‖R(xi)−R(xj)‖ ∑ 1≤j≤4, i6=j |yi − yj | )
+ 124 ∑
1≤i,j≤4, i6=j
‖R(xi)−R(xj)‖ ∑
1≤i,j≤4, i6=j
|yi − yj | − V[R(x),y] . (18)
Then, the centered U -statistics V̂n[R(x),y]− V[R(x),y] can be represented as
Un(h̄R) = 1
C4n ∑ 1≤i1<i2<i3<i4≤n h̄R(Õi1 , Õi2 , Õi3 , Õi4).
Our goal is to bound the supremum of the centeredU -process Un(h̄R) with the nondegenerate kernel h̄R. By the symmetrization randomization Theorem 3.5.3 in De la Pena & Giné (2012), we have
E[ sup R∈ND,W,S,B |Un(h̄R)|] ≤ C5E[ sup R∈ND,W,S,B | 1 C4n ∑ 1≤i1<i2<i3<i4≤n i1 h̄R(Õi1 , Õi2 , Õi3 , Õi4)|],
(19) where, i1 , i1 = 1, ...n are i.i.d Rademacher variables that are also independent with Õi, i = 1, . . . , n. We finish the proof by upper bounding the above Rademacher process with the matric entropy of ND,W,S,B. To this end we need the following lemma.
Lemma B.7. If ξi, i = 1, ...m are m finite linear combinations of Rademacher variables j , j = 1, ..J . Then
E j ,j=1,...J max 1≤i≤m |ξi| ≤ C6(logm)1/2 max 1≤i≤m
( Eξ2i )1/2 . (20)
Proof. This result follows directly from Corollary 3.2.6 and inequality (4.3.1) in De la Pena & Giné (2012) with Φ(x) = exp(x2).
By the boundedness assumption on y and the boundedness of R ∈ ND,W,S,B, we have that the kernel h̄R is also bounded, say
‖h̄R‖L∞ ≤ C7(2B1L1 √ p+ log n) √ d. (21)
∀R, R̃ ∈ ND,W,S,B define a random empirical measure (depends on Oi, i = 1, . . . , n)
en,1(R, R̃) = E i1 ,i1=1,...,n| 1
C4n ∑ 1≤i1<i2<i3<i4≤n i1(h̄R − h̄R̃)(Õi1 , . . . , Õi4)|.
Condition on Oi, i = 1, . . . , n, let C(N , en,1, δ)) be the covering number ofND,W,S,B with respect to the empirical distance en,1 at scale of δ > 0. Denote Nδ as the covering set of ND,W,S,B with
cardinality of C(N , en,1, δ)). Then,
E i1 [ sup R∈ND,W,S,B | 1 C4n ∑ 1≤i1<i2<i3<i4≤n i1 h̄R(Õi1 , Õi2 , Õi3 , Õi4)|]
≤ δ + E i1 [ sup R∈Nδ | 1 C4n ∑ 1≤i1<i2<i3<i4≤n i1 h̄R(Õi1 , Õi2 , Õi3 , Õi4)|]
≤ δ + C6 1
C4n (logC(N , en,1, δ))1/2 max R∈Nδ [ n∑ i1=1 ∑ i2<i3<i4 (h̄R(Õi1 , Õi2 , Õi3 , Õi4)) 2]1/2
≤ δ + C6C7(2B1L1 √ p+ log n) √ d(logC(N , en,1, δ))1/2 1
C4n [
n(n!)2
((n− 3)!)2 ]1/2
≤ δ + 2C6C7(2B1L1 √ p+ log n) √ d(logC(N , en,1, δ))1/2/ √ n
≤ δ + 2C6C7(2B1L1 √ p+ log n) √ d(VCN log 2eBn δVCN )1/2/ √ n ≤ δ + C6C7C10(B1L1 √ p+ log n) √ d(DS logS log Bn
δDS logS )1/2/
√ n.
where the first inequality follows from the triangle inequality, the second inequality uses (20), the third and fourth inequalities follow after some algebra, and the fifth inequality holds due to C(N , en,1, δ) ≤ C(N , en,∞, δ) and the relationship between the metric entropy and the VCdimension of the ReLU networks ND,W,S,B (Anthony & Bartlett, 2009), i.e.,
logC(N , en,∞, δ)) ≤ VCN log 2eBn δVCN ,
and the last inequality holds due to the upper bound of VC-dimension for the ReLU network ND,W,S,B satisfying C8DS logS ≤ VCN ≤ C9DS logS, see Bartlett et al. (2019). Then (17) holds by the selection of the network parameters and set δ = 1n and some algebra.

heading: Lemma B.8.
text: E[ sup R∈ND,W,S,B
|D̂f (µR(x)||γd)−Df (µR(x)||γd)|] ≤ C14(L2 √ d+B2 +B3)(n − 2 2+p + lognn− 2 2+d ) (22)
Proof. ∀R ∈ ND,W,S,B, let r(z) = dµR(x) dγd (z), gR(z) = f ′(r(z)). By assumption gR(z) : Rd → R is Lipschitz continuous with the Lipschitz constant L2 and ‖gR‖L∞ ≤ B2. Without loss of generality, we assume supp(gR) ⊆ [− log n, log n]d. Then, similar to the proof of Lemma B.2 we can show that there exists a D̄φ̄ ∈ MD̃,W̃,S̃,B̃ with depth D̃ = 9 log n + 12, width W̃ = max{8d(n d 2+d / log n) 1 d + 4d, 12n d 2+d / log n + 14}, and size S̃ = n d−2 d+2 /(log4 npd), B̃ = 2L2 √ d log n+B2 such that for z ∼ γd and z ∼ µR(x)
Ez[|D̄φ̄(z)− gR(z)|] ≤ 160L2 √ d log nn− 2 d+2 . (23)
∀g : Rd → R, define
E(g) = Ex∼µx [g(R(x))]− EW∼γd [f∗(g(W ))],
Ê(g) = Ê(g,R) = 1 n n∑ i=1 [g(R(xi))− f∗(g(Wi))].
By (6) we have E(gR) = Df (µR(x)||γd) = sup
measureable D:Rd→R E(D). (24)
Then,
|Df (µR(x)||γd)− D̂f (µR(x)||γd)|
= |E(gR)− max Dφ∈MD̃,W̃,S̃,B̃ Ê(Dφ)|
≤ |E(gR)− sup Dφ∈MD̃,W̃,S̃,B̃ E(Dφ)|+ | sup Dφ∈MD̃,W̃,S̃,B̃ E(Dφ)− max Dφ∈MD̃,W̃,S̃,B̃ Ê(Dφ)|
≤ |E(gR)− E(D̄φ̄)|+ sup Dφ∈MD̃,W̃,S̃,B̃ |E(Dφ)− Ê(Dφ)|
≤ Ez∼µR(x) [|gR − D̄φ̄|(z)] + EW∼γd [|f ∗(gR)− f∗(D̄φ̄)|(W )] + sup Dφ∈MD̃,W̃,S̃,B̃ |E(Dφ)− Ê(Dφ)| ≤ 160(1 +B3)L2 √ d log nn− 2 d+2 + sup
Dφ∈MD̃,W̃,S̃,B̃ |E(Dφ)− Ê(Dφ)|
where we use the triangle inequality in the first inequality, and we use E(gR) ≥ supDφ∈MD̃,W̃,S̃,B̃ E(Dφ) followed from (24) and the triangle inequality in the second inequality, the third inequality follows from the triangle inequality, and the last inequality follows from (23) and the mean value theorem. We finish the proof via bounding the empirical process
U(D,R) = E[ sup R∈ND,W,S,B,D∈MD̃,W̃,S̃,B̃ |E(D)− Ê(D)|].
Let S = (x, z) ∼ µx ⊗ γd and Si, i = 1, . . . , n be n i.i.d copy of S. Denote b(D,R;S) = D(R(x))− f∗(D(z)). Then E(D,R) = ES [b(D,R;S)] and
Ê(D,R) = 1 n n∑ i=1 b(D,R;Si).
Let
G(M×N ) = 1 n E{Si, i}ni
[ sup
R∈ND,W,S,B,D∈MD̃,W̃,S̃,B̃ | n∑ i=1 ib(D,R;Si)| ] be the Rademacher complexity of MD̃,W̃,S̃,B̃ × ND,W,S,B (Bartlett & Mendelson, 2002). Let C(M× N , en,1, δ)) be the covering number of MD̃,W̃,S̃,B̃ × ND,W,S,B with respect to the empirical distance (depends on Si)
dn,1((D,R), (D̃, R̃)) = 1
n E i [ n∑ i=1 | i(b(D,R;Si)− b(D̃, R̃;Si))|]
at scale of δ > 0. LetMδ ×Nδ be such a converging set ofMD̃,W̃,S̃,B̃ ×ND,W,S,B. Then,
U(D,R) = 2G(M×N ) = 2ES1,...,Sn [E i,i=1,...,n[G(N ×M)|(S1, ..., Sn)]]
≤ 2δ + 2 n ES1,...,Sn [E i,i=1,...,n[ sup (D,R)∈Mδ×Nδ | n∑ i=1 ib(D,R;Si)||(S1, . . . , Sn)]
≤ 2δ + C12 1
n ES1,...,Sn [(logC(M×N , en,1, δ))1/2 max (D,R)∈Mδ×Nδ [ n∑ i=1 b2(D,R;Si)] 1/2]
≤ 2δ + C12 1
n ES1,...,Sn [(logC(M×N , en,1, δ))1/2
√ n(2L2 √ d log n+B2 +B3)]
≤ 2δ + C12 1√ n
(2L2 √ d log n+B2 +B3)(logC(M, en,1, δ) + logC(N , dn,1, δ))1/2
≤ 2δ + C13 L2 √ d log n+B2 +B3√
n (DS logS log Bn δDS logS + D̃S̃ log S̃ log B̃n δD̃S̃ log S̃ )1/2
where the first equality follows from the standard symmetrization technique, the second equality holds due to the iteration law of conditional expectation, the first inequality follows from the triangle inequality, and the second inequality uses equation 20, the third inequality uses the fact that b(D,R;S) is bounded, i.e., ‖b(D,R;S)‖L∞ ≤ 2L2 √ d log n+B2 +B3, and the fourth inequality follows from some algebra, and the fifth inequality follows from C(N , en,1, δ) ≤ C(N , en,∞, δ) (similar result for M) and logC(N , en,∞, δ)) ≤ VCN log 2eBnδVCN , and ND,W,S,B satisfying C8DS logS ≤ VCN ≤ C9DS logS, see Bartlett et al. (2019). Then (22) follows from the above display with the selection of the network parameters ofMD̃,W̃,S̃,B̃,ND,W,S,B and with δ = 1 n .
Finally, Theorem 4.3 is a direct consequence of (11) in Lemma B.1 and (16) in Lemma B.5. This completes the proof of Theorem 4.3.

Review summaries:
The authors present a new representation learning algorithm that trades off between a sufficiency condition (that is, the label should be independent of the input conditioned on the representation) and what they call a "disentangling" condition - that the representation vectors should be independent of one another and rotationally invariant. While the first condition has been used to define disentangled representations, the second is not standard. From the condition of rotational invariance, they require that the distribution over representations is isomorphic to a uniform Gaussian. They arrive at a loss with two terms, the first is a distance correlation between labels and representation, and the second is a divergence between the representation and a uniform Gaussian. In this sense, the regularization term looks quite similar to a VAE while the loss term looks quite similar to standard classification losses. The regularization is represented as a maximum over another loss, leading to a GAN-like coupled optimization problem.