Year: 2021

Abstract:
ABSTRACT This work proposes a new computational framework for automatically learning a closed-loop transcription between multi-class multi-dimensional data and a linear discriminative representation (LDR) that consists of multiple multi-dimensional linear subspaces. In particular, we argue that the optimal encoding and decoding mappings sought can be formulated as the equilibrium point of a two-player minimax game between the encoder and decoder. A natural utility function for this game is the so-called rate reduction, a simple information-theoretic measure for distances between mixtures of subspace-like Gaussians in the feature space. Our formulation avoids expensive evaluating and minimizing approximated distances between arbitrary distributions in either the data space or the feature space. To a large extent, conceptually and computationally this new formulation unifies the benefits of Auto-Encoding and GAN and naturally extends them to the settings of learning a both discriminative and generative representation for complex multiclass and multi-dimensional real-world data. Our extensive experiments on many benchmark datasets demonstrate tremendous potential of this framework: under fair comparison, visual quality of the learned decoder and classification performance of the encoder is competitive and often better than existing methods based on GAN, VAE or a combination of both.

heading: 1 INTRODUCTION AND RELATED WORK
text: One of the most fundamental tasks in machine learning is to learn and model complex distributions (or structures) of real-world data, such as images or texts, from a set of observed samples. By “learn and model”, one typically means that we want to establish a (parameteric) mapping between the distribution of the real data, say x ∈ RD, and a more compact random variable, say z ∈ Rd:
f(·, θ) : x ∈ RD 7→ z ∈ Rd or the inverse g(·, η) : z ∈ Rd 7→ x ∈ RD, (1) where z has certain standard structure or distribution (e.g. normal distributions). The so-learned representation or feature z would be much easier to use for either generative or discriminative purposes. Be aware that the support of the distribution of x (and that of z) can be low-dimensional hence the above map(s) may not be so well defined off the support in the space RD (or Rd).

heading: 1.1 LEARNING GENERATIVE MODELS VIA AUTO-ENCODING OR GAN
text: Auto-Encoding. In the machine learning literature, roughly speaking, there have been two representative approaches to such a distribution-learning task. One is the classic “Auto Encoding” (AE) approach (Kramer, 1991; Hinton & Zemel, 1993) that aims to simultaneously learn an encoding mapping f from x to z and an (inverse) decoding mapping g from z back to x:
X f(x,θ)−−−−−−→ Z g(z,η)−−−−−−→ X̂. (2)
Here we use bold capital letters to indicate a matrix of finite samples X = [x1, . . . ,xn] ∈ RD×n of x and their mapped features Z = [z1, . . . ,zn] ⊂ Rd×n, respectively. Typically, one wishes for two properties: firstly, the (empirical) distribution of the mapped samples Z, denoted as p̂(z |X, θ), is close to certain desired distribution p(z), say some much lower-dimensional disentangled multivariate Gaussian;1 and, secondly, the decoded samples X̂ are “similar” or close to the original X .
In the feature space z, to enforce the learned distribution p̂(z) to be close to a target p(z), one normally minimizes a certain “distance” between the two distributions, say the KL-divergence: minDKL(p̂, p). However, it is very difficult, often computationally intractable, to precisely compute or minimize such a distance D between two arbitrary degenerate distributions in high-dimensional
1The classical PCA can be viewed as a special case of this task. In fact, the original auto-encoding is precisely justified as nonlinear PCA (Kramer, 1991).
spaces. So in practice, one typically chooses to minimize instead certain approximate bounds (surrogates) of the distance, such as a variational bound in variational auto-encoding (VAE) (Kingma & Welling, 2013; Zhao et al., 2017) and the earth mover’s (Wasserstein) distance. In this work, we will show that if we impose specific requirements on the (distribution of) learned feature z (e.g. being subspace-like Gaussians), natural distances can be given in closed-form.
GAN. Compared to measuring distribution distance in the (often controlled) feature space z, a much more challenging issue is how to effectively measure the distance between the decoded samples X̂ and the original X in the data space x. For instance, for visual data such as images, despite extensive studies in the computer vision and image processing literature (Wang et al., 2004), it remains elusive to find a good measure for similarity of real images that is both efficient to compute and effective in capturing visual quality and semantic information of the images equally well. Generative Adversarial Nets (GAN) (Goodfellow et al., 2014) offers an ingenious idea to resolve this difficulty by utilizing a powerful discriminator d, usually modeled and learned by a deep network, to discern differences between the generated samples X̂ and the real ones X:
Z g(z,η)−−−−−−→ X̂,X d(x,θ)−−−−−−→ Y ∈ 0,1}. (3)
Such a discriminator minimizes the Jensen-Shannon divergence, between the data X and X̂ . But as shown in (Arjovsky et al., 2017), the JS-divergence can be highly ill-conditioned to optimize when the distributions are low-dim. So in practice, one may choose to replace the JS-divergence with the earth mover’s distance or other variants. The original GAN aims to directly learn a mapping g, called a generator, from a standard distribution (say a low-dim Gaussian random field) to the real (visual) data distribution in a high-dim space. However, distributions of real-world data can be rather sophisticated and often contain multiple classes and multiple factors in each class (Bengio et al., 2013). That makes learning the mapping g rather challenging in practice, suffering difficulties such as mode-collapse (Srivastava et al., 2017). As a result, many variants of GAN have been subsequently developed in order to improve the stability and performance in learning multiple modes and disentangling different factors in the data distribution, such as Conditional GAN (Mirza & Osindero, 2014; Sohn et al., 2015; Mathieu et al., 2016; Van den Oord et al., 2016; Wang et al., 2018), InfoGAN (Chen et al., 2016; Tang et al., 2021), or Implicit Maximum Likelihood Estimation (IMLE) (Li & Malik, 2018; Li et al., 2020)
In particular, to learn a generator for multi-class data, prevalent conditional GAN literature requires label information as inputs during training (Mirza & Osindero, 2014; Odena et al., 2017; Dumoulin et al., 2016; Brock et al., 2018). Recently Wu et al. (2019b;a) has proposed to train a k-class GAN by generalizing the two-class cross entropy to a (k + 1)-class cross entropy. In this work, we will introduce a more refined 2k-class measure for the k real and k generated classes. In addition, to avoid features for each class to collapse to a singleton, we will use the so-called rate reduction measure that promotes multi-dimension in the learned features (Yu et al., 2020).
Another line of research is about how to stablize the training of GAN. SN-GAN (Miyato et al., 2018) has shown spectral normalization on the discriminator is rather effective, which we will adopt in our work. PacGAN (Lin et al., 2018) shows that the training stability can be significantly improved by packing a pair of real and fake images together for the discriminator. In this work, we show how to naturally generalize this idea to discriminating an arbitrary number of pairs of real and decoded samples. Also, Wu et al. (2019a) has shown that optimizing the latent features leads to state of the art visual quality. There are strong reasons to believe that their method essentially utilizes the Compressed Sensing principle to implicitly exploit low-dimensionality of the feature distribution. Our framework will explicitly impose and exploits such low-dimensional structures of the learned feature distribution.
Combination of AE and GAN. Although AE (VAE) and GAN have started with somewhat different motivations, they have evolved into popular and effective frameworks for learning and modeling complex distributions of many real-world data such as images. Many recent efforts tend to combine both Auto-Encoding and GAN to generate more powerful generative frameworks for more diverse data sets, such as Larsen et al. (2015); Rosca et al. (2017); Srivastava et al. (2017); Bao et al. (2017); Huang et al. (2018); Ulyanov et al. (2018); Vahdat & Kautz (2020). As we will see, in our framework, AE and GAN can be naturally interpreted as two segments of a closed-loop data transcription process. But unlike GAN or VAE, the distribution of the feature z is learned from the data x and its low-dim support in Rd is explicitly modeled as a union of discriminative subspaces.

heading: 1.2 LEARNING LINEAR DISCRIMINATIVE REPRESENTATION VIA RATE REDUCTION
text: Recently, Chan et al. (2021) has proposed a new objective for deep learning that aims to learn a linear discriminative representation (LDR) for multi-class data. The basic idea is to map distributions of real data, potentially on multiple nonlinear submanifolds, to a family of canonical models consisting of multiple incoherent (or orthogonal) linear subspaces. To some extent, this generalizes nonlinear PCA (Kramer, 1991) to the more general/realistic settings where we simultaneously apply multiple nonlinear PCAs to data on multiple nonlinear submanifolds. Unlike conventional discriminative methods that only aim to predict class labels, the LDR aims to learn the likely multi-dimensional distribution of the data hence is potentially suitable for both discriminative and generative purposes. It has been shown that this can be achieved via maximizing the so-called “rate reduction” objective.
MCR2. More precisely, consider a set of data samples X = [x1, . . . ,xn] ∈ RD×n from k different classes and we use Πj , j = 1, . . . , k to denote the memberships of the samples in k classes X = ∪kj=1Xj . One seeks a continuous mapping f(·, θ) : x 7→ z from X to an optimal representation Z = [z1, . . . ,zn] ⊂ Rd×n that maximizes the following coding rate reduction objective, known as the MCR2 principle (Yu et al., 2020):
max Z
∆R(Z,Π, ) . =
1 2 log det
( I + αZZ∗ ) ︸ ︷︷ ︸
R(Z, )
− k∑ j=1 γj 2 log det ( I + αjZΠ jZ∗ )
︸ ︷︷ ︸ Rc(Z, |Π)
, (4)
where α = dn 2 , αj = d tr(Πj) 2 , γj = tr(Πj) n for j = 1, . . . , k. In this paper, for simplicity we denote ∆R(Z,Π, ) as ∆R(Z) assuming Π, are known and fixed. The first term R(Z | ) is the coding rate of the whole feature set Z (coded as a Gaussian source) with a prescribed precision ; the second term Rc(Z |Π, ) is the average coding rate of the k subsets of features Zj = f(Xj) (each coded as a Gaussian). As it has been shown by Yu et al. (2020), maximizing the difference between the two terms will “expand” the whole feature set while “compressing and linearizing” features of each of the k classes. If the mapping f maximizes the rate reduction, it maps the features of different classes into independent (orthogonal) subspaces in Rd. Figure 1 illustrates a simple example of data with k = 2 classes (on two submanifolds) mapped to two incoherent subspaces (solid black lines).

heading: 2 DATA TRANSCRIPTION VIA RATE REDUCTION

heading: 2.1 LDR TRANSCRIPTION
text: One issue with this one-sided LDR learning is that maximizing the above objective tends to maximize the dimension of the learned subspace for features in each class.2 To verify whether the learned features are good, we may consider learning a decoder g(·, η) : z 7→ x from the representation Z = f(X, θ) back to the data space x: X̂ = g(Z, η), and check how close X and X̂ are or how close their features Z and Ẑ are. The overall pipeline can be illustrated by the following diagram:
X f(x,θ)−−−−−−→ Z g(z,η)−−−−−−→ X̂ f(x,θ)−−−−−−→ Ẑ, (5)
where the overall model has parameters: Θ = {θ, η}. Notice that in the above process, the segment from X to (X̂,X) resembles a typical Auto-Encoding process although, as we will soon see, our MCR2-based encoder f plays an additional role as a discriminator. The segment from Z to (Ẑ,Z) draws resemblance to the typical GAN process although, in our context, the distribution of the latent variable z will be learned from the data x, which often is a distribution with multiple modes and each mode with multiple factors.
Here, in the specific context of rate reduction, we name this special auto-encoding process LDR Transcription since the maximal rate reduction principle explicitly transcribes the data X , via f , to features Z on a linear discriminative representation (LDR), which can be subsequently decoded back to the data space X̂ , via g. Hence, the encoding and decoding maps f and g together form a closed-loop process, as illustrated in Figure 1. We wish this closed-loop transcription process to have the following good properties:
• Injectivity: the generated x̂ = g(f(x, θ), η) ∈ X̂ to be as close to (ideally the same as) the original data x ∈X , in terms of certain measure of similarity or distance.
2If the dimension of the feature space d is too high, maximizing the rate reduction may over-estimate the dimension of each class. Hence to learn a good representation, one needs to pre-select a proper dimension for the feature space, as done in the experiments in (Yu et al., 2020).
• Surjectivity: for all mapped images z = f(x) ∈ Z of the training data x ∈X , there are decoded samples ẑ = f(g(z, η), θ) ∈ Ẑ close to (ideally the same as) z.

heading: 2.2 MEASURING DISTANCES IN THE FEATURE SPACE AND DATA SPACE
text: Contractive Measure. For the second item in the above wishlist, as the representations in the feature space z are by design linear subspaces or (degenerate) Gaussians, we have geometrically or statistically meaningful metrics for both samples and distributions in the feature space z. For example, we care about distance between distributions between the features of the original data Z and the transcribed Ẑ. Since the features of each class, Zj and Ẑj , are subspace/Gaussian-like, their “distance” can be measured by the rate reduction, with (4) restricted to two sets of equal size):
∆R ( Zj , Ẑj ) . = R ( Zj ∪ Ẑj ) − 1
2
( R ( Zj) +R ( Ẑj) ) , (6)
which, as shown in Fig. 1, measures the space volume between a pair of black lines and blue lines, as per interpretation of rate reduction in (Yu et al., 2020). Then for the “distance” of all, say k, classes, we simply sum the rate reduction for all pairs:
d(Z, Ẑ) . = min
η k∑ j=1 ∆R ( Zj , Ẑj ) = min η k∑ j=1 ∆R ( Zj , f(g(Zj , η), θ) ) , (7)
where Zj = f(Xj , θ) and Ẑj = f(X̂j , θ). Obviously, a main goal of the learned decoder g(·, η) is to minimize the distance between these distributions. Notice that if the encoder f preserves (i.e. injective for) the intrinsic structures of the original data X ,3 this criterion essentially aims to ensure there will be some decoded sample x̂ close to every data sample x – hence the decoder should be “surjective.” According to the ideas of IMLE (Li & Malik, 2018), such a requirement could effectively help avoid mode-collapsing or mode-dropping.
Contrastive Measure. For the first item in our wish-list, however, we normally do not have a natural metric or “distance” for similarity of samples or distributions in the original data space x.4 To alleviate this difficulty, we can measure the similarity or difference between X̂ and X through their mapped images Ẑ and Z in the feature space (again assuming f is structure-preserving). If we are interested in discerning any difference in the distributions of the original and transcribed samples, we may view the MCR2 feature encoder f(·, θ) as a “discriminator” to magnify any difference between all pairs of Xj and X̂j , by simply maximizing, instead of minimizing, the same quantity in (7):
d(X, X̂) . = max
θ k∑ j=1 ∆R ( Zj , Ẑj ) = max θ k∑ j=1 ∆R ( f(Xj , θ), f(X̂j , θ) ) . (8)
That is, a “distance” between X and X̂ can be measured as the maximally achievable rate reduction between all pairs of classes in these two sets. In a way, this measures how well or bad the decoded X̂ aligns with the original data X – hence measuring the goodness of “injectivity”. Notice that such a discriminative measure is consistent with the idea of GAN (Goodfellow et al., 2014). Nevertheless, here the MCR2-based discriminator f naturally generalizes to cases when the data distributions are multi-class and multi-modal, and the discriminativeness is measured with a more refined measure – the rate reduction, instead of the typical two-class loss (e.g. cross entropy) used in GANs.
3This is typically the case for MCR2-based feature representation (Yu et al., 2020). 4As mentioned before, finding proper metrics or distance functions on natural images has always been an
elusive and challenging task (Wang et al., 2004).
One may wonder the reason why we need the mapping f(·, θ) to function as a discriminator between X and X̂ by maximizing maxθ ∆R ( f(X, θ), f(X̂, θ) ) ? Figure 2 gives a simple illustration: there might be many decoders g such that f ◦g is an identity mapping f ◦g(z) = z for all z in the subspace Sz in the feature space. However, g ◦ f is not necessarily an auto-encoding map for x in the original distribution Sx (here for simplicity drawn as a subspace). That is, g ◦ f(Sx) 6⊂ Sx (let alone g ◦ f(x) = x). One should expect, without careful control of the image of g, this would typically be the case especially when the support of the distribution of x is extremely low-dimensional in the original high-dimensional data space (e.g. in the millions for images).

heading: 2.3 ENCODING AND DECODING AS A TWO-PLAYER MINIMAX GAME
text: Comparing (7) and (8), we see the roles of the encoder f(·, θ) and the decoder g(·, η) naturally as a “a two-player game”: while the encoder f tries to magnify the difference between the original data and the transcribed; the decoder g aims to minimize the difference. Now for convenience, let us define the “closed-loop encoding” function:
h(x, θ, η) . = f ( g ( f(x, θ), η ) , θ )
: x 7→ z. (9) Ideally, we want this function to be very close to f(x, θ). With this notation, combining (7) and (8), a closed-loop notion of “distance” between X and X̂ can be computed as an equilibrium point to the following Min-Max program for the same utility in terms of rate reduction:
D(X, X̂) .= min η max θ k∑ j=1 ∆R ( f(Xj , θ), h(Xj , θ, η) ) . (10)
Notice that this only measures the difference between (features of) the original data and its transcribed version. It does not measure how good the representation Z (or Ẑ) is for the multiple classes within X (or X̂). To this end, we may combine the above distance with the original MCR2-type objectives (4): namely, the rate reduction ∆R(Z) and ∆R(Ẑ) for the learned LDR Z for X and Ẑ for the decoded X̂ . The overall multi-class Min-Max program for learning the LDR model is:
min η max θ
∆R ( f(X, θ) )︸ ︷︷ ︸ Expansive encode + ∆R ( h(X, θ, η) )︸ ︷︷ ︸ Compressive decode + k∑ j=1 ∆R ( f(Xj , θ), h(Xj , θ, η) )︸ ︷︷ ︸ Contrastive encode & Contractive decode . = TX(θ, η).
(11) Notice that, without the terms associated with the generative part h or with all such terms fixed as constant, the above objective is precisely the original MCR2 objective.5 Also, notice that the minimax objective function depends only on the data X hence one can learn the encoder and decoder (parameters) without the need of sampling any other distribution (as needed by GANs)!
As a special case, if X only has one class, the above Min-Max program reduces6 to a special “binary” form between X and X̂:7
Binary: min η max θ
∆R ( f(X, θ), h(X, θ, η) ) . (12)
Sometimes, even when X contains multiple classes/modes, one could still view all classes together as one class. Then the above binary objective is to align the union distribution of all classes with their decoded X̂ . This is typically a simpler task to achieve than the multi-class one (11) since it does not require to learn a more refined multi-class LDR for the data, as we will later see in experiments.
5In an unsupervised setting, if we view each sample (and its augmentations) as its own class, the above formulation remains exactly the same! The number of classes k is simply the number of independent samples.
6as the first two terms become zero. 7Notice that this binary case resembles formulation of the original GAN (3) by viewing X and X̂ as two classes {0,1}. Nevertheless, instead of using cross entropy, our formulation adopts a more refined rate reduction measure, which has been shown to promote diversity in the learned representation (Yu et al., 2020).

heading: 2.4 DISCUSSIONS ABOUT THE MINIMAX OBJECTIVE
text: There are many outstanding questions that one may wonder about the above formulation: When does the game minθ maxη TX(θ, η) have a well-defined Nash-equilibrium point (θ?, η?)? What good characteristics the associated optimal encoder f? and g? have? For instance, under what conditions, will their composition be or close to be a true auto-encoding g? ◦ f? ≈ Id, and will the learned features Z? = f(X, θ?) form orthogonal or highly incoherent subspaces?
In this paper, we will not be able to give complete rigorous answers to these questions. The main purpose of this paper is to provide compelling empirical evidence that answers to the above questions are highly likely to be positive under fair broad conditions (see Section 3). Here we will give some preliminary justification based on characteristics of each team in TX(θ, η) and will also give reasons why a rigorous characterization of the optimality conditions is mathematically challenging.
Let us first consider maximizing the first two rate reduction terms in TX(θ, η), assuming η fixed and Z(θ) and Ẑ(θ) are aligned, i.e., ∆R ( Zj(θ), Ẑj(θ) ) reaches its minimum 0 for j ∈ [k] (to be
discussed below). Then they essentially maximize the rate reduction ∆R(Z(θ)) + ∆R(Ẑ(θ)) for features of the k classes in Z(θ) = f(X, θ) and Ẑ(θ) = h(X, θ, η). Based on the results in Yu et al. (2020), a sufficient condition for the rate reduction to reach a (local) maximum is when features of all classes belong to subspaces orthogonal to each other8: Z>i Zj = 0 and Ẑ > i Ẑj = 0 for i 6= j.
Now let us consider minimizing the k rate reduction terms in the third term of TX(θ, η), assuming θ fixed and the ∆R(Ẑ) achieves its maximum (as mentioned above). Then, from the property of the log det(·) function, we know that each of the k terms in the contractive measure: according to Lemma 10 of Chan et al. (2021), ∆R ( Zj , h(Ẑj(η) ) reaches its minimum (0) when the covariance matrices of Zj and Ẑj are the same, i.e., ZjZ>j = ẐjẐ > j . If one further assumes that the encoder is
sufficiently injective, perfect alignment of Zj and Ẑj suggests perfect alignment of Xj and X̂j .
Based on the above properties, one may show that when the k classes in each of Z and Ẑ form k orthogonal subspace and each corresponding pair of Zj and Ẑj aligns perfectly, then one reaches a critical point9 of the minimax game minθ maxη TX(θ, η). It remains open whether such a critical point is a strict saddle point – a (local) Nash equilibrium.
There are more subtle questions that one could or should ask too, such as: how are the learned features Z and Ẑ distributed within the subspaces? Are features for individual sample zi ẑi (hence xi and x̂i) well aligned? Results in Yu et al. (2020) suggest that covariance of the features in each subspace needs to be nearly isotropic in order for a rate reduction term to be maximized. When minimizing a rate reduction term, results about the so-called Brascamp-Lieb inequalities (Jonathan Bennett et al., 2007) suggest that a necessary condition for the minimum to be reached is when the distribution (density) within each subspace becomes Gaussian. Hence from these results one may conjecture that the optimal distribution in each subspace for the minimax game is Gaussian too. Although theoretically nothing is known about relationships between the features zi ẑi of individual samples, (multi-class) experiments in next section suggest that they are actually very close.
Despite all these challenging open mathematical questions, one may notice that the aggregated effect of all the terms associated with the decoder g(·, η) is precisely “opposite” to the effect of the terms associated with the encoder f(·, θ). All together, the Min-Max program (11), or (12), aims to strike a good tradeoff between maximizing the expressiveness and discriminativeness of the learned representation as well as minimizing any unnecessary cost, in terms of the overall coding rate and errors in decoding, all measured in rate reduction (assuming features being subspace Gaussians).

heading: 3 EMPIRICAL VERIFICATION ON REAL-WORLD IMAGERY DATASETS
text: The experiment section serves three purposes: First, we empirically justify the new formulation by demonstrating good properties of the learned encoder, decoder, and representations. Second, we compare our method with several representative methods from the GAN family and VAE family. Finally, we evaluate the learned LDR through visualization and classification tasks.
8Here one must assume such a mapping f exists which puts conditions on the original data distribution X and the family of functions that f(·, θ) can represent.
9That is, gradients of the utility T are zero w.r.t. θ and η because ∂∆R ∂Z all vanishes.
Datasets. We provide extensive qualitative and quantitative experimental results on the following datasets: MNIST (LeCun et al., 1998), CIFAR-10 (Krizhevsky et al., 2009), STL-10 (Coates et al., 2011), CelebA (Liu et al., 2015), LSUN bedroom (Yu et al., 2015), and ImageNet ILSVRC 2012 (Russakovsky et al., 2015). The network architectures and implementation details can be found in Appendix A.1 and corresponding appendix section for each dataset.

heading: 3.1 EMPIRICAL JUSTIFICATION OF LDR TRANSCRIPTION
text: To empirically validate our new framework, we conduct experiments from a small low-variety dataset (MNIST), to a small dataset of diverse real-world objects (CIFAR-10), to higher resolution images (CelebA, LSUN-bedroom), to a large-scale image set (ImageNet). The results are evaluated both quantitatively and qualitatively. Implementation details and more results are given in the Appendix. Comparison (IS and FID) with other formulations. First, we conduct five experiments to fairly compare our formulation with GAN (Radford et al., 2015) and VAE(-GAN) (Larsen et al., 2016) on MNIST and CIFAR-10. Except for the objective function, everything else is exactly the same for all methods (e.g. networks, training data, optimization method). These experiments are: 1). GAN; 2). GAN with its objective replaced by that of the LDR-Binary (12); 3). VAE-GAN ; 4). Binary LDR (12); and 5). Multi-class LDR (11). Some visual comparison is given in Fig. 3. IS (Salimans et al., 2016) and FID (Heusel et al., 2017) scores are summarized in Table 1.
As we see from the above Table 1, replacing cross-entropy with the Equation (12) can improve the generative quality. The two LDR formulations are clearly on par with the others in terms of IS and significantly better in FID. Finally, with the same training datasets, quality of LDR-Multi is lower than LDR-Binary. This is expected as the multi-class task is more challenging. Nevertheless, as we will see soon, images decoded by LDR-Multi align much better with their classes than Binary.
Visualizing correlation of features Z and decoded features Ẑ. We visualize the cosine similarity between Z and Ẑ learned from the multi-class objective (11) on MNIST and CIFAR-10, which indicates how close ẑ = f ◦ g(z) is from z. Results in Fig 4 show that Z and Ẑ are aligned very well within each class. The block-diagonal patterns for MNIST are sharper than those for CIFAR-10, as images in CIFAR-10 have more diverse visual appearances.
Visualizing auto-encoding of the data X and the decoded X̂ . We compare some representative X and X̂ on MNIST, CIFAR-10 and ImageNet (10 classes) to verify how close x̂ = g ◦ f(x) is to x. The results are shown in Fig 5, and visualizations are created from training samples. Visually, the auto-encoded x̂ faithfully captures the visual features from its respective training sample x. There also exist some auto-encoded images that are almost identical to the original. We refer the reader to Appendix A.2 A.4 A.7 for
more visualizations of results on these datasets, including similar results on transformed MNIST digits and real-life images decoded by randomly sampling the learned feature subspaces.

heading: 3.2 COMPARISON TO EXISTING GENERATIVE METHODS
text: Table 2 gives a quantitative comparison of visual quality of our method with others on CIFAR-10, STL-10, and ImageNet. In general, there is a large difference in terms of FID and IS scores between the GAN family and the VAE family of models. SNGAN (Miyato et al., 2018) are commonly used methods in most generative applications while LOGAN (Wu et al., 2019a) is the state-of-the-art method on ImageNet in terms of FID and IS. As we see, even if the rate reduction is not specifically designed nor engineered for visual quality10, our method is still rather competitive in terms of these metrics.

heading: 3.3 BENEFITS OF THE LEARNED LDR TRANSCRIPTION MODEL
text: As we have argued before, the learned LDR transcription model (including the feature z, the encoder f , and the decoder g) can be used for both generative and discriminative purposes.
Decoding samples from the feature distribution. Using the CIFAR-10 and CelebA datatsets, we visualize images decoded from samples of learned feature subspace. For the CIFAR-10 dataset, for each class j, we first compute the top-4 principal components of the learned features Zj (via SVD). For each class j, we then compute |〈zij ,vlj〉|, the cosine similarity between the l-th principal direction vlj and feature sample z i j . After finding the top-5 z i j according to |〈zij ,vlj〉| for each class j, we reconstruct images x̂ij = g(z i j). Each row of Fig. 6 is for one principal component. We observe that images in the same row share many common features; images in different rows differ significantly in characteristics like shape, background, and style. See Appendix A.4 for more visualization of principal components learned for all 10 classes of CIFAR-10.
For the CelebA dataset, we calculate the principal components of all learned features in the latent space. Fig 7(a) shows some decoded images along these principal directions. These components seem to clearly disentangle visual attributes/factors such as wearing a hat, changing hair color and wearing glasses. More examples can be found in Appendix A.6. The results are consistent with the property of MCR2 that promotes diversity of the learned features. Fig. 7(b) shows interpolating features between pairs of training image samples, where for two training images x1 and x2, we reconstruct based on their interpolated feature representations by x̂ = g(αf(x1)+(1−α)f(x2)), α ∈ [0, 1]. The decoded images show continuous morphing from one sample to another in terms of visual characteristics, as opposed to merely a superposition of the two images.
10In our current implementation, the original objective is used without any other heuristics or regularization.
Encoded features for classification. Notice that not only is the learned decoder good for generative purpose, but also the encoder is good for discriminative tasks. In this experiment, we evaluate the discriminativeness of the learned LDR model by testing how well the encoded features can help classify the images. We use features of the training images to compute the learned subspaces for all classes, then classify features of the test images based on a simple nearest subspace classifier. Results in Table 3 show that our model gives competitive classification accuracy on MNIST, compared to some of best VAE-based methods. We also tested the classification on CIFAR-10, the accuracy is currently about 80.7%. As expected, the representation learned with the multi-class objective is very discriminative and good for classification tasks. This demonstrates the learned LDR model is not only generative but also discriminative.

heading: 3.4 CONCLUSION
text: This work provides a novel formulation for learning a both generative and discriminative representation for multi-class multi-dimensional real-world data. We have provided compelling empirical evidence that the distribution of most data can be effectively mapped to an LDR, a union of incoherent subspaces. The main purpose of this paper is to demonstrate the conceptual simplicity and practical potential of this new representation learning framework, instead of to strive for state of the art performance. Nevertheless, with our preliminary implementation, LDR can be effectively learned for a variety of real-world datasets, from small to large, from domain-specific to diverse. In addition, the so-learned decoder g already enjoys the benefit of GAN for its good generative visual quality and the encoder f with the benefit of AE for its discriminative property. From our experience, the rate reduction based objective can be stably optimized across a wide range of datasets and network architectures without any additional regularizations or engineering tricks. One may notice that there are many ways our method can be significantly improved. For one, in this work, we have simply adopted networks that were designed for GAN or AE, but they may not be optimal for the rate reduction type objectives. Also notice that compared to GAN and AE, our method leads to an explicit model for the feature distribution: a mixture of incoherent subspace Gaussians. Such an explicit model has the potential of making many subsequent tasks much easier and better: importance sampling for decoding, classification, or even incremental learning. We leave all these directions, together with all the open mathematical problems, for future investigation.

heading: ETHICS STATEMENT
text: All authors agree and will adhere to the conference’s Code of Ethics. We do not anticipate any potential ethics issues regarding the research conducted in this work.

heading: REPRODUCIBILITY STATEMENT
text: Settings and implementation details of network architectures, optimization methods, and some common hyper-parameters are described in the Appendix A.1. More detailed experimental settings for different datasets can be found in their corresponding Appendix section entitled with the name of the dataset. We will also make our source code available upon request by the reviewers or the area chairs.

heading: A APPENDIX

heading: A.1 EXPERIMENT SETTINGS AND IMPLEMENTATION DETAILS
text: Network backbones. For MNIST, we use the standard CNN models in Table 4 and Table 5, following the DCGAN architecture (Radford et al., 2015). We resize the MNIST image resolution from 28 × 28 to 32 × 32 to fit DCGAN architecture. All α in lReLU (lReLU is short for Leaky-ReLU) of the encoder are set to 0.2.
We adopted conv ResNet architectures for CIFAR-10 in Tables 6 and 7, and STL-10 shown in Tables 8 and 9. Each ResBlock up is same as Resnet, but add an up-sampler after the first conv layer. All batch normalization layers of ResBlock in encoder are replaced with spectral normalization layer.
Finally, we use the same architecture for CelebA, LSUN-bedroom, ImageNet-128 (see Tables 10 and 11) as all three datasets have the same 128×128 resolution. Again, each ResBlock up is same as Resnet, but add an up-sampler after the first conv layer. And all batch normalization layers in encoder are replaced with spectral normalization layer. All experiments utilize this lightweight PyTorch library that provides implementations of popular state-of-the-art GANs and evaluation metrics.
Table 4: Decoder for MNIST.
z ∈ R1×1×128 4 × 4, stride=1, pad=0 deconv. BN 256 ReLU 4 × 4, stride=2, pad=1 deconv. BN 128 ReLU 4 × 4, stride=2, pad=1 deconv. BN 64 ReLU
4 × 4, stride=2, pad=1 deconv. 1 Tanh
Table 5: Encoder for MNIST.
Gray image x ∈ R32×32×1 4 × 4, stride=2, pad=1 conv 64 lReLU 4 × 4, stride=2, pad=1 conv 128 lReLU 4 × 4, stride=2, pad=1 conv 256 lReLU
4 × 4, stride=1, pad=0 conv 128
Table 6: Decoder for CIFAR-10.
z ∈ R128 dense −→ 4 × 4 × 256
ResBlock up 256 ResBlock up 256 ResBlock up 256
BN, ReLU, 3 × 3 conv, 3 Tanh
RGB image x ∈ R48×48×3 ResBlock down 64
ResBlock down 128 ResBlock down 256 ResBlock down 512
ResBlock 1024 ReLU
Global sum pooling dense −→ 128
Training details. Across all of our experiments, we use Adam (Kingma & Ba, 2014) as our optimizer for all experiments, with hyperparameters β1 = 0, β2 = 0.9. The initial value of learning rate is 0.00015 and is scheduled with linear decay. We choose 2 = 0.5 for both equation 11 and 12 in all LDR experiments. For all LDR-Multi experiments on ImageNet, we only choose 10 classes. The details of the 10 classes as shown in Table 12. Most experiments are trained on RTX 3090ti GPUs.

heading: A.2 MNIST
text: Settings. On MNIST dataset, we train our model using DCGAN (Radford et al., 2015) architecture with our proposed models LDR-Multi (11) and LDR-Binary (12). We set the learning rate to 10−4, batch size to 2048, and training 15,000 iterations. Due to the advantages of the LDR objective, we can achieve between-class discriminative representations while the within-class diversity of these representations can be preserved, which are shown in the following experimental results.
More results illustrating auto-encoding. Here we give more reconstruction results, or X̂ , of our LDR-Multi and LDR-Binary models, compared to their corresponding original input X . As shown in the Fig.8, for the LDR-Binary model, it can generate clean digit-like images but the decoded X̂ might resemble digits from similar but different classes to the input data X since LDR-Binary tends to only align the distribution of all digits.
In contrast, with the LDR-Multi objective, the decoded X̂ not only are coherent with the correct class with the input data X , but also show very clear one-to-one mapping between individual sample x and x̂ although the objective 11 does not enforce that! Comparing with the results from the VAE-GAN (Larsen et al., 2015), our decoded images preserve much better the individual characteristics of the original samples.
Images decoded from random samples on the learned multi-class LDR. Since our LDR-Multi objective function maps input data of each class into a different (orthogonal) subspace in the feature space, we can generate images conditioned on each class by random sampling z in the subspace of each class and then decode them back to the input space as x̂.
To do random sampling in the learned subspace, we first calculate the mean feature z̄j and the singular vectors vij of the SVD (or principal components) of the learned features Zj of the class j of the training data, where index i represents the ith principal components. We only use top r = 8 principal
components of each class on MNIST dataset. These statistics of the subspace can be used for guiding the random sampling. Then we sample z randomly along the principal components and around the mean feature as
zrandom j = z̄j + α r∑ i=1 ni ∗ σij ∗ vij , (13)
where z̄j is the mean feature of class j, σij and v i j are the i-th singular value and principal component of class j, ni are i.i.d. GaussianN (0, 1) random variables. That is, the feature in each subspace/class is modeled by a r-dimensional multivariate Gaussian, with variances σij which characterize variances of the training data in the feature space. Here, α is a hyper-parameter that controls the sampling range. As for visualization of random generated images g(zrandom j) conditioned on the given class, we compare our method with some other conditional generation method such as ACGAN (Odena et al., 2017) and InfoGAN (Chen et al., 2016) (For ACGAN and InfoGAN, we generate images conditioned on class labels with randomly sampled latent z according the procedures mentioned in their respective works). Our model can give realistic and correct conditional generation results with high diversity in each class, while other methods may make mistakes in the generation between some similar classes such as classes 3 and 5 for InfoGAN.

heading: A.3 TRANSFORMED MNIST
text: Settings. In this experiment, we verify that our LDR-Multi model can preserve diverse data modes in the learned feature embeddings. We construct a transformed MNIST dataset with 5 modes: normal, large(1.5 ×), small(0.5 ×), rotate 45◦ left, and rotate 45◦ right. Each image data will be randomly transformed to one of the modes. Representative examples of such training data can be found in Figure 10(a). We train the model with learning rate 1e-4 and batch size 2048 for 15,000 iterations.
Auto-encoding results. Figure 10(b) gives the decoded results of the training data with different modes. Even though the data are now much more diverse for each class, decoder learned from the LDR-Multi objective can still achieve high sample-wise similarity to the original images.
Identifying different modes. Similar to the earlier experiments of Fig. 6 for CIFAR-10 in the main paper, we find the top principal components of features of each class Zj (via SVD) and generate new images using the learned decoder g from features of the training images aligned the best with these components.
In Fig 11, we select three classes 0, 1, 2 and visualize samples from top r = 8 principal components for each class. Each row represents one principal components direction. As it can be seen in the figure, the decoded images along each principal component shows similar mode and the modes along different component directions are rather incoherent. All major modes of the original data can be identified as one of these principal directions. This clearly shows that our LDR-Multi model can keep the different modes within each class of the data Xj as the principal component directions of Zj , and these modes can also be retained in the decoded images X̂j .
A.4 CIFAR-10
Settings. For all experiments on CIFAR-10, we follow the common training hyper-parameters in section A.1. Beyond that, for each experiment, we run 450,000 iterations with mini-batch size 1600.
Images decoded from random samples on the learned multi-class LDR. We sample z in the feature space randomly along the principal components and around the mean feature of each class Zj as in the MNIST case, according to equation (13). The generated images from the sampled features are illustrated in Fig 12, one row per class. As we see, the generator learned from the LDR-multi objective is capable of generating diverse images for each class.
Further, for visualization of random generated images g(zrandom j) conditioned on the given class, we also compare our method with some other conditional generation method such as ACGAN (Odena et al., 2017) and InfoGAN (Chen et al., 2016). For all three experiments, we have randomly sampled 8 images per class in CIFAR-10. For more complex dataset like CIFAR-10, our model can give more realistic conditional generation results for different classes with high diversity within each class.
Generating image along different PCA components for each class. For each class, we first compute top-10 principal components (singular vectors of the SVD) of Z and then for each of the top singular vectors, we display in each row the top-10 reconstructed image X̂ whose Z are closest to the singular vector using method described in the main body of the paper, Section 3.3. The results are given in Figure 13 below.
A.5 STL-10
Settings. For all experiments on STL-10, we follow the common training hyper-parameters in section A.1. For LDR-Binary setting, we train 150,000 iterations. For LDR-Multi setting, we initialize the weights from the 20,000-th iteration of LDR-Binary checkpoint and train for another 80,000 iterations (with the LDR-Multi objective). The IS and FID scores on the STL-10 dataset are reported in Table 13, on par or even better than existing methods such as SNGAN (Miyato et al., 2018) or DC-VAE (Parmar et al., 2021).
Visualizing auto-encoding property for LDR-Binary. We visualize the original images x and their decoded x̂ using the LDR model learned from LDR-Binary objective. The results are shown in Figure 14 for STL-10.

heading: A.6 CELEB-A AND LSUN
text: To verify that our formulation works on images of higher-resolution, we conduct experiments on the Celeb-A and LSUN datasets, which have a resolution of 128× 128.
Settings. For all experiments on these datasets, we follow the common training hyper-parameters in Section A.1. We choose a 300 mini-batch size for Celeb-A and LSUN. Both of them are trained with the LDR-Binary formulation, and for 450,000 iterations.
Generating image along different PCA components. We calculate the principal components of the learned features Z in the latent subspace. We manually choose 3 principle components which are related to hat, hair color, and glasses (see Fig 15). The three components are 9-th, 19-th, and 23-th respectively from the overall 128 principal components. These principal directions seem to clearly disentangle visual attributes/factors such as wearing hat, changing hair color, and wearing glasses.
Images generated from random sampling of the feature space. We sample z randomly according to the following Gaussian model:
zrandom = z̄ + α r∑ i=1 ni ∗ σi ∗ vi, (14)
where z̄ is the mean feature, σi and vi are the ith singular value and singular vector, ni are i.i.d. Gaussian N (0, 1) random variables. As before α is a hyper-parameter to control the sampling range. We use top r=100 principle components for random sampling. The random generated images are realistic and diverse. (see Fig 16)
Visualizing auto-encoding property for LDR-Binary. We visualize the original image x and their decoded x̂ using the LDR model learned from LDR-Binary formulation. The results are shown in Fig 17 and Fig 18 for the Celeb-A dataset and the LSUN dataset, respectively. The LDR-Binary formulation can give very good visual quality for x̂ but cannot ensure sample to sample alignment. Nevertheless, the decoded x̂ seems to be similar in style to the original x. We believe it manages to align only the dominant principal component(s) associated with the most salient visual attributes, say pose of the face for Celeb-A or layout of the room for LSUN, between features of X and X̂ .
A.7 IMAGENET
Settings. To verify that our formulation works on large-scale datasets, we train a model on the entire ImageNet. For all experiments on the ImageNet, we follow the common training hyper-parameters in Section A.1.
We first train our model with LDR-Binary (12) with a mini-batch size 1800 on the whole ImageNet ILSVRC 2012 dataset. The number of training iterations is 450,000.
After that, we fine-tune the Binary-pretrained model with LDR-Multi (11), on 10 selected classes. Information about the 10 classes can be found in Table 12. The fine-tune mini-batch size is 1024, and we train another 35,000 iterations. This experiment took 8 A100-SXM4 GPUs, each with 40GB of CUDA memory for 120 GPU hours. Note that our choice of mini-batch size is substantially larger than those commonly adopted in other works while training on the ImageNet (e.g. 128 in Miyato et al. (2018)). We empirically observe that training with a larger mini-batch generates images of better quality and clearer class alignment. This is consistent with the proposed LDR framework as the LDR-Multi objective explicitly encourages alignment of class distributions, therefore benefiting from a larger batch that better captures overall data distributions. We leave a more rigorous study of the effect of batch size for future work.
Due to the heavy computation of such large batch size, we present the following intermediate results obtained at 35,000 iterations whereas most existing methods run with magnitude larger number of iterations. Nevertheless, these intermediate results already verify the efficacy of our framework. In addition, we present the full version of the comparison with existing generative methods in Table 13. We see the SI and FID scores for LDR-Multi degraded a little after the finetuning. This is expected as learning a more refined separation and alignment of 10 classes is a more challenging task than 2 classes. This is consistently observed from experiments on other datasets too.
Visualizing feature similarity for LDR-Multi. We visualize the cosine similarity among features Z of different classes learned from the LDR-Multi objective in Figure 19. In addition, we provide the visualization of alignment between features Z and decoded features features Ẑ. These results demonstrate that not only the encoder has already learnt to discriminate between classes, Z and Ẑ also are aligned clearly within each class.
Visualizing auto-encoding property for LDR-Multi. We visualize the original image x and their decoded x̂ using the LDR model fine-tuned using LDR-Multi formulation. The results are shown in Fig 20 for the selected 10 classes in ImageNet. The LDR-Multi formulation can give good visual quality for x̂ as well as decent sample-to-sample alignment.

heading: A.8 ABLATION STUDY ON OBJECTIVE FUNCTIONS
text: To empirically validate the importance of the rate reduction (∆R) loss, we keep the closed-loop formulation but replace all rate reduction (∆R) loss terms in (11) with corresponding cross-entropy loss.
To replace the rate reduction (∆R) terms in the objective function (11) with cross-entropy, we introduce a linear mapping W ∈ Rd×k to map Z ∈ Rd×n from feature space to logits γ = Z>W . We then calculate the softmax cross-entropy function on logits γ and one hot label matrix Y . Here H(γ,Y ) = ∑n i=1 ∑k j=1 Yij log eγij∑k j=1 e γij is the formulation of softmax cross-entropy function and
Y ∈ Rn×k is one hot label matrix. Then, we can replace the first two terms of (11) (∆R ( Z )
and ∆R ( Ẑ ) ) with H(Z>W ,Y ) and H(Ẑ>W ,Y ). For the third term of (11), we extract j-th class
one hot feature γj = Z>j W , γ̂j = Ẑ > j W from Z and Ẑ, and define the distance D(γj , γ̂j) =
eγj
eγj+eγ̂j of them. For the third term of (11), we further introduce k linear layers as discriminators {Dj}kj=1 for each class. Then, we replace the third term with the GAN’s objective function as∑k j=1 E[logDj(Zj)] + E[log(1−Dj(Ẑj))] 11. Now, we have the cross-entropy version objective function (15) for closed loop framework. We denote the closed loop framework with cross-entropy as Closed-loop-CE.
11E[X] denote the expectation of X .
min η max θ,W ,D
TX(θ, η,W ,D) . = H ( Z>W ,Y ) +H ( Ẑ>W ,Y ) +
k∑ j=1 E[logDj(Zj)] + E[log(1−Dj(Ẑj))]. (15)
We run the experiments on MNIST and CIFAR10. The architectures of MNIST and CIFAR10 are given in Table 4 to Table 712.
Results on MNIST. The training hyper-parameters of LDR-Multi and Closed-loop-CE on MNIST are following Section A.1. Comparisons between LDR-Multi and Closed-loop-CE are listed in Figure 21, 22, and 23.
Figure 21(b) and 21(c) show the reconstructed images X̂ from Closed-loop-CE and LDR-Multi. Both methods can give sample-wise reconstruction results due to the transcription framework. However, comparing training images whose features are best aligned with the principal components of class ‘2’ in Figure 22, we see that the principal components of CE features do not correspond to consistent visual attributes of the images, whereas ours do.
From the heatmaps in Figure 23(a) and 23(b), we see the features learned by rate reduction possess clear orthogonal subspace structures, whereas those learned by Closed-loop-CE do not. Moreover, Figure 23(c) and 23(d) show that the learned features of LDR-Multi have higher singular values for the top principal components of each class, corresponding to a more linearized and diverse feature distribution, whereas those by Closed-loop-CE do not.
Failed Attempts on CIFAR-10 with Cross Entropy. The training hyper-parameters of Closed-loopCE on CIFAR10 follow Section A.1. We do the grid search on three hyper-parameters: learning rate {1.5× 10−2, 1.5× 10−3, 1.5× 10−4}, mini batch size (800 or 1600), and inner loop (1,2,3,4), conduct 24 experiments in total. All cases of the Closed-loop-CE fail to converge or experience model collapse on the CIFAR-10 dataset.
12In the context of this section, we use the term Decoder and Generator interchangeably; similarly for Encoder and Discriminator.

Review summaries:
The paper "Closed-Loop Data Transcription To An LDR via Minimaxing Rate Reduction" introduces a two-player minimax game between an encoder and a decoder to yield a linear discriminative representation (LDR). It achieves this by building on top of the recently proposed MCR2 rate reduction principle, and then motivates a contractive and contrastive measure to yield a minimax game. The paper shows generative empirical results on MNIST, CIFAR-10 and ImageNet, as well as discriminative classification accuracies on MNIST.