Closed-Loop Data Transcription to an LDR via Minimaxing Rate Reduction
Authors: Xili Dai
Shengbang Tong
Mingyang Li
Ziyang Wu
Kwan Ho Ryan Chan
Pengyuan Zhai
Yaodong Yu
Michael Psenka
Xiaojun Yuan
Heung-Yeung Shum
Yi Ma
Venue: ICLR 2022 Submitted (2022)
Keywords: Linear discriminative representation
Generative model

Abstract: This work proposes a new computational framework for automatically learning a closed-loop transcription between multi-class multi-dimensional data and a linear discriminative representation (LDR) that consists of multiple multi-dimensional linear subspaces. In particular, we argue that the optimal encoding and decoding mappings sought can be formulated as the equilibrium point of a two-player minimax game between the encoder and decoder. A natural  utility function for this game is the so-called rate reduction, a simple information-theoretic measure for distances between mixtures of subspace-like Gaussians in the feature space. Our formulation avoids expensive evaluating and minimizing approximated distances between arbitrary distributions in either the data space or the feature space. To a large extent, conceptually and computationally this new formulation unifies the benefits of Auto-Encoding and GAN and naturally extends them to the settings of learning a both discriminative and generative representation for complex multi-class and multi-dimensional real-world data. Our extensive experiments on many benchmark datasets demonstrate tremendous potential of this framework: under fair comparison, visual quality of the learned decoder and classification performance of the encoder is competitive and often better than existing methods based on GAN, VAE or a combination of both. 

Under review as a conference paper at ICLR 2022
CLOSED-LOOP DATA TRANSCRIPTION TO AN LDR VIA MINIMAXING RATE REDUCTION
Anonymous authors Paper under double-blind review
ABSTRACT
This work proposes new computational framework for automatically learning closed-loop transcription between multi-class multi-dimmnnsonal data and a linear discriminative representation (LDR) that consists of multiple multi-dimensional linear subspaces. In particular, we argue that the optimal encoding and decoding mappings sought can be formulated as the equilibrium point of rwo-player minimax game between the encoder and decoder. A natural utility function for this game is the so-called rate reduction, simple distances between mixtures of subspace-like like Gaussians in the feature space. Our formulation avoids expensive evaluating and minimizing approximated distances between arbitrary distributions in either the data space or the feature space. To a large extent, conceptually and computationally this new formulation unifies the benefits of Auto-Entce anto-Encoding and GAN and naturally extends them to the settings of learning a both discriminative and generative representation for complex multi- class and lintiiiiimnsiooal real-world data. Our extensive experiments many benchmark datasets demonstrate tremendous potential of this framework: under fair comparison, visual quality of the learned decoder and classification performance of the encoder is competitive and often better than existing methods based on GAN, VAE or a combination of both.
coormationnthhoreee measure for
INTRODUCTION AND RELATED WORK
One of the most fundamental tasks in machine learning is to learn and model complex distributions (or structures) of real-world data, such images or texts, from set of observed samples. By "learn and model", one typically means that we want to establish a (parameterie) mapping between the distribution of the real data, say x E R,D, and more compact random variable, say 2 E Rd:
f(:0): x xE E + E Rdd or the inverse g(:nn): 2 E Rd + E
(1)
where = has certain standard structure or distribution (e.g. normal distributions). The so-learned representation feature would be much easier use for either generative or discriminative purposes. Be aware that the support of the distribution of æ (and that of 2) can be low-dimensional hence the above map(s) may not be so well defined off the support in the space F. (or Ed).
1.1 LEARNING GENERATIVE MODELS VIA AUTO-ENCODING OR GAN
Auto-Encoding. the machine learning literature, roughly speaking, there have been two repre- sentative approaches to such a distribution-learnie approach (Kramer, 1991; Hinton & Zemel, 1993) that aims to simultaneously learn an encoding mapping from and an (inverse) decoding mapping y from back to a::
task. One is the classic "Auto Encoding" (AE)
x f(x,0) Z g(2.7)
(2)
Here we use bold capital letters to indicate a matrix of finite samples X = [x= I..,..., x²) E Rx of and their mapped features Z [z] [=!....,>*) ="] Rdxn, respectively. Typically, wishes for two properties: firstly, the (empirical) distribution of the mapped samples Z, denoted as p(z X Ø), is close to certain desired distribution p(z), say some much lower-dimensional disentangled multivariate Gaussian: and, secondly, the decoded samples X are "similar" or close to the original X.
In the feature space to enforce the learned distribution p(z) to be close to target p(z), one normally minimizes a certain "distance" between the two distributions, say the KL-divergence: (p. min D ( p). However, it is very difficult, often computationally intractable, to precisely compute or minimize such a distance 'D between two arbitrary degenerate distributions in high-dimensional
'The classical PCA can be viewed as special case of this task. In fact, the original auto-encoding is precisely justified as nonlinaarr PCA (Kramer, 1991).

---

Under review as a conference paper at ICLR 2022
spaces. So in practice, one typically chooses to minimize instead certain approximate bounds (surrogates) of the distance, such as a variational bound in variational auto-encoding (VAE) (Kingma & Welling, 2013; Zhao et al.. 2017) and the earth mover's 's (Wasserstein) distance. In this work, we will show that if we impose specific requirements on the (distribution of) learned feature z (e.g. being subspace-like Gaussians), natural distances can be given in closed-form.
GAN. Compared to measuring distribution distance in the (often controlled) feature space 2, a much more challenging issue is how effectively measure the distance between the decoded samples X and the original X in the data space æ. For instance, for visual data such as images, despite extensive studies in the computer vision and image processing literature (Wang et al., 2004), it remains elusive to find a good measure for similarity of real images that is both efficient to compute and effective in capturing visual quality and semantic information of the images equally well. Generative Adversarial Nets (GAN) (Goodfellow et al., 2014) offers an ingenious idea to resolve this difficulty by utilizing a powerful discriminator d, usually modeled and learned by a deep network, to discern differences between the generated samples X and the real ones X:
Z g(z.n)
d(xr0) Y 0.
Such a discriminator minimizes the Jensen- Shannon divergence, between the data X and X. But as shown in (Arjovsky et al., 2017), the JS-divergence can be highly ill-conditioned to optimize when the distributions are low-dim. So practice, one may choose to replace the JS-divergence with the earth mover's distance or other variants. The original GAN aims to directly learn a mapping g. called generator, from a standard distribution (say a low-dim Gaussian random field) to the real (visual) data distribution in high-dim space. However, distributions of real-world data can be rather sophisticated and often contain multiple classes and multiple factors in each class (Bengio et al., 2013). That makes learning the mapping g rather challenging in practice, suffering difficulties such as mode-collapse (Srivastava et al., 2017). 1).. As a result, many variants of GAN have been subsequently developed in order to improve the stability and performance in learning multiple modes and disentangling different factors in the data distribution, such as Conditional GAN (Mirza & Osindero, 2014; Sohn et al., 2015; Mathieu et al., 2016; Van den Oord et al., 2016; Wang al., 2018), InfoGAN (Chen et al., 2016; Tang et al., 2021), or Implicit Maximum Likelihood Estimation (IMLE) (Li & Malik, 2018; Li et al., 2020)
In particular, to learn a generator for multi-class data, prevalent conditional GAN literature requires label information as inputs during training (Mirza & Osindero, 2014; Odena et al., 2017; Dumoulin et al., 2016; Brock et al., 2018). Recently Wu et al. (2019h:a) has proposed to train a k-class GAN by generalizing the two-class cross entropy to (k + )--class cross entropy. In this work, we will introduce a more refined 2k-class measure for the k real and & generated classes. In addition, to avoid features for each class to collapse to singleton, we will use the so-called rate reduction measure that promotes multi-dimensian dimension in the learned features (Yu et al., 2020).
Another line of research is about how to stablize the training of GAN. SN-GAN (Miyato et al., 2018) has shown spectral normalization on the discriminator is rather effective, which we will adopt in our work. PacGAN (Lin et al., 2018) shows that the training stability can be significantly improved by packing a pair of real and fake images together for the discriminator. In this work, we show how to naturally generalize this idea to discriminating an arbitrary number of pairs of real and decoded samples. Also, Wu et al. (2019a) has shown that optimizing the latent features leads to state of the art visual quality, There are strong reasons believe that their method essentially utilizes the Compressed Sensing principle to implicitly exploit low- dimensionalyy of the feature distribution. Our framework will explicitly impose and exploits such dimenninnlo structures of the learned feature distribution.
Combination of AE and GAN. Although AE (VAE) and GAN have started with somewhat different motivations, they have evolved into popular and effective frameworks for learning and modeling complex distributions of many real-world data such as images. Many recent efforts tend to combine both Auto-Encoding and GAN to generate more powerful generative frameworks for more diverse data sets, such as Larsen et al. (2015); Rosca et al. (2017); Srivastava et al. (2017); Bao et al. (2017); Huang et al. (2018); Ulyanov et al. (2018); Vahdat & Kautz (2020). As we will see, in our framework, AE and GAN can be naturally interpreted as two segments of closed-loop data transcription process. But unlike GAN or VAE, the distribution of the feature z is learned from the data x and its low-dim support in Rd is explicitly modeled union of discriminative subspaces.
2

---

Under review as a conference paper at ICLR 2022
1.2 LEARNING LINEAR DISCRIMINATIVE REPRESENTATION VIA RATE REDUCTION
Recently, Chan et al. (2021) has proposed new objective for deep learning that aims to learn linear discriminative representation (LDR) for multi-class data. The basic idea is to map distributions of real data, potentially on multiple nonlinear submanifolds, to family of canonical models consisting of multiple incoherent (or orthogonal) linear subspaces. To some extent, this generalizes nonlinear PCA (Kramer. 1991) to the more general/realistic settings where we simultancously apply multiple nonlinear PCAs to data on multiple nonlinear submanifolds. Unlike conventional discriminative methods that only aim to predict class labels, the LDR aims to learn the likely multi- multi-dimeensonnl distribution of the data hence is potentially suitable for both discriminative and generative purposes. It has been shown that this can be achieved via maximizing the so-called "rate reduction" objective.
MCR². More precisely, consider a set of data samples X = [ x"] E IR Dxn from & different classes and we use IT', j .,...: k denote the memberships of the samples in A classes X L One seeks a continuous mapping, f((,,)) 0) æ + z from X to an optimal representation ="1
Z [x] :...;""' C R that maximizes the following coding rate reduction objective, known as the MCR² principle (Yu et al., 2020):
max AR(Z, = log det 1T+a22
Yi log det (T (T+ ZII 2
(4)
R(Z,e)
R2 II)
tr((1)) for j = 1....,k k. In this paper, for simplicity we denote AR(Z Z, e) as AR(Z Z) assuming II, are known and fixed. The first term R( Z(Z] (e) is the coding rate of the whole feature set Z (coded as a Gaussian source) with a prescribed precision e; the second is the average coding rate of the l subsets of features Z, = f(X,) )) (each coded as a Gaussian). As it has been shown by Yu et al. (2020), maximizing the difference between the two terms will "expand" the whole feature set while "compressing and linearizing" features of each of the k classes. If the mapping ʃ maximizes the rate reduction, it maps the features of different classes into independent (orthogonal) subspaces in Rd Figure illustrates a simple example of data with & = 2 classes (on two submanifolds) mapped to two incoherent subspaces (solid black lines).
where a nez 03 ) 7/
d
term (Z II,
2 DATA TRANSCRIPTION VIA RATE REDUCTION
2.1 LDR TRANSCRIPTION
One issue with this one-sided LDR learning is that maximizing the above objective tends to maximize the dimension of the learned subspace for features in each class.? 2 To verify whether the learned features are good,
+ from the representation Z f(x,)) 0) back the data space æ: X = =g(Z, 17), and check how close X and X or how close their features Z and Z are. The overall pipeline can be illustrated by the following diagram: f(
may consider learning decoder g(:,n)
Z ((x.)) X [(2.0) Z.
X
(5)
where the overall model has parameters: θ = [0, n). Notice that in the above process, the segment from X to (X resembles a typical Auto-Encodee Encoding process although, as we will soon see, our -bsed encoder plays additional role discriminator. The segment from Z to (Ž, Z) draws resemblance to the typical GAN process although, in our context, the distribution of the latent variable will be learned from the data œ. which often is a distribution with multiple modes and each mode with multiple factors.
Here, in the specific context of rate reduction, we name this special auto-encoding process LDR Transcription since the maximal rate reduction principle explicitly transcribes the data X, via J. to features Z on a linear discriminative representation (LDR), which can be subsequently decoded back to the data space X, via g. Hence, the encoding and decoding maps f and g together form a closed-loop process, as illustrated in Figure 1. We wish this closed-loop transcription process to have the following good properties:
Injectivity: the generated a- = g( 0), 1) E X to be as close to (ideally the same as) the original data x E X, in terms of certain measure of similarity or distance.
'If the dimension of the feature space d is too high, maximizing the rate reduction may over-estimate the dimension of each class. Hence to learn good representation, one needs to pre-select proper dimension for the feature space, done in the experiments in (Yu et al., 2020).
3

---

Under review as a conference paper at ICLR 2022
æ
2
f(z, 0)
g 7))
RD
R
9
Figure 1: Closed-loop LDR Transcription. The encoder has dual roles: it learns an LDR for the data æ and also discerns any discrepancy in the data a and the decoded á.
Surjectivity: for all mapped images f(()) E Z of the training data E X, there decoded samples f(g(z. n), 0) 2 close to (ideally the same as)
2.2 MEASURING DISTANCES IN THE FEATURE SPACE AND DATA SPACE
Contractive Measure. For the second item in the above wishlist, as the representations in the feature space z are by design linear subspaces or (degenerate) Gaussians, we have geometrically or statistically meaningful metrics for both samples and distributions in the feature space z. For example, we care about distance between distributions between the features of the original data Z and the transcribed Z. Since the features of each class, Z, and 2j. are subspace/ Jubspace/Gaussian-liiee their "distance" can be measured by the rate reduction, with (4) restricted to two sets of equal size);
AR(2) 2, 2,) R(Z; 2,) - (R(Z, + R(2,)).
which, as shown in Fig. 1, measures the space volume between pair of black lines and blue lines, as per interpretation of rate reduction in (Yu al., 2020). Then for the "distance" of all. say k. classes, we simply sum the rate reduction for all pairs:
k
d(z, 2
min Σ AR Zz. 2,) = min Σ f(g(Z,
1=1
where Z, f(,, and Z, f(), Obviously, main goal of the learned decoder g(- to minimize the distance between these distributions. Notice that if the encoder f preserves (i.e. injective for) the intrinsic structures of the original data X,3 this criterion essentially aims to ensure there will be some decoded sample & close to every data sample x hence the decoder should be "surjective." According to the ideas of IMLE (Li & Malik, 2018), such a requirement could effectively help avoid mode-collapsing or mode-dropping.
For the first item our wish-list, however, we normally do not have a natural metric or "distance" for similarity of samples or distributions in the original data space a. To alleviate this difficulty, we can measure the similarity or difference between X and X through their mapped images Ž and Z in the feature space (again assuming f is structure- -rrettrrrpprsrrnngg)) If we interested in discerning any difference in the distributions of the original and transcribed samples, we may view the MCR² feature encoder f(: 0) as "discriminator" magnify any difference between all pairs of X, and X,, by simply maximizing, instead of minimizing, the same quantity in (7):
Contrastive Measure.
d(x, X) - mAX Σ AR 2,) max Σ 1),
(8)
j=1
That is, "distance" between X and X can be measured as the maximally achievable rate reduction between all pairs of classes in these two sets. In a way, this measures how well or bad the decoded X aligns with the original data X hence measuring the goodness "injectivity". Notice that such a discriminative measure is consistent with the idea of GAN (Goodfellow et al., 2014). Nevertheless, here the -CRe-based discriminator naturally generalizes to cases when the data distributions are multi-class and multi-modal. and the discriminativenes, is measured with more refined measure- the rate reduction, instead of the typical two-class loss (e.g. cross entropy) used in GANs.
3 Thii typically the case for MCR² -based feature representation (Yu et al., 2020).
As mentioned before, finding proper metrics or distance functions natural images has always been an clusive and challenging task (Wang al., 2004).
4

---

Under review as a conference paper at ICLR 2022
*ER
One may wonder the reason why need the mapping f(,, 8) to function as a discrim- inator between X and X by maximizing f( f .)) Figure 2 gives simple illustration: there might be many decoders g such that f Jog is an identity mapping I f=)()) = for all z the subspace Sz in the feature space. is not necessarily an auto-encoding map for æ in the original distribution Se (here for simplicity drawn o f(Sw) z Se (let alone g o f(z) = z). One should expect, without careful control of the image of g. this would typically be the case especially when the support of the distribution of æ is extremely don-dimensional in the original high-dimensions data space (e.g. the millions for images).
S² -Img(Se
f(x)
-(*)
R
max; AR(f(XX ())
82
Img (9s)
Null space of f
However,
go f Id, but fog g Id
subspace). That is,
Figure 2: S.. (red) is the submanifold for the orig- inal data æ; S² (blue) is the image S₂ under the mapping f. representing the learned feature 2; and the green curve is the image of the fcature z under the decoding mapping g.
2.3 ENCODING AND DECODING AS TWO-PLAYER MINIMAX GAME
Comparing (7) and (8), we the roles of the encoder f(;,0) and the decoder g(:, naturally as a "a two-player game": while the encoder I tries to magnify the difference between the original data and the transcribed; the decoder g aims to minimize the difference. Now for convenience, let us define the "closed-loop encoding" function:
h(r, 0,,)) (a((((r))))
x
Ideally, we want this function to be very close to f(x, 0). With this notation, combining (7) and (8), closed-loop notion of "distance" between X and X can be computed an equilibrium point the following Min-Max program for the same utility in terms of rate reduction:
D(X X) = min max O),
(10)
Notice that this only measures the difference between (features of) the original data and its transcribed version. It does not measure how good the representation Z (or 2) is for the multiple classes within X (or X). To this end, we may combine the above distance with the original MCR2-type objectives (4): namely, the rate reduction AR( Z) and AR( 2) for the learned LDR Z for X and Ž for the decoded X. The overall multi-class Min-Max program for learning the LDR model is:
min max A + AR( 0,))
AR( 0), (,,,((X,,nn)) = Tx(0,)) n).
Expansive encode
Compressive deccde
Contrustive encode & Contractive decode
(11)
Notice that, without the terms associated with the generative part or with all such terms fixed constant, the above objective is precisely the original MCR² objective." 5 Also, notice that the minimax objective function depends only on the data X hence one can learn the encoder and decoder (parameters) without the need of sampling any other distribution (as needed by GANs)!
As a special case, if X only has one class, the above Min-Max program reduces to a special "binary" form between X and X.7
Binary: min max (f(((X (()
(12)
0 Sometimes, even when contains multiple classes/modes, one could still view all classes together as one class. Then the above binary objective is to align the union distribution of all classes with their decoded X. This is typically a simpler task to achieve than the multi-class one (11) since it does not require to learn a more refined multi-class LDR for the data, as we will later see in experiments.
In an unsupervised setting, we view each sample (and its augmentations) as own class, the above formulation remains exactly the same! The number of classes is simply the number of independent samples.
as the first two terms become zero.
'Notcee that this binary case resembles formulation of the original GAN (3) by viewing X and X as two classes (0. 1). Nevertheless, instead of using cross entropy, our formulation adopts more refined rate reduction measure, which has been shown promote diversity in the learned representation (Yu et al., 2020).
5

---

Under review as a conference paper at ICLR 2022
2.4 DISCUSSIONS ABOUT THE MINIMAX OBJECTIVE
There are many outstanding questions that one may wonder about the above formulation: When does the game ming max, Tx (6, 1) have well-defined Nash-equilibrium point ,²") What good characteristics the associated optimal encoder I* and g" have? For instance, under what conditions, will their composition be or close to a true auto-encoding g" o o x Id, and will the learned features 2² f(X, (*) form orthogonal or highly incoherent subspaces?
In this paper, we will not be able to give complete rigorous answers to these questions. The main purpose of this paper is to provide compelling empirical evidence that answers to the above questions are highly likely to be positive under fair broad conditions (see Section 3). Here we will give some preliminary justification based characteristics of each team in Tx (6. n) and will also give reasons why a rigorous characterization of the optimality conditions is mathematically challenging.
Let us first consider maximizing the first two rate reduction terms in Tx (0. 17), assuming 7) fixed and (0) and (0) are aligned, i.e., AR( , (0), (0) reaches its minimum for E [K] (to be discussed below). Then they essentially maximize the rate reduction AR(Z + AR( (0)) for features of the k classes in (0) f(x; 0) and (0) hi (X, 0, n1). Based on the results in Yu et al. (2020), a sufficient condition for the rate reduction to reach a (local) maximum is when features of all classes belong to subspaces orthogonal to each other": Z Zj- = 0 and 2, 2j = 0 for i + j.
Now let us consider minimizing the k rate reduction terms in the third term of Tx (0, 1). assuming 0 fixed and the AR( 2) achieves its maximum (as mentioned above). Then, from the property of the log det (.) function, we know that each of the l terms in the contractive measure: according to Lemma 10 of Chan et al. (2021), A Zj. h 2 (n)) reaches its minimum (0) when the covariance matrices of Z, and 2, are the same, i.e., Z 2, If one further assumes that the encoder is sufficiently injective, perfect alignment of Z, and Z, suggests perfect alignment of X, and X,.
Based on the above properties, one may show that when the lt classes in each of Z and Z form k orthogonal subspace and each corresponding pair of Z, and 2, aligns perfectly, then one reaches a critical point" of the minimax game ming max, Tx (0, 17). It remains open whether such a critical point is strict saddle point- a (local) Nash equilibrium.
There are more subtle questions that one could or should ask too, such as: how are the learned features Z and Z distributed within the subspaces? Are features for individual sample z; &, (hence æ; and 2;)) well aligned? Results in Yu et al. (2020) suggest that covariance of the features in each subspace needs to be nearly isotropic in order for a rate reduction term to be maximized. When minimizing a rate reduction term, results about the so-called Brascamp-Lieb inequalities (Jonathan Bennett et al., 2007) suggest that a necessary condition for the minimum to be reached is when the distribution (density) within each subspace becomes Gaussian. Hence from these results one may conjecture that the optimal distribution in each subspace for the minimax game is Gaussian too. Although theoretically nothing is known about relationships between the features 2; E, of individual samples, (multi-class) experiments in next section suggest that they are actually very close.
Despite all these challenging open mathematical questions, one may notice that the aggregated effect of all the terms associated with the decoder g(- -7) is precisely "opposite" to the effect of the terms associated with the encoder [(, 0). All together, the Min-Max program (11), or (12), aims to strike a good tradeoff between maximizing the expressiveness and discriminativeness of the learned representation as well as minimizing any unnecessary cost, in terms of the overall coding rate and errors in decoding, all measured in rate reduction (assuming features being subspace Gaussians).
3 EMPIRICAL VERIFICATION ON REAL- WORLD IMAGERY DATASETS
The experiment section serves three purposes: First, we empirically justify the new formulation by demonstrating good properties of the learned encoder, decoder, and representations. Second, we compare our method with several representative methods from the GAN family and VAE family. Finally, we evaluate the learned LDR through visualization and classification tasks.
'Here one must assume such mapping f exists which puts conditions on the original data distribution X and the family of functions that f(- 0) can represent.
That is, gradients of the utility T are zero w.r.t. 0 and r7 hecause PAR all vanishes.

---

Under review as a conference paper at ICLR 2022
Itpe
6
0
5
0
7
6
hpp
6
0
0
7
6
VAS-AN
W
0 0
5
0
0
2
0
10R-fharry
6
0
0
7
(a) MNIST
(b) CIFAR-10
(c) ImageNet
Figure 3: Qualitative comparison on MNIST, CIFAR- 10 and ImageNet. First row: original X; Other rows:
reconstructed X for different methods.
Table 1:
Quantitative comparison on MNIST and CIFAR-IO. Average Inception scores (IS) (Salimans al., 2016) and FID scores (Heusel el al., 2017). 1 means higher is better. 4 means lower is better.
Method
GAN GAN (LDR -DDary
VAE-GAN LDR-Binary LDR-Multi
IS 1
2.08
1.95
2.21
2.02
2.07
MNIST
FID +
24.78
20.15
33.65
16.43
16.47
IS f
7.32
7.23
7.11
8.11
7.13
CIFAR-10
FID +
26.06
22.16
43.25
19.63
23.91
Datasets. We provide extensive qualitative and quantitative experimental results on the following datasets: MNIST (LeCun al., 1998), CIFAR-10 (Krizhevsky et al., 2009), STL-10 (Coates et al., 2011), CelebA (Liu et al., 2015), LSUN bedroom (Yu al., 2015), and ImageNet ILSVRC 2012 (Russakovsky et al., 2015). The network architectures and implementation details can be found in Appendix A.1 and corresponding appendix section for each dataset.
3.1 EMPIRICAL JUSTIFICATION OF LDR TRANSCRIPTION
To empirically validate our new framework, we conduct experiments from small low-variety dataset (MNIST), to small dataset of diverse real-world objects (CIFAR-10), to higher resolution images (CelebA, LSUN-bedroom), to large-scale image set (ImageNet). The results are evaluated both quantitatively and qualitatively. Implementation details and more results are given in the Appendix.
Comparison (IS and FID) with other formulations. First, we conduct five experiments to fairly compare formulation with GAN (Radford et al., 2015) and VAB(-GAN) (Larsen et al., 2016) MNIST and CIFAR-10. 10. Except for the objective function, everything else is exactly the same for all methods (e.g- networks, training data, optimization method). These experiments are: 1). GAN; 2). GAN with its objective replaced by that of the LDR-Binary (12); 3). VAE-GAN ; 4). Binary LDR (12); and 5). Multi-class LDR (11). Some visual comparison is given in Fig. 3. IS (Salimans et al., 2016) and FID (Heusel et al., 2017) scores are summarized in Table 1.
As we see from the above Table I, replacing cross-entropy with the Equa- tion (12) can improve the generative quality. The two LDR formulations are clearly on par with the others terms of IS and significantly better in FID. Finally, with the same training datasets, quality of LDR-Multi is lower than LDR-Binnry. This is expected as the multi-class task more challenging. Nevertheless, we will see soon, images decoded by LDR-Multi align much better with their classes than Binary.
Visualizing correlation features Z and decoded features 2. We visual- ize the cosine similarity between Z and Ž learned from the multi-class objec- tive (11) on MNIST and CIFAR-10, which indicates how close & = f g(z) from z. Results in Fig 4 show that Z and Z are aligned very well within each class. The block ddck-diiggonal patterns for MNIST are sharper than those for CIFAR-10. as images in CIFAR-10 have more diverse visual appearances.
Figure 4: Visualizing the alignment hetween Z and Z: Iz 2 and in
Visualizing auto-encoding the data X and the decoded X. We com- pare some representative X and X on MNIST, CIFAR-10 and ImageNet (10 classes) to verify how close a o f(x ) is to æ. The results are shown in Fig 5, and visualizations are created from training samples. Visually, the auto-encoded & faithfully captures the visual features from its respective training sample æ. There also exist some auto-encoded images that are almost identical to the original. We refer the reader to Appendix A.2 A.4 A.7 for
feature space for MNIST (top) and CIFAR-10 (bottom).
7

---

Under review as a conference paper at ICLR 2022
504 /
5 0
1
3 7
2
3 2
7 3
3
6
9
3
3
0
+
(a) MNIST X
(b) MNIST X (c) CIFAR-10 10 X (d) CIFAR-IO X (e) ImageNet X (f) ImageNet X
Figure 5: Visualizing the auto-encoding property of the learned LDR (a x ē = g f(z)) on MNIST,
C0,AR-10, and ImageNet (zoom in for better visualization).
Table 2:
Comparison on CIFAR-I0 10 and STL-10. Comparison with more existing methods and on ImageNet can be found in Table 13 in the Appendix.
Method
GAN based methods
VAE/GAN based methods
SNGAN CSGAN LOGAN
VAE-GAN
NVAE DC- VAE LDR-Binary LDR-Multi
IS f
7.4
8.1
8.7
7.4
8.2
8.1
7.1
CIFAR-10 -0
FID +
29.3
19.6
17.7
39.8
50.8
17.9
19.6
23.9
IS f
9.1
8.1
8.4
7.7
STL-10
FID 1
40.1
41.9
38.6
45.7
more visualizations of results these datasets, including similar results transformed MNIST digits and real-life images decoded by randomly sampling the learned feature subspaces.
3.2 COMPARISON EXISTING GENERATIVE METHODS
Table 2 gives a quantitative comparison visual quality of our method with others on CIFAR- 10, STL-10, and ImageNet. In general, there is large difference in terms of FID and IS scores between the GAN family and the VAE family of models. SNGAN (Miyato et al., 2018) are commonly used methods in most generative applications while LOGAN (Wu al., 2019a) is the state-of-the-an method on ImageNet in terms of FID and IS. As we see, even if the rate reduction is not specifically designed nor engineered for visual quality 10, our method is still rather competitive in terms of these metrics.
3.3 BENEFITS OF THE LEARNED LDR TRANSCRIPTION MODEI.
As we have argued before, the learned LDR transcription model (including the feature z, the encoder f, and the decoder g) can be used for both generative and discriminative purposes.
Decoding samples from the feature distribution. Using the CIFAR-10 and CelebA datatsets, we visualize images decoded from samples of learned feature subspace. For the CIFAR-10 dataset, for each class j. we first compute the top-4 principal components of the learned features Z, (via SVD). For each class j, we then compute [<z], 1/)1. the cosine similarity between the L-th principal direction v` and feature sample z). After finding the top-5 zj according to [z],, v for each class j. reconstruct images é = g(2)). Each row Fig. is for one principal component. We observe that images in the same row share many common features; images in different rows differ significantly characteristics like shape, background, and style. See Appendix A.4 for more visualization of principal components learned for all 10 classes of CIFAR-10.
For the CelebA dataset, we calculate the principal components of all learned features in the latent space. Fig 7(a) shows some decoded images along these principal directions. These components seem to clearly disentangle visual attributes/factors such as wearing hat, changing hair color and wearing glasses. More examples be found in Appendix A.6. The results consistent with the property of MCR² that promotes diversity of the learned features. Fig. 7(b) shows interpolating features between pairs of training image samples, where for two training images 2, and Ø2, reconstruct based on their interpolated feature representations by & = g(aa (1 ox) (x) (x2)), )), a E [0, 1]. The decoded images show continuous morphing from sample to another in terms of visual characteristics, as opposed to merely a superposition of the two images.
In current implementation, the original objective is used without any other heuristics regularization.
8

---

Under review as a conference paper at ICLR 2022
(a) Horse
(b) Ship
Figure 6: CIFAR-10 dataset. Visualization top-5 reconstructed (()) based on the closest distance of
x to each row (top-4) principal components of data representations for class 7- t-'Horse' and class 8- 'Ship".
Hat
Hair Color
Glasses
(a) Disentangled attributes principal components
(b) Interpolation between distinct samples
Figure 7: CelebA dataset. (a): Sampling along three principal components that seem correspond to different
visual attributes; (h): Samples decoded by interpolating along the line between features two distinct samples.
Encoded features for classification. Notice that not only is the learned decoder good for generative purpose, but also the encoder is good for discriminative tasks. In this experiment, we evaluate the discriminativeness of the learned LDR model by testing how well the encoded features help classify the images. We use features of the training images to compute the learned subspaces for all classes, then classify features of the test images based on a simple nearest subspace classifier. Results in Table 3 show that our model gives competitive classification accuracy on MNIST, compared to some of best VAE-based methods. We also tested the classification on CIFAR- 10, the accuracy is currently about 80.7%. As expected, the representation learned with the multi-class objective is very discriminative and good for classification tasks. This demonstrates the learned LDR model is not only generative but also discriminative.
Table 3: Classification accuracy MNIST, comparing classifier based VAE methods (Parmar al., 2021).
Most those VAE-based methods require auxiliary classifiers to boost classification performance.
Method
VAE
Factor VAE
Guide VAE
DC- VAE
-DR-Binary LDR-Mule
MNIST
97.12%
93.65%
98.51%
98.71%
89.12%
98.30%
3.4 CONCLUSION
This work provides a novel formulation for learning both generative and discriminative represen- tation for multi-class cilti-dimensionall real-world data. We have provided compelling empirical evidence that the distribution of most data can be effectively mapped to an LDR, union of inco- herent subspaces. The main purpose of this paper is to demonstrate the conceptual simplicity and practical potential of this new representation learning framework, instead of to strive for state of the art performance. Nevertheless, with our preliminary implementation. LDR can be effectively learned for variety of real-world datasets, from small large, from domain-specific In addition, the so-learned decoder g already enjoys the benefit of GAN for its good generative visual quality and the encoder f with the benefit of AE for its discriminative property. From our experience, the rate reduction based objective can be stably optimized across a wide range of datasets and network architectures without any additional regularizations or engineering tricks. One may notice that there are many ways our method can be significantly improved. For one, in this work, we have simply adopted networks that were designed for GAN or AE, but they may not be optimal for the rate reduction type objectives. Also notice that compared to GAN and AE, our method leads to an explicit model for the feature distribution: a mixture of incoherent subspace Gaussians. Such an explicit model has the potential of making many subsequent tasks much easier and better: importance sampling for decoding, classification, or even incremental learning. We leave all these directions, together with all the open mathematical problems, for future investigation.
diverse.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
All authors agree and will adhere to the conference's Code of Ethics. We do not anticipate any potential ethics issues regarding the research conducted in this work.
REPRODUCIBILITY STATEMENT
Settings and implementation details of network architectures, optimization methods, and some common hyper-parann parempparaeeters are described in the Appendix A. More detailed experimental settings for different datasets can be found in their corresponding Appendix section entitled with the name of the dataset. We will also make our source code available upon request by the reviewers or the area chairs.
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 2017.
Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, and Gang Hua. Cvae-gan: fine-grained image generation through asymmetric training. In Proceedings of the IEEE international conference computer vision, pp. 2745-2754, 2017.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8):1798-1822 2013.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv:1809. 2018.
Kwan Ho Ryan Chan, Yaodong Yu, Chong You, Haozhi Qi, John Wright, and Yi Ma. Redunet: A white-bot box deep network from the principle of maximizing rate reduction. CoRR, 110.2100511444 2021. URL https: / arxiv. org. abs /2105.1151000 10446.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2172-2180, 2016.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp.
JMLR Workshop and Conference Proceedings, 2011.
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. arXiv preprint 1r1:02661..07629, 2016.
Ian Goodfellow, Jean Abudet-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems, pp. 2672-2680, 2014.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637, 2017.
Geoffrey E. Hinton and Richard S. Zemel. Autoencoders, minimum description length and helmholtz free energy. In Proceedings of the 6th International Conference Neural Information Processing Systems, NIPS 93, pp. 3-10, San Francisco, CA, USA, 1993. Morgan Kaufmann Publishers Inc.
Huaibo Huang, Ran He, Zhenan Sun, Tieniu Tan, et al. Introvae: Introspective variational autoen- coders for photographic image synthesis. Advances in Neural Information Processing Systems, 31, 2018.
Jonathan Jonathan Bennett, Anthony Carbery, Michael Christ, and Terence Tao. The brascamp- lieb inequalities: Finiteness, structure and extremals. Geometric and Functional Analysis, 17, 2007.
10

---

Under review as a conference paper at ICLR 2022
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.61 14, 2013.
Mark A. Kramer. Nonlinear principal component analysis using autoassociative neural networks. Aiche Journal, 37:233 -7,233-243,, 1991.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Anders Boesen Lindbo Larsen, Søren Kaac Sønderby, Hugo Larochelle, and Ole Winther. Au- toencoding beyond pixels using learned similarity metric. arXiv preprint arXiv:1512.0995 2015.
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoen- coding beyond pixels using a learned similarity metric. In International conference machine learning, pp. 1558-1566. PMLR, 2016.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
Ke Li and Jitendra Malik. Implicit maximum likelihood estimation, 2018.
Ke Li, Shichong Peng, Tianhao Zhang, and Jitendra Malik. Multimodal image synthesis with conditional implicit maximum likelihood estimation. International Journal of Computer Vision, 2020.
Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. Pacgan: The power of two samples in generative adversarial networks. In Advances in neural information processing systems, pp. 1498-1507, 2018.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Michael F Mathicu, Junbo Jake Zhao, Junbo Zhao, Aditya Ramesh, Pablo Sprechmann, and Yann LeCun. Disentangling factors of variation in deep representation using adversarial training. In Advances in Neural Information Processing Systems, pp. (040-5048, 2016.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1785 2014.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.
Spectral normalization for generative adversarial networks. International Conference on Learning Representations, 2018.
URL htwp::ttp::.... / /ww.eeeveeeeeeeeeeeeeeeeee net. fentt/foouut forum? d= QRgziT-
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier GANs. In International conference machine learning, pp.
2017.
Gaurav Parmar, Dacheng Li, Kwonjoon Lee, and Zhuowen Tu. Dual contradistinctive generative autoencoder. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 823-833 2021.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, arXiv preprint
2015.
Mihaela Rosca, Balaji Lakshminarayanan, David Warde-Farley, and Shakir Mohamed. Variational approaches for auto-encoding generative adversarial networks. arXiv preprint arXiv: 04987, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal computer vision, 15(3):211 1-252, 2015.
11

---

Under review as a conference paper at ICLR 2022
Tim Salimans, lan Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In Advances in neural information processing systems, pp. 2016.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Advances in neural information processing systems, pp. 3483-3491, 2015.
Akash Srivastava, Lazar Valkoz, Chris Russell, Michael U Gutmann, and Charles Sutton. VeeGAN: Reducing mode collapse in GANs using implicit variational learning. In Advances in Neural Information Processing Systems, pp. 3310-3320, 2017.
Shichang Tang, Xu Zhou, Xuming He, and Yi Ma. Disentangled representation learning for control- lable image synthesis: An cnoommtiionthhorettee perspective. In 2020 25th International Conference on Puttern Recognition (ICPR), pp. 10042-10049, 2021. doi: 10. 109/1CPR48806 .941 1925.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. It takes (only) two: Adversarial generator- encoder networks. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Arash Vahdat and Jan Kautz. Nvae: deep hierarchical variational autoencoder. arXiv preprint
arXiv:2007.03888, 2020.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with PixelCNN decoders. In Advances in neural information processing systems, pp. 4790-4798, 2016.
Ting- Chun Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional GANs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7798-8807, 2018
Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions Image Processing, 13(4):600 2004. doi: 10.1100/TPP.2033.0 819861.
Yan Wu, Jeff Donahue, David Balduzzi, Karen Simonyan, and Timothy Lillicrap. Logan: Latent optimisation for generative adversarial networks. arXiv preprint arXiv:1912.00953, 2019a.
Yan Wu, Mihaela Rosca, and Timothy Lillicrap. Deep compressed sensing. In International Conference on Machine Learning, pp. 1860- PMLR, 2019b.
Fisher Yu, Ari Seff, Yinda Zhang. Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arxiv: /55. 003600336, 2015.
Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. In Advances in neural information processing systems, 2020.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. InfoVAE: Information maximizing variational autoencoders. arXiv preprini arXiv: 1706.0226, 2017.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.I EXPERIMENT SETTINGS AND IMPLEMENTATION DETAILS
Network backbones. For MNIST, we use the standard CNN models in Table 4 and Table 5, following the DCGAN architecture (Radford et al., 2015). We resize the MNIST image resolution from 28 x 28 to 32 x 32 fit DCGAN architecture. All IReLU (IReLU is short for Leaky-ReLU) of the encoder are set to 0.2.
We adopted conv ResNet architectures for CIFAR-10 in Tables and 7, and STL-10 shown in Tables and 9. Each ResBlock up is same as Resnet, but add an up-sampler after the first conv layer. All batch normalization layers of ResBlock in encoder are replaced with spectral normalization layer.
Finally, we use the same architecture for CelebA, LSUN-bedroom, ImageNet- 128 (see Tables 10 and 11) as all three datasets have the same 128 x 128 resolution. Again, each ResBlock up is same as Resnet, but add an up-sampler after the first conv layer. And all batch normalization layers in encoder are replaced with spectral normalization layer. All experiments utilize this lightweight Py PoTrhh library that provides implementations of popular tate-offtheeart GANs and evaluation metries.
Table 4: Decoder for MNIST.
Table 5: Encoder for MNIST.
x E PIXIXT2
Gray image æ E R 2Xx32X
4 4, stride=1, pad=0 deconv. BN 256 ReLU 4 x 4, stride=2, pad=1 deconv. BN 128 ReLU 4 x 4, stride=2, pad=1 deconv. BN 64 ReLU
x 4, stride=2, pad=1 conv 64 IReLU
4 x 4, stride=2, pad= 1 conv 128 IReLU
x 4, stride=2, pad=1 conv 256 IReLU
4 4, stride=2, pad=1 deconv. Tanh
4 x 4, stride=1, pad=0 conv 128
Table 6: Decoder for CIFAR-10.
Table 7: Encoder for CIFAR-10.
2 E R2288
RGB image x E 322x3233
dense + 4 x 4 x 256
ResBlock down 128
ResBlock up 256
ResBlock down 128
RosBlock up 256
ResBlock 128
ResBlock up 256
ResBlock 128
BN, ReLU, 3 x 3 conv, 3 Tanh
ReLU
Global sum pooling
dense - 128
Table 8: Decoder for STL-10.
Table 9: Encoder for STL-10.
R R228
RGB image E 15x11823
dense + 6 x 6 x 512
ResBlock down 64
ResBlock up 256
ResBlock down 128
ResBlock up 128
ResBlock down 256
ResBlock up 64
ResBlock down 512
BN, ReLU, x 3 cony, 3 Tanh
ResBlock 1024
ReLU
Global sum pooling
dense - 128
Training details. Across all of our experiments, we use Adam (Kingma & Ba, 2014) optimizer
for all experiments, with byperparameters 31 = 0, 32 = 0.9. The initial value of learning rate is 0,00015 and is scheduled with linear decay. We choose c² = 0.5 for both equation 11 and 12 in all LDR experiments. For all LDR- Multi experiments on ImageNet, we only choose 10 classes. The details of the 10 classes as shown in Table 12. Most experiments trained RTX 3090ti GPUs.
13

---

Under review as a conference paper at ICLR 2022
Table 10: Decoder for CelebA-128,
Table 11: Encoder for CclebA- 128, ASUN-bedroom-128, and ImageNet- 128.
BUUN-bedroom-1222 128, and ImageNet-
128.
R R22
RGB image E R 1225x12823
dense 4 x 4 x 1024
ResBlock down 64
ResBlock up 1024
ResBlock down 128
ResBlock up 512
ResBlock down 256
ResBlock up 256
ResBlock down 512
ResBlock up 128
ResBlock down 1024
ResBlock up 64
ResBlock 1024
BN, ReLU, 3 x 3 conv, 3 Tanh
ReLU
Global sum pooling dense 128
Table 12: ID and correspond category for 10 classes of ImageNet
ID
Category
n02930766
cab, hack, taxi, taxicab wok
n04596742
n02974003
car wheel
n01491361
tiger shark, Galeocerdo cuvieri
n0151 48859
hen
n09472597
volcano lemon
n07749582
n09428293
seashore, coast, seacoast, sea-coast
m02504458
African elephant, Loxodonta africana
n04285008
sports car, sport
A.2 MNIST
Settings. On MNIST dataset, train model using DCGAN (Radford et al., 2015) architecture with our proposed models LDR-Multi (11) and LDR-Binary (12). We set the learning rate to 10-1 batch size 2048, and training 15,000 iterations. Due the advantages of the LDR objective, can achieve between-clate discriminative representations while the within-class diversity of these representations can be preserved, which are shown in the following experimental results.
More results illustrating auto-encoding. Here we give more reconstruction results, or X, of our LDR-Multi and LDR- LDR-Binan Binary models, compared to their corresponding original input X. As shown in the Fig.8, for the LDR- Binary model, it can generate clean digit-like images but the decoded X LDR-Binary might resemble digits from similar but different classes to the input data X since LDR-Binary tends to only align the distribution of all digits.
In contrast, with the MDR-Multi objective, the decoded X not only coherent with the correct class with the input data X, but also show very clear one-to-one mapping between individual sample æ and although the objective 11 does not enforce that! Comparing with the results from the VAE-GAN (Larsen et al., 2015), our decoded images preserve much better the individual characteristics of the original samples.
Images decoded from random samples on the learned multi-class LDR. Since our LDR-Multi objective function maps input data of each class into different (orthogonal) subspace in the feature space, we can generate images conditioned on each class by random sampling z in the subspace of each class and then decode them back the input space as ă.
To do random sampling in the learned subspace, we first calculate the mean feature 2, and the singular vectors v of the SVD (or principal components) of the learned features Z, of the class j of the training data, where index represents the ith principal components. We only use top principal
14

---

Under review as a conference paper at ICLR 2022
4
/
2
5
0
4
/
3
1
3
3
4
9
6
8
9
2
3
1
4
3
5
3
6
7
9
3
3
6
1
/
4
8
5
3
0
7
4
3
3
6
7
2
8
9
M
0 7
9
/
2
8
1
4
0
9
/
6
&
0
0
2
8
6
4
0
9
1
2
3
2
2
7
3
3 5
1
3
332
1
2
6
9
0
5
6
0
6
0
6
0
9
0
0
5
2
6
9
0
5
7
1
7
9
9
8
5
8
7
7
4
8
5
7
7
9
15
8
7
3
9
3
3
3
0
9
8
3
0
4
8
/
8
0
7
9
2
3 3
7
9
O
9
4
/
4
6
0
O
9
4
/
9
6
0
3 9
4
4
A
0 0
O
9
4
4
8
0
(a) Original X
(b) VAE/GAN X
(c) LDR-Binary X
(d) LDR-Mulli X
Figure The comparison of the reconstruction results of different methods with the input data.
components of each class on MNIST dataset. These statistics of the subspace can be used for guiding the random sampling. Then we sample randomly along the principal components and around the mean feature as
2 Zrandom = 2j +
n4 * Vj.
(13)
Xm
where žj the mean feature of class j. J. and vj are the i-th singular value and principal component of class j. n, are i.i.d. Gaussian (o,, 1) random variables. That is, the feature each subspace/class modeled by r-dimensional multivariate Gaussian, with variances of which characterize variances of the training data in the feature space. Here, a is a nyper-parameter that controls the sampling range. As for visualization of random generated images g ((Zrandom.)) conditioned on the given class, we compare our method with some other conditional generation method such as ACGAN (Odena al., 2017) and InfoGAN (Chen et al., 2016) (For ACGAN and InfoGAN, generate images conditioned on class labels with randomly sampled latent z according the procedures mentioned in their respective works). Our model can give realistic and correct conditional generation results with high diversity in each class, while other methods may make mistakes in the generation between some similar classes such as classes and for InfoGAN.
0
0
000
0
0
0
1
/
1
/
1
1
1
/
7
1
1
1
11
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
/
3
3
3
3
3
3
3 3
5
3
3
3
5
5
3
3
3
3
3
3 4
4
4
9
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4 5
5
5
6
5
5
5
5 6
5
3
3
5
5
5
5
3
5
5
5
5
5
5
5
5
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7 8
8
8
8
8
8
8
P
2
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
9
9
9
5
9
0
9
9
9 9
9
9
9
9
9
9
8
9
9
9
9
9
9
(a) ACGAN
(b) InfoGAN
(c) LDR- Multi
Figure 9: Comparison randomly generated images conditioned on each class.
A.3 TRANSFORMED MNIST
Settings. In this experiment, verify that our LDR-Multi model can preserve diverse data modes in the leamned feature embeddings. We construct a transformed MNIST dataset with 5 modes: normal, large(1.5 x), small(0.5 x), rotate 45e left, and rotate 45" right. Each image data will be randomly transformed to one of the modes. Representative examples of such training data can be found in Figure 10(a). We train the model with learning rate le-4 and batch size 2048 for 15,000 iterations.
15

---

Under review as a conference paper at ICLR 2022
Auto-encoding results. Figure 10(b) gives the decoded results of the training data with different modes. Even though the data are now much more diverse for each class, decoder learned from the LDR-Multi objective can still achieve high sample-wise similarity to the original images.
Tem
p
2
3
50 x
& 5 9 3 8
2
1
3
6
1
3
3
6 0 7
/ 9
1 /
8
6
9
4
3
8
b
4 2 0 و
1
2 9
4 0 A
1
V 4 9
6
b
6
1
8
9 0007
0
u
9
3
4
4 44
3
1
7 4
\ 4
0
,
4
& *
0
y 6
(a) Original X
(b) Decoded X
Figure 10: Original (training) data X and their decoded version X on transformed MNIST.
Identifying different modes. Similar to the earlier experiments of Fig. for CIFAR-10 in the main paper, we find the images using the learned decoder g from features of the training images aligned the best with these components.
top principal components of features of each class Zj (via SVD) and generate new
In Fig 11, we select three classes 0, 1, and visualize samples from top r = 8 principal components for each class. Each row represents one principal components direction. As it can be seen in the figure, the decoded images along each principal component shows similar mode and the modes along different component directions identified as one of these principal directions. This clearly shows that our LDR-Multi model can keep the different modes within each class of the data X, the principal component directions of Zj. and these modes can also be retained in the decoded images X,.
rather incoherent. All major modes of the original data can be
O
O
2
2
O
O
O
2
Z
2
e
(a) Components of class "0"
(b) Components of class "I"
(c) Components class "2"
Figure 11:
The reconstructed images X from the features Z best aligned along top-8 principal components on the transformed MNIST dataset. Each row represents a different principal component.
A.4 CIFAR-10
Settings, For all experiments on CIFAR-10, we follow the common training hyper-parameee parameters in section A.1. Beyond that, for each experiment, run 450,000 iterations with mini-batch size 1600.
16

---

Under review as a conference paper at ICLR 2022
Images decoded from random samples the learned multi-class LDR. We sample in the feature space randomly along the principal components and around the mean feature of each class Zj in the MNIST case, according equation (13). The generated images from the sampled features are illustrated Fig 12, one row per class. As we see, the generator learned from the L.R-muuti objective is capable of generating diverse images for each class.
Further, for visualization of random generated images g(Zrandom.s) conditioned on the given class, also compare our method with some other conditional generation method such ACGAN (Odena et al., 2017) and InfoGAN (Chen et al., 2016). For all three experiments, we have randomly sampled images per class CIFAR- 10. For more complex dataset like CIFAR-10. our model can give more realistic conditional generation results for different classes with high diversity within each class.
(a) ACGAN
(b) InfoGAN
(c) LDR-Multi
Figure 12: Comparison of randomly generated images conditioned on each class.
Generating image along different PCA components for each class. For each class, we first compute 10 principal components (singular vectors of the SVD) of Z and then for each of the top singular vectors, we display in each row the top- 10 reconstructed image X whose Z are closest to the singular vector using method described the main body of the paper, Section 3.3. The results are given in Figure 13 below.
A.5 STL-10
Settings. For all experiments on STI.-10, we follow the common training hyper-parameters. in section A.I. For LDR-Binary setting, we train 1500,00 iterations. For I.DR-Multi setting, we initialize the weights from the 20,000-th iteration of LDR- L-DR-Binary checkpoint and train for another 80,000 iterations (with the LDR-Multi objective). The and FID scores on the STL-10 dataset are reported in Table 13, par even better than existing methods such SNGAN (Miyato et al., 2018) or DC- VAE (Parmar et al., 2021).
Visualizing auto-encoding property for LDR-Binary, We visualize the original images and their decoded using the LDR model learned from IDR-Binary objective. The results are shown in Figure 14 for STI-100
A.6 CELEB-A AND LSUN
To verify that formulation works images of higher-resolution, conduct experiments on the Celeb-A and LSUN datasets, which have a resolution of 128 x 128.
Settings. For all experiments these datasets, follow the common training in Section A.I. We choose 300 mini-batch size for Celeb-A and LSUN, Both of them are trained with the LDR-Binary formulation, and for 450,000 iterations.
17

---

Under review as a conference paper at ICLR 2022
Airplane
(b) Automobile
(c) Bird
(d) Cat
(c) Deer
Dog
(g) Frog
(h) Horse
ii Ship
Truck
Figure 13: Reconstructed images X from features Z close to the principal components learned for 10 classes CIF
18

---

Under review as a conference paper at ICLR 2022
(a) Original X
(b) Decoded X
Visualizing the original æ and corresponding decoded results on STL- dataset. Note the model is trained from LDR-Binary (12) hence sample class wise correspondence is relatively poor. But the decoded image quality is very good.
Figure 14:
(a) Hat
(b) Hair Color
(c) Glasses
Figure 15: Sampling along the 9-th, 19-th, and 23-th principal components of the learned features Z seems to manipulate the visual attributes for generated images, on the CelebA dataset.
Generating image along different PCA components. We calculate the principal components of the learned features Z in the latent subspace. We manually choose 3 principle components which are related hat, hair color, and glasses (see Fig 15). The three components are 9-th, 19-th, and 23-th respectively from the overall 128 principal components. These principal directions seem to clearly disentangle visual attributes/factors such as wearing hat, changing hair color, and wearing glasses.
Images generated from random sampling of the feature space. We sample randomly according the following Gaussian model:
Zrandom + a n4 *
(14)
i=1
where E is the mean feature, o; and v; are the ith singular value and singular vector, are i.i.d. Gaussian (0, 1) random variables. As before a is hyper-paramete to control the sampling range. We use top 1=100 principle components for random sampling. The random generated images are realistic and diverse. (see Fig 16)
Visualizing auto-encoding property for LDR-Binary. We visualize the original image and their decoded à using the LDR model learned from LDR- LDR-Binary formulation. The results are shown in Fig 17 and Fig 18 for the Celeb-A dataset and the LSUN dataset, respectively. The LDR -DnaBry formulation can give very good visual quality for & but cannot ensure sample to sample alignment. Nevertheless, the decoded & seems to be similar in style to the original æ. We believe it manages to align only the dominant principal component(s) associated with the most salient visual attributes, say pose of the face for Celeb-A or layout the room for LSUN, between features of X and X.
A.7 IMAGENET
Settings. To verify that our formulation works large-scale datasets, train model the entire ImageNet. For all experiments on the ImageNet, we follow the common training prprr-paraeeters in Section A.1.
We first train our model with LDR- LDR-Brnary (12) with a mini-batee size 1800 on the whole ImageNet SLSC 2012 dataset. The number of training iterations is 450,000.
19

---

Under review as a conference paper at ICLR 2022
Figure 16: Images decoded from randomly sampled features, as a learned Gaussian distribution (14). for the CelebA dataset.
(a) Original X
(b) Decoded X
Figure 17: Visualizing the original x and corresponding decoded å results on Celeb-A dataset. The LDR model trained from LDR-Bo LDR-Binary (12).
(a) Original X
(b) Decoded X
Figure 18: Visualizing the original æ and corresponding decoded a results on LSUN-bedroom dataset. The LDR model is trained from LDR-Binary (12).
20

---

Under review as a conference paper at ICLR 2022
Table 13: Comparison on CIFAR-10, STL-10, and ImageNet.
CIFAR-10
STL-10
ImageNet
Method
1st
FID4
ISt
FID4
IS
FID
GAN based methods
DCGAN Radford et al. (2015)
6.6
7.8
SNGAN Miyato al. (2018)
7.4
29.3
9.1
40.1
48.73
CSGAN Wu et al. (2019b)
8.1
LOGAN
(2019a)
8.7
17.7
VAE/GAN based methods
VAE Kingma & Welling (2013)
3.8
115.8
VAE/GAN Larsen al. (2016)
7.4
39.8
NVAE Vahdat & Kautz (2020)
50.8
OC-VEE Parmar et al. (2021)
8.2
8.1
41.9
LLDR-Binary (ours)
8.1
19.6
8.4
38.6
7.74
46.95
LDR-Muultt (ours)
7.1
23.9
7.7
45.7
6.44
55.51
After that, we fine-tune the Information about the classes can be found Table 12. The fine-tune mini-batch size is 1024, and we train another 35,000 iterations. This experiment took 8 A A100-SXM4 GPUs, each with 40GB of CUDA memory for 120 GPU hours. Note that our choice mini-bater bati-batch size substantially larger than those commonly adopted in other works while training et al. (2018)). We empirically observe that training with a larger mini-bated generates images of better quality and clearer class alignment. This is consistent with the proposed LDR framework as the LDR-Muu objective explicitly encourages alignment of class distributions, therefore benefiting from a larger batch that better captures overall data distributions. We leave more rigorous study of the effect of batch size for future work.
model with LDR-Multi (11), on 10 selected classes.
the ImageNet (e.g. 128 in Miyato
Due the heavy computation of such large batch size, we present the following intermediate results obtained at 35,000 iterations whereas most existing methods run with magnitude larger number of iterations. Nevertheless, these intermediate results already verify the efficacy of our framework. In addition, we present the full version of the comparison with existing generative methods in Table 13. We see the and FIID scores for LDR-Multi degraded little after the finetuning. This is expected as learning a more refined separation and alignment of 10 classes is more challenging task than 2 classes. This is consistently observed from experiments on other datasets too.
Visualizing feature similarity for LDR-Multi. We visualize the cosine similarity among features Z of different classes learned from the LDR-Multi objective Figure 19. In addition, we provide the visualization of alignment between features and decoded features features Ž. These results demonstrate that not only the encoder has already learnt to discriminate between classes, Z and 2 also aligned clearly within each class.
Visualizing auto-encoding property for LDR-Multi. We visualize the original image æ and their decoded & using the LDR model fine-tuned using LDR-Multi formulation. The results are shown in Fig 20 for the selected 10 classes in ImageNet. The LDR-Multi formulation can give good visual quality for & as well decent sample-to-sample alignment.
A.8 ABLATION STUDY ON ORJECTIVE FUNCTIONS
To empirically validate the importance of the rate reduction (AR) loss, we keep the closed-loop formulation but replace all rate reduction (AR) loss terms in (11) with corresponding cross-entropy
loss.
To replace the rate reduction (AR) terms in the objective function (11) with cross-entropy, introduce a linear mapping W E gaxk to map Z dd from feature space to logits 7 = 2 W.
We then calculate the softmax cross-entropy function logits 7 and one hot label matrix Y. Here H(y,, Y)= E Ef Yúj log "
edij is the formulation of softmax cross-entropy function and
21

---

Under review as a conference paper at ICLR 2022
1.0
2600
2600
5200
0.5
7800
7800
13000
13000
2000
5200
7800
10400
13003
(a) Z
2 2
Figure 19; Visualizing feature alignment: (a) among features IZ Z). (b) between features and decoded features I 2 These results obtained after 200,000 iterations.
(a) Original X
(b) Decoded x
Figure 20: Visualizing the original x and corresponding decoded o results on ImageNet (10 classes).
The LDR model fine-tuned using LDR-Multi (11).
Y Rank is one hot label matrix. Then, we can replace the first terms of (11) (AR(Z) and A AR(2 2)) with H(2 W, Y) and H W, Y). For the third term of (11), we extract j-th class one hot feature 7 Z W. 9, Z W from and 2, and define the distance D(7j, 7/) +5 of them. For the third term of (11), we further introduce l linear layers discriminators
e²
for each class. Then, we replace the third term with the GAN's objective function as == Eloog D, E[log(111111 - Dj (2,))) [2,) 1I, Now, we have the cross-entropy version objective function (15) for closed loop framework. We denote the closed loop framework with cross-cntrope as Closed-loop-CE.
(Dr} 1==1
Σ
"E "X[ denote the expectation of X.
22

---

Under review as a conference paper at ICLR 2022
5
0
4
/
9
2
3
5
0
4
9
2
1
3
3
0 4
/
9
2
3
1
4
3
5
3
6
7
1
4
6
3
6
1
7
/
4
3
5
3
6
7
2
8
6
9
4
0
2
8
6
9
4
0
9 3
/
2
8
6
9
4
0
/
1
2
4
3
2
3
8
1
2
4
3
2
7
8
1
2
4
3
2
3
8
6
9
0
5
6
0
7 8 9
6
6
9
0
5
6
0
7
6
6
9
0
5
6
0
7
6
1
8
1
9
3
9
5
1
8
7
9
3
9
8
5
1
8
1
9
3
8
5 8 0
9
3
3
0
7
4
8
5
3
0
7
4
9
8
3
3
3
0
7
4
9
0
9
4
/
4
4
6
0
O
9
4
4
4
6
O
O
9
4
/
4
8
6
(a) Original
(b) X by Closed-loop-CE
(c)
X by LDR-Multi
Figure 21: The comparison of sample-wise reconstruction between Close-loop-CE and LDR-Multi.
min max Tx (x W, = H W. Y) + (,, 0.W,D
W w,Y)+
Ellog (2 + B|log(1 D,(2,))
(15)
We run the experiments on MNIST and CIFARIO. The architectures of MNIST and CIFARIO are given Table 4 to Table 712.
Results MNIST. The training hyper-paranne seper-parameters of LDR-Multi and Closed-loop-CE on MNIST are following Section A.1. Comparisons between LDR-Multi and Closed-loop-CE are listed in Figure 21, 22, and 23.
Figure 21(b) and 21(c) show the reconstructed images X from Closed- Closed-loop-CE and LDR-Multi. Both methods can give sample-wise reconstruction results due to the transcription framework. However, comparing training images whose features best aligned with the principal components of class '2' in Figure 22, we see that the principal components of CE features do not correspond to consistent visual attributes of the images, whereas ours do.
From the heatmaps in Figure 23(a) and 23(b), we see the features learned by rate reduction possess clear orthogonal subspace structures, whereas those learned by Closed-loop-CE do not. Moreover, Figure 23(c) and 23(d) show that the learned features of LDR-Multi have higher singular values for the top principal components of each class, corresponding to a more linearized and diverse feature distribution, whereas those by Closed-loop-CE do not.
Failed Attempts on CIFAR-10 with Cross Entropy. The training hyper-parameters Closed-loop- CE on CIFAR10 follow Section A.1. We do the grid search three rate {1.5 x 10-2, 1.5 x 10-3 1.5 x 10-4), mini batch size (800 or 1600), and inner loop (1,2,3,4), conduct 24 experiments in total. All cases of the Closed-loop-CT fail converge experience model collapse on the CIFAR-10 dataset.
learning
In the context of this section, use the term Decoder and Generator interchangeably; similarly for Encoder and Discriminator.
23

---

Under review as a conference paper at ICLR 2022
2
2
2
2
2 2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
3
P
2
3_
2
3
2
2
2
2
2
2
2
2
2
2
3
2
7
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
3
2
2
2
(a) Closed-loop- -oop----oo-EEEEE- -E
(b) LDR- LDR-Multi
Figure 22: Training samples along different principal components of the learned features of digit '2".
2000
3003
1000
3902
5000
5003
10000
20000
30003
40000
5900
10300
20003
30030
40300
52009
IZ Z from Closed-loop-CE.
Z Z from LDR-Multi
75-
70-
65-
65-
60-
5%
15-
s0ulsr values
50-
45-
40-
35
15-
10
25
20
25-
20-
20
components
companents
(c) PCA: Closed-loop-CE learned feature ev-(d) PCA: LDR-Multi learned feature for every cry class
class
Figure 23: Comparison Closed-loop- CE and LDR-Mult 1 ž and PCA singular values.
24


Reviews:
[{"review_id": "yGtE5V9mJuk", "reviewer": "Reviewer_hk2J", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths.\n- the core idea of the paper is novel as far as I can tell. I'd argue it has some resemblance to CycleGan (Zhu et al. 2017), but the setting on top of rate reduction and motivation is significantly different.\n- the experiments demonstrate that the method can work well as a generative method, even though it formulates its losses only in latent space.\n\nWeaknesses. \n- The paper is not as self-contained as it could be, i.e. it is near impossible to understand without reading the core referred literature first. It would help readability a lot if core intuitions and concepts would be briefly discussed when introduced. E.g. when introducing MCR2 in equation 4, it would help the reader a lot what the target of each of the two terms is. Another example, after equation 6 it is stated that this measures the volume - why? (if this is not easy to explain, then there should be a reference to where this is explained). The math also needs more clarification, for example what is the union in equation 6 (Z and \\hat Z are both \\in Rˆ{dxn}, or)? Is the \\Delta R in equation 4 and equation 6 really the same (or is it semantically overloaded)?  writing (and logic of the writing) needs to be worked on.\n- Experiments. The comparisons on generative models in Table 1 are done with rather old baselines (all being at least 4 years old). E.g. DCGAN (Radford 2015) could be replaced by StyleGAN2/3 etc (similar for the VAE methods). It should also be reported what the training time for the method on the datasets is, and how it scales with dataset size and resolution. Also, it should be discussed that the model only roughly encodes the semantics, and sometimes disregards even color (e.g. as seen in Figure 5 where the red car turns yellow). In Figure 14 in the abstract it can be seen that the the model seems to sometimes collapse inputs into the same output? If so this should be discussed.\n- Already in the Abstract it is claimed that the model learns a discriminative representation, and it is shown in Table 3 to perform in the ballpark of VAE methods on MNIST data. As the model is also trained on ImageNet (see Fig5), why is no discrimnative comparison performed on ImageNet? It'd be interesting to see if the model can scale up to more classes and more complex datasets (what is the scaling behaviour theoretically of the model in the latent space? Does the dimensionality need to be proportional to the number of classes to ensure the possibility of orthogonality?) . \n\nMinor Issues.\n- The Introduction should start with a section contextualizing the work, i.e. review of context and background, what has been done by others broadly in the field that led to the current work. \n- Equation 2,3 etc. - the result of function g is not a tuple (X, \\hat X). The figure should be changed.\n- After eq. 3 it is written \"later studies [...] surrogate to earth mover's distance\". I think this is wrong. The original GAN paper showed that GAN's minimize Jensen-Shannon distance, and Arjovsky et al showed how to build the WGAN that operates on earth mover's distance.\n- Also on pg 2. - Combination of AE and GAN: \".. started with somewhat different motivation, they have evolved...\". While true that both methods are quite different, as both are generative models, it was this motivation (generative modeling) that they become successful in modeling of real-world data. So, the logic behind this sentence is confusing. \n", "summary_of_the_paper": "The paper \"Closed-Loop Data Transcription To An LDR via Minimaxing Rate Reduction\" introduces a two-player minimax game between an encoder and a decoder to yield a linear discriminative representation (LDR). It achieves this by building on top of the recently proposed MCR2 rate reduction principle, and then motivates a contractive and contrastive measure to yield a minimax game. The paper shows generative empirical results on MNIST, CIFAR-10 and ImageNet, as well as discriminative classification accuracies on MNIST.", "summary_of_the_review": "While the paper proposes an interesting and promising method to learn an encoder-decoder with a novel two-player minimax game, the paper is not yet well enough written and lacks some experimental results that allow for comparison with state of the art methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "wSxRc7geXGt", "reviewer": "Reviewer_ykKP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n\n(1) The proposed adversarial objective function is novel and interesting. \n\n(2) The experimental results are solid. \n\nWeakness: \n\n(1) No background is given on rate reduction and MCR2. As a result, the paper is not self-contained, and is not easy to understand. \n\n(2) The objective function is less simple and clean than VAE and GAN loss functions, which are closely related to log-likelihood of a generative model or a discriminative model. The objective function in this paper is quite complex. \n\n(3) No theoretical analysis of the proposed method is given. \n\n(4) When generating a new x, is z sampled from N(0,I)? The probabilisitic generative model is not very explicit in this paper. A probabilistic generative/decoder model is a natural representation of the observed data, with encoder being interpreted as approximated posterior inference. The auto-encoding point of view appears rather limited in comparison. \n", "summary_of_the_paper": "This paper proposes to learn auto-encoder using an adversarial objective based on rate reduction. The experiments show that the proposed method achieves competitive performance on reconstruction, generation, and discrimination. ", "summary_of_the_review": "The paper is an extension of MCR. It appears a bit complex. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "9b2BOqH2UbU", "reviewer": "Reviewer_beUC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of viewing the encoder and the decoder in a close-loop auto-eoncoder as generator and discriminator is quite interesting. \n\nOne of the critical parts of the proposed method is viewing the encoder as a discriminator and measuring the distance between X and \\hat{X} as the maximization of the rate reduction.  The authors explain that the maximization can avoid that g\\circ f is not an auto-encoding map. However, it is still not super clear that why such maximization is reasonable to make g(f(x)) = x. In other words, why not use eqn (7) together with conventional reconstruction loss. This seems like also provide desired maps between multi-class data and LDR. It is a bit hard for me to see the advantages of using maximization there. \n\nSome minor issues:\nIn eqns (2), Z maps to \\hat{X}, it is a bit strange to write g maps Z to (\\hat(X),X). Similar issue for eqn (5). \n\nIn Section 1.2, X has been amused for different sets. \n\n", "summary_of_the_paper": "The manuscript considers learning a closed-loop auto-encoder between multi-class multi-dimension data and LDR. It also provides various experiments to verify the proposed method. ", "summary_of_the_review": "The explanation of the maximization part or two-player game part is not super clear to me. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors describe an approach to modeling data via an implicit representation that lives on a union of linear subspaces.  While the reviewers consider the authors' approach as novel and having potential, they (and myself) consider that the exposition and notation could be improved, and that the paper as it is is hard to understand and contextualize.
Decision: Reject