Closed-Loop Data Transcription to an LDR via Minimaxing Rate Reduction

1 INTRODUCTION AND RELATED WORK . One of the most fundamental tasks in machine learning is to learn and model complex distributions ( or structures ) of real-world data , such as images or texts , from a set of observed samples . By “ learn and model ” , one typically means that we want to establish a ( parameteric ) mapping between the distribution of the real data , say x ∈ RD , and a more compact random variable , say z ∈ Rd : f ( · , θ ) : x ∈ RD 7→ z ∈ Rd or the inverse g ( · , η ) : z ∈ Rd 7→ x ∈ RD , ( 1 ) where z has certain standard structure or distribution ( e.g . normal distributions ) . The so-learned representation or feature z would be much easier to use for either generative or discriminative purposes . Be aware that the support of the distribution of x ( and that of z ) can be low-dimensional hence the above map ( s ) may not be so well defined off the support in the space RD ( or Rd ) . 1.1 LEARNING GENERATIVE MODELS VIA AUTO-ENCODING OR GAN . Auto-Encoding . In the machine learning literature , roughly speaking , there have been two representative approaches to such a distribution-learning task . One is the classic “ Auto Encoding ” ( AE ) approach ( Kramer , 1991 ; Hinton & Zemel , 1993 ) that aims to simultaneously learn an encoding mapping f from x to z and an ( inverse ) decoding mapping g from z back to x : X f ( x , θ ) −−−−−−→ Z g ( z , η ) −−−−−−→ X̂ . ( 2 ) Here we use bold capital letters to indicate a matrix of finite samples X = [ x1 , . . . , xn ] ∈ RD×n of x and their mapped features Z = [ z1 , . . . , zn ] ⊂ Rd×n , respectively . Typically , one wishes for two properties : firstly , the ( empirical ) distribution of the mapped samples Z , denoted as p̂ ( z |X , θ ) , is close to certain desired distribution p ( z ) , say some much lower-dimensional disentangled multivariate Gaussian ; 1 and , secondly , the decoded samples X̂ are “ similar ” or close to the original X . In the feature space z , to enforce the learned distribution p̂ ( z ) to be close to a target p ( z ) , one normally minimizes a certain “ distance ” between the two distributions , say the KL-divergence : minDKL ( p̂ , p ) . However , it is very difficult , often computationally intractable , to precisely compute or minimize such a distance D between two arbitrary degenerate distributions in high-dimensional 1The classical PCA can be viewed as a special case of this task . In fact , the original auto-encoding is precisely justified as nonlinear PCA ( Kramer , 1991 ) . spaces . So in practice , one typically chooses to minimize instead certain approximate bounds ( surrogates ) of the distance , such as a variational bound in variational auto-encoding ( VAE ) ( Kingma & Welling , 2013 ; Zhao et al. , 2017 ) and the earth mover ’ s ( Wasserstein ) distance . In this work , we will show that if we impose specific requirements on the ( distribution of ) learned feature z ( e.g . being subspace-like Gaussians ) , natural distances can be given in closed-form . GAN . Compared to measuring distribution distance in the ( often controlled ) feature space z , a much more challenging issue is how to effectively measure the distance between the decoded samples X̂ and the original X in the data space x . For instance , for visual data such as images , despite extensive studies in the computer vision and image processing literature ( Wang et al. , 2004 ) , it remains elusive to find a good measure for similarity of real images that is both efficient to compute and effective in capturing visual quality and semantic information of the images equally well . Generative Adversarial Nets ( GAN ) ( Goodfellow et al. , 2014 ) offers an ingenious idea to resolve this difficulty by utilizing a powerful discriminator d , usually modeled and learned by a deep network , to discern differences between the generated samples X̂ and the real ones X : Z g ( z , η ) −−−−−−→ X̂ , X d ( x , θ ) −−−−−−→ Y ∈ 0,1 } . ( 3 ) Such a discriminator minimizes the Jensen-Shannon divergence , between the data X and X̂ . But as shown in ( Arjovsky et al. , 2017 ) , the JS-divergence can be highly ill-conditioned to optimize when the distributions are low-dim . So in practice , one may choose to replace the JS-divergence with the earth mover ’ s distance or other variants . The original GAN aims to directly learn a mapping g , called a generator , from a standard distribution ( say a low-dim Gaussian random field ) to the real ( visual ) data distribution in a high-dim space . However , distributions of real-world data can be rather sophisticated and often contain multiple classes and multiple factors in each class ( Bengio et al. , 2013 ) . That makes learning the mapping g rather challenging in practice , suffering difficulties such as mode-collapse ( Srivastava et al. , 2017 ) . As a result , many variants of GAN have been subsequently developed in order to improve the stability and performance in learning multiple modes and disentangling different factors in the data distribution , such as Conditional GAN ( Mirza & Osindero , 2014 ; Sohn et al. , 2015 ; Mathieu et al. , 2016 ; Van den Oord et al. , 2016 ; Wang et al. , 2018 ) , InfoGAN ( Chen et al. , 2016 ; Tang et al. , 2021 ) , or Implicit Maximum Likelihood Estimation ( IMLE ) ( Li & Malik , 2018 ; Li et al. , 2020 ) In particular , to learn a generator for multi-class data , prevalent conditional GAN literature requires label information as inputs during training ( Mirza & Osindero , 2014 ; Odena et al. , 2017 ; Dumoulin et al. , 2016 ; Brock et al. , 2018 ) . Recently Wu et al . ( 2019b ; a ) has proposed to train a k-class GAN by generalizing the two-class cross entropy to a ( k + 1 ) -class cross entropy . In this work , we will introduce a more refined 2k-class measure for the k real and k generated classes . In addition , to avoid features for each class to collapse to a singleton , we will use the so-called rate reduction measure that promotes multi-dimension in the learned features ( Yu et al. , 2020 ) . Another line of research is about how to stablize the training of GAN . SN-GAN ( Miyato et al. , 2018 ) has shown spectral normalization on the discriminator is rather effective , which we will adopt in our work . PacGAN ( Lin et al. , 2018 ) shows that the training stability can be significantly improved by packing a pair of real and fake images together for the discriminator . In this work , we show how to naturally generalize this idea to discriminating an arbitrary number of pairs of real and decoded samples . Also , Wu et al . ( 2019a ) has shown that optimizing the latent features leads to state of the art visual quality . There are strong reasons to believe that their method essentially utilizes the Compressed Sensing principle to implicitly exploit low-dimensionality of the feature distribution . Our framework will explicitly impose and exploits such low-dimensional structures of the learned feature distribution . Combination of AE and GAN . Although AE ( VAE ) and GAN have started with somewhat different motivations , they have evolved into popular and effective frameworks for learning and modeling complex distributions of many real-world data such as images . Many recent efforts tend to combine both Auto-Encoding and GAN to generate more powerful generative frameworks for more diverse data sets , such as Larsen et al . ( 2015 ) ; Rosca et al . ( 2017 ) ; Srivastava et al . ( 2017 ) ; Bao et al . ( 2017 ) ; Huang et al . ( 2018 ) ; Ulyanov et al . ( 2018 ) ; Vahdat & Kautz ( 2020 ) . As we will see , in our framework , AE and GAN can be naturally interpreted as two segments of a closed-loop data transcription process . But unlike GAN or VAE , the distribution of the feature z is learned from the data x and its low-dim support in Rd is explicitly modeled as a union of discriminative subspaces . 1.2 LEARNING LINEAR DISCRIMINATIVE REPRESENTATION VIA RATE REDUCTION . Recently , Chan et al . ( 2021 ) has proposed a new objective for deep learning that aims to learn a linear discriminative representation ( LDR ) for multi-class data . The basic idea is to map distributions of real data , potentially on multiple nonlinear submanifolds , to a family of canonical models consisting of multiple incoherent ( or orthogonal ) linear subspaces . To some extent , this generalizes nonlinear PCA ( Kramer , 1991 ) to the more general/realistic settings where we simultaneously apply multiple nonlinear PCAs to data on multiple nonlinear submanifolds . Unlike conventional discriminative methods that only aim to predict class labels , the LDR aims to learn the likely multi-dimensional distribution of the data hence is potentially suitable for both discriminative and generative purposes . It has been shown that this can be achieved via maximizing the so-called “ rate reduction ” objective . MCR2 . More precisely , consider a set of data samples X = [ x1 , . . . , xn ] ∈ RD×n from k different classes and we use Πj , j = 1 , . . . , k to denote the memberships of the samples in k classes X = ∪kj=1Xj . One seeks a continuous mapping f ( · , θ ) : x 7→ z from X to an optimal representation Z = [ z1 , . . . , zn ] ⊂ Rd×n that maximizes the following coding rate reduction objective , known as the MCR2 principle ( Yu et al. , 2020 ) : max Z ∆R ( Z , Π , ) . = 1 2 log det ( I + αZZ∗ ) ︸ ︷︷ ︸ R ( Z , ) − k∑ j=1 γj 2 log det ( I + αjZΠ jZ∗ ) ︸ ︷︷ ︸ Rc ( Z , |Π ) , ( 4 ) where α = dn 2 , αj = d tr ( Πj ) 2 , γj = tr ( Πj ) n for j = 1 , . . . , k. In this paper , for simplicity we denote ∆R ( Z , Π , ) as ∆R ( Z ) assuming Π , are known and fixed . The first term R ( Z | ) is the coding rate of the whole feature set Z ( coded as a Gaussian source ) with a prescribed precision ; the second term Rc ( Z |Π , ) is the average coding rate of the k subsets of features Zj = f ( Xj ) ( each coded as a Gaussian ) . As it has been shown by Yu et al . ( 2020 ) , maximizing the difference between the two terms will “ expand ” the whole feature set while “ compressing and linearizing ” features of each of the k classes . If the mapping f maximizes the rate reduction , it maps the features of different classes into independent ( orthogonal ) subspaces in Rd . Figure 1 illustrates a simple example of data with k = 2 classes ( on two submanifolds ) mapped to two incoherent subspaces ( solid black lines ) .

Summary: The paper "Closed-Loop Data Transcription To An LDR via Minimaxing Rate Reduction" introduces a two-player minimax game between an encoder and a decoder to yield a linear discriminative representation (LDR). It achieves this by building on top of the recently proposed MCR2 rate reduction principle, and then motivates a contractive and contrastive measure to yield a minimax game. The paper shows generative empirical results on MNIST, CIFAR-10 and ImageNet, as well as discriminative classification accuracies on MNIST.