Year: 2019

Abstract:
We present a generative adversarial network (GAN) that conducts manifold learning and alignment (MLA): A task to learn the multi-manifold structure underlying data and to align those manifolds without any correspondence information. Our main idea is to exploit the powerful abstraction ability of encoder architecture. Specifically, we define multiple generators to model multiple manifolds, but in a particular way that their inverse maps can be commonly represented by a single smooth encoder. Then, the abstraction ability of the encoder enforces semantic similarities between the generators and gives a plausibly aligned embedding in the latent space. In experiments with MNIST, 3D-Chair, and UT-Zap50k datasets, we demonstrate the superiority of our model in learning the manifolds by FID scores and in aligning the manifolds by disentanglement scores. Furthermore, by virtue of the abstractive modeling, we show that our model can generate data from an untrained manifold, which is unique to our model.

text: We present a generative adversarial network (GAN) that conducts manifold learning and alignment (MLA): A task to learn the multi-manifold structure underlying data and to align those manifolds without any correspondence information. Our main idea is to exploit the powerful abstraction ability of encoder architecture. Specifically, we define multiple generators to model multiple manifolds, but in a particular way that their inverse maps can be commonly represented by a single smooth encoder. Then, the abstraction ability of the encoder enforces semantic similarities between the generators and gives a plausibly aligned embedding in the latent space. In experiments with MNIST, 3D-Chair, and UT-Zap50k datasets, we demonstrate the superiority of our model in learning the manifolds by FID scores and in aligning the manifolds by disentanglement scores. Furthermore, by virtue of the abstractive modeling, we show that our model can generate data from an untrained manifold, which is unique to our model.

heading: 1 INTRODUCTION
text: Real-world datasets generally have multi-manifold structure. Smoothly varying features such as the size or the pose of an animal introduce smooth manifold structure, while the existence of discrete features such as species makes the structure not a single manifold but multiple disconnected manifolds (e.g., one manifold per species) (Khayatkhoei et al., 2018). Importantly, these manifolds are correlated to each other in their structure since the transformation rules according to the size (stretching the foreground patch) or the pose (rotating the patches of body parts) are similar even for different species. Hence, our desire is to build a model that learns not only the multi-manifold structure but also the correlations between the manifolds.
As for the learning of manifold structure, generative adversarial networks (GANs) (Goodfellow et al., 2014) have been known for their remarkable performance. In particular, recently proposed GAN models have successfully demonstrated multi-manifold learning by using multiple generator networks (Khayatkhoei et al., 2018; Ghosh et al., 2017; Hoang et al., 2018) or giving a mixture density on the latent space (Xiao et al., 2018; Gurumurthy et al., 2017). However, none of these models have taken the correlations between the manifolds into account.
On the other hand, a field of research named manifold alignment (Ham et al., 2003) has its primary concern in learning the correlations between the manifolds. In particular, unsupervised manifold alignment (Cui et al., 2014; Wang & Mahadevan, 2009) aims to learn plausible correspondences between the data points of different manifolds, without any ground-truth correspondence information is given. However, most existing methods focus only on finding the correspondences, not on learning the structure of the manifolds itself.
In this work, we propose a model that performs both of the tasks, which we call as manifold learning and aligning GAN (MLA-GAN). MLA-GAN is similar to the GANs with multiple generators (Khayatkhoei et al., 2018; Ghosh et al., 2017; Hoang et al., 2018), but the generators are restricted such that their inverse maps can be represented by a single deep encoder. Due to the abstracting property of deep encoders, this restriction guides the inverse maps to learn semantically similar mappings, and the model achieves a semantically plausible manifold alignment. The main contributions of this work can be summarized as follows.
• The MLA problem: Up to our knowledge, MLA-GAN is the first model that performs both multi-manifold learning and alignment with real-world datasets. It shows state-ofthe-art FID scores in data generation and plausible alignments with disentangled features.
• Generalization to an untrained manifold: Due to the abstracting nature of an encoder, the smooth features can be inferred even for data lying on an untrained manifold. Exploiting this, one can generate other data points lying on this manifold.
• Easily applicable: Due to the new regularizer we propose, the inverse maps used for restricting the generators do not need to be actually computed. This makes MLA-GAN easily applicable to any existing GAN architectures.

heading: 2 MANIFOLD LEARNING AND ALIGNMENT WITH GAN

heading: 2.1 BACKGROUNDS
text: Multi-Manifold Learning GANs Conventional GANs (Goodfellow et al., 2014) model data by transforming a latent-space distribution p(z) with a generating map fG : Z → X . If fG is smooth and injective1, the image of fG is a smooth manifold M ⊂ X , thus the data are modeled as they are lying on this manifold. To deal with multiple manifolds, recent extensions (Khayatkhoei et al., 2018; Ghosh et al., 2017; Hoang et al., 2018) employ multiple generating maps {f (i)G }Ai=1 where each f (i)G accounts for the i-th manifold M (i).
Unsupervised Manifold Alignment Most existing methods formulate the manifold alignment problem as finding embeddings of manifolds into a common low-dimensional space Y . This set of embeddings {r(i)E : M (i) → Y }i essentially defines an alignment of manifolds; data points that are mapped to the same point in Y correspond to each other. To achieve a plausible alignment, the embeddings are trained to minimize several loss terms guiding geometrical similarity (see Sec. 3).
1Although these conditions are not guaranteed for general neural networks, they are approximately met or observed in practice (Shao et al., 2017). Additionally, the support of p(z) needs to be a completely separable Hausdorff space, but this condition is met for the usual choices of p(z) (e.g., unimodal Gaussian).

heading: 2.2 MAIN IDEA
text: To model multiple manifolds, we use multiple generating maps {f (i)G : Z → M (i)}i as with the aforementioned GAN models. Interestingly, the inverses2 of these maps {(f (i)G )−1 : M (i) → Z}i can be regarded as a set of embeddings (where Y is replaced by Z), which means that the manifold alignment is automatically defined as we define the manifolds. However, this alignment is not at all semantically plausible unless properly regulated. Here, we regulate it by restricting the generating maps such that their inverses to be commonly represented by a single encoder hE : X → Z. Formally, we restrict the generating maps as follows:
{f (i)G }i : A set of functions (Z →M (i) ⊂ X) s.t.
∃hE ∈ H satisfying (f (i)G ) −1(x) = hE(x) ∀i,∀x ∈M (i). (1)
Here, H is a certain set of functions (to be clearer soon), and hE being an element of H is an important constraint. Without this constraint, one can always find hE satisfying the above condition, for any sets of {f (i)G }i, because hE is merely a union of functions {(f (i) G ) −1}i extending the domain
from ⋃
iM (i) ⊂ X to X . However, if H is, for example, a set of smooth functions and hE ∈ H,
then {(f (i)G )−1}i will be restricted to be similar to each other and the alignment will become more plausible. In our model, H is defined as a set of functions represented by deep encoder networks. We explain the intuition behind this design in what follows.
Intuition Consider a dataset containing images of cats and dogs. It is not difficult to imagine a deep encoder network that takes an image from this dataset and outputs the size and pose regardless of the species. At least, we could have trained this encoder easily if the size and pose labels were given. Also, even if we train the encoder using the cat data only, we still expect it works fairly well for the dogs due to the powerful abstraction and generalization abilities of deep encoders. We want to exploit these abilities of deep encoders for aligning the manifolds. If we let H be a set of deep encoders, its element hE , once well-trained, will abstract the species away and outputs the size and pose. This gives a nice manifold alignment, where a cat and a dog images with the same size and pose correspond to each other.
Remaining Question Is H being a set of deep encoders enough to obtain the abstraction and generalization? An answer to this question is related to how deep networks generalize so well despite the large number of parameters they have. Several works conjecture that it is due to the SGD (Zhang et al., 2016; Brutzkus et al., 2018), but this is not a fully developed theory yet. At least, we can assume that deep encoders learn simpler, generalization-friendly maps among others, based on various cases (e.g., VGG-Net (Simonyan & Zisserman, 2014), Fast R-CNN (Girshick, 2015)).

heading: 2.3 MLA-GAN
text: What remains is to implement the generating maps with deep neural networks. The main challenge is to make the deep networks comply with the constraint given in Eq. 1. In this section, we explain how this challenge can be achieved in the proposing MLA-GAN model in detail. We first focus on a single linear layer and prove the following (detailed in Proposition 1):
If linear generating maps are defined with the same weight and the same tangential components of bias, their inverses can always be represented by a single linear encoder. Thus, the maps comply with Eq. 1.
Then, to realize the above, we introduce a new form of regularizer. After, we move on to the entire network and derive MLA-GAN by applying the same principles as the linear layer.

heading: 2.3.1 SINGLE LINEAR LAYER
text: Consider a set of linear generating maps {f (i)G : Z →M (i) ⊂ X}i; each of the maps is defined as
f (i) G (z) := U (i)z + a(i),
2With the same assumption as Footnote 1.
where U (i) ∈ RdX×dZ (dX > dZ) and a(i) ∈ RdX are the weight and bias parameters respectively. We assume U (i) is a full-rank matrix (rank-dZ) such that f (i) G is injective (see Footnote 1). Then, the inverse (f (i)G ) −1 : M (i) → Z can be derived as
(f (i) G ) −1 = (U (i))+(x(i) − a(i)) (2) where (U (i))+ := ( (U (i))>U (i) )−1 (U (i))> denotes the pseudo-inverse of U (i). As noted in Eq. 1,
our desire is to restrict the inverse maps {(f (i)G )−1}i such that they can be represented by a single encoder hE : X → Z. One simple way to achieve this is to use the following proposition. Proposition 1. If the linear generating maps {f (i)G (z)}i are restricted to have the same weight U and to have the same tangential components of bias a‖ (i.e., a(i) = a‖ + a (i) ⊥ ), then their inverses {(f (i)G )−1}i can be represented by a single linear encoder hE(x) := W>x + b where W = U(U>U)−1 and b = −W>a‖.
Proof. Note the bias a(i) can be decomposed into the tangential component a(i)‖ and the normal component a(i)⊥ to the column space of U >. Substituting the condition a(i) = a‖+a (i) ⊥ and the same weight U , Eq. 2 is written as
(f (i) G ) −1 = U+x− U+(a‖ + a (i) ⊥ )
= U+x− U+a‖ = W>x+ b.
Bias Regularizer When implementing Proposition 1, making {U (i)}i the same is as trivial as setting the same weight U for all f (i)G , but making {a (i) ‖ }i the same is not trivial since the tangential direction keeps changing while training. One solution would be to use a regularizer minimizing the sum of the variance: trace(cov(a(i)‖ )). However, computing this term is intractable due to the inversion (U>U)−1 inside of a(i)‖ = U(U >U)−1U>a(i).
Theorem 1. The following inequality holds: trace ( cov(U>a(i)) ) ≥ 1 dz H({λk}dzk=1)trace ( cov(a(i)‖ ) )
where {λk}dzk=1 denotes the eigenvalues of U>U and H(·) denotes harmonic mean.
Proof. See Appendix B.
As the harmonic mean in Theorem 1 is constant from the perspective of a(i)‖ , we can minimize the original term by minimizing the upper bound instead. With an additional log function to match the scale due to the dimensionality, we propose the upper bound as a regularizer to make a(i)‖ the same:
Regularizer: Rbias = log ( trace ( cov(U>a(i)) )) . (3)

heading: 2.3.2 ENTIRE NETWORK
text: A single linear layer is, of course, insufficient as linear generators can only model the linear manifolds, and a linear encoder can align the manifolds only linearly by removing the normal components. So, we implement the generating maps with deep networks to model complex nonlinear manifolds. As usual, the deep networks are constructed by interweaving linear layers and nonlinear activations, and optionally batch-normalization layers. But, each linear layer here implements the conditions of Proposition 1. That means, for the l-th linear layer of each deep generating map f (i)G , their weights are set as the same, denoted as Ul, and their biases {a(i)l }i are regularized to have the same tangential components using Eq. 3.
Note that the above set of deep generating maps comply with Eq. 1. As all the linear layers, nonlinear activations (we use LeakyReLU; see Appendix C), and batch-norm layers are injective, the deep generating maps have inverses. These deep inverse maps can be represented by a single deep encoder, as the inverses of all the component-linear maps can be represented by single encoders.
Training As above, let us denote the weight of the l-th linear layer asUl and the biases as {a(i)l }Ai=1 . Then, we can express the data generating distribution as an ancestral sampling:
x ∼ f (i)G ( z; {Ul, a(i)l } L l=1 ) where z ∼ p(z), i ∼ πi.
Here, πi stands for the probability of selecting the i-th bias. This probability could be also learned using the method proposed in Khayatkhoei et al. (2018), but it is beyond our scope and we fix it as 1/A. Now, denoting the real data distribution as pR and the fake distribution that the above sampling presents as pG, we define our GAN losses as:
MLA-GAN Loss:
{ LG = −Ex∼pG [D(x)] + λ ∑L l=1 log ( trace ( cov(U>l a (i) l ) ))
LD = Ex∼pG [D(x)]− Ex∼pR [D(x)]
where LG and LD are the generator and the discriminator losses respectively and λ is a regularization weight. We use Wasserstein GAN (WGAN) Arjovsky et al. (2017) so the discriminator D(x) is restricted to a k-Lipschitz function.
Encoding Due to the bias regularizer, we do not need to concretize the encoder hE by inverting f (i) G during training. But, when encoding is needed, we can obtain the latent codes and biases by minimizing the distance along with the bias regularizer as follows (as suggested in Ma et al. (2018)).
z, {al}Ll=1 = arg min z̃,{ãl}Ll=1 ∥∥x− fG (z̃; {ãl}Ll=1)∥∥2 + µ L∑ l=1 log (∥∥U>l al − U>l āl,‖∥∥2) , (4)
Importantly, one can encode an image that is considerably different from the trained images using this method. By virtue of the abstraction ability of the encoder, the encoded latent code is actually semantically plausible. Moreover, as the encoded biases fully describe the generating map, one can virtually generate any data lying on the same manifold. This is unique to our model and to be further discussed in Sec. 4.3.

heading: 3 RELATED WORK
text: Multi-Manifold Learning GANs There exist roughly three different methods to handle multiple manifolds with GANs (see also Appendix A). Using multiple generators (DMWGAN (Khayatkhoei et al., 2018); Ghosh et al. (2017); Hoang et al. (2018)) is the simplest one, but it cannot align the manifolds as the generators are independent to each other without any regulations. Giving a mixture density for the latent space prior (Xiao et al., 2018; Gurumurthy et al., 2017) is another method; although this method uses a single generating map, it hardly aligns the manifolds since the manifolds are modeled by different regions in the latent space where the mappings are effectively different from each other. The last method (InfoGAN (Chen et al., 2016)) introduces discrete latent variables and concatenate them with the original continuous latent variables. Here, the discrete information is slowly blended in, layer by layer, and modulates the generating map. As the overall structure is shared among the manifolds to some degree, this method does align the manifolds; but as it is not explicitly guided, the performance is not as good as MLA-GAN (Table 1).
Unsupervised Manifold Alignment The existing methods (Cui et al., 2014; Wang & Mahadevan, 2009) find a correspondence between the points of different manifolds by learning embeddings, with consideration for geometrical similarity. As the geometry is captured by adjacency matrices, one shortcoming is that the alignment can be done only in a point-to-point manner. Moreover, as the embeddings are modeled as linear functions, it is inevitable to lose information about the manifolds.
Deep generative models involving an encoder Utilization of an encoder is a notable feature of MLA-GAN, but there are several other deep generative models using encoders, such as adversarial auto-encoders (Makhzani et al., 2015), variational auto-encoders (Kingma & Welling, 2013; Higgins et al., 2016) and BiGAN (Donahue et al., 2016). However, there is a large difference: The encoders of these models capture all the necessary information in order to reconstruct the data, while the encoder of MLA-GAN removes the discrete information in the latent space to abstract the manifolds.

heading: 4 EXPERIMENTS
text: Datasets We experiment on MNIST (Lecun et al., 1998), 3D-Chair (Aubry et al., 2014) and UTZap50k (Yu & Grauman, 2014) image datasets. 3D-Chair contains 1393 distinct chairs rendered from 62 different viewing angles (total 86,366 images); in experiments, only front-looking 44,576 images are used and rescaled to 64x64 grayscale images. UT-Zap50k contains images of 4 different types of shoes (total 50,025 images); the images are rescaled to 32x32.
Model Architecture We use DCGAN (Radford et al., 2015)-like model architectures for all the datasets (see Appendix C for the complete information). For each of the linear layers in the generator, the number of biases A is set as 10 (MNIST), 20 (3D-Chair), and 4 (UT-Zap50k). Although our multi-biased linear layer can be applied to both fully-connected and transposed-convolution layers, we apply it only to the former. This is sufficient for our purpose since discrete features rarely exist for such small-sized kernel patches. In the discriminator, we use spectral normalization (Miyato et al., 2018) to achieve the k-Lipschitz condition for WGAN. For training and encoding, Adam (Kingma & Ba, 2014) is used with the defaults except for the learning rate, 0.0002.

heading: 4.1 MULTI-MANIFOLD LEARNING
text: MLA-GAN shows great performance in multi-manifold learning and alignment, both qualitatively and quantitatively. Looking at Fig. 3 row-wise, we can see that MLA-GAN learns distinct manifolds well, where each manifold accounts for a different discrete class: e.g., different digits; rolling vs. armchairs; boots vs. flat shoes. Column-wise, we can see that the smooth features are well aligned among the manifolds: e.g., stroke weight, the slant of numbers; viewing angle of chairs; colors of shoes. Note that these alignments are qualitatively much better than that of the existing alignment methods given in Fig. D.2. For quantitative evaluation, we compute FID scores (Heusel et al., 2017), which is widely used to measure the diversity and quality of the generated image samples, reflecting the manifold learning performance overall. Table 1 shows that our model gives better or comparable scores than others.

heading: 4.2 MANIFOLD ALIGNMENT AND DISENTANGLED FEATURES
text: To further investigate the manifold alignment performance, we examine how much the learned latent features are disentangled. We take a few images from the dataset and manually change one of the smooth features that corresponds to a known transformation (e.g., sheer transform). Then, we encode these images to the latent codes using our model, analyze the principal direction of the change, and compute the linearity of the change as the disentanglement score (see Appendix E). Fig. 4 shows that the learned smooth features are well disentangled along the principal changing directions. Table 1 shows that our model gets better scores than other models, but sometimes not as good as β-VAE. However, it should be noted that the FID scores of β-VAE is significantly worse.

heading: 4.3 STYLE TRANSFER
text: We demonstrate a style-transfer between images using our model by matching the latent features (Fig. 5). Specifically, we encode the source and target images, then use the inferred biases of the source and the inferred latent codes of the target to generate the style-transferred image.
Interestingly, MLA-GAN is even able to style-transfer the images that are off the trained distribution (see the red boxes). This is an exclusive property of MLA-GAN due to the encoder. If the encoder generalizes well (which apparently does), it can recognize the smooth features of an image even if the image has noises such as added frames or color modification. Once the smooth features are recognized, the model knows how to transform this image smoothly, similar to the original data, but on this new manifold involving noises (as described in Eq. 4 and below).

heading: 5 CONCLUSION
text: We proposed MLA-GAN that performs multi-manifold learning and alignment. To our knowledge, MLA-GAN is the first model that performs both of the tasks for real-world datasets, and it mostly showed superior performance. Unique to MLA-GAN is that it utilizes the abstraction ability of an encoder, from which it showed its potential to infer the untrained manifolds in the style-transfer experiment. If it is trained with larger datasets such as ImageNet in the future, we expect MLA-GAN would become a more versatile tool that can infer the manifold structure of general images.
3Here, we also train on RGB-MNIST dataset: a simple extension of MNIST by coloring with (R, G, or B).

heading: A COMPARISON OF VARIOUS GAN MODELS

heading: B THE PROOF OF THEOREM 1
text: Theorem 1. The following inequality holds
trace ( cov(U>a(i)) ) ≥ 1 dz H({λk}dzk=1)trace ( cov(a(i)‖ ) ) where {λk}dzk=1 are the eigenvalues of U>U and H(·) denotes a harmonic mean.
Proof. Note that trace ( cov(a(i)‖ ) ) = trace ( cov(U(U>U)−1U>a(i)) )
= trace
( 1
A− 1 A∑ i=1
U(U>U)−1U>(a(i) − ā)(a(i) − ā)>U(U>U)−1U> )
= trace
( 1
A− 1 A∑ i=1
U>(a(i) − ā)(a(i) − ā)>U(U>U)−1 )
= trace ( cov(U>a(i))(U>U)−1 )
≤ trace ( cov(U>a(i)) ) trace ( (U>U)−1 ) ,
where the second and the fourth lines use the definition of the covariance, third line is obtained from the cyclic property of trace and the last line is obtained from the Cauchy-Schwarz inequality of the positive semi-definite matrices. Thus,
trace ( cov(U>a(i)) ) ≥ trace ( cov(a(i)‖ ) ) /trace ( (U>U)−1 ) = 1
D H({λd}Dd=1)trace
( Var(a(i)‖ ) ) where {λd}Dd=1 are the eigenvalues of U>U and H(·) denotes the harmonic mean.

heading: C MODEL ARCHITECTURE AND EXPERIMENTING ENVIRONMENTS
text: We used machines with one NVIDIA Titan Xp for the training and the inference of all the models.
C.1 MNIST
We use A = 10 distinct decoding biases in the model. In the training, we set the regularization weight λ = 0.05 and use the Adam optimizer with learning rate 0.0002. In the encoding, we use the Adam optimizer with learning rate 0.1, and the set the regularization weight µ = 0.1.
C.1.1 NOTES ON THE OTHER COMPARED MODELS
Overall, we match the architecture of other models with our model for fair comparison. Some differences to note are:
• DMWGAN: We used 10 generators. Each generator has the same architecture as ours except the number of features or the channels are divided by 4, to match the number of trainable parameters. Note that 4 is the suggested number from the original paper.
• β-VAE: We used Bernoulli likelihood.
• InfoGAN: Latent dimensions consist of 1 discrete variable (10 categories), 2 continuous variables and 8 noise variables.
C.2 3D-CHAIR
We use A = 20 distinct decoding biases in the model. In the training, we set the regularization weight λ = 0.05 and use the Adam optimizer with learning rate 0.0002. In the encoding, we use the Adam optimizer with learning rate 0.1, and the set the regularization weight µ = 0.1.
C.2.1 NOTES ON THE OTHER COMPARED MODELS
• DMWGAN: We used 20 generators. Each generator has the same architecture as ours except the BatchNorms are removed and the number of features or the channels are divided by 4, to match the number of trainable parameters. Note that 4 is the suggested number from the original paper. Note that this was the best setting among what we have tried (division number 2; ones with the BatchNorms).
• β-VAE: We used Bernoulli likelihood.
• InfoGAN: Latent dimensions consist of 3 discrete variables (20 categories), 1 continuous variable and 10 noise variables.
C.3 UT-ZAP50K
We use A = 4 distinct decoding biases in the model. For the regularization weight in the training, we start with λ = 5e− 6 then raise to λ = 5e− 4 after 300 epochs.

heading: D MANIFOLD ALIGNMENT RESULTS OF THE EXISTING METHODS
text: We show the manifold alignment results of Wang & Mahadevan (2009) for MNIST dataset.

heading: E DISENTANGLEMENT SCORE
text: To compute the disentanglement score, we first take 500 images from the dataset and manually change one of the smooth features that corresponds to a known transformation. For example, we change the slant of the MNIST digits by taking a sheer transform. With 11 different degrees of the transformation, we obtain 5500 transformed images in total. We encode these images to obtain the corresponding latent codes and subtract the mean for each group of the images (originates from the same image) to align all the latent codes. Then, we conduct Principal Component Analysis (PCA) to obtain the principal direction and the spectrum of variations of the latent codes. If the latent features are well disentangled, the dimensionality of the variation should be close to one. To quantify how much it is close to one, we compute the ratio of the first eigenvalue to the second eigenvalue of the PCA covariance, and set it as the disentanglement score.

heading: F EFFECT OF THE NUMBER OF GENERATORS, A
text: To investigate the effect of the number of generators (or biases), A, we train our model on MNIST with different A values then compute the FID and the disentanglement scores (Fig. F.3). It can be seen that our model performs consistently better than the baseline, WGAN, regardless of the different A values.
Figure F.3: The FID scores (right axis, the smaller the better) and the disentanglement scores (left axis, the larger the better) of MLA-GAN with varying A are shown for MNIST dataset. The dashed lines show the mean scores of the baseline model (WGAN).
G EFFECT OF THE REGULARIZATION WEIGHT, λ, IN TRAINING
To investigate the effect of the regularization weight, λ, we train our model on MNIST with different λ values. It can be seen that our model performs consistently better than the other compared models — MLA-GAN with no regularization (λ = 0), DMWGAN, MADGAN-like (see the next paragraph)
— regardless of the different λ values (other models are omitted for better readability; see Table 1 for the omitted ones). It can be also seen that the scores are not very sensitive to the different choices of λ’s; this is beneficial in that one may choose any reasonable value for λ when training the model with a new dataset.
Here, MADGAN-like is a DMWGAN model, but has a similar parametrization to the MADGAN (Ghosh et al., 2017): The parameters of the first three layers (from the top) are shared for all the generators. In contrast, MLA-GAN shares the parameters in all the layers except the biases in the fully-connected layers. Thus, in a sense, one can say that MLA-GAN shares only the last few layers, whereas MADGAN-like shares only the first few layers (of course, there is another difference due to the regularizers). Although the both models have the shared structures among the generators, MLA-GAN performs much better in both of the scores as seen in Figure G.4. This suggests that a simple parameter sharing is not enough to obtain a good performance, and the bias regularizer is indeed required.
(a) FID Scores
(b) Disentanglement Score (Slant) (c) Disentanglement Score (Width)
Figure G.4: The FID scores (the smaller the better) and the disentanglement scores (the larger the better) of MLA-GAN with varying λ are shown for MNIST dataset. Note the other models are positioned in the center (shaded in gray) to be visually comparable with the best-performing MLAGAN model (λ = 0.05).
H EFFECT OF THE REGULARIZATION WEIGHT, µ, IN ENCODING
To investigate the effect of the regularization weight, µ, in encoding (Eq. 4), we take a trained model and encode an image from the train set using different µ values. Then, plugging in the encoded (estimated) biases, {al}Ll=1, we randomly generate the samples from this new manifold and compare the qualities for different µ’s.
From Figure H.5 (b), we can see that the quality of the encoding tends to improve as µ gets smaller. This might seem opposite to what we expect, as weaker regularization gives better results. However, looking closer, we can see that the features like the slant and the stroke are more aligned with a
stronger regularization of µ = 0.05, when comparing with the other manifolds in the bottom pane. Thus, a trade-off exists here between the image quality of the samples and the alignment of the manifold to the others. Note we chose to use µ = 0.05 in all the other experiments.
(a) Left: An image taken from the training dataset. Middle: The same image with a rectangle noise. Right: Regenerated image from the estimated biases, {al}Ll=1, and the estimated latent code z, from Eq. 4.
(b) Top pane: Generated samples from the estimated biases, {al}Ll=1. The biases are estimated with different µ’s, (5.0, 0.5, 0.05, 0.005, 0.0005, 0.0), from the top to the bottom. Bottom pane: Generated samples from the originally-learned biases, {{a(i)l } A i=1}Ll=1 (only 5 out of 10 are shown). Note the images in the same column have the same latent value z.
Figure H.5: Effect of the regularization weight µ in encoding is shown with a MNIST-trained MLAGAN model.

heading: I EFFECT OF THE BIAS REGULARIZER
text: To examine the effectiveness of our bias regularizer, we visualize the raw values of biases {a(i)l }i,l and their (pseudo-)tangential component {U>l a (i) l }i,l (see Fig. I.6, I.7). In all figures, we see that the biases are diverse, but their tangential components are well aligned due to the bias regularizer (left). On the contrary, without the regularizer, the tangential components are not aligned (right).
J SAMPLES GENERATED FROM VARIOUS MODELS
(a) MLA-GAN (b) MLA-GAN (λ = 0)
(c) DMWGAN (d) WGAN
(e) β-VAE (f) InfoGAN
Figure J.8: MNIST image samples generated from the trained models
(a) WGAN (b) β-VAE (c) InfoGAN
Figure J.9: 3D-Chair image samples generated from the trained models
Figure J.10: 3D-Chair image samples generated from the trained MLA-GAN
Figure J.11: 3D-Chair image samples generated from MLA-GAN (λ = 0)
Figure J.12: 3D-Chair image samples generated from the trained DMWGAN

Review summaries:
The paper suggests performing simultaneous manifold learning and alignment by multiple generators that share common weight matrices and a constructed inverse map that is instantiated by a single encoder. The method utilizes a special regularizer to guide the training. It has been empirically well tested on a multi-manifold learning task, manifold alignment, feature disentanglement, and style transfer.