Title: MANIFOLD LEARNING AND ALIGNMENT WITH GENERATIVE ADVERSARIAL NETWORKS

Abstract: We present a generative adversarial network (GAN) that conducts manifold learning and alignment (MLA): A task to learn the multi-manifold structure underlying data and to align those manifolds without any correspondence information. Our main idea is to exploit the powerful abstraction ability of encoder architecture. Specifically, we define multiple generators to model multiple manifolds, but in a particular way that their inverse maps can be commonly represented by a single smooth encoder. Then, the abstraction ability of the encoder enforces semantic similarities between the generators and gives a plausibly aligned embedding in the latent space. In experiments with MNIST, 3D-Chair, and UT-Zap50k datasets, we demonstrate the superiority of our model in learning the manifolds by FID scores and in aligning the manifolds by disentanglement scores. Furthermore, by virtue of the abstractive modeling, we show that our model can generate data from an untrained manifold, which is unique to our model.

1. INTRODUCTION:

Real-world datasets generally have multi-manifold structure. Smoothly varying features such as the size or the pose of an animal introduce smooth manifold structure, while the existence of discrete features such as species makes the structure not a single manifold but multiple disconnected manifolds (e.g., one manifold per species) (Khayatkhoei et al., 2018) . Importantly, these manifolds are correlated to each other in their structure since the transformation rules according to the size (stretching the foreground patch) or the pose (rotating the patches of body parts) are similar even for different species. Hence, our desire is to build a model that learns not only the multi-manifold structure but also the correlations between the manifolds.

As for the learning of manifold structure, generative adversarial networks (GANs) (Goodfellow et al., 2014) have been known for their remarkable performance. In particular, recently proposed GAN models have successfully demonstrated multi-manifold learning by using multiple generator networks (Khayatkhoei et al., 2018; Ghosh et al., 2017; Hoang et al., 2018) or giving a mixture density on the latent space (Xiao et al., 2018; Gurumurthy et al., 2017) . However, none of these models have taken the correlations between the manifolds into account.

On the other hand, a field of research named manifold alignment (Ham et al., 2003) has its primary concern in learning the correlations between the manifolds. In particular, unsupervised manifold alignment (Cui et al., 2014; Wang & Mahadevan, 2009) aims to learn plausible correspondences between the data points of different manifolds, without any ground-truth correspondence information is given. However, most existing methods focus only on finding the correspondences, not on learning the structure of the manifolds itself.

In this work, we propose a model that performs both of the tasks, which we call as manifold learning and aligning GAN (MLA-GAN). MLA-GAN is similar to the GANs with multiple generators (Khayatkhoei et al., 2018; Ghosh et al., 2017; Hoang et al., 2018) , but the generators are restricted such that their inverse maps can be represented by a single deep encoder. Due to the abstracting property of deep encoders, this restriction guides the inverse maps to learn semantically similar mappings, and the model achieves a semantically plausible manifold alignment. The main contributions of this work can be summarized as follows.

Figure 1 : The proposing MLA-GAN model is illustrated along with multi-manifold learning GANs (abbr. mmGANs) and unsupervised manifold alignment methods (abbr. UMA). In MLA-GAN, the latent features of cats and dogs manifolds are aligned. This is achieved by restricting the inverses of the generative maps (f (1) ) -1 , (f (2) ) -1 to be represented by a common encoder h E (see Eq. 1). In mmGANs, the latent features are not aligned as f (1) and f (2) do not share any structures. In UMA, the embedding coordinates are trained to be aligned, but as r

EQUATION

E are linear maps, they lose information about the structure of the manifolds.

• The MLA problem: Up to our knowledge, MLA-GAN is the first model that performs both multi-manifold learning and alignment with real-world datasets. It shows state-ofthe-art FID scores in data generation and plausible alignments with disentangled features.

• Generalization to an untrained manifold: Due to the abstracting nature of an encoder, the smooth features can be inferred even for data lying on an untrained manifold. Exploiting this, one can generate other data points lying on this manifold.

• Easily applicable: Due to the new regularizer we propose, the inverse maps used for restricting the generators do not need to be actually computed. This makes MLA-GAN easily applicable to any existing GAN architectures.

2. MANIFOLD LEARNING AND ALIGNMENT WITH GAN:



2.1. BACKGROUNDS:

Multi-Manifold Learning GANs Conventional GANs (Goodfellow et al., 2014) model data by transforming a latent-space distribution p(z) with a generating map f G : Z → X. If f G is smooth and injective 1 , the image of f G is a smooth manifold M ⊂ X, thus the data are modeled as they are lying on this manifold. To deal with multiple manifolds, recent extensions (Khayatkhoei et al., 2018; Ghosh et al., 2017; Hoang et al., 2018) employ multiple generating maps {f

(i) G } A i=1 where each f (i)

G accounts for the i-th manifold M (i) .

Unsupervised Manifold Alignment Most existing methods formulate the manifold alignment problem as finding embeddings of manifolds into a common low-dimensional space Y . This set of embeddings {r (i) E : M (i) → Y } i essentially defines an alignment of manifolds; data points that are mapped to the same point in Y correspond to each other. To achieve a plausible alignment, the embeddings are trained to minimize several loss terms guiding geometrical similarity (see Sec. 3).

2.2. MAIN IDEA:

To model multiple manifolds, we use multiple generating maps {f (i) G : Z → M (i) } i as with the aforementioned GAN models. Interestingly, the inverses2 of these maps {(f (i) G ) -1 : M (i) → Z} i can be regarded as a set of embeddings (where Y is replaced by Z), which means that the manifold alignment is automatically defined as we define the manifolds. However, this alignment is not at all semantically plausible unless properly regulated. Here, we regulate it by restricting the generating maps such that their inverses to be commonly represented by a single encoder h E : X → Z. Formally, we restrict the generating maps as follows:

EQUATION

Here, H is a certain set of functions (to be clearer soon), and h E being an element of H is an important constraint. Without this constraint, one can always find h E satisfying the above condition, for any sets of {f

(i) G } i , because h E is merely a union of functions {(f (i) G ) -1 } i extending the domain from i M (i) ⊂ X to X.

However, if H is, for example, a set of smooth functions and h E ∈ H, then {(f (i) G ) -1 } i will be restricted to be similar to each other and the alignment will become more plausible. In our model, H is defined as a set of functions represented by deep encoder networks. We explain the intuition behind this design in what follows.

Intuition Consider a dataset containing images of cats and dogs. It is not difficult to imagine a deep encoder network that takes an image from this dataset and outputs the size and pose regardless of the species. At least, we could have trained this encoder easily if the size and pose labels were given. Also, even if we train the encoder using the cat data only, we still expect it works fairly well for the dogs due to the powerful abstraction and generalization abilities of deep encoders. We want to exploit these abilities of deep encoders for aligning the manifolds. If we let H be a set of deep encoders, its element h E , once well-trained, will abstract the species away and outputs the size and pose. This gives a nice manifold alignment, where a cat and a dog images with the same size and pose correspond to each other.

Remaining Question Is H being a set of deep encoders enough to obtain the abstraction and generalization? An answer to this question is related to how deep networks generalize so well despite the large number of parameters they have. Several works conjecture that it is due to the SGD (Zhang et al., 2016; Brutzkus et al., 2018) , but this is not a fully developed theory yet. At least, we can assume that deep encoders learn simpler, generalization-friendly maps among others, based on various cases (e.g., VGG-Net (Simonyan & Zisserman, 2014) , Fast R-CNN (Girshick, 2015) ).

2.3. MLA-GAN:

What remains is to implement the generating maps with deep neural networks. The main challenge is to make the deep networks comply with the constraint given in Eq. 1. In this section, we explain how this challenge can be achieved in the proposing MLA-GAN model in detail. We first focus on a single linear layer and prove the following (detailed in Proposition 1):

If linear generating maps are defined with the same weight and the same tangential components of bias, their inverses can always be represented by a single linear encoder. Thus, the maps comply with Eq. 1.

Then, to realize the above, we introduce a new form of regularizer. After, we move on to the entire network and derive MLA-GAN by applying the same principles as the linear layer.

2.3.1. SINGLE LINEAR LAYER:

Consider a set of linear generating maps {f

(i) G : Z → M (i) ⊂ X} i ; each of the maps is defined as f (i) G (z) := U (i) z + a (i) , 𝑀 (#) 𝑀 (%) 𝒂 (𝟏) 𝒂 (𝟐) 𝑧 𝑈𝑧 𝒙 (𝟏) = 𝑼𝒛 + 𝒂 (𝟏) 𝒙 (𝟐) = 𝑼𝒛 + 𝒂 (𝟐) Generation 𝑎 ∥ 𝑥 3 (#) 𝑥 3 (%) 𝑥 ∥ 𝒙 (𝟐) 𝑊 # 𝑊 % 𝒙 (𝟏) 𝑀 (#) 𝑀 (%) 𝑂 (Origin) 𝑊 6 𝑥 ∥ = 𝑊 6 𝑥 (#) 𝑧 = 𝑊 6 𝑥 ∥ + 𝑏 𝑏 Encoding 𝑥 ∥ = 𝑊 6 𝑥 (%) 𝑈 # 6 𝑈 % 6 Figure 2:

The generators and the encoder of a linear layer are illustrated. Generation: A latent point z is transformed by the same weight U , then different biases a (1) and a (2) are added to generate x (1) and x (2) for different manifolds. The biases are regularized to have the same tangential component (to the column space of W or U ) a to comply with Eq. 1. Encoding: The points x (1) and x (2) lying on different manifolds are mapped to the same z if they have the same tangential components x . Note this becomes false if the biases are not regularized.

where

U (i) ∈ R d X ×d Z (d X > d Z

) and a (i) ∈ R d X are the weight and bias parameters respectively. We assume

U (i) is a full-rank matrix (rank-d Z ) such that f (i)

G is injective (see Footnote 1). Then, the inverse (f

EQUATION

where (U (i) ) + := (U (i) ) U (i) -1 (U (i) ) denotes the pseudo-inverse of U (i) . As noted in Eq. 1, our desire is to restrict the inverse maps {(f

(i) G

) -1 } i such that they can be represented by a single encoder h E : X → Z. One simple way to achieve this is to use the following proposition.

Proposition 1. If the linear generating maps {f (i) G (z)} i are restricted to have the same weight U and to have the same tangential components of bias a (i.e., a (i) = a + a

(i) ⊥ ), then their inverses {(f (i) G ) -1 } i can be represented by a single linear encoder h E (x) := W x + b where W = U (U U ) -1 and b = -W a .

Proof. Note the bias a (i) can be decomposed into the tangential component a (i) and the normal component a

(i)

⊥ to the column space of U . Substituting the condition a

(i) = a + a (i)

⊥ and the same weight U , Eq. 2 is written as

(f (i) G ) -1 = U + x -U + (a + a (i) ⊥ ) = U + x -U + a = W x + b.

Bias Regularizer When implementing Proposition 1, making {U (i) } i the same is as trivial as setting the same weight U for all f (i)

G , but making {a (i) } i the same is not trivial since the tangential direction keeps changing while training. One solution would be to use a regularizer minimizing the sum of the variance: trace(cov(a (i) )). However, computing this term is intractable due to the

inversion (U U ) -1 inside of a (i) = U (U U ) -1 U a (i) .

Theorem 1. The following inequality holds:

trace cov(U a (i) ) ≥ 1 d z H({λ k } dz k=1 )trace cov(a (i) )

where {λ k } dz k=1 denotes the eigenvalues of U U and H(•) denotes harmonic mean.

Proof. See Appendix B.

As the harmonic mean in Theorem 1 is constant from the perspective of a (i) , we can minimize the original term by minimizing the upper bound instead. With an additional log function to match the scale due to the dimensionality, we propose the upper bound as a regularizer to make a (i) the same:

Regularizer: R bias = log trace cov(U a (i) ) .

(3)

2.3.2. ENTIRE NETWORK:

A single linear layer is, of course, insufficient as linear generators can only model the linear manifolds, and a linear encoder can align the manifolds only linearly by removing the normal components. So, we implement the generating maps with deep networks to model complex nonlinear manifolds. As usual, the deep networks are constructed by interweaving linear layers and nonlinear activations, and optionally batch-normalization layers. But, each linear layer here implements the conditions of Proposition 1. That means, for the l-th linear layer of each deep generating map f

(i)

G , their weights are set as the same, denoted as U l , and their biases {a

(i)

l } i are regularized to have the same tangential components using Eq. 3.

Note that the above set of deep generating maps comply with Eq. 1. As all the linear layers, nonlinear activations (we use LeakyReLU; see Appendix C), and batch-norm layers are injective, the deep generating maps have inverses. These deep inverse maps can be represented by a single deep encoder, as the inverses of all the component-linear maps can be represented by single encoders.

Training As above, let us denote the weight of the l-th linear layer as U l and the biases as {a

(i) l } A i=1

. Then, we can express the data generating distribution as an ancestral sampling:

x ∼ f (i) G z; {U l , a (i) l } L l=1 where z ∼ p(z), i ∼ π i .

Here, π i stands for the probability of selecting the i-th bias. This probability could be also learned using the method proposed in Khayatkhoei et al. (2018) , but it is beyond our scope and we fix it as 1/A. Now, denoting the real data distribution as p R and the fake distribution that the above sampling presents as p G , we define our GAN losses as:

MLA-GAN Loss: L G = -E x∼p G [D(x)] + λ L l=1 log trace cov(U l a (i) l ) L D = E x∼p G [D(x)] -E x∼p R [D(x)]

where L G and L D are the generator and the discriminator losses respectively and λ is a regularization weight. We use Wasserstein GAN (WGAN) Arjovsky et al. (2017) so the discriminator D(x) is restricted to a k-Lipschitz function.

Encoding Due to the bias regularizer, we do not need to concretize the encoder h E by inverting f (i) G during training. But, when encoding is needed, we can obtain the latent codes and biases by minimizing the distance along with the bias regularizer as follows (as suggested in Ma et al. (2018) ).

EQUATION

Importantly, one can encode an image that is considerably different from the trained images using this method. By virtue of the abstraction ability of the encoder, the encoded latent code is actually semantically plausible. Moreover, as the encoded biases fully describe the generating map, one can virtually generate any data lying on the same manifold. This is unique to our model and to be further discussed in Sec. 4.3. 2018)) is the simplest one, but it cannot align the manifolds as the generators are independent to each other without any regulations. Giving a mixture density for the latent space prior (Xiao et al., 2018; Gurumurthy et al., 2017) is another method; although this method uses a single generating map, it hardly aligns the manifolds since the manifolds are modeled by different regions in the latent space where the mappings are effectively different from each other. The last method (InfoGAN (Chen et al., 2016) ) introduces discrete latent variables and concatenate them with the original continuous latent variables. Here, the discrete information is slowly blended in, layer by layer, and modulates the generating map. As the overall structure is shared among the manifolds to some degree, this method does align the manifolds; but as it is not explicitly guided, the performance is not as good as MLA-GAN (Table 1 ).

Unsupervised Manifold Alignment The existing methods (Cui et al., 2014; Wang & Mahadevan, 2009) find a correspondence between the points of different manifolds by learning embeddings, with consideration for geometrical similarity. As the geometry is captured by adjacency matrices, one shortcoming is that the alignment can be done only in a point-to-point manner. Moreover, as the embeddings are modeled as linear functions, it is inevitable to lose information about the manifolds.

Deep generative models involving an encoder Utilization of an encoder is a notable feature of MLA-GAN, but there are several other deep generative models using encoders, such as adversarial auto-encoders (Makhzani et al., 2015) , variational auto-encoders (Kingma & Welling, 2013; Higgins et al., 2016) and BiGAN (Donahue et al., 2016) . However, there is a large difference: The encoders of these models capture all the necessary information in order to reconstruct the data, while the encoder of MLA-GAN removes the discrete information in the latent space to abstract the manifolds.

4. EXPERIMENTS:

Datasets We experiment on MNIST (Lecun et al., 1998) Model Architecture We use DCGAN (Radford et al., 2015) -like model architectures for all the datasets (see Appendix C for the complete information). For each of the linear layers in the generator, the number of biases A is set as 10 (MNIST), 20 (3D-Chair), and 4 (UT-Zap50k). Although our multi-biased linear layer can be applied to both fully-connected and transposed-convolution layers, we apply it only to the former. This is sufficient for our purpose since discrete features rarely exist for such small-sized kernel patches. In the discriminator, we use spectral normalization (Miyato et al., 2018) to achieve the k-Lipschitz condition for WGAN. For training and encoding, Adam (Kingma & Ba, 2014) is used with the defaults except for the learning rate, 0.0002.

Figure 3 : Images generated from the trained MLA-GANs. Each i-th row presents samples from the i-th manifold (only 8 out of 20 are shown for 3D-Chair) and each column presents the samples generated from the same latent code z, which is randomly sampled from p(z). See also Appx. J.

4.1. MULTI-MANIFOLD LEARNING:

MLA-GAN shows great performance in multi-manifold learning and alignment, both qualitatively and quantitatively. Looking at Fig. 3 row-wise, we can see that MLA-GAN learns distinct manifolds well, where each manifold accounts for a different discrete class: e.g., different digits; rolling vs. armchairs; boots vs. flat shoes. Column-wise, we can see that the smooth features are well aligned among the manifolds: e.g., stroke weight, the slant of numbers; viewing angle of chairs; colors of shoes. Note that these alignments are qualitatively much better than that of the existing alignment methods given in Fig. D .2. For quantitative evaluation, we compute FID scores (Heusel et al., 2017) , which is widely used to measure the diversity and quality of the generated image samples, reflecting the manifold learning performance overall. Table 1 shows that our model gives better or comparable scores than others.

4.2. MANIFOLD ALIGNMENT AND DISENTANGLED FEATURES:

To further investigate the manifold alignment performance, we examine how much the learned latent features are disentangled. We take a few images from the dataset and manually change one of the smooth features that corresponds to a known transformation (e.g., sheer transform). Then, we encode these images to the latent codes using our model, analyze the principal direction of the change, and compute the linearity of the change as the disentanglement score (see Appendix E). Fig. 4 shows that the learned smooth features are well disentangled along the principal changing directions. Table 1 shows that our model gets better scores than other models, but sometimes not as good as β-VAE. However, it should be noted that the FID scores of β-VAE is significantly worse.

4.3. STYLE TRANSFER:

We demonstrate a style-transfer between images using our model by matching the latent features (Fig. 5 ). Specifically, we encode the source and target images, then use the inferred biases of the source and the inferred latent codes of the target to generate the style-transferred image. Interestingly, MLA-GAN is even able to style-transfer the images that are off the trained distribution (see the red boxes). This is an exclusive property of MLA-GAN due to the encoder. If the encoder generalizes well (which apparently does), it can recognize the smooth features of an image even if the image has noises such as added frames or color modification. Once the smooth features are recognized, the model knows how to transform this image smoothly, similar to the original data, but on this new manifold involving noises (as described in Eq. 4 and below).

5. CONCLUSION:

We proposed MLA-GAN that performs multi-manifold learning and alignment. To our knowledge, MLA-GAN is the first model that performs both of the tasks for real-world datasets, and it mostly showed superior performance. Unique to MLA-GAN is that it utilizes the abstraction ability of an encoder, from which it showed its potential to infer the untrained manifolds in the style-transfer experiment. If it is trained with larger datasets such as ImageNet in the future, we expect MLA-GAN would become a more versatile tool that can infer the manifold structure of general images. 

E DISENTANGLEMENT SCORE:

To compute the disentanglement score, we first take 500 images from the dataset and manually change one of the smooth features that corresponds to a known transformation. For example, we change the slant of the MNIST digits by taking a sheer transform. With 11 different degrees of the transformation, we obtain 5500 transformed images in total. We encode these images to obtain the corresponding latent codes and subtract the mean for each group of the images (originates from the same image) to align all the latent codes. Then, we conduct Principal Component Analysis (PCA) to obtain the principal direction and the spectrum of variations of the latent codes. If the latent features are well disentangled, the dimensionality of the variation should be close to one. To quantify how much it is close to one, we compute the ratio of the first eigenvalue to the second eigenvalue of the PCA covariance, and set it as the disentanglement score.

F EFFECT OF THE NUMBER OF GENERATORS, A:

To -regardless of the different λ values (other models are omitted for better readability; see Table 1 for the omitted ones). It can be also seen that the scores are not very sensitive to the different choices of λ's; this is beneficial in that one may choose any reasonable value for λ when training the model with a new dataset.

Here, MADGAN-like is a DMWGAN model, but has a similar parametrization to the MADGAN (Ghosh et al., 2017) : The parameters of the first three layers (from the top) are shared for all the generators. In contrast, MLA-GAN shares the parameters in all the layers except the biases in the fully-connected layers. Thus, in a sense, one can say that MLA-GAN shares only the last few layers, whereas MADGAN-like shares only the first few layers (of course, there is another difference due to the regularizers). Although the both models have the shared structures among the generators, MLA-GAN performs much better in both of the scores as seen in Figure G .4. This suggests that a simple parameter sharing is not enough to obtain a good performance, and the bias regularizer is indeed required. 

H EFFECT OF THE REGULARIZATION WEIGHT, µ, IN ENCODING:

To investigate the effect of the regularization weight, µ, in encoding (Eq. 4), we take a trained model and encode an image from the train set using different µ values. Then, plugging in the encoded (estimated) biases, {a l } L l=1 , we randomly generate the samples from this new manifold and compare the qualities for different µ's.

From Figure H.5 (b) , we can see that the quality of the encoding tends to improve as µ gets smaller. This might seem opposite to what we expect, as weaker regularization gives better results. However, looking closer, we can see that the features like the slant and the stroke are more aligned with a 

:

Although these conditions are not guaranteed for general neural networks, they are approximately met or observed in practice(Shao et al.,

2017). Additionally, the support of p(z) needs to be a completely separable Hausdorff space, but this condition is met for the usual choices of p(z) (e.g., unimodal Gaussian).

With the same assumption as Footnote 1.

Here, we also train on RGB-MNIST dataset: a simple extension of MNIST by coloring with (R, G, or B).

Figures and Tables:

Figure 4: Disentangled features. Images are arranged the same as Fig. 3, except the columns show linear changes in the latent space along the first eigenvector of the disentanglement score analysis (see Sec. 4.2). Slant, width (MNIST), height and brightness (3D-Chair) components are shown.

Figure 5: Style transfer results 3 . In each source image set, the fourth and the fifth images are derived from the third image by adding frames, rectangles or changing colors. The derived source images are clearly off the trained distribution, but their style-transfer results are good and consistent with the original style-transfer results (highlighted in red boxes).

Figure D.2: The manifold alignment results of Wang & Mahadevan (2009) for MNIST dataset.

investigate the effect of the number of generators (or biases), A, we train our model on MNIST with different A values then compute the FID and the disentanglement scores (Fig. F.3). It can be seen that our model performs consistently better than the baseline, WGAN, regardless of the different A values.

Figure F.3: The FID scores (right axis, the smaller the better) and the disentanglement scores (left axis, the larger the better) of MLA-GAN with varying A are shown for MNIST dataset. The dashed lines show the mean scores of the baseline model (WGAN).

Figure G.4: The FID scores (the smaller the better) and the disentanglement scores (the larger the better) of MLA-GAN with varying λ are shown for MNIST dataset. Note the other models are positioned in the center (shaded in gray) to be visually comparable with the best-performing MLA-GAN model (λ = 0.05).

Figure J.8: MNIST image samples generated from the trained models

Figure J.9: 3D-Chair image samples generated from the trained models

FID (smaller is better) and Disentanglement (larger is better) scores are shown. We compare WGAN(Arjovsky et al., 2017), DMWGAN(Khayatkhoei et al., 2018), β-VAE(Higgins et al., 2016), InfoGAN(Chen et al., 2016)  with our model. The mean and std. values are computed from 10 (MNIST) and 5 (3D-Chair) replicated experiments.

Table content: <table><tr><td/><td/><td>WGAN</td><td>DMWGAN</td><td>β-VAE</td><td>InfoGAN</td><td colspan="2">MLA-GAN (Ours) MLA-GAN, λ = 0</td></tr><tr><td>FID</td><td>MNIST</td><td>10.13 ± 3.16</td><td colspan="2">5.41 ± 0.34 58.43 ± 0.23</td><td>12.17 ± 1.30</td><td>5.69 ± 0.89</td><td>15.74 ± 10.00</td></tr><tr><td/><td>3D-Chair</td><td colspan="4">125.32 ± 1.16 184.5 ± 31.5 217.12 ± 0.55 187.94 ± 9.51</td><td>125.27 ± 4.34</td><td>128.44 ± 7.06</td></tr><tr><td/><td>MNIST (slant)</td><td>1.62 ± 0.41</td><td>1.08 ± 0.04</td><td>5.04 ± 1.19</td><td>1.24 ± 0.15</td><td>2.15 ± 0.17</td><td>1.76 ± 0.35</td></tr><tr><td>Disent.</td><td>MNIST (width)</td><td>1.68 ± 0.49</td><td>1.11 ± 0.06</td><td>5.63 ± 0.75</td><td>1.18 ± 0.11</td><td>2.93 ± 0.60</td><td>2.75 ± 0.67</td></tr><tr><td/><td>3D-Chair (height)</td><td>2.14 ± 0.20</td><td>1.14 ± 0.05</td><td>8.10 ± 0.20</td><td>1.41 ± 0.34</td><td>3.27 ± 1.73</td><td>2.76 ± 0.31</td></tr><tr><td/><td>3D-Chair (bright.)</td><td>3.53 ± 0.80</td><td>1.20 ± 0.14</td><td>3.96 ± 0.20</td><td>3.02 ± 0.88</td><td>4.45 ± 0.66</td><td>4.24 ± 0.54</td></tr><tr><td colspan="3">3 RELATED WORK</td><td/><td/><td/><td/><td/></tr></table>

Peer review (iclr2020):
The paper suggests performing simultaneous manifold learning and alignment by multiple generators that share common weight matrices and a constructed inverse map that is instantiated by a single encoder. The method utilizes a special regularizer to guide the training. It has been empirically well tested on a multi-manifold learning task, manifold alignment, feature disentanglement, and style transfer. Overall, this is an interesting idea with a motivated approach, however, I would like several points to be addressed before I could increase a score. 1. It seems that the method makes direct use of the number of classes in the datasets used. How would it fare compared to other models when the number of manifolds is not known (e.g. CelebA dataset)? 2. In MADGAN (Ghosh et al., 2017) generators share first layers which possibly makes them not independent as claimed in the paper, thus it is worth checking if MADGAN exhibits any kind of manifold alignment and could be a baseline for disentanglement with multiple generators. 3. There are hyperparameters lambda and mu for the regularizers in the model. It would be helpful to study their effect of different values on the training and encoding. 4. Is there a reason for DMWGAN/InfoGAN scores being omitted in Table 1? Minor remark - there are a number of typos in the text.