Manifold Learning and Alignment with Generative Adversarial Networks
Authors: Jiseob Kim
Seungjae Jung
Hyundo Lee
Byoung-Tak Zhang
Venue:  (2020)
Keywords: Generative Adversarial Networks
Manifold Learning
Manifold Alignment

Abstract: We present a generative adversarial network (GAN) that conducts manifold learning and alignment (MLA): A task to learn the multi-manifold structure underlying data and to align those manifolds without any correspondence information. Our main idea is to exploit the powerful abstraction ability of encoder architecture. Specifically, we define multiple generators to model multiple manifolds, but in a particular way that their inverse maps can be commonly represented by a single smooth encoder. Then, the abstraction ability of the encoder enforces semantic similarities between the generators and gives a plausibly aligned embedding in the latent space. In experiments with MNIST, 3D-Chair, and UT-Zap50k datasets, we demonstrate the superiority of our model in learning the manifolds by FID scores and in aligning the manifolds by disentanglement scores. Furthermore, by virtue of the abstractive modeling, we show that our model can generate data from an untrained manifold, which is unique to our model.

Under review as a conference paper at ICLR 2020
MANIFOLD LEARNING AND ALIGNMENT WITH GENERATIVE ADVERSARIAL NETWORKS
Anonymous authors Paper under double-bling review
ABSTRACT
We present generative adversarial network (GAN) that conducts manifold learn- ing and alignment (MLA): A task to learn the multi-manifold structure underlying data and to align those manifolds without any correspondence information. Our main idea is to exploit the powerful abstraction ability of encoder architecture. Specifically, define multiple generators model multiple manifolds, but in a particular way that their inverse maps can be commonly represented by single smooth encoder. Then, the abstraction ability of the encoder enforces semantic similarities between the generators and gives plausibly aligned embedding in the latent space. In experiments with MNIST, 3D-Chair, and UT-Zap50K datasets, we demonstrate the superiority of our model in learning the manifolds by FID scores and in aligning the manifolds by disentanglement scores. Furthermore, by virtue of the abstractive modeling, we show that our model can generate data from an untrained manifold, which is unique to our model.
INTRODUCTION
Real-world datasets generally have multi manifilm structure. Smoothly varying features such as the size or the pose of an animal introduce smooth manifold structure, while the existence of dis- crete features such as species makes the structure not a single manifold but multiple disconnected manifolds (e.g.. one manifold per species) (Khayatkhoei et al.) 2018). Importantly, these manifolds are correlated each other in their structure since the transformation rules according to the size (stretching the foreground patch) or the pose (rotating the patches of body parts) similar for different species. Hence, our desire to build a model that learns not only the multi-manifold structure but also the correlations between the manifolds.
As for the learning of manifold structure, generative adversarial networks (GANs) (Goodfellow et al. 2014) have been known for their remarkable performance. In particular, recently proposed GAN models have successfully demonstrated multi-manifold learning by using multiple generator networks (Khayatkhoei et al. 2018;; Ghosh et al.] 2017 Hoang al. 2018) or giving mixture density on the latent space (Xiao et al.) [018]] Gurumurthy et al. (017)) However, none of these models have taken the correlations between the manifolds into account.
On the other hand, a field of research named manifold alignment (Ham et al. 2003) has its primary concern in learning the correlations between the manifolds. In particular, unsupervised manifold alignment (Cui et al. 2014] Wang & Mahadevan] 2009) aims to learn plausible correspondences between the data points different manifolds, without any ground- -ruth correspondence informa- tion is given. However, most existing methods focus only on finding the correspondences, not on learning the structure of the manifolds itself.
In this work, we propose a model that performs both of the tasks, which we call as manifold learn- ing and aligning GAN (MLA-GAN), MLA-GAN is similar to the GANs with multiple generators (Khayathhoei et al. 2018 Ghosh et al. 2017 Hoang et al. 2018), but the generators are restricted such that their inverse maps can be represented by a single deep encoder. Due to the abstracting property of deep encoders, this restriction guides the inverse maps to learn semantically similar mappings, and the model achieves a semantically plausible manifold aligmment. The main contribu- tions of this work can be summarized as follows.

---

Under review as a conference paper at ICLR 2020
Latent Space
Latent Space
Embedding Space
Powe
Pose
Size
22)
(
f
(
fG
Fg
(1)
E
2
fG
Pose
Pose
Posc
Size
Size
Size
MLA-GAN (Ours)
Multi-Manifold Learning GANs
Unsupervised Manifold Alignment
Figure 1: The proposing MLA-GAN model is illustrated along with multi-manifold learning GANs (abbr. mmGANs) and unsupervised manifold alignment methods (abbr. UMA). In MLA-GAN, the latent features of cats and dogs manifolds are aligned. This achieved by restricting the inverses of the generative maps (f())=1 (f(2))-1 to be represented by common encoder hg (see Eq. I In mmGANs, the latent features not aligned f(1) and f(2) do not share any structures. In UMA, the embedding coordinates are trained to be aligned, but as r TF are linear maps, they lose information about the structure of the manifolds.
The MLA problem: Up our knowledge, MLA-GAN the first model that performs both multi-manifold learning and alignment with real-world datasets. It shows state-of- the-art FID scores in data generation and plausible alignments with disentangled features.
Generalization to an untrained manifold: Due to the abstracting nature of an encoder, the smooth features be inferred even for data lying this, one can generate other data points lying on this manifold.
untrained manifold. Exploiting
Easily applicable: Due to the new regularizer propose, the inverse maps used for stricting the generators not need to be actually computed. This makes MLA-GAN easily applicable any existing GAN architectures.
2 MANIFOLD LEARNING AND ALIGNMENT WITH GAN
2.1 BACKGROUNDS
Multi-Maniffod Learning GANs Conventional GANs (Goodfellow et al. 2014) model data by
transforming latent-space distribution p(z) with a generating map fG Z X. fo is smooth and injective"] the image of fc is smooth manifold M C X, thus the data are modeled as they are lying on this manifold. To deal with multiple manifolds, recent extensions (Khayatkhoei et al.)
2018 Ghosh et al. 2017 Hoang et al. 2018) employ multiple generating maps { } where each f accounts for the i-th manifold M(),
Unsupervised Manifold Alignment Most existing methods formulate the manifold alignment problem finding embeddings of manifolds into common now-dimensional space Y. This set of embeddings {r() (t) - Y}; essentially defines an alignment of manifolds; data points that mapped to the same point in Y correspond to each other. To achieve plausible alignment, the embeddings are trained to minimize several loss terms guiding geometrical similarity (see Sec. 3
Although these conditions are not guaranteed for general neural networks, they are approximately met or
observed in practice (Shao al. 2017). Additionally, the support of p(2) needs to be completely separable Hausdorff space, but this condition met for the usual choices of p(z) (c.g., unimodal Gaussian).
2

---

Under review as a conference paper at ICLR 2020
2.2
MAIN IDEA
To model multiple manifolds, we use multiple generating maps Ue 2 - M j. as with the afforementioned GAN models. Interestingly, the inverse of these maps {( M Z}: can be regarded as a set of embeddings (where Y is replaced by Z), which means that the manifold alignment is automatically defined as we define the manifolds. However, this aligmment is not at all semantically plausible unless properly regulated. Here, we regulate it by restricting the generating maps such that their inverses to be commonly represented by a single encoder hE X - Z. Formally, restrict the generating maps as follows:
(fG h A set of functions (Z - M00 CX) s.t.
(1) Here, H is a certain set of functions (to be clearer soon), and hg being an element of H. is an important constraint. Without this constraint, one can always find lgg satisfying the above condition, for any sets of (JG)) J., because lgg is merely a union of functions {(T6)) )-), extending the domain from U, M C X to X. However, if H is, for example, set of smooth functions and hg H, then {(G)) (1/6)-1) ) will be restricted to be similar to each other and the alignment will become more plausible. In our model, H. is defined as a set of functions represented by deep encoder networks. We explain the intuition behind this design in what follows.
ahg E H. satisfying (z) = hg(t) Vi, E MO.
Consider a dataset containing images of cats and dogs. is not difficult to imagine a deep encoder network that takes an image from this dataset and outputs the size and pose regardless of the species. At least, we could have trained this encoder easily if the size and pose labels were given. Also, even if we train the encoder using the cat data only, we still expect it works fairly well for the dogs due to the powerful abstraction and generalization abilities of deep encoders. We want to exploit these abilities of deep encoders for aligning the manifolds. If we let 7 be set of deep encoders, its element hg, once well-trained, will abstract the species away and outputs the size and pose. This gives nice manifold alignment, where cat and dog images with the same size and pose correspond to each other.
Intuition
Remaining Question Is H being a set of deep encoders enough obtain the abstraction and generalization? An answer to this question is related to how deep networks generalize so well despite the large number of parameters they have. Several works conjecture that is due to the SGD (Zhang et al. [016)] Brutzkus et al., 2018). but this is not fully developed theory yet. At least, we can assume that deep encoders learn simpler, generalization-friendi maps among others, based on various cases (e.g., VGG-Net (Simonyan & Zisserman 2014), Fast R-CNN (Girshick) 2015)).
2.3 MLA-GAN
What remains is to implement the generating maps with deep neural networks. The main challenge is to make the deep networks comply with the constraint given in Eq. T In this section, we explain how this challenge can be achieved in the proposing MLA-GAN model in detail. We first focus on single linear layer and prove the following (detailed in Proposition] T
If linear generating maps are defined with the same weight and the same tangential components of bias, their inverses can always be represented by a single linear encoder. Thus, the maps comply with Eq.
Then, to realize the above, we introduce a new form of regularizer. After, we move on to the entire network and derive MLA-GAN by applying the same principles as the linear layer.
2.3.1 SINGLE LINEAR LAYER
Consider a set of linear generating maps (5(1) 7 + M ( C Che
each of the maps is defined as
fa (2) ((1)) + (,)
With the same assumption as Footnote
3

---

(4) Under review as a conference paper at ICLR 2020
W
W'
WT x
WT x
Generation
Encoding
MUN
M(1)
x00 a(1)
M
(2)
W
x2) Uz al2)
W1
(Origin)
TJ
Figure 2: The generators and the encoder of linear layer are illustrated. Generation: A latent point 2 is transformed by the same weight U, then different biases a') and (²2) are added to generate (1)) and 22(2) for different manifolds. The biases are regularized to have the same tangential component (to the column space of W or ) ajj to comply with Eq. Encoding: The points x()) and x
lying different manifolds are mapped the same if they have the same tangential components zil- Note this becomes false if the biases are not regularized.
where U(1) E IRdx xdz (dx > dz) and (0) E Rdx are the weight and bias parameters respectively. We assume U(9 is full-rank matrix (rank-dz) such that ((() is injective (see Footnote 1]. Then, the inverse (f())-1, (0 + : can be derived as (fl)) (W6)) (2)10 - (210-a")) al))
where (w+0))+ = ((U())TTT1))) (U()T denotes the pseudo-invers of U1)) As noted in Eq. our desire is to restrict the inverse maps { -'); such that they can be represented by a single encoder hg
Z. One simple way to achieve this is use the following proposition.
Proposition 1. If the linear generating maps (fG ( (z)}s are restricted to have the same weight
U and to have the same tangential components of bias on (i.e, al ag + a. then their
inverses (8) )-)) can be represented by single linear encoder hx(x) = W x + b where W = UU U)- and h = -T atI-
(6) and the normal Proof. Note the bias (i) can be decomposed into the tangential component aj (3)
component aj to the column space of UT. Substituting the condition ( an a 1 and the same weight U, Eq. 2 is written as
(1())1 = U+ - U (a +
-++ U
= W x +b
Bias Regularizer When implementing Proposition making (17)); the same is as trivial as setting the same weight U for all f(() but making {a ) the same not trivial since the tangential direction keeps changing while training. One solution would be to use regularizer minimizing the sum of the variance: trace(cov(a (())) However, computing this term is intractable due to the inversion (U- inside of a (6) -CUT))-'0" T.
Theorem 1. The following inequality holds:
trace (cov(U) (oo))
HH(()t (Ax Idz trace trrce (cor(a()))
dz
4

---

Under review as a conference paper at ICLR 2020
where {Ak 1d: denotes the eigenvalues of U U and H() denotes harmonic mean.
Proof. See Appendix B
Oij As the harmonic mean in Theorem is constant from the perspective of (t) we can minimize the
original term by minimizing the upper bound instead. With an additional log function to match the scale due the dimensionality, propose the upper bound regularizer make dj
(5) the same:
Rosas = log ttaaee (cov(UT a())))
Regularizer:
2.3.2 ENTIRE NETWORK
A single linear layer is, of course, insufficient linear generators can only model the linear man- ifolds, and linear encoder can align the manifolds only linearly by removing the normal com- ponents. So, we implement the generating maps with deep networks to model complex nonlinear manifolds. As usual, the deep networks are constructed by interweaving linear layers and nonlinear activations, and optionally norccahoomalizttionn layers. But, each linear layer here implements the conditions of Proposition That means, for the (-th linear layer of each deep generating map f((
(8) are regularized have the their weights are the same, denoted U/, and their biases {at same tangential components using Eq.
Note that the above set of deep generating maps comply with Eq. As all the linear layers, nonlinear activations (we use LeakyReLU; see Appendix and batch-norm layers are injective, the deep generating maps have inverses. These deep inverse maps be represented by single deep encoder, as the inverses of all the component- maps can be represented by single encoders.
Training As above, let us denote the weight of the I-th linear layer as Uj and the biases as (a) ) Then, express the data generating distribution ancestral sampling:
P()) x (z (U 1=.1) where ~ p(>), i i~ n;
Here, my stands for the probability of selecting the i-th bias. This probability could be also learned using the method proposed Khayatkhoei et al. (2018). but it is beyond our scope and we fix it as 1/A. Now, denoting the real data distribution as PR and the fake distribution that the above sampling presents as Pc. define our GAN losses as:
S = -Egmna [D(x)) + log (trace (ov(UUT a, d("))
MLA-GAN LOss:
( = -C==pepp [D(x)] - Ex-pa [D(x)]
where La and LD the generator and the discriminator losses respectively and is regulariza- tion weight. We use Wasserstein GAN (WGAN) Arjovsky et al. (2017) so the discriminator ((x)
restricted k-Lipschitz function.
Encoding Due the bias regularizer, we do not need concretize the encoder 4g by inverting
(0) during training. But, when encoding is needed, we can obtain the latent codes and biases by minimizing the distance along with the bias regularizer follows (as suggested in Ma al. (018)) )
2, (1r)/== = arg min H.
|x - Ja (z: (a))))))
+ Σ log (IUT at UT a (.. (4)
1=1
Importantly, one can encode image that is considerably different from the trained images using this method. By virtue of the abstraction ability of the encoder, the encoded latent code is actually semantically plausible. Moreover, the encoded biases fully describe the generating map, virtually generate any data lying on the same manifold. This is unique to our model and to further discussed in Sec.
5

---

Under review as a conference paper at ICLR 2020
Table FID (smaller is better) and Disentangiement (larger is better) scores are shown. We compare WGAN (Arjovsky et al. 2017), DMWGAN (Khayatkhoei et al. 2018), B-VAE (Higgins et al.) 2.16 InfoGAN (Chen et 2016) with our model. The mean and std. values are computed from 10 (MNIST) and 5 (3D-Chair) replicated experiments.
WGAN
DMWGAN
B-VAE
InfoGAN
MLA-GAN (Ours)
MLA-GAN, À
MINSTT
10.13
5.11
38.1320222
12.17 1.30
5.69 = 0.89
15.74 = 10.00
FID
3D-Chair
125.32 # 1.16
1813 = 31.5
217.12
187.91 = 9.51
125.27 1.34
128.44 = 7.06
MNIST (slant)
1.62 1 0.41
1.08 0.004
5.04
1.19
1.24 0.15
2.15 1 0.17
1,76 1. 0.35
Disent.
MNIST (width)
1.68 1. 0.49
0.006
5.63 1. 0.75
1.18 1. 0.11
2.93 1 0.60
2.75 1. 0.67
3D Chair (height)
2.14 1. 0.20
1.14 0.055
8.10 0.220
1.41 1 0.34
3.27 1.73
2.76 1. 0.31
3D-Chair (bright.)
3.53 + 0.50
0.14
3.96
3.02 0.88
4.45 0.66
4.24 + 0.54
3 RELATED WORK
Multi-Manifole Learning GANs There exist roughly three different methods to handle multiple
manifolds with GANs (see also Appendix A Using multiple generators (DMWGAN (Khayatkhoei et al. 2018); Ghosh et al.] (2017): Hoang et al. (2018) is the simplest one, but it cannot align the manifolds as the generators are independent to each other without any regulations. Giving a mixture density for the latent space prior (Xiao et al. 2018) Gurumurthy et al. (017) is another method: although this method uses a single generating map, hardly aligns the manifolds since the manifolds are modeled by different regions in the latent space where the mappings are effectively different from each other. The last method (InfoGAN (Chen et al., 2016)) introduces discrete latent variables and concatenate them with the original continuous latent variables. Here, the discrete information is slowly blended in, layer by layer, and modulates the generating map. As the overall structure is shared among the manifolds some degree, this method does align the manifolds; but as it is not explicitly guided, the performance is not as good as MIA-GAN (Table I
Unsupervised Manifold Alignment The existing methods (Cui et al. 2014 Wang & Mahadevan 2009) find a correspondence between the points of different manifolds by learning embeddings, with consideration for geometrical similarity. As the geometry captured by adjacency matrices, one shortcoming is that the alignment can be done only in point-to-point manner. Moreover, as the embeddings are modeled as linear functions, is inevitable to lose information about the manifolds.
Deep generative models involving encoder Utilization of an encoder is notable feature of MLA-GAN, but there are several other deep generative models using encoders, such as adversarial auto-encoders (Makhzani et al. 2015), variational auto-encoders (Kingma & Welling) 2013 Higgins et al. 2016) and BiGAN (Donahue et al. 2016). However, there is a large difference: The encoders of these models capture all the necessary information in order reconstruct the data, while the encoder of MLA- removes the discrete information in the latent space to abstract the manifolds.
4 EXPERIMENTS
Datasets We experiment on MNIST (eecn et al. 1998). 3D-Chair (Aubry et al. 2014) and UT- Zap50k YYu & Grauman 2014) image datasets. 3D-Chair contains 1393 distinct chairs rendered from 62 different viewing angles (total 86,366 images); in experiments, only front-looking 44,576 images are used and rescaled to 64x64 grayscale images. UT-Zap50k contains images of 4 different types of shoes (total 50,025 images): the images are rescaled to 32x32.
Model Architecture We use DCGAN (Radford et al., 2015)-like model architectures for all the datasets (see Appendix for the complete information). For each of the linear layers in the gener- ator, the number of biases is set as 10 (MNIST), 20 (3D-Chair), and 4 (UT-Zap50k). Although our multi-biased linear layer can be applied to both fully-connected and transposed-convolutie lay- ers, we apply it only to the former. This is sufficient for our purpose since discrete features rarely spectral normalization (Miy-
exist for such small-sized kernel patches. In the discriminator, ato et al.] (0188) to achieve the k-Lisschitz condition for WGAN. For training and encoding, Adam (Kingma & Ba 2014) is used with the defaults except for the learning rate, 0.0002.

---

Under review as a conference paper at ICLR 2020
/
H
9
9
9
9
9
9
3
9
9
9
4
4
4
4
4
4
4
9
4
4
2
2
2
_
7
2
3
2
2
2
5
3
3
5
5
3
6
5
5
4p
8
8
8
8
8
8
6
R
8
8
0
O
O
O
O
O
Q
0
0
3
3
3
3
5
3
5
S
8
5
6
6
6
6
6
6
6
6
6
6
4
7
7
7
7
7
7
7
7
J
Figure 3: Images generated from the trained MLA-GANs. Each i-th row presents samples from the i-th manifold (only 8 of 20 are shown for 3D-Chair) and each column presents the samples generated from the same latent code z, which is randomly sampled from p(z). See also Appx. ]
4.1 MUL.TI-MANIFOI LEARNING
MLA-GAN shows great performance in multi-manifole learning and alignment, both qualitatively and quantitatively. Looking at Fig. rowwwwsee we can see that MLA-GAN learns distinct manifolds well, where each manifold accounts for different discrete class: e.g., different digits; rolling armchairs; boots vs. flat shoes. Column-wise, we can see that the smooth features are well aligned among the manifolds: e.g., stroke weight, the slant of numbers; viewing angle of chairs; colors of shoes. Note that these alignments are qualitatively much better than that of the existing alignment methods given in Fig. D.2 For quantitative evaluation, we compute FID scores (Heusel al. 2017), which is widely used to measure the diversity and quality of the generated image samples, reflecting the manifold learning performance overall. Table shows that our model gives better or comparable scores than others.
4.2 MANIFOLD ALIGNMENT AND DISENTANGLED FEATURES
To further investigate the manifold alignment performance, we examine how much the learned latent features are disentangled. We take few images from the dataset and manually change one of the smooth features that corresponds to known transformation (e.g., sheer transform). Then, we encode these images the latent codes using change, and compute the linearity of the change as the disentanglement score (see Appendix E}. Fig. 4 shows that the learned smooth features are well disentangled along the principal changing directions. T shows that our model gets better scores than other models, but sometimes not as good as B-VE. However, it should be noted that the FID scores of 3-VAE significantly worse.
model, analyze the principal direction of the
4.3 STYLE TRANSFER
We demonstrate style-transfer between images using model by matching the latent features (Fig. 5). Specifically, we encode the source and target images, then use the inferred biases of the source and the inferred latent codes of the target generate the ttyle-transferred image.

---

Under review as a conference paper at ICLR 2020
9
+
A
*
4
0
6
6
6
R
7
7
7
*
9
9
9
9
9 4
4
4
4
Mp My Pr
14
Hp
0
0
0 6
6
6
7
Figure 4: Disentangled features. Images are arranged the same as Fig. except the columns show linear changes in the latent space along the first eigenvector of the disentanglement score analysis (see Sec.
Slant, width (MNIST), height and brightness (3D-Chair) components are shown.
Targets
0
O
5
3
9
3
3
3
0
0
3
3
7
7
7
7
7
7
7
7
Sources
9
9
9
7
3
9
9
7
7
7
9 9
9
9.
9
9
9
9
4
6
9
4
4
7
9
3
3
3
3
3
2
O
O
0
0
0
C
0
O 0 0
Figure 5: Style transfer results In each source image set, the fourth and the fifth images are derived from the third image by adding frames, rectangles or changing colors. The derived source images are clearly off the trained distribution, but their style-transfer results are good and consistent with the original style-transfer results (highlighted in red boxes).
Interestingly, MLA-GAN is even able to style-transfed the images that are off the trained distribution (see the red boxes). This an exclusive property of MLA-GAN due the encoder. the encoder generalizes well (which apparently does), it can recognize the smooth features of an image even the image has noises such as added frames or color modification. Once the smooth features are recognized, the model knows how to transform this image smoothly, similar to the original data, but on this new manifold involving noises (as described in Eq. and below).
CONCLUSION
We proposed MLA-GAN that performs multi-manifold learning and alignment. To our knowledge, MLA-GAN is the first model that performs both of the tasks for real-world datasets, and it mostly showed superior performance. Unique to MLA-GAN is that it utilizes the abstraction ability of encoder, from which it showed its potential infer the untrained manifolds in the style-transfer experiment. If is trained with larger datasets such as ImageNet in the future, we expect MLA-GAN would become versatile tool that can infer the manifold structure of general images.
Here, also train RGB-MNIST dataset: simple extension of MNIST by coloring with (R, G. B).
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. January 2017.
Mathieu Aubry, Daniel Maturana, Alexei A. Efros, Bryan C. Russell, and Josef Sivic. Seeing 3D
Chairs: Exemplar Part-Based 2D-3D Alignment Using a Large Dataset of CAD Models. In 2014 IEEE Conference Computer Vision and Pattern Recognition, pp. 3762-3769, Columbus, OH, USA, June 2014. IEEE. ISBN 978-14479--11885 doi: 10. 00.1009/VVPR.204444887
Alon Brutzkus, Amir Globerson, Eran Malach, and Shai Shalev-Shwartz. SGD LEARNS OVER- PARAMETERIZED NETWORKS THAT PROVABLY GENERALIZE ON LINEARLY SEPARA- BLE DATA. pp. 17, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets. In D. D. Lee, Sugiyama, U. V. Luxburg. I. Guyon, and R. Garnett (eds.), Advances Neural Information Processing Systems 29, pp. 2172-2180. Curran Associates, Inc., 2016.
Zhen Cui, Hong Chang, Shiguang Shan, and Xilin Chen. Generalized Unsupervised Manifold Align- ment. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 2429-2437. Curran Associates, Inc., 2014.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell.
Adversarial
Feature Learning.
arXiv: 1605.09782 [es, stat), May 2016.
Arnab Ghosh, Viveka Kulharia, Vinay Namboodiri, Philip H. S. Torr, and Puneet K. Dokania. Multi- Agent Diverse Generative Adversarial Networks. arXiv:1704.02900 [cs, stat], April 2017.
Ross Girshick. Fast R- R-CNN. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1440-1448, 2015.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural In- formation Processing Systems 27, pp. 2672-2680. Curran Associates, Inc., 2014.
Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, and Venkatesh Babu. DeLiGAN: Gener- ative Adversarial Networks for Diverse and Limited Data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 166-174, 2017.
Ji Hun Ham, Daniel Lee, and Lawrence Saul. Learning High Dimensional Correspondences from Low Dimensional Manifolds. Departmental Pupers (ESE), August 2003.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter: GANs Trained by Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 6626-6637. Curran Asso- ciates, Inc., 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. Beta- Vae: Learning Basic Visual Concepts with a Constrained Variational Framework. November 2016.
Quan Hoang, Tu Dinh Nguyen, Trung Le, and Dinh Phung. MGAN: Training Generative Adversar- ial Nets with Multiple Generators. February 2018.
Mahyar Khayatkhoei, Maneesh K. Singh, and Ahmed Elgammal, Disconnected Manifold Learning for Generative Adversarial Networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianche and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 7343-7351 Curran Associates, Inc., 2018.
Diederik P. Kingma and Jimmy Ba. Adam: Method for Stochastic Optimization. December 2014.

---

Under review as a conference paper at ICLR 2020
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. arXiv:1312.6114 [es, stat], December 2013.
Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, pp. 2278-2324, 1998.
Fangehang Ma, Ulas Ayaz, and Sertac Karaman, Invertibility of Convolutional Generative Networks from Partial Measurements. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances Neural Information Processing Systems 31, pp. 9628- 9637. Curran Associates, Inc., 2018.
Alireza Makhzani, Jonathon Shiens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial Autoencoders. (oXiv:1511.0564/ [cs], November 2015.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral Normalization for Generative Adversarial Networks. arXiv:1802.0599) [cs, stat]. February 2018.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. November 2015.
Hang Shao, Abhishek Kumar, and P. Thomas Fletcher. The Riemannian Geometry of Deep Gener- ative Models. arXiv:1711.08014 [cs, stat), November 2017.
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv: 1556 [cs], September 2014.
Chang Wang and Sridhar Mahadevan. Manifold Alignment without Correspondence. In Twenty- First International Joint Conference on Artificial Intelligence, June 2009.
Chang Xiao, Peilin Zhong, and Changxi Zheng. BourGAN:
Generative Networks with Metric Em- beddings. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchh and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 2269-2280. Curran Asso- ciates, Inc., 2018
Aron Yu and Kristen Grauman. Fine-Grained Visual Comparisons with Local Learning. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 192-199, Columbus, OH, USA, June 2014. IEEE. ISBN 978-1-4799-511888 doi: 10.1 10.1199CCVRR.204..22.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv: 1,03530 [cs], November 2016.
10

---

Under review as a conference paper at ICLR 2020
Appendix
A
COMPARISON OF VARIOUS GAN MODELS
(
fc
(x)
000
000
100 00
000
GAN
DeLiGAN
InfoGAN
DMWGAN
MLA-GAN (Ours)
Figure A.1: Various GAN models trained on 2-manifold data are shown schematically. Conven- tional GAN (Goodfellow al., 2014) learns highly nonlinear mapping to compensate for the disconnected region with small density. But it is still not perfect compared to the true disconnec- tion (red-dashed). DeLiGAN (Gurumurthy et al.) 2017) and InfoGAN (Chen et al. 2016) use non-connected topology in the latent space. DMWGAN (Khayatkhoei al. (208)) uses distinct generating maps for each manifold. MLA-GAN (Ours) is similar but regulated that the inverses of the generating maps be represented common single encoder.
B THE PROOF OF THEOREM
Theorem 1. The following inequality holds
trace (cov(UT a (())) 2 d.
H(AA 1k= trace (cov(a) (cov(a()) (f)))
where {A are the eigenvalues of U and H() denotes harmonic mean.
Proof. Note that
trace (cov(a())) trace
()) a!))
U(UTT)--U -a)(0 -a -000000--
trace
A A-1
A
A-1 UT (a - -) a)TT(UTT))
trace
(cov(UT ())
trace
< trace (Cov(UT a(1)) trace (IU U)-1),
where the second and the fourth lines use the definition of the covariance, third line is obtained from the cyclic property of trace and the last line obtained from the Cauchy-Schward inequality of the positive semi-definite matrices. Thus,
trace (cov(UT aa)))) 2 trace (cov(a())) /trace ((U ((UT
1 - (12)) (Var(a()))
where (Aa) are the eigenvalues of TTU and H(-) denotes the harmonic mean.

---

Under review as a conference paper at ICLR 2020
C MODEL ARCHITECTURE AND EXPERIMENTING ENVIRONMENTS
We used machines with one NVIDIA Titan Xp for the training and the inference of all the models.
C.I MNIST
We use 10 distinct decoding biases in the model. the training, we set the regularization
weight À - 0.05 and use the Adam optimizer with learning rate 0.0002. In the encoding, we use the Adam optimizer with learning rate 0.1, and the set the regularization weight μ = 0.1.
Table C.I: MLA-GAN architecture used for MNIST dataset
Generator
Discriminator
Input(8)
Input(1,28,28)
Full(1024), BN, LReLU((..2)
Conv(c=64, k=4, s=2, p=1), BN, LReLU((..2)
Full(6272), BN, LReLU(0.2)
Conv(c=128, k=4, s=2, p=1), BN, LReLU(00.2
ReshapeTo((228.77)
ReshapeTo(62722
s=2, p=1), BN, LReLU(0..2
Full(1024), BN, LReLU((..2)
ConvTis(c=32, k=4, s=2, p=1), BN, LReLU(0.22
Full(1)
ConvTrs(c=1, k=3, s=1, p=1), Tanh
C.1.1 NOTES ON THE OTHER COMPARED MODELS
Overall, we match the architecture of other models with our model for fair comparison. Some differences to note are:
We used 10 generators. Each generator has the same architecture as ours except the number of features or the channels are divided by 4, match the number of trainable parameters. Note that is the suggested number from the original paper.
DMWGAN:
3-VAE: We used Bernoulli likelihood.
InfoGAN:
Latent dimensions consist of I discrete variable (10 categories), continuous variables and noise variables.
C.2 3D-CHAIR
We use A 20 distinct decoding biases in the model. In the training, set the regularization
weight À = 0.05 and use the Adam optimizer with learning rate 0.0002. In the encoding, we use the Adam optimizer with learning rate 0.1, and the the regularization weight μ 0.1.
12

---

Under review as a conference paper at ICLR 2020
Table C.2: MLA-GAN architecture used for 3D-Chair dataset.
Generator
Discrininator
Input(10)
Input( (1,64,64)
Full(256), BN, (ReLU((02)
Conv(c=64, k=4, s=2, p=1), BN, LReLU(0.2)
Full(8192), BN, LReLU((.2)
Conv(c=128, k=4, s=2, p=l), BN, LReLU(0.22
1eshapeTTo(128.8.88
Conv(c=128, k=4, s=2, p=1), BN, (Re
Conv Mon/TTs(c=64, k=4 s=2, p=1), BN, LReLU((0.2)
ReshapeToo(119)
ConvTis(c=32, k=4, s=2, p=1), BN, (Rel.U(0.2))
Full(1024), BN, LReI.U(0.2)
ConvTTis(==66, k=4, s=2, p=1), BN, LRe.UU(0.2)
Full(1)
ConvTrs(c=1, k=3, s=1, p=1), Tanh
C.2.1 NOTES ON THE OTHER COMPARED MODELS
We used 20 generators. Each generator has the same architecture as ours except the BatchNorms removed and the number of features the channels are divided by 4, to match the number of trainable parameters. Note that 4 is the suggested number from the original paper. Note that this was the best setting among what we have tried (division number 2; ones with the BatchNorms).
DMWGAN:
3-VAE: We used Bernoulli likelihood.
InfoGAN:
Latent dimensions consist of discrete variables (20 categories). continuous variable and 10 noise variables.
C.3 UT-ZAP50K
We use A = 4 distinct decoding biases in the model. For the regularization weight in the training, we start with A = 5e 6 then raise to À 5e 4 after 300 epochs.
Table C.3: MLA-GAN architecture used for UT-ZZap50k dataset.
Generator
Discriminator
Input(8)
Input(3.32,32)
Full(512), BN, LReLU(0.2)
Conv(c=128, k=4, s=2, p=1), BN, LReLU(0.22)
Full(1024), BN, (R.2)((0.2)
Conv(c=256, k=4, s=2, p=1), 1), BN, LReLU(0.22
Full(8192), BN, I.ReI.((0.))
Conv(c=512, k=4, s=2, p=1), BN, (ReI.U(0.2)
Reshape/1o(512.444)
Reshape'To(819)
Conv Com-Tns(==256,k44,, s=2. p=1), 1), BN, LReLU(0.22
Full(1024), BN, LReLU(0.22
ConvThs(c=128, k=4, s=2, p=1), BN, (ReILU(0.2)
Full(512), BN, LReLU((0.2)
ConvTrs(c=64, k-4, s=2, p=1), BN, LReLU(0.22)
Full(1)
ConvTrs(c=3, k=3, s=1, p=1), Tanh
D MANIFOLD ALIGNMENT RESULTS OF THE EXISTING METHODS
We show the manifold alignment results of Wang & Mahadevan 2009) for MNIST dataset.
13

---

Under review as a conference paper at ICLR 2020
8
8
8
8
8
8
8
8
8
7
8
8
4
8
8
8
8
8
8
8
3
3
3
5
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
6
6
6
6
6
6
6
6
6
C
6
6
6
6
6
6
6
6
6
6 6
0
O
o
0
4
O
O
0
O
O
O
O
a
O
o
O
0
0
6
6
6
6
6
6
6
6
6
6
6
6
6
6
4
6 8
8
8
8
8
8
8
8
8
8
8
6
8
8
8
8
8
Figure D.2: The manifold alignment results of Wang & Mahadevan (2009) for MNIST dataset.
E DISENTANGLEMENT SCORE
To compute the disentanglement score, we first take 500 images from the dataset and manually change one of the smooth features that corresponds known transformation. For example, we change the slant of the MNIST digits by taking sheer transform. With 11 different degrees of the transformation, we obtain 5500 transformed images total. We encode these images to obtain the corresponding latent codes and subtract the mean for each group of the images (originates from the same image) to align all the latent codes. Then, we conduct Principal Component Analysis (PCA) to obtain the principal direction and the spectrum of variations of the latent codes. If the latent features are well disentangled, the dimensionality of the variation should be close to one. To quantify how much it is close one, compute the ratio of the first eigenvalue the second eigenvalue of the PCA covariance, and set it as the disentanglement score.
F EFFECT OF THE NUMBER OF GENERATORS, A
To investigate the effect of the number of generators (or biases), A, we train our model on MNIST with different values then compute the FIID and the disentanglement scores (Fig. F3.. It can be seen that our model performs consistently better than the bascline, WGAN, regardless of the different values.
10
Disent. Score
FID Score
2
Sheer Width
FID
10
25
Number of Biases (A)
Figure F.3: The FID scores (right axis, the smaller the better) and the disentanglement scores (left axis, the larger the better) MLA-GAN with varying A are shown for MNIST dataset. The dashed lines show the mean scores of the baseline model (WGAN).
G EFFECT OF THE REGULARIZATION WEIGHT, X, TRAINING
To investigate the effect of the regularization weight, X, train model on MNIST with different values. It can seen that our model performs consistently better than the other compared models MLA-GAN with regularization (A = 0), DMWGAN, MADGAN-like (see the next paragraph)
14

---

Under review as a conference paper at ICLR 2020
regardless of the different À values (other models omitted for better readability; Table for the omitted ones). It can be also seen that the scores are not very sensitive to the different choices of A A's; this is beneficial in that one may choose any reasonable value for À when training the model with a new dataset.
Here, MADGAN- is DMWGAN model, but has similar parametrization the MADGAN (Goshh et al.) 2017): The parameters of the first three layers (from the top) are shared for all the generators. In contrast, MLA-GAN shares the parameters in all the layers except the biases in the fully-connected layers. Thus, in a sense, one can say that MLA-GAN shares only the last few layers, whereas MADGAN-like shares only the first few layers (of course, there is another difference due to the regularizers). Although the both models have the shared structures among the generators, MLA-GAN performs much better in both of the scores as seen in Figure G.4 This suggests that simple parameter sharing is not enough to obtain a good performance, and the bias regularizer is indeed required.
18
16
14
FID Score
12
10
4-
MLA-GAN (Ours)
DMWGGAN
MLA-GAN 12 0)
0.0005
0.005
0.05
0.5
5.0
Bias Regularization Weight (A)
(a) FID Scores
Disent. Score (Slant)
4.0
E
NLA-GAN
DMWGAN MADGAN-ike
Disent. Score (Width)
MLA-GAN IA
MADGAN- like
3.5
4
3.0
w
2.0
1.5
1.0
0.0
0.0005
0.005
0.05
0.5
5.0
0.0005
0.005
0.05
0.5
5.0
Bias Regularization Weight (A)
Blas Regularization Weight (A)
(b) Disentanglement Score (Slant)
(c) Disentanglement Score (Width)
Figure G.4: The FID scores (the smaller the better) and the disentanglement scores (the larger the better) of MLA-GAN with varying À are shown for MNIST dataset. Note the other models are positioned in the center (shaded in gray) be visually comparable with the best-performing MLA- GAN model (A = 0.05).
H EFFECT OF THE REGULARIZATION WEIGHT, J1, IN ENCODING
To investigate the effect of the regularization weight, μ,, in encoding (Eq. take trained model and encode an image from the train set using different values. Then, plugging in the encoded (estimated) biases, (ai)/,, randomly generate the samples from this new manifold and compare the qualities for different ytss.
that the quality of the encoding tends to improve gets smaller. This might seem opposite to what we expect, as weaker regularization gives better results. However, looking closer, can that the features like the slant and the stroke more aligned with
From Figure H.5 (b),
15

---

Under review as a conference paper at ICLR 2020
stronger regularization of μ 0.05, when comparing with the other manifolds in the bottom pane. Thus, a trade-off exists here between the image quality the samples and the alignment of the manifold to the others. Note chose μ 0.05 in all the other experiments.
3
3
3
(a) Left: An image taken from the training dataset. Middle: The same im- age with rectangle noise. Right: Regenerated image from the estimated biases, {as jun and the estimated latent code from Eq.
3
2
8
3
3
3
3
3
3
5
3
2
?
3
3
3
3
3
3
3
3
7
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
5
5
5
5
6
O
5
5
5
5
6
6
6
6
6
6
6
6
6
6
2
7
2
2
2
2
2
2
2
2
3
3
5
5
3
3
3
3
3
5
9
7
9
7
9
3
3
7
9
9
(b) Top pane: Generated samples from the estimated biases, {ar] The biases are estimated with different ju's 'x,, (5.0, 0.5, 0.05, 0.005, 0.0005, 0.0), from the top to the bottom. Bottom pane: Generated samples from the
originally leamned biases, {fa (6) ) (only out of 10 are shown).
Note the images in the same column have the same latent value 2.
Figure H.5: Effect of the regularization weight y in encoding is shown with MNIST-trained MI.A- GAN model.
I EFFECT OF THE BIAS REGULARIZER
To examine the effectiveness of our bias regularizer, we visualize the raw values of biases {a Je, (2)
and their Itangental component (U, )1r (see Fig. 1.6 1.7 In all figures, we see that the biases are diverse, but their tangential components are well aligned due to the bias regularizer (left). On the contrary, without the regularizer, the tangential components are not aligned (right).
J SAMPLES GENERATED FROM VARIOUS MODELS
16

---

Under review as a conference paper at ICLR 2020
(
2.0
15
Values
Values
-15
-2.0
200
400
600
800
1000
200
400
600
800
1000
Dimensions
Dimensions
U 0
(
U a
10:
Values
Values
005
-10
-20
-1.5
-40
Dimensions
Dimensions
a
a
Values
Values
1000
2000
3000
4000
5000
6000
1000
2000
3000
4000
5000
6000
Dimensions
Dimensions
UTa (
UTa
Values
Values
-10
200
400
600
800
1000
200
400
600
800
1000
Dimensions
Dimensions
Figure 1.6: Biases o and their (aande--aangettiall components U. a of the MLA-GAN models, trained on MNIST. Individual curve indicates each i-th bias. Left Parameters of MLA-GAN Right Parameters of MLA-GAN without the regularizer (A 0). It can be that the regularizer makes the tangential components of the biases well aligned. 17

---

Under review as a conference paper at ICLR 2020
(1)
Values
Values
50
100
150
200
250
50
100
150
200
250
Dimensions
Dimensions
U a
UTa
0.1
Values
-0.1
Values
-022
-0.4
-05
Dimensions
Dimensions
a
Values
Values
2000
4000
6000
8000
2000
4000 Dimensions
6000
8000
Dimensions
UTa
UTa
(
150
100
Values
Values
-50
150
100
200
Dimensions
Dimensions
Figure 1.7: Biases di and their (oseudo-)Jtangentn component U,T dj of the MLA-GAN models, trained on 3D-Chair. Individual curve indicates each i-th bias. Left Parameters of MLA-GAN Right Parameters of MLA-GAN without the regularizer (A = 0). It can be seen that the regularizer makes the tangential components the biases well aligned.

---

Under review as a conference paper at ICLR 2020
6
6
6
6
6
6
6
6
6 6
5 5
5
$
5
8
S 3 5
2 2
2
2
3
2
2 2
4
4
4
4
9
4
4 0000000
4
4
9 0 r 3 2 23 6 / /
O 00000000
0 6
0
0
0 9 2 6
0
0
5
5
5
5
5
5
3 9
9 2 6
9
9
4 2 6
99999
9
9
9
9
9
9
2
8
8
4
4
4
4
4
4
6
6
6
6
6 6 7 7
8888 8
8
8
8
8 7
8
/
2
2
7
2
/
77777771
7
7
1
1
1
(
/
/
1
/
1
1
3
3
3
8
3
3
5
3
3 7
3 7
3
3
3
3
3
3 3
3
3
3
7
7
7
7
9
9
7
7
(a) MLA-GAN
(b) MLA-GAN 0)
4
4
1
1
9
/
6
9
7
3
2
8
4
6 0
3
3
2
3
2
2
2
3
3
2
8
4
7
6
6
0
9
/
4
4
/
/
4
/
4
6
9
4
7
6
3
7
8 1 5
7
9
7
8
7
7
7
7
8
7
5
7
2
6
9
3
0
9
9
8
3
8
3
3
5
3
5 0 2
9
8
8
9
6
6
7
4
0
6
O
O
C
0
0
0
0
0
0
9
J
9
5
9
8
4
7 3
a
2
9
9
9
4
9
9
7
9
2
9
6
2
5
0
7 4
9 6
6
5
6
6
5
6
6
6
6 5 9
6
4
3
9
2
0
/
7
5
9
5
7
/
3
5
4
7
4
7 6
0 1
8
8
6
7
4
7
8
7
2
/ 3
6
6
2
7
(c) DMWGAN
(d) WGAN
9
3
2
4
4
q
3
6
O 05000
5
5
0
5
6
4
9
<
2
8
7
3
7
4
9
4
4
/
9
6
6
I
1
9
6
2
Q
9
8
8
3
5
2
2
/
3
3
3
3
2
/
3
3 7 6
1
3
O
9
2
7
4
7
4 1 4
8
7
7
7
7
7
7
7
7
7
0
2
4
9
2
2
9
1
G
6
6
6
6
6
6
5
6
4
9
9
0
3
7
3
7
2
3
2
2
3
Z
2
2
2
3
8
3
9
2
a
6
6
2 9 2
9
9
4
4
9
4
9
9
9
9
Q
3
9
5
6
8
2
9
9
9 8 8
4 0
9
4
I
9 6
4
9
8
2
2
7
7
0
2
8
0
3
0
6
0
0
7
5
7
3
7
8
5
6
/
0
8
8
8
8
8
8
8
8
8
(e) 3-VAE
InfoGAN
Figure J.8: MNIST image samples generated from the trained models
19

---

Under review as a conference paper at ICLR 2020
1
%
H
3
A
4
WGAN
(b) 3-VAE
(c) InfoGAN
Figure J.9: 3D-Chair image samples generated from the trained models
20

---

Under review as a conference paper at ICLR 2020
3
J
H
Figure J.10: 3D-Chair image samples generated from the trained MLA-GAN
21

---

Under review as a conference paper at ICLR 2020
M
R
R
4
4
4
R
A
4
R
Figure J.11: 3D-Chair image samples generated from MLA-GAN () 0)
22

---

Under review as a conference paper at ICLR 2020
Figure J.12: 3D-Chair image samples generated from the trained DMWGAN
23


Reviews:
[{"review_id": "S1lO7zYNtr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "A paper's idea is to train joint Wasserstein GAN for k datasets given in R^n (corresponding, e.g., to different classes of objects), together with manifold alignment of k manifolds. \n\nThe idea is to align points whose corresponding latent points are the same. This induces a natural constraint: k encoder functions (inverse of generators) should be consistent with each other. This is done by adding a regularization term. A paper demonstrates a clear motivation and a working solution for the problem. Experiments are convincing.\n\nThe only question is that the regularization term forces something stronger than just consistency of encoders. It seems, the requirement that \"all  tangential components of biases are the same\" means that k images of latent space (under k generator functions) are either coincide or non-intersecting. This is much stronger than just consistency, which is the weakest part of the approach.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "BJlU_xM-5S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "EDIT: Updated score to weak Accept in lieu of author's response.  See below for more details.\n\nThe authors propose a GAN architecture that aims to align the latent representations of the GAN with different interpretable degrees of freedom of the underlying data (e.g., size, pose).  While the text, motivation, and experiments are fairly clear, there are some spelling/grammar mistakes throughout, and the draft could use a solid pass for overall clarity.\n\nWhile the idea of using the log trace covariance as a regularizer for the manifold is certainly interesting, it seems fairly incremental upon previous work (i.e., DMWGAN).  Even modulo the work being incremental, I still have concerns regarding the comparison to baselines/overall impact, and thus I suggest a Weak Rejection.\n\nTable 1 seems to indicate that the author's proposed method is on par with or worse than every method compared against except for 3D chair (bright).  Additionally, the lack of comparison against DMWGAN for every task (except the first) is a bit concerning, considering its similarity to the proposed method.  If the authors could check DMWGAN's performance for all of their tasks and report it, I would be more likely to raise my score.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1evrRiTKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper suggests performing simultaneous manifold learning and alignment by multiple generators that share common weight matrices and a constructed inverse map that is instantiated by a single encoder. The method utilizes a special regularizer to guide the training. It has been empirically well tested on a multi-manifold learning task, manifold alignment, feature disentanglement, and style transfer.\nOverall, this is an interesting idea with a motivated approach, however, I would like several points to be addressed before I could increase a score.\n1. It seems that the method makes direct use of the number of classes in the datasets used. How would it fare compared to other models when the number of manifolds is not known (e.g. CelebA dataset)?\n2. In MADGAN (Ghosh et al., 2017) generators share first layers which possibly makes them not independent as claimed in the paper, thus it is worth checking if MADGAN exhibits any kind of manifold alignment and could be a baseline for disentanglement with multiple generators.\n3. There are hyperparameters \\lambda and \\mu for the regularizers in the model. It would be helpful to study their effect of different values on the training and encoding.\n4. Is there a reason for DMWGAN/InfoGAN scores being omitted in Table 1?\n\nMinor remark - there are a number of typos in the text.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a GAN architecture that aims to align the latent representations of the generator with different interpretable degrees of freedom of the underlying data (e.g., size, pose).

Reviewers found this paper well-motivated and the proposed method to be technically sound. However, they cast some doubts about the novelty of the approach, specifically with respect to DMWGAN and MADGAN. The AC shares these concerns and concludes that this paper will greatly benefit from an additional reviewing cycle that addresses the remaining concerns. 

Decision: Reject