Deep Quotient Manifold Modeling
Authors: Jiseob Kim
Seungjae Jung
Hyundo Lee
Byoung-Tak Zhang
Venue:  (2021)
Keywords: deep generative models
manifold learning

Abstract: One of the difficulties in modeling real-world data is their complex multi-manifold structure due to discrete features. In this paper, we propose quotient manifold modeling (QMM), a new data-modeling scheme that considers generic manifold structure independent of discrete features, thereby deriving efficiency in modeling and allowing generalization over untrained manifolds. QMM considers a deep encoder inducing an equivalence between manifolds; but we show it is sufficient to consider it only implicitly via a bias-regularizer we derive. This makes QMM easily applicable to existing models such as GANs and VAEs, and experiments show that these models not only present superior FID scores but also make good generalizations across different datasets. In particular, we demonstrate an MNIST model that synthesizes EMNIST alphabets.

Under review as a conference paper at ICLR 2021
DEEP QUOTIENT MANIFOLD MODELING
Anonymous authors Paper under double-bling blind review
ABSTRACT
One of the difficulties modeling real-world data is their complex multi-manifold structure due to discrete features. In this paper, we propose quotient manifold modeling (QMM), a new data-modeling scheme that considers generic manifold structure independent of discrete features, thereby deriving efficiency in modeling and allowing generalization over untrained manifolds. QMM considers a deep encoder inducing an equivalence between manifolds; but we show it is sufficient to consider it only implicitly via bias-regularizer we derive. This makes QMM easily applicable to existing models such GANs and VAEs, and experiments show that these models not only present superior FID scores but also make good generalizations across different datasets. In particular, we demonstrate MNIST model that synthesizes EMNIST alphabets.
INTRODUCTION
Real-world data are usually considered to involve multi-manifai structure by having discrete fea- tures as well continuous features; continuous features such size location induce smooth manifold structure in general, whereas discrete features such as digit-class or a new object in the background induce disconnections in the structure, making it single (Khayatkhoei et al. 2018). While this multiplicity makes modeling data a difficult problem, recently proposed deep generative models showed notable progresses by considering each manifold separately. Extending the conventional models by using multiple generators (Khayatkhoei et al.,
disjoint manifolds instead of
2018
Ghosh et al.] 2017;: Hoang et al., 2018), discrete latent variables (Chen |0.6 Dupont
2018 Jeong and Song (2019), or mixture densities (Gurumurthy et al. 2017 Xiao et al. 2018 Tom-
czak and Welling] 2018), they exhibit improved performances image generations and learning
high-level features.
There are, however, two additional properties little considered by these models. First, since discrete features are both common and combinatorial, there can be exponentially many manifolds that are not included in the dataset. For example, an image dataset of a cat playing around in a room would exhibit simple manifold strueture according to the locations of the cat, but there are also numerous other manifolds derivable from it via discrete variations- toy, turning on light their combinations-that Second, while the manifolds to model are numerous considering such variations, they usually have the same generic structure since the underlying continuous features remain the same; regardless of the chair, toy, or light, the manifold structures are equally due to the location of the cat.
as placing a new chair, displacing not included in the dataset (see Fig. D-
Considering these properties, desired is model that can handle a large number of resembling man- ifolds, but the aforementioned models show several inefficiencies. They need proportionally many generators or mixture components to model large number of manifolds; each of them requires much data, only to learn the manifolds having the same generic structure. Moreover, even if they are successfully trained, new discrete changes are very easy to made, yet they cannot generalize beyond the trained manifolds.
In this paper, we propose quotient manifold modeling (QMM) new generative modeling scheme that considers generic manifold structure independent of discrete features, thereby deriving effi- ciency in modeling and allowing generalization over untrained manifolds. QMM outwardly follows the
scheme (Khayatkhoei et al., 2018 Ghosh et al., 2017;; [Hoang et al.) 2018); but it involves a new regularizer that enforces encoder compatibiliry a condition that the inverse maps of the generators to be presented by single deep encoder. Since deep encoders usually exhibit

---

Under review as a conference paper at ICLR 2021
Encoding Map h
Ag (x)
Latent space,
-CCCat -Cat
Car Right
Cat Left
M -Ct w/ Chai
10 M()
IZ M00
Generative Maps
Data space, x
Data space, X
Figure 1: An illustration of quotient manifold modeling (QMM). Images of a cat moving around in room would form an 1-D manifold due to the location of the cat; but the structure can become multi-manifold by having different discrete features (a chair or vase in the background), The idea of QMM to consider the generic structure shared by the manifolds using an encoding map. The encoding map induces equivalence relation that can be curves). The quotient of the relation (or the orthogonal curve to the map) gives the manifold structure for untrained image (shown in purple, dotted).
contour map (green dotted
good generalizability, this condition not only makes a generic structure be shared among the gener- ators also makes it generalizable to untrained manifolds. In particular, it induces generalizable equivalence relation between data, and the manifold structure of out-of-sample data can be derived by taking the quotient of this relation, hence the name QMM.
Since the implementation of QMM is essentially adding regularizer, it can be casily applied to existing deep generative models such as generative adversarial networks (GANs; (Goodfellow al., 2014)), variational auto-encoders (VAEs; (Kingma and Welling, 2013)), and their extensions. We demonstrate that these QMM-applied models not only show better FID scores but also show good generalizations.
Our contributions can be summarized as follows:
We propose QMM, new generative modeling scheme that considers generic manifold structure, thereby allowing generalizations over untrained manifolds.
We derive regularizer enforeing encoder compatibility, an essential condition for QMM.
We show that GANs and VAEs implementing QMM show superior FID scores and gener- alize across different datasets.
BACKGROUND
2.1 MANIFOLD MODELING IN GANS AND VAES
While generative adversarial networks (GANs) (Goodfellow et al.) 2014) and variational auto- encoders (VAEs) (Kingma and Welling 2013) are two different strands of models, they have the same data-modeling scheme that leads to a manifold structure (though VAEs involve stochasticity). They model data E X transformation of low-dimensional latent code E Z via gen- crative (decoding) map fo Z + X, which makes every datum they consider lie on a subspace M (Z) C X. Since fa can be assumed be smooth and injective in practice (Shao et al. 2017). M accords with the mathematical definition a smooth manifold.
But to deal with multi-manifole data, these models need approximate disconnections in the struc- ture with low densities. This requires them to have a highly nonlinear fa. which is difficult to learn and often leads to either low-quality model mode collapse (Khayatkhoei al. 2018)
2

---

Under review as a conference paper at ICLR 2021
2.2
EXTENSIONS
To better model the multi-manifold structure, several studies proposed extended GAN and VAE models that consider each of the manifolds separately. According to which component is extended, the approaches can be broken down into the below three. While these approaches have advantages in dealing with multi-manifold data, they still show limiting performance in learning generic structure and do not allow generalization over untrained manifolds.
1) sonulii-eeeratrr---ae ((0) = f6 (x) Khayatkhoei et al. 2018 Ghosh et al.
2017 Hoang et al, 2018). In this approach, each manifold modeled by separate generator. The generators usually independent, but some models design them to share the weight parameters in a subset of the layers. This in part contributes to the learning of generic structure, but lacks theoretical grounds
and shows inferior performances (see Appendix F). 2) Mixture density (2())
fc((()). ) where
200) p()) /urumurthy al., 2017| Xiao et al. 2018 Tomczak and Welling (0018) In this approach, each manifold is modeled by a separate mode of the latent distribution. While the modes outwardly share the generator, the actual mappings are effectively different from each other they reside in different regions in Z. 3) Discrete lutent variables-z!"
Sa ([=; d]) (Chen et al. 2016 Duponi, 2018| Jeong and Song 2019). this approach, discrete random variables are explicitly defined and concatenated the continuous variable. Since discrete information is slowly blended in layer by layer, it can learn the generic structure to some degree, but not as clear (see Table TT.
3 QUOTIENT MANIFOLD MODELING (QMM)
QMM inherits the multi-generator scheme (Khayatkhoei et al. 2018 Ghosh et al. 2017 Hoang
jet al. 2018), but involves an additional regularizer enforcing the encoder compatibllit. Leaving the regularizer for the next section, generative maps. Then, we see how a plausible equivalence relation can be defined using deep encoder. Lastly, explain how new manifold can be obtained by taking the quotient of the relation.
first explain the role of this compatibility as binding the
3.1 ENCODER COMPATIBILITY
Definition 1. Let H. be a set of encoding maps (X- + 2) that can be represented by a deep encoder. We say that generative maps (1() 7 - (1) X k have encoder compatibility if there exists hg E H. satisfying (f(() (xx) (z(z) for all E M and i.
With this condition satisfied, the generative maps (f() h but share a single X < Z translation rule represented by the deep encoder h5 E H. However, this binding is meaningful only when H has certain property; otherwise, hg just an extension of functions ((f()) giving no useful signal to {fe (1)) h:-
(())
longer independent to each other
In practice, H. indeed involves an important property that its elements deep encoders have good generalizability. Having numerous parameters, deep encoders could overfit data, but in practice they find the smoothest function exhibiting generalizability. For example, if a deep encoder is trained on images illustrated in Fig. T to output the position of the cat, we expect it would work fairly well after we place vase dition. While this generalizing property is not fully understood to date (there are several compelling theories (Zhang et al. 2016 Brutzkus et al. 2018)). it has been consistently demonstrated by many deep recognition models (e.g.. VGG-Net (Simonyan and Zisserman 2014). Fast R-CNN (Girshick, 2015). In this regard, we assume its continuity and generalizability developing our model, and verify them later from the experiments.
light, generalizing over the discrete changes of the room
3.2 EQUIVALENCE RELATION AND QUOTIENT MANIFOLDS
Putting the generalizability of fg and the compatibility ( = hg together, we expect he to output the continuous features given data æ. Then, there is naturally induced equivalence relation
3

---

Under review as a conference paper at ICLR 2021
W xx
b
W"
Generation
Uz
-
Eacoding
(12)
MC
U2
Art1y
W
11 U.
(Ongin))
Figure 2: The generators and the encoder of a linear layer are illustrated. Generation: A latent point is transformed by the same weight U, then different biases a(!) and (²2) are added generate (x!) and z²(2) for different manifolds. The biases are regularized to have the same tangential component (to the column space of W or U alj- which makes the generators encoder-compatible (Def. [ Encoding: The points x²)) and (² lying different manifolds are mapped to the same if they have the same tangential components ziI- Note this becomes false if the biases are not regularized.
(called the kernel of hg) x(1) ~bs 22(2) = hg(x()) (z(2(2)), which effectively groups data having the same continuous features together regardless the manifolds they are lying on.
This can be seen more concretely in Fig. [ Assuming the model is well trained, it should have captured the crucial continuous feature the location of the cat-in Z. Due to the encoder com- patibility, such feature should be encoded in Z smoothly, meaning that hg outputs the location information of the cat in a consistent and generalizable way, if not exactly calibrated. Given this, images having the cat on the same location, regardless of the chair, will have the same hæ(r),, thus being equivalent under ~iE (ones on green dotted curve). Since hgg can generalize, the equiv- alence relation is consistent for data lying on an untrained manifold as well (ones with the vase, drawn in purple).
Now that the data lying on different manifolds are made equivalent under ~ig manifold structures are described as quotients of this relation. In implementations, taking the quotient is the same as taking the orthogonal directions the equivalence-relation contours. In Fig. [ we can see that (1) and (2), the manifolds that are included in the dataset, are already orthogonal to the contours formed from hg. When given an untrained image (shown in purple), we can obtain the new manifold just by following the orthogonal directions. It will be explained later that this manifold can be described by new generator, whose bias parameters are optimized for the manifold to pass through the given image (see Sec. 55.
BLAS-ALIGNING REGULARIZER
To implement the discussed scheme, the main issue is to make the generators have the encoder com- patibility. We first examine a simplified case where each generator is single-layered and derive that sufficient condition for this is that the biases of the generators sition [). To achieve the alignment, we introduce a bias-aligning regularizer, the main component of QMM. After, we explain how the regularizer can be extended to the multi-layer case.
aligned in certain way (Propo-
4.1 ENCODER COMPATIBILITY FOR SINGLE LINEAR LAYER
Consider set of linear generative maps (1(0) 7 + ( c X):; each of the maps is defined as (z) U02 2 + a where U+i) E Rdx xdz (dx > d2) and ( E Rdx are the weight and bias parameters respectively. We assume (1t) is full-rank matrix (rank-dz) such that f({ injective.
Then, the inverse (F(()) (() + Z can be derived as
(1(0))-= = (W+)) (2x11) (1))
4

---

Under review as a conference paper at ICLR 2021
denotes the pseudo-inverse of U To achieve the encoder compatibility (Def. 1 our desire is to restrict the inverse maps {( h such that they can be represented by a single encoder he : X + Z. One simple way to achieve this is to use the following proposition.
where (U = (( (U())T
Proposition 1. If the linear generating maps [z)): are restricted to have the same weight ((((()) U and to have the same tangential components of bias api then their inverses {(fG))-1) can be represented by a single linear encoder hx(#) = W + b, where W U) and b b- an-
(8) and L1 denote the tangential and the normal components of a(() (to the column Proof. Let "I
space of UT), respectively. Then, the restrictions can be expressed (1(i) U and (t)) a + +o for all i. Substituting these in Eq. 1
(G)) U --- (al + = Uta
WT 1 b.
4.2 ALAS-ALIG REGULARIZER FOR SINGLE LINEAR LAYER
When implementing Proposition 1 making (U(+); the same is as trivial as setting the weight U for all CG but making {ai
(i) 3. the same is nontrivial since the tangential direction keeps changing while training. One solution would be to use a regularizer minimizing the sum of the variance: trace(cov(ar) (a ())). However, computing this term is intractable due to the inversion (U (UT inside of a (1) = UU UU at.).
Theorem 1. The following inequality holds:
trace (cov(UTa())) Ta d()))² dz
(T(AA)E= (cov(a()))
where {A+)d= ,d denotes the eigenvalues of T and H() denotes harmonic mean.
Proof. See Appendix A.
(6) As the harmonic mean in Theorem is constant from the perspective of Olj we can minimize the original term by minimizing the upper bound instead. With an addition of log to match the scale due to the dimensionality of the layer, we propose this upper bound as regularizer make (i) the same:
BA-regularizer: RBA log (traee (cov(UT (ATUTa()))) d0)))
4.3 MULTI-LAYER NETWORK
The encoder compatibility for multi-layer networks can be enforced straightforwardly by applying Proposition [ all the linear layers. That means, for the 1-th linear layer of all the generators, their weights are shared, U,() = U. and their biases are regularized to have the same tangential components, a(!) dr, + a via Eq. 2 other layers- nonliner activation functions (we use LeakyRel.U) and natch-normalizztion layers-ard are simply set to be shared.
This design guarantees the inverses the generator networks to be represented by a single deep encoder (though we do not actually compute the inversion), inducing the encoder compatibility. Since all the layers are invertible, the entire networks are invertible; also, due to Eq. _ inverses of 1-th linear layers can be represented by a single linear layer (W(I)T x + B), and other layers can be trivially inverted and represented by the same layers.
5

---

Under review as a conference paper at ICLR 2021
Table FID (smaller is better) and Disentangiement (larger is better) scores are shown. We compare WGAN (Arjovsky et al. 2017), DMWGAN [Khayatkhoei et al. 2018), B-VAE (Higgins et al., 2016 InfoGAN (Chen et 2016) with our model. The mean and std. values are computed from 10 (MNIST) and 5 (3D-Chair) replicated experiments.
GANs
VAIs
WGAN
DMWGAN
IdfoAAN
Q WGAN (Ours)
WGAN,
A-0
A-VAE
QU VAE (Ours)
Q VAL 1--
FTD
MNIST
10.33 3.11
0.34
12.17 1.30
5.69 0.88
15.74 1 10000
58.43 0.23
41. 14.44
40.74_2.00
3D Chair
125.32
184.5 1 51.5
18799 1
125.27 4.34
128.44 7.0
217.12
211.30 7.42
210.05 4.26
Divetangle
MNIST (slent)
1.62+00.1
1.08 0.04
1.24 + 0.15
2.15 0.11
1,70 + 0.35
5.0
6.98 1.78
5.18
MNIST (width)
1,68 0:40
L111
1.18 0.1.1
2:93 0.60
2.75 + 0.67
5:63 1:8-:003 0.75
5.71
5.00
3D-Char (height)
2.34
1,14 0.05
1.41 = 0.34
3.27 = 1:78
2.76 = 0.31
8.30 0.20
8.88 +
6.99 =
3DSChair (wight.)
3.53++0.80
1,20 0.34
3.02 + 0.95
4.45 0.66
4.24 0.54
3.96 1.66=1..3 0.20
7:27 + 1.48
5,88 + 1.00
DEEP QUOTIENT GENERATIVE MODELS
Now that we described the QMM scheme, we explain how can be applied to the concrete models. We present its applications on Wasserstein GANs and 3-VAEs among others.
Generation As above, let us denote the weight of the 1-th linear layer as U2 and the biases as (6)
{a 1== Then, express the data generating distribution in the form of ancestral sampling:
(()) (z: (U HEI) where p<(2,
n.--
Here, Ty stands for the probability of selecting the i-th bias. This probability could be also learned using the method proposed Khayatkhoei et al. (2018), but it is beyond our scope and we fix it as 1/A. We denote the distribution due to this process PG-
Encoding (Deriving Quotient Manifolds) Due to the bias regularizer, we do not need con- cretize the encoder 4gg by actually inverting (() during training. But, when encoding is needed, we can obtain the latent codes and biases by minimizing the Euclidean distance (as suggested in Ma et al. (2018)) along with a similar bias regularization as follows.
| fa (z: (ar + Ps Σ log U|, U₁ aail
faille arg min
(3)
2,fa:
Q-WGAN Applying QMM to the Wasserstein GAN (WGAN; (Arjovsky et al. 2017 we can define the QMM-applied WGAN losses by simply adding the regularizer:
LG -- (Dxx)) + log (trace(cov(UTT of
1=1
where La and LD Ex-pa [D(x)] Ex-pa D(x) (pR being the real data distribution) are the regularization weight, and D(x)
generator and the discriminator losses, respectively, k-Lipschitz discriminator function.
Q-8-VAE We use multi-generatee version of 3- BAE, which use EM algorithm for training (de- tailed in Appendix C). Applying QMM can be similarly done by adding the BA-regularizer:
L = -= -=Eagg(ele)
Σ
(i) log Pa (a]=z) + DDx (a(=)a))l<(=)) + RoA((a)) ().))
EXPERIMENTS
Datasets We experiment MNIST (Lecun al. 1998), 3D-Chair (Aubry al. 2014) and UT-
Zap50k YY and Grauman 2014) image datasets. 3D-Chair contains 1393 distinct chairs rendered from 62 different viewing angles (total 86,366 images): in experiments, only front-looking 44,576 images are used and rescaled to 64x64 grayscale images. UT-Zap50k contains images of 4 different types of shoes (total 50,025 images); the images rescaled to 32x32.

---

Under review as a conference paper at ICLR 2021
4 9
/
I
/
7
7
7
7
7
7
7
7
7
9
9
9
4
4 7
9
4
4
9 0 7
0
O
O
0
C
O
O /' 0 5 6 6
8
8
8
8
8
8
6
8
A
M
0
0
O
6
O
O
O
Q
0
3
3
3
S
5
6
6
6
6
6
6
6
22
7
7
7
7
7
7
7
Figure 3: Images generated from the trained Q- V-Vs (first row) and Q-GANs. Each i-th row presents samples from the i-th manifold (only out 20 are shown for 3D-Chair) and each column presents the samples generated from the same latent code which is randomly sampled from p(z).
Model Architectures For GAN models, we use DCGAN (Radford et al. 2015)-like model archi- tectures for all the datasets (see Appendix B for the complete information). In the discriminator, we use spectral normalization (Miyato al. 2018) to achieve the k-Lipschitz condition. For VAE models, the architectures are generally the same as GANs, except that the batch-norms are not used.
We use 3 4 and ten EM steps. In both models, Adam (Kingma and Ba 2014) is used for training and encoding. with the default values except for the learning rate, 0.0002.
In the QMM-applied models, the number of biases, A, is set to 10 (MNIST), 20 (3D-Chair), and (UTT-ap55k)), respectively. Although our multi biased linear layer can be applied to both fully- connected and convolutional layers, apply it only to the former. This was sufficient for our pur- pose since disconnections rarely exist for such small-sized (thus low-dimensional) kernel patches.
6.1 BASIC MALIIOD LEARNING
QMM-applied models show great performance in multi-manifold learning with good alignment, both qualitatively and quantitatively. Looking at Fig. 3 row-wise, we can see that Q-WGAN and B-B- learn distinct manifolds well, where each manifold accounts for different discrete class: e.g., different digits; rolling vs. armchairs; boots vs. flat shoes. Column-wise, we can see that the continuous features are well aligned among the manifolds: e.g., stroke weight, the slant of numbers; viewing angle of chairs; colors of shoes. For quantitative evaluation, we compare FID score (Heusel 2017). widely used metric examine the diversity and quality of the generated image samples, reflecting the manifold learning performance overall. As seen in Table I our models give better or comparable scores than others.
6.2 DISENTANGLEMENT OF LATENT CODES
To further investigate the manifold alignment performance, we examine how much the learned latent features are disentangled. We take a few images from the dataset and manually change one of the smooth features that corresponds to known transformation (e.g., sheer transform). Then, encode these images to the latent codes using our model, analyze the principal direction of the change, and compute the linearity of the change the disentanglement score (see Appendix D). Fig. shows that the learned continuous features are well disentangled along the principal changing directions. Table T shows that our models get better scores than other models.

---

Under review as a conference paper at ICLR 2021
9
+
A
*
4
0
6
6
6
A
R
7
7
7
*
9
9
9
9
9
4
4
4
4 6
%
Mp My Pr
142
Hp
0
0
0
6
6
6
7
7
Figure 4: Disentangled features. Images are arranged the same as Fig. except the columns show linear changes in the latent space along the first eigenvector from the disentanglement analysis (see Slant, width (MNIST), height and brightness (3D-Chair) components are shown.
Sec. 6.2
A
#
9
4
7
9
3
0
5
0
2
3
3
3 3
3
3
3
3
3
7
7
7
7
7
7
+
O 0
O
9
9
9
9
9
9
3 3 9
*
0 C
O
O
0
Q
7
9
9
7 9
7
9
0 0
O
9
9
9
9
9
Figure 5: Manifolds derived for noise-added data ?highlighted in red boxes). Here, also train on RGB-MNIST dataset: simple extension of MNIST by coloring with (R, G, or B).
6.3 GENERALIZATION OVER UNTRAINED MANIFOLDS
Using the encoding method deriving quotient manifolds (Eq. 3 we examine if QMM-applied models well generalize over untrained manifolds. In both case of added noises (Fig. 5 and different datasets (Fig. 6 they indeed show fair performances in generalization, sharing generic structure with the trained manifolds (aligning continuous features column-wise).
7 CONCLUSION
We proposed QMM that performs Unique to QMM is that it utilizes the generalization ability of deep encoder, from which it showed its potential to infer the untrained manifolds even across different datasets. is trained with larger datasets such as ImageNet the future, expect QMM-applied models would become a more versatile tool that can derive the manifold structure of images in wild.
learning in consideration for the generic structure.
3
3
3
3
2
2
3
2
4
5
5
4
5
4
5
4
6
6
3
6
3
3
6
3
6
3
0
9
0
0
0
0
9
5
4
5
4
5
4
5
4
5
/
6
6
6
6
3
6
6
6
3
6
3
6
9
0
9
0
9
5
T
5
5
4
7
4
4
7
2
7
7
7
7
>
8 1
8
8
/
8
8
4
4
4
4
9
7
9
7
8
8
8
8
7
8
7
8
8
3
J
V
V
J'
V
V
V
6
0
6
0
6
6
C
6
O
6
0
0
6
6
Figure 6: Left: QMM-applied models trained on MNIST generates alphabet 'v" included in EM- NIST dataset. Right: QMM-applied models trained on two-digit MNIST (multiples of 9) dataset, generating one-digit MNIST and non-multiples.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. January 2017.
Mathieu Aubry, Daniel Maturana, Alexei A. Efros, Bryan C. Russell, and Josef Sivic. Seeing 3D Chairs: Exemplar Part-Based 2D-3D Alignment Using Large Dataset of CAD Models. In 20/4 IEEE Conference on Computer Vision and Pattern Recognition, pages 3762-3769, Columbus, OH, USA, June 2014. IEEE. ISBN 978-1-4799-5188- doi: 10.1109/CVPR220444445
Alon Brutzkus, Amir Globerson, Eran Malach, and Shai Shalev-Shwan SGD LEARNS PARR--PAAAEEEEZEE NETWORKS THAT PROVABLY GENERALIZE ON LINEARLY SEPARA- BLE DATA. page 17, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 2172-2180. Curran Associates, Inc., 2016.
Emilien Dupont, Learning Disentangled Joint Continuous and Discrete Representations. 10X0001804.00104 [cs, stat], October 2018.
Arnab Ghosh, Viveka Kulharia, Vinay Namboodiri, Philip H. Torr, and Puneet K. Dokania. Multi- Agent Diverse Generative Adversarial Networks. arXiv:1704.02900 [cs, stat], April 2017.
Ross Girshick. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision, pages 1440-144, 2015.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q Weinberger, editors, Advances in Neural Information Processing Systems 27, pages
Curran Associates, Inc., 2014.
Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, and R. Venkatesh Babu. DeLiGAN Gener- ative Adversarial Networks for Diverse and Limited Data. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 166-174, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Guyon, U. V. Luxburg, Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 6626-6637. Curran Asso- ciates, Inc., 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. Vat::Vae: Learning Basic Visual Concepts with Constrained Variational Framework. November 2016.
Quan Hoang, Tu Dinh Nguyen, Trung Le, and Dinh Phung. MGAN: Training Generative Adversar- ial Nets with Multiple Generators. February 2018.
Yeonwoo Jeong and Hyun Song. Learning Discrete and Continuous Factors of Data via Alter- nating Disentanglement. arXiv:1905.09433 [cs, stat], May 2019.
Mahyar Khayatkhoei, Maneesh K. Singh, and Ahmed Elgammal. Disconnected Manifold Learning for Generative Adversarial Networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 7343-7353. Curran Associates, Inc., 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. December 2014.
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. arXiv:1312.6114 [cs, stat], December 2013.
Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, pages 2278-2324, 1998.

---

Under review as a conference paper at ICLR 2021
Fangchang Ma, Ulas Ayaz, and Sertac Karaman. Invertibility of Convolutional Generative Networks from Partial Measurements. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 9628-9637. Curran Associates, Inc., 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral Normalization for Generative Adversarial Networks. arXiv:180..05957 [cs, stat]. February 2018.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. November 2015.
Hang Shao, Abhishek Kumar, and P. Thomas Fletcher. The Riemannian Geometry of Deep Gener- ative Models. 1rXiv:1711.08014 [cs, stat), November 2017.
Karen Simonyan and Andrew Zisserman, Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv: 1556 [cs], September 2014.
Jakub Tomezak and Max Welling. VAE with a VampPrior. arXiv:1705.07 [es, stat], February 2018.
Chang Xiao, Peilin Zhong, and Changxi Zheng. BourGAN:
Generative Networks with Metric Em- beddings. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Besa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 2269-2662222222 2280. Curran Asso- ciates, Inc., 2018
Aron Yu and Kristen Grauman. Fine-Grained Visual Comparisons with Local Learning. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 192-199, Columbus, OH, USA, June 2014. IEEE. ISBN 278-1-4799-5118- doi: 10. 10.1109/CVPR22014...2
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv:1611.0352: [cs], November 2016.
10

---

Under review as a conference paper at ICLR 2021
A THE PROOF OF THEOREM
Theorem 1, The following inequality holds
trace (cov(UT a (
- dz
H(Aa (1(A))E- dd= FR= ttraee (cov(af)))
where {>k 1x the eigenvalues of UT and H() denotes harmonic mean.
Proof. Note that
trace (cov(a = trace (Co((U(UTU)~'
U) (T)
(a a)((()) -a) atTottTTTU U(U 11
trace
A-1
trace
A- 1 -000000000000100111
1
Σ
(cov(UTa
trace
trace
(cov(UTafe (P(UUTa))) trace ((U U)-").
where the second and the fourth lines use the definition of the covariance, third line is obtained from the cyclic property of trace and the last line obtained from the Cauchy-Schwarz inequality of the positive semi-definite matrices. Thus,
trace (cov(U) aa)))) Z trace (cov(a())) /trace UUU
1 ((Aa)a)))) (Var(af)))
where (dal are the eigenvalues of U T and (-) denotes the harmonic mean.
B MODEL ARCHITECTURE AND EXPERIMENTING ENVIRONMENTS
We used machines with one NVIDIA Titan Xp for the training and the inference of all the models.
B.1 MNIST
We use = 10 distinct decoding biases in the model. the training, we set the regularization weight À 0.05 and use the Adam optimizer with learning rate 0.0002. In the encoding, Adam optimizer with learning rate 0.1, and the set the regularization weight /z = 0.1.
the
Table B.1: Q-WGAN architecture used for MNIST dataset (Q-3-VAE architecture is similar. See below for the differences.)
Generator
Diserininator
Input(8)
Input( 1, 1,28,28)
Full(1024), BN, LReLU(0.2)
Conv(c=64, k=4, s=2, p=1), BN, LReLU(0.2)
Full(6272), BN, LReLU((.22)
Conv(c=128, k=4, s=2, p=1), BN, LReLU(0.22
ReshapeTTo(128.77)
ReshapeToo66222)
ConvTrs(c=64, k=4, k=4, s=2, p=1), BN, LReeU((0.2)
Full(1024), BN, LReLU(0..2)
ConvThs(c=32, k=4, 2=2, p=1), BN, LReLU(0..2)
Full(1)
ConvTrs(c=1, k=3, s=1, p=1), Tanh

---

Under review as a conference paper at ICLR 2021
B.1.1 NOTES ON THE OTHER COMPARED MODELS
Overall, match the architecture of other models with model for fair comparison. Some differences to note are:
We used 10 generators. Each generator has the same architecture as ours except the number of features or the channels divided by 4, match the number of trainable parameters. Note that is the suggested number from the original paper.
DMWGAN:
InfoGAN: Latent dimensions consist of discrete variable (10 categories), continuous variables and noise variables.
3-VAE & V-B: We used the same architecture as the generators of Q-WGAN, except the BatchNorm layers are removed. We used the Bernoulli likelihood.
B.2 3D-CHAIR
We use 20 distinct decoding biases in the model. the training, we set the regularization
weight À = 0.05 and use the Adam optimizer with learning rate 0.0002. In the encoding, we use the Adam optimizer with learning rate 0.1, and the set the regularization weight μ = 0.1.
Table B.2: Q- Q-WGAN architecture used for 3D-Chair dataset.
Generator
Discriminator
Input(10)
Input(1,64,64))
Full(256), BN, LReLU(0..2
Conv(c=64, k=4, s=2, p=1), BN, LReLU(0.2)
Full(8192), BN, LReLU(0..)
Conv(c=128, k=4, s=2, p= 1), BN, LReLU(()2)
ReshapeTT(11288888
Conv(c=128, k=4, s=2, p=l), BN, LReLU((0.2)
ConvTrs(c=64, k=4, s=2, p=1), BN, LReLU(0.22)
ReshapeTo(8192)
ConvThs(c=32. k=4, s=2, 1),, BN, (0.2)(0.22)
Full(1024), BN, LReLU((..2)
ConvTis(c=16, k=4, s=2, p=l), BN, (ReLU((..2)
Full(1)
ConvTis(c=1, k=3, s=1, p=1), Tanh
B.2.1 NOTES ON THE OTHER COMPARED MODELS
We used 20 generators. Each generator has the same architecture as ours except the BatchNorms removed and the number of features the channels are divided by 4, to match the number of trainable parameters. Note that 4 is the suggested number from the original paper. Note that this was the best setting among what we have tried (division number 2; ones with the BatchNorms).
DMWGAN:
InfoGAN: Latent dimensions consist of discrete variables (20 categories), continuous variable and 10 noise variables.
3-VAE & Q-R-VAR: We used the architecture the generators of Q-WGAN, except the NatchNorm layers are removed. We used the Bernoulli likelihood.
B.3 UT-ZAP50K
We use A 4 distinct decoding biases the model. For the regularization weight the training, start with À 5e 6 then raise to À = 5e 4 after 300 epochs.
12

---

Under review as a conference paper at ICLR 2021
Table B.3: Q-WGAN architecture used for UT-Zap500k dataset.
Generator
Discriminator
Input(8)
Input(3,32.32)
Fu2) BN, LReLU(0.22
Conv(c=128, k=4, s=2, p=1), BN, LReLU((..2
FulI(1024), BN, (ReL.U(0.2)
Conv(c=256, k=4. s=2, p=1), BN, LReLU((..2)
Full(8192), BN, (Re2.U(0.2)
Conv(c=512, k=4, s=2, p=1), BN, LReLU(0..2)
ReshapeTo((112.4.4
ReshapeTo((1192
Con'Tis(c=256, k=4, s=2, p=1), BN, (.ee.U(0.2)
Full(1024), BN, (ReI.U(0.2)
Con/Tis(c=128 k=4, s= p=1), BN, (ReI.U(0.2)
Full(512), BN, LRe..U(0.2)
ConvTrs(==64, s=2, p=1), BN, LReLU(0.2)
Full(1)
ConvTrs(c=3, k=3, s=1, p=1), Tanh
C Q-3-VAE MODEL
Q-B-VAE model adopts a multi-generrtor (multi-decoder) version of 3-VAE along with the BA- regularizer. To maximize the marginal likelihood given the multiple generators, uses EM-like algorithm:
E-Step:
4 Q(G) Σ
(0) (() log Pg (xle), where 7(1) =P (x/²)/ Σ PG (alz)
M-Step:
- (z)g()))) [Q(G)) + BDKL (d(2)=))p(()) + RBa(fa ( h.).
where RBA((a; (8) (1.) = a EL_ log (rrace(cov(U,) ( )(")).
In the E-Step, it takes the expectation over the different generators; the responsibilities of each gen- erator can be computed as presented above since ( p(ilar, =) -.P (xll=//] E, म.P& (æ|2) PG (xl=)/> 3; Pg (z]2). In the M-Step. we plug in the computed expectation Q(G) as the marginal likelihood term (7(1) is fixed). Repeating and M step multiple times (we take repeats), finish the gradient step for single mini-batch.
D DISENTANGLEMENT SCORE
To compute the disentanglement score, first take 500 images from the dataset and manually change one of the smooth features that corresponds to known transformation. For example, we change the slant of the MNIST digits by taking sheer transform. With 11 different degrees of the transformation, we obtain 5500 transformed images in total. We encode these images to obtain the corresponding latent codes and subtract the mean for each group of the images (originates from the same image) to align all the latent codes. Then, we conduct Principal Component Analysis (PCA) to obtain the principal direction and the spectrum of variations of the latent codes. If the latent features well disentangled, the dimensionality of the variation should be close to one. To quantify how much it is close to one, we compute the ratio of the first eigenvalue to the second eigenvalue of the PCA covariance, and set it the disentanglement score.
13

---

Under review as a conference paper at ICLR 2021
E EFFECT OF THE NUMBER OF GENERATORS, A
To investigate the effect of the number of generators (or biases), A, we train our model on MNIST with different A values then compute the FID and the disentanglement scores (Fig. E.T. It can be seen that our model performs consistently better than the baseline, WGAN, regardless of the different values.
Disent. Score
FID FID Score
Sheer Width
2
FID
10
25
Number of Biases (A)
Figure E.1: The FID scores (right axis, the smaller the better) and the disentanglement scores (left axis, the larger the better) of Q-WGAN with varying A are shown for MNIST dataset. The dashed lines show the mean scores of the baseline model (WGAN).
F EFFECT OF THE REGULARIZATION WEIGHT, X, IN TRAINING
To investigate the effect the regularization weight, X, we train our model on MNIST with differ- values. It can be seen that our model performs consistently better than the other compared models-Q-WGA. with no regularization (A = 0), DMWGAN, MADGAN-like (see the next reggglas of the different À values (other models omitted for better readability: see Table for the omitted ones). It can be also seen that the scores are not very sensitive to the different choices of X this is beneficial in that one may choose any reasonable value for when training the model with a new dataset.
Here, MADGAN-like is DMWGAN model, but has similar parametrization to the MADGAN [6]: The parameters of the first three layers (from the top) are shared for all the generators. In contrast, Q- WGAN shares the parameters in the layers except the biases in the fully-connected layers. Thus, in sense, one can say that Q-WGAN shares only the last few layers, whereas MADGAN-like shares only the first few layers (of course, there is another difference due to the regularizers). Although the both models have the shared structures among the generators. Q-WGAN performs much better in both of the scores as seen in Figure F22 This suggests that simple parameter sharing is not enough obtain good performance, and the bias regularizer is indeed required.
14

---

Under review as a conference paper at ICLR 2021
16
14
FID Score
12
10
4
Q-WGAN (Curs) Q-WCIAN m0)
MADGAN-iike
5.0
0.5
0.05
0.005
0.0005
Bias Regularization Weight (A)
(a) FID Scores
4.5
+
C-WGAN (Ours)
DRWGAN
G-WGAN (Duns)
DMWGAN
Disent: Score (Slant)
MADGAN
Disent. Score (Width)
like
3.0
2.5
2.0
2
1.0
0.5
0.0
5.0
0.5
0.05
0.005
0.0005
5.0
0.5
0.05
0.005
0.0005
Bias Regularization Weight (A)
Bias Regularization Weight (A)
(b) Disentanglement Score (Slant)
(c) Disentanglement Score (Width)
Figure F.2: The FID scores (the smaller the better) and the disentanglement scores (the larger the better) of Q-WGAN with varying À are shown for MNIST dataset. Note the other models positioned in the center (shaded in gray) to be visually comparable with the best-performing Q- WGAN model (A 0.05).
G EFFECT OF THE REGULARIZZTION WEIGHT, 14, IN ENCODING
To investigate the effect of the regularization weight, μ. encoding (Eq. 3), we take trained model and encode image from the train set using different values. Then, plugging in the encoded (estimated) biases, {ar] we randomly generate the samples from this new manifold and compare the qualities for different n's.
From Figure G.3 (b), we can see that the quality the encoding tends to improve as gets smaller. This might seem opposite to what we expect, as weaker regularization gives better results. However, looking closer, we can see that the features like the slant and the stroke are more aligned with a stronger regularization of μ 0.05, when comparing with the other manifolds in the bottom pane. Thus, a trade-off exists here between the image quality the samples and the alignment of the manifold to the others. Note chose use μ 0.05 in all the other experiments.
15

---

Under review as a conference paper at ICLR 2021
3 3
3
(a) Left: An image taken from the training dataset. Middle: The same im- age with rectangle noise. Right: Regenerated image from the estimated biases, {ar (a))fi and the estimated latent code 2, from Eq.
3
8
8
3
3
3
3
3
3
5
3
2
3
3
3
3
3
3
3
3
7
7
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
5
3
5
5
6
6
S
5
5
6
6
6
6
6
6
6
6
6
6
2
2
2
2
2
2
2
2
2
2
3
3
5
5
3
3
3
3
3
5
9
7
9
7
9
3
3
7
9
9
(b) Top pane: Generated samples from the estimated biases, [ar] The biases are estimated with different y's, (5.0, 0.5, 0.05, 0.005, 0.0005, 0.0), from the top the bottom. Bottom pane: Generated samples from the higinallly-ccamee biases, Hia (8) 34 > (only out of 10 are shown).
Note the images in the same column have the same latent value
Figure G.3: Effect of the regularization weight μ in encoding shown with a MNIST-trained Q- WGAN model.
H EFFECT OF THE BIAS REGULARIZER
To examine the effectiveness of bias regularizer, we visualize the raw values of biases {a Je, and their (pseudo- -taneeniaa component (U, ) )at (see Fig. H.4 H.5). In all figures, we see that the biases are diverse, but their tangential components are well aligned due to the bias regularizer (left). On the contrary, without the regularizer, the tangential components are not aligned (right).
I SAMPLES GENERATED FROM VARIOUS MODELS
16

---

Under review as a conference paper at ICLR 2021
(
2.0
15
Values
Values
-15
-2.0
200
400
600
800
1000
200
400
600
800
1000
Dimensions
Dimensions
U 0
(
U a
10:
Values
Values
005
-10
-20
-1.5
-40
Dimensions
Dimensions
a
a
Values
Values
1000
2000
3000
4000
5000
6000
1000
2000
3000
4000
5000
6000
Dimensions
Dimensions
UTa (
UTa
Values
Values
-10
200
400
600
800
1000
200
400
600
800
1000
Dimensions
Dimensions
Figure H.4: Biases aj and their (pseudo- Manneenia components U, aa of the Q-WGAN models, trained on MNIST. Individual curve indicates each i-th bias. Left Parameters of Q-WGAN Right Parameters of Q-WGAN without the regularizer (A 0). It can be seen that the regularizer makes the tangential components of the biases well aligned. 17

---

Under review as a conference paper at ICLR 2021
(1)
Values
Values
50
100
150
200
250
50
100
150
200
250
Dimensions
Dimensions
U a
UTa
0.1
Values
-0.1
Values
-022
-0.4
-05
Dimensions
Dimensions
a
Values
Values
2000
4000
6000
8000
2000
4000 Dimensions
6000
8000
Dimensions
UTa
UTa
(
150
100
Values
Values
-50
150
100
200
Dimensions
Dimensions
Figure H.5: Biases df and their (iseudo-)tangentne component U,T aj of the Q-WGAN models, trained on 3D- 3D-Chair. Individual curve indicates each i-th bias. Left Parameters Q- WGAN Right Parameters of Q-WGAN without the regularizer (A = 0). It can be seen that the regularizer makes the tangential components the biases well aligned.
18

---

Under review as a conference paper at ICLR 2021
6
6
6
6
6
6
6
6
6 6
5 5
5
$
5
8
S 3 5
2 2
2
2
3
2
2 2
4
4
4
4
9
4
4 0000000
4
4
9 0 r 3 2 23 6 / /
O 00000000
0 6
0
0
0 9 2 6
0
0
5
5
5
5
5
5
3 9
9 2 6
9
9
4 2 6
99999
9
9
9
9
9
9
2
8
8
4
4
4
4
4
4
6
6
6
6
6 6 2 7
8888 8
8
8
8
8 7
8
/
2
2
7
2
/
77777771
7
7
1
1
1
(
/
/
|
1
/
1
1
3
3
3
8
3
3
5
3
3 7
3 7
3
3
3
3
3
3 3
3
3
3
7
7
7
7
9
7
7
7
(a) Q-WGAN
(b) Q-WGAN (A
0)
4
4
1
9
/
6
9
7
3
2
8
4
6
0 0
3
3
2
3
2
2
2
3
3
2
8
4
7
6
6
0
9
/
4
4
/
/
4
/
4
6
9
4
7
6
3
7
8 1 5
7
9
7
8
7
7
7
7
8
7
5
7
2
6
9
3
0
9
9 6
8
3
8
3
8
5
3
5 0 2
9
8
8
9
6
6
7
4
0
O
O
C
0
0
0
0
0
0
9
J
9
5
9
8
4
7 3
a
2
9
9
9
4
9
9
7
9
2
9
6
2
5
0
7 4
9 6
6
5
6
6
5
6
6
6
6 5 9
6
4
3
9
2
0
/
7
5
9
5
7
/
3
5
4
7
4
7 6
0 1
8
8
6
7
4
7
8
7
2
/ 3
6
6
2
7
(c) DMWGAN
(d) WGAN
9
3
2
4
4
q
3
6
O 05000
5
5
0
5
6
4
9
<
2
8
7
3
7
4
9
4
4
/
9
6
6
I
1
9
6
2
Q
9
8
8
3
5
2
2
/
3
3
3
3
2
/
3
3 7 6
1
3
O
9
2
7
4
7
4 1 4
8
7
7
7
7
7
7
7
7
7
0
2
4
9
2
2
9
1
G
6
6
6
6
6
6
5
6
4
9
9
0
3
7
3
7
2
3
2
2
3
Z
2
2
2
3
8
3
9
2
a
6
6
2 9 2
9
9
4
4
9
4
9
9
9
9
Q
3
9
5
6
8
2
9
9
9 8 8
4 0
9
4
I
9 6
4
9
0
8
2
2
7
7
0
2
8
0
3
0
6
0
0
7
5
7
3
7
8
5
6
/
0
8
8
8
8
8
8
8
8
8
(e) 3-VAE
InfoGAN
Figure L.6: MNIST image samples generated from the trained models
19

---

Under review as a conference paper at ICLR 2021
1
%
H
3
A
4
WGAN
(b) 8- 8-VAE
(c) InfoGAN
Figure 1.7: 3D-Chair image samples generated from the trained models
20

---

Under review as a conference paper at ICLR 2021
3
J
H
Figure 1.8: 3D-Chain image samples generated from the trained Q- Q-WGAN
21

---

Under review as a conference paper at ICLR 2021
M
R
R
4
4
4
R
A
4
R
Figure 1.9; 3D-Chair image samples generated from Q-WGAN (A 0)
22

---

Under review as a conference paper at ICLR 2021
Figure 1.10: 3D-Chair image samples generated from the trained DMWGAN
23


Reviews:
[{"review_id": "ifhu9Z__rHc", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Natural images may lie on the disjoint multi manifolds rather than one globally connected manifold, and this can cause several difficulties for the training of Generative Adversarial Networks (GANs) and variational autoencoders (VAEs).\nThe paper proposes QMM, a new generative modeling scheme that inherits the multi-generator scheme but involves an essential regularizer enforcing the encoder compatibility. QMM considers generic manifold structure by the generalizable equivalence relation between data, thereby taking the quotient of this relation and driving the manifold structure of untrained data.\n\nClarity: The paper is well written and easy to follow.\n\nNovelty:  Inheriting multiple generators scheme for generate model training has been proposed in many previous works. There are some recent work for applying multiple generators to VAEs or GANs, e.g., Pan et al., Latent Dirichlet allocation based generative adversarial networks, Neural Networks, 2020,  etc.   A major difference in this paper is that the additional regularizer enforcing the encoder compatibility and the quotient of the plausible equivalence relation. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "iaxyw2muyfX", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "_Summary_:\nThe author extends generative models with multi-generators by restricting the generators to share weights and all bias to be regularized in order to enforce that the inverse maps of the generators can be represented by a single encoder. The regularizer proposed minimizes an upper bound the the sum of the bias variances. This extension is evaluated on a set of visual datasets (MNIST, 3DChair and UT-Zap50k) with respect to density estimation (evaluated with the FID score) and disentanglement (evaluated with their own disentanglement score).\n\n_Strengths_:\n- The method proposes a regularizer which is easy to understand and implement. This makes it easy to apply to existing generative models.\n- The evaluation showed that sample diversity w.r.t. FID score is comparable/superior to baseline generative models.\n\n_Weaknesses_:\nFor me, the main problems are the motivation of encoder compatibility, the clarity of the paper and limited evaluation:\n1) Motivation of encoder compatibility: I don't understand why the inverse of all generators needs to represented by one (!) single encoder? Why can't the generators be in general invertible and thus there exists a mapping for each observed variable to the latent space? What is the advantage of one encoder?\n2) The paper needs to improve its preciseness and clarity. I found it at parts confusing to understand because it introduces terms without really explaining or justifying them, e.g., \"encoder compatibility\", \"good generalizability\", \"equivalence relation\". Further, background is lacking to give details about the model that this paper extends on. This also hinders readability of the section where the authors introduced their own method. Further, assumption of continuity and generalizability have been made without defining these terms or reference.\n3) The evaluation is limited to generators w.r.t. generator architecture (fully-connected layers only). Further, FID score show high standard deviation and thus are not necessarily superior compared to baselines and the introduced disentangle metric are only reported for specific attributes of MNIST and 3D-Chair.\n\nMore detailed comments are below.\n\n_Overall assessment_: \nUnfortunately, I do not believe this paper is ready for acceptance. Therefore I am recommending a rejection. I do believe it is an interesting approach. And with improvement in terms of clarity and evaluation, I am happy to improve my score if the authors can make the necessary improvement.\n\n_Detailed comments and questions_:\n- \"since discrete features are both common and combinatorial\" (Introduction): Common to what?\n- \"they usually have the same generic structure since the underlying continuous features since the underlying continuous features remain the same\" (Introduction): Has this been shown in any work before (if so, please cite)? I can also imagine this being depending on the specific learning task.\n- \"it induces a generalizable equivalence relation between data, and the manifold structure of out-of-sample data can be derived by taking the quotient of this relation\" (Introduction): What is a generalizable equivalence relation between data? Is the quotient of this relation defined somewhere?\n- \"Since deep encoders usually exhibit good generalizability\" (Introduction): What is the generalizability considered with respect to? Can you elaborate?\n- Multi-generator scheme (Sections 2.2, 3): Can you elaborate on the actual model used for Quotient Manifold Modelling (QMM)? The paper mentions that a multi-generator scheme is used but there are no more details than that. It would be helpful to know what the learning objective is, how the (multiple) generator (and discriminator) are used.\n- \"Let H be a set of encoding maps $(X \\to Z)$ that can be represented by a deep encoder\" (Section 3.1): Why is being represented by one encoder important?\n- \"this binding is meaningful only when H has a certain property\": Which one? Is that the generalizability that is mentioned in the next paragraph? \n- \"its elements--deep encoders--have good generalizability\": Similar to the claim above, what does that mean to have generalizability? How can generalizability be quantified?\n- High standard deviation of FID scores of proposed methods: The proposed method have high standard deviation. Can you explain why this is? Is this an optimization problem?\n- Proposed disentanglement metric: There is a vast literature of evaluation for disentanglement, why did you choose to propose your own disentanglement metric? How does it compare to the existing disentanglement metrics (advantages, similarity, etc.)?\n- Table 1 (disentanglement): Can you also report overall disentanglement of the dataset? For disentanglement specifically you can also use existing benchmarks and datasets from Locatello et al. (2018).\n\n_Minor_:\n- Definition of generative maps ${f_G^{(i)}:Z->M [...]\\}_i^A$ (Section 3.1): I believe $A$ was not defined before, is that just the number of generators?\n\n_Post-Rebuttal_:\nUnfortunately, the authors neither did update their paper nor addresses my comments. Therefore, I'm keeping my recommendation of rejection.", "extra_scores": {}, "extra_text": {}}, {"review_id": "YLCR3IAFpJT", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. The \"orthogonal directions  to  the  equivalence-relation  contours\" in section 3.2. should be properly defined. I can understand that the authors want to say that the horizontal spaces along the sections are perpendicular to the vertical spaces along the fibers. But still need clarification for a general audience.\n\n2. The tangential components are ``parallel'' to U and hence in prop. 1, they need to be same. This is not obvious and need to be properly defined. Along this direction, the tangent and normal of a are with respect U. On the other hand, along the fibers there exists a group operation, which is translation for a linear layer and justify Prop. 1. So Prop. 1 is not surprising or worth proving in the paper. \n\n3. The same nonlinear activation functions (we useLeakyReLU) and batch-normalization layers constraint the explanability of the network. This implies one needs more linear layers. To me, this needs further investigation, and while reading the paper, I thought some interesting ideas will come up during developing non-linearity. Although the design using quotient structure is interesting, the choice of sharing same vertical component is a easy consequence of principal fiber bundle theory. It will be interesting to study ``better'' choice of non-linearity. Can authors comment on that?\n\n4. The experimental setup is a bit under-developed. The disentnglement scores have only been shown on MNIST images, what about for more complicated images like CelebA, e.g., \"Learning Disentangled Representations via Independent Subspaces\", CVPR 2019.\n\n5. Similar kind of simplistic setup has been used for showing generalizability on untrained datasets.\n\n6. These kind of simplistic experimental setup points to the simplicity of just changing liner layer and using same non-linearity. Can authors please verify whether the method ``works'' (in such a simplistic manner) on more complicated datasets, i.e., do we need a very deep network? If yes is the training still stabilized or does converge?", "extra_scores": {}, "extra_text": {}}, {"review_id": "IGuLKWdhriO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a method to train deep generative models on quotient manifolds and show improved performance on some simple standard test sets.\n\nThe article makes a strong and compelling case for the need of working with quotient manifolds, and shows good results over baselines in all the test cases investigated. The method is simple enough to implement and it’s theoretical motivation plausible (although far from conclusively proven).\n\nHowever, I feel the descriptions of the quotient manifolds investigated (e.g. the continuous-discrete differences) are somewhat lacking. Figure 1 strongly implies the quotient spaces are w.r.t. some continuous quantity, while the method is discrete. It does feel like the method is more aimed towards manifolds with multiple disjoint components than true quotient manifolds. The experiments are also lacking. The baselines are old, and for some of them I know for sure that there are better results available out there. I would be much happier if the ablations were one-to-one, e.g. using the exact same architecture except for this change.\n\nOverall I find this an OK contribution, but it would be strengthened by either giving more sharp theoretical results or by using more up to date experimental methods.\n\nSome smaller comments:\n* It’s not quite clear that the regulariser will force the parallel components to behave as expected. Wouldn’t another possible way for the network to solve this be to make the U’s very small?\n* The architecture seems to assume a standard feed-forward setup. This isn’t really state of the art (almost everyone does resnets). Would it be possible to extend the setup to this slightly more general setting?\n* The domain of “i” should be defined\n* The definition of the pseudo inverse uses only holds for full rank matrices. How do you make sure this is the case in practice?\n* GANs, VAEs, etc have reasonably strong theoretical foundations and simply adding a regulariser might break them in weird ways. Would it be possible to incorporate the proposed methods more rigorously?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an interesting method dubbed quotient manifold modeling to handle the "multi-manifold" structure of natural data and generalize to new manifolds that arise from novel discrete combinations. While some of the methods and ideas were appreciated by reviewers, there were a number of experimental and clarity concerns. The authors's did not submit a rebuttal, and the many unaddressed concerns (especially around experimental baselines) lead me to recommend rejecting this work.
Decision: Reject