Conditional Generative Modeling via Learning the Latent Space
Authors: Sameera Ramasinghe
Kanchana Nisal Ranasinghe
Salman Khan
Nick Barnes
Stephen Gould
Venue: ICLR 2021 Poster (2021)
Keywords: Multimodal Spaces
Conditional Generation
Generative Modeling

Abstract: Although deep learning has achieved appealing results on several machine learning tasks, most of the models are deterministic at inference, limiting their application to single-modal settings. We propose a novel general-purpose framework for conditional generation in multimodal spaces, that uses latent variables to model generalizable learning patterns while minimizing a family of regression cost functions. At inference, the latent variables are optimized to find solutions corresponding to multiple output modes.  Compared to existing generative solutions, our approach demonstrates faster and more stable convergence, and can learn better representations for downstream tasks. Importantly, it provides a simple generic model that can perform better than highly engineered pipelines tailored using domain expertise on a variety of tasks, while generating diverse outputs. Code available at https://github.com/samgregoost/cGML.

Published as a conference paper at ICLR 2021
CONDITIONAL GENERATIVE MODELING VIA LEARN- ING THE LATENT SPACE
Sameera Ramasinghe f* Kanchana Ranasinghel, Salman Khan Nick Barnes and Stephen Gould
i Australian National University, "Dat661 (CSIRO), $ Mohhmedd bin Zayed University of AI ameera. ramasinghe@anaa edu.
ABSTRACT
Although deep learning has achieved appealing results on several machine learning tasks, most of the models are deterministic at inference, limiting their applica- tion to single-modal settings. We propose a novel general-purpose framework for conditional generation in multimodal spaces, that uses latent variables to model gen- eralizable learning patterns while minimizing a family of regression cost functions. At inference, the latent variables are optimized to find solutions corresponding multiple output modes. Compared to existing generative solutions, our approach demonstrates faster and more stable convergence, and can learn better represen- tations for downstream tasks. Importantly, it provides a simple generic model that can perform better than highly engineered pipelines tailored using domain expertise on a variety of tasks, while generating diverse outputs. Code available at https: / / github...... com/ http://aamrgooststoooo cCML.
INTRODUCTION
Conditional generative models provide natural mechanism to jointly learn a data distribution and optimize predictions. In contrast, discriminative models improve predictions by modeling the label distribution. Learning to model the data distribution allows generating novel samples and is considered preferred way to understand the real world. Existing conditional generative models have generally been explored in single-modal settings, where one-to-one mapping between input and output domains exists (Nalisnick et al., 2019; Fetaya et al., 2020). Here, we investigate continuous multimodal (CMM) spaces for generative modeling, where one-to-many mappings exist between input and output domains. This is critical since many real world situations inherently multi- modal, e.g., humans can imagine several completions for a given occluded image. In a discrete setting, this problem becomes relatively easy to tackle using techniques such as maximum -ikeuum-iieliohod--- estimation, since the output can be predicted as a vector (Zhang et al., 2016), which is not possible in continuous domains. One way to model CMM spaces is by using variational inference, e.g., variational autoencoders (VAE) (Kingma & Welling, 2013). However, the approximated posterior distribution of VAEs are often restricted to the Gaussian family, which hinders its ability to model more complex distributions. As solution, Maalóe et al. (2016) suggested using auxiliary variables to improve the variational distribution. To this end, the latent variables are hicrarchically correlated through injected auxiliary variables, which can produce non-Gaussian distributions. A slightly similar work by Rezende & Mohamed (2015) proposed Normalizing Flows, that can hierarchically generate more complex probability distributions by applying series of bijective mappings to original simpler distribution. Recently, Chang et al. (2019) proposed a model, where a separate variable can be used to vary the impact of different loss components at inference, which allows diverse outputs. For a more detailed discussion on these methods see App. 1.
In addition to the aforesaid methods, in order to model CMM spaces, prominent approach in the literature is to use a combination of reconstruction and adversarial losses (Isola et al., 2017; Zhang et al., 2016; Pathak al., 2016). However, this entails key shortcomings. 1) The goals of adversarial and reconstruction losses are contradictory (Sec. 4), hence model engineering and numerous regularizers are required support convergence (Lee al., 2019; Mao et al., 2019),
"Correpooniigg author.

---

Published as a conference paper at ICLR 2021
thereby resulting in less-generti models tailored for specific applications (Zeng et al., 2019; Vitoria et al., 2020). 2) The adversarial loss based models are notorious for difficult convergence due to the challenge of finding Nash equilibrium of non-convex min-max game in high high-dimensioos (Barnett, 2018; Chu et al., 2020; Kodali et al., 2017). 3) The convergence is heavily dependent on the architecture, hence such models show lack of scalability (Thanh-Tung
al., 2019; Arora & Zhang, 2017). 4) The promise of assisting downstream tasks remains challenging, with a large gap in performance between the generative modelling approaches and their discriminative counterparts (Grathwohl et al., 2020; Jing & Tian, 2020).
In this work, we propose a ceneral-purposs framework Conditional Generation by Modeling the Latent Space (cGML) for modeling CMM spaces using set of domain-agnosto regression cost functions instead of the adversarial loss. This improves both the stability and eliminates the incompatibility between the adversarial and reconstruction losses, allowing more precise outputs while maintaining diversity. The underlying notion is to learn the 'behaviour of the latent variables' in minimizing these cost functions while converging to an optimum mode during the training phase, and mimicking the same at inference. Despite being a novel direction, the proposed framework showcases promising attributes by: (a) achieving state-of-the-an results on diverse set of tasks using a generic model, implying generalizability, (b) rapid convergence to optimal modes despite architectural changes. (c) learning useful features for downstream tasks, and (d) producing diverse outputs via traversal through multiple output modes at inference.
2 PROPOSED METHODOLOGY
We define a family of cost functions {Ei, = d(v ((z), w))}, where Ty ~ X is the input, vf, ~ is the jth ground-truth mode for Tz. G is a generator function with weights w, and d(::) .) is distance function. Note that the number of cost functions E(-5) for a given 2y can vary over X- Our aim here is to come up with a generator function 9(x,, w), that can minimize each F.,- Vi as (zz,, w) - vi, However, since G is deterministic function (r and w are both fixed at inference). it can only produce a single output. Therefore, we introduce a latent vector z to the generator function, that can be used to converge Di, G(zj, W, 2i.7) towards ground truth Fit,) at inference, and possibly, to multiple solutions. Formally, the family of cost functions now becomes: (Éi, = d(yl, G(r), W, zig)),, Vzi.j ~ Ç. Then, our training objective can be defined as finding set of optimal 2 E Ç and w" E w by minimizing Ex~ [É:g], where I is the number of possible solutions for æy. Note that w" is fixed for all i and different 2f exists for each i. Considering all the training samples x ~ X, our training objective becomes,
(2ig), wn
arg min EiErerJ [Eagl-
Zug Ew
Eq. can be optimized via Algorithm 1 (proof App. 2.2). Intuitively, the goal of Eq. is to obtain a family of optimal latent codes { each causing a global minima in the corresponding É., as vi, = (Z2,, w, 2. Consequently, at inference, we can optimize Tk. to converge to an optimal mode in the output space by varying z. Therefore, we predict estimated ž. at inference,
En, R min Paij
for each v²,3. which turn can be used to obtain the prediction G(#,, w, #..s) R vl,y In other words, for a selected xy, let vi be the initial estimate for Øt.j At inference, z can traverse gradually towards an optimum point vī, in the space, forcing " v. in finite steps (n).
However, still critical problem exists: Eq. 2 depends on vi,y- which is not available inference. As remedy, we enforce Lipschitz constraints on G over (3r,, which bounds the gradient norm as,
[ o7, -ttay mo)
<IIVASla,, wt, (ntte,,,""n()))))))))))))) dt < C.
(3)
where 2u Ç is an arbitrary random initialization, C is a constant, and 7(-) is straight path from 20 to =ī, (proof in App. 2.1). Intuitively, Eq. 3 implies that the gradients V.G(zz, w", =0) along the path 7(.) do not tend to vanish or explode, hence, finding the path to optimal 2/,5 in the space becomes a fairly straight forward regression problem. Moreover, enforcing the Lipschitz constraint
2

---

Published as a conference paper at ICLR 2021
%
H
Cost
(a) Training
(b) Inference
Figure Training and inference process. Refer to Algorithm 1 for the training process. At inference, z iteratively updated using the predictions of Z and fed to G to obtain increasingly fine-tuned outputs (see Sec. 3).
encourages meaningful structuring of the latent space: suppose zï.j and = are two optimal codes corresponding two ground truth modes for a particular input. Since || zī,ز is lower bounded by |[ ) .ff,))l where L is the Lipschitz constant, the minimum distance between the two latent codes is proportional to the difference between the corresponding ground truth modes. In practice, we observed that this encourages the optimum latent codes be placed sparsely (visual illustration in App. 2), which helps a network to learn distinctive paths towards different modes.
2.1 CONVERGENCE AT INFERENCE
We formulate finding the convergence path of at inference as a regression problem, i.e., Zt4-1 xz), We implement r(-) as a recurrent neural network (RNN). The series of predicted values first-order Markov chain requiring no memory for the
{Z(+4)): 2., .N can be modeled RNN. We observe that enforcing Lipschitz continuity on & over leads to smooth trajectories even in high dimensional settings, hence, memorizing more than one step into the history is redundant. However, Z+++1 is not state variable, i.e., the existence of multiple modes for output prediction leads to multiple possible solutions for 2++1- On the contrary, 1[2+1] is a state variable w.r.t. the state (#z,z) which can be used as approximation to reach the optimal x² at inference. Therefore, instead of directly learning r(-), we learn a simplified version r' (a,,, x) = [[[=+1]]. Intuitively, the smooth surface the training
whole process can be understood as observing the behavior of stage, and predicting the movement at inference. A key aspect of r'r (', is that the model is capable of converging to multiple possible optimum modes at inference based the initial position of z.
2.2 MOMENTUM AS A SUPPLEMENTARY AID
Based on See. 2.1, z can now traverse to an optimal position =" during inference. However, there exist rare symmetrical positions in the where 1[2++1] 24 0, although far away from {=*}, forcing 2++1 = 24. Simply, the above phenomenon can occur if some 2 has traveled in many non- orthogonal directions, the vector addition of 2ą-1 | 0. This can fool the system falsely identify convergence points, forming phantom optimum point distributions amongst the true distribution (see Fig. 3). To avoid such behavior. we learn the expected momentum E[p(2,, x,)] aB[[]2+1 2 [ey]] at each (zz, xz) during the training phase, where a is an empirically chosen scalar. In practice, (()(),,,,) - 0 as 24+1, 2t > {2*) Thus, avoid phantom distributions, we improve the update
as,
r' (2,, 24 (2),)))) 24
2+1 2 + Elp(21,
Since both Ejp(=;,zz)) xr and r'( II) are functions on (ze, Kz). we jointly learn these two functions using a single network Z(2,, æ,). Note that coefficient B[p(z,, x,)] serves two practical purposes: I) slows down the movement of near true distributions, 2) pushes 2 out of the phantom distributions.
3 OVERALL DESIGN
The proposed model consists of three major blocks as shown in Fig. 1: an encoder H. generator G. and Z. The detailed architecture diagram for 128 128 is shown in Fig. 2. Note that for derivations in Sec. 2, we used x instead of H(x), as h is a high-level representation of æ. The training
process is illustrated in Algorithm 1. At each optimization 24+1 2 PV =, [E.,]. J. 2 is trained separately to approximate (Z++1, p). At inference, x is fed to H. and then optimizes the output ğ by updating 2 for a pre-defined number of iterations of Eq. 4. For É(..), we use L, loss. Furthermore, it is important to limit the search space for 2411, to improve the performance of Z. To this end,
3

---

Published as a conference paper at ICLR 2021
hedd
Janv
lowd
44
300
Figure 2: Overall architecture for 128 128 inputs.
Algorithm 1: Training algorithm
sample inputs (#1, T2, T2, 35 E x; sample outputs (y1, 12, yu} Y; for epochs for in x do for steps do
update = {21, ..., /: V.
b> Freeze H. G, Z and update z
update 3: V V.Li [(E411, p), ((2, H()))
>> Freeze H. G, a and update Z
update G. H. V
P Freeze Z, z and update H. g
sample z from the surface of the n-dimensional sphere (S"). Morcover, to ensure faster convergence of the model, force Lipschitz continuity on both Z and the g (App. 2.4). For hyper-parametee and training details, see App. 3.1.
MOTIVATION
Here, we explain the drawbacks of conditional GAN methods and illustrate our idea via toy example.
Incompatibility of adverariaal and reconstruction losses: cGANs use combination of adversarial and reconstruction losses. We note that this combination is suboptimal to model CMM spaces.
Remark: Consider a generator G(x, 2) and discriminator D(x, 2). where and z are the input and the noise vector, respectively. Then, consider arbitrary input Ti and the corresponding set of ground-truths { -1, .... Further, us define the optimal generator G* (x. 2) = 2 9 E (u LGAN = E, [log D(v E. [log(1 D(G((,, (((((,,,)) =))] and T = Es,=llv,,, G( =)] Then, G* +G +G where G arg min max LGAN ALg, AA a (Proof in App. 2.3).
The incompatibility of above mentioned loss functions demands domain specific design choices from models that target high realism in CMM settings. This hinders the generalizability across different tasks (Vitoria et al., 2020; Zeng et al., 2019). We further argue that due this discrepancy, cGANs learn sub-optimal features which are less useful for downstream tasks (Sec. 5.3).
Generalizability:
Convergence and the sensitivity the architecture: The difficulty of converging GANs to the Nash equilibrium of non-convex min-max game in high-ddmeessioal spaces is well explored (Barnett, 2018: Chu et al., 2020; Kodali et al., 2017). Goodfellow et al. (2014b) underlines if the discriminator has enough capacity, and is optimal at every step of the GAN algorithm, then the generated distribution converges to the real distribution; that cannot be guaranteed a practical scenario. In fact, Arora al. (2018) confirmed that the adversarial objective can easily approach to an equilibrium even the generated distribution has very low support, and further, the number of training samples required to avoid mode collapse can be in order of exp(d) (d is the data dimension).
4

---

Published as a conference paper at ICLR 2021
Figure 3: Toy Example: Plots generated for each dimension of the CMM space T. (a) Ground- truth distributions. (b) Model outputs for L1 loss. (c) Output when trained with the proposed objective (without p correction). Note the phantom distribution identified by the model. (d) Elp] heatmap on (z, y). E'p] is lower near the truc distribution and higher otherwise. (e) Model outputs after correction.
11=4400
32 14-41
10=
(a) GT
(b)
(c)
(d) Elpl heatmup
(e)
Multimodality: The ability generate diverse outputs, i.e., convergence to multiple modes in the output space, is an important requirement. Despite the typical noise input, cGANs generally lack the ability to generate diverse outputs (Lee et al., 2019). Pathak al. (2016) and lizuka al. (2016) even state that better results are obtained when the noise is completely removed. Further, variants of cGAN that target diversity often face a trafe-off between the realism and diversity (He et al., 2018), as they have to compromise between the reconstruction and adversarial losses.
toy example: Here, we experiment with the formulations Sec. 2. Consider a 3D CMM space (z, _², )²"). Then, we construct multi-layer perceptrons (MLP) with three layers to represent each the functions, H. G. and 2, and compare the proposed method against the In loss. Figure illustrates the results. As expected, L1 loss generates the line y 0. and is inadequate model the multimodal space. As explained in Sec. 2.2, without momentum correction, the network is fooled by phantom distribution where [[ removes the phantom distribution and refines the output to closely resemble the input distribution. As implied in Sec. 2.2, the momentum is maximized near the true distribution and minimized otherwise.
0 at training time. However, the push of momentum
EXPERIMENTS AND DISCUSSIONS
The distribution of natural images lies on a high dimensional manifold, making the task of modelling it extremely challenging. Moreover, conditional image generation poses an additional challenge with their constrained multimodal output space (a single input may correspond to multiple outputs while not all of them are available for training). In this section, fair comparison with a similar capacity GAN, we use the encoder and decoder architectures used in Pathak et al. (2016) for 76 and G respectively. We make two minor modifications: the channel-wise fully connected (FC) layers are removed and U-Net style skip connections are added (see App. 3.1). We train the existing models for maximum of 200 epochs where pretrained weights not provided, and demonstrate the generalizability of our theoretical framework in diverse practical settings by using generic network for all the experiments. Models used for comparisons are denoted follows: PN (Zeng et al., 2019), CA (Yu et al., 2018b), DSGAN (Yang et al., 2019), CIC (Zhang et al., 2016), RFR (Li al., 2020), Chroma (Vitoria et al., 2020), P2P (Isola et al., 2017). lizuka (lizuka al., 2016), CE (Pathak et al., 2016), CRN (Chen & Koltun, 2017a), and B-GAN (Zhu et al., 2017b).
experiment several such tasks. For
5.1 CORRUPTED IMAGE RECOVERY
We design this task as image completion, i.c., given a masked image as input, our goal recover the masked area. Interestingly, we observed that the MNIST dataset, in its original form, does not have a multimodal behaviour, i.e., fraction of the input image only maps to a single output. Therefore, we modify the training data as follows: first, we overlap the top half of an input image with the top half of another randomly sampled image. We carry out this corruption for 20% of the training data. Corrupted samples are not fixed across epochs. Then, apply a random sized mask to the top half, and ask the network to predict the missing pixels. We choose two competitive baselines here: our network with the L1 loss and CE. Fig. 4 illustrates the predictions. As shown, model converges to the most probable non-corrue nor-corrpteed mode without any ambiguity, while other baselines give sub-optimal results. In the next experiment, add small white box the top part of the ground-truth images
5

---

Published as a conference paper at ICLR 2021
Method
STL.
User study
Net
Turing test ImageNer
Tizuka et al.
Method
LPIP
0.18
PiaAPP
2.37
STL
|SSIM
088
[PSNR
24.30
LPIP
0.17
PicAPP
2.47
ImageNet
SSIM
0.87
PSNR
18.43
lizuka
32.28
P2P
1.21
2.69
0.73
17.80
2.01
2.80
0.87
18.43
Chroma
62,40
31.67
CIC
0.18
2.81
0.71
22.04
0.19
2.56
0.71
19.11
Ours
45.71
36,05
31.66
0.16
0.91
25,57
2.13
Ours
0.12
1.47
0.95
27,03
0.16
2.64
0.92
24.51
Table 1: Colorization: Psychophysi- cal study and Turing test results. All performances in %.
Ours (wdo
0.16
1.90
0.89
25.02
0.20
2.11
0.88
23.31
Table 2: Colorization: Quantitative analysis of our method against the
state-of-the-art. Ours perform better on a variety of metrics.
Method
10%
corruption
corruption PicAPP
25%
corruption
LPIP
PicAPP
PSNR
SSIM
LPIP
PSNR
SSIM
LPIP
PicAPP
+
PSNR
SSIM
DSGAN
0.101
1.577
20.13
0.67
0.189
2.970
18.45
0.55
0.213
3.54
16.44
0.49
PN
0.045
0.639
27.11
0.88
0.084
0.680
20.50
0.71
0.147
0.764
19.41
0.63
CE
0.092
1.134
22.34
0.71
0.134
2.134
19.11
0.63
0.189
2.717
17.44
0,51
P2P
0.074
22.33
0.79
0.101
0.70
0.185
0.57
CA
0.048
0.731
26.45
0.83
0.091
0,933
20.12
0.72
0.166
0.822
21.43
0.72
RFR
0.051
0.743
29.31
0.85
0.097
1,033
19.22
0.70
Ours (wlo
0.053
0.799
27.77
0..83
0.085
0.844
23.22
0.76
0.141
0.812
22.31
0.74
Ours
0.051
0.727
27.83
0.89
0.080
0.740
26.43
0.80
0.129
0.760
24.16
0.77
Table
Image completion: Quantitative analysis of our method against state-of-ther on a variety metries.
different rates. At inference, our model was able to converge to both the modes (Fig. 5), depending on the initial position of 2, as the probability of the alternate mode reaches 0.3.
5.2 AUTOMATIC IMAGE COLORIZATTIO
Deep models have tackled this problem using semantic priors (lizuka et al., 2016; Vitoria et al., 2020), adversarial and L1 losses (Isola et al., 2017; Zhu et al., 201 7a; Lee et al., 2019), or by conversion to a discrete form through binning of color values (Zhang al., 2016). Although these methods provide compelling results, several inherent limitations exist: (a) use of semantic priors results in complex models, (b) adversarial loss suffers from drawbacks (see Sec. 4), and (c) discretization reduces the precision. In contrast, we achieve better results using a simpler model.
The input and the output of the network are and (a, b) planes respectively (LAB color space). However, since the color distributions of a and b spaces are highly imbalanced over natural dataset (Zhang et al., 2016), we add another constraint to the cost function E to push the predicted a and b colors towards uniform distribution: E = ||agt all + l bll A(lossat + LOSSRI, s), where lossMr = KL (0, 1)). Here, I-I) is the KL divergence and (0, 1) is a uniform distribution (see App. 3.3). Fig. 7 and Table 2 depict our qualitative and quantitative results, respectively. We demonstrate the superior performance of our method against four metrics: LPIP, PieAPP, SSIM and PSNR (App. 3.2). Fig. 10 depicts examples of multimodality captured by model (more examples in App. 3.4). Fig. 6 shows colorization behaviour as the z converges during inference.
User study: We also conduct two user studies to further validate the quality of generated samples (Table 1). a) In the PSYCHOPHYSICAL STUDY, we present volunteers with batches of images, each generated with a different method. A batch is displayed for secs and the user has to pick the most realistic image. After 5 secs, the next image batch is displayed. b) We conduct TURING TEST to validate our output quality against the ground truth, following the setting proposed by Zhang et al. (2016). The volunteers are presented with a series of paired images (ground-trui -ruth and our output). The images are visible for sec, and then the user has an unlimited time pick the realistic image.
3
X
4
9
8
3
7
7
7
/
hypa
2
3
o 7
2
^
7
7
D
H
H
H
8
3
O
7
-
4
3
7
3
O
O
3
X
4
4
8
(105)
Inper
Oape
3
O
O
3
8
4
4
8
3
Figure 5: With 330%
0
02
alternate mode data, Figure 6: The prediction
Figure 4: Performance with 20% corrupted data. Our model our model can con- quality increases as the tra- demonstrates better convergence compared to L1 loss and verge both the input verses to an optimum posi- similar capacity GAN (Pathak al., 2016).
modes (cols 4-5).
tion at the inference.

---

Published as a conference paper at ICLR 2021
GT
Lzuka
F2P
Chroma
CIC
Ours
Figure 7: Qualitative comparison against the state-of-the- the- on ImageNet (left 5 columns) and STL (right 5
columns) datasets. Our model generally produces more vibrant and balanced color distributions.
AAA
Output
Figure Image completion on Celeb-HQ (left) Facade (right) datasets. used fixed cen- ter masks and random irregular masks (Liu et al., 2018) for Celeb-HQ and Facades datascts, respec- tively.
Input
P2P
PN
Ours
Figure 9: Qualitative comparison for image completion with 25% missing data (models trained with random sized square masks).
Mode
Mode
Mide:
Mode2
Made
Mode 2
Figure 10: Multiple colorization modes predicted by model for single input. (Best viewed color).
Figure 11: Multi-modality our predictions on
Celeb-HQ dataset. (Best viewed with zoom)

---

Published as a conference paper at ICLR 2021
Figure 14: Map to aerial image transla- tion. From left: GT, Input and Output. Also see App. 5.2.
Figure 12: Translation from hand-bag sketches to images.
Figure 13: Translation from shoe sketches
to images.
Figure 16: Translation from Figure 17: Translation from facial landmarks to faces.
Figure 15: Diversity: Quantitative comparisons.
surface-normals to pet faces.
5.3 IMAGE COMPLETION
In this case, we show that our generic model outperforms similar capacity GAN (CF) as well as task-specific GANs. In contrast to task-specific models, we do not use any domain-specific modifications to make our outputs perceptually pleasing. We observe that with random irregular and fixed-sized masks, all the models perform well, and were not able to visually observe considerable difference (Fig. 8, see App. 3.11 for more results). Therefore, we presented models with more challenging task: train with random sized square-shapee masks and evaluate the performance against masks of varying sizes. Fig. 9 illustrates qualitative results of the models with 25% masked data. As evident, model recovers details more accurately compared to the state-of-the-art Notably, all models produce comparable results when trained with a fixed sized center mask, but find this setting more challenging. Table 3 includes a quantitative comparison. Observe that in the case of smaller sized masks, PN performs slightly better than ours, but worse otherwise. We also evaluate the learned features of the models against a downstream classification task (Table 5). First, we train all the models on Facades (Tyleček & Šára, 2013) against random masks, and then apply the trained models on CIFARIO (Krizhevsky et al., 2009) to extract bottleneck features, and finally pass them through FC layer for classification (App. 3.7). We compare PN and ours against an oracle (AlexNet features pre-trainn pre-trained on ImageNet) and show our model performs closer to the oracle.
Method
Pretext
Acc. (%)
ResNet
ImageNet Cls.
74.2
Sharma et (2016)
80.5%
75.5%
PN
Im. Completion
40.3
3
Han et (2019)
92.2%
90.2%
Ours
Im. Completion
62.5
65
Achlioptas al. (2017)
95.3%
85.7%
-00 -Ch
Yang et al. (2018)
94.4%
88.4%
Table 5: Comparison down-
Sauder
(2019)
Ramasinghe al. (2019c)
93.1%
stream task (CIFARIO cls). (")
Khan et (2019)
92.2%
denotes the oracle case.
Ours
92.4%
90.9%
Method
MIU
M40
Figure 18: Convergence on im- age completion (Paris view). Our model exhibits rapid and stable convergence compared to state-of the (PN, CE, P2P, CA).
CE
10.3
4.6
Table 4: Downstream 3D object classifi- cation results on ModelNet10 10 and Model- Net40 using features learned in unsuper- vised manner. All results % accuracy.
<VAE
4.2
Our
84.2
79,4
Table Reconstruction mAP
of 3d spectral denoising.
8

---

Published as a conference paper at ICLR 2021
Model
CF
PN
Chroma
CIC
P2P
Tizuka et al.
RFR
Ours
FLOPS (1 x 10")
0.634
0.946
1.275
52.839
0.732
14,082
25.64
0.638
Table 7: Model complexity comparison.
5.3.1 DIVERSITY AND OTHER COMPELLING ATTRIBUTES
We also experiment on diverse set of image translation tasks to demonstrate our generalizability.
Fig. 12, 13, 14, 16 and 17 illustrate the qualitative results of kketch-tohhandbag, sketch-to-shoes, map-to-arial, simmarks-to-face and surface- poftucceooomalsstoopaa show the ability of our model to converge to multiple modes, depending on the z initialization. Fig. 15 demonstrates the quantitative comparison against other models. See App. 3.4 for further details experiments. Another appealing feature of our model is its strong convergence properties irrespective of the architecture, hence, scalability to different input sizes. Fig. 19 shows examples from image completion and colorization for varying input sizes. We add layers to the architecture to be trained on increasingly high-resolution inputs, where our model was able to converge to optimal modes at each scale (App. 3.8). Fig. 18 demonstrates our faster and stable convergence. Table 7 compares the number FLOPS required by the models for batch size of 10.
tasks. Fig. 10, 11, 12. 13, 16 and 17
5.4 DENOISING OF 3D OBJECTS IN SPECTRAL SPACE
Spectral moments of 3D objects provide a compact representation, and help building fight-weight networks (Ramasinghe et al., 2020; 2019b; Cohen al., 2018; Esteves al., 2018). However, spectral information of 3D objects has not been used before for self-supervised learning, key reason being the difficulty of learning representations in the spectral domain due to the complex structure and unbounded spectral coefficients. Here, we present an efficient pretext task that is conducted in the spectral domain: denoising 3D spectral maps. We use two types of spectral spaces: spherical harmonics and Zernike polynomials (App. 4). We first convert the 3D point clouds to spherical harmonic coefficients, arrange the values as 2D map, and mask or add noise to a map portion (App. 3.12). The goal is recover the original spectral map. Fig. 20 and Table 6 depicts qualitative and quantitative results. We perform favorably well against other methods. To evaluate the learned features, we use Zernike polynomials, as they harmonics (Ramasinghe et al., 2019a). We first train the network on the 55k ShapeNet objects by denoising spectral maps, and then apply the trained network on the ModelNet10 & 40. The features are then extracted from the bottleneck (similar to Sec. 5.3), and fed to a FC classifier (Table 4). We achieve state-of-the-- results in ModelNet40 with simple pretext task.
discriminative compared to spherical
Input
332 x 32
D4 X
G4
122 128
25G 206
Ieput
CF
<VAF
Oum
Figure 20: Qualitative comparison 3D spectral denoising. The results are converted to the spatial domain for a clear visualization.
Figure 19: Scalability: we subsequently add layers to the archi- tecture to be trained on increasingly high-resolution inputs
6 CONCLUSION
Conditional generation in multimodal domains is a challenging task due to its ill-posed nature. In this paper, we propose a novel generative framework that minimizes a family of cost functions during training. Further, it observes the convergence patterns of latent variables and applies this knowledge during inference to traverse to multiple output modes during inference. Despite using a simple and generic architecture, we show impressive results diverse set of tasks. The proposed approach demonstrates faster convergence, scalability, generalizability, diversity and superior representation learning capability for downstream tasks.

---

Published as a conference paper at ICLR 2021
REFERENCES
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Representation learning and adversarial generation of point clouds. arXiv preprint arXiv:170..02392, 2017. 8
Sanjeev Arora and Yi Zhang. Do gans actually learn the distribution? an empirical study. arXiv preprint 1706.
2017. 2
Sanjeev Arora, Andrej Risteski, and Yi Zhang. Do GANs learn the distribution? some theory and empirics, In International Conference on Learning Representations, 2018.
Aayush Bansal, Xinlei Chen, Bryan Russell, Abhinav Gupta, and Deva Ramanan. Pixelnet: Repre- sentation of the pixels, by the pixels, and for the pixels. arXiv preprint arXiv: 2017a. 21
Aayush Bansal, Yaser Sheikh, and Deva Ramanan. Pixelnn: Example-based image synthesis. arXiv preprint arXiv:1708.0553 05349, 2017b. 21
Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, and Gang Hua. Cvae-gan: Fine-grained image generation through asymmetric training. In The IEEE International Conference Computer Vision (ICCV), Oct 2017. 15
Samuel A Barnett. Convergence problems with generative adversarial networks (gans), arXiv preprint arXiv: 13382, 2018. 2,
Simyung Chang, Sk Park, John Yang, and Nojun Kwak. Sym-parameterizee dynamic inference for mixed-domain image translation. In Proceedings of the IEEE International Conference Computer Vision, pp. 1803-4810 2019. 1, 16
Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks.
In Proceedings of the IEEE international conference on computer vision, pp. 1511-1520, 2017a.
Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017b. 15
Casey Chu, Kentaro Minami, and Kenji Fukumizu. Smoothness and stability in gans. arXiv preprint
arXiv:2002.044 04 185, 2020. 2.4 4
Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. arXiv preprint 10X11:1801.10130, 2018. 9
Aditya Deshpande, Jiajun Lu, Mao-Chuang Yeh, Min Jin Chong, and David Forsyth. Learning diverse image colorization. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017. 15
James Driscoll and Dennis M Healy. Computing fourier transforms and convolutions on the 2-sphere. Advances in applied mathematics, 15(2):202-250, 1994. 28
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. Alché-Buc, E. Fox, and R. Garnett (cds.), Advances in Neural Information Processing Systems 32, pp. 3608-3618 Curran Associates, Inc., 2019. 15
Carlos Esteves, Christine Allen-Blanchette, Ameesh Makadia, and Kostas Daniilidis. Learning so (3) equivariant representations with spherical enns. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 52-68, 2018. 9
Ethan Fetaya, Jörn-Henrik Jacobsen, Will Grathwohl, and Richard Zemel. Understanding the limita- tions of conditional generative models. In International Conference on Learning Representations, 2020. 1
Arnab Ghosh, Viveka Kulharia, Vinay P. Namboodiri, Philip H.S. Torr, and Puneet K. Dokania. Multi-agent diverse generative adversarial networks. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018. 15
10

---

Published as a conference paper at ICLR 2021
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems 27, pp. 2672-2680. 2014a. 15
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- fion processing systems, pp. 2672-2680, 2014b. 4, 17
Will Grathwohl, Kuan-Chieh Wang, Joern- Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In International Conference on Learning Representations, 2020. 2
Zhizhong Han, Mingyang Shang, Yu-Shen Liu, and Matthias Zwicker. View inter-prediction gan: Unsupervised representation learning for 3d shapes by learning global shape memories to support local view predictions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 8376-8384, 2019. 8
Yang He, Bernt Schiele, and Mario Fritz. Diverse conditional image generation by stochastic regression with latent drop-out codes. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 406-421, 2018. 5
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image translation. In The European Conference on Computer Vision (ECCV), September 2018. 15
Satoshi lizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Let there be color! joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Transactions on Graphics (ToG), 35(4):1-11, 2016. 5, 6, 15,28
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-imad translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pultern recognition, pp. 1125-1134, 2017. 1,5, 6, 15,
Longlong Jing and Yingli Tian. supe-supe visual feature learning with deep neural networks: survey. IEEE transactions on pattern analysis and machine intelligence, 2020. 2
Salman H Khan, Yulan Guo, Munawar Hayat, and Nick Barnes. Unsupervised primitive discovery for improved 3d generative modeling. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 9739-9748, 2019.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114 2013.
arXiv preprint
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. CoRR, abs/1312.6114, 2014. 15
Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. On convergence and stability of gans. arXiv preprint arXiv: 1o0.:v/1705007.555 2017. 2,
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). 2009. URL http::/tp///e //wp://wwww///..///////////... cs www.ott.ttooooooooooooooooo
html. 8
Hsin-Yine Ying Lee, Hung- Yu Tseng. Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Diverse image-to-image translation via disentangled representations. In The European Conference on Computer Vision (ECCV), September 2018. 15
Soochan Lee, Junsoo Ha, and Gunhee Kim. Harmonizing maximum likelihood with GANs for multimodal conditional generation. In International Conference Learning Representations, 2019. 1, 5, 6, 15
Jingyuan Li, Ning Wang. Lefei Zhang. Bo Du, and Dacheng Tao. Recurrent feature reasoning for image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7760-7768, 2020. 5
11

---

Published as a conference paper at ICLR 2021
Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting- Chg- Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. In Proceedings of the European Conference Computer Vision (ECCV), pp. 85- 2018. 7
Antonio Loquercio, Mattia Segü, and Davide Scaramuzza. A general framework for uncertainty estimation in deep learning. arXiv preprint arXiv:/907.06890 2019. 19
Lars Maaløc, Casper Kaac Sønderby, Søren Kaae Sønderby, and Ole Winther. Auxiliary deep generative models. arXiv preprint arXiv: 162.006602.05733 2016. 1, 15
Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling. In Advances in neural information processing systems, pp. 6548-6558, 2019. 15
Qi Mao, Hsin- Hsin-Ying Lee, Hung- Yu Tseng, Siwei Ma, and Ming- Ming-Hsuan Yang. Mode seeking generative adversarial networks for diverse image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pp. 1429-1437, 2019.
Michaël Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. CoRR, abs/151 511.05440, 2015. 15
Mchdi Mirza and Simon Osindero. Conditional generative adversarial nets. ArXiv, abs/ abs//abl.//aa... 1481.1784 2014. 15
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't 't know? International Conference on Learning Reprexentations, 2019. 1
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012. 21
Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei Efros. Context encoders: Feature learning by inpainting. 2016. 6, 15
Nathanaël Perraudin, Michaël Defferrard. Tomasz Kacprzak, and Raphael Sgier. Deepsphere: Effi- cient spherical convolutional neural network with healpix sampling for cosmological applications. Astronomy and Computing, 27:130-146, 2019. 31
Ekta Prashnani, Hong Cai, Yasamin Mostofi, and Pradeep Sen. Pieapp: Perceptual image- error assessment through pairwise preference. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, pp. 1808-1817, 2018. 21, 23
Sameera Ramasinghe, Salman Khan, and Nick Barnes. Volumetric convolution: Automatic represen- tation learning in unit ball. arXiv preprint 190100990 2019a. 9,31
Sameera Ramasinghe, Salman Khan, Nick Barnes, and Stephen Gould. Representation learning on unit ball with 3d roto-translational equivariance. International Journal of Computer Vision, pp. 1-23, 2019b. 9
Sameera Ramasinghe, Salman Khan, Nick Barnes, and Stephen Gould. Spectral- -anns for high- resolution 3d point-cloud generation. arXiv preprint arXiv: 19/2.0000 2019c. 8, 31
Sameera Ramasinghe, Salman Khan, Nick Barnes, and Stephen Gould. Blended convolution and synthesis for efficient discrimination of 3d shapes. In The IEEE Winter Conference on Applications of Computer Vision, pp. 21-31, 2020. 9
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv: 2015. 1, 15
Herbert E. Robbins. A stochastic approximation method. Annals of Mathematical Statistics, 22:
400-407, 2007. 17
Min-cheol Sagong, Yong-goo goo Shin, Seung wook Kim, Seung Park, and Sung-jea Ko. Pepsi: Fast image inpainting with parallel decoding network. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. 15
12

---

Published as a conference paper at ICLR 2021
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2242, 2016. 23
Jonathan Sauder and Bjame Sievers. Self-supervised deep learning on point clouds by reconstructing space. In Advances in Neural Information Processing Systems, pp. 12942-12952, 2019. 8
Abhishek Sharma, Oliver Grau, and Mario Fritz. Vconv-dae: Deep volumetric shape learning without object labels. In European Conference Computer Vision, pp. 236-250. Springer, 2016. 8
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems 28. 2015.
15
Hoang Thanh-Tungg Truyen Tran, and Svetha Venkatesh. Improving generalization and stability of generative adversarial networks. arXiv preprint arXiv:1902.03984, 2019. 2
Radim Tyležek and Radim Sára. Spatial pattern templates for recognition of objects with regular structure. In Proc. GCPR, Saarbrucken, Germany, 2013. 8
Patricia Vitoria, Lara Raad, and Coloma Ballester. Chromagan: Adversarial picture colorization with semantic class distribution. In The IEEE Winter Conference on Applications of Computer Vision, pp. 2445-2454, 2020. 2, 4, 5, 6, 15, 28
Yi Wang, Xin Tao, Xiaojuan Qi, Xiaoyong Shen, and Jiaya Jia. Image inpainting via generative multi-column convolutional neural networks. In Advances Neural Information Processing Systems 31. 2018. 15
Zhou Wang, Eero Simoncelli, and Alan C Bovik, Multiscale structural similarity for image quality assessment. In The Seventh Asilomar Conference on Signals, Systems & Computers, 2003, volume 2, pp. 1398-1402. Ieee, 2003. 21
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4)6000-612, 2004. 21
You Xie, Erik Franz, Mengyu Chu, and Nils Thuerey, tempoGAN: A Temporally Coherent, Volumet- ric GAN for Super-resolution Fluid Flow, ACM Transactions on Graphics (TOG), 37(4):95, 2018. 15
Dingdong Yang. Seunghoon Hong, Yunseok Jang, Tianchen Zhao, and Honglak Lee. Diversity- sensitive conditional generative adversarial networks. arXiv preprint urXiv:1901.09000 2019.
5
Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2018. 8
Jiahui Yu, Zhe Lin, Jimei Yang. Xiaohui Shen, Xin Lu, and Thomas S. Huang. Generative image inpainting with contextual attention. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018a. 15
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5505-5514, 201 8b.
Yanhong Zeng, Jianlong Fu, Hongyang Chao, and Baining Guo. Learning pyramid-contex, encoder network for high-quality image inpainting. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1486-1494, 2019. 2, 4, 23,
Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang. Fsim: feature similarity index for image quality assessment. IEEE transactions Image Processing, 20(8):2378-238) 223266.. 2011 1. 21
13

---

Published as a conference paper at ICLR 2021
Richard Zhang. Phillip Isola, and Alexei A Efros. Colorful image colorization. In European
conference on computer vision, pp. 149-666. Springer, 2016. 1,5, 6, 15, 21, 28
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018. 21, 23
Song-Yang Zhang, Zhifei and Hairong Qi. Age progression/regresionn by conditional adversarial autoencoder. In IEEE Conference Computer Vision and Pattern Recognition (CVPR). IEEE, 2017.
Xian Zhang, Xin Wang, Bin Kong, Youbing Yin, Qi Song, Siwei Lyu, Jiancheng Lv, Canghong Shi, and Xiaojie Li. Domain embedded multi-model generative adversarial networks for image-based face inpainting. ArXiv, abs/2002.02999, 2020. 15
Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros. Generative visual manipula- tion on the natural image manifold. Proceedings of European Conference on Computer Vision (ECCV), 2016. 15
Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. Advances in Neural Information Processing Systems 30. 2017a. 6, 15
Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. In Advances in neural information processing systems, pp. 465-476 2017b. 5
14


Reviews:
[{"review_id": "zL5AHm0azHV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Quality：\nThe proposed general-purpose framework for modeling CMM spaces is worthwhile and insightful. By using a set of domain-agnostic regression cost functions instead of the adversarial loss, it improves both the stability and eliminates the incompatibility between the adversarial and reconstruction losses, allowing more precise outputs while maintaining diversity.\n\nHowever, it would be interesting to see the qualitative and quantitative comparison with the latest related works. For example, the following two CVPR2020 papers(For reference only):\n[1] Zheng C, Cham T J, Cai J. Pluralistic image completion[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019: 1438-1447.\n[2] Li J, Wang N, Zhang L, et al. Recurrent Feature Reasoning for Image Inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 7760-7768.\n\nClarity：\nThe paper is very well-written and well-structured and is friendly for readers to understand. It starts off by pointing out the key shortcomings of the use of a combination of reconstruction and adversarial losses and then clarifies the framework proposed in this paper for modeling CMM spaces. Continuously, it explains the drawbacks of conditional GAN methods and illustrates the idea via a toy example. After that, with an extensive set of experiments, this paper experiment on several such tasks with different datasets. It illustrates the outperformance in both qualitative and quantitative experimental comparison and demonstrates state-of-the-art performance.\n\nBy the way, it may be more friendly for readers to read the paper if the layout of diagrams and tables can be reformatted in a clearer way. \n\nOriginality and Significance：\nDue to the ill-posed nature of conditional generation in multimodal domains, this paper proposed a novel generative framework with a simple and generic architecture instead of the adversarial loss. The proposed approach demonstrates faster convergence, scalability, generalizability, diversity, and superior representation learning capability for downstream tasks. At the same time, the comparable performance has been validated on different datasets both quantitatively and qualitatively. And in most of the experiments, it achieves state-of-the-art performance.\n\nBased on the above considerations, I think it is a good paper that marginally above the acceptance threshold. And it may be worthwhile to be accepted if more latest experimental comparison can be shown, and still have the outperformance.\n\n-----------------------------------------------------------------------------------------------\nCompared with the latest related work, the author added the qualitative and quantitative comparison against \"Recurrent Feature Reasoning for Image Inpainting\" in the image completion task. And it shows the outstanding performance of this paper. \nThey have also reformatted the paper so the paper becomes clearer for readers to read. \nConsider the above all, I think it’s a good paper and is worthwhile to be accepted.\n\n\nSo I improving my rating to “7: Good paper, accept”\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "pL7cEXOg1T", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a family of cost functions and a framework for modeling a continuous multimodal (CMM) space.\nThe proposed model converges more stably and faster than conventional methods and shows high-quality results in several tasks. Also, this method can generate diverse outputs at inference with a single model. It was partially possible with many GAN methods, but there is a significant improvement in diversity.\n\n- Clear motivation and well-defined method\\\nThe problem of modeling the CMM space is well defined, and the limitation of the previous methods are described in detail. And the intuition to resolve this problem is also highly convincing.\n\n- Various and extensive experiments\\\nThe experimental settings and results support the effectiveness of the proposed method. The toy example in Figure 3 clearly shows that the proposed method can model CMM space properly.\nEach experiment for downstream tasks also has a detailed explanation, showing good qualitative and quantitative performance.\n\nThere are no major weaknesses in the overall content. However, it is necessary to correct that table 5 and table 6 are overlapped, and many figures and tables are mixed in a somewhat disorderly manner.\n\nIt is somewhat similar to [1,2] in that a separate variable is defined for generating multiple outputs in a deterministic function. It's good to discuss this.\n\nIt would be great if there is a follow-up study to see if it could be extended to generate an image from random variables, like Conditional GAN. And some naming is required to represent the method.\n\n[1] Auxiliary deep generative models, L. Maaløe, ,et al.\n[2] Sym-parameterized Dynamic Inference for Mixed-Domain Image Translation, S. Chang, et al.", "extra_scores": {}, "extra_text": {}}, {"review_id": "owqPRvyqAr8", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nIn this paper, the authors proposed a general-purpose framework for conditional generation in multimodal space. The proposed method is optimized to find multi-modal optimal solutions at inference time. This general method can be applied to a lot of down-stream tasks, improving inference performance without the need to carefully design a network structure.\n\nPros:\n1. The proposed method is very general. It is evaluated on quite a few different tasks, including sketch to image generation, image imputation. image colorization, etc.\n2. The proposed method shows superior results both qualitatively and quantitatively compared with baseline models. It is quite impressive how comprehensive the evaluation is. The multi-modality perspective is especially well presented.\n\nCons:\n1. The authors should carefully audit the paper. There are quite a few typos and presentation flaws, which makes it much harder to read. For example, the numbering of the figures is not consistent. There is not figure 2, 4, 6, 8 etc. \n2. I understand that the authors want to show as many results as possible, but the presentation becomes very crowded, especially on page 8. I would recommend the authors to leave the critical experimental results here and move some to the supplementary materials.\n3. Despite the impressive experimental results, I think readers would benefit more from a deep discussion. For example, what is the relationship between this model with a VAE model? The similarity is of course the continuous latent space. By varying the latent factor, VAE models can generate diverse data as well. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "PqvdAIZa-EL", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addressed the deterministic inference issue and enabled the conditional generation in multimodal spaces. The authors also explained the `` 'generalizability' advantage over cGANS, that is capable of learning more task-agnostic representations.\nThis paper is very dense. The authors conducted many experiments to show 1) the presented conditional generative modeling approach is capable of learning diverse representation and hence lead to diverse generation, 2) the proposed methods worked quite well across different tasks, both subjectively and objectively. Ablation studies are also performed to show ```'momentum' as a supplementary aid” is helpful.\n\nAt this moment, I don’t have major concerns regarding this submission. I incline to accept this paper, and am willing to further change my rating. Below are some minor comments:\n1.\tBased on the description in section 2 to 4, and also the strong experimental results, I am convinced that the proposed approach converges faster than cGAN, and can learn diverse representations and enables multimodal generation. However, it is a bit surprising/interesting that proposed method worked consistently better than other prior methods in almost all the downstream tasks explored. Can you elaborate more on why learned representations worked consistently well in almost all downstream tasks that have tried in the paper.\n2.\tThe experimental study part is very dense. It would be good to have a short section clearly compare the model size/capacity of your models and the counterpart models. Also, is it good to move at least one model architecture you used to main text?\n3.\tThe authors can consider comparing and contrasting with the normalizing flow related work. When normalizing flow applied for inference/learning latent representations, it seems (weakly) related to your design described on section 2.1\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a model and a training mechanism for multimodal generation. The reviews are generally positive: they praise the generality of the method, the extensive experimental evaluation, and the good empirical results. Overall, no major concerns were raised, and all reviewers recommend acceptance.

A couple of concerns remain, in my view:
- The method is generally heuristic, and intuitively rather than theoretically motivated. This is compensated of course by the empirical evaluation, which is thorough.
- The paper could be better written. The reviewers suggested some minor improvements which were implemented in the updated version, but I believe there is room for further improvement.

Due to the above concerns, I consider the rating of reviewer #3 (10: Top 5% of accepted papers, seminal paper) to be unjustifiably high. On balance, however, I'm happy to recommend acceptance.

Message to the authors:

In the abstract you write: "a simple generic model that can beat highly engineered pipelines". Please be aware that the word "beat" evokes competition, winners and losers, so it's not appropriate in the context of scientific evaluation. Please consider replacing it with something neutral, such as "a simple generic model that can perform better than ...".
Decision: Accept (Poster)