Generative Restricted Kernel Machines
Authors: Arun Pandey
Joachim Schreurs
Johan A.K. Suykens
Venue:  (2020)
Keywords: Generative models
Kernel methods
Deep learning

Abstract: We introduce a novel framework for generative models based on Restricted Kernel Machines (RKMs) with multi-view generation and uncorrelated feature learning capabilities, called Gen-RKM. To incorporate multi-view generation, this mechanism uses a shared representation of data from various views. The mechanism is flexible to incorporate both kernel-based, (deep) neural network and convolutional based models within the same setting. To update the parameters of the network, we propose a novel training procedure which jointly learns the features and shared representation. Experiments demonstrate the potential of the framework through qualitative evaluation of generated samples.

Under review as a conference paper at ICLR 2020
GENERATIVE RESTRICTED KERNEL MACHINES
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce novel framework for generative models based on Restricted Kernel Machines (RKMs) with multi-view generation and uncorrelated feature learning capabilities, called Gen-RKM. To incorporate multi-view generation, this mech- anism uses a shared representation of data from various views. The mechanism is flexible to incorporate both kernel-based, (deep) neural network and Convo- lutional based models within the same setting. To update the parameters of the network, we propose a novel training procedure which jointly learns the features and shared subspace representation. The latent variables are given by the eigen- decomposition of the kernel matrix, where the mutual orthogonality of eigenvec- tors represents uncorrelated features. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of generated sam- ples various standard datasets.
INTRODUCTION
In the past decade, interest in generative models has grown tremendously, finding applications in multiple fields such as, generated art, on-demand video, image denoising (Vincent et al.] (011))
exploration in reinforcement learning (Florensa et al. 2018), collaborative filtering (Salakhutdinoy et al. 2007), inpainting (Yeh al. 2017) and many more.
Some examples of graphical models based on a probabilistic framework with latent variables are Variational Auto-Eincoders (Kingma & Welling 2014) and Restricted Boltzmann Machines (RBMs) (Smolensky] 1986; [Salakhutdinov & Hinton 2009). More recently proposed models are based on adversarial training such as Generative Adversarial Networks (GANs) (Goodfellow et al. 2014) and its many variants. Furthermore, auto-reggt rugo-reeresssve models such as Pixel Recurrent Neural Networks (PixeIRNNs) (Van Den Oord et al., 2016) model the conditional distribution of every individual pixel given previous pixels. All these approaches have their own advantages and disadvantages. For example, RBMs perform both learning and Bayesian inference in graphical models with latent vari- ables. However, such probabilistic models must be properly normalized, which requires evaluating intractable integrals over the space of all possible variable configurations (Salakhutdinov Hinton, 2009). Currently GANs are considered the tfte-of-theear for generative modeling tasks, pro- ducing high-quality images but are more difficult to train due to unstable training dynamics, unless more sophisticated variants applied.
Many datasets are comprised of different representations of the data, or views. Views can corre- spond different modalities such as sounds, images, videos, sequences of previous frames, etc. Although each view could individually be used for learning tasks, exploiting information from all views together could improve the learning quality (Pu et al., 2016 Liu & Tuzel 2016: Chen & De- noyer 2017). Also, it is among the goals of the latent variable modelling to model the description of data terms of uncorrelated or independent components. Some classical examples are Indepen- dent Component Analysis; Hidden Markov models (Rabiner & Juang 1986); Probabilistic Principal Component Analysis (PCA) (Tipping & Bishop, 1999); Gaussian- Celussian-Process Latent variable model (Lawrence] 2005) and factor analysis. Hence, when learning a latent space in generative models, it becomes interesting to find a disentangled representation. Disentangled variables are generally considered to contain interpretable information and reflect separate factors of variation in the data for e.g. lighting conditions. style, colors, etc. The definition of disentanglement in the literature is not precise, however many believe that representation with statistically independent variables is good starting point (Schmidhuber) 1992, Ridgeway 2016). Such representations extract informa- tion into compact form which makes it possible to generate samples with specific characteristics

---

Under review as a conference paper at ICLR 2020
(Chen et al. 2018 Bouchacourt et al. 2018 Tran al. 2017 Chen al. representations have been found to generalize better and be more robust against adversarial attacks (Alemi al. 2017).
2016). Additionally, these
In this work, we propose an alternative generative mechanism based on the framework of Restricted Kernel Machines (RKMs) (Suykens 2017), called Generative RKM (Gen-RKM)) RKMs yield representation of kernel methods with visible and hidden units establishing links between Kernel PCA, Least-Squares Support Vector Machines (LS-SVM) (Suykens al. 2002) and RBMs. This framework has a similar energy form as RBMs, though there is non-probabilistic training proce- dure where the eigenvalue decomposition plays the role of normalization. Recently, Houthuys &
Suykens (2018) used this framework to develop tensor- rensor-based multi-view classification models and Schreurs & Suykens 2018) showed how kernel PCA fits into this framework.
Contributions. 1) A novel multi-view generative model based on the RKM framework where mul- tiple views of the data can be generated simultaneously. 2) Two methods are proposed for computing the pre-image of the feature vectors: with the feature map explicitly known unknown. We show that the mechanism is flexible to incorporate both kernel-based, (deep) convolutional neural network based models within the same setting. 3) When using explicit feature maps, we propose training algorithm that jointly performs the feature-selection and learns the common -anumon--ubspace representation in the same procedure. 4) Qualitative and quantitative experiments demonstrate that the model is capable of generating good quality images of natural objects. Further experiments on multi-view datasets exhibit the potential of the model. Thanks to the orthogonality of eigenvectors of the kernel matrix, the learned latent variables are uncorrelated. This resembles disentangled representation, which makes it possible generate data with specific characteristics.
This paper is organized as follows. In Section 2 we discuss the Gen-RKM training and generation mechanism when multiple data sources are available. In Section 3 we explain how the model incorporates both kernel methods and neural networks through the use of implicit and explicit feature maps respectively. When the feature maps are defined by neural networks, the Gen-RKM algorithm is explained in Section 4 In Section 5] we show experimental results of our model applied on various public datasets. Section 6 concludes the paper along with directions towards the future
work. Additional supplementary materials are given in the Appendix A
2 GENERATIVE RESTRICTED KERNEL MACHINES FRAMEWORK
The proposed Gen-RKM framework consists of two phases: a training phase and a generation phase which occurs after another.
2.1 TRAINING
Similar to Energy-Based Models (EBMs, see LeCun al. (2004) for details), the RKM objective function captures dependencies between variables by associating scalar energy to each configura- energy function in which the observed con-
tion of the variables. Learning consists of finding figurations of the variables are given lower energies than unobserved ones. Note that the schematic representation, shown in Figure similar to Discriminative RBMs (Larochelle & Bengio 2008) and the objective function J, (defined below) has an energy form similar to RBMs with additional regularization terms. The latent space dimension the RKM setting has a similar interpretation as the number of hidden units in restricted Boltzmann machine, where in the specific case of the RKM these hidden units are uncorrelated.
We assume a dataset D = {æ,, y. J.N. with E Rd y/ IR comprising of N data points. Here y; may represent an additional view of 2,, e.g., an additional image from a different angle, the caption of image
class label. Starting from the RKM interpretation of Kernel PCA, which gives upper bound on the equality constrained Least-Square Kernel PCA objective function (Suykens) 2017) and applying the feature-maps Φ1 Rd + Rd and de T&P + RR to the input data points,
2

---

Under review as a conference paper at ICLR 2020
F
oxf-)
Vi()
dz(+)
Figure 1: Gen-RKM schematic representation modeling common subspace H between two data sources X' and V. The Φ1, Φe are the feature maps (Fx and Fg represent the feature-spaces) corre- sponding to the two data sources. While 1h, 1/2 represent the pre-image maps. The interconnection matrices U, V model dependencies between latent variables and the mapped data sources.
the training objective function J, for generative RKM is given by
Ji = Σ 1
(Zx
-2( Uh UUh;
A h hi)) + Tr(U U) ( Vh, +
12 Tr(V V). (1) 2
2
where U E XR and V E xR are the unknown interaction matrices, and h, E R are the latent variables modeling common subspace H. between the two input spaces X' and y (see Figure
D. The derivation of this objective function is given in the Appendix A.
Given 1/1 0 and 7/2 0 regularization parameters, the stationary points of Je are given by:
a 0 =
Xh; = U Φ1 (x.) + V 02(2)), Vi I.....N
0h.
0Jr 0
= U 1 - (xx
30
n
03,
= 0
V E 22 (v:)
ØV
Substituting U and V in the first equation above, denoting = S N, yields the following eigenvalue problem:
E R with
1
KI
K2 H H
(3)
171
where H [_]..... AN] R with N is the number of selected principal components and data source Based on Mercer's theo- rem (Mercer 1909). positive-definite kernel functions k1 Rd x IRd + I, K2 Rº x R2 + R can be defined such that kj xy) (Ø1 (z;). ). Φ1 (x,) )), and (2 yy = (0z(y)), dh (y,)). )), Vi,j 1.....N .N forms the elements of corresponding kernel matrices. The feature maps and Φ2. map- ping the input data the high-dimensional feature space (possibly infinite) are implicitly defined by kernel functions. Typical examples of such kernels are given by the Gaussian RBF kernel k(I,, 2;)) -- -1= (1,12(222)) or the Laplace kernel k( 2.y) e ---laa-=,,ll. lla/ just name few (Scholkopf & Smola (001). However, one can also define explicit feature maps, still preserving the positive-definitenes of the kernel function by construction (Suykens et al. 2002).
K1, K2 E N are the kernel matrices corresponding
2.2 GENERATION
In this section, we derive the equations for the generative mechanism. RKMs resembling energy- based models, the inference consists in clamping the value of observed variables and finding con- figurations of the remaining variables that minimizes the energy (LeCun et al.] 2004). Given the
For convenience, is assumed that all the feature vectors centered in the feature space using Ō(x)) ((x)) ΣN Φ(æ:). Otherwise, centered kernel matrix could be obtained using Eq. 17 (Appendix A.4 WWiile above section we have assumed that only two data sources (namely and })) are available for learning, the above procedure could be extended to multiple data-sources. For the M views or data-sources,
this yields the training problem:
Ei=1 TI
M
Kr H"
H A.
3

---

Under review as a conference paper at ICLR 2020
learned interconnection matrices U and V, and given latent variable h", consider the following objective function:
- - - Un* -02(1) Vh +
2 220((²²)) x1(x")) "0(0=")((((""") + 2 "2((""))"(x") o2(3"),
(4)
with additional regularization term data sources. Here Jg denotes the objective function for generation. The given latent variable h* can be the corresponding latent code of a training point, a newly sampled hidden unit specifically determined one. Above cases correspond to generating the reconstructed visible unit, generating random new visible unit or exploring the latent space by carefully selecting hidden units respectively. The stationary points of Jg are characterized by:
03, (2 8J
$1(x") = Uh
(S)
x2(1²) Vh
002()))
Using U and V from Eq. 2 we obtain the generated feature vectors:
(1x"))
1 7/
s (a, h", 22(1") = 1/2
- (2 h:
(6)
To obtain the generated data, one now needs to compute the inverse images of the feature maps (-(+) and (b(-) in the respective input spaces, i.e. solve the pre-image problem. We seek to find the functions th IR4y + IRd and t/2: RPs + Rº corresponding to the two data-sources, such that (v/1 (x=) = xx and (th2 a o2) Re y², where Φ1 (xx" and d2 (y") are calculated using Eq. 6
When using kernel methods, explicit feature maps are not necessarily known. Commonly used kernels such as the radial-basis function and polynomial kernels map the input data very high dimensional feature space. Hence finding the pre-image, in general, is known to be an ill-conditioned
problem (Mika et al. 1999). However, various approximation techniques have been proposed (Bui ett al. 2019 Kwok & Tsang] 2003 Honeine Richard [001] Weston et al. 2004) which could be used to obtain the approximate pre-image æ of Φ1 (1 In section 3.1 we employ one such technique to demonstrate the applicability in our model, and consequently generate the multi-view data. One could also define explicit pre-image maps. In section we define parametric pre-image maps and learn the parameters by minimizing the appropriately defined objective function. The next section describes the above two pre-image methods for both cases, i.e., when the feature map explicitly known or unknown, in greater detail.
IMPLICIT & EXPLICIT FEATURE MAP
3.1 IMPLICIT FEATURE MAP
As noted in the previous section, since x² may not exist, we find an approximation à. A possible technique is shown by Schreurs & Suykens 2018). Left multiplying Eq, by &i (æ.)) T and (21))
Vi 1, I,...,N, we obtain:
1 K, HH h²; by 1 KK, k
(7)
71
where, kn- [k<211 xx),, "..,.... k(aN. aanNaa
represents the similarities between & (x*) and training NxN represents the centered kernel matrix of X.
data points in the feature space, and K1 Similar conventions follow for ) respectively. Using the kernel-smoother method (Hastie et al.) 2001], the pre-images are given by:
a = 4/2 (01(2)) = E)= (ay af Ej= (x(x)
ÿ= U2 (dz(1))) = Re (8)
E K2(
where ki xx,, and kz y²) are the scaled similarities (see Eq. 8 between 0 and 1 and n, the number of closest points based the similarity defined by kernels k and kzz
4

---

Under review as a conference paper at ICLR 2020
3.2 EXPLICIT FEATURE MAP
While using an explicit feature map, Mercer's theorem is still applicable due to the positive semi- definiteness of the kernel function by construction, thereby allowing the derivation of Eq. 3 In the experiments, we use a set of (convolutional) neural networks as the feature maps (ee()) Another (transposed convolutional) neural network used for the pre-image map the (+) (Dumoulin & Visin 2016). The network parameters {0, {} are learned by minimizing the reconstruction errors defined by La 111, (δle, (z))) and C2 2/2c, (#20, (y))). our experiments, we use the mean-squared errors L1 (I,, 111,, (0ie, (x))) = -==ll ||= 1h,, (010, (a:))]l2 and L2 1/2c, (#20, (v))
1. +E yi D22co (02ag (w:)) however, in principle, one can use any other loss appropriate to the dataset. Here @lo, (z;) and d22_ (3:) are computed from Eq. 6 i..., the generated points in feature space from the subspace H.
N
Adding the loss function directly into the objective function Te is not suitable for minimization. Instead, we use the stabilized objective function defined as Tstab = Js + Catub JJ², where Cytab E R+ is the regularization constant (Suykens 2017). This tends to push the objective function Te towards zero, which also the case when substituting the solutions X, h; back into J. (see Appendix [A.3] for details). The combined training objective is given by:
N
min Jo Istab + 01, 02.
Cace 2N
Σ
(ai, (0le, (ax))) + L2(yis 1/2c (d2o, (w()))
where Cace E RR+ is regularization constant to control the stability with reconstruction accuracy. In this way, we combine feature-selection and subspace learning within the same training procedure.
There is also intuitive connection between Gen-RKM and autoencoders. Namely, the properties of kernel PCA resemble the objectives of the variations of an autoencoder: standard (Kramer) 1991). VAE (Kingma& & Welling 2014) and 8-VAE (Higgins et al. 2017). 1) Similar to autoen- coder, Gen-RKM minimizes the reconstruction error in the loss function (see Eq. 99. where kernel PCA which acts denoiser (the information is compressed in the principal components). 2) By interpreting kernel PCA within the LS-SVM setting (Suykens et al. 2002). the PCA analysis can take the interpretation of a one-class modeling problem with maximizes the variance (Suykens et al. (000)). When choosing a good feature map, one expects the latent variables to be normally distributed around zero. This property resembles the added reg- ularization term in the objective of the VAE (Kingma & Welling 2014), which is expressed as the Kullback-Leibler divergence between the encoder's 's distribution and a unit Gaussian as a prior on the latent variables. 3) Kernel PCA gives uncorrelated components in feature space. While it was already shown that PCA does not give a good disentangled representation for images (Eastwood & Williams| 2018 Higgins et al., 2017). Hence by designing good kernel (through appropriate feature-mapp and doing kernel PCA, it is possible to get a disentangled representation for images as we show on the example in Figure 5 The uncorrelated components enhances the interpretation of the model.
target value around which one
4 THE GEN-RKM ALGORITHM
Based on the previous analysis, we propose a novel algorithm, called the Gen-RKM algorithm, combining kernel learning and generative models. We show that this procedure is efficient to train and evaluate.
also scalable to large datasets when using explicit feature maps. The training procedure simultaneously involves feature selection, common-subxpace learning and pre-image map learning. This is achieved via an optimization procedure where one iteration involves an eigen- decomposition of the kernel matrix which is composed of the features from various views (see Eq. 3). The latent variables are given by the cigenvectors, which are then passed via a pre-image map to reconstruct the sample. Figure T shows schematic representation of the algorithm when two data sources are available.
Thanks to training in mini-batches, this procedure is scalable large datasets (sample size N) with training time scaling super-linearly with T,, c I instead of T₁ cNV1, where y zs 3 for algorithms based on decomposition methods, with some proportionality constant c. The training time could be further reduced by computing the covariance matrix (size (dd instead
5

---

Under review as a conference paper at ICLR 2020
of kernel matrix (size N ). when the sum of the dimensions of the feature-spaces is less than the samples in mini-batch i.c. dy + ps S While using neural networks as feature maps, d, and Pf correspond to the number of neurons in the output layer, which are chosen as hyperparameters by the practitioner. Eigendecomposition of this smaller covariance matrix would yield U and V as eigenvectors (see Eq. 10 and Appendix A.2 for detailed derivation), where computing the h; involves only matrix-multiplication which readily parallelizable on modern GPUs:
t
T
d + y
U V
U
Φx == [ ( .................... 01 (xx)]
T T
m
T +
A,
(10)
1 1/2
by (02(y)) ( x2(N))
T2
Algorithm Gen-RKM
Input: be). mh 72. feature map o5(+)) explicit implicit via kernels (, for (1,2) Output: Generated data x",, y
I: procedure TRAIN
procedure GENERATION
2:
if (bg()) = Implicit then
2: Select h"
Hyperparameters: kernel specific
(+)()) Implicit then
4:
Solve Eq.
4: Hyperparameter: Tir
5:
Select principal components
5: Compute hz ky (Eq.
6:
else if =xppiiiiiii then
Explicit
6: Get Φ (Eq. g)
while not converged do
else (.)+) Explicit then
8:
{æ, y +- {Get banc-
8: do steps 11-12
9:
(1z) + æ; (22 +
9: end
10:
do steps 4-5
10:
end procedure
11:
{di (a)), db(y)) t h (Fq.
(a. y) + (6) (0)(x)), th2 (z2()(
13:
AO, x Te A0, x VozJo
14:
AC: x Je: A62 Vc
15:
end while
16:
end if
17:
end procedure
EXPERIMENTS
To demonstrate the applicability of the proposed framework and algorithm, we trained the Gen- RKM model variety of datasets commonly used to evaluate generative models: MNIST (Le- Cun & Cortes 2010), Aushion-MNIS" (Xiao et al.) 2017), CIFAR-10 (Krizhevsky) 2009), CelebA
(Liu al., 2015). Dsprites (Matthey al.] 2017) and Teapot (Eastwood & Williams 2018). The experiments were performed using both the implicit feature map defined by Gaussian kernel and parametric explicit feature maps defined by deep neural networks, either convolutional fully con- nected. As explained in Section in case of kernel methods, training only involves constructing the kernel matrix and solving the eigenvalue problem in Eq. 3 In our experiments. we fit a Gaussian mixture model (GMM) with 1 components to the latent variables of the training set, and randomly sample a new point h" for generating views using kernel smoother. case of explicit feature maps, define L1.. and v'1c, as convolution and transposed-convolution neural networks, respectively (Dumoulin & Visin (2066); and d2e,, and uhic, as fully-connectes networks. The particular archi- tecture details are outlined in Table the Appendix. The training procedure in case of explicitly defined maps consists of minimizing J. using the Adam optimizer (Kingma & Ba 2014) to update the weights and biases. To speed-up learning, we subdivided the datasets into m mini-batches, and within each iteration of the optimizer, Eq. Bis solved to update the value of H. Information on the datasets and hyperparameters used for the experiments is given Table in the Appendix.
Generation:
Qualitative examples: Figure 2 shows the generated images using a convolutional neural network and
neural network the feature map and pre-image map respectively. The first column yellow-boxt rellow-boxes shows the training samples and the second column on the right shows the reconstructed samples. The other images shown are generated by random sampling from

---

Under review as a conference paper at ICLR 2020
5
S
7
1 4
7
9
3
3
6
2 5 3 5 4 0
8 2 0
6
6
/
8
4
4
7
5
5
9
(a) MNIST
(b) Fashion Fashion-MNIST
(c) CIFAR-10
(d) CelebA
5
5'
(e) Bilinear interpolation: MNIST
(f) Bilinear interpolation: CelebA
Figure 2: Generated samples from the model using CNN as explicit feature map in the kernel func- tion. (a). (b), (c). (d) the yellow boxes in the first column show training examples and the adjacent boxes show the reconstructed samples. The other images (columns 3-6) are generated by random sampling from the fitted distribution over the learned latent variables. (e) and (I) shows the generated images through bilinear interpolations in the latent space.
Male
Moush elightly open No heard Narrow eyes auilliug
Young No. heard Narrow agoes P'olock slandow
No beard Big mose Mouth alighely open Oval Gooc
Vlage under eye Narrow eyex.
Side Luno
Figure Multi-view generation on CelebA dataset showing images and attributes.

---

Under review as a conference paper at ICLR 2020
8
7
3
6
3
4
5
/
8
3
6
3
4
2
5
(a) MNIST: Implicit feature maps with Gaussian kernel are used during training. For generation, the pre-images are computed using the kernel-smoother method.
9
6
4
8
2
917
7
4
&
2
J 3 7
(b) MNIST: Explicit feature maps and the corresponding pre-image maps are defined by the Convolutional Neural Networks.
horse bird horse dog cat plane horse se bird car plane ship deer
(c) CIFAR-10: Explicit feature maps Convolutional Neural Networks, Pre-images computed using Transposed CNNs.
Figure 4: Multi-view Generation (images and labels) on various datasets using implicit and explicit feature maps.
GMM over the learned latent variables. Notice that the reconstructed samples are of better quality visually than the other images generated by random sampling. To elucidate that the model has merely memorized the training examples, we show the generated images via bilinear-interpolations the latent space [n]
Comparison: We compare the proposed model with the standard VAE (Kingma & Welling 2014). For fair comparison, the models have the same encoder/decoder architecture, optimization param- eters and trained until convergence, where the details given in Table We evaluate the performance qualitatively by comparing reconstruction and random sampling, the results are shown in Figure X in the Appendix. In order to quantitatively assess the quality of the randomly gener- ated samples, we use the Fréchet Inception Distance (FID) introduced by Heusel et (207)). The results reported in Table [ Experiments were repeated for different latent-space dimensions (haim) and we observe empirically that FID scores are better for the Gen-RKM. This is confirmed by the qualitative evaluation in Table 8 where the VAE generates smoother images. An interesting trend could be noted that as the dimension of latent-space is increased, VAE gets better at gener- ating images whereas the performance of Gen-RKM decreases slightly. This is attributed to the eigendecomposition of the kernel matrix whose eigenvalue spectrum decreases rapidly depicting that most information is captured in few principal components, while the rest is noise. The presence of noise hinders the convergence of the model. It is therefore important to select the number of latent variables proportionally to the size of the mini-bate and the corresponding spectrum of the kernel matrix (the diversity within mini-batch affects the eigenvalue spectrum of the kernel matrix).
Table 1: FID Scores (Heusel et al. 2017) for randomly generated samples (smaller is better).
Dataset
Algorithm
FID score
hasm 10
hdsm
30
hdim 50
Gen-RKM
89,825
130.497
131,696
MNIST
VAE
250
234.749
205.282
Gen-RKM
103.299
84.403
85.121
CelebA
VAE
286.039
245.738
225.783
Multi-view Generation: Figures 3 demonstrate the multi-view generative capabilities of the model. these datasets, labels or attributes are seen as another view of the image that provides extra information. One-hot encoding of the labels was used to train the model. Figure Catureeashhr shows the generated images and labels when feature maps are only implicitly known ie. through a Gaussian kernel. Figures 4c shows the same when using fully-connected networks parametric functions
8

---

Under review as a conference paper at ICLR 2020
to encode and decode labels. We that both the generated image and the generated label matches in most cases, albeit not all.
Figure 5: Exploring the learned sotorelated-feturree by traversing along the eigenvectors. The first column shows the scatter plot of latent variables using the top two principal components. The green lines within, show the traversal the latent space and the related rows show the corresponding reconstructed images.
Table 2: Disentanglement Metric on DSprites and Teapot dataset with Lasso and Random Forest regressor (Fastwood Williams, 2018). For disentanglement and completeness higher score better, for informativeness, lower is better.
Lasso
Random Forest
hdlim
Algorithm
Disent.
Comple.
Inform.
Disent.
Comple.
Inform.
Gen-RKM
0.30
0.10
0.87
0.12
0.10
0.28
DSprites
10
VAE
0.11
0.09
0.17
0.73
0.54
0.06
3-VAE (B
0.53
0.18
0.18
0.58
0.36
0.06
RKM
0.72
0.71
0.64
0.05
0.19
0.03
VAE
0.04
0.01
0.87
0.01
0.13
0.11
3-VAE (B 3)
0.13
0.40
0.71
0.00
0.26
0.09
Gen-RKM
0.28
0.23
0.39
0.48
0.39
0.19
10
VAE
0.28
0.21
0.36
0.30
0.27
0.21
Теаpot
3-VAE (8 = 3)
0.33
0.25
0.36
0.31
0.24
0.20
Gen-RKM
0.22
0.23
0.74
0.08
0.09
0.27
VAE
0.16
0.14
0.66
0.11
0.14
0.28
3-VAE (3
0.31
0.25
0.68
0.13
0.15
0.29
Disentanglement:
Qualitative examples: The latent variables are uncorrelated, which gives an indication that the model could resemble disentangled representation. This is confirmed by the empirical evidence Fig- B where we explore the uncorrelated features learned by the models on the Dsprites and celebA dataset. In our experiments, the Dsprites training dataset comprised of 32 x 32 positions of oval and heart-shaped objects. The number of principal components chosen were 2 and the goal was findout whether traversing along the eigenvectors, corresponds to traversing the generated im-

---

Under review as a conference paper at ICLR 2020
age in particular direction while preserving the shape of the object. Rows 1 and 2 of Figure show the reconstructed images of an oval while moving along first and second principal component respectively. Notice that the first and second components correspond the y and z positions spectively. Rows 3 and 4 show the same for hearts. On the celehA dataset, we train the Gen-RKM with 15 components. Rows 5 and 6 shows the reconstructed images while traversing along the prin- cipal components. When moving along the first component from left-to-right, the hair-color of the women changes, while preserving the face structure. Whereas traversal along the second compo- nent, transforms a man to woman while preserving the orientation. When the number of principal components were 2 while training, the brightness and background light-source corresponds to the two largest variances in the dataset. Also notice that, the reconstructed images are more blurry due to the selection of less number of components to model H.
Comparison: To quantitatively assess disentanglement performance, we compare Gen-RKM with VAE (Kingma & Welling, 2014) and beta- VAE (Higgins et al. 2017) on the Dsprites and Teapot datasets (Eastwood & Williams 2018). The models have the same encoder/decoder architecture, op- timization parameters and are trained until convergence, where the details are given in Table 3 The performance is measured using the proposed framework of [Eastwood & Williams/ 2018). which gives 3 measures: disentanglement, completeness and informativeness. The results are depicted in Table 2 Gen-RKM has good performance on the Dsprites dataset when the latent space dimension is equal to 2. This is expected as the number of disentangled generating factors in the dataset is also equal 2, hence there are no noisy components in the kernel PCA hindering the convergence. The opposite happens in the case hadi 10, where noisy component are present. The above is confirmed by the Relative Importance Matrix on Figure 6 in the Appendix, where the 2 generating factors are well separated in the latent space of the Gen-RKM. For the Teapot dataset, Gen-RKM has good performance when htam = 10. More components are needed to capture all variations in the dataset, where the number of generating factors is now equal to 5. In the other cases, Gen-RKM has a performance comparable to the others.
6 CONCLUSION AND FUTURE WORK
The paper proposes a novel framework, called Gen-RKM, for generative models based on RKMs with extensions to multi-view generation and learning uncorrelated representations. This allows for a mechanism where the feature map can be implicitly defined using kernel functions or explicitly by (deep) neural network based methods. When using kernel functions, the training consists of only solving an eigenvalue problem. In the case of a (convolutional) neural network based explicit feature map, we used (transposed) networks the pre-image functions. Consequently, training procedure was proposed which involves joint feature-selection and subspace learning. Thanks to training mini-batches and capability of working with covariance matrices, the training is scalable to large datasets. Experiments on benchmark datasets illustrate the merit of the proposed framework for generation quality as well as disentanglement. Extensions of this work consists of adapting the model to more advanced multi-view datatsets involving speech, images and texts; further analysis on other feature maps, pre-image methods, loss-functions and uncorrelated feature learning. Finally, this paper has demonstrated the applicability of the Gen-RKM framework, suggesting new research directions to be worth exploring.
'Code and dataset available at att.pp///ghhbbscccm/c/cmsttoooos
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Alex Alemi, lan Fischer, Josh Dillon, and Kevin Murphy. Deep variational information bottleneck. In ICLR, 2017.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. In Thirty-Second AAAI Con- ference on Artificial Intelligence, 2018.
Anh Tuan Bui, Joon-Ku Im, Daniel W. Apley, and George C. Runger. Projection-Free Kernel Prin- cipal Component Analysis for Denoising. Neurocomputing, 2019. ISSN 0925-2312.
Mickaël Chen and Ludovic Denoyer. Multi-view generative adversarial networks. In Joint Euro- pean Conference on Machine Learning and Knowledge Discovery in Databases, pp. 175-188.
Springer, 2017.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In Advances in Neural Information Processing Systems, pp- 2610-2620, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172-2180, 2016.
Vincent Dumoulin and Francesco Visin. A guide to convolution arithmetic for deep learning, arXiv preprint arXiv: 1603.008588 2016.
Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018.
URL httph:tt://stt /ttp:rs//ppeeeevviee net. tnne/fooree htt////rrmmm/// Forrmmidd-y--dae 7d
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic Goal Generation for Reinforcement Learning Agents. In Proceedings of the 35th International Conference on Ma- chine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1515-1528, Stock- Jul 2018. PMLR.
holmsmssan, Stockholm Sweden, 10-15 5
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative Adversarial Nets. Advances in Neural formation Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 2672-2680, 2014.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning. Springer New York Inc., New York, NY, USA, 2001.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by time-scale update rule converge local nash equilibrium. In Proceed- ings of the s1 International Conference on Neural Information Processing Systems, NIPS' 17,
pp. 6629-6640, USA. 2017, Curran Associates Inc. ISBN 778-1-5108-6096-41 URL http:
7/ 7/dl.80 acm. hwtp:a//ggggcatttiioooooooooaoot ffm sotimmidd3222222 3295408
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick,
Shakir Mohamed, and Alexander Lerchner. Beta-VAF: Learning basic visual concepts with a constrained variational framework. ICLR, 2(5)66, 2017.
Paul Honeine and Cedric Richard. Preimage Problem in Kernel-Based Machine Learning. IEEE Signal Processing Magazine, 28(2):77-88, March 2011. ISSN 1053-5888.
Lynn Houthuys and Johan K Suykens. Tensor learning in multi-view kernel PCA In 27th International Conference on Artificial Neural Networks ICANN, Rhodes, Greece, volume 11140, pp. 205-215, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 444266998, 2014.
11

---

Under review as a conference paper at ICLR 2020
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Mark A Kramer. Nonlinear principal component analysis using autoassociative neural networks.
AIChE journal, 37(2):233-243, 1991.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
James T. Kwok and Ivor Wai-Hung Tsang. The pre-image problem in kernel methods. IEEE Trans- actions on Neural Networks, 15:1517-1525, 2003.
Hugo Larochelle and Yoshua Bengio. Classification using discriminative restricted Boltzmann ma- chines. In Proceedings of the 25th International Conference on Machine Learning- ICML '08, pp. 536-543, Helsinki, Finland, 2008. ACM Press. ISBN 978-1-60555-2055
Neil Lawrence. Probabilistic non-linear principal component analysis with gaussian process latent variable models. JMLR, 6:1783 December 2005. ISSN 1532-4435. URL http://dl.
acm. orq/ http:/iiaiiiiionoo cfm? htt.aoo.........000440111.111111
Yann LeCun and Corinna Cortes. 2010.
MNIST
handwritten
digit
database.
URL httpp://yann lecun. wwmmexdb/
mnist//
Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pp. II-97-104 Vol.2, 2004.
Ming-Yu Yu Liu and Oncel Tuzel. Coupled Generative Adversarial Networks. In Advances in Neural Information Processing Systems 29, pp. 469-477. Curran Associates, Inc., 2016.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. https://github.ccm http://tww.oommeeeeeepnpptteeeee 2017.
James Mercer. Functions of Positive and Negative Type, and Their Connection the Theory of Integral Equations. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 209(441-458):411 44544441-454:4454446 January 1909.
Sebastian Mika, Bernhard Schölkopf, Alex Smola, Klaus-Robert Müller, Matthias Scholz, and Gun- nar Rätsch. Kernel PCA and De-noising Feature Spaces. In Proceedings of the 1998 Conference on Advances in Neural Information Processing Systems II, pp. 536-542. MIT Press, 1999.
Yunchen Pu, Zhe Gan, Ricardo Henao, Xin Yuan, Chunyuan Li, Andrew Stevens, and Lawrence Carin. Variational Autoencoder for Deep Learning of Images, Labels and Captions, NIPS' 16, pp. 2360-2368. Curran Associates Inc., USA, 2016. ISBN 978-1-5108-388 10--1-8--8--
Lawrence R Rabiner and Bwing-Hwang Juang. An introduction to Hidden Markov models. JEEE ASSP magazine, 3(1):4-16, 1986.
Karl Ridgeway. A survey of inductive biases for factorial representation-lcammin CoRR, abs/1612.05299, 2016.
Ralph Tyrrell Rockafellar. Conjugate Duality and Optimization. SIAM, 1974.
Ruslan Salakhutdinov and Geoffrey Hinton. Deep Boltzmann Machines. Proceedings of the 12th International Conference on Artificial Intelligence and Statistics, Volume 5 of JMLR, 2009.
Ruslan Salakhutdinov. Andriy Mnih, and Geoffrey Hinton. Restricted Boltzmann machines for collaborative filtering. In ICML 00, pp. 791-798, Corvalis, Oregon, 2007. ACM Press.
12

---

Under review as a conference paper at ICLR 2020
Jürgen Schmidhuber. Learning factorial codes by predictability minimization. Neural Computation, 4(6):863-877 879,, 1992.
Bernhard Scholkopf and Alexander J. Smola. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge, MA, USA, 2001. ISBN 0262194759.
Joachim Schreurs and Johan A. K. Suykens. Generative Kernel PCA. In European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning pp. 139-134, 2018.
Paul Smolensky. Parallel distributed processing: Explorations in the microstructure of cognition, vol. 1. chapter Information Processing in Dynamical Systems: Foundations of Harmony Theory, pp. 194-281. MIT Press, Cambridge, MA, USA, 1986. ISBN 0-262-68053-X.
Johan A. K. Suykens. Deep Restricted Kernel Machines using Conjugate Feature Duality, Neural Computation, 1223:221,,2 August 2017. ISSN 0899-7667, 1530-888X.
Johan A. K. Suykens, Tony Van Gestel, Jos De Brabanter, Bart De Moor, and Joos Vandewalle. Least Squares Support Vector Machines. World Scientific, River Edge, NJ, January 2002. ISBN 978-981-22 1-88-81--151--
Johan A. K. Suykens, Tony Van Gestel, Joos Vandewalle, and Bart De Moor. A support vector ma- chine formulation to PCA analysis and its kernel version. JEEE Transactions on neural networks, 14(2):47-450, 2003.
Michael E. Tipping and Chris M. Bishop. Probabilistic principal component analysis. Journal Of The Royal Statistical Society, series B, 61(():611-622 1999.
Luan Tran, Xi Yin, and Xiaoming Liu. Disentangled representation learning GAN for pose-invariant face recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 1415-1424, 2017.
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel Recurrent Neural Net- works. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML' 16, pp. 1747-1756. JMLR.org, 2016.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with Local Denoising Criterion. Journal of Machine Learning Research, 11:3371-3408, 2010.
Jason Weston, Bernhard Schölkopf, and Gökhan H. Bakir. Learning to Find Pre-Images. In NIPS 16, pp. 449 456. 2004.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: Novel Image Dataset for Bench- marking Machine Learning Algorithms. 2017.
Raymond A. Yeh, Chen Chen, Teck Yian Lim, Alexander G. Schwing, Mark MasseaawaJJhhnon and Minh Do. Semantic image inpainting with deep generative models. In The IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), July 2017.
13

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 DERIVATION OF GEN-RKM OBJECTIVE FUNCTION
Given D (æi Yı j= where a; Rd Ye E Rp and feature-map 61 I - Rdd and Φ : R RP,, the -east-Squares Support Vector Machine (LS-SVM) formulation of Kernel PCA (Suykens)
al. 2002) for the data sources can be written as:
N
min TI(UTU) U) U,V. 2
12 TT 2
1
e
e;
2
(11)
s.t. e, x1 + (1)
Vi =
N,
where U E TRdx and V E TRxx are the interconnection matrices.
Using the notion of conjugate feature duality introduced in Suykens 2017), the error variables e; are conjugated to latent variables h; using:
Tee n hhhze 2A
Ve, h E R
(12)
which is also known as the Fenchel- Young inequality for the case of quadratic functions (Rockafel- lar 1974). By eliminating the variables e; from Eq. [[]and using Eq. [2] we obtain the Gen-RKM
traming objective function:
J.
(21()x) 1=1
"2 A + h4 + th UU+UU)++ Vh, + UU, Uhh-6260 Vh,
Tr(V IV(VV).. V). (13)
2
A.2 KERNEL PCA IN THE PRIMAL
From Eq.
eliminating the variables h; yields the following:
1
(1(x))01
U
di ( V
XU,
7
(14)
25 0z (ax))
U
=V.
1/2
Denote Φx |ob ( di (xN)). by (x)), (2NN) and A = diag , Xa} Raxs with s < N. Now, composing the above equations matrix form, we get the following cigen-decomposition problem:
1
1 T
U
U v
T $
A.
(15)
V
Here the size of the covariance matrix is (dy + pf) x (d, py). The latent variables h; can be computed using Eq. 2 which simply involves matrix multiplications.
A.3 STABILIZING THE OBJECTIVE FUNCTION
Proposition 1. All stationary solutions for H.A in Eq. of Je lead to Te = 0.
Proof. Let >; h, are given by Eq.
Using Eq. to substitute V and U in Eq.
I yields:
N
J(V.U,A, =
h
h
+
Tr
ha (x:)
T
Φn h
2
1=1
i=
f=1
N
N
172 Tr 2
+
h4 (y:)
Se ( h3
112
1=1
1=1
14

---

Under review as a conference paper at ICLR 2020
N
h
h4 +
I1 T
- 1
H
+
72
2
T
HKgH
2
1/
i=1
N
1
h2
h4
+ - Ir
H
K1
- K2
H
1/1
172
i=1
From Eq.
we get:
JV
Σ
X h
2 (HH'
h;
- 1
-
2 h
X
h;
X
h h 0.
2
f=1
i=1
1=1
Proposition 2. Let J(x) TRN + R be smooth function, for all E RN and for RR define ((x) = J(z) + 2 {z)². Assuming (1 + cx)) # 0, then x² is the stationary points of (x) iff x² is the stationary point for ((x).
Proof. Let x* be stationary point of (ax), meaning that vJ((x) = 0. The stationary points for [(x) can be obtained from:
d.J
(VJ(() + cJ = cI)
(16)
dac
dJ dx mm
It is easy to see from Eq. 2 that if x", VJ(x*) 0, we have that
0, meaning that all
the stationary points of J (x) are stationary points of [(x).
To show the other way, let x* be stationary point of J(x) i.e. vJ((x*) = 0. Assuming (1 + (I( o 0, then from Eq. for all e R2 we have
(1+ c(("))))) VJ(x") 0,
implying that v=(x") O.
Based the above propositions, stabilize our original objective function Eq. to keep it bounded and hence is suitable for minimization with Gradient-descent methods. Without the re- construction errors, the stabilized objective function is
min Jr + 5. U.V.
Denoting J Tt Satch Tz. Since the derivatives of Ts given by Eq. the stationary points of J are:
8V = (1 + Tetab
N or, + m V =
= V di
0J
(1+ TotabiT)
DN 02(3)) hT + nna
U Σn" d2(y)h
aU
Xh; V (1
aJ
(1 + CstabrF) (- (-VT (1x)) - U (1))) + Ah.) =
Sh.
U (2(1))
assuming + CstabiJ * O. Elimination of V and yields
1 K1 + K H = H A, which
indeed the same solution for Catab 0 in Eq. T and Eq. 3
A.4 CENTERING OF KERNEL MATRIX
Centering of the kernel matrix is done by the following equation:
K. K N - + 2 K11T,
(17)
where 1 denotes N-dimensional vector of ones and K is either K2 K2.
15

---

Under review as a conference paper at ICLR 2020
A.5 ARCHITECTURE DETAILS
See Table 3 and 4 for details on model architectures, datasets and hyperparameters used in this paper. The PyTorch library in Python was used as the programming language with 8GB NVIDIA QUADRO P4000 GPU.
Table 3: Details of model architectures used in the paper. All convolutions and transposed- convolutions are with stride and padding 1, Unless stated otherwise, the layers have Parametric- RELU (a 0.2) activation function, except the output layers of the pre-image maps which has sigmoid activation function.
Dataset
Optimizer (Adam)
Architecture
Input
28x28x1
10 (One-hot encoding)
Conv 32x4x4;
MNIST
le-3
Feature-man (fim)
Conv 64x4x4;
FC 15, 20 (Linear)
FC 128 (Linear) reverse of fi
Pre-image map Latent space dim.
reverse of fm
500
Input
28x28x1
10 (One-hot encoding)
Fashion -MNIST
Conv 32x4x4;
1e-3
Feature-map
64x4x4;
FC 15, 20
FC 128 (Linear)
Pre-image map (fmm)
reverse of fm
reverse of fm
Latent space dim.
100
Input
32x32x3
10 (One-hot encoding)
Conv 64x4x4;
CIFAR-10
le-3
Feature-map (fm)
Conv 128x4x4;
FC 15, 20
FC 128 (Linear)
Pre-image map Latent space dim. Input
reverse fm
reverse of fm 500
64x64x3
Conv 32x4x4;
CelebA
le-4
Couv 64x4x4;
Feature-man maau (fm)
Conv 128x4x4;
Conv 256x4x4;
FC 128 (Linear)
Pre-image map Latent space dim. Input
reverse of fm
15
64x64x1
Conv 20x4x4;
Dsprites
le-4
Conv 40x4x4;
Feature-map (fm)
Conv 80x4x4;
FC 128 (Linear) reverse of fm
Pre-image map Latent space dim. Input
2/10
64x64x3
Conv 30x4x4;
Teapot
le-4
Conv 60x4x4;
Feature-map (fm)
Cony 90x4x4;
FC 128 (Linear)
Pre-image map Latent space dim.
reverse fm
5/10
16

---

Under review as a conference paper at ICLR 2020
Table 4: Datasets and hyperparameters used for the experiments. The bandwidth of the Gaussian kernel for generation corresponds to the bandwidth that gave the best performance determined by valsdavai on the MNIST classification problem.
Dataset
N
d
Nubsse
m71
0
Thp
MNIST
60000
28 28
5000
500
50
1.3
10
Fashion-MNIST
60000
28 x 28
500
100
5
/
10
CIFAR-10
60000
32 x 32 x
500
500
/
10
CelebA
202599
128 X 128
500
15
5
/
20
Dsprites
737280
64 X 64
1024
2/10
/
Теapot
200000
64 x 64 x 3
1000
5/10
100
/
A.6 BILINEAR INTERPOLATION
Given four vectors h1 h2, h3 and 14 (reconstructed images from these vectors are shown at the edges of Figs. 2e 21 the interpolated vector h" is given by:
h- (1 - (1 - h) + a(1 - h2 + r(1 - a) + yahs, 0 0 1
This h" is then used in step of the generation procedure Gen-RKM algorithm (see Algorithm 1 to compute x²..
A.7 VISUALIZING THE DISENTANGLEMENT METRIC
In this section we show the Hinton plots to visualize the disentaglement scores as shown in Table Following the conventions of Eastwood & Williams (0018). represents the ground-truth data generating factors. Figs. 6 & shows the Hinton plots on DSprites and Teapot datasets using Lasso and Random Forest regressors for various algorithms. Here the square size indicates the magnitude of the relative importance of the latent code h; in predicting 2,.
17

---

Under review as a conference paper at ICLR 2020
Lasso VAE
Random Forest
Gen-RKM
3 VAE
Gen-RKM
VAE
8 VAE
h
h
h
2
8
2
(a) haim
10
Lasso VAE
Random Forest
Gen-RKM
3-VAE
Gen-RKM
VAE
3-VAE
(b) hasm
Figure 6: Relative importance matrix computed by Lasso and Random Forest regaressors
DSprites dataset for hdim = {10, 2} against the underlying data generating factors Zdim = {2} corresponding to x, y positions of object.
Lasso VAE
Random Forest
Gen-RKM
B- B-VAE
Gen-RKM
VAE
8- 3-VAE
h
Dx
(a) hatim 10
Lasso VAE
Random Forest
Gen-RKM
B-VAE
Gen-RKM
VAE
8-VAE
(b) hddm
Figure 7: Relative importance matrix as computed by Lasso and Random Forest regaressors on Teapot dataset for haim 110, 5) against the underlying data generating factors 2dim corresponding to azimuth, elevation and colors red, green and blue the teapot object.
{5)
18

---

Under review as a conference paper at ICLR 2020
A.8
FURTHER EMPIRICAL RESULTS
Gen-R Gen-RKKM
VAE
5
5
8
8
8
8
2 2
5
5
8
8
8
8
Reconstructions MNIST
6
6
9
9
4
4
1
6 6
9
9
4
4
1
3 4
3 45334400
557700'7
5 5
7
77223377
7 77223377
5
5
8
8
/
/
8 8
5 5 8
/
/
8
8
Random Generation MNIST
1
4
6
8
0
9
3
0 7 8
0
2
4
5 8
9
7
9
4 7
6 0 2 0 1
1
1
8 3
8
4
6
6 6 0 3 6
2
1 , 509
3
0 9
1
0
0
7
9
6
3
2 0
5 4 5
1 5
4 7
4
, 4
8 8
2 8 7
0 3 0
0
8
0 4 1
4
9 1 6
7
0
8
Reconstructions CelebA
Random Generation CelebA
Figure 8: Comparing Gen-RKM and standard VAE for reconstruction and generation quality. construction MNIST and reconstruction CelebA, uneven columns correspond to the original image, even columns to the reconstructed image.
19


Reviews:
[{"review_id": "HJgqjEXpFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a model and training framework for generating samples based on restricted kernel machines. It is extended to multi-view generation and uncorrelated feature representation learning. \n\n- The paper is well-written and well-organized. Notations and claims are clear.\n\n- The idea of a multi-view generation model based on restricted kernel machines is interesting. However, the paper seems to be limited to model definition and algorithm overview without a performance evaluating analysis. \n\n\n- The experimental evaluations are not satisfactory. Although it is claimed in the paper that the model is able to generate high quality images, it is very hard to be confirmed with these experiments. There is no concrete attempt at comparing the performance of the model to the other used methodologies. Generating high quality images with multiple views is an interesting problem, and there are good works in the field addressing the issues. To name a few:\nZhu, Z., Luo, P., Wang, X., Tang, X.: Multi-view perceptron: a deep model for learning face identity and view representations. In: Advances in Neural Information Processing Systems (NIPS). pp. 217–225, (2014)\nKan, M., Shan, S., Chen, X.: Multi-view deep network for cross-view classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4847–4855, (2016) \nYin, X., Liu, X.: Multi-task convolutional neural network for pose-invariant face recognition. IEEE Transactions on Image Processing (2017) \nYim, J., Jung, H., Yoo, B., Choi, C., Park, D., Kim, J.: Rotating your face using multitask deep neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 676–684, (2015)\nYu Tian, Xi Peng , Long Zhao, Shaoting Zhang , ,Dimitris N. Metaxas , CR-GAN: Learning Complete Representations for Multi-view Generation, arXiv: 1806.11191, 2018.\n\nThere might be differences between these works and the paper, it is common to evaluate the quality of the generation to other models in terms of accuracy or in the classification tasks. Unfortunately, there is no such quantitative analysis in the paper. So the advantages of the proposed model is not very clear since there is not enough quantitative performance analysis. It would be interesting to see complexity analysis to evaluate the computational costs.\n\nOverall, I do not recommend this paper for publication. The experimental results are not satisfactory, and the paper needs improvements in that regard.\n\n** update:\nI would like to thank the authors for their comments. However, I still see major issues in the paper unresolved and my review remains the same. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJly_2s4cB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "There exists two papers:\n[1] Multimodal Learning with Deep Boltzmann Machines, http://jmlr.org/papers/volume15/srivastava14b/srivastava14b.pdf\n[2] Deep Restricted Kernel Machines Using Conjugate Feature Duality, ftp://ftp.esat.kuleuven.ac.be/stadius/suykens/reports/deepRKM1.pdf\n\nIn particular [2] considers a model, which is similar to a Boltzmann machine, but at the same time it is based on kernel features, and uses structure of the corresponding optimization problem to obtain a solution in a semi-explicit way.\n\nThe authors of the considered paper \n1) generalise a multimodal variant of the Boltzmann machine from [1] (which uses a special cross-product term to take into account dependency between modalities) to the case of kernel machines,\n2) demonstrate on several typical datasets that using explicit deep network features it is possible to model images and data with two modalities (faces/textual description of faces).\n\nComments:\n- From the description of the functionals L1 and L2 (bottom of page 4 and top of the page 5) the reader can think that the authors tune parameters (zeta_1,theta_1) and (zeta_2,theta_2) for each sample point separately\n- The authors claimed that the experiments were done both for kernel features and for explicit features based on neural networks. However, in the experimental section there are no results obtained when using implicit kernels. Nothings is told on how to select kernel parameters\n- The authors claimed that thanks to PCA-like definition of latent vectors they are orthogonal which is similar to disentangle representations. However, there are no any empirical evidences whether it is possible to benefit somehow from that orthogonal property, as well as there is no comparison with approaches to construct disentangleв latent representation for other types of generative models.\n\nConclusions:\n- In general the text is accurately written, the work is well organised.\n- Still I was not able to understand the main idea of the paper. \na) if the main idea of the paper that the authors propose some new method for generative modeling of multi-modal data, then the authors should make significantly more diverse experiments and ablation studies. Actually, this is not the case of the current work. The authors did not provide any quantitate measure and comparison with existing approaches;\nb) if the main idea is to present a new approach, then still I would not call the approach completely new, as it is based on well-known ideas and its benefits are completely not obvious.\n\nI guess that the paper can be published, but only after issues in a) are addressed.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgubSH1qS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is a very good paper, building on the idea of Restricted Kernel Machines (drawing a nice parallel between Restricted Bolzman Machines and tools available in the Kernel modelling literature). In this manuscript, the author(s) extend the work to a generative model setting to achieve multi-view generation -- a generative model that can explain correlated variables from a common subspace. The manuscript is well-written and easy to follow and the algorithmic details are clear. Image generation is illustrated on standard datasets (MNIST / CIFAR / CelebA). \nWhile the framework and learning algorithm are good, and novel extensions to what appears to be previous work of the authors, I am less persuaded by the empirical work. Latent variable-based generative modes such as this (and this is motivated in the introduction to the paper) should be judged on if they can extract anything useful about the problem domain in the latent representations that we can interpret. This is not the case here --  the results presented are examples of images that the models can generate. No critical appraisal is given about when the models might fail or when one ought to resort to this approach and not a sample from the plethora of variants of VAE we read about. What have we learnt about images / hand-written characters / faces of popular people from a study like this?\nFrom the above empirical results point of view, I do not think this manuscript is ready for publication, despite what I see as the elegance of the framework. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a way to use kernel method for multi-view generation. The points are mapped into a common subspace (with CNN feature extractor and kernel on top), and then a generation procedure from a latent point is given. 
I found the paper not easy to ready and follow; the idea of using CNN + kernel methods have been around for some years (for example, see "Impostor networks" by Lebedev et. al), and explicit feature map shows that kernel is just an additional layer to the network. Overall, the approach is straightforward, the generation can be quite slow and the benefits are not clear. The reviewers are mildly negative, so I think this time this paper can not be accepted.
Decision: Reject