Title: MULTI-VIEW GENERATIVE ADVERSARIAL NET-WORKS

Abstract: Learning over multi-view data is a challenging problem with strong practical applications. Most related studies focus on the classification point of view and assume that all the views are available at any time. We consider an extension of this framework in two directions. First, based on the BiGAN model, the Multi-view BiGAN (MV-BiGAN) is able to perform density estimation from multi-view inputs. Second, it can deal with missing views and is able to update its prediction when additional views are provided. We illustrate these properties on a set of experiments over different datasets.

1. INTRODUCTION:

Many concrete applications involve multiple sources of information generating different views on the same object (Cesa-Bianchi et al., 2010) . If we consider human activities for example, GPS values from a mobile phone, navigation traces over the Internet, or even photos published on social networks are different views on a particular user. In multimedia applications, views can correspond to different modalities (Atrey et al., 2010) such as sounds, images, videos, sequences of previous frames, etc... The problem of multi-view machine learning has been extensively studied during the last decade, mainly from the classification point of view. In that case, one wants to predict an output y based on multiple views acquired on an unknown object x. Different strategies have been explored but a general common idea is based on the (early or late) fusion of the different views at a particular level of a deep architecture (Wang et al., 2015; Ngiam et al., 2011; Srivastava & Salakhutdinov, 2012) .

The existing literature mainly explores problems where outputs are chosen in a discrete set (e.g categorization), and where all the views are available. An extension of this problem is to consider the density estimation problem where one wants to estimate the conditional probabilities of the outputs given the available views. As noted by Mathieu et al. (2015) , minimizing classical prediction losses (e.g Mean square error) will not capture the different output distribution modalities.

In this article, we propose a new model able to estimate a distribution over the possible outputs given any subset of views on a particular input. This model is based on the (Bidirectional) Generative Adversarial Networks (BiGAN) formalism. More precisely, we bring two main contributions: first, we propose the CV-BiGAN (Conditional Views BiGAN -Section 3) architecture that allows one to model a conditional distribution P (y|.) in an original way. Second, on top of this architecture, we build the Multi-view BiGANs (MV-BiGAN -Section 4) which is able to both predict when only one or few views are available, and to update its prediction if new views are added. We evaluate this model on different multi-views problems and different datasets (Section 5). The related work is provided in Section 6 and we propose some future research directions in Section 7.

2. BACKGROUND AND GENERAL IDEA:



2.1. NOTATIONS AND TASK:

Let us denote X the space of objects on which different views will be acquired. Each possible input x ∈ X is associated to a target prediction y ∈ R n . A classical machine learning problem is to estimate P (y|x) based on the training set. But we consider instead a multi-view problem in which different views on x are available, x being unknown. Let us denote V the number of possible views and xk the k-th view over x. The description space for view k is R n k where n k is the number of features in view k. Moreover, we consider that some of the V views can be missing. The subset of available views for input x i will be represented by an index vector s i ∈ S = {0, 1} V so that s i k = 1 if view k is available and s i k = 0 elsewhere. Note that all the V views will not be available for each input x, and the prediction model must be able to predict an output given any subset of views s ∈ {0; 1} V . In this configuration, our objective is to estimate the distributions p(y|v(s, x)) where v(s, x) is the set of views xk so that s k = 1. This distribution p will be estimated based on a training set D of N training examples. Each example is composed of a subset of views s i , v(s i , x i ) associated to an output y i , so that D = { y 1 , s 1 , v(s 1 , x 1 ) , ..., y N , s N , v(s N , x N ) } where s i is the index vector of views available for x i . Note that x i is not directly known in the training set but only observed through its associated views.

2.2. BIDIRECTIONAL GENERATIVE ADVERSARIAL NETS (BIGAN):

We quickly remind the principle of BiGANs since our model is an extension of this technique. Generative Adversarial Networks (GAN) have been introduced by Goodfellow et al. (2014) and have demonstrated their ability to model complex distributions. They have been used to produce compelling natural images from a simple latent distribution (Radford et al., 2015; Denton et al., 2015) . Exploring the latent space has uncovered interesting, meaningful patterns in the resulting outputs. However, GANs lack the ability to retrieve a latent representation given an output, missing out an opportunity to exploit the learned manifold. Bidirectional Generative Adversarial Networks (BiGANs) have been proposed by Donahue et al. (2016) and Dumoulin et al. (2016) , independently, to fill that gap. BiGANs simultaneously learn both an encoder function E that models the encoding process P E (z|y) from the space R n to a latent space R Z , and a generator function G that models the mapping distribution P G (y|z) of any latent point z ∈ R Z to a possible object y ∈ R n . From both the encoder distribution and the generator distribution, we can model two joint distributions, respectively denoted P E (y, z) and P G (y, z): P G (y, z) = P (z)P G (y|z) P E (y, z) = P (y)P E (z|y)

(1) assuming that P (z) = N (0, 1) and P (y) can be estimated over the training set by a uniform sampling. The BiGAN framework also introduces a discriminator network D 1 whose task is to determine whether a pair (y, z) is sampled from p G (y, z) or from p E (y, z), while E and G are trained to fool D 1 , resulting in the following learning problem:

EQUATION

It can be shown, by following the same steps as in Goodfellow et al. (2014) , that the optimization problem described in Equation 2 minimizes the Jensen-Shanon divergence between P E (y, z) and P G (y, z), allowing the model to learn both a decoder and a generator over a training set that will model the joint distribution of (y, z) pairs. As proposed by Dumoulin et al. (2016) , we consider in the following that P G (y|z) is modeled by a deterministic non-linear model G so that G(z) = y, and P E as a diagonal Gaussian distribution E(z) = (µ(y), σ(y)). G, µ and σ are estimated by using gradient-based descent techniques.

2.3. GENERAL IDEA:

We propose a model based on the Generative Adversarial Networks paradigm adapted to the multiview prediction problem. Our model is based on two different principles:

z E(y) H(x) G(z) y x G E H H(x), x y, E(y) G(z), z D 1 D 2 E(y),

x

Figure 1 : The CV-BiGAN Architecture. The two top levels correspond to the BiGAN model, while the third level is added to model the distribution over the latent space given the input of the CV-BiGAN.

The discriminator D 2 is used to constraint P (z|y) and P (z|x) to be as close as possible.

Conditional Views BiGANs (CV-BiGAN): First, since one wants to model an output distribution based on observations, our first contribution is to propose an adaptation of BiGANs to model conditional probabilities, resulting in a model able to learn P (y|x) where x can be either a single view or an aggregation of multiple views. If conditional GANs have already been proposed in the literature (see Section 6) they are not adapted to our problem which require explicit mappings between input space to latent space, and from latent space to output space.

Multi-View BiGANs (MV-BiGAN): On top of the CV-BiGAN model, we build a multi-view model able to estimate the distribution of possible outputs based on any subset of views v(s, x). If a natural way to extend the Conditional BiGANS for handling multi-view is to define a mapping function which map the set of views to a representation space (see Section 4.1) the resulting model has shown undesirable behaviors (see Section 5.1). Therefore, we propose to constrain the model based on the idea that adding one more view to any subset of views must decrease the uncertainty on the output distribution i.e the more views are provided, the less variance the output distribution has. This behavior is encouraged by using a Kullback-Leibler divergence (KL) regularization (see Section 4.2).

3. THE CONDITIONAL BIGAN MODEL (CV-BIGAN):

Our first objective is to extend the BiGAN formalism to handle an input space (e.g a single observed view) in addition to the output space R n . We will denote x the observation and y the output to predict. In other words, we wish to capture the conditional probability P (y|x) from a given training dataset. Assuming one possesses a bidirectional mapping between the input space and an associated representation space, ie. P E (z|y) and P G (y|z), one can equivalently capture P (z|x).

The CV-BiGAN model keeps the encoder E and generator G defined previously but also includes an additional encoder function denoted H which goal is to map a value x to the latent space R Z . Applying H on any value of x results in a distribution P H (z|x) = N (µ H (x), σ H (x)) so that a value of z can be sampled from this distribution. This would then allow one to recover a distribution P (y|x).

Given a pair (x, y), we wish a latent representation z sampled from P H (z|x) to be similar to one from P E (z|y). As our goal here is to learn P (z|x), we define two joint distributions between x and z: P H (x, z) = P H (z|x)P (x)

EQUATION

H(.) φ 1 (x 1 ) H H(.) φ 1 (x 1 ) + φ 3 (x 3 ) H KL constraint

Figure 2 : The MV-BiGAN additional components. In this example, we consider a case where only x1 is available (top level) and a second case where both x1 and x3 are available. The distribution P (z|x1 , x3 ) is encouraged to be "included" in P (z|x 1 ) by the KL constraint. The aggregation of the views is made by the φ k functions that are learned conjointly with the rest of the model.

Minimizing the Jensen-Shanon divergence between these two distributions is equivalent to solving the following adversarial problem:

min E,H max D2 E x,y∼p(x,y),z∼p E (z|y) [log D 2 (x, z)] + E x,y∼p(x,y),z∼p H (z|x) [1 -log D 2 (x, z)] (4)

Note that when applying stochastic gradient-based descent techniques over this objective function, the probability P (x, y) is approximated by sampling uniformly from the training set. We can sample from P H (x, z) and P E (x, z) by forwarding the pair (x, y) into the corresponding network.

By merging the two objective functions defined in Equation 2 and 4, the final learning problem for our Conditionnal BiGANs is defined as: min

EQUATION

The general idea of CV-BiGAN is illustrated in Figure 1 .

4. MULTI-VIEW BIGAN:



4.1. AGGREGATING MULTI-VIEWS FOR CV-BIGAN:

We now consider the problem of computing an output distribution conditioned by multiple different views. In that case, we can use the CV-BiGAN Model (or other conditional approaches) conjointly with a model able to aggregate the different views where A is the size of the aggregation space. Instead of considering the input x, we define an aggregation model Ψ. Ψ(v(s, x)) will be the representation of the aggregation of all the available views xk 1 :

Ψ(v(s, x)) = V k=1 s k φ k (x k ) (6)

where φ k is a function that will be learned that maps a particular view in R n k to the aggregation space R A . By replacing x in Equation 5, one can then simultaneously learn the functions φ k and the distributions P H , P E and P D , resulting in a multi-view model able to deal with any subset of views.

4.2. UNCERTAINTY REDUCTION ASSUMPTION:

However, the previous idea suffers from a very high instability when learning, as it is usually noted with complex GANs architectures (see Section 5). In order to stabilize our model, we propose to add a regularization based on the idea that adding new views to an existing subset of views should reduce the uncertainty over the outputs. Indeed, under the assumption that views are consistent one another, adding a new view should allow to refine the predictions and reduce the variance of the distribution of the outputs.

Let us consider an object x and two index vectors s and s such that v(x, s) ⊂ v(x, s ) ie. ∀k, s k ≥ s k . Then, intuitively, P (x|v(x, s )) should be "included" in P (x|v(x, s)). In the CV-GAN model, since P (y|z) is deterministic, this can be enforced at a latent level by minimizing KL(P (z|v(x, s ) || P (z|v(x, s)). By assuming those two distributions are diagonal gaussian distributions (ie. P (z|v(x, s ) = N (µ 1 , Σ 1 ) and P (z|v(x, s) = N (µ 2 , Σ 2 ) where Σ k are diagonal matrices with diagonal elements σ k(i) ), the KL divergence can be computed as in Equation 7and differentiated.

EQUATION

Note that this divergence is written on the estimation made by the function H and will act as a regularization over the latent conditional distribution.

The final objective function of the MV-BiGAN can be written as:

min G,E,H max D1,D2 E s,x,y∼P (s,x,y),z∼P E (z|y) [log D 1 (y, z)] + E z∼P (z),y∼P G (y|z) [1 -log D 1 (y, z)] +E s,x,y∼P (s,x,y),z∼P E (z|y) [log D 2 (v(x, s), z)] + E s,x,y∼P (s,x,y),z∼P H (z|v(x,s)) [1 -log D 2 (v(x, s), z)] +λE x∼P (x) s,s ∈Sx ∀k,s k ≥s k KL(H(v(x, s ))||H(v(x, s)))

(8) where λ controls the strength of the regularization. Note that aggregation models Ψ are included into H and D 2 and can be optimized conjointly in this objective function.

4.3. LEARNING THE MV-BIGAN:

The different functions E, G, H, D 1 and D 2 are implemented as parametric neural networks and trained by mini-batch stochastic gradient descent (see Appendix for more details concerning the architectures).We first update the discriminators networks D 1 and D 2 , then we update the generator and encoders G, E and H with gradient steps in the opposite direction.

As with most other implementation of GAN-based models, we find that using an alternative objective proposed by Goodfellow et al. (2014) for E, G and H instead leads to more stable training. The new objective consist of swapping the labels for the discriminators instead of reversing the gradient. We also find that we can update all the modules in one pass instead of taking alternate gradient steps while obtaining similar results.

Note that the MV-BiGAN model is trained based on datasets where all the V views are available for each data point. In order to generate examples where only subsets of views are available, the ideal procedure would be to consider all the possible subsets of views. Due to the number of data points that would be generated by such a procedure, we build random sequences of incremental sets of views and enforce the KL regularization over successive sets.

5. EXPERIMENTS:

We evaluate our model on three different types of experiments, and on two differents datasets. The first dataset we experiment on is the MNIST dataset of handwritten digits. The second dataset is the CelebA (Liu et al., 2015) dataset composed of both images of faces and corresponding attributes. The MNIST dataset is used to illustrate the ability of the MV-BiGAN to handle different subset of views, and to update its prediction when integrating new incoming views. The CelebA dataset is used to demonstrate the ability of MV-BiGAN to deal with different types (heterogeneous) of views.

5.1. MNIST, 4 VIEWS:

We consider the problem where 4 different views can be available, each view corresponding to a particular quarter of the final image to predict -each view is a vector of R (14×14) . The MV-BiGAN Figure 3 illustrates the results obtained for some digits. In this figure, the first column displays the input (the subset of views), while the other columns shows predicted outputs sampled by the MV-BiGAN. An additional view is added between each row. This experiment shows that when new views are added, the diversity in the predicted outputs decreases due to the KL-contraint introduced in the model, which is the desired behavior i.e more information implied less variance. When removing the KL constraint (Figure 4 ), the diversity still remains important, even if many views are provided to the model. This show the importance of the KL regularization term in the MV-BiGAN objective. 

5.2. MNIST, SEQUENCE OF INCOMING VIEWS:

We made another set of experiments where the views correspond to images with missing values (missing values are replaced by 0.5). This can be viewed as a data imputation problem -Figure 5 . Here also, the behavior of the MV-BiGAN exhibits interesting properties: the model is able to predict the desired output as long as enough information has been provided. When only non-informative views are provided, the model produces digits with a high diversity, the diversity decreasing when new information is added.

5.3. CELEBA, INTEGRATING HETEROGENEOUS INFORMATION:

At last, the third experiment aims at measuring the ability of MV-BiGAN to handle heterogeneous inputs. We consider two views: (i) the attribute vector containing information about the person in the picture (hair color, sex, ...), and (ii) a incomplete face. Figure 9 illustrates the results obtained on two faces. The first line corresponds to the faces generated based on the attribute vector. One can see that the attribute information has been captured by the model: for example, the sex of the generated face is constant (only women) showing that MV-BiGan has captured this information from the attribute vector. The second line corresponds to the faces generated when using the incomplete face as an input. One can also see that the generated outputs are "compatible" with the incomplete information provided to the model. But the attribute are not considered (for example, women and men are generated). At last, the third line corresponds to images generated based on the two partial views (attributes and incomplete face) which are close to the ground-truth image (bottom left). Note that, in this set of experiments, the convergence of the MV-BiGAN was quite difficult to obtain, and the quality of the generated faces is still not satisfying.

6. RELATED WORK:

Multi-view and Representation Learning: Many application fields naturally deal with multi-view data with true advantages. For example, in the multimedia domain, dealing with a bunch of views is usual (Atrey et al., 2010 ): text, audio, images (different framings from videos) are starting points of these views. Besides, multimedia learning tasks from multi-views led to a large amount of fusionbased ad-hoc approaches and experimental results. The success of multi-view supervised learning approaches in the multimedia community seems to rely on the ability of the systems to deal with the complementary of the information carried by each modality. Comparable studies are of importance in many domains, such as bioinformatics (Sokolov & Ben-Hur, 2011) , speech recognition (Arora & Livescu, 2012; Koço et al., 2012) , signal-based multimodal integration (Wu et al., 1999) , gesture recognition (Wu et al., 2013) , etc. Moreover, multi-view learning has been theoretically studied mainly under the semi-supervised setting, but only with two facing views (Chapelle et al., 2006; Sun, 2013; Sun & Taylor, 2014; Johnson & Zhang, 2015) . In parallel, ensemble-based learning approaches have been theoretically studied, in the supervised setting: many interesting results should concern multi-view learning, as long as the ensemble is built upon many views (Rokach, 2010; Zhang & Zhang, 2011) . From the representation learning point of view, recent models are based on the incorporation of some "fusion" layers in the deep neural network architecture as in (Ngiam et al., 2011) or (Srivastava & Salakhutdinov, 2012) for example. Some other interesting models include the multiview perceptron (Zhu et al., 2014) .

Estimating Complex Distributions: While deep learning has shown great results in many classification task for a decade, training deep generative models still remains a challenge. Deep Boltzmann Machines (Salakhutdinov & Hinton, 2009) are un-directed graphical models organized in a succession of layers of hidden variables. In a multi-view setting, they are able to deal with missing views and have been used to capture the joint distribution in bi-modal text and image data (Srivastava & Salakhutdinov, 2012; Sohn et al., 2014) . Another trend started with denoising autoencoder (Vincent et al., 2008) , which aims to reconstruct a data from a noisy input have been proved to possess some desirable properties for data generation (Bengio et al., 2013) . The model have been generalized under the name Generative Stochastic Networks by replacing the noise function C with a mapping to a latent space (Thibodeau-Laufer et al., 2014) . Pulling away from the mixing problems encountered in previous approaches, Variational Autoencoders (Kingma & Welling, 2013) attempts to map the input distribution to a latent distribution which is easy to sample from. The model is trained by optimizing a variational bound on the likelihood, using stochastic gradient descent methods. The Kullback-Leibler regularizer on the latent Gaussian representations used in our model is reminiscent of the one introduced in the variational lower bound used by the VAE.

The BiGAN model (Donahue et al., 2016; Dumoulin et al., 2016) that serves as a basis for our work is an extension of the Generative Adversarial Nets (Goodfellow et al., 2014) . A GAN extension that captures conditional probabilities (CGAN) has been proposed in (Mirza & Osindero, 2014) . However, as noted by (Mathieu et al., 2015) and (Pathak et al., 2016) , they display very unstable behavior. More specifically, CGAN have been able to generate image of faces conditioned on an attribute vector (Gauthier, 2014) , but fail to model image distribution conditioned on a part of the image or on previous frames. In both CGAN and CVBiGAN, the generation process uses random noise to be able to generate a diversity of outputs from the same input. However, in a CGAN, the generator concatenate an independent random vector to the input while CV-BiGAN learns a stochastic latent representation of the input. Also, some of the difficulties of CGAN in handling images as both inputs x and outputs ỹ stem from the fact that CGAN's discriminator directly compares x and y. In CV-BiGAN, neither discriminators has access to both x and y but only to a latent representation z and either x or y.

7. CONCLUSION AND PERSPECTIVES:

We have proposed the CV-BiGAN model for estimating conditional densities, and its extension MV-BiGAN to handle multi-view inputs. The MV-BiGAN model is able to both handle subsets of views, but also to update its prediction when new views are added. It is based on the idea that the uncertainty of the prediction must decrease when additional information is provided, this idea being handled through a KL constraint in the latent space. This work opens different research directions: the first one concerns the architecture of the model itself since the convergence of MV-BiGAN is still difficult to obtain and has a particularly high training cost. Another direction would be to see if this family of model could be used on data streams for anytime prediction.

The discriminator D 1 has two fully connected layers with a Leaky ReLU after the first one, and a sigmoid at the output, build on top of a fractionally-strided convolutions network. z is concatenated at the output level of the convolutional network.

As in the MNIST experiments, the discriminator D 2 is similar to E and H except it uses Leaky ReLUs instead of ReLUs and a sigmoid at the output level. z is concatenated directly to the aggregation vector Ψ(v(x, s)).

Aggregation space is of size 1000. λ is set to 1 • 10 -3 , and minibatch size is 16. The model has been trained for 15 epochs.

ADDITIONAL RESULTS:



:

Note that other aggregation scheme can be used like recurrent neural networks for example.

Figures and Tables:

Figure 3: Results of the MV-BiGAN on sequences of 4 different views. The first column corresponds to the provided views, while the other columns correspond to outputs sampled by the MV-BiGAN.

Figure 4: MV-BiGAN without KL-constraint (i.e λ = 0).

Figure 5: MV-BiGAN with sequences of incoming views. Here, each view is a 28 × 28 matrix (values are between 0 and 1 with missing values replaced by 0.5).

Figure 6: Results obtained on the CelebA dataset for two examples. The first line corresponds to the images generated based on the attribute vector, the second line corresponds to images generated based on the incomplete face, the third line corresponds to the images generated based on the two views. The groundthruth face is given in the bottom-left corner, while the incomplete face is given in the top-left corner.

Peer review (iclr2017):
This paper builds in the bidirectional GAN (BiGAN) to obtain an extension which can handle multiple views of data. To this end, the authors extend the BiGAN for multiple view aggregation. This is an easy task and just requires the introduction of the additional distribution and accompanying discriminator. The main challenging and novel part is in regularizing the model to avoid instabilities. The authors propose a novel KL divergence based constraint. As mentioned above, the approach builds quite heavily on previous ones but it has enough novel elements, in particular the constraint for regularization. This constraint is a reasonable assumption an in practice seems to work well. One downside is that it comes with a parameter lambda which controls its strength and which is not obvious how to find efficiently. For example, for the MNIST data it takes a very small value, 10^-5 and for the CELEBA it*s 10^-3. It could be that the results are not very sensitive to this value, but there*s no discussion concerning this aspect. Apart from the regularization term, the rest of the model construction is well motivated, I agree with the authors that a multi-view approach employing GANs is an interesting topic to consider. Presentation is in general good although at parts readability is hindered. I feel that the notation is unnecessarily complicated, and some parts of the text too. Furthermore, it wasn*t immediately obvious to me what is considered as 	ilde{x} and what is y in the experiments. Additionally, most of the discussion on the well-studied area of multi-view learning (intro and sec. 6) omits reference to important prior work which is not based on neural networks. Indeed, some of the issues mentioned as common in today*s methods (discrete outputs only, no density estimation...) do not actually exist in many non-neural network approaches. There are too many works to suggest including in the discussion, but I guess the most relevant ones are in the field of probabilistic and non-linear multi-view learning, which is also what MV-BiGAN is doing. The experiments presented in the paper are nice illustrations but unfortunately insufficient. Firstly, they only cover the task of generating (small) images and correspond to non-real world settings (I*d actually consider all of the experiments as toy experiments). Furthermore, the most difficult of these experiments (sec. 5.3) is quite unconvincing. If Fig. 6 shows some of the best results that can be achieved, then this is rather disappointing. Important details are also missing: what is exactly the attribute vector used? How many instances exist? The two other experiments on 5.1 and 5.2 are well executed. It*s important that the authors show a comparison with lambda=0. However, beyond showing the validity of the KL term, one can*t conclude much about the overall merit of the method as a multi-view learning approach, given the above experiments. Overall this was a nice paper to read, but seems somehow incomplete.