Year: 2018

Abstract:
Many of the zero-shot learning methods have realized predicting labels of unseen images by learning the relations between images and pre-defined class-attributes. However, recent studies show that, under the more realistic generalized zero-shot learning (GZSL) scenarios, these approaches severely suffer from the issue of biased prediction, i.e., their classifier tends to predict all the examples from both seen and unseen classes as one of the seen classes. The cause of this problem is that they cannot properly learn a mapping to the representation space generalized to the unseen classes since the training set does not include any unseen class information. To solve this, we propose a concept to learn a mapping that embeds both images and attributes to the shared representation space that can be generalized even for unseen classes by interpolating from the information of seen classes, which we refer to shared manifold learning. Furthermore, we propose modality invariant variational autoencoders, which can perform shared manifold learning by training variational autoencoders with both images and attributes as inputs. The empirical validation of well-known datasets in GZSL shows that our method achieves the significantly superior performances to the existing relationbased studies.

heading: 1 INTRODUCTION
text: The recent high performance of deep neural networks on image classification and object recognition depends greatly on whether we can obtain sufficiently labeled images of classes to predict. However, it is difficult to operate such a AI system as-is in the real world because the number of existing classes is huge and, as long as human beings create or develop new objects, their number can be expected to continue to increase day by day, resulting in difficulty obtaining labeled data of all these classes. Human beings have the excellent ability to infer objects that they have seen only a few by exploiting and transferring the knowledge learned in the past so as to survive in the harsh real world. In the machine learning context, this problem is formulated as few-shot learning.
The extreme problem setting of few-shot learning is especially known as zero-shot learning (ZSL), which is to train by labeled set from certain classes called seen classes, and then to predict completely unseen classes not included in the training set. It is usually accomplished by preparing pre-defined semantic representations of all classes such as attributes, and learning the relations between images and class-attributes, which we call the relation-based method in this paper (Lampert et al., 2009; 2014; Frome et al., 2013; Akata et al., 2013; 2015; Romera-Paredes & Torr, 2015; Kodirov et al., 2017). However, this conventional setting is unnatural in application to the real world because the objects we encounter are not always new but are rather likely to have been experienced in the past. Therefore, a more natural setting that the test set includes examples of both seen and unseen classes has been proposed, which is called generalized zero-shot learning (GZSL) (Chao et al., 2016).
It is known that GZSL is a more difficult setting than conventional ZSL because seen classes are also candidates for prediction. Especially, it is pointed out that prediction to unseen classes becomes fail severely in the case of relation-based methods (Chao et al., 2016; Xian et al., 2018). The main reason is that they cannot obtain the generalized representation space such that examples of unseen classes are properly mapped because this information is not included in the training set. For example,
if unseen classes are arranged to overlap with the seen classes in the representation space, which means this space is not generalized for unseen classes, all examples of the unseen classes might be biased to be predicted as one of the seen classes.
Therefore, in this study, we propose a new solution of GZSL called shared manifold learning. At first, we aim to learn a mapping that embeds each image and attribute in the same representation space, i.e., shared representation learning of images and attributes. This contributes to the prediction of the relations in shared representation that is robust against the representation of the inputs. In addition, we make sure that this mapping is also generalized, which means that the inputs of the unseen classes are embedded in appropriate places in shared representation by interpolating from the information of seen classes. That is, it can be regarded as manifold learning in shared representation. If such shared manifold learning can be performed properly, we can estimate the correct classattributes from the test image by embedding it in the shared manifold representation, regardless of whether this image is seen or unseen.
To realize this, we propose modality invariant variational autoencoders (MIVAE), which are based on variational autoencoders (VAE) (Kingma & Welling, 2013; Rezende et al., 2014). In this method, first of all, to obtain shared representations from inputs of images x and class-attributes ay , we define mappings from each input to the latent space z as the deep probabilistic distributions, qλ(z|x) and qλ(z|ay). Next, to learn manifold representation from these two inputs, we prepare a VAE with both images and attributes as inputs. In learning of MIVAE, we not only maximize the lower bound of this VAE but also learn so that each mapping approximates the encoder of VAE, qφ(z|x,a), and so that each mapping also approaches another. This makes it possible to obtain the mappings that embed each input in the shared manifold representation. See figure 1 as the outline of shared manifold learning with MIVAE. If we consider images and attributes as different modalities, then MIVAE can be regarded as an extension of works examining multimodal VAE (Suzuki et al., 2016; Vedantam et al., 2017).
The contribution of this research is as follows.
• We discuss that the biased problem is related to the reason why the relation-based methods in GZSL fails and propose shared manifold learning as a concept to solve it.
• We propose MIVAE, which extended VAE as a method to realize shared manifold learning. • We confirm that MIVAE can perform shared manifold learning properly and that it can
contribute to the higher performance of GZSL compared to existing relation-based GZSL methods.

heading: 2 PROBLEM FORMULATION
text: We assume that the dataset Dtr = {xi, yi}Ntri=1 is given as the training set, where xi ∈ X is the input data, e.g. an image, and yi ∈ Ys = {1..., S} is the corresponding label data. The objective
Training Test Training Test
of ZSL is to train the classifier using Dtr and to predict labels ŷj ∈ Y from the example xj ∈ X in the test set Dts = {xj}Ntsj=1. In the original ZSL, it is assumed that the classes of the test set are completely unseen in the training set, which means Y = Yu = {S + 1, ..., U}. On the other hand, our goal is to train in the setting of GZSL, which includes both seen and unseen classes in the test set (Y = Ys ∩ Yu). We also assume that we have the class-attribute matrixA ∈ RM×(S+U) as the semantic information of classes, where each column represents the M -dimensional attribute vector ac ∈ A = RM of each class c = 1, ..., S + U . Using this attribute vector, the training set Dtr can be replaced as {xi,ayi}Ntri=1.

heading: 3 PROPOSED METHOD

heading: 3.1 BIASED PROBLEM IN RELATION-BASED GZSL AND SHARED MANIFOLD LEARNING
text: In this section, we discuss the reason why the prediction of test data fails in relation-based GZSL and how to solve it.
The relation-based methods learn the relations between the image and the attribute by mapping them to the representation space. Depending on each method, this representation space is considered as attribute space (image-attribute projection) or another shared space (shared representation learning). For details, see the section 4.
What is important in learning both ZSL and GZSL is to obtain the relations between images and attributes generalized to test data, that is, to map each pair of examples to the same place in the representation space. Furthermore, in the setting of GZSL, since the test set includes the seen classes, it is necessary to learn a generalized representation where the unseen classes do not overlap with the seen classes. However, this is clearly difficult because information of the unseen classes does not exist in the training set. If the generalization between classes fails and the representations of the unseen classes overlap the seen classes, all unseen examples may be classified biased toward the seen classes (see figure 2(a)). Therefore, in GZSL, we have to consider means to learn generalized representations not only between images and attributes but also between classes. Note that ZSL does not include a biased problem like GZSL because the test set does not include the seen classes (see figure 2(b)).
In this paper, we propose the framework of learning to solve this problem as follows.
First, we consider mapping from each image and attribute to the same space. At this time, we enforce the images and attributes to be the same representations in the embedding space (shared representation learning). One of the advantages of mapping to the shared space is that we can think of relations between examples in a space with more information by integrating images and attributes. It is thought that we can estimate the location of unseen data better by performing manifold learning as described later in such a space. Also, Chao et al. (2016) argues that improvement of the representation of attributes is vital for GZSL, which can be realized by the shared representation integrated with the images. Another advantage is that since shared space does not depend on input dimensions or representation, we can make robust inference to the complexity of inputs.
Next, we perform manifold learning to obtain a shared representation generalized in the unseen classes. In manifold learning, continuous mapping from input space to shared space can be learned. Therefore, unseen data also can be mapped to an appropriate place in the shared space by interpolating from the prior knowledge of the seen classes.
We designate these series of learning as shared manifold learning of images and attributes. We claim that we can deal with the bias-related shortcoming of GZSL by learning shared manifold representation.

heading: 3.2 MULTIMODAL INVARIANT VARIATIONAL AUTOENCODER
text: Next, we propose modality invariant VAE (MIVAE), which is a method of learning mapping from two inputs to a space to be shared and manifold representations.
Here, we consider that the dataset of ZSL {xi,ayi}Ntri=1 has different modalities, which means that x and ay are different in their spaces and distributions, i.e., X 6= A and p(X) 6= p(A),where X = {x1, ..., xNtr} and A = {ay1 , ..., ayNtr }. At first, we consider two Gaussian distributions parameterized with deep neural networks qλ(z|x) and qλ(z|ay) as mappings from each of X andA to Z , where λ is the parameter of the distribution. See appendix A for the way of parameterization with deep neural networks. In MIVAE, we aim to learn so that these mappings embed inputs in shared manifold representation. Note that it is important to set these mappings to the deep probabilistic distributions in obtaining shared representations. For a detailed discussion on this, see appendix C.
Next, we use variational autoencoders (VAEs) (Kingma & Welling, 2013; Rezende et al., 2014), which has been widely used as a model of representation learning and manifold learning. Moreover, we consider VAE with two modalities as input, which is called joint VAE (Vedantam et al., 2017).
Joint VAE has a latent concept z that is common to the two modalities x,ay , each of which is generated conditionally and independently from the concept. That is, their generative processes are z ∼ p(z) and x,ay ∼ pθ(x,ay|z) = pθ(x|z)pθ(ay|z), where θ corresponds to model parameters. These processes indicate that the latent variable z can be regarded as a shared representation containing different modality information.
To train this, we should maximize the joint distribution p(x,ay) = ∫ pθ(x,ay|z)p(z)dz over the training set. However, it is difficult to perform this maximization directly because this joint distribution is intractable. Therefore, we instead maximize the following evidence lower bound (ELBO).
Lvae = −DKL(qφ(z|x,ay)||p(z)) + Eqφ(z|x,ay)[log pθ(x|z)] + Eqφ(z|x,ay)[log pθ(ay|z)], (1) where qφ(z|x,ay) is an approximate distribution of posterior and φ represents parameters. We call qφ(z|x,ay) as encoder and pθ(x|z) and pθ(ay|z) as decoder. These models are also parameterized by deep neural networks. The distribution of the encoder is Gaussian, and that of the decoder depends on the distribution of the input data.
From the encoder of trained VAE, we can extract shared representations that integrate the information of different modalities. In addition, the VAE can obtain the manifold representation of the input in the latent space and appropriately place the unseen input on the latent space by interpolating the seen information. Because we use images and attributes as inputs to joint VAE here, we can obtain the encoder that integrates and maps them into manifold representation, i.e., we can perform shared manifold learning.
Therefore, if we train each modality’s distribution to minimize the distance between the encoder, then we can obtain such representations from the distribution of each modality. We define this distance using Kullback-Liebler (KL) divergence as
Ldist = DKL(qφ(z|x,ay)||qλ(z|x)) +DKL(qφ(z|x,ay)||qλ(z|ay)). (2)
Intuitively, because the encoder has more information to be conditioned, it seems that the mapping distributions, especially qλ(z|ay), might degenerate if each is brought closer to the encoder. Actually, Vedantam et al. (2017) showed that minimization of Epdata(x,ay)[DKL(qφ(z|x,ay)||qλ(z|ay))] is equivalent to minimization of
Epdata(x,a)[DKL(q avg φ (z|ay)||qλ(z|ay))], where q avg φ (z|ay) = Epdata(x|ay)[qφ(z|x,ay)], i.e., this is the average of the encoder over all x corresponding to a. Therefore, the minimization of equation 2 encourages each mapping to close to the distribution complementing the other modality of the encoder, thereby resulting in stabilization.
When equation 2 becomes minimum, both qλ(z|x) = ∫ qφ(z|x,a)da and qλ(z|a) =∫
qφ(z|x,a)dx are satisfied. However, if the objective function of VAE is also optimized simultaneously, then these minimization may not be performed properly due to the influence of the VAE terms during training. Therefore, to force the two distributions to be explicitly close at the training time, we also minimize the following equation:
Lmap = DKL(qλ(z|x)||qλ(z|ay)). (3) Therefore, the objective function of MIVAE is
L = −Lvae + λdistLdist + λmapLmap, (4) where λdist and λmap are hyper-parameters. When λmap = 0, this objective function is the same as JMVAE (Suzuki et al., 2016), which means that this model can be regarded as an extension of JMVAE.
After training MIVAE, a label of test image can be simply predicted as ŷj = argminy∈Y DKL(qλ(z|xj)||qλ(z|ay)).

heading: 4 RELATED WORKS
text: ZSL is one of the greatest attention problems among image recognition tasks, and various methods have been proposed. Here, we explain them separately on relation-based and synthetic-based approaches.

heading: 4.1 RELATION-BASED
text: Relation-based methods are intended to train relations between images and class-attributes F (x, ay) from training data and to predict a class whose relation is strong in test data, i.e. ŷ = F (x, ay). This approach can be classified further into the following two methods: training of mapping from image space to attribute space (image-attribute projection) and training of shared representation that embed both image and attribute space (shared representation learning).
Image-attribute projection In ZSL, the simplest and most widely adopted approach is to obtain mapping from image space to class-attribute space with training data. Some early studies used classifiers such as SVM to learn mapping from images to class attributes (Lampert et al., 2009; 2014), but many other methods use linear embedding into attributes (or semantic representations) and typically set a ranking loss as the objective function to train such embedding (Frome et al., 2013; Akata et al., 2013; 2015; Romera-Paredes & Torr, 2015; Kodirov et al., 2017). Actually, RomeraParedes & Torr (2015) adds a regularization term of the mapping to this objective function. Also, Kodirov et al. (2017) minimizes the embedding error in both directions with the objective. Recently, nonlinear models are also used Socher et al. (2013) to consider non-linearity between images and attributes. However, classifiers tend to predict only one class from many images at testing, which is known as the hubness problem because of the difference in the distribution of the two modalities. To avoid this difficulty, some works have proposed training of reverse mapping from attribute to image space with such as deep neural networks (Zhang et al., 2017). Verma & Rai (2017) suggests that mapping from attributes to an image are a conditional distribution by exponential family rather than deterministic, which can incorporate consideration of uncertainty among modalities. Therefore the hubness problem can be relaxed further.
Shared representation learning Another means of obtaining relations is to learn mapping that embeds images and attributes into a shared latent representation. This learning can be realized by shared
representation learning in which images and attributes are regarded as different modalities. In ZSL, several methods of learning shared embedding spaces have been proposed (Zhang & Saligrama, 2015; 2016), but few methods use deep neural networks as embedding maps. VZSL (Wang et al., 2017) learns shared representations of images and attributes using deep generative models, which most closely approximates our method. As with Verma & Rai (2017), uncertainty can be taken into consideration, and furthermore, deep neural networks are used.
Although these relation-based methods work well in conventional ZSL settings, in GZSL, their performance is much worse, as shown in Chao et al. (2016). The approach of shared representation learning is robust to the representation of attributes, but it does not perform manifold learning at two inputs, which means that it does not perform shared manifold learning. Indeed, VZSL performs manifold learning only with image input, which results in poor generalization performance in examples of unseen classes, as shown later in an experiment.

heading: 4.2 SYNTHESIS-BASED
text: Recently, several synthesis-based methods have been proposed (Verma et al., 2018; Mishra et al., 2017). These approaches are attempts to generate images from class-attributes and to train a classifier of unseen classes by these synthesis samples. To generate samples from attributes properly, they train conditional deep generative models of images given class-attributes in training data. It is apparently difficult to generate realistic test images that are useful to train the classifier. However, in many ZSL settings, it is common to use feature extraction representations from images as input so that they can generate these inputs with high quality.
The greatest advantage of the synthesis-based approaches is that they can achieve high performance in a GZSL setting because they train classification with a new classifier after generating all samples of seen or unseen classes from the trained deep generative model. For that reason, we no longer need to consider differences between seen and unseen classes. However, training the classifier requires generation of multiple images of all classes. Therefore, if the image size increases or the number of classes increases, then it might be costly to generate these synthetic samples. Moreover, this method works well only if the input representations of images are easy to generate. In recent years, several deep generative models have been proposed that can produce high-quality images (Kingma & Dhariwal, 2018; Karras et al., 2017), but it still might be difficult to generate arbitrary types of inputs to be useful for classification of unseen classes.
In this paper, we focused on the relation-based approach and proposed the concept of shared manifold learning to deal with the biased problem while retaining the advantage of relation-based GZSL. Table 1 presents a brief comparison of our method with others.

heading: 5 EXPERIMENTS
text: For experiment, we use the following four datasets commonly used for ZSL: Animals with Attributes (AWA1) (Lampert et al., 2014), CUB-200-2011 Bird (CUB) (Wah et al., 2011), SUN Attribute (SUN) (Patterson & Hays, 2012), and Attribute Pascal and Yahoo (aPY) (Farhadi et al., 2009). In addition, because the images of AWA1 were not public available, Xian et al. (2018) additionally introduced Animals with Attributes 2 (AWA2) composed of images collected from public sites. Therefore, in this experiment, we will use five datasets including AWA2. Each dataset contains images of each class, where each class is represented by semantic attributes. Table 2 presents the numbers of images, attributes, and classes in respective datasets. For fair comparisons, we use 2048- dim top-layer embedding of the 101-layered ResNet (He et al., 2016) provided by Xian et al. (2018) as the image vector. Furthermore, for the class-attribute representation we use the continuous valued attributes between 0 and 1 provided with each dataset. We followed the split proposed in Xian et al. (2018) for splitting each dataset into train, validation, and test. The hyper-parameter selection of
our model was based on this train-validation split, whereas in training of GZSL both were used as training data.
For the metric of evaluation at the test time, we use the average per-class top-1 accuracy on both the seen and unseen classes (referred as accYs and accYu ). In addition, to evaluate the performance on GZSL, we calculate the harmonic mean of accYs and accYu , which is H = (2 · accYs · accYu)/(accYs + accYu). We used the Adam optimization algorithm (Kingma & Ba, 2014) with a learning rate of 10−3. Also, in order to suppress over-fitting, we applied batch normalization to each layer and added L2 regularization to the parameters, where the coefficient of this term is 10−4. In all experiments, we trained for 200 epochs and evaluated the average of 5 experiments with different seeds. For details of network architectures, see appendix A. In all experiments, we set λdist = 1, which works well in multimodal learning according to reports of related studies (Suzuki et al., 2016; Vedantam et al., 2017). For another parameter λmap, we will verify the effect by changing this value in the experiments. All models in this paper were implemented with PyTorch (Paszke et al., 2017).
We conduct two experiments. First, compared with some methods of shared representation learning, we demonstrate that MIVAE with shared manifold learning can learn with high performance qualitatively and quantitatively in GZSL. The second experiment compares our method with the state-of-the-art relation-based methods and confirms that our method has much higher performance than those methods.

heading: 5.1 COMPARISON WITH SHARED REPRESENTATION LEARNING METHODS
text: To the best of our knowledge, VZSL (Wang et al., 2017) is the only existing method of ZSL used to train shared representation learning with deep probabilistic embedding maps, qλ(z|x) and qλ(z|ay). Furthermore, we consider the following three methods including VZSL as shared representation learning to compare.
Minimization of embedded divergence (MED) This simply minimizes KL divergence of the two distributions, i.e. the loss function of MED is DKL(qλ(z|x)||qλ(z|ay)).
MED-VAE To train the manifold representation in the latent space, we add the reconstruction error of the VAE whose input is the image vector to that of the MED, i.e., −Eqλ(z|x)[log pθ(x|z)] +DKL(qλ(z|x)||qλ(z|ay)).
VZSL (Wang et al., 2017) Wang et al. (2017) proposed margin regularization that encourages separation of the true class from the next class. The loss function of VZSL is the addition of the margin regularization to that of MED-VAE, i.e.,
−Eqλ(z|x)[log pθ(x|z)] +DKL(qλ(z|x)||qλ(z|ay))− λvzsl log S∑ c=1 exp(−DKL(qλ(z|x)||qλ(z|ac)),
where λvzsl is a hyper-parameter and set to 1.
This experiment used AWA2 for the dataset. The distribution of the embedding maps was Gaussian. The networks structures were set the same in all methods. Furthermore, the prediction methods of the classes of each example in the test data are all the same: ŷj = argminy∈Y DKL(qλ(z|xj)||qλ(z|ay)).
Figure 3 shows learning curves obtained with these three methods and MIVAE in GZSL. First, for seen classes, all models except MED have nearly equal accuracy. However, because the accuracy of MED is markedly low, it is apparent that manifold learning on latent representation by VAEs is important. In MIVAE, we confirm that the accuracy score is higher when λmap is set to 1.
Next, the result of the unseen classes shows that both the MED and the MED-VAE deteriorate considerably during early training. Regarding MED, failure to train generalized representations during training caused bias to seen classes, leading to bad prediction in unseen classes. However, in MED-VAE, accuracy becomes very poor despite manifold training of images, probably because we were unable to train the connection between attributes and images on the embedding space. VZSL tries to link images and corresponding attributes more closely during training, which contributes to making them better than MED-VAE. However, as training progresses, the accuracy drops, i.e., the training becomes unstable.
Finally, it is apparent that MIVAE is markedly more accurate than these methods. From these results, we confirmed that, to predict the unseen classes in the GZSL setting, it might be important to train a shared manifold representation that integrates images and attributes. Moreover, we found that MIVAE can properly obtain such a representation.
The result of the harmonic mean is almost identical to that of the accuracy of unseen classes. In addition, in these AWA2 experiments, the MIVAE’s results on accuracy of the unseen classes and the harmonic mean are apparently less dependent on the λmap value.
Figure 4 shows representations mapped respectively by the embedding distribution trained with MED-VAE, VZSL, and MIVAE. From this result as well, it is apparent that MIVAE can obtain representations of seen and unseen classes in the latent space.
These results clarify that shared representation learning is insufficient in GZSL and that shared manifold learning is important for high performance.

heading: 5.2 COMPARISON WITH STATE-OF-THE-ART METHODS
text: Next, we compare MIVAE with state-of-the-art relation-based ZSL. All results of these methods were reprinted from Xian et al. (2018).
As table 3 shows, our proposed method largely outperforms all existing methods of relation-based. Relation-based methods are known to tend to be biased to seen classes, so that the results of unseen classes become considerably worse. Particularly, table 3 shows that performance is poor in datasets with few seen classes such as AWA (both 1 and 2) and aPY. By contrast, despite being a relationbased method, MIVAE can properly generalize unseen classes even in these difficult datasets. These results suggest that our approach solves the bias problem in GZSL. Also, for seen classes, our method is equal to or higher than other methods, so that for harmonic mean, the proposed method is much superior to the others.
In MIVAE, the result of λmap = 1 is better than that of λmap = 0 in both SUN and CUB, but there is not much difference between them in terms of AWA and aPY; rather it is better when λmap = 1.
Note that we do not reprint the results of VZSL in Wang et al. (2017) to table 3 because the train/test split and the evaluation metric in Wang et al. (2017) is different from here. In our reproduction of VZSL in the same split and metric here, the harmonic means in each dataset and their average (SUN, CUB, AWA1, AWA2, aPY, Average) are (20.9, 30.7, 16.7, 18.7, 11.6, 19.7) respectively, which are all inferior to our method.
Finally, we evaluated the performance in the conventional ZSL setting. As table 4 shows, our proposed method has nearly the same performance as that of existing methods in most datasets. These results show that our method is directly addressing the bias-related shortcoming of relation-based approaches on GZSL.

heading: 6 CONCLUSION
text: In this paper, we addressed the setting of generalized zero-shot learning (GZSL) and specifically demonstrated that shared manifold learning is important to solve the biased problem. Furthermore, to demonstrate that point, we proposed modality invariant variational autoencoders (MIVAE), which realizes shared manifold learning using variational autoencoders that take both images and attributes as inputs. We first compared MIVAE with some ZSL methods that perform only shared representation learning. Thereby, we confirmed that performing manifold learning with both images and attributes is important for good performance of GZSL. Next, we experimented with several benchmark datasets. Results showed that MIVAE has markedly higher performance than state-of-the-art relation-based ZSL methods, which suggested that the bias-related difficulty can be mitigated. Furthermore, the fact that the performance in the conventional ZSL is not much different from that of the existing method means that MIVAE directly confronts difficulties arising with related-based GZSL. Actually, MIVAE is a robust and extensible method for the input information format. Therefore, in the future, we plan to deal with more realistic and complicated images directly and to use text information etc. as more sophisticated side information of classes.

heading: A PARAMETERIZATION OF DISTRIBUTIONS WITH DEEP NEURAL NETWORKS
text: The Gaussian distribution can be parameterized with deep neural networks as
N (z;µ,diag(σ2)), µ = fµ(fMLP(x)),
σ2 = Softplus(fσ2(fMLP(x))),
where fµ and fσ2 are respectively denote linear single layer neural networks and where fMLP represents a deep neural network with an arbitrary number of layers. Moreover, applying the softplus function for each element of a vector is denoted as Softplus.
The Bernoulli distribution is parameterized as
pθ(x|z) = B(x;µ),µ = Sigmoid(fµ(fMLP(z))), where Sigmoid is represents the sigmoid function.

heading: B NETWORK ARCHITECTURES
text: For the notation of model structures, we denote a linear fully-connected layer with k units, batch normalization, and ReLU as DkBR. Also, we denote DkBRwithout batch normalization and ReLU as Dk. In addition, the process of applying J after I is denoted as I-J, and the process of concatenating the last layers of the two networks I, J into one layer is denoted as (I,J).
Therefore, the network structures of distributions of MIVAE are as follows:
• p(x|z) (Gaussian, where σ2 = 1) – fµ: D2048 – fMLP: z-D1000BR
• p(a|z) (Bernoulli) – fµ: DdimA – fMLP: z-D1000BR
• q(z|x,a) (Gaussian) – fµ and fσ2 : D1000 – fMLP: (x,a)-D1000BR-D1000BR
• q(z|x) (Gaussian) – fµ and fσ2 : D1000 – fMLP: x-D1000BR-D1000BR
• q(z|a) (Gaussian) – fµ and fσ2 : D1000 – fMLP: a-D1000BR-D1000BR

heading: C DISCUSSION ON PROBABILISTIC MAPPING TO OBTAIN SHARED REPRESENTATION
text: In shared representation learning on ZSL, we consider nonlinear functions that map each modality as input into the same space Z , i.e. fx : X → Z and fa : A → Z . If each mapping can be trained so that the same representation can be extracted from each modality of the same example, i.e. ∀izxi = zayi , where zxi = fx(xi) and zayi = fa(ayi), we should be able to predict labels of the images in the test set by selecting the class that corresponds to the closest representation, i.e. ŷj = argminy∈Y dis(zxj , zay), where dis(a, b) is an arbitrary distance function between a and b. Therefore, the simplest approach is to train the mappings so that the distance of the representations of the two modalities is small on the training set, i.e. minfx,fa ∑Ntr i=1 dis(fx(xi), fa(ayi)).
However, this method is highly likely to fail for the following two reasons. First, although only one example of attributes corresponding to a class exists, there are countless examples of images corresponding to that class. Therefore, if we attempt to make representations of attributes and images contained in one class closer, then these representations will degenerate to one point. Another reason is that if one can train to correspond well between each modality of training data, then that representation might not generalize to test data.
To solve these problems, we consider the uncertainty of each example of each modality. To account for this uncertainty, we replace the deterministic embedding functions, fx and fa with the deep probabilistic distributions, p(zi|xi) and p(zi|ayi). The benefits of considering distributions for each example are twofold. First, we can prevent the degeneracy of representation which occurs when making examples of modalities that do not closely correspond one-to-one. This is true because we can consider the distance between distributions of each example, not between each example. Second, minimizing the distance between distributions of examples of each modality implies that the distribution of each modality is closer, i.e.,, ∀i ∈ {1..., Ntr}[p(zi|xi) = p(zi|ayi)] ⇒ p(Zx) = p(Za), where Zx = {zx1..., zxNtr} and Za = {zay1 ..., zayNtr }. This point might be readily apparent from ∏Ntr i=1 p(zi|xi) = p(Z|X) = p(Zx). Therefore, if we can train to bring each distribution closer, then we can obtain shared representation of them.

Review summaries:
This paper proposes a multimodal VAE model for the problem of generalized zero shot learning (GZSL). In GZSL, the test classes can contain examples from both seen as well as unseen classes, and due to the bias of the model towards the seen classes, the standard GZSL approaches tend to predict the majority of the inputs to belong to seen classes. The paper proposes a multimodal VAE model to mitigate this issue where a shared manifold learning learn for the inputs and the class attribute vectors.