Correlating Variational Autoencoders Natively For Multi-View Imputation

Authors: ella s c orme
marina  evangelou
ulrich  paquet
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-24 00:01:32

Abstract: Multi-view data from the same source often exhibit correlation. This is mirrored in correlation between the latent spaces of separate variational autoencoders (VAEs) trained on each data-view. A multi-view VAE approach is proposed that incorporates a joint prior with a non-zero correlation structure between the latent spaces of the VAEs. By enforcing such correlation structure, more strongly correlated latent spaces are uncovered. Using conditional distributions to move between these latent spaces, missing views can be imputed and used for downstream analysis. Learning this correlation structure involves maintaining validity of the prior distribution, as well as a successful parameterization that allows end-to-end learning.

Main paper: 
 introduction: 
 Data from multiple sources describing the same subjects arises in a wealth of settings. This can be clinical information of patients alongside genetic information and scan data. Datasets consisting of multiple views are referred to as multi-view or multi-modal data. There are instances where not all views are always available for every realisation. For example, a patient may miss an appointment or machinery may falter, resulting in no reading for a specific view. Missing data results in smaller usable datasets and reduced statistical power with many methods only applicable to full datasets [1] and can result in reduced performance in downstream analysis [2] . This manuscript presents a multi-view imputation approach, where its aim is to impute the realisations of a missing view by incorporating the information learnt from the other view.The proposed multi-view imputation method, named Joint Prior Variational Autoencoder (JPVAE), is based on variational autoencoders (VAEs) [3] , with the views connected solely through a joint prior on the VAEs' latent embeddings. Standard autoencoders seek to encode a latent representation of data, and from this encoding reconstruct the original data via a decoder. Multi-view approaches allow the latent representation of a missing view to be obtained, from which the reconstruction can be used as an imputation of the missing view. Several multi-view imputation approaches exist in the literature based on autoencoders [4] . However, as variational autoencoders learn a continuous embedding, they provide better interpolation of the latent space than standard autoencoders, making them a more suitable approach for an imputation method. The proposed joint prior in JPVAE incorporates a non-zero correlation structure that is found to increase the observed correlation between views in the latent spaces. This allows for successful movement between latent spaces, improving the ability to impute missing views.Various approaches have been proposed extending VAEs to the multi view case, including those by [5] [6] [7] [8] , with missing data imputation included as a feature of these methods. Most recently, proposed methods differ via their method of approximating a joint posterior. Daunhawer et al. [9] discuss the undesirable upper bound these methods put on a lower limit of the log-likelihood used as the objective function in VAEs. In contrast to existing approaches, which largely assume some joint posterior and/or shared latent space, our work has the novelty that is based on a joint prior between the latent variables. To the best of our knowledge, this is the first attempt made to correlate the latent spaces of VAEs natively through a joint prior. JPVAE is most similar in structure to the private version of Deep Variational Canonical Correlation Analysis (VCCA-private), a multi-view VAE approach by Wang et al. [10] . However their model contains both private and shared variables in the latent space and doesn't have a suitable approach for imputing missing views. JPVAE can also be seen as an application of the ideas from self-supervised learning techniques such as Barlow Twins [11] , where the embeddings from noisy versions of the same input are driven to be highly correlated via a loss function.Figure 1 illustrates an application of the proposed approach JPVAE, on imputing view 2 of the data that corresponds to the bottom half of MNIST digits using the top half of the digits data (view 1). Through the proposed approach, reconstructions of missing views can be obtained. This is achieved through the conditional distribution between the two latent variables, as illustrated in Figure 2 . By incorporating a joint prior with a non-zero cross-correlation structure, we observe better quality results in imputing view 2. Further realisations of image imputation, including imputation of view 1 from view 2, can be found in Appendix A.5. A novel multi-view imputation approach based on variational autoencoders, named JPVAE, is proposed that in contrast to existing work assumes a joint prior between the latent variables. 1 As the multiple views are correlated, the generated latent spaces are found also to be correlated. JPVAE takes advantage of this correlation and whilst marginals for each VAE are assumed to follow a standard normal distribution, a joint prior is assumed with a non-zero cross correlation. The imposed correlation structure is learnt natively by forcing a stronger correlation in the latent space. Section 2 presents the proposed method and related theorems that showcase the validity of the proposed method. As part of our work, we present and prove theorems that enable the parameterization of differentiable positive definite matrices that allow end-to-end learning. Through our conducted experiments presented in Section 3, learning the correlation structure leads to improved imputation ability, lower reconstruction loss and better predictive likelihood. Imputed views from JPVAE can be used for downstream tasks, with improved classification accuracy demonstrated. Lastly, better VAE models are learnt with JPVAE preventing posterior collapse, a common problem observed with VAEs [12] .Throughout this paper matrices are denoted by capital letters and column vectors are denoted by lower case letters, both emboldened. X = {x i } n i=1 with vector x i ∈ R c represents an entire dataset in R n×c , with x i an individual realisation. A diagonal matrix with vector a along the diagonal is represented by diag(a). A block diagonal matrix with matrices A i along the diagonal is represented by bdiag(A i ). Vertical concatenation of column vectors a and b is denoted by (a; b) i.e. (a; b) = (a T ; b T ) T . The eigenvalues and singular values of matrix A are denoted by λ i (A) and σ i (A) respectively, indexed by subscript i in order of decreasing magnitude. I denotes an identity matrix of relevant dimension. 
multi-view variational autoenconder with a joint prior: 
 This section presents the novel multi-view VAE approach JPVAE where each view has a separate associated VAE. Before discussing the proposed approach in detail, the main concepts of VAEs are introduced. Subsequently the single-view VAE is extended to the multi-view setting and the joint prior is presented. The different components of JPVAE are presented including theoretical work on matrix parameterization which enables end-to-end learning.A standard VAE seeks to encode data X in a probabilistic latent space and decode from this space to reconstructed data X [3] . The goal is for X to be as close as possible to X. As the latent space is a probabilistic embedding rather than one obtained by a deterministic mapping, it has the advantage that it can be explored fully, including the sampling of new realisations from the same distribution as X. These realisations are assumed to be taken from the same underlying distribution as X, p θ (•) = p(•|θ) where θ is the set of parameters defining the distribution.VAEs are a type of variational Bayesian method that seek to find a lower bound for this marginal probability, p θ (X), through a Bayesian framework. The likelihood of x ∈ R c given latent variable z ∈ R d is denoted by p θ (•|z) and the prior for the latent variables, usually taken to be a standard normal, is denoted by p θ (z). The encoder and decoder are neural networks that seek to learn the posterior distribution of z given x, p θ (•|x), and the likelihood of x given z respectively, given the assumed prior over the latent space. In order to make the posterior learnable, an approximate posterior distribution is used that usually is a multivariate normal distribution. The approximation p θ (•|x) ≈ q ϕ (•|x) is made where ϕ denotes the parameters of the probabilistic encoder, q ϕ (•|x). The encoder maps an input x to a mean vector µ(x) ∈ R d and to the log of the vector of variances, σ 2 (x) ∈ R d . A sample is drawn from the approximate posterior distribution and this is used as an input to the neural network which acts as the decoder, D θ . As ϕ appears within the distribution of the latent variables, derivatives cannot simply be taken inside the expectation term. Differentiability of the loss function is required for the loss-driven parameter update steps and therefore a reparamaterization trick needs to be implemented [3] .Instead of drawing z directly, ϵ ∼ N (0, I) is drawn and z = µ(x) + diag(σ 2 (x))ϵ is determined.The distribution over which we take expectation is now independent of the parameters for which we take derivatives, and derivative update steps may now be implemented. The neural network decoder then seeks to reconstruct x from the latent variable z.A maximum likelihood principle is then applied to the marginal likelihood of the data p θ (X) to obtain estimates for the parameters within the encoders and decoders. As the likelihood is intractable a lower bound on the log-likelihood, known as the Evidence Lower Bound (ELBO) is instead maximised, given by:EQUATIONwhere D KL (r|s) denotes the Kullback-Leibler (KL) divergence between distributions r and s [13] .E z∼q ϕ (•|x) (•)denotes the expectation with respect to the conditional distribution of z given x.Maximising the ELBO is equivalent to finding a balance between an approximate posterior that is close to the prior and putting weight on the latent variable space that maximises the likelihood of the data given these same variables, ln(p θ (x|z)). Without the KL term, the delta function is returned as the approximate posterior and the autoencoder is recovered.With this VAE formulation the KL term often becomes very small or 'vanishes' leading to the posterior equalling the prior (posterior collapse). This is referred to as KL vanishing and leads to a decoder that is largely independent of the latent variables. See [14] for further discussion of this problem. To combat this, we implement KL annealing -a procedure where a weight β is introduced on the KL term and gradually increased, typically from 0, as learning occurs [15] . The objective function becomes:EQUATIONIt is now assumed two data views with shared row dimension are present and represented byX = {X 1 , X 2 } with X i ∈ R N ×ci .For brevity, the sample subscript is dropped and x i ∈ R ci refers to an individual realisation from view i. Here N is the number of rows in both views and c i is the number of columns in view i = 1, 2. This shared row dimension implies the views are paired, with row j representing the same individual/sample in both views. This allows for meaningful correlation in the latent space. The notation introduced in this section for single view VAEs is used in the following sections where multi-view VAEs are presented. Subscripts are used to denote the relevant views e.g. the encoder and decoder in view i are represented by E iϕi and D iθi respectively.As the different views in a multi-view dataset are from a common source, correlation exists between the views. This translates to correlation between the latent spaces of each view from independently trained VAEs. JPVAE takes advantage of this correlation, enforcing the relationship between the two views via a joint prior on the latent variables, as illustrated in Figure 3 . Enforcing the proposed correlation structure between the latent spaces ensures we can move from the original space where data are highly correlated in a non-linear fashion, to a space where the correlation is linear, and back to the reconstructed space that has a non-linear correlation. The marginal prior on the latent variables corresponding to each view is a standard normal, as with the traditional VAE. However, a cross-covariance matrix C is assumed between the latent variables z 1 and z 2 . Having separate VAE structures for each view assumes conditional independence in both directions: (a) given the data the latent variables are independent, and (b) given the latent variables the data are independent. This allows the unique features of each view to be encoded and decoded.As the latent spaces are linearly correlated, it is possible to move between them via the conditional distribution, as illustrated in Figure 2 . This allows for imputation of missing views, obtaining a reconstruction of x 2 solely from realisation x 1 (and vice versa). If a joint prior is not assumed, separate VAEs are trained on each view and there is no correlation enforced between latent spaces. Whilst some correlation exists between the latent spaces, it is not as strong, and therefore may not produce as accurate reproductions, as illustrated in the numerical experiments of Section 3.By assuming that latent variables for each view are independent given the data, the approximated posterior distributed can be expressed as:EQUATIONThe individual approximate posteriors q ϕi (•|x i ) are multivariate Gaussians with mean and covariance matrix determined by the output of E iϕi . For input x i , these are µ i (x i ) and diag(σ 2 i (x i )) respectively. As the posteriors are assumed to be independent, the joint distribution is multivariate Gaussian with mean (µ 1 (x 1 ); µ 2 (x 2 )). The covariance matrix is represented by bdiag(Σ i (x i )) withΣ i (x i ) = diag(σ 2 i (x i )).Similarly, by assuming the data is independent given the latent variables, the likelihood function can be expressed as:EQUATIONThis is equivalent to having separate encoders and decoders for each view but with a joint prior.Assuming independence means the expectation term in the ELBO can be separated into terms corresponding to the separate views:E z∼q ϕ (•|x) [lnp θ (x|z)] = E z1∼q ϕ 1 (•|x1) [lnp θ1 (x 1 |z 1 )] + E z2∼q ϕ 2 (•|x2) [lnp θ2 (x 2 |z 2 )] .The objective function therefore becomes:EQUATION-βD KL (q ϕ1 (•|x 1 )q ϕ2 (•|x 2 ) ∥ p θ (•)).There are assumed to be n i latent variables in the VAE for view i, represented by the random vector z i ∈ R ni . Without loss of generality it is that assumed n 1 ≤ n 2 . The joint prior distribution of the random vector z = (z 1 , z 2 ) ∈ R n1+n2 is assumed to be a multivariate normal with mean 0 and covariance matrix, Σ C :EQUATIONwhere C ∈ R n2×n1 is the cross-covariance matrix encapsulating the relationship between the two latent spaces. I i is the n i × n i identity matrix. As Σ C is a covariance matrix, it needs to satisfy two conditions: symmetry and positive semi-definiteness. With the defined structure it is clearly symmetric and so it is sufficient to require Σ C to be a positive semi-definite matrix to ensure it is a well-defined covariance matrix. For a covariance matrix with structure as defined in Eq. 6, it is useful to note that Σ C is positive definite if and only if I 1 -C T C (or I 2 -CC T )As discussed earlier, for calculating the ELBO for the proposed prior, the KL divergence between q ϕ1 (•|x 1 )q ϕ2 (•|x 2 ) and the prior on (z 1 , z 2 ), p C , is required. The following general result on KL divergence between multivariate Gaussians is applied in this setting.Between two k-dimensional multivariate Gaussians r and s with respective means µ r and µ s and respective covariance matrices Σ r and Σ s , the KL divergence is given by [18] :EQUATIONAs this assumes the existence of Σ -1 r and Σ -1 s , both covariance matrices must be positive definite. By utilising the result of Eq. 7 with the prior over z = (z 1 , z 2 ) set as r = p C , and the approximate posterior distribution as s = q ϕ1 (•|x 1 )q ϕ2 (•|x 2 ), the KL term of Eq. 2 is obtained. Explicitly, p C = N (0, Σ C ) and q ϕ1 (•|x 1 )q ϕ2 (•|x 2 ) = N ((µ 1 (x 1 ), µ 2 (x 2 )), Σ q ) with Σ q = bdiag(Σ i ). For the utilisation of Eq. 7, it is assumed that Σ C is positive definite and the variances of the approximate posterior are positive.Using the block matrix inverse result from [19, p. 18] (stated in Appendix A.1.1) the inverse of Σ C is given by:EQUATIONwhereD 1 = (I 1 -C T C) -1 and D 2 = (I 2 -CC T ) -1. These inverses are guaranteed to exist, given the assumption of positive definiteness on Σ C . Using Eq. 8 and properties of the trace, we obtain:tr Σ -1 C Σ q = tr D 1 Σ 1 -D 1 C T Σ 2 -D 2 CΣ 1 D 2 Σ 2 = tr {D 1 Σ 1 } + tr {D 2 Σ 2 } . Additionally, as Σ C is a block matrix, |Σ C | = |I 1 -C T C| = 1/|D 1 | = 1/|D 2 | [20, p. 114].Similarly. as Σ q is a diagonal matrix we have|Σ q | = |Σ 1 ||Σ 2 |. Writing µ i = µ i (x i) and noting that µ p = 0, we obtain:D KL (q ϕ1 (•|x 1 )q ϕ2 (•|x 2 )||p C ) = 1 2 µ T 1 D 1 µ 1 -ln |Σ 1 | -n 1 + tr {D 1 Σ 1 } (9) + 1 2 µ T 2 D 2 µ 2 -ln |Σ 2 | -n 2 + tr {D 2 Σ 2 } - 1 2 ln |D 1 | + µ T 1 D 1 C T µ 2 + µ T 2 D 2 Cµ 1 .The matrix C is unknown and therefore must be either chosen apriori or be optimised. In our work, C is updated along with all other parameters at each update step, using the ELBO function as the loss function. EQUATION=⇒ 1 ≤ λ k (I 1 -C T C) -λ n1+1-k (-C T C) ≤ 1which combined with the relations above gives λ k (I 1 -C T C) = 1 + λ n1+1-k (-C T C) = 1 -σ 2 n1+1-k (C). Therefore all eigenvalues of I 1 -C T C are non-negative if and only if all singular values of C are bounded by 1.If the inequality on σ 1 (C) is replaced by a strict inequality then Σ C is guaranteed to be positive definite. This is clear as all eigenvalues of I 1 -C T C are now positive which ensures positive definiteness. This guarantees the applicability of Eq. 9. A further restriction, outlined in Theorem 2.2, can be made which enables a different parameterization of C which assumes a scaled orthogonal relationship between the views.Theorem 2.2. If C = α C ∈ R n2×n1 where |α| ≤ 1 and C is a semi-orthogonal matrix ( CT C = I 1 ) then Σ C is positive semi-definite.Proof. Given the assumed structure on C, we have the following:EQUATION)As I 1 is positive semi definite, if |α| ≤ 1 then, so too isI 1 -C T C.Alternatively, this could be seen as a corollary to Theorem 2.1 -due to the semi-orthogonality of C, all singular values of C are equal to 1 which means σ i (C) = α with |α| ≤ 1. Again, if the condition on α is replaced with a strict inequality (|α| < 1)) this guarantees applicability of Eq. 9. A simplification of Eq. 9 assuming this orthogonality condition can be found in Appendix A.2.To ensure applicability of Eq. 9, we can either require As discussed by Shepard et al. [22] , there are four popular parameterizations of orthogonal matrices. All methods in their preliminary form parameterize at most a subset of the orthogonal matrices (at maximum those with determinant +1 or -1). These parameterizations must therefore be extended to map to the entire orthogonal matrix space. As it is the only one-to-one mapping, the rational Cayley transform has been chosen for use within JPVAE. In its original form, the Cayley transform tells us that for all orthogonal matrices with no eigenvalues equal to 1, there exists a unique skew-symmetric matrix A ∈ R m×m such that O = (I + A)(I -A) -1 [22] . To extend the parameterization to the full space of orthogonal matrices, an extra matrix consisting of 1s and -1s along the diagonal is needed [23, 24] . Let J = diag([1 m-r ; -1 r ]) where r = floor(m/(1 + exp(-s))) for s ∈ R. Then O = J (I +A)(I -A) -1 is a mapping onto the space of orthogonal matrices. C is therefore parameterized in full by (n -1) 2 /2 + 1 parameters:({a ij ∈ R : i < j for i, j ∈ {1, • • • , n}} , {s ∈ R}).Once the model has been learned on the training data, missing views can be imputed. Assume data is available for view j, but not view i, given x j we can impute the missing value of x i . Data x j is fed into the encoder for view j, E jϕ , and latent variables are sampled giving z j = a. An estimate of z i can be obtained using the conditional distribution of z i given z j . This is fed through decoder i, D iθ , to produce an estimate of x i given x j , xi|j .The joint marginal of z 1 and z 2 is assumed to be a multivariate normal with mean [µ 1 ; µ 2 ] and covariance matrix Σ. The maximum likelihood estimator for the mean and covariance are obtained and denoted by [ μ1 ; μ2 ] and Σ respectively. As any subset of variables from a multivariate normal conditioned on a known second subset of variables also follows a multivariate normal distribution, the conditional distribution can be found explicitly. For i ̸ = j, the distribution of z i given z j = a is observed is:EQUATIONwhere Σkl ∈ R n k ×n l is the submatrix of Σ corresponding to the variables associated with view k and view l. The conditional mean E(z i |z j = a) = μi + Σij Σ-1 jj (a -μj ) is then used as an estimate of the latent variables in latent space i and fed into D iθ to obtain xi|j , the imputed value of x i given x j . 
numerical experiments: 
 Through a series of experiments the performance of JPVAE is explored for both imputation purposes as well as for downstream analyses like classification. As JPVAE enables imputation of missing views ( Xi|j ), this ability is investigated alongside reconstruction of views ( Xi ).A multi-view dataset was created from the binary version of the popular MNIST dataset, which consists of handwritten digits from 0 to 9 [25] . For each image, the top half was taken as view 1 and the bottom half was taken as view 2. This dataset of halved images is referred to as hvdMNIST. The hvdMNIST dataset has the desirable property of having a strong correlation between views. The dataset contains 50, 000 training images and 10, 000 test images. Experiments are repeated with 5 different random seeds and the average and standard deviation reported. Details on the model architecture and training details can be found in Appendices A.3 and A.4 respectively. Two variants of JPVAE are explored, where in each one the validity of Σ C is enforced via different ways. The first variant imposes a bound on singular values (such that σ 1 (C) < 1). The second variant enforces a scaled orthogonality where CC T = C T C = α 2 I, with the value of α set as 0.95. The two variants of JPVAE are compared with the C = 0 case, which corresponds to completely disjoint VAEs for each view.Without explicitly learning a correlation structure, correlation is present between the two generated latent spaces. By incorporating a joint prior with a non-zero cross-correlation as presented in Eq. 6 into the loss function, JPVAE increases the correlation between views in the latent space as shown in Figure 4 . Table 1 illustrates the improvement in the ability to reconstruct view 2 given view 1 and vice versa when correlation structure is learnt. Enforcing the orthogonality restriction improves the performance of JPVAE compared with simply applying the singular value bound. The improved performance of the method is enabled by the learnt correlation that prevents posterior collapse. Following the work by [26] , the phenomenon of posterior collapse is indicated by the percentage of active units (AU). The activity of unit u in the latent space is measured by A u = Cov x E u∼q(u|x) [u] . A unit is considered active, i.e. to not have suffered from posterior collapse, if A u ≥ 10 -2 . A higher percentage of AUs are preserved when C is learnt, with the best case scenario observed with the orthogonality constraint (Table 2 ). Wang et al. [12] proved that posterior collapse is equivalent to latent variable non-identifiability. This indicates that by enforcing the orthogonality restriction, we make the latent variable space identifiable. CC T = C T C = α 2 I 98.5 (2.2)The imputed views, X1|2 and X2|1 , can be used for downstream tasks. If not all views are available for an individual, this allows techniques requiring all views to be applied. As an illustration of the performance of the imputed data in downstream tasks, a basic multi-layer perceptron classifier was trained and tested on different combinations of reconstructed and imputed views. The performance of a classifier trained on the reconstructed data indicates that when C is learnt, the relevant signal remains present in both the reconstructed and the imputed data. The performance on imputed data is greatly improved by learning C, and in particular by enforcing the orthogonality constraint (Table 3 ; results with standard deviation can be found in Figure 9 .). Table 3 : Results for (Y , Z) represent classification accuracy % for model trained on the training split of Y and tested on the test split of Z. Accuracies for (X 1 , X 1 ) and (X 2 , X 2 ) with standard deviation in brackets are 93.59% (0.25) and 90.83% (0.23) respectively. Best results in bold. For clarity, results with standard deviation reported can be found in Figure 9 .View 1 View 2 1 ). This may be due to the increased use of the latent space, as evidenced by the higher percentage of active units. Whilst a classifier trained on the imputed data from C = 0 demonstrates the retention of signal, the low accuracy seen for a classifier trained on reconstructed data and high reconstruction loss indicates that it does not retain the specific signature of the input. For example, taking the top half of a digit '2' as the input, it may correctly reconstruct the bottom half of a realisation of the digit '2' but not correctly reconstruct the specific realisation (as seen in Figure 1a ).EQUATIONUsing the joint prior we see that the view with stronger signal (view 1) is bolstering the classification of the view with weaker signal (view 2). Whilst classification accuracy on x 1 is higher than that on x 2 , the accuracy on the imputed data X2|1 sees a smaller drop, and is greater than that of X1|2 . Notably, the accuracy on imputed data ( X2|1 , X2|1 ) is comparable to that on ( X2 , X2 ) whilst the same for view 1 experiences a drop in performance. 
conclusions: 
 A novel multi-view VAE approach has been proposed that natively strengthens the correlation between latent spaces via a joint prior. This is the first time that a connection between multi-view VAEs is made through a joint prior rather than a joint posterior, as has previously been implemented in the literature. Theoretical guarantees and parameterizations are presented that allow for end-to-end learning. By simultaneously preventing posterior collapse, JPVAE returns superior models and demonstrates a promising ability to impute missing data suitable for downstream tasks. 
 
 Appendix: Additional results are presented in this section. Preprint.The code for implementation of JPVAE and the numerical experiments contained in this manuscript can be found at https://github.com/eso28599/JPVAE. 


Decision: In light of the positive reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: A good idea of learning joint priors that model the correlation between latent spaces in the VAEs
main_review: main_review: The paper proposes an end-to-end training framework for learning joint priors that captures the correlations between latent spaces of multi-view data in VAEs.

The idea is described clearly, however it lacks sufficient results and visuals that shows that the idea generally works as expected. Moreover, the experiments were conducted on MNIST dataset, which is pretty simple and non-comprehensive. The authors also do not show enough visual results from the VAE multi-view imputation.

Overall, the idea is good, but it needs to be analyzed on more complex datasets, which can be the scope of a future publication.
title: title: Useful extension of the VAE framework for use in data from 2 views
main_review: main_review: In this paper the authors present a method to use VAEs in order to generate data coming from two distinct views, an example being medical data where the same patient is scanned by different scanners or under different conditions. Their method presents a way to train two VAEs such that the latent space distribution between both views are correlated, and through experiments demonstrate improvements in reconstruction over using two uncorrelated VAEs. The authors also demonstrated that by using the VAE structure, they are able to impute a missing view given the first one. 

While the paper presents good ablation comparisons and contributes to the workshop, I believe that the paper needs to be compared to a model that performs the same task but is part of another family, rather than a special case of the same model.
title: title: The paper is well written, background, theoretical part and method is very clear and concise. The scope fits very well to the workshop. Few questions regarding the results though.
main_review: main_review: There is very little to add to the introduction and the background.

1) I am missing a theoretical link to other alignment loss functions, as the proposed covariance structure shows similarities to other self-supervised loss functions (e.g. contrastive loss, barlow-twins...).

2) As a qualitative evaluation of Figure 1, a classifier could be trained on concatenated native views vs correlated views.

3) Does the correlation increase if one trains a single decoder with shared parameters for $z_1$ and $z_2$, instead of two separate decoders?

Comments: title: Review reply
Review reply
Review reply
comment: Many thanks for your positive comments and your time taken in reviewing our work. In response to your questions:
1. Thank you for pointing this out,  a comment has been added to the introduction mentioning the similarity to self-supervised learning techniques such as Barlow twins. However, as our method is developed for multi-view data rather than noisy versions of the same view, no further discussed is had. 
2. This has been implemented with detailed results also contained in the appendix. The results show that classification performed on the concatenation of the two views is greatly improved by enforcing correlation between the latent spaces.
3. As implementing a single decoder corresponds to a very different model structure and the implementation of a joint posterior (which is the focus of entire methods) this is beyond the scope of this work. However, this would be a worthwhile comparison, and we plan to compare the approach with other popular multi-view VAE methods (including those that implement joint posteriors) in future work.
Many thanks for your time in reviewing our work. We agree it would be helpful to compare the model to a method from another family. This manuscript has focussed on the presentation of this novel method and any comparisons with other methods will be the focus of future work.
Many thanks for your time in reviewing our work. In response to your concerns regarding:
- Insufficient results:  we have now performed repeats and applied the classifier to the image re-stitched together to further illustrate successful reconstruction/imputation. 
- Insufficient visuals: we have added to the appendix a figure illustrating imputing the top half of the image from the bottom, as well as replicates.
- Lack of complexity on analysed dataset – we agree, MNIST has been used here to illustrate proof of concept and it is our plan to analyse its performance on more complicated data in future work.