Deep Incomplete Multi-view Learning via Cyclic Permutation of VAEs
Authors: Xin Gao
Jian Pu
Venue: ICLR 2025 Poster (2025)
Keywords: Multi-View Learning
Representation Learning
Multimodal VAEs
Generative Models
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: This paper proposes the Multi-View Permutation of VAEs (MVP), designed to learn more sufficient and consistent representations from incomplete multi-view data.

Abstract: Multi-View Representation Learning (MVRL) aims to derive a unified representation from multi-view data by leveraging shared and complementary information across views. However, when views are irregularly missing, the incomplete data can lead to representations that lack sufficiency and consistency. To address this, we propose Multi-View Permutation of Variational Auto-Encoders (MVP), which excavates invariant relationships between views in incomplete data. MVP establishes inter-view correspondences in the latent space of Variational Auto-Encoders, enabling the inference of missing views and the aggregation of more sufficient information. To derive a valid Evidence Lower Bound (ELBO) for learning, we apply permutations to randomly reorder variables for cross-view generation and then partition them by views to maintain invariant meanings under permutations. Additionally, we enhance consistency by introducing an informational prior with cyclic permutations of posteriors, which turns the regularization term into a similarity measure across distributions. We demonstrate the effectiveness of our approach on seven diverse datasets with varying missing ratios, achieving superior performance in multi-view clustering and generation tasks.

Published as a conference paper at ICLR 2025
DEEP INCOMPLETE MULTI-VIEW LEARNING VIA CYCLIC PERMUTATION OF VAES
Xin Gao
Jian Pu
Fudan University
Fudan University
gaoxin238m fudan edu. en
jianpuž fudan. eduu en
ABSTRACT
Multi-View Representation Learning (MVRL) aims derive unified represen- tation from multi-view data by leveraging shared and complementary informa- tion across views. However, when views are irregularly missing, the incomplete data can lead to representations that lack sufficiency and consistency. To address this, we propose Multi-View Permutation of Variational Auto-Encoders (MVP), which excavates invariant relationships between views in incomplete data. MVP establishes inter-view correspondences Encoders, enabling the inference of missing views and the aggregation of more sufficient information. To derive a valid Evidence Lower Bound (ELBO) for learn- ing, apply permutations randomly reorder variables for cross-view genera- tion and then partition them by views to maintain invariant meanings under per- mutations. Additionally, we enhance consistency by introducing an informational prior with cyclic permutations of posteriors, which turns the regularization term into a similarity measure across distributions. We demonstrate the effectiveness of our approach on seven diverse datasets with varying missing ratios, achieving superior performance in multi-view clustering and generation tasks.
the latent space Variational Auto-
INTRODUCTION
Multi-view data are prevalent in real-world applications', capturing various aspects of a shared subject. Examples include observing a 3D object from multiple angles, applying diverse image feature descriptors, or reporting the same supervised signals that enable the extraction of meaningful patterns. Multi- View Representation Learning (MVRL) aims maps this data into latent space, integrating information from multiple views into unified representation for downstream tasks like clustering, classification, and genera- tion (Li et al., 2018). However, in practice, not all views are available for every sample, presenting the challenge of Incomplete Multi- View Representation Learning (IMVRL) (Tang al., 2024). The incomplete data complicates the integration of information from different views, making difficult to derive high-quality representations under varying missing ratios.
in different languages. This offers valuable self-
Among MVRL methods, Multimodal Variational Auto-Encoders (MVAEs) stand out for their ro- bustness in modeling the latent distributions of multi-view data (Aguila & Altmann, 2024). Their flexibility in handling incomplete data stems from mean-based fusion strategies, such as Mixture- of of-Experts (MoE) and Product-of-Expee Experts (PoE) (Wu & Goodman, 2018; Shi et al., 2019; Sutter et al., 2021), which can accommodate varying numbers views and have been validated by Hwang al. (2021). Furthermore, some MVAE variants enhance inter-view consistency by incorporating data-dependent priors, which facilitate better information integration (Sutter et al., 2020; 2024) and reduce reliance on missing views (Hwang et al., 2021). Despite these advancements, the sufficiency and consistency of learned representations are still not assured in the presence of incomplete data. Samples with fewer views inherently provide less information, which hampers the integration and undermines inter-view consistency. As a result, these methods experience significant performance degradation and semantic incoherence across views as the rate of missing data increases.
Multi-view: In this paper. we follow Hwang et al. (2021), using the term "view" broadly to refer to what other works may call view (Lin et al., 2023; Hwang et al., 2021), modality (Sutter al., 2021; Huh et al., 2024), any perspective describing different aspects of common entity.

---

Published as a conference paper at ICLR 2025
To address the challenge in IMVRL, an increasing number of studies have proposed inferring miss- ing views through cross- gooss-geneattion from available ones. These methods leverage the invariant rela- tionship between views. meaning that converting one view into another preserves the sample-specific information while altering only the style (Zhang et al., 2020; Lin et al., 2023; Cai et al., 2024; Jiang et al., 2023). This intrinsic property of multi-view data allows for deeper extraction of information from it. Furthermore, Huh et al. (2024) observe phenomenon of representational convergence, indicating that inter-view transformation can be easily achieved using simple mappings in an well- aligned representative space. This occurs naturally in MVAEs, where multiple encoders map dif- ferent views into latent space and align their representations by promoting inter-view consisteney. Building on these insights, we aim to explicitly establish inter-view correspondences (Huang et al., 2020) MVAEs, effectively learning and enriching the latent space with invariant relationships between views, thus enabling the inference of representations for missing views.
In this paper, we propose the Multi- Vieww Permutation of VAES (MVP), designed to learn more suf- ficient and consistent representations from incomplete multi-view data. MVP captures inter-view relationships by modeling correspondences between views, which enables latent variables to be transformed from one view another. To facilitate these transformations, that randomly shuffle variables within each view-either through self-view encoders or cross-view correspondences. Next, partition variables by view preserve their invariant meanings under permutation, which helps factorize the joint posterior and derive a valid Evidence Lower Bound (ELBO) for optimization. Additionally, we propose an informational prior based on cyclic permu- tations of posteriors, which converts the Kullback-Leible (KL) divergence term into a similarity measure among distributions. We validate the effectiveness of our method through experiments in multi-view clustering and generation tasks. The key contributions of this work are:
apply permutations
We enhance MVAEs by modeling inter-view correspondences in the latent space to infer missing views. Our novel approach of applying permutations and partitions to the latent variable set leads to the derivation of a valid ELBO for optimization.
We introduce an informational prior using cyclic permutations of posteriors. This results in the regularization term into a similarity measure to enhance consistency between views.
Quantitative and qualitative results on seven diverse datasets, across different missing tios, show that our approach learns more sufficient and consistent represeetttesssss compared to other IMVRL methods and MVAEs
2 RELATED WORKS
Incomplete Multi- View Representation Learning (IMVRL) Early IMVRL approaches addressed incomplete data by grouping available views and applying classical methods like CCA (Hotelling, 1992). DCCA (Andrew et al., 2013) introduced nonlinear representations via correlation objectives, while DCCAE (Wang et al., 2015) enhanced reconstruction with autoencoders. As missing rates in- creased, the need for handling incomplete information grew. Methods like DIMVC (Xu et al., 2022) projected representations into high-dimensional spaces to improve complementarity, and DSIMVC (Tang & Liu, 2022) used bi-level optimization to impute missing views. Completer (Lin al., 2021; 2023) maximized mutual information and minimized conditional entropy to recover missing views, while CPSPAN (Jin et al., 2023) aligned prototypes across views to preserve structural consistency. ICMVC (Chao et al., 2024) proposed high-confidence guidance to enhance consistency, and DVIMC (Xu al., 2024) introduced coherence constraints to handle unbalanced information.
Multimodal Variational Auto-Encoders (MVAEs) MVAES are generative models that maximize the log-likelibar likelihoood of observed data through latent variables. MVAE (Wu & Goodman, 2018) models the joint posterior using PoE (Hinton, 2002), though this may hinder unimodal posterior optimiza- tion. MMVAE (Shi et al., 2019) and mmJSD (Sutter et al., 2020) use MoE for the joint posterior, with MMVAE applying pairwise optimization for reconstructing all views, but struggling to ag- gregate information efficiently. mmJSD addresses this with a dynamic prior, replacing regulariza- tion with Lensen-Shannoe Divergence. Sutter et al. (2021) propose Mixture-of-Prodductoff Product-of-Exxee -f-ppetss (MoPoE) to decompose KL divergence into 2 terms, while MVTCAE (Hwang et al., 2021) intro- duces an information- -eeematton--thoretii objective using forward KL divergences. MMVAE+ (Palumbo et al., 2023) extends MMVAE by separating shared and view- new-pecullar information in latent subspaces and incorporating cross-view reconstructions.
2

---

Published as a conference paper at ICLR 2025
Informational Priors in VAE Formulations Tomezak & Welling (2018) first introduced data- dependent prior into VAE, which was later extended to multimodal VAEs for better inter-view con- sistency. Sutter et al. (2020) employed dynamic prior combined with the joint posterior to define Jensen-Shannon divergence regularization. Hwang et al. (2021) used view-specific posteriors as priors, regularizing the joint posterior ensure representations could be inferred from all views. Sutter et al. (2024) develop an MoE prior for soft-sharing of information across view-specific rep- resentations rather than simply aggregation. They relies fusion of posteriors and enforce strict alignment, while we encourage soft consistency between views after transformations.
3
METHOD
In this section, we present the core components of our method, which aims to capture relationships between views in incomplete multi-view data. Our approach models inter-view correspondences in the latent space of MVAEs, enabling the inference of missing views and the generation of more sufficient and consistent representations. To facilitate transformations between views, we apply permutations reorder the latent variables and introduce two partitions based on their encoding in- formation (Section 3.1). We then use these partitions to factorize the posterior and derive the ELBO for optimization (Section 3.2). Finally, we define priors using the permuted variables in the regular- ization term. By applying cyclic permutations to the posteriors, we transform the regularization into a similarity measure, enforcing consistency across views (Section 3.3).
3.1 INTER- VIEW CORRESPONDENCE AND LATENT VARIABLE PARTITION
Given incomplete multi-view dataset {X,}== 1=1 where each X: = (x(1)) ) ve,, consists of multiple
views, we denote by I, subset of the complete view indices [4] (1,2...., L} (i.e. I4 S [4). Each observed view (1) is a vector in Rd, For simplicity in derivation, we drop the subscript i from I₁ and use Z. We then encode the data from each view into latent variable set, where each view contributes complementary information. These latent variables, 2 E RR,, are derived using encoders parameterized by {o.).-. following standard MVAEs.
We adopt the term "correspondences" from Huang et al. (2020). but extend it to explicitly construct a mapping channel, referred as inter-view correspondences. Specifically, we introduce multiple nonlinear mappings from the w-th view to the I-th view, represented by functions (fie), parameter- ized by }. For each pair where v * 1, a unique mapping ftv establishes a direct relationship between the source view and the target view (. This allows for cross-view transformations, where information from one view informs the representation of another. The latent variables are then or- ganized into a set Z = =2( (=.")(.,DEIII))) where -00) denotes the representation of the 1-th view, with subseript indicating its source view. Specifically: (1) If o = I, 216) is directly encoded from the observed view (²)), following d-dimensional Gaussian distribution NV(=l") p(x(("),, ((
E(=("))), denoted as g(26") (=()); (o). (2) If u # 1, 2 is transformed from (() using fiv, following a Gaussian distribution (222)) fwo p(x(")), fiv E(x("))). denoted as q(21 (z(")) av))))
To organize the encoded latent variables, we construct a matrix Zo. as shown in Figure 1. The diago- nal elements of Zu correspond to the self-view encoding (case (1)), while the off-diagonal elements correspond to cross-view transformations (case (2)). The core idea is that variables with the same superscript / encode similar information about the I-th view, regardless of whether they are directly encoded or transformed. Thus, even if the columns of Zo are reordered, as illustrated in the transition from Zo Z1 in Figure 1, each column continues to represent the same view, and the information encoded by elements at corresponding positions remains invariant. This observation motivates us to group the latent variables by views. leading partition of the set Z. Mathematically, partition P of set X is a collection of non-empty, mutually disjoint subsets of X, also known as a "set of sets". Accordingly, we define the single-view partition for Z:
Definition 1 (Single-view Partition). A single-view partition of 2, denoted as Pa(Z), is defined as (S such that Ul_1 S = 2. The I-th single-view cell S = (z0 Yoez and S [Z].
(1)
Accordingly, each sample has a unique PZ( where each single-view cell S, consists of all vari- ables with the same superseript, representing the 1-th view. More specifically, these variables are all
3

---

Published as a conference paper at ICLR 2025
Eneoders & Inter-view Correspondences
Cyelic Permutation & Consensus Fusion
(c) Decoders
(i @ ding((5)
Incomplete Muln-view dataset [34)
1
40
05
01
g()
10
14
05
2
%,
2
x
O2
g
05
x22
ding
x33
x9
+
x
270
101
201
10
x, = [xx*)
ØL 8
8/2
O2
g0i
KL(Z,
KL(O,
Figure I: Overview of our method: (a) Incomplete multi-view data X is fed into encoders to gen- erate the diagonal elements of matrix Zo. while off-diagonal elements are derived through inter-view correspondences. (b) Latent variables are partitioned by columns for single-view partition {S,) and by rows for complete-view partition )C with each row aggregated into a consensus variable w, capturing shared information across views. A cyclic permutation within each column transforms Zo into Z1, generating new partitions (See Figure 5 for transformation details.). Regularization is applied by comparing distributions at the same positions before (Z)) and after (Z1) permutation. (c)
Each view z(()) is reconstrueted from its latent representation 2 (()) and a consensus variable w.
drawn from the /-th column of matrix Zo or Z1. The set S is expected to be homogeneous, meaning the distributions of the variables within it should be as close possible.
To combine complete information across views for downstream tasks, we randomly select variables from different views in Z. In practice, this is achieved by selecting row from matrix Zo or Z1. each representing a subset of Z that contains complete information from all L views. This approach leads to another partition based on combinations of complete views:
Definition 2 (Complete-ve view view Partition). A complete- partition of 2, denoted as (2))), is de- fined as (CrInEI such that UnET Cn = 2. Each complete-view cell Cn is given by {2v (1) L 1
where the index set In S I and UnEI Jn = I. The size of each cell is [c,l=I.
For each sample, multiple complete-view partitions satisfy Definition 2. This is evident in the matrix, where we can divide Zo by rows; each row corresponds to complete-view cell C, which consists of variables with superscripts ranging from to L and subscripts indicating their sources. After randomly reordering variables in each column, we obtain new (2( by similarly dividing Zı by rows. This reordering action is rigorously described as permutation of set, which is bijection from set to itself. Applying permutations each column in matrix Zo generates different PZ( facilitating the selection of any complete-viee -iew combination from each row.
To aggregate the shared information from L views, introduce consensus variable w, which is obtained from a cmmplete-view cell C,, We assume that the first k dimensions of z capture information common to all views, such as categorical features, and compute the geometric mean of the k-diimnnionnll marginal distributions within the set C,,. This approach combines the L complete views into a single, sharper k-dimensional Gaussian distribution with explicitly computed mean and covariance (Cochran, 1954), represented q(wn Cn, (2(")) Thus, a complete-view partition (Z) with [Z] cells produces a set Ω = {wn (wnn ~ q(wn, Cn, (z) ) C, E PZ) n E I}. The set Ω should also be homogeneous, meaning that any combination of L views fused in this way should encode consistent information.
.V& (W an, An), E I.
By defining these two partitions, we enable the factorization of the posterior in the following sec- tions, forming the foundation for our learning objective.
4

---

Published as a conference paper at ICLR 2025
3.2 POSTERIOR FACTORIZATION AND THE DERIVATION OF ELBO
Given the latent variables 2 and Ω our goal is to infer their posterior distribution p(2,0 S (x(*)) However, this exact posterior involves an intractable integral, so we approximate it with a variational posterior (2, (~("))) For a given complete-view partition Pc(Z) = IneI assume that the density factorizes follows:
(2, (z(")
II
NEI g(wn Cn. (z((1)) J,,)((Cn (x.()))3n)
II I Im,. {(>!) w
II NET g(wn, Cns {x.))))))
To optimize this factorized posterior, we minimize the KI. divergence between the true posterior and the variational posterior, expressed as:
KL
[ ( (x }z) ((, {z(") = logglgppeee p( (1 CELBO((a }z);
where ( Iz) is the Evidence Lower Bound (ELBO) of the log-likelihood of incomplete multi-view data (x(0)) Jz. Maximizing the ELBO effectively minimizes the KL divergence, thereby improving the approximation of the true posterior. Next, we use decoders parameterized by {0, (
reconstruct the observations. Each view (x)) is reconstructed using the latent variable z.") (rep- resent the n-th view) and a consensus variable wn. The likelihood can be written as:
p(a (C, = (x))
Cn Sn:WWi 0n)
where Cn ns, (()) is the diagonal element of Z1, with its source * determined by the permuta- tion. Then the ELBO expressed as (with detailed derivations in Appendix A.4):
CELBO [z)
[logp(((n)) (((cc,... nEI
E
Cnn n&n, wn)
(1)
KL (((()) (0) || ||
- Σ KL g(win Cas (2ll)) Jam) [[ nEZ
1=1 VEI
The ELBO in Eq. (1) consists of three terms. The first term represents the reconstruction loss for all observed views, ensuring that the variable 2(1) C, S,S, encodes information n-th view, while captures shared aspects. By applying permutations to obtain a new Pz(Z), we can derive another
CELBO- This randomness permutation results in 24 with different source views *, facilitating both self-view reconstruction and cross-view generation. We can even use convex combination of different LELBO. discussed in Appendix A.4. Different partitions produce varying w that serve the same role, ensuring that the first dimensions capture shared features across views. The remaining two terms are regularization terms for z and w, further discussed Section 3.3.
3.3 PRIOR SETTING USING CYCLIC PERMUTATIONS OF POSTERIORS
The regularization terms in the ELBO for and w, when their priors are properly set, support our learning objective of establishing inter-view correspondences in MVAEs. In the first term for 2, the outer summation spans all single-view cells S,, while the inner summation includes all 1-th view latent variables z in S or equivalently, all variables in the I-th column of matrices Zo or Z1. With eell-established inter-view correspondences, the I-th view variables should encode similar information and have distributions that are as close as possible, regardless of their sources.
In Section 3.1, we introduced the idea of permuting the columns of matrix Zo to generate different complete-view partitions. When these permutations follow a cyclic structure, the permuted poste- riors can used as informational priors. For example, if we have three distributions Pl, P2, Pa. can set the prior for Pi as Pg. for P2 as Pt, and for Pa as Pz. As these pairs converge, the distributions become more similar due to the cyclic nature of the permutation (P==P:=P:=P)) This cyclic permutation of indices ensures that the latent variables in each single-view cell become more homogeneous. Cyclic permutations can be efficiently generated before training using Sattolo's Algorithm with time complexity O(N) (Wilson, 2005), defined follows:
5

---

Published as a conference paper at ICLR 2025
Table 1: Summary of the seven multi-view datasets. These benchmarks exhibit diverse character- isties, with the last two image datasets enabling better observation through visualization.
Dataset
#View
Type
Dimension
#Sample
#Class
Handwritten
Fou, Fac, Kar, Zer, Pix, Mor
76, 216, 64, 47, 240, 6
2000
10
CUB
2
Image. Text
1024. 300
600
10
1.
GIST, PHOG, LBP
20, 59, 40
4485
15
Reuters
Text: Eng.. Fr., Ger., It., Sp. Acoustic, Scismic
10, 10, 10, 10, 10 50, 50
18758
SenslT Vehicle
98528
PolyMNIST
RGB Image
3x28x28
70000
10
MVShapeNet
RGB Image
3x64x64
25155
Definition 3 (Cyelic Permutation (Gross, 2016)). A cyclic permutation of set X is a bijection X X. For any X, (x) = and o (z) + x for all k < |x], N..
In our model, for each single-view cell S,, we define the prior for 2, as a cyclic permutation of its corresponding posterior. Specifically, we compute the KL divergence between the distributions at the same positions in Zu and Z1 (before and after permutation). This leads to a new measure of similarity within each single-view cell, which we define as Permutation Divergence, aimed minimizing the heterogeneity among distributions.
Definition 4 (Permutation Divergence). Let N 2 2 be fixed natural number. Given cyclic permutation a on the index set [N] and a set P of probability distributions on the same measure space, the Permutation Divergence of order N is mapping d from PN to the extended real line R U (o defined as follows for any PI, Pa...., PN P:
Ps... PN;O) = Σ 1= KL[P, Pooo)
The proof that Permutation Divergence valid similarity measure is provided in Appendix A.3. The key property is that d(P1. P..... PN) = 0 if and only if Pi P3 the Permutation Divergence ensures that the distributions of the latent variables in each single-view cell become as similar as possible, whether they are self-encoded or cross-transformed. This reflects how well the model captures inter-view correspondences, Translatability. This approach also enforces a form of soft consistency in the latent space, which encourages easier transformation between views rather than strict alignment.
PN. Minimizing
property we refer to as Inter-View
The second regularization term in Eq. (1) targets the consensus variable w, deterministically derived from via the geometric mean of marginal distributions in C,. Serving as a central anchor across L views, w actually acts as additional regularizer for Each Wn captures shared information across views and should be similar, with its prior set as the posterior fused from another complete- view cell C, Minimizing it ensures the similarity of all wn. property refer to Consensus Concentration. For visualizations of their impacts, please refer to Appendix A.5.
EXPERIMENTS
We extensively evaluated the proposed method across seven diverse multi-view datasets, summa- rized in Table 1. These datasets encompass a variety of view types with different dimensions, originating from diverse sensors descriptors, different angles. PolyMNIST (Sutter et al., 2021) consists of five images per data point, all sharing the same digit label but varying in handwriting style and background. ShapeNet is large-scale repository of 3D CAD models of objects (Chang et al., 2015). For each object, we rendered five images from viewpoints spaced 45 degrees apart around the front of the object. Then we selected five representative categories to create multi-view dataset called ShapeNNet. The missing pat- terns are predetermined and saved as masks before training. Specifically, for each missing rate ? {0.1, 0.3, 0.5, 0.7), randomly select r) len(dataset) samples to be incomplete, ensuring that each incomplete sample retains at least one view while missing at least one. For more details on the generation of missing patterns and masks, please refer to Appendix C.2.
well as real-world perspectives captured from
In Section 4.1, we perform clustering in the latent space and compare our method to eight state- of-the-art IMVRL approaches. The results, consistent across various missing ratios, underscore the

---

Published as a conference paper at ICLR 2025
Table Clustering results of nine methods on five multi-view datasets with missing rates 1j {0.1, 0.3, 0.5, 0.7}. The first and second best results are indicated in bold red and blue, respectively. Each experiment was run five times, with the means reported here due to space limit. The complete table, including standard deviations, is provided in Table 3 from Appendix B.
Missing Rate
0.1
0.3
0,5
0.7
Metries
ACC1
NMI
ARI
ACC1
NML1
ARI
ACC
NMI
ARI
ACC1
NMI1
ARI
DOCA
73.83
71.10
55.13
68.02
64.93
44.76
63.25
60.27
38.93
59.07
56.44
33.58
DCCAE
73.61
72.22
54.61
68.12
65.76
43.42
63.36
60.72
37.32
59.31
56.89
32.55
DIMVC
89.13
80.06
77.96
85.24
74.67
70.83
82.76
71.17
66.81
79. 66
68 94
63.
DSIMVC
81.27
79.47
71.59
73.36
81.39
79.23
77.38
Completer
82.18
77.73
68.92
78.70
69.08
58. 14
74.73
67.44
50.21
68,
62.41
41.06
CPSPAN
78.43
71,84
72.28
80.23
83,90
75.25
ICMVC
88,86
82. 19
79,51
80.95
74.53
68. 15
73.83
67.95
58. 72
67. .484
61 .9
48.66
DVTMC
87.89
79.66
78.50
85.01
78.50
74.85
MVP(Ours)
90.55
87.08
84,46
88.69
84,86
81.59
90,76
85.44
83,53
86.74
82.56
78.75
DCCA
58.93
59.59
40.87
55.65
54.23
35.65 .3.63
48, 60
46.71
27.885
42.35
39.42
21.77
DCCAE
57.27
61.04
43,09
53.43
$4.59
36,48
47.21
47.00
28.37
41.52
40.50
22.66
DIMYC
66.03
61.70
48.96
57.20
56.00
41.25
60.65
55.75
42.81
56.08
51.07
36.65
DSIMVC
72.93
67.82
$5.89
66.83
61.78
47.71
68.37
61.55
48.21
67.33
59, 89
46.31
Completer
52.97
65.47
45,98
60.773
68.88
$2.78
$1,90
61.84
45.11
19.43
17.37
0.73
CPSPAN
58.77
62.27
45,35
61.30
64.21
48.93
60.07
64.18
46,42
58.60
62.16
45.23
ICMVC
29.23
38.31
211.55
24.33
25.74
14.17
19.87
10.52
16.
7.79
DVIMVC
44.53
41.88
23,78
43.37
45.18
28.29
39.57
34,39
20.52
39.47
36.71
22.41
MVP(Ours)
78.67
77.67
66.73
74.97
73.09
61.35
74.20
71.40
59.11
66.53
50.59
DCCA
38.22
41.20
19.89
36.16
39.46
17.12
34,05
37.26
14,48
30.84
33.93
12.60
DCCAE
39.46
42.08
20.36
36.73
39.80
17.06
34.49
37.66
14.57
31.16
34,21
12.64
DIMVC
42.51
41.53
24.45
40.37
38.57
20.84
40.17
35,95
20.59
36.01
32.57
16.29
DSIMVC
29.43
30.38
14,86
31.38
32.54
16.29
27.24
28.68
13.38
28.42
29.09
13.85
Completer
37.00
41.89
23.60
40.04
42.41
24.22
36.64
38.99
19.70
35.37
37.05
17.58
CPSPAN
42,69
38.79
24.56
43.21
39.42
24.94
43.44
39.19
24.96
42.53
38.41
24.38
ICMVC
43.88
40.03
25.80
43.14
38.06
24.74
37.96
33.45
20.34
36.70
30
18.35
DVIMC
45.16
45.06
28.64
43.68
42.32
26.68
41.13
39.58
25.03
39.59
66
21.56
MVP(Ours)
45,70
43.77
27.90
45,81
42.54
27.53
45,28
41.84
26.80
43.14
39.53
24.68
DOCA
47.66
23.93
15.46
46.28
20.62
12.71
44.10
22.63
11.04
43.36
22. 90
10.03
DCCAE
42.70
23.84
7,59
43.71
26.07
8.15
42.32
24.30
6.80
41.32
23.11
5.90
DIMVC
48.83
28.94
25.78
50.54
29.86
26.89
48.51
27.29
24.74
46.94
25.79 79
23.24
DSIMVC
51,26
35.56
28.21
$1.33
34,88
26.61
50.78
36.85
28.27
47.12
33.57
25.51
Completer
41.08
21.38
7.92
40.56
22.48
10.32
41.77
20.41
9.80
42.27
22.47
11.51
CPSPAN
38.35
14.35
10.94
38.51
13.11
10.47
38.21
11.80
11.30
37.86
12.03
10.16
ICMVC
54.01
36.52
29.44
51.09
30.71
25.66
47.59
28.43
23.56
47.67
26.83
22.14 14
DVIMC
44.06
16.08
15.21
43.06
10.84
11.77
35.37
5.14
4.98
32.18
3.02
3.15
MVP(Ours)
57.83
37.25
32.20
55.70
37.02
31,35
53.67
35.43
30.24
55.16
36,00
34.66
DCCA
57.11
11.60
14.26
57.76
16.62
$3.89
11.01
12.79
50.69
8.47
9.75
DCCAE
57.93
12.84
15.28
60.32
19.42
22. 46
54.08
13.32
15.40
51.33
10.31
11.81
SensiT Vchiele
DIMVC
59.72
17.31
21.82
62.38
27.93
22.08
26.21
60.57
21.36
25.44
DSIMVC
69.82
33.40
34.88
69.24
32.95
33.50
68.05
31.49
31.56
66.54
30.08
29.73
Completer
52.63
5,33
55.59
11.29
13.96
12.52
56.37
14.77
14.66
CPSPAN
63.48
28.43
32.32
64.03
28.10
32.33
65.47
28.62
32.25
66.16
28 60
31.28
ICMVC
71.50
34,53
36.41
70.79
32.95
33.63
67.80
29.11
29.36
54.11
19.39
18.93
DVTMC
69.48
30.41
34.98
58
30.31
35.26
67.80
29.27
34.00
61.91
25 84
28.59
MVP(Ours)
72.08
34.81
41.10
71.28
33.57
39.76
70.21
32.05
38.06
68.87
30.08
36.23
structural robustness and informativeness of our learned representations. To further evaluate the effectiveness our posterior and prior settings within the VAE framework, Section 4.2 compares method with six MVAEs using two image datasets. PolyMNIST and MVShapeNet. Through multi-view clustering and generation tasks on different missing combinations, we demonstrate that method learns representations with greater sufficiency and consistency. Additionally, an ablation study on the use of permutation, permutation types, and prior settings is provided in Appendix B.3, which confirms that our cyclic approach delivers the best results.
4.1 ENHANCED PERFORMANCE FOR INCOMPLETE MULTI-VIEW CLUSTERING
To evaluate the ability of our method to handle incomplete multi-view data, we first assess clus- tering performance under various missing ratios, following Zhang et al. (2020); Lin et al. (2023); Cai et al. (2024). We apply K-means clustering to the consensus representation w and compare our method with eight IMVRL approaches: DCCA, DCCAE, DIMVC, DSIMVC, Completer, CPSPAN, ICMVC, and DVIMC (see Section 2 and Table 9 in Appendix D. for their modeling details). Clustering performance is measured by Accuracy (ACC), Normalized Mutual Information (NMI), and Adjusted Rand Index (ARI) in previous works.
7

---

Published as a conference paper at ICLR 2025
The experimental results in Tables 2 and 3 show that our method consistently achieved the best (bold red) or second-best (bold blue) performance across various missing ratios and datasets. This highlights the robustness of our method on both large and small datasets with varying class num- bers. In contrast, other methods fluctuated significantly across datasets. Notably, on smaller datasets like Handwritten, with six views provided rich self-supervised information, our method maintained strong performance even as missing rates increased. On CUB, where image-text modality dispar- ity is larger, our approach excelled moderate missing rates (17 0.1, ACC: 78.67% vs. 72.93%; = 0.3, ACC: 74.97% vs. 66.83%) by better integrating complementary information from dif- ferent views. However, as 1) rose, the limited views and small sample size led to a faster decline in performance than on other datasets. Still, our method stayed competitive with DSIMVC, which uses bi-level optimization to impute missing views, far outperforming the VAE-based DVIMC. On SensIT Vehicle, with also two views but a larger sample size, our method experienced a smaller performance drop and maintained the best results even at higher missing ratios. The Reuters, with its large size but more views, more clearly highlighted our method's 's advantage, achieving an 8.04% lead in ACC over the second-best method at = 0.7. For datasets with more classes, like Scene 15 method performed comparably to DVIMC, which uses Gaussian Mixture
and Handwritten, for explicit class modeling in the VAE. However, as increased, DVIMC struggled with incom- plete information due to missing views, while our method mitigated this by inferring complete-view information, maintaining robustness.
4.2 COMPARING WITH OTHER MVAES USING TWo IMAGE DATASETS
In this section, we compare our method with six other MVAES that utilize different posterior and prior modeling techniques: m (Wu & Goodman, 2018), mm VAE (Shi et al., 2019). MoPoE (Sutter et al., 2021), MmJSD (Sutter et al., 2020), MVTCAE (Hwang et al., 2021), and MMVAE+ (Palumbo et al., 2023). These models can naturally adapt to incomplete scenarios because their mean-based fusion can accommodate any number of views, validated by Hwang et al. (2021). We perform experiments on two tasks: multi-view clustering and generation, to demonstrate that our learned representation is able to extract more sufficient information from multiple views and infer missing views from incomplete observations while maintaining consistent semanties. We conduct our evaluation using two image datasets, PolyMNIST and ShapeNee..
4.2.1 POLYMNIST: PRESERVING CONSISTENT SEMANTICS ACROSS DIVERSE STYLES
For the PolyMNIST dataset, the shared information across its five views is the digit ID, while view- specific details include handwriting styles and backgrounds. Although the digit is present in each view, it can be obscured or unclear in some images, making it crucial to aggregate complementary information from all views for accurate recognition. We use the original split with 60K tuples for training and 10K for testing, All models are trained on incomplete observations (17 0.5), with 50% of samples having to 4 views missing. We evaluate model performance on the testing data across all possible incomplete view combinations, totaling C.. + C5 + C C5. 30 cases.
Evaluation protocol At test time, given the incomplete subset (z.(1)),, we extract the represen- tation using [Z] encoders and evaluate its quality. We perform K K-means clustering directly and use Normalized Mutual Information (NMI) as the performance metrie. Next, we generate all views (x(()) }1o using the corresponding decoders. To assess consistent semantics across views, we mea- sure coherence accuracy by feeding the generated views into a pretrained CNN-based classifier and checking if the predictions match the labels of the given subsets. Finally, we use the Structural Sim- ilarity Index Measure (SSIM) to compare the similarity between the reconstructions and the ground truth. All results are averaged across subsets of the same size.
The left plot in Figure 2 shows that our method consistently outperforms others in clus- tering across various incomplete scenarios. As the number of missing views increases, PoE- and MoE-based fusion methods experience sharp performance declines due to severe incomplete infor- mation. In contrast, only our method and MVTCAE maintain high levels of structural information. Our approach explicitly establishes inter-view correspondences to compensate for missing infor- mation, encoding different views into a latent space that facilitates easier transformations between them. MVTCAE penalizes latent information that cannot be inferred from other views to retain only highly correlated details. Both methods enforce a form of consistency in the representation, ensuring that the aggregated information is less affected by the presence of missing views.
Results
8

---

Published as a conference paper at ICLR 2025
Clustering
Nomaltoee Munual Informat 1863
100
Structue Sinitertty Index Measure
3.
00
60
24
menikE
nerjp
HoFot
NoTisE
MoPSE
20-
HY/CCE
20
NoTCAE
30-
MNTCCCE
HiTYOura))
NoTrOrs!
Number Micsing View
Number of Micsing View
Number of Missing View
Figure 2: Quantitative results the PolyMNIST dataset compared to six MVAEs. Evaluations were conducted on all incomplete subsets of the testing set, averaged across same-sized subsets.
laper
RVAL ER 205050
BRYAR (TUMP))
Melak (T2M05)
MDNVAR (2AS29)
MVP
+
4
2
3
5
3
3
3
3
S
3
3
2
5
9
6
7
7
7
7
7
7
7
7
7
7
7
9
7
8
8
8
0
8
8
8
8
8
0
7
Figure 3: Multi-view sample generation conditioned view 2. The leftmost column shows in- put images of view 2, randomly selected from digit classes 0 to 9. The following columns display multi-view samples (five views per sample) generated by various models. Ideally, the conditional generated digits should match the input digit, with yellow boxes highlighting inconsistencies. Accu- racy scores, shown in parentheses, derived from pre-trained classifiers the generated images.
The middle plot in Figure 2 shows the semantic coherence results. Our method exhibits a smaller performance drop and surpasses others as missing views increase. In contrast, other models show an obvious decline when up to four views are missing. Perversely. MMVAE and mmJSD slightly improve as more views are lost due MoE's limitations in aggregating information (Hwang al., 2021), which favors single-view identification. Figure 3 illustrates a task where only view 2 was provided, with representations extracted from incomplete observations to generate full views. Our method achieved 83.56% accuracy in maintaining semantic consistency with the input, significantly outperforming other models. In contrast, other methods showed notable inconsistencies across the five-view tuples, highlighted by the yellow boxes. m mVA struggled maintain consistent semantics across views due to the precision miscalibration of each view caused by PoE fusion (Shi et al., 2019). MoE-based models like mm VAE, mmJSD, and MoPoE maintained some semantic coherence but failed with more challenging samples, producing blurry backgrounds. MVTCAE generated clearer, more varied backgrounds but still showed semantic inconsistencies in several samples. Although designed to retain highly correlated information between views, missing views made it harder to maintain consistency, especially with only one view available. MMVAE+ produced clearer back- grounds than mm VAE by separating shared and view-specific subspaces, but sampling missing-view information from auxiliary priors caused severe category confusion.
The right plot in Figure shows the SSIM results. Also, mm VAE and mmJSD exhibit improved reconstruction quality as the number of missing views increases-a counterintuitive trend, yet con- sistent with theirs in semantic coherence. MMVAE+ performs moderately, likely because we used the same network structure for all models rather than its original ResNet architecture, suggesting may overly rely powerful decoders for good generation. Since SSIM primarily reflects the quality of dominant backgrounds in the PolyMNIST dataset, our method is the only one that consistently balances high semantic coherence with diverse background styles.

---

Published as a conference paper at ICLR 2025
Figure 4: Multi-vies samples generated by our method on the MVShape dataset. Categories include table, chair, car, airplane, and rifle, with each sample consisting five views from different angles. The model was trained with missing rate 0.5 and tested with only view 5 available.
4.2.2 MVSHAPENET: CAPTURING DETAILED INFORMATION FROM VARIOUS A ANGLES
We further evaluate our method on the MVShapeNet dataset. Unlike PolyMNIST, where views share few common pixels depicting the same digit against various background styles, Shappeeee presents smaller inter-view gap and greater consistency due to its uniform white backgrounds with the same object captured from different the same experimental settings as in Section 4.2.1.
angles. We use an 80:20 train-test split and apply
In this setting, MVAEs are less prone to semantic inconsistencies observed with PolyMNIST, mean- ing the generated object generally matches the input. However, expect the learned representa- tions to preserve more detail such as furniture hollowing, textures, and lightnings, which remains a challenge under high missing ratios. To evaluate this, we compare our method with other MVAEs at missing rates of r = 0.1 and 0.5, testing across all possible incomplete combinations. For quan- titative evaluation,
average SSIM to evaluate the basic structure of generated images. Addi- tionally, we pretrain two CNN-based classifiers on all views to evaluate whether the decoded images accurately capture object categories and perspective angles. As demonstrated in Table 4 of Appendix B. our method consistently performs well, regardless of whether it is trained at high or low missing rates and tested any incomplete combination. In contrast, other methods either show dramatic performance drop as the number of missing views increases or rely on memorizing incomplete sam- ples without effectively aggregating complementary information from additional views. As shown in Figure 4, when only view 5 is provided, MVP leverages inter-view correspondences to transform latent representations and successfully infer the missing views. Further visualizations in Figure 10 illustrates that other models tend to produce blurry reconstructions with lack of detail or unclear perspective angles. In contrast, our method clearly infers more accurate details in missing views, such as the placement of table legs at different angles, changes in light and shadow, and hollowed- out armrests on chairs. All these results suggest that our method learns representation with more sufficient and consistent information from incomplete multi-view data.
5 CONCLUSION
In this paper, we presented Multi- Permutation of VAEs (MVP), a novel framework to address the challenges of incomplete multi-view learning. By explicitly modeling inter-view correspon- dences in the latent space, MVP effectively captured invariant relationships between views. We derived valid ELBO for efficient optimization by applying permutation and partition operations to the latent variable set. Notably, these operations on multi-view representations are not limited to the VAE framework and can be extended into non-generative models. Additionally, the introduction of an informational prior using cyclic permutations of posteriors resulted in regularization terms with both practical meanings and theoretical guarantees. Extensive experiments on seven diverse datasets demonstrated the robustness and superiority of MVP over existing methods, particularly in scenarios with high missing ratios. These findings underscore its potential to reveal more informative latent spaces and fully unlock the capability of MVAEs handle incomplete data.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
Special thanks to Yuxin Li, Hangqi Zhou, Hanru Bai, An Sui, Fuping Wu, Yuanye Liu, Yibo Gao, and Ruofeng Mei for their invaluable feedback on the manuscript.
REFERENCES
Ana Lawry Aguila and Andre Altmann. A tutorial on multi-view autoencoders using the multi- view-ae library. arXiv preprint PXXiv:2403.07466 2024.
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International conference on machine learning, pp. 1247-1255. PMLR, 2013.
Miklós Bóna. Combinatorics of permutations. ACM SIGACT News, 39(4):21 2008.
Richard A Brualdi. Introductory combinatorics. Pearson Education India, 2004.
Hongmin Cai, Weitian Huang, Sirui Yang. Siqi Ding, Yue Zhang, Bin Hu, Fa Zhang, and Yiu-Ming Cheung. Realize generative yet complete latent representation for incomplete multi-view learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: arXiv:1512.03012. 2015.
Guoqing Chao, Yi Jiang, and Dianhui Chu. Incomplete contrastive multi-view clustering with high- confidence guiding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 1221-11229, 2024.
William G Cochran. The combination of estimates from different experiments. Biometrics, 10(1): 101-129, 1954.
Jonathan Gross. Combinatorial methods with computer applications. CRC Press, 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basie visual concepts with constrained variational framework. ICLR (Poster), 3, 2017.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1711-100 2002.
Harold Hotelling. Relations between two sets of variates. In Breakthroughs in statisties:
methodology and distribution, pp. 162-190. Springer, 1992.
Zhenyu Huang, Peng Hu, Joey Tianyi Zhou, Jiancheng Lv, and Xi Peng. Partially view-aligned clustering. Advances in Neural Information Processing Systems, 33:2892-2902,, 2020.
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. International Conference Machine Learning, 2024.
HyeongJoo Hwang, Geon-Hyeone Kim, Seunghoon Hong, and Kee-Eung Kim. Multi-view rep- resentation learning via total correlation objective. Advances in Neural Information Processing Systems, 34:12 34:12194-12207 2021.
Nicholas Jardine and Robin Sibson. Mathematical taxonomy. Wiley, 1971.
Xinke Jiang, Zidi Qin, Jiarong Xu, and Xiang Ao. Incomplete graph learning via attribute-structur decoupled variational auto-encoder. In WSDM oral, pp. 304-312. 2023.
Jiaqi Jin, Siwei Wang. Zhibin Dong, Xinwang Liu, and En Zhu. Deep incomplete multi-view clus- tering with cross-view partial sample and prototype alignment. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 11600-11609 2023.
Solomon Kullback. Information theory and statistics, 1959.

---

Published as a conference paper at ICLR 2025
Yingming Li, Ming Yang, and Zhongfei Zhang. A survey of multi-view representation learning. IEEE transactions on knowledge and data engineering, 31(10):186 11600):883-1888, 2018.
Yijie Lin, Yuanbiao Gou, Zitao Liu, Boyun Li, Jiancheng Lv, and Xi Peng. Completer: Incomplete multi-view clustering via contrastive prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11174-1118,, 2021.
Yijie Lin, Yuanbiao Gou, Xiaotian Liu, Jinfeng Bai, Jiancheng Lv, and Xi Peng. Dual contrastive prediction for incomplete multi-view representation learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4447-4461, 2023.
Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision, pp. 366-384. Springer, 2025.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 201 pp. 4. Granada, 2011.
Emanuele Palumbo, Imant Daunhawer, and Julia E Vogt. Mmvae+: Enhancing the generative quality of multimodal vaes without compromises. In The Eleventh International Conference Learning Representations. OpenReview, 2023.
Emanuele Palumbo, Laura Manduchi, Sonia Laguna, Daphné Chopard, and Julia E Vogt. Deep generative clustering with multimodal diffusion variational autoencoders. In The Twelfth International Conference on Learning Representations, 2024.
Andrea Sgarro, Informational divergence and the dissimilarity of probability distributions. Calcolo, 18(3):293-302 1981.
Yuge Shi, Brooks Paige, Philip Torr, et al. Variational mixture-off of of-eexeets autoencoders for multi- modal deep generative models. Advances neural information processing systems, 32, 2019.
Robin Sibson. Information radius. Zeitschrift für Wahnscheinlichkeisshheoeee und verwandte Gebiete, 4492);19--160, 1969.
Thomas Sutter, Imant Daunhawer, and Julia Vogt. Multimodal generative learning utilizing jensen- shamnon-divergence. Advances in neural information processing systems, 33:6100-6111 6110, 2020.
Thomas M Sutter, Imant Daunhawer, and Julia E Vogt. Generalized multimodal elbo. International Conference on Learning Representations, 2021.
Thomas M Sutter, Yang Meng, Norbert Fortin, Julia Vogt, and Stephan Mandt, Unity by diversity: Improved representation learning in multimodal vaes. arXiv preprint 2XXi:2443300500 2024.
Huayi Tang and Yong Liu. Deep safe incomplete multi-view clustering: Theorem and algorithm. In International Conference on Machine Learning, pp. 21090-211100 PMLR, 2022.
Jingjing Tang, Qingqing Yi, Saiji Fu, and Yingjie Tian. Incomplete multi-view learning: Review, analysis, and prospects. Applied Soft Computing, pp. 111278, 2024.
Jakub Tomezak and Max Welling. Vae with a vampprior. International conference on artificial intelligence and statistics, pp. 1214-1223. PMLR, 2018.
Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. On deep multi-view representation learning. In International conference machine learning, pp. 1083-1092. PMLR, 2015.
Mark Wilson. Overview of sattolo's algorithm. In Algorithms Seminar, 2002-2004, pp. 105. Citeseer, 2005.
Mike Wu and Noah Goodman. Multimodal generative models for scalable ing. Advances in neural information processing systems, 31, 2018.
learn-
12

---

Published as a conference paper at ICLR 2025
Gehui Xu, Jie Wen, Chengliang Liu, Bing Hu, Yicheng Liu, Lunke Fei, and Wei Wang. Deep vari- ational incomplete multi-view clustering: Exploring shared clustering structures. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 16147-16155, 2024.
Jie Xu, Chao Li, Yazhou Ren, Liang Peng, Yujie Mo, Xiaoshuang Shi, and Xiaofeng Zhu. Deep in- complete multi-view clustering via mining cluster complementarity. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 8761-8766 2022.
Changqing Zhang. Yajie Cui, Zongbo Han, Joey Tianyi Zhou, Huazhu Fu, and Qinghua Hu. Deep partial multi-view learning, IEEE transactions on pattern analysis and machine intelligence. 44 (5):2402-2415, 2020.
13

---

Published as a conference paper at ICLR 2025
Appendices
Theoretical Analysis
15
A.1 Partition and Permutation of Set
15
A.2 Cyclic Permutation and Sattolo's Algorithm
16
A.3 The Similarity Measure over Distributions
17
A.4 Derivation and Analysis of the Evidence Lower Bound (ELBO)
19
A.5
Latent Space Dynamics During Training
23
Additional Experimental Results
24
B.1
Quantitative and Qualitative Results MVShapeNet
24
B.2
Additional Experiment Results on CUB Dataset
27
B.3
Ablation Study of Permutation and Priors Setting
28
B.4 Coefficients 3 of Regularization Terms
29
B.5 The Choice of Latent Representation Dimensions
29
Implementation Details of Our Method
C.I Network Architectures
31
C.2 Missing Patterns and Mask Generation
32
C.3 Pre-computation of Cyclic Permutation Indices and Batch Processing
32
C.4 Overall Training Pipeline of Our Method
32
C.5 Inference of Missing Views with Available Views
33
D Comparative Methods
33
D.I Incomplete Multi-View Learning Methods in Section 4.
................................
33
D.2 Multimodal VAEs in Section 4.2
34
Dataset Information and Construction Details
34
14

---

Published as a conference paper at ICLR 2025
A THEORETICAL ANALYSIS
In this section, we present a comprchensive thcoretical analysis of the proposed method, offering additional details to complement the main text.
A.1 PARTITION AND PERMUTATION OF SET
In mathematics, a partition of a set refers to a division of its elements into non-empty, mutually exclusive subsets, such that each element of the original set belongs exactly one of these subsets. In simpler terms, a partition is a "set of sets", where each subset is known as a cell.
Definition 5 (Partition of A Set (Brualdi, 2004)). A family of sets P is a partition of the set X if and only if all of the following conditions hold:
(1) P does not contain the empty (i.e., & P).
(2) The union of the sets in P is equal to X (i.e., UAED X). The sets in P said to exhaust or cover X.
(3) The intersection of any two distinct sets in P is empty (i.e., VA, E An B 0). The elements P are said to be pairwise disjoint or mutually exclusive.
In this work, we introduce two specialized types of partitions applied to the latent variable set Z: the single-view partition P. (2) and the complete-view partition (Z) These partitions are tailored to the particular structure and requirements of the problem under study.
The visualization of these two partitions is facilitated by arranging the variables in matrix and dividing them according to rows and columns. In Figure 5, the left matrix Zo contains diagonal elements directly derived from observed data, while aff-diagonal elements represent transformations
of the diagonal elements. Each column consists of variables =0)) corresponding to the I-th view, derived from different sources. Hence, the single-view partition of Z corresponds the set of columns in the matrix. In contrast, the complete-viey partition can be represented by dividing the matrix by rows, where each row encompasses all views. However, this partition is not unique; by reordering the elements within each column and then partitioning the matrix by rows, we obtain new complete-view partition P [)), as illustrated by the right matrix Z₁ in Figure 5.
Zo
Z
(1)
(1)
(2)
21
(3)
2
(4)
(S)
25
11)
22
(3)
Z5
44)
23
(5)
C1
(2)
22
2/
22
27
(4)
(1)
4
12)
33
5
C2
x
3
2/
2/1
23
(8)
2
2
22
(8)
C3
x
24
(1)
12)
24
(1)
24
(2)
(3)
2
(4)
(5)
C4
x
(1)
(2)
25
25
23
(1)
73
(3)
22
(3)
22
(9)
(3)
C5
51
52
83
84
85
82
83
84
8g
Figure 5: An illustration of column- permutations to generate different complete- parti- tions. In the first column (red box), the permutation ai = (1532) is applied, and its inverse
(01)-1 (2351)(4) reverses the cycle order. This results in 0х ([21))
(1) 20) (1) (1)
2g 21 23 (1)
(1) The same procedure is applied to the other columns. Partitioning each row (purple box) yields the complete-view partition.
[2g :1 Ag 223
(1) (1)
Next, define permutations, which are central to generating new partitions.
Definition (Permutation of A Set (Bóna, 2008)). A permutation of a set X is a fiiectiveffunticcn X X. In other words, it is one-lo-one mapping of the set X onto itself.
common method for representing permutations cycle notation, where a permutation is ex- pressed product of disjoint cycles. Each cycle indicates how the permutation rearranges
15

---

Published as a conference paper at ICLR 2025
subset of elements, moving each element to the position of the next one in the cycle. For example,
consider a permutation ø of the set X = 1,2, 3, 4, defined by o(1) = 2, o(2) = 3, o = 1. and (4) The cycle (123) indicates
(4) 4. In cycle notation, this permutation written that mapped to 2, to 3, and 3 back to 1. The element remains fixed, represented as (4), often referred as a fixed point.
A.2 CYCLIC PERMUTATION AND SATTOLO'S AL.GORITHM
A cyclic permutation is specific type of permutation that consists of exactly eycle in its cycle notation, with the cycle length equal to the size of the set (Gross, 2016). For example, a cyclic permutation a of the set X {1, 3, 4] can be written as (kyk2kg4)) where definition is given in Definition 3. In cyclic permutation, each element of a set with more than one element is cyclically shifted, meaning each element mapped to another, and after a number of mappings equal to the set size, every element returns to its original position. Cyclic permutations are particularly useful in our setting, as they guarantee convergence of the regularization term (see Appendix A.3) and can be efficiently generated using Sattolo's Algorithm (Wilson, 2005), which operates with linear time complexity.
X. formal
Algorithm 1 Sattolo's 's Algorithm for Cyclic Permutation
Input: Array A of size n Output: A cyclic permutation of A
Initialize array length | do 2: Randomly select from 0 to i - 1;
1: for
3: Swap A(i] and A[j]:
4: end for
5: return A;
The algorithm starts with the identity permutation (0) = Id. For each i {1,...,, by (() the permutation obtained after the first steps. Step consists in choosing random integer k; in (1...., n - i} and swapping the values of (i-1) at places k, and --it+1. 1. In this way, obtain permutation (()) which is equal n-++ (i-1), where Tkg is the transposition exchanging k; and n - i + Finally, the algorithm returns the permutation
n-11; 1}.
denote
(n-1)) This process is captured in Algorithm 1. Figure provides an example where n = 5. and the sequence of random swaps is 3, 1, 2, 1. The resulting cyelie permutation is - 5 3 - 4 1.
1
2
3
4
1
2
5
4
4
2
5
4
5
2
1
5
4
2
1
3
Figure 6: Execution of Sattolo's algorithm for n = 5. where the sequence of random swaps is 3, 3, (Wilson, 2005).
Proposition The mapping produced by Sattolo's 's algorithm a cyclic permutaaion, and every cyclic permutation can be obtained using Sattolo' 's algorithm.
Proof. The correctness of Sattolo's 's algorithm follows from the fact that it generates a unique decom- position of a cyclic permutation a as a product of transpositions, Tk .0...00. OTk,m-ii41 . Tki i) for 1 <
where k; E (I,...,.e
1.
For n = 2, the permutation o = Th. is clearly a cyclic permutation. Assuming Sattolo's algorithm works for sets with fewer than elements, we now demonstrate that it also holds for set of size n.
16

---

We prove this using a convenient feature of the cycle notation. Consider a cyclic permutation The inverse permutation o- is obtained by reversing the order of the elements in the cycle, yielding (kL, "z-1.".. "....,))))..... Since this Published as a conference paper at ICLR 2025
Consider set of n elements. The permutation Tk., 1.2 0.... - consists of n transpositions. The first n transpositions act on the set {l..... n n- 1--1. excluding A1 where 1 is swapped with By the inductive hypothesis, these form eyclic permutation n elements, which can be represented as single cycle: (n, Txx2))
k 7ki,
.... Th2.n-1
Applying the final transposition Th swaps 1 with n, yielding:
(n, 71," 22-2) Thi, (n, k1, PI,
which is a eyclic permutation of elements. By induction, Sattolo's algorithm always produces a cyclic permutation for any set size
Since Sattolo's 's algorithm generates (n - 1)! distinct cyclic permutations for a set of size n, and this precisely the number of all possible cyclic permutations on elements, every cyclic permutation can be obtained using this method. This completes the proof.
Next, we explain how permutations generate different complete-view partitions. Let v represent rows and / columns, with the matrix form of the set 2 expressed as Zo = [2+ LxL' We assume the missing views preserve the matrix's 's square form, simplifying notation. Let the observed views be I = ( }n S [L]. with missing views given by [L]
(0)
fri...., ,'m complete-view partition is obtained by dividing Zo by rows: Pa Id) {C JEZ. where ch {=n
(() } with Id representing the identity mapping. We call Pa (2 Id) the basic complete-view partition.
The matrix Zo can also be expressed by columns as [20"] LXL = -xx0)....,22)) Consider
permutations a {at]t=1 defined on the index set [L], each being cyclic permutation on kn (ki.... with fixed points {ri,... .Tm). J. A new matrix Zi can then be constructed by per-
muting the row indices within each column [2(0) [=ott)] = [ ..... _14]], where =()) contains
the same elements 2()), which implies that single-view partition is unique. And this results in new oz) = (Ca)nez, where C, = {==)(-))
As illustrated in Figure 5, if we disregard the missing views treated as fixed points, applying a cyclic permutation transforms matrix Zo into Z1. Notably, applying the inverse the permutation, which is also cyclic, restores Z₁ back to Zo. This highlights a symmetric relationship between the two matrices, governed by cyclic permutations and their inverses.
Proposition 2. The inverse of cyclic permutation is also cyclic permutation.
Proof.
defined on the set [L], with cycle notation (k1, ..,... kL).
reversed sequence still forms a single cycle, -1 is indeed a cyclic permutation.
A.3 THE SIMILARITY MEASURE OVER DISTRIBUTIONS
In this section, introduce similarity measure between distributions, referred to as the Dissimi- larity Coefficient (d.c.), which we use to reduce the value of the KL divergence term in the ELBO. Consequently, this reduction minimizes the dissimilarity between distributions, effectively maxi- mizing their similarity. We also explain how the informational prior in our method transforms the regularization term into new d.c..
In various statistical fields-such as hypothesis testing, cluster analysis, and pattern recognition-1 essential to distinguish between probability distributions using appropriate dissimilarity coeffi- cients (or separation measures, denoted as d.c.). A d.c. for a set of N probability distributions quantifies their "degree of heterogeneity". For N 2, a d.c. can be interpreted as a "distance" between two distributions, though it may not always represent a metric distance in the strict sense.
Definition 7 (Dissimilarity Coefficient (Sgarro, 1981)). Let P denote the set of probability mea- sures measurable space F). F). Let 2 2 be a fixed natural number. A dissimilarity coefficient (d.c.) of order N is a mapping d PN + IR U RRU{+ {+00} that satisfies the following properties for any Pı, Pa. P2...., Py P:
17

---

Published as a conference paper at ICLR 2025
(1) Non-negativity: d dP,,, (P...P......N 20; 0;
(2) Identity of Indiscernible: d P2 ...... Pw) 0f = PN:
(3) Symmetry: d ((P,,P2..... Px)=d (P (1011,PP(2)) Pa(2).... Po(N)) for any permutation & of [N].
In some cases, condition (2) can be strengthened to:
(2') d P2, 2..... Pv) 0 if and only if Pi P2
PN.
The Kullback-Leible (KL) divergence (Kullback, 1959), while not symmetric. is widely regarded as a fundamental statistical measure for distinguishing between two probability distributions. To extend its application to multiple distributions (N 2), several symmetric dissimilarity coefficients based on the KL divergence have been proposed. For example, the average divergence sums the KL divergence over all N(N 1) pairs of distributions (Kullback, 1959), while the information radius resembles the Jensen-Shannon divergence for multiple distributions (Sibson, 1969; Jardine & Sibson, 1971). Additionally, Sgarro (1981) introduced the minimum divergence, which measures the KL divergence for the pair with the smallest difference.
In this work, we reuse the permuted posteriors generated during the construction of complete-view partitions and set them priors within the multimodal VAE framework. This transforms the KL di- vergence term the ELBO into a new d.c., which call the Permutation Divergence (Definition 4). We prove that this is valid d.c.:
Theorem 1. The Permutation Divergence defined in Definition is a dissimilarity coefficient.
Proof. Consider cyclic permutation a defined on [N] with cycle notation (1, PN-1) The corresponding permutation divergence is given by:
N
d(Pi. P2..... PN:o) Σ KL[P, ||
Pa()).
Since the divergence is sum of KL divergences, property (1) of non-negativity is satisfied.
To verify the identity of indiscernible, note that d(P1, .e....,,PPrr. in the sum is zero. This implies KL[P. Patel = 0 for all i, which occurs if and only if P. = Pa()- Consequently, P1 Pr.
Pv: o)
0 if and only if each term
P.N-.. satisfying property (2').
Finally, for any permutation & of [N]:
d (Pac)), Pag).... Pa(Nji)= =
Σ KL( Po(ot))]
i=1
= Σ
KL[P, Pown d(Pi, P2....... PN;o).
J=1
Thus, the Permutation Divergence satisfies the symmetry property (3).
The proof of property (2') demonstrates why cyclic permutations are used instead of general permu- tations: the one-cycle structure ensures that the divergence reaches its minimum when all distribu- tions are identical.
Proposition 3. The of dissimilarity coefficients defined on the of distributions is itself a dissimilarity coefficient.
Proof. The proofs of these properties for the sum of d.c.'s follow directly from the corresponding properties of the individual coefficients. For brevity, these straightforward proofs are omitted here.
As a result, the sum of two Permutation Divergences, each defined with different cyclic permutations on the same set distributions, remains a valid d.c.. Importantly, this applies to the case where the cyclic permutation and its inverse used together.
18

---

Published as a conference paper at ICLR 2025
Proposition 4. The sum of the Permutation Divergences defined with cyclic permutation a and its inverse is also dissimilarity coefficient and is composed of symmetric KL divergences.
Proof. Consider cyclic permutation defined on [N]. The corresponding permutation divergence is given by:
N
d(Pi. Pa PN;o) Σ
KLIP, Pa(o)
||
According to Proposition 3, 01 also a cyclic permutation, thus
NV
d(Pi. P2, PN;o-= Σ KL[P, || P. 26)
For any j [N], there exists a unique & E [N] such that o(k) =. Therefore, we have:
N
KL[P: Potol + Σ KL[P, Po-1(s)
N KL|Pa ((k) || Po-1co(m))
KLIP: Po(o) +
k=1
(KL[P, || Patel (L(Po() P:))
A d(P, PN. PN;
where each term symmetric with respect the pair P. and Pa(t)-
As illustrated in Figure 5, cyclic permutation transforms complete-vies view partition into one, while its inverse restores the original partition. This allows for the interchangcability of the matrices, Zo and Z1, where Consequently, examining the sum of the Permutation Divergences defined by a cyclic permutation and its inverse highlights this symmetry, which will be further explored in the next section.
be used for posterior factorization and the other for priors.
A.4 DERIVATION AND ANALYSIS OF THE EVIDENCE LOWER BOUND (ELBO)
With the complete-view partition factorizing the joint approximate posterior and the permuted uni- modal posteriors serving as informational priors, we are now ready derive the ELBO for in- complete multi-view data (x((1) I. To facilitate this derivation, which involves types of latent variables, we first present a useful lemma that establishes the chain rule for KL divergence. Lemma (Chain Rule of KL divergence).
KL(g(ı, 1)l y)) KL(g(=)ll(<))) + ([(((o)=)))(()=)))
Proof. The proof follows from the definition of the KL divergence and the factorization of joint distributions:
g(2, v) p(z.v) dy dx (x))((/>)) p(z)p(u(x)
KL(g(æ, 1)(l,( v)) =
q(x, 8) log
= / ((0, log
dy dr
/ / g( v) log p(x)
g(x) dy do / c(x,)) log ((v1z) dy dx
p(y(æ)
u(x) log ((x)
p(x) dx + f alo)) (lul)))og log ((ylx)
x))x)) dy dr
=LL((())l)(())) KL(g(vle))lo(()r))
19

---

Published as a conference paper at ICLR 2025
Theorem 2. Equation (1) is the evidence lower bound (ELBO) for incomplete multi-view data {2(")/
Proof. We begin with the log- log-likelihood of the incomplete multi-view data fx(")) assuming two sets of latent variables, Z and ST. For any joint distribution 22, S2), the following equation holds:
log pfx (2( / (2,52) log p( (x }2)
/ (2, log (((x(0)) }z 2, 2,50)((255) S2) (3,9)
12/2
( }z)
(3,0)
/ (( log ((2l0))z. 2,0)
dzdΩ KL (1 3, 12) ((2,5) {z (z))))2)
(2,2)
True posterior
Evidence Lower Bound (ELBO)
We use encoders to model the distribution (3, given the observed data, denoted as q S2 (ax()) The first term in this equation represents the ELBO, which serves lower bound on the log-likelihood of the data. By maximizing the ELBO, the KL. term becomes smaller, meaning that the learned distribution approximates the true posterior p 2 S2
(x(() Jz). For a given complete-
view partition o)) (CnInEZ, where C, ( H we can factorize the joint posterior
as:
(2,0 (x) (z) &
II g(w,, C.. (20 )g() (o))))a))
nEZ
II q(w,, Cn. (c(()) II {(x!") (.)).
nET
Next, we assume the generative process as:
p((x(1)) 12, 2,52
-
II p(xln) NET
Cns W (w.pp(C...n))
II p
wn) p(wa) II P (:4))
Here, we again explain why we use C, ns,, the diagonal of the matrix of Z, for reconstruction. From the derivation, we partition according to (Cu)neI, where each Cn contains I. latent variables
representing different views. Among them, only *o:(n) C, n ns, is related to x(()) (with the
_(x)
same superscript). This approach also simplifies practical implementation, we can directly use the diagonal of the matrix of to extract all the C, SSS..
/ (2 Q (z) log p((z(")) Jz, 2,52) dzdn
(2,52 S {x(1))z)
p(x(1) Cn,ww))(un) It p(=()) dC,, duwn
Σ
((Cr (zn))))))) log
g(wm C.. (a(1)) Ion) II ((26)) (z())
nET
(og C,, (nn
NEZ
/ q(wn Cn, (2(1)) Joa log
p(wn)
Cu, {a(1))onn) dwn
glwin
IL p(z(.) dC,,
/ II" II ((2(0) (1)) log
II ((21)) x(o))
Σ nEZ
(al(o)))) p(z!")) Cnn n (p:
1.
KL
(g=(() ((01) || p(((00]
Σ KL (g(un, | Cns (zn))) ll plun))
[=1 VEI
nEI
20

---

Published as a conference paper at ICLR 2025
The split of the final two KL terms follows directly from the chain rule provided in the lemma. At this point, we have derived Eq. (1), but we can further simplify by removing redundant variables (which essential for modeling and derivation but not necessary for exposition) and explicitly defining the prior settings, as outlined in Section 3.3. Specifically, we denoteq( 2!!) Ie()),, which encodes
the (-th view's information using the e-th view as the source, (t) (z). his notation represents the encoding and transformation of the distribution, where qo (z) ~ N2; Sin o uux(()), Siv o E(x(+))), where five id when I = Similarly, we denote the prior p(=t") as pe (z). We set this prior to Pe (>) % (w) (z), where of 1 is the inverse of the permutation used to obtain the complete-vien partition and acts as cyclic permutation on the incomplete index set I. This transforms the first KL term into:
L
ΣΣ KT. 2o (x) 11 (0)
(z)
Σ 1=1
d(G
Ok. or
1= VET
where {ki...., }n represents the observed views, and d is the permutation divergence, ensuring that distributions encoding the same view from different sources remain as close as possible. For practical implementation, we only need to compute the KL divergence between the corresponding positions of the distributions in the left and right matrices shown in Figure 5.
The latent variable w derived from the fusion of the marginal Gaussian distributions in C.,. In other words, can be deterministically determined by the variable making the regularization of effectively a regularization of z. We simplify the notation for the posterior distribution g(w Cn), which represents the distribution obtained by fusing the marginal k-dimensional distributions. Specifically, g(w C.) ~ N(w; an An). where
A, = E L
an AA, L [1: k1 ::) (1:(1:))) k3). E Jn. [
Here, we rely on two well-known results: first, the marginal distribution of multivariate Gaus- sian remains Gaussian, and second, the geometric mean of several Gaussian random variables also follows Gaussian distribution, with parameters that are straightforward to compute.
For the prior setting of w, since its posterior is derived from the combination C,, (1) (n) H
we can similarly fuse the priors of 2, which have already been defined. is easy see that repressssnnii the cell of the basic complete-view partition, t7, "(.. = {z1l H {
which corresponds to the reeppemmutation position in the matrix. Therefore, set the prior of g(w C, to g(w C.)). In practice, this simply requires calculating the KI. divergence between the w variables obtained from the two matrices shown in Figure
Finally, for given permutations and their resulting complete-view partition, we can express the ELBO as follows:
CELBO (fz:)
Σ nEI
[oo (ogp(a!()) (w
(0(0) [=o())))))) [0]
(2)
I
KL
11 (z) |1
(z)
KL (g(w C.) g(w (c Σ NEI
1=1 DEI
Since we can factorize the joint posterior using any complete-view partition, we can alternatively the basic partition D (Z) (C") where Cº 1 In this case, there are cyclic permuted posteriors to set the prior. Thus, assuming arbitrary cyclic permutations {ai)/ on
the incomplete index set I, we can derive the basic ELBO as follows:
CR (1 (1x = Σ
[logp(z<(m) 22("),,)
nEI
(3)
KL Le (z) (x)
Σ nEt
KL (g(w
co) g(w C.))
1=1 VEI
21

---

formations. In contrast, in E2.(2), (n) comes from a cyclically permuted complete-view partition, where (n) is not equal to but instead corresponds to another observed view obtained via inter- view correspondences. This can be interpreted as a cross-view generation. If Eq.(2) iterates over Published as a conference paper at ICLR 2025
Fecenmattin Levs
X0V
Wing Eo- G1
Figure/ 7: Loss evolution curves during the training process on the Handwritten under missing rate 7) 0.1, illustrating the trends of different loss components. The first subplot (Loss all) highlights the warm-up phase using Eq. (3) during the initial 100 epochs (shaded yellow). Afterward, training transitions to Eq. (4), which is defined as 0.5 (Eq. (2) + Eq. (3)). Subplots for Reconstruction Loss, KLD,z, z. and KLD_w show the progressive change in their respective values over 500 epochs.
There are subtle differences between the basic ELBO in (3) and Eq. The first term, rep- resenting the reconstruction loss, shows that in the basic ELBO, the variable ((n) generates x(())
(n) is directly encoded from x:(n) without any trans- through self-view reconstruction, meaning Zn
(n))
all possible permutations, it implies that within the loss function, all observed views generate other views via cross goossgeenerttio..
The form of the remaining two terms suggests that types of ELBOs, assuming they use the same set of a and their inverses. This leads to the following expression:
form convex combination of these two
(F (x) Fzi - (( (z) + (FL.oO(( }r)
Σ5
E won (lgg((.(a)) 1 (mm) + ( (log((((m) (0)m)
=o((m))
2
REI
KL (x) 9 (o)
KL g (2) || (o((m) (=)
(KL (g(w C.) || g(w ch)) + KL (g(w chi g(w (n)))
IEI
(4)
In practical optimization, we use E3.(3) for warm up and Eq. as the final learning objective, selecting different permutation at each iteration.Thi approach offers three main advantages. First, combining the two ELBOs simultaneously promotes both self-view reconstruction and cross- view generation, which leads to more comprehensive learning process. Second, by using cyclic permutations or and exploiting the fact that their inverses are also cyclic, we can efficiently obtain two complete-view partitions. This allows the distributions at corresponding positions, both pre- and post-permutation, to supervise each other without the need for additional computations, making the optimization process more computationally efficient. Finally, the convex combination of the two regularization terms introduces a higher degree of symmetry. For the term involving 2, based on Proposition 4, we can rewrite it as a more symmetric dissimilarity coefficient:
L
ΣΣ KL (2) ll (0)) EELEEI
(z) + KL (o00
1 (=)
Σ
d(al))
Ok. oz o,
Similarly, the term involving w is also expressed as sum of symmetric KL divergences. In prior work, such as Sutter et al. (2020) and Hwang et al. (2021), the unimodal posteriors are fused to obtain joint posterior. Subsequent generations rely on sampling from this joint posterior, which becomes the primary optimization target. As a result, Hwang et al. (2021) suggests that the asymmetry of the KI. divergence makes the forward KL more suitable than the reverse KI. for this context.
22

---

Published as a conference paper at ICLR 2025
Figure 8: Visualization of the effects of the two regularization terms five-view sample with one missing view. Each circle represents a set of homogeneous latent variables. In S, markers of the same shape indicate variables from the 1-th view, while colors represent their source views. In Ω consensus variables are fused from all five views, as shown by the dashed purple arrows. Bidirectional black arrows illustrate the cyclic convergence of variables within each set.
In contrast, our approach maintains the unimodal posteriors throughout the factorization process, with subsequent reconstructions dependent on these individual subspaces. Thus, optimizing all uni- modal posteriors becomes essential. The symmetric KL divergence ensures that both distributions encouraged to move toward each other's high- sigh-probability regions, fostering more stable conver- gence during training.
A.5 LATENT SPACE DYNAMICS DURING TRAINING
The two regularization terms, Inter- View Translatability and Consensus Concentration, play dis- tinct roles in the training process. These terms impact the arrangement and interaction of latent variables in the learned space, as illustrated in the following visualizations.
In Figure 8, we depict the impact of the regularization terms a five-view sample with one miss- ing view. In the single-view cell S₁, markers of the same shape represent the latent variables corresponding to the 1-th view, while the different colors indicate their source views, whether self- encoded or cross-transformed. Note that each set contains as many variables as there are observed views (four in this case), they can only be encoded or transformed from available views. As this term diminishes, the variables within each S cyclically converge, indicating that variables from different views can effectively transform into each other, thereby establishing inter-view correspon- dences. This process also enforce a form of soft consistency, as representations from different views encouraged approach each other after being transformed, rather than aligning directly. The learning of inter-view correspondences avoids collapsing into identity mappings because the recon- struction loss ensures that variables retain unique information specific each view.
The Consensus Concentration term aims to ensure that consensus variables derived from different combinations remain consistent, as shown in Figure 8. Each w S2 obtained from a complete combination of all views. Over time, the regularization promotes closer alignment of these consen- sus variables, facilitating the aggregation of shared information across the views.
Figure illustrates the evolution of the latent space during training. The top row depicts the latent variables for five views across different training epochs, with each color representing a different view. Each colored cluster contains all variables in S,, whether self-encoded or cross- trasffrransf...d... Initially, these variables are scattered, but over time, they coalesce into five distinet clusters, indi- cating the emergence of inter-view correspondences. In the middle row, the consensus variables w, derived from various complete combinations of the five views, are shown. At the start, these variables are widely dispersed, as the views are not yet able to transform into each other effectively, leading to inconsistencies in the captured information. As training progresses, inter- view correspon- dences established, and the first i dimensions of reliably encode shared information across views. As a result, all combinations the five views produce similar w' s, with their representations converging into indistinguishable, uniformly distributed clusters.
23

---

Published as a conference paper at ICLR 2025
Epoch
Epoch
Epoch
Epoch
Figure 9: T-SNE visualization of latent space dynamics during training on the PoLyMNIST dataset. Top row: Latent variables z for five views at different training stages, with colors repre- senting each view. As training progresses, the initially scattered representations gradually cluster, indicating the establishment of inter-view correspondences. Middle row: Consensus variables w, derived from different combinations of the five views, shown with different marker shapes. These scattered representations gradually align and become more consistent. Bottom row: Average consensus representations, colored by digit class, become more distinet over time, which reflects enhanced clustering and effective information sharing across views.
B ADDITIONAL EXPERIMENTAL RESULTS
In this section, we present additional experimental results to complement those in the main text. A complete version of the clustering results five datasets in Section 4.1, including standard devia- tions from five experimental runs, is provided in Table 3.
B.1 QUANTITATIVE AND QUALITATIVE RESULTS ON MVSHAPENET
In this section, present both quantitative and qualitative comparisons the MVShapeNet dataset to assess the performance of our method, MVP, alongside several prominent MVAE-based ap- proaches. These include the models discussed in Section 4.2.2, such m mVA,, VmV, mmJSD, MoPoE, and MVTCAE. Although MMVAE+ is strong method, it did not perform well on this dataset using the same CNN-based architecture, irrespective of whether the Normal or Laplace dis- tribution was applied. For this reason, we chose not to include it in our direct comparisons.
It's important note that MMVAE+ excels in handling complete datasets by leveraging auxiliary priors to facilitate cross-modal reconstructions, making it highly effective in real-world generation tasks. However, its design, which prioritizes robustness to hyperparameters that control the capac- ity of mopelity- subspaces, seems less suited for scenarios where data is missing. In such cases, its auxiliary prior may struggle to compensate for missing information, affecting ability to perform well under these conditions. This key distinction highlights the difference in focus between MMVAE+ and our approach, with each serving different objectives.
Figure 10 illustrates multi-view sample generation conditioned on view 5 across different object categories. All models were trained with a missing rate of = 0.5, representing a complex sce- nario where maintaining geometric consistency and capturing fine details across views is particu- larly challenging. MVP stands out by producing sharper and more consistent multi-view samples across categories compared to the other models.
24

---

Published as a conference paper at ICLR 2025
Table 3: Complete clustering results of nine methods on five multi- view datasets with missing rates of = 0.1. 0.1, 0.3, 0.5. 0.5, and 0.7. The first and second best results are indicated in bold red and blue, respectively. Each experiment was run five times using different random seeds.
33.58414.25
1.25
32 12.55+14.33
63.12.4.17
66 06.84:4.35
41.0611.88
75.25
48,644466 48.6613.10 19
74.85a4.00 06
78.75&4.77
21 21.77114.99 14.94
36.65 23
46.3 46.31111.9 1.99
0,73+0.31
45.23=1.64 1.66
7.79#1.60 1.7
22.41
00.59:1.. 1.84
12,6045.00 08
12.6425.13 13
16, 16.29-2.22 24
13 13855002 21
17.58:0099 99
24.38
18. 18.35+1 32
21.56624.04 0.4
24.68 1.58
10.03 69
5.90+2.88 28
23.24 49
11
11.51 1.884
10, 84
22.1441.44
3.15:2.72
30 60.66411 1.07
9.75+12.28
11.81411.79 7,99
25 11.111.21
29.73.0.72
14 14.66#12.71 71
31 11.2840.79
18.93-3.51 51
28. 78.59a2..15 35
36.23t00.9 0.39
ART
33
22.
50.5
21
25.
22.
NM
56.4129.0 86
56 36.89+10.22
68,94 20
74. 74,80=3,1 12
02.41
80,60 98
61.9912.97
80.78=1.84
82. 82.5662..31
39 19-42416.37
37
40. 40.50011.27
51.07=1.54 1.54
59 59.89.0.95
17.37+1.25 12
62. 1.43
16.10=1.80 1080
36,71 60
63.11#1.211
339336655 55
34.21
32.57=1.08 08
29.09:0.88 81
37.05 1.03
38 38.41+2.43
35 15,80=1 30
62
36,6624.62
39. 39.53e0.67
22.90a5.12 13
23, 23.11 34
25.7912.77
33.57+3.00
22.47=1.17
12.03+4.51 50
26.83&1.54
3.02+2488
36.0020.551
8 37
10.314111.5
21.36-2.44 41
30.0880.22
14. 14.77=11.10 10
28.0440.79 79
19.3923.44 .444
25.84 90
30 50.08:0.099
0.7
56.4
28
36.
ACC²
59 59.07=10.23
39.31
79.6624.22 28
77,38 35
68.8612.41
83,90
67.4813.8 83
8 81.6625.02 02
86. 86.74e5.11
42.35 12.35+14.35
41 11.52413.12
56,08 56,08 16
67 67.3342.29
19.43+1.55
58.0 11
19.43+2 62
39.47-2.61
66.5363.73
308416.3 35
31.1626,75
36.01 1.2
28.4 28.42 78
35.37
42.53+2.56
36 76-202222
39.59 39.5924.41
43.14=2.20
43.3644.33 35
41.32
46.9423.41
47.12+2.08
42.27 73
37 66
47.67
32.18:4.07
55.1642.93
50,60 05
51.33 86
60.57:4.44
04.0.22.
56.37 36
66,1641.76
54.11
61.91 =2.77
68.8740.32
60.5
38.93
37.32+13
66.8111.19
71 71.8433 94
50.2146.37
75. 75.3465.62
58.7240.44
78.50144006
83.5323.69
27. 27.88+12.17
28. 1R.37412.99
42 42.81:4.777 4.77
48.2144.00
15.11+1.28
46.424141 11
10.52+1.76
20-5243.40 44
59.11#2.33
14,4844.45
14.5724.44
20. 10.5942.08
13 13.38:0.64
19.70 19.70+1.3 335
24.96 24.9621 46
20.34H0.78
25.03a2.42
26.50 1,2
11.0413 3.63
6. 6.8022.77 78
24.7441 .67
28.2700881
9.8013.53
11 11.30 35
23.5642.01
4,9822.51 51
30.2440.93
12.79112.00
15.40&14.21
26.21+13.13
12.52+12.63
32.2540.55
99.3642.48
34,00250443 43
38.0650.12 12
ARI
45
46
12
29.
NMD
60.2728.42 42
71. 71.1741.06 06
79 79.23+2247
67.4914.05
80 80.23+3310 10
67. 67.9544.223
82.9612.41
85.44
46.71
$7.05412.10
55.75 62
61. 61.5543.00
61 11.8440.8 89
6-. 1..1862.50 50
19,87#103
34. 19.3944.68
7140 100
37.2643 49
37 77.66+3.00
35 15.95+2.50 50
28 68.80
38 19991.00
39.1 39.19 85
3345-0991
29.5882..25 25
41.84 79
22.6344.8
24. 24.30+6 O8
27. 27.29 25
30.5+11.2
20.41
11 11.8083. 58
28-4361000
5.1422.33 33
35.334..16
11.01+1088
13.3241 1.89
22.08+1 11
31.4950.25
13. 13.96211.70 7.
28 38.62460.78
29. 29.11+2.47
29.27 26
20
32.55:0.00
0.5
60.
60.
7
39
ACC1
63 63.25 30
82 62.7610.71
81 31.3942.83
83
74.7344.51
84. 84.6415547
73,8340.46
85.0114.49
90.7624.69
48. 33.00-10.85
47.21
60.66 60.65+3..0 62
68.3745 67
51 $1.0+2.44 4.47
60.07+4.51 51
22.90 22.90=2.80 2,3
39 19-5745.54 54
74,20 3121
34.05 5.50
34,4943.00 96
40.1 10.1742,14 14
27.24+1.2
36.5441.0 11
43,441143 44
37.96+1 187
41.1323.32 33
45.28 1.4
44. 14.1044-44
42. +2.32443.13 13
48.51 14.54
50-78-22.16
41. 41.7742.28
38.21 44
47.59&1.68
35.3753.00 .9
53.6742.88
53 $3,8948..0 00
54,0819 96
61 61.0916.02 02
680559.85
55.0 55.09 18
63 654470.88
67.80 07.8042.33 36
67. 67.8950046 46
70.21=0.09
M.76211.45
70.83:2.10 10
73, 73,36:3 40
72.28=6.00 09
68. 1541811
78.5 08.5035.33
81 81.59a5.77
35.63:5 86
41 11.2543.77 74
47.71
52.78+2.71
48 48.9342.94 94
14. 14.17=4.28
28. 25.294111 80
61.3513.6
17. 77.1242.00
17.06-1333 32
20. 20.8441 54
16.29
24 24.2240224
24.9410.85
24.7 24
26, 26.6861100
27.53+1.80 1.56
12.71
8.1522.47 47
26.89+1.90 99
26.611181
10. 14.3242.34
10.47=2.09
25 25.6642.38
11.7 11.77=1006
31.35a3.30
16.02=13117
22 12.33112.31
27.93=12.99
11 11.29912.69
33.6340.43
35.26660.3 38
39.7 39.7650.24
ARI
32.3
44
33.
0.3
NMD-
64.9326-000 40
65.76x6665 62
74.67+1.47 1.77
80 95
69.0844.17
79.08:3.61 16
74.5341155
+2.8243382
84.8622733
54,23:5 71
54.5966.82
56.00 36.00=2.88 89
61.7842.21
08.88:1.94
64.2 64.2142.72 12
25.7445.44 40
45, 15,18 185
73. 73.0912.55
39. 09-46411 1.79
39 09.80+2.35 2.35
38, 25,5741555 30
32.5 22.54
42.4 12032
39 59.4220.56
38. 08.0600551
42.3242.11
42.54x1.02
20.6224.66
26 26.0755.37 37
29 89.852258
14.88=1.00
22. 22.4841.99 1.97
13.1 13.11:4475
30.7142.09
10.8420.11
37.0242.08
14 1.04
19.4 19.4219.93 94
23. 18.18:10.38
22.9560.41
12.09=11.52 1.52
28. 10.1000332 32
$2.9540.59
30.3 10.31=00.2
33.57:0.09
65.7
74,
79.0
68.0 08.02455.87
K5 15.24+1.44 120
81 818223.27
7R. 18.701-.22
22
79 8026.76
80 N0.9542.75
55.3645,55
88.6955.89
55 55.554449
57 57.20+3.90 98
00.7 60.7342.49
61 11.00+1 41
24.3344.17
53.3742.57
74.72-4.58
36. 11166.177
36.7 36.734 7342322.6 86
40.3 10.37+1. 85
31.341 .2
40.04-0-67
43.2111.511 1111
43 33.14+10 O0
43.6821.54 1.44
45.81=2755
46.2841.9 95
43.7 722.92
20.54122.11
11.33-2.28
40. 40.5612.008
38.5 51+.23300
51 11.0942.33
43 13.06+1.02
00
55.7045.10
60.321-1 70.3214477
62. 22.3845.0 996
55 $5.5925.06 566
64.0341. 14
70,7920.50
69 5K.5811.21 21
71.28:0.23
ACC1
66.834477
$7. 67.76+5
43.2
68.
51
69
55.1 33,1342.01
44.61+2.49
9.962220666
71 1..5941 92
68. 92.9211199 12211.9
71 11.8426.71
79 99.5166.58
19.6622881
84. 16.665..16
40 00.87+1223 1.3
43 1..02.3.22
48, 96:5.55555
55. 49
45. 95.94-4445
45, 15.3542.40
40
21 11.55+0.88
23, 23.7842.59
66. 76.7310.71
19 29.990022
20. 00.36440227
24 24.4522.33
14, 14.8640559
23 8..60-0844
84
24. 24.5642.15
25.811154 S4
28. 28.6211.52
27.90=1.38
1.38
15 15.4515555
7,59=1.61
25. 13.7842.01
28 21+2.0
7.92 7,92+2.18
10 10.9422.70 720
29.446113
15.21=4.70
32.200+2.02
14.26=11.24
10.21
21 87:4444
3. 3.72=3.01
32. 52.3240.59
M4-124442
34, 98
41. 11.1050.11
ARI
77.
0.1
NMC
71.174199
92
/2.22+1 162
00.0621.55 96
19.47+1300 30
71-7341.08
78.43:3.90 90
82.1943.94
83.51+198
87. 87.0822448
59. 0,.99+1 08
61. 11-0442.59
61. 70:00-4000
67,824 07.8243.07
-7444111
62. 18
38 18.31
41 41.8343499
49
77. 77.6720.66
41 11:0040552
42. 77.00-0555
41 31.53+160 60
30 00.38:0.95
41 89490-411
38. 28.7912.49
40.03=1.14
45, 35.06681.33 133
43.7750993
13.9344-52
23 8448
28, 94.294
35. 15.56+199
15
21 11.384444
14,35+3.10 10
5.5221137
16. 16.0855599
37. 37.2561.17
11002:777 78
12. 12.4417.71
17.3 17.31=13900
40400.62
5.33 96
28 4340441
34.5 44.5340.38
30.4 52
34.81=0,211
80.0
79
78.
13.8300.66
0..6+0065
89. 11
81 11.2741 .48
82. 1842.06
80. 3.3
88 86.55.5539
39
87 8943.28
900 5.
58 1
S7 2.22743.22
66 0.003:5.04
5.04
72. 2.93355.28
52. 54
5% 12.7723.12
29 23+0.33
44.5 44.5344000
78.67 23.6722.39
38 22. .00
39. 44440.8444
42.5 22.5122.22
29. 43 .21
37,00 7,000 .8
42. 22.6911.77 1.77
43 83.28.3
45.1 26.16.24.40 40
45. 15.70=1.6 1-63
42, 70+ .20
48 2.33
51 11.26+33.45
41 11..709997
38.35 N8.35+5.07
54.01+1.67
44 11.06E11 1.87
57 7788333.66 6.66
57. 11+5.77
57 13
59. 72.7228.27
52. 2.6
63 13.48&1.66 1.65
71. 11.5020.46 46
69. 49-4660-44
72. 72.08:0.10
ACC
47 77.6611
69
Missing rate
Metries
DCCA
DCCAF
DINVC
DSIMVC
Completer
CPSPAN
ICMVC
DVIMC
MVP(Our))
DCCA
DCCAE
DINVC
DSINVC
Completr
CPSPAN
ICMVC
DV
MVP(Ours)
DCCA
DCCAE
DINVC
DSIMVC
Completer
CPSPAN
1CMVC
DVINC
MVP(Ours)
DCCA
DCCAE
DIMVC
DSIMVC
Completer
CPSPAN
1CMVC
DVIMC
MVP(Ours)
DCCA
DXCAA
DIMVC
DSINVC
Completer
CPSPAN
ICMVC
DVINC
MVP(Our))
CUB
Seouicts
25

---

Published as a conference paper at ICLR 2025
Input
DVAE
mmYAE
BEBISD
MoPol
MVTCCAE
MYF
Figure 10: Multi-view sample generation conditioned view 5. The leftmost column shows input images from view 5, randomly selected from five categories: table, rifle, chair, airplane, and car. The following columns display five-view samples generated by different models.
Table 4: Quantitative results on the MVShapeNet dataset. The metries 'ACC' and "View' rep- resent the classification accuracy of generated images for object categories and perspective angles, respectively, using two pretrained classifiers. The metric 'SSIM' measures the structural similarity of generated images compared to the ground truth. Models were trained with incomplete data at missing rates 0.1 and 0.5, and evaluated across different combinations of missing views. Results are averaged over same-sized subsets such 'missing view', which includes five cases: (2,3,4.5) (1,3,4.5), [1,2,4.5), and (1,2,344.
Testing Subset
Missing 1 view
Missing 2 views
Missing 3 views
Missing 4 views
Method
ACC1
View
SSIM
ACC1
View
SSIM
ACC
View
SSIM
ACC1
View
SSIM
mVAE
90.56
81.34
0.8406
86.60
78.56
0.8139
79.11
73.97
0.7737
62.27
66.70
0.7045
Vml
91.56
81.28
0.8141
91.55
81.39
0.8132
91.56
81.42
0.8126
91.78
81.51
0,8141
DmnJDD
41.16
53.03
0,7217
40.91
53.11
0.7210
40.77
53.24
0.7207
40.55
53.16
0.7215
MoPoE
86.45
0.8252
86.34
0.8223
0.8159
84.40
0.7961
MVTCAE
93.65
83.45
0.8570
93,39
83,41
0,8440
92.71
83.07
0.8308
89.90
81.70
0.7858
MVP(Ours)
94.66
84.34
0.8349
94.76
0.8336
94.77
0.8274
94.47
84.12
VAA
79.04
76.14
0.8111
73.02
11.88
0.7802
63.76
66.88
0.7386
49.28
61.26
0.6800
mm mmmVAE VAE
91.22
81.67
0.8088
91.26
81.63
0,8081
91.30
81.51
0.8076
91.38
81.40
0.8092
DinJDD
92.40
82.49
0.8118
92.41
82.46
0,8110
92.40
82.52
0.8105
92.37
82.47
0,8192
0.5
MoPoE
92.84
82.46
0.8314
92.71
82.31
0.8292
92.
82,00
0.8247
90.55
80.91
0.8129
MVTCAE
93.38
83.09
0.8482
93.07
82.91
0.8396
92.22
82.53
0,8540
89.03
80.86
0.7865
MVP(Ours)
94.57
84.60
0.8239
94.72
84.64
0.8276
94.75
84.66
0.8268
94.45
84.50
0.8126
Table 4 presents the quantitative evaluation on the MVShapeNet dataset. The 'ACC' and 'View' metrics represent classification accuracy for object categories and perspective angles, respectively, while 'SSIM' measures the structural similarity between generated and ground truth images. As seen from the visualization, SSIM primarily captures the basic structure of the generated images, result- ing in relatively minor differences across most methods. Our method, MVP, consistently achieves competitive SSIM scores and surpasses all other models in both classification accuracy metrics. This robustness is particularly evident compared VAE, which suffers a notable drop in performance as the number of missing views increases. Figure 10 illustrates this, showing how VAE, relying on PoE fusion, loses the structural integrity of the input object when fewer views are available.
Interestingly, VAE, mmJSD, and MoPoE perform better in scenarios with higher missing rates. This can be attributed to their reliance on fusing all available views in their posterior or informational priors, making them dependent on missing data to memorize different incomplete combinations. Thus, they struggle to fully utilize the complementary information provided by additional views. On the other hand, VAE, which uses MoE fusion, shows improved performance with more available views during both training and testing. However, its fusion strategy struggles to effectively aggregate information from multiple views, resulting in smoother generated objects that may lack the distinct angles necessary to capture fine details.
MVTСAE demonstrates stable performance across all conditions by enforcing strict consistency between views, making it resilient to missing views. However, this strict consistency can result in the loss of unique, view-specific details. Our MVP method, by enforcing consistency after trans-
26

---

Published as a conference paper at ICLR 2025
leae
lngg
2o Sra Doaniciiingg coeling
laan plaes
wig and ded outer
ar
and delk
gryboul
with ang
furtin Joee
serther
bocyy ce
bre
tia Eot
boak
ligh
Figure 1I: Conditional generation by our method on the CUB dataset given only text.
formations, not only maintains robust performance but also effectively integrates information from additional views while preserving the sharp, distinctive details of each view.
B.2 ADDITIONAL EXPERIMENT RESULTS ON CUB DATASET
We conducted additional experiments the raw-text version of the CUB dataset to evaluate the generative capabilities of our method on datasets with non-RGB views (Netzer et al., 2011; Shi et al., 2019). This version contains 88,550 training and 29,330 testing samples, comprising paired bird images and textual descriptions. Following MMVAE+ (Palumbo al., 2023), we adopted the same network architecture and latent dimension (64 dimensions).
The results, shown in Figure 10, demonstrate that our method effectively generates images aligned with textual descriptions in the incomplete setting. Clear semantic alignments observed in at- tributes such as colors (e.g.. black, white, brown) and structures (c.g.. belly, beak, wings). However, the generated images exhibit blurry backgrounds and contours, consistent with observations in MM-
27

---

Published as a conference paper at ICLR 2025
WoL sociis
Figure 12: Conditional generation by our method on the CUB dataset given only images.
Table 5: Ablation study of permutation and different priors in MVAEs with inter-view corre- spondences. Clustering results on the Handwritten dataset with missing rate of 0.5, averaged over five training runs. "Perm." refers to the use of permutation, which reorders the variables of the same views (indicated by the same superscript) for reconstruction. The "Regularization" column indicates the use of different prior settings. The last row represents our proposed model.
Model
Reconstruction
Regularization
ACC1
NMI1
ARIT
Random Perm. Random Perm.
Diagonal Random
31.58:10.75
30.98
16.9448.81
2
59.37+4.30
58.7042.73
44.6944.33
3
w/o Perm.
Cyclic
79.80=1.26
81.35+1.00
73.98+1.48
Cyclic Perm.
N NO,01 1)
86.96#6.44
82.49+2.80
79.2345.65
5
Cyclic Perm.
Fusion
87.71
84.47+1.57
80.77+4.61
Cyclic Perm.
Diagonal
88.5326.43
85.00+2.86
81.7726.14
Ours
Cyclic Perm.
Cyclic
90.7624.69
85.44+1.55
83.53+3.69
VAE+, which stem from the limitations of single-step VAE-based generation. Advanced approaches, such as D-CMVAE (Palumbo et al., 2024), address this issue using diffusion models, though such refinements are beyond the scope this study.
Quantitative evaluation remains challenging, as traditional metrics like diversity or reconstruction scores fail capture cross-modal consistency. While MMVAE+ introduced metric based HSV color alignment with textual descriptions, it does not measure higher-level consistencies, such as specific bird features (e.g., wings, beak) or environmental contexts (e.g.. sky, water). Future di- rections could explore more precise metrics or employ large vision-language models for automated evaluation (Lin et al., 2025).
B.3 ABLATION STUDY PERMUTATION AND PRIORS SETTING
We conducted an ablation study to validate the key design decisions in our method, and as shown in Table 5, our proposed approach achieves the best performance. For MVAEs that explicitly model inter-view correspondences, given incomplete input data (x(*)) J.ez, we derive set of latent vari- ables {2(") corresponding to different views, organized into matrix Zo for clarity. In this notation, the superscript in = indicates that it represents the /-th view, while the subscript denotes the source view. Thus, 20() used to reconstruct the I-th view's observation and can be regularized by other latent variables associated with the 1-th view, all of which share the same superscript in the I-th single-view cell S }
Model 1 serves as a simple baseline. In this model, we randomly select a latent variable from the
1-th view reconstruct the observation (x()) for that view, using z(") (with matching superscripts and subscripts) to regularize all variables for the I-th view. However, this approach performs poorly
because the random selection can pick either the self-view encoded z () or the cross-view transformed z (v * (), leading to difficulties in reconstruction and confusing the learning process.
(!) Additionally, the regularization term for 2 simply vanishes entirely.
28

---

Published as a conference paper at ICLR 2025
Model 2 relaxes the prior by replacing the strict 2i with posterior derived from a random permu- tation. This improves performance due to the added randomness. However, since random permu- tations do not guarantee that elements are moved from their original positions, some regularization terms for 20 may still vanish.
In Model 3, we exclude permutation during reconstruction. For the 1-th view (9),, we encode into 2/" and use it directly for decoding. Other latent variables, derived through inter-view corre- spondences, are aligned with the target view's variable solely through regularization terms. This approach slightly improves performance, as it effectively adds information from other views during decoding. However, it cannot establish and learn correspondences between views effectively.
In Models 4-6 and our proposed model, we apply cyclic permutation to the matrix Zo. using both the diagonals of matrices Zo and 1 for reconstruction, effectively combining self-view reconstruction with cross-view generation. The use of different informational priors ('Fusion', 'Diagonal', and 'Cyclic') ) demonstrates clear advantages over the standard normal distribution N (0, 1), which lacks the flexibility to adapt to varying samples.
'Fusion' involves using the geometric mean of posteriors within homogeneous set as the prior for each posterior in that set, kind of like reducing the variance of all unimodal posteriors. For example, the geometric mean of {&n Juez is used to regularize each 2/ However, when views are missing, the varying set sizes |Z] for different samples result in imbalanced fusion and increased computational complexity.
'Diagonal' regularizes all {z. Jvez using the distribution of 2 and the geometric mean of (z (z(") Jiez regularize all w. However, when the regularization term fails, leading to suboptimal performance.
'Cyclic', as used in method, efficiently reuses the permuted posteriors priors and fully leverages the relationships between views. Since no element remains in its original position, this method maximizes inter-view correspondences and achieves the best perfor- mance. Additionally, it avoids extra fusion computation for informational priors by simply caleulating the KL divergence of Zo and Z1.
In all, our model outperforms all others, demonstrating the effectiveness of our learning strategy.
B.4 COFFFICIENTS OF REGULARIZATION TERMS
As is common in the VAE literature, the objective function in Eq. (4) can be rewritten the of a reconstruction term and two KL-divergence terms, each weighted by the coefficients 31 and B3 (Higgins al., 2017). As shown in Figure 13, conduct sensitivity analysis examine how different combinations of 31 (for regularization) and 32 (for w regularization) impact clustering accuracy. Since is deterministically computed from fusion of z, both terms essentially act as regularization on but with different objectives. The figure shows that the best performance achieved when 31 5.0 and Bg 2.5, resulting in clustering accuracy of 90.76. This indicates that moderate weighting of both regularization terms strikes an optimal balance between promoting smooth latent space transitions (Inter-Vies Translatability) and ensuring latent space consistency (Consensus Concentration).
the coefficients
Interestingly, the performance degrades when either 31 or f2 is set too high, as seen when 31 10 or |8g = 10. This likely results from overly constraining the latent space, which could reduce the flexibility needed for effective inter-view transformations. Conversely, lower values of 31 and 3n (e.g., B1 1.0, 32 = 1.0) show suboptimal performance, indicating insufficient regularization and thus poorer structure in the latent space.
B.5 THE CHOICE OF LATENT REPRESENTATION DIMENSIONS
In this section, we examine the choice of latent variable dimensions across different datasets and compare the selection of the first à dimensions for encoding shared information in the PolyMNIST and MVShapeNet datasets. For the clustering task, since requires high consistency between views to preserve shared information and reveal the underlying category structure, use 100% of the
29

---

Published as a conference paper at ICLR 2025
70
50
2.0
1.5
1.0
Figure 13: Sensitivity analysis of the coefficients for the two regularization terms the ELBO. 31 controls the regularization for and f2 controls the regularization for w. Clustering accuracy (ACC) is reported on the Handwritten dataset with missing rate 0.5, averaged over five training runs.
Table 6: Clustering results across different representation dimensions (2). Clustering accuracy (mean + standard deviation) shown for various latent dimensions with missing rate of y = 0.5, averaged over five runs. The green cell marks the selected dimension.
Dataset
Dimension of latent representation
16
24
32
Handwritten
85.88+6.15
90.76144.69
88.3245.99
86.41#5.63
16
32
64
CUB
67.30+5.36
74.20+33.2
73.03+33.9
69.40+2.95
16
32
64
Scene15 12.58+1.31
45.28+1.44
44.44+0.69
43.74*1.79
16
32
48
Reuters10
$2.60+2.66
$3.6742.88
53.89+5.32
43.01+4.26
16
32
64
SensIT Vehicle
70.33+0.48
69.9540.28
70.21
70.00+0.21
dimensions of z to encode shared information, meaning k = d. with w having the same dimension- ality z. Table presents the clustering results for various latent representation dimensions across different datasets. Rather than always selecting the best-performing dimension, we strike a balance between model complexity and the inherent structure of the dataset for simplicity.
Table reports the reconstruction performance on the PolyMNIST and MVShapeNet datasets as we vary the proportion of dimensions (k) used encode shared information. The results show that performance remains robust across different ratios, allowing us to select the ratio based on each dataset's characteristics. For PolyMNIST, which exhibits greater variability between views due to diverse background styles and colors, smaller k/d ratio (50%) for shared information encoding more effective. This because only a small portion the pixel data (representing the digit) is consistent across views, while the rest varies significantly. In contrast, MVShapeNet, with more consistent views (mainly different angles of the same object against a plain background), is better suited to higher k/d ratio (75%).
30

---

Published as a conference paper at ICLR 2025
Table 7: Reconstruction results across different shared feature dimensions (k) in the d- dimensional x. The models are trained on the PolyMNIST and MVShapeNet datasets with a missing rate of rj = 0.5. SSIM reported on the complete test set. The green cell highlights the selected dimension.
k/d
d
25%
50%
75%
100%
PolyMNIST
96
0.6023
0.6188
0.6137
0.6134
MVShapeNet
256
0.8383
0.8384
0.8381
0.8374
C IMPLEMENTATION DETAILS OF OUR METHOD
C.I NETWORK ARCHITECTURES
For the experiments in Section 4.1, we employed fully connected neural networks similar to those used in previous studies. The choice of architecture depends on the dataset's characteristies, such as input dimension and number of samples. We used one of the following network configurations:
d, 256 - 256 1024 d
de - 512- 1024-0144
d,, 512 5122-512-- 2048 d
de 1024-02-- 1024 2048
Here, d, is the input dimension, and dl is the latent dimension. Each fully connected layer is followed by a ReLU activation function to introduce non-linearity, except for the final output layer, which uses Tanh activation function to normalize the latent representation within a bounded range.
For the experiments in Section 4.2, we utilized simple convolutional neural networks (CNNs) com- bined with fully connected layers, ReLU activations, and Tanh activations. In both the PolyMNIST and MVShapeNet datasets, each view is encoded and decoded using separate Variational Autoen- coders (VAEs). The architecture of single VAE's encoder and decoder is summarized in Table 8. The same architecture is applied to each of the five views. The filter sizes (Sfilter tupleS) for PolyM- NIST (input shape [3, 28, 28)) and MVShapeNet (input shape [3, 64, 64)) are (64, 128, 256, 512) and (64, 128, 256, 512, 512), respectively.
Table 8: CNN architecture used for each VAE the PolyMNIST and MVShapeNet datasets. Description
Module
Layer Type
Conv2D Layers
4 layers with increasing filter sizes Sfilter-tuleeS and stride of 2. Each followed by ReLU activation.
Encoder
Flatten
Flatten output to vector of size 2048.
Linear
Fully connected layer to latent dimension (d), followed by ReLU. Separate layers for mean and log variance of latent distribution.
Linear (Mu, Vor)ar)
Linear
Fully connected layer from concatenated latent vector (d + k) to output size of 2048.
Decoder
Conv ConyTranspose2D
layers with decreasing filter sizes $Reverse filter tupleS, fol- lowed by ReLU activations.
Layers
Conv TonnTrranppss21)
Final layer with filters, followed by Tanh activation to output RGB image.
(Output)
The correspondence between each pair of views is modeled using a simple fully connected network with linear layers and LeakyReLU activations. For all datasets in Section 4.1,
the architecture 128 d. For MV MVShapeNet, we use d - 256 512 256 d, and for PolyMNIST, we d 256 1024 256 d. The larger network for PolyMNIST reflects the greater differences between its views, requiring more parameters to effectively model the conrespondence.
d- 128 256
For more implementation details, please refer to the code provided in the supplementary material.
31

---

Published as a conference paper at ICLR 2025
C.2 MISSING PATTERNS AND MASK GENERATION
We follow standard practices in incomplete multi-view learning, where missing-view masks are generated before training to inform the model of the missing patterns. Following the approach outlined in the public repository by Tang & Liu (2022), for a dataset with L views and a missing rate 17 we randomly select 7) x len(dataset) samples to be incomplete. For each of these samples, we randomly remove between 1 and L 1 views, ensuring that every incomplete sample retains at least one view while missing least one. The missing- masks (e.g.. '00101") are generated for the entire dataset before training and stored in a "fingerprint" file for each missing rate 1). When the dataset loaded, the corresponding masks are applied as long as 7) is specified. This ensures consistent missing patterns across different models, enabling fair and reproducible evaluations.
C.3 PRE-COMPUTATION OF CYCLIC PERMUTATION INDICES AND BATCH PROCESSING
Given an index set A = [1,2,.... n). Sattolo's Algorithm can efficiently generate a cyclic permu- tation, as outlined in Algorithm 1. By leveraging stack operations, all possible cyclic permutations of the index set can be precomputed.
For incomplete multi-view datasets with fixed missing patterns, each sample has distinct mask (e.g., '01101"), indicating the available views (.... Precomputing the cyclic permutations for these masks reduces computational overhead during training. Treating missing views fixed points (e.g., views and 4 in this case) ensures that the permuted index sets maintain the same length (c.g., [L, 2, 3, 4, 5] + [1,3, 5, 4, 2)). simplifying batch operations. These precomputed permutations are also stored in the "fingerprint" file for easy retrieval.
During training, these precomputed permutation indices can be directly applied to arrays containing values for each view, enabling efficient rearrangement of data. For a dataset with L views, cyclic permutations are applied to the latent variables of each sample by permuting elements within each column of an T. x T. matrix Zn- As shown in Figure 5, the precomputed cyclic permutations are chosen, structured as an I. x L. array and are used to reorder Zo into Z1. Since these permutations are stored with the dataset, the transformation process can be efficiently executed in batches using indexing, allowing for fast and seamless operations during training.
C.4 OVERALL TRAINING PIPELINE OF OUR METHOD
The general pipeline of method is outlined in Algorithm 2. The goal is to learn multiple encoders, decoders, and inter-view correspondences. To implement more easily, we organize the variables into a matrix Zo. The pipeline consists of the following four steps for an L-view dataset:
1. Self-view Encoding and Cross-view Transformation for Zo: We employ L encoders to obtain (n) ((((((")), I(a(")), v (I....,L). L). Then, using all inter-view corre- spondences {fiv]. we compute 2 ~ Nifie o p(zl")), ) fiv I(x(")) )), for I * v. where I, E (1...., L}. These values are organized into a matrix Zo = |=00] IxLx where the row index denotes the subscript and the column index denotes the superscript.
2. Cyclic Permutation for Z1: We apply L pre-generated cyclic permutations {ar) pro-
vided with the dataset, to form an permuted index array of size L x L. This array is used (1)
to transform Zo into 21 2ox(t) LXL'
3. Fusion for W with Complete-view Partition: For both Zu and Z,, we fuse each row by computing the geometric mean of the k-dimensional marginal distributions, resulting in 24 = (wp) and Ω = (wr) We also extract the diagonal elements from Zo and Z,, (1) H/m and (= respectively, where (1()) LL.
corresponding to {=)
4. Decoding with w and 2: We use to reconstruct the view indicated by its superseript, combining it with consensus variable w. Specifically, we apply L. decoders to reconstruct
the views as follows: 100) Decoder([w]] 22") 2(11) and ( Decoder([wr, =(1) "
5. Masked BLBO Update: We apply masks to ignore terms related to missing views and update the model using the ELBO objective (Eq. 4).
32

---

Published as a conference paper at ICLR 2025
Algorithm 2 Multi- View Permutation of VAΣs for Incomplete Data
Input:
Incomplete Multi- View dataset (X:)==) latent variable dimension d, and shared feature dimension & (k < d).
Output: MVAEs with inter-view correspondences.
Initialize parameters { ). {0e). and (oto), where v, I E [7].
1: while maximum epochs not reached do
2:
for in available views
3: Generate (u²²) Ee
Encoder(x<(1) through v-th encoder;
4: Compute (Mt
(1) > using fiv for of u, 2., L};
5: end for
6:
For derived Zu. apply cyclic permutations to obtain Z1;
7:
for in available views do
8:
Calculate geometric mean (0n, An) from Cn and sample win
Sample ( from single-view cell Sn;
9:
10: Generate f(() = Decoder(wn, ( through n-th decoder;
11: end for
12:
Update { {0.}, and {arv> by maximizing the ELBO (Bq. 4);
13:
end while
Table 9: Overview of comparative methods for incomplete multi-view learning. This table summarizes the key modeling details of the methods used for comparison.
Method
Year
Venue
#View
Description
DCCA
2013
ICML
Two
Baseline, CCA-base method
DCCAE
2015
ICML
Two
Baseline, CCA-based method
DIMVC
2022
AAAI
Mutiple
High-dimensional nonlinear mapping, EM
DSIMVC
2022
ICML
Mutiple
Bi-level opimization
Completer
2023
CVPR/TPAMI
Two
Information theory, contrastive learning
CPSPAN
2023
CVPR
Mutiple
Prototype alignment
ICMVC
2024
AAAI
Two
GNN, high-confidence guiding
DVIMC
2024
AAAI
Mutiple
GMM+ VAES
C.5 INFERENCE OF MISSING VIEWS WITH AVAILABLE VIEWS
During inference, given incomplete input sample {2(")) }z, first derive the latent vari- able set (20)) ((.))exx(1] For each view E {1...., L}. we compute an averaged latent vari- able z()) by taking the geometric mean of the distributions from the available views, z()
Geometric Average((2)) (")) Jeez). This process provides complete set of L latent variables {
Next, we compute a consensus variable w from this complete set by taking the geometric mean of their marginal distributions over the first dimensions. Finally, we concatenate w with each g(1) and use the /-th decoder to reconstruct (1)), allowing us infer all views.
D COMPARATIVE METHODS
D.1 INCOMPLETE MULTI- VIEW LEARNING METHODS IN SECTION 4.1
For the methods listed in Table 9, follow their original implementations provided in the spective public repositories. For methods that are limited to handling only two views, we train them on all possible combinations of two views and report the best performance for each dataset.
33

---

Published as a conference paper at ICLR 2025
Table 10: Comparative methods for different multi-modal VAEs. This table outlines key details of various multi-modal VAE approaches used for comparison, including the latent variable dimen- sion, joint posterior fusion strategy, and prior setting.
Method
Year
Venue
Dimension
Joint Posterior
Prior and KL divergence term
mVAE
2018
NeurIPS
512
PoE
Spherical Gaussian: (V,0,)
mm VAE
2019
NeurIPS
512
MoE
Laplace distritbution
mmJSD
2021
NeurIPS
512
PoE/MoE
JS divergence with Geometric Mixture
MoPoE
2021
ICLR
512
MoPoE
Spherical Gaussian: ('00, 1)
МVTСАЕ
2021
NeurIPS
512
PoE
Spherical Gaussian: WiO, 1), Total Correlation
objective resulting in forward KL
VME+
2023
ICLR
96/256
MoE
Laplace distritbution
D.2 MULTIMODAL VAES SECTION 4.2
Table 10 summarizes the key modeling details of the various multi-modal VAE approaches used in our comparisons, including latent space dimensions, joint posterior fusion strategies, and prior settings. We adhere to the implementations provided in their respective public repositories. For the first five models, we set the latent dimension to 512 for both the PolyMNIST and MVShapeNet datasets. For VAE+, matching the settings used in our method. Following its original implementation, 50% of the latent dimension is allocated the shared subspace, with the remaining 50% designated for view-specific subspaces. All methods were trained for 300 epochs across all datasets.
use latent dimensions of 96 for PolyMNIST and 256 for MVShapeNet,
E DATASET INFORMATION AND CONSTRUCTION DETAILS
Table provides an overview of the statistics for all datasets used in this study, all of which are publicly available through their respective repositories and the Huggingface website. For PolyM- NIST (Sutter et al., 2021), we present sample images in Figure 14.
The MVShapeNet dataset was constructed from the ShapeNet dataset (Chang et al., 2015) using the rendering tool developed by Stanford?. We selected five categories: table, chair, car, airplane, and rifle, with total of 25155 samples. The number of samples for each category is 8445, 6778, 3514, 4045, and 2373, respectively.
To convert the 3D point cloud data into multiple 2D views, we employed directional lighting setup. The primary light source was configured as a sun light with shadows disabled, a specular factor of 1.0, and an energy level of 10.0. To ensure consistent lighting across surfaces not directly illuminated by the primary source, secondary sun light was added with a low energy level of 0.015, positioned 180' relative to the primary light. Both light sources had shadows disabled to maintain uniform illumination. The camera was placed at coordinates (0, 0.6). with focal length of 35mm and a sensor width of 32mm. To capture multiple views, the camera was constrained to track an empty object the origin, which We selected five views, each taken from viewpoints spaced 45 degrees apart around the front of the object. Example images from the MV MVShapeNet dataset shown in Figure 15.
rotated in 8 increments (360%/45 steps) around the Z-axis.
whw.. http:htwwwwoooeeeeeeettetteeeeee
34

---

Published as a conference paper at ICLR 2025
C
Claos
Dos
Cloo
Gosc
Cloa
Claa 9
C
4
9
Ven
2
3
4
S
6
7
8
S
3
//
6
7
1
3
5
6
9
Viww 4
3
U
6
Figure 14: Visualization of randomly selected MNIST samples across 10 digit classes (0-9) and modalities (m0-m4). Each column represents a digit class, and each row shows a different modality.
table
table
chai
chair
sirplane
airplane
car
View1
Vinwd
View5
Figure 15: Visualization of randomly selected MVShapeNet samples from 5 object categories. Fach
row shows 5 different views (View1-View5) of two randomly selected samples from each category.
35


Reviews:
[{"review_id": "liqELyKOKJ", "reviewer": "Reviewer_o4Jy", "summary": "The paper introduces a novel model, Multi-View Permutation of Variational Autoencoders (MVP), designed to address incomplete multi-view data by establishing robust inter-view relationships through cyclic permutations within the VAE latent space. MVP uniquely integrates cyclic permutations and latent variable partitions to encode both shared and view-specific information, enabling inference for missing views without sacrificing inter-view coherence. By incorporating an informational prior through cyclic permutations, MVP transforms the regularization term into a similarity measure, enhancing the consistency and sufficiency of representations across views. Experimental results on seven benchmark datasets demonstrate that MVP outperforms existing IMVRL methods in clustering and generation tasks, showcasing its ability to generate coherent and robust representations even with high missing-view rates.", "questions": "Did the authors employ an analytical form or a sample-based estimation for the KL divergence terms in the ELBO computation?\n\nHow does the computational complexity of MVP scale with dataset size, and are there specific optimizations implemented to mitigate the computational overhead introduced by cyclic permutations?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The application of cyclic permutations in the VAE latent space presents a unique and innovative approach to modeling inter-view relationships. This method appears more robust than prior approaches, effectively capturing and maintaining consistency across incomplete multi-view data.\n\nMVP is rigorously tested across multiple datasets under varying missing rates, demonstrating strong adaptability and consistently superior performance over previous methods in both partially and fully observed data settings. The results confirm MVP’s effectiveness in handling diverse levels of data incompleteness.\n\nThe paper includes thorough experiments and analyses, such as ablation studies and additional tests on relatedness, offering comprehensive insights into MVP’s performance. The authors strengthen their findings by running models multiple times with different random seeds and presenting deviations on the plots, adding credibility to the robustness and reliability of their results.", "weaknesses": "Although the paper centers on incomplete multi-view learning, much of the processing related to incomplete data, such as section C1, is relegated to the appendix. This structure may hinder readability and comprehension. A more detailed description of the incomplete data processing steps in the main text would improve accessibility and clarity for readers.\n\nWhile the proposed method’s effectiveness in handling random arrangements of latent variables is supported by experiments, it also introduces additional computational complexity, particularly in the cumulative computation of the multi-view variational lower bound function. MVP’s complexity seems closer to that of MMVAE and notably higher than MVAE and MVTCAE. A detailed analysis of this computational cost would be beneficial, as it has important implications for scalability in multi-view applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6vRN4hYl2F", "reviewer": "Reviewer_dPkn", "summary": "This paper introduces the MVP framework named of Multi-View Permutation of Variational Auto-Encoders, which focuses on a practical problem for incomplete multi-view learning. MVP can use MVAEs to establish view relationships in the latent space, thereby aggregating more comprehensive information while inferring missing views. Compared with the existing methods, the authors arrange and partition the variables and use a circular permutation approach to transform regularization into a measure of distribution similarity, thereby enhancing the consistency between different perspectives. The experimental results on several real-world datasets demonstrate the effectiveness of the proposed model.", "questions": "None", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper is well-motivated since missing is a common and significant problem in real-world multi-view learning. The modelling of the missing problem in this paper is very well designed and the method of applying permutations and segmentations in the latent space is very novel.\n2. The paper provides a well-structured overview that is easy for the reader to understand. In addition, implementation details of the selected technology are presented in detail.\n3. The paper provides comprehensive experimental results to validate the effectiveness of the proposed method. The model is tested on seven different datasets and compared with several SOTA methods to demonstrate its usefulness and robustness", "weaknesses": "1. The complexity of the proposed method is not adequately discussed. It would be helpful to compare the computation cost of the proposed method to the baselines.\n2. This paper assumes that the first k dimensions of z capture information common to all views, so how is it set on different datasets?\n3. This paper only shows the results generated on the PolyMNIST and MVShapeNet datasets, where the views are all RGB type. How are the results generated on datasets with other types of views, such as the CUB Dataset?\n4. For the experimental results, why are the experimental results you provided lower than the original papers? For example, DVIMC on the Scene 15 dataset with missing rates η=0.1 is lower than the published papers. In addition, how were the experimental results of Completer obtained? The original Completer is proposed for two view data which cannot be applied on the datasets you exploited in the paper directly.\n5. The convergence analysis can be added in the experiment, which can be adopted to better the loss function.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1OtTXEx7np", "reviewer": "Reviewer_7w7m", "summary": "In this paper, the authors design a multi-view permutation of variational auto-ecoders for incomplete multi-view clustering, which termed MVP. The variational auto-encoder is used for extracting the invariant features.  Randomly reoder variables are used for cross-view generation.", "questions": "Please see weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Extensive experiments have demonstrated the effectiveness of the methodology designed in the paper.\n\n2. The paper is well-organized.\n\n3. The motivation is clearly described.", "weaknesses": "1. The use of VAE for invariant feature learning has been extensively studied [1,2,3, 4]. Please analyze the differences.\n\n2. The paper does not provide the code, yet the results in Table 2 show significant performance of the proposed method. The authenticity and fairness of the experiments are questionable. Therefore, please release the code during the rebuttal process, as this will be a key criterion in my evaluation.\n\n3. The construction details of the PolyMNIST and MVShapeNet datasets in line 406 are unclear. Is using new datasets to test previous methods fair?\n\n4. How can the impact of randomness introduced by random padding, as mentioned in line 20, be mitigated?\n\n\n5. The novelty needs to be further clarified. The use of Variational Autoencoders and ELBO has been widely studied in the IMVC field.\n\n[1] Xu G, Wen J, Liu C, et al. Deep Variational Incomplete Multi-View Clustering: Exploring Shared Clustering Structures[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(14): 16147-16155.\n\n[2] Cai H, Huang W, Yang S, et al. Realize Generative Yet Complete Latent Representation for Incomplete Multi-View Learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.\n\n[3]Chen M, Huang H, Li Q. Towards Robust Uncertainty-Aware Incomplete Multi-View Classification[J]. arXiv preprint arXiv:2409.06270, 2024.\n\n[4] Xu J, Ren Y, Tang H, et al. Multi-VAE: Learning disentangled view-common and view-peculiar visual representations for multi-view clustering[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 9234-9243.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fHEnoKTkmW", "reviewer": "Reviewer_yAd5", "summary": "This paper introduces the Multi-View Permutation of Variational Autoencoders (MVP) for incomplete multi-view representation learning. By leveraging cyclic permutations of posteriors, MVP enhances inter-view consistency and infers missing views effectively. Key elements include partitioning variables for view invariance, deriving an Evidence Lower Bound (ELBO) for optimization, and implementing an informational prior through cyclic permutation to align distributions across views. Experimental validation across multiple datasets and missing data scenarios shows MVP’s superiority over existing methods in clustering and generation tasks, particularly under high missing rates.", "questions": "See Major problems in Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- MVP introduces a cyclic permutation approach to VAEs, leveraging view-invariant transformations in the latent space to address the challenges posed by incomplete multi-view data. This method successfully captures inter-view relationships by modeling correspondences and creating a more robust latent space, which significantly enhances the ability to infer missing views and aggregate information effectively.\n- The paper provides rigorous theoretical analysis and proofs.\n-  The extensive experiments, particularly in scenarios with high missing data ratios, provides compelling evidence of MVP’s ability to handle incomplete data effectively. The authors not only validate the approach across different missing ratios but also include quantitative metrics and visualizations to showcase MVP’s advantage over competing models.", "weaknesses": "##### Major\n\n- \"Even if we reorder the variables within each column of Z0, as demonstrated in the transition from Z0 to Z1in Figure 1, the underlying semantic information remains invariant\"? The explanation of “underlying semantic information” remains unclear. A detailed clarification on what specific semantic information remains invariant during these transformations would strengthen the paper.\n- MVP’s performance could be sensitive to cyclic permutation settings and regularization parameters, as they directly influence view consistency and the similarity measure. A deeper sensitivity analysis of these parameters would provide insight into MVP’s stability and adaptability.\n- The cyclic permutation technique appears conceptually similar to disturbance and re-alignment strategies as proposed in *Partially View-Aligned Clustering* (NeurIPS ’20). A discussion of distinctions and similarities between MVP and this approach would benefit readers.\n- Network architectures used for MVP are not fully detailed in the paper, limiting reproducibility and making it challenging for readers to understand the baseline structures supporting MVP’s performance.\n\n##### Minor\n\n- Highlighting the Single-view Partition and Complete-view Partition in Figure 1 would improve clarity.\n- The transition from Z_0 to Z_1 in Figure 1 is difficult to follow; linking this to Section A.2 might assist readers.\n- It is not easy to distinguish the views with the samples in Fig.1b. It would be better to use different number of views and samples.\n- The reference to “The second term” in Line 303 would be clearer if accompanied by Equation 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)