CLAP: Collaborative Adaptation for Patchwork Learning
Authors: Sen Cui
Abudukelimu Wuerkaixi
Weishen Pan
Jian Liang
Lei Fang
Changshui Zhang
Fei Wang
Venue: ICLR 2024 spotlight (2024)
Keywords: Patchwork learning
robustness
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: This paper studies the missing modality imputation from the patchwork.

Abstract: In this paper, we investigate a new practical learning scenario, where the data distributed in different sources/clients are typically generated with various modalities. Existing research on learning from multi-source data mostly assume that each client owns the data of all modalities, which may largely limit its practicability. In light of the expensiveness and sparsity of multimodal data, we propose patchwork learning to jointly learn from fragmented multimodal data in distributed clients. Considering the concerns on data privacy, patchwork learning aims to impute incomplete multimodal data for diverse downstream tasks without accessing the raw data directly. Local clients could miss different modality combinations. Due to the statistical heterogeneity induced by non-i.i.d. data, the imputation is more challenging since the learned dependencies fail to adapt to the imputation of other clients. In this paper, we provide a novel imputation framework to tackle modality combination heterogeneity and statistical heterogeneity simultaneously, called ``collaborative adaptation''. In particular, for two observed modality combinations from two clients, we learn the transformations between their maximal intersection and other modalities by proposing a novel ELBO. We improve the worst-performing required transformations through a Pareto min-max optimization framework. In extensive experiments, we demonstrate the superiority of the proposed method compared to existing related methods on benchmark data sets and a real-world clinical data set.

Published as a conference paper at ICLR 2024
CLAP: COLLABORATIVE ADAPTATION FOR PATCH- WORK LEARNING
Sen Cui
Wuerkaixi Abudukelimu'
Weishen Pan?
Jian Liang
Lei Fang* Changshui Zhang*1 Fei Wang
Institute for Artificial Intelligence, Tsinghua University (THUAI)
Beijing National Research Center for Information Science and Technology (BNRist)
Department of Automation, Tsinghua University, Beijing, P.R.Chins
2 Depparmeent of Population Health Sciences,
Weill Cornell Medical College, Cornell University, New York, USA
3 Independent Researcher
4 DataCanvas Technology Co., Ltd.
ABSTRACT
this paper, we investigate a new practical learning scenario, where the data dis- tributed in different sources/clients are typically generated with various modalities. Existing research on learning from multi-source data mostly assume that each client owns the data of all modalities, which may largely limit its practicability. In light of the expensiveness and sparsity of multimodal data, we propose "patchwork learning" jointly learn from fragmented multimodal data in distributed clients. Considering the concerns on data privacy, patchwork learning aims to impute incomplete multimodal data for diverse downstream tasks without accessing the raw data directly. Local clients could miss different modality combinations. Due the statistical heterogeneity induced by non-i.i.d. data, the imputation is more challenging since the learned dependencies fail to adapt to the imputation of other clients. this paper, we provide novel imputation framework to tackle modality combination heterogeneity and statistical heterogeneity simultaneously, called "col- laborative adaptation". In particular, for two observed modality combinations from two clients, we learn the transformations between their maximal intersection and other modalities by proposing a novel ELBO. We improve the worst-performin required transformations through a Pareto min-max optimization framework. In extensive experiments, we demonstrate the superiority of the proposed method compared existing related methods benchmark data and real-world clinical data set.
INTRODUCTION
Multi-modal learning (Ngiam al., [001I) which refers to the paradigm of learning from the data with multiple modalities, has gained growing interest for its practical significance in facilitating real-world applications (Valindria et al. 2018 Yang al. (022). reality, multimodal data are typically generated from various users/clients with private information. Given the costly and sparse nature of such multimodal data, an interesting problem is how learn from distributed multimodal data without sacrificing privacy.
One challenge of learning from distributed multimodal data modality heterogeneity. In many applications, local data on clients are generated with various modalities. More importantly, it is usually hard to require all modalities to exist each client in reality. One example collaborative learning in the clinical research network (CRN) involving multiple hospitals (Fleurence et al.) 2014). where each hospital can be viewed as a local client. If we want to collaboratively build a mortality prediction model for COVID-19 patients, different hospitals can have various medical records due to
"These authory contributed equally to this work. "Corresponding authors

---

Published as a conference paper at ICLR 2024
different diagnoses and treatments. Hence, it becomes crucial to enhance scalability by dealing with different combinations of modalities.
Another challenge, which has been studied in federated learning (FLMMcMahhn al., 2017). is statistical heterogeneity (i.c., non-identically distributed data from local clients). As the remote data sources could be gathered from various users in reality, there is research showing that global model can suffer severe performance degradation when the local data distributions drift dramatically (Deng al. 2020).
DAD
v Y
(a) distributed multimodal learning
(b) patchwork learning
Figure I: Illustrations of the two learning paradigms. Gray grids denote the observed modalities and white grids are missing modalities. Y is the label to be predicted. (a). distributed multimodal learning; the data in local clients are multimodal and all modalities are observed. (b). patchwork learning; different clients have different multimodal data. The goal is to jointly learn from the "patchwork" to impute all missing modalities without direct access to the raw data.
With the aforementioned considerations, we propose general multimodal learning problem in practical setting, called patchwork learning (Rajendran et al.) 2023), where data from distributed clients have different modality combinations. Instead of learning a prediction model using complete multimodal data in existing distributed multimodal learning (Zhao et al. 2021 Xiong et al. 2022). where each client has the data of all modalities shown in Figure [ (a), patchwork learning aims to learn from incomplete data shown in Figure (b). Each client owns one or multiple modali- ties. Different clients have different modality combinations, which introduces additional difficulty when collaboratively learning models. Our goal is to impute the unobserved modalities to provide convenience for various downstream tasks without direct access to the raw data.
Modality imputation from observed data becomes even more challenging when it associates with statistical heterogeneity. As the missing modalities of a specific target client are unavailable during training, the imputation process depends on the learned dependencies among modalities from other clients, known as source clients. This may bring bias because of the statistical heterogeneity. To tackle the statistical heterogeneity and modality heterogeneity simultaneously, we propose novel imputation framework called "collaborative adaptation" (CLAP). For statistical heterogeneity, we propose to learn common modality VAE for each modality for all clients. We mitigate the distribution discrepancy of the generated representations from modality VAEs by balancing the distribution distance of all clients. For different modality combinations in local clients, we propose clinnt-adaptation VAE (CA-VAE) for approximating the required dependencies between the maximal intersection of observed modalities in source and target clients, which maximally utilizes the observed modalities for the missing modality imputation.
We empirically evaluate our framework on a series of benchmark data sets, and a real-world electronic health record (EHR) data repository eICU, which includes patient EHR data in ICU from multiple hospitals. The results show our framework outperforms existing relevant methods in the modality imputation tasks and classification downstream tasks. The source codes of our framework are made publicly available at
http:ggtthuu..co.ooooeeeoe com/ wtt../////ccccn//c.n..
RELATED WORK
Learning from diverse modalities has the potential to improve the performance of machine learning algorithms as they could provide complementary information. For example, the visual appear- ance and tactile impression of an object could converge on a more invariant abstract characteriza- tion (Yildirim 2014). While there research aiming bridge modalities with fully-supervised
2

---

Published as a conference paper at ICLR 2024
complete data (Pandey & Dukkipati 2017: Ledig et al. 2017). important problem is that multi- modal data is usually expensive and sparse Wu Goodman, 2018). To overcome this obstacle, Wu al. firstly propose MVAE(WWu & Goodman 2018) to yield generalizable representations to capture the relationship across modalities in a weakly supervised setting, where there is only subgroup of examples with all modalities present. MVAE flexibly handles missing data in scalable way for arbitrary multiple views. Recently, several variants of MVAE are proposed to strengthen the generation quality and the coherence between modalities (Shi al. 2019; Sutter et al., 2020b a Hwang et al. 202T Daunhawer et al. (021). As existing multimodal representation learning methods were proposed for traditional learning scenario, there no distribution discrepancies and privacy concerns, which limit their practicability in patchwork learning scenario.
The concept of federated learning (FL. was first proposed by McMahan et al. McMahan al. 2017). Recently, statistical heterogeneity in FL has drawn much attention in machine learning community and a variety of efforts have been made to tackle There are research proposing to
calibrate local objectives by regulating the deviation between local models and the global model (Li
ct al. 2021 Smith et al. 2017). D. For example, Smith et al. aim to model the information-sharing
mechanism between local objectives by multi-task regularization(SSiite et al.) (017). Another line of research focus on a better trade-off between local and global learning (Liang et al. 2020
Collins al. 202T Khodak al. 2019 Fallah al., 2020). For example, some studies assume the heterogeneity comes from label shift and propose to use a multi-head network to model common feature embedding (Collins al., (02I). In existing work, usually assumed that data from all clients share a common feature space. Besides statistical heterogeneity, more universal problem is how to learn when different clients own diverse data modalities.
Considering the diverse data modalities in real life, there are few research investigating the task of distributed multi-modal learning, i.e., collaboratively learning models on distributed sources containing multimodal data [Xiong et al. [022] Zhao et al. (202).. In particular, Zhao er al. present a multimodal FedAvg algorithm aggregate the representation extracted by local autoencoders for downstream (Zhoo et al. (202T). Xiong ef al. propose co-attention mechanism (Xiong et al.) 2022) to fuse different modalities under the assumption that the multimodal data are completely observed shown in Figure (a). In this work, we are interested in addressing more practical and upstream learning task: how to impute a patchwork when local clients have various incomplete multimodal data.
M
M2
M3
M4
M1
M2
Mg
M4
M
Mi
C
M2
c
Ms
c2
33
Mi
c3
Ma
(a) connected graph
(b) unconnected graph
Figure 2: Illustrations of Assumption 2 (a). connected sub- graph; {M1 MM2 is connected sub- graph, because al has both modalities. Similarly, [M2, M2} (fM2 Ma)) } is a connected sub graph because of c² (c.). (b). connected sub-graph; M1. M3 and M4 are isolated points.
3 NOTATIONS AND PROBLEM DEFINITION
3.1 NOTATIONS
Suppose there are N clients with K modalities in patchwork. The input space X is not shared across all clients. Different local clients can have different subsets of all modalities. An example is shown in Figure bb) X (M, M3, M4} but X (Ma, Ms). ). We use x' { [mf. mi, m'x denote the sample variable of X where m² is the j-th modality in the i-th client. For example, x² {m] m3, } the sample variable of x shown in Figure (b). The output space y denotes the missing modalities. Note that the missing modalities Y are not accessible during training.
3

---

Published as a conference paper at ICLR 2024
3.2 PATCHWORK LEARNING
Suppose each modality Mp is node of graph, and there is an undirected edge from Mp to M3 if there exists a client e² owning both modalities. An edge between Mp and My means that we could model the dependency between them.
To enable the possibility of the imputation of blank grids in patchwork, we propose the following assumptions.
Assumption 1. For the row in the patchwork, each client owns the data with at least one modality.
Assumption 1 means that the observed modalities cannot be an empty set for each client.
Assumption 2. There exists connected sub- sub-graph in the patchwork.
We give an example shown in Figure 2(a) When there exist sub-graphs in the patchwork, it guarantees the dependency between missing modalities Mp E Y and the observed modalities X could be learned from other clients.
missing modality data my in the client et imputed relying on the learned dependency between Mj and the observed modalities X and the completed imputation of the patchwork means that all missing modalities could be imputed. We have the following proposition.
Proposition 1. (informal, proof in Appendix) Assumption 7 and a connected graph are the necessary and sufficient conditions for the completed imputation of the patchwork.
Since Y is not available during training, the imputation of the target client e relies on the dependen- cies learned from other clients. general, patchwork learning faces the following challenges that need to be addressed:
statistical heterogeneity; the multimodal data of local clients are typically non-i.i.d The model may fail adapt the learned dependencies target clients;
modality combination heterogeneity; local clients can have various modality combinations.
The learned dependencies from X is hard to be used for other clients with different combinations X);
modality com combination vulnerability; the learned imputation method could be vulnerable to the modality combinations and the imputation quality significantly varies for two similar combinations;
4 METHODOLOGY
4.1 PRELIMINARIES: VARIATIONAL AUTOENCODER (VAE), PRODUCT-OF-EXPERTS (PoE)
AND PARETO MIN-MAN MAX OPTIMIZATION
VAE. VAE (Kingma& & Welling 2013) is proposed as a generative model, which assumes the latent variable is sampled from a prior standard normal distribution p(2) := N NO, 1). VAE consists of an encoder (=(2(z) with the parameters 0, and decoder Po (x]=)) with the parameters ø) During training, VAE maximizes log p(x) by optimizing the evidence lower bound (ELBO) formulated as
max ELBÖ(z), where ELBO(r) (x)()))) (=(1)) DK (=0()))) 0,0
(1)
where D (((2))),p(())) denotes the Kullback-Leibler divergence between go and p(z).
PoE. Wu et al. propose PoE(WW & Goodman 2018 for multimodal variational autoencoder (MVAE).
MVAE assumes that the multimodal data are conditionally independently generated from a joint latent representation z. Suppose there are modalities X (M1, MK,..,Kkk). The generative model has the following factorized form:
p (m1,
2) (m=)p(m11z) p (m²]=) ...p (mxle 2).
According to Eq.2) Wu al. propose to employ PoE aggregate the representations of multiple modalities. For any modality combination X c X, PoE obtains the aggregated representation (<2xx*) as follows.
(x|2²) = Im e (2 p(2)
(3)
I I. p(2)
4

---

Published as a conference paper at ICLR 2024
where g(2 mm and p(z) are both Gaussian, and the product and quotient distributions are solvable in closed form!
Pareto min-max optimization. Suppose (h) 14 ( (/)] represents the loss vector on N learning tasks with hypothesis h, is Pareto solution there is no hypothesis h that dominates h: h x h, i.e.
Ah' E H. S.L. Vi 4 (M) - 4(h)) and Aj 1 (h') < 1,)h)
A Pareto Min-max solution means that Pareto solution achieves the lowest loss on the maximal value in (h)). To obtain such a solution, there are research proposing to use the entropy function method achieve min-max optimality and then continue optimize all losses with projected gradient for Pareto optimality (Cui et al.) (02T), which indicates that each objective cannot be further optimized without degrading others. Pareto min-max optimization maximizes the utility of the worst-performing objective.
M1
M2
M3
M4
M1
A6.
Mr
E
Mr
C
Parsto (- ELBO(m)))
2²
(a) two clients in the patchwork
modality VAE
M2 Ms
M4
Ms
Mj < (M2. Ma)
E
10
PoE
BLBO(
{M Ma + Mr
(b) the optimal transformations
(d) client-udaptation VAE.
Figure 3: (a) and (b) illustrate the identification of the required dependencies for two clients. (a) the patchwork with two clients, where colored grids represent observed modalities and blank grids represent missing modalities: (b) the identified dependencies for the imputation.
(c) and (d) illustrate our implementation framework CLAP. (c). modality VAE: each modality learns a VAE for obtaining the representation. The VAEs trained by Pareto min-max optimization.
(d). client-adaptation VAE (CA-VAE): CA-VA shares the encoder (E) with modality VAE and has an extra decoder D
4.2
AN OVERVIEW OF THE REALIZATION OF CLAP
From Figure 3 (c) and (d), our framework consists of modality VAE and client-adaptation VAE, which synergistically address the three key challenges stated in Sec. 3.2
Modality VAF. From Figure 3 (c). we learn VAE for each modality M, to obtain the representation Zi, which has the encoder E = {Ei, .... Ex } and the decoder D
(D
D }.
For the training of each modality VAE (E, and D, the data of the i-th modality M, come from multiple clients (e.g., m ...., m, and are non-i.i.d. An averaged loss on all data may harm particular clients. The learned modality VAE could generate biased representation during imputation.
As shown in Figure (c), to mitigate the statistical heterogeneity of the learned representation, share the encoder for clients. Meanwhile, we propose to balance the KL divergence of the learned representations by maximizing the client with the maximal KL divergence, that each modality VAE encodes various data distributions of clients to a common latent space, which we call vertical Pareto min-max optimization.
Client-Adappationn VAE (CA-VAE). From Figure 3 (d), we learn a CA-VAE to model the de- pendencies among modalities, which shares the parameters of the encoder with modality VAE E : {E,,... Ex} and has an additional decoder DC := [D ..., D& }. By using the two de- coders (DM and _) separately, we relieve the potential conflicts when rebuilding the unimodal data modality VAEs and multimodal data in CA-VAE
Mooe details about the computation of (=)x*) could be found in Wu & Goodman 2018
5

---

Published as a conference paper at ICLR 2024
To model the dependencies between modalities, we use the subset of the observed modalities X X' to infer other modalities and X itself. Specifically, the subsets are identified as the intersection of the observed modalities between X and other clients. For the example shown in Figure 3 (d), we use the modality combination X n X = {M, M2} to rebuild X3. Then we use Pol. to aggregate the representations 21 he, (mi) and 22 ho, (m²), where ho, denotes the encoder E in Figure (d). The aggregated : is input to DC to rebuild all observed modalities X
The optimization of CA-VARE While poor transformation learned from one client could worsen the imputation of multiple clients, to overcome the combination vulnerability, we propose horizontal Pareto min-max optimization to balance the dependency learning across the diverse combinations.
The training of CLAP. We train CLAP following the framework in federated learning (McMahan al. 2017). Only the sum of the gradients allowed to be transferred between the server and clients. We summarize the algorithm for the training of CLAP in Algorithm I in Appendix. The Pareto min-max optimization and other implementation details could also be found in Appendix because of the page limit.
4.3 MODALITY VAE AND VERTICAL PARETO MIN-MAX OPTIMIZATION
Multimodal data {=m]]==}=- >
i=1 mostly have different dimensions when computing. To enhance the scalability of the imputation framework, learn a modality VAE for each modality, which encodes mý to a latent representation = = ho, (m]) with fixed dimension, i.e.,
min - -R (s1m)) [log 0g.0g
Po, Pe, (m/1l=)] DE (go, (=lm))) where i = 1,2, N.
(5)
Statistical heterogeneity and vertical Pareto min-max optimization. From the equation 5 the KL divergence measures the difference between the distribution of the learned representations 4o, (=|m²) and the spherical Gaussian p(z). The presence of diverse KL divergence among local clients suggests a significant statistical heterogeneity.
We propose vertical Pareto min-max optimization to address the statistical heterogeneity. Specifically, since there is a conflict when a global model minimizes all KL divergences, minimizing the maximal KL divergence uniform KI. divergences across all clients. Thus, the maximum KI. divergence is an indicator of the uniformity of the KL divergences. We propose enhance the worst client with minimal ELBO to minimize the maximal
the worst client can increase the KL divergences of other clients, leading to more
KL divergence.
Pareto min max
- (E (x1m)) [logro (m/) [c]] - DKL (xlm)), p(=))), mij is observable. (6)
4.4 CLIENT-APATATION VAE AND HORIZONTAL PARETO MIN-MAX OPTIMIZATION
Modality combination heterogeneity and CA-VAE. Given patchwork to be completed, the trainable transformations are defined. For client x' with & observed modalities, there are 2 subsets and 2 corresponding transformations"" 0 While learning all transformations could have an exponential computing complexity, natural problem is how to identify the optimal transformations efficiently.
Identify the required dependencies on two clients. To answer the above problem, we first consider p there are two clients c² and c² with the modality combinations X and X² shown in Figure 3 (a).
Both clients learn the dependencies on their observed modalities for the imputation of the other. From Figure 3 (a), to impute mř for e², c² could learn the transformation M, + M, (or M3 + M,2 using its data x However, the optimal dependencies for c² may be [M2, M2} > Mj, which uses the intersection of X and X to infer other modalities. By using the intersection, the dependencies learned from X maximize the utilization of the observed data in X when imputing the missing modalities for c².
From the above analysis, the identified optimal dependencies learned by transformations shown in Figure 3 (b) are as follows.
for c:: x' x' x2 for e: x2 x X2 x
(7)
We also rebuild X for the rationality of the encoder.
For each subset x C X", there corresponding transformation X X' X'1X" 1x

---

Published as a conference paper at ICLR 2024
where X X denotes X removes the elements in X². For two clients with different modality combinations, the identified dependencies mitigate the modality combination heterogeneity by using all common modalities, which is the intersection X X This improves the transferability of the learned dependencies when applied to the target clients with different combinations.
Identify the required dependencies multiple clients. For multiple clients, we identify the dependencies to be modeled according to Eq.(7) for each pair of clients. The required dependencies be modeled by each client et formulated follows.
for c² X + X X2, Vj E (1,
(8)
Learning the identified dependencies by CA-VAE. We consider the multimodal data {m²}
in each client is generated by an unknown random process with a latent representation To learn the dependencies formulated in Eq.(8). we propose to approximate X using X n X', We use xn to denote the subset of the variable x², which only contains the modality combination X n X We optimize the KL-divergence between the learned and the true posterior distribution (p (=|=²) ) to model the dependencies in (8.
min DKL (g0 (2)2*Ti) 1lp (z(z"))
(9)
Since p (=|=²) in Eq. 9) is intractable, we optimize Eq. 9 by giving the following proposition.
Proposition 2. (proof in Appendix.) Eq. Eq.( formulates an appropriate ELBO, and minimizing Eq. 10p minimizes the Kl-divergence formulated formullatte in Eq.. 9 ie.,
Cj (0, (0,);='== == = (Ego(() [log [x)() De (4o (z)a [Ip(z)))
(10)
where 0 denotes the parameters of the encoder E and denotes the parameters of the decoder D
shown in Figure 3 (d). go (2(="n0) ) is calculated using PoE according to Eq. p(2) is the standard normal distribution.
For client c², we approximate the dependencies in Eq.(8) by optimizing the sum of the KI.-divergence between g( g(2(x=()) and p(z[==),, where j is the selected other clients with different modality conbina- tions (X * X
min Li dex' Ej (0. 0,0
(11)
Modality combination vulnerability and horizontal Pareto min-max optimization. While a poor transformation learned from one client could worsen the imputation of multiple clients. We propose to mitigate the worst-case effect by Pareto min-max max optimization, i.c.
Pareto min max L (0, 0b xz)").
(12)
6.0
EXPERIMENTS
Baselines. As we tackle new problem and there is no existing method that focuses on it, we modify the training scheme of existing methods on the modality imputation using an existing distributed learning scheme (e.g.. FedAvg (McMahan al. 2017)) as ours. In particular, in each communication round, each client trains its local model using the observed modalities, then the server averages the model weights to update the global model. The baselines include MVAE (Wu & Goodman) 2018).
MMVAE (Shi et al. 2019) and MoPoE-Vas VAE (Sutter et al. 2020b).
3
f
9
3
9
3
7
0
7
3
9
>
7
>
3
7
>
3
7
0 7
3
7
3
3
3
3
3
7
7
O
7
3 7
O
7
3 7
O 7
3 7
MVAE
MMAAE
MoPoR
CLAP
Figure 4: Imputation results of methods on PolyMNIST. Each column represents a sample. Red boxes indicate the missing modalities.
Datasets.
Following the work (Sutter et al., 2020b), we evaluate our method with baselines on benchmark datasets with various modalities, including PolyMNIST, MNIST-SVIN-TEXT.. CelebA
7

---

Published as a conference paper at ICLR 2024
and CUB. More importantly, the practicability of our method is validated real-world clinical distributed dataset eICU (Pollard et al., 2018).
Evaluation metrics. We evaluate the performance of all methods from two aspects. 1. generation quality; the models take the observed modalities as inputs to generate missing modalities. We assess the coherence of generated modalities with pre-trained classifiers to evaluate whether the imputed samples could be classified correctly. Meanwhile, we also approximate the log- log-likelihoods of the output data reflect the generation quality. For the eICU dataset, which is tabular data and has many modalities, we report the mean square error (MSE) between the generated results and the ground truth. representation quality; the latent space of the data is supposed to be informative. We build the downstream tasks by using the learned classifiers to measure the quality of learned representations. All experiments are run 5 times to calculate the average results with stds. More experimental results, more ablation studies and discussions could be found in Appendix.
5.1 EXPERIMENTS ON POLYMNIST
The PolyMNIST dataset (Sutter al., 2020b) is synthesized with the MNIST dataset. Each sample contains 5 image modalities with different backgrounds and styles with the same digit label. There 60,000 samples in the training and 10,000 samples in the test set. We create the distributed environment by partitioning the dataset into clients. Each client contains 6 classes of samples. Different clients involve different classes of data. In each client, we mask less than 2 modalities random.
Table I: Imputation coherence accuracy (%) of 5 modalities the PolyMNIST dataset.
Model
Mi
M2
Ms
M4
Ma
MVAE
22.1
19.3
15.2
68.7+122
19.6410
MMVAE
36.3+0.1
60.5
61.3+1.0
64.442.
34,4
MoPoE
38.540.6
69.421.4
72.421.4
75.720.5
45.621.0
CLAP
45.2+0.2
69.4g1.1
74.5
79.3
50.5
Table 2: Linear classification accuracy (%) the representation of the modality subsets the PolyMNIST dataset.
Model
{
{Ma, Ms}
(M2, M4 Ms}
Average
MVAE
81.410.4
75.7
76.720.2
77.9
MMVAE
86,9+1.1
78.9
83.2+1.u
83.0
MoPoE
79.240.4
80.1
97.9
85.7
CLAP
87.640.2
91.8+b.4
95.9
91.940.4
The experimental results on PolyMNIST dataset are shown in Table and 2 From Table MVAE has worse imputation coherence accuracy because it fails to learn the dependencies among modali- ties. MoPoB achieves better imputation coherence than other baselines as it learns all transformations among modalities. CLAP achieves the best imputation coherence as it models the required depen- dencies for imputation, which mitigates the potential modality combination vulnerability. For the quality of the learned representation shown in Table 2 CLAP still maintains the best accuracy as our method addresses the heterogeneity by learning a common latent representation for all clients. We visualize the generated result in Figure From Figure 4 the baseline methods are easily confused by the images '7' and '9', and CLAP correctly impute the missing modalities.
5.2 EXPERIMENTS ON MORE COMPLICATED DATASET BIMODAL CELEBA
Bimodal CelebA The Bimodal CelebA dataset (Sttter et al. 2020a) is extended from CelebA dataset (Liu et al. 2015). Each face image is attached with a textual modality. is generated with dataset, the face
the 40 attributes describing the face. Compared with the images the Bimodal CelebA dataset are more complex. Also, the text strings are longer and more complicated. We split the dataset into 5 clients. For the setting of statistical heterogeneity, we set different portions of Male samples in local clients.
The results of imputation coherence and latent space classification shown in Table CLAP sur- passes all baselines and achieves the highest imputation coherence and latent space classification accuracy.
8

---

Published as a conference paper at ICLR 2024
Table 3: Imputation coherence and latent space classification results of 2 modalities / (Image) and T (Text) on the Bimodal CelebA dataset. The mean average precision (mAP) (%) is averaged over all 40 attributes.
Coherence
Latent Classification
Model
T
T
MVAE
35.5
34.7
16.0_a.3
7.1
MMVAE
34.4+0.2
32.7+0.3
43.7-0.2
41.5+0.1
MoPoE
34.940.3
33.7.0.5
43.8-a.1
41.2+0.3
CLAP
37.4
35.210.1
44.2_0.1
45.4
Table 4: Results on eICU dataset. MSE and latent classification AUC (%) are averaged over 13 modalities and certain subsets. The log- log-likelihoods of joint distribution X are reported.
Model
MSE
Latent Classification
Likelihood
MVAE
0.0661
0002 0002
S6.3
+0.5
MMVAE
0.0681 0000004
23.0
-340.44..
MoPoE
CLAP 0.0653 0002
62.3
.3.0.210.7
5.3 EXPERIMENTS ON A REAL-WORLD CLINICAL DATASET BICU
To further confirm the practicability CLAP, use real- world clinical dataset eICU (Pollard et al. 2018). eICU contains the records of the patients to ICUs with hospital information, and each hospital a client (Cui et al. 2022). The data is preprocessed as in (Sheikhalishahi al., (019). We select 13 features (e.g., FiO2, 02 Saturation, etc.) as 13 modalities. Each instance has the 48-hour monitoring data of the features. The label is binary indicating whether the patient is alive. The experimental results are shown in Table 4 in which MoPoE
not work the eICU dataset due to the vast number (2 of modality subsets. We use AUC to evaluate the classification results due to the severe label imbalance (more than 90% samples have negative labels). From Table CLAP outperforms baselines in all three metrics.
5.4 ABLATION STUDIES
Vertical and horizontal Pareto min-max optimization. We conduct ablation studies on PolyMNIST to validate the effect of Pareto min-max optimization. In particular, replace Pareto min- max optimization in 6) and Eq. (2). The averaged results over all 5 modalities shown Table where w/o both denotes that both losses replaced with the averaged losses. w/o modality means that we only replace vertical Pareto min-max optimization in Eq. with the averaged loss, and w/o client means that we only replace horizontal Pareto min- max optimization in Eq. 66 with the averaged loss. The results are shown in Table 5 The ablation studies on more clients could be found in Appendix.
the averaged loss
min-max
Table Ablation study on the PolyMNIST dataset. Coherence accuracy, log likelihood and latent classification accuracy (%).
Method w/o both
Coherence
Likelihood
Latent
56.440.2
-667241
53.3
w/o modality
57.3
-6642+3
54.4
w/o client
56.9+0.1
-6633+1
55.1
CLAP 58.7+0.2
-6603+4
56.640.1
CONCLUSION
In this paper, we propose a practical multimodal learning problem. Given an incomplete patchwork, we propose scalable framework, CLAP, to achieve efficient and valid imputation by jointly learning from the patchwork vertically and horizontally without direct access to the raw data. Em pirical results both benchmark and real-world medical data sets demonstrated the effectiveness, superiority, and practicability of our proposed method. We hope this work opens venues for future research, for example, large models in multimodal learning, unsupervised multimodal learning, etc.

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGMENTS'
This work is funded by the Natural Science Fundation of China(NSFC. No. 62176132) and the Guoqiang Institute of Tsinghua University, with Grant No. 2020GQG0005.
REFERENCES
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared represen- tations for personalized federated learning. In International Conference on Machine Learning, pp. 2089-2099 PMLR. 2021.
Sen Cui, Weishen Pan, Jian Liang, Changshui Zhang, and Fei Wang. Addressing algorithmic disparity and performance inconsistency in federated learning. Advances in Neural Information Processing Systems, 34:26091-26102, 2021.
Sen Cui, Jian Liang, Weishen Pan, Kun Chen, Changshui Zhang, and Fei Wang. Collaboration equilibrium in federated learning. In Proceedings the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 241-251, 2022.
Imant Daunhawer, Thomas M. Sutter, Ricards Marcinkevics, and Julia E. Vogt. Self-supervised disentanglement of modality sppecifi and shared factors improves multimodal generative models. In Zeynep Akata, Andreas Geiger, and Torsten Sattler (eds.), Pattern Recognition 42nd DAGM German Conference, DAGM GCPR 2020, Tübingen, Germany, September 28 October 1, 2020, Proceedings, volume 12544 of Lecture Notes in Computer Science, pp. 459-473. Springer, 2020.
Imant Daunhawer, Thomas M Sutter, Kieran Chin-Cheong, Emanuele Palumbo, and Julia E Vogt. On the limitations of multimodal vaes. arXiv preprint arXiv:2110.04121 2021.
Li Deng. The MNIST database of handwritten digit images for machine learning research [best of the web]. JEEE Signal Process. Mag., 29(6): 141-142, 2012.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning, arXiv preprint 2ttt::2003.1346111 2020.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, 33:3557-3568, 2020.
Rachael L Fleurence, Lesley H Curtis, Robert M Califf, Richard Platt, Joe V Selby, and Jeffrey S Brown. Launching pcornet, national patient-centered clinical research network. Journal of the American Medical Informatics Association, 21(4):578-55 2014.
HyeongJoo Hwang, Geon-Hyeong Kim, Seunghoon Hong, and Kee-Eung Kim. Multi-view represen- tation learning via total correlation objective. Advances in Neural Information Processing Systems, 34, 2021.
Mikhail Khodak, Maria-Florina Balcan, and Ameet Talwalkar. Adaptive gradient-based meta-learning methods. arXiv preprint 1xXi::996.002/11 2019.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114, 2013.
arXiv preprint
Christian Ledig, Lucas Theis, Ferene Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2017.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pp. 6357-6366 6368, PMLR, 2021.
10

---

Published as a conference paper at ICLR 2024
Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B Allen, Randy P Auerbach, David Brent, Ruslan Salakhutdinov, and Louis-PPilippe Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.01523, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 3730-3736 IEEE Computer Society, 2015.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effiiiennn learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning, 2011.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lec, and Andrew Y Ng. Multi- modal deep learning. In Proceedings of the 28th international conference on machine learning (ICML-II), pp. 689-696, 2011.
Gaurav Pandey and Ambedkar Dukkipati. Variational methods for conditional multimodal deep learning. In 2017 International Joint Conference on Neural Networks (LJCNN), pp. 308-315. IEEE, 2017.
Tom J Pollard, Alistair EW Johnson, Jesse D Raffa, Leo A Celi, Roger G Mark, and Omar Badawi. The eicu collaborative research database, a freely available multi-center database for critical care research. Scientific data, 5(1):1-13, 2018.
Suraj Rajendran, Weishen Pan, Mert R Sabuncu, Jiayu Zhou, and Fei Wang. Patchwork learning: paradigm towards integrative analysis across diverse biomedical data sources. arXiv preprint 1.Xiv:2305.06217, 2023.
Seyedmostafa Sheikhalishahi, Vevake Balaraman, and Venet Osmani. Benchmarking machine learning models eicu critical care dataset. arXiv preprint arXiv: 1cXiv:1910000966 2019.
Yuge Shi, Brooks Paige, Philip Torr, et al. Variational mixture-of-exppee autoencoders for multi- modal deep generative models. Advances in Neural Information Processing Systems, 32, 2019.
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet S Talwalkar. Federated multi-task learning. Advances in neural information processing systems, 30, 2017.
Thomas Sutter, Imant Daunhawer, and Julia Vogt. Multimodal generative learning utilizing jensen- shannon-divergence Advances in Neural Information Processing Systems, 33:6100-611 10006010,,0100 2020a.
Thomas M Sutter, Imant Daunhawer, and Julia E Vogt. Generalized multimodal elbo. In International Conference Learning Representations, 2020b.
Vanya V Valindria, Nick Pawlowski, Martin Rajchl, Ioannis Lavdas, Eric O Aboagye, Andrea G Rockall, Daniel Rueckert, and Ben Glocker. Multi modal learning from unpaired images: Applica- tion to multi-organ segmentation in et and mri. In 2018 IEEE winter conference on applications of computer vision (WACV). pp. 547-556. IEEE, 2018.
Mike Wu and Noah Goodman. Multimodal generative models for scalable supeli-su learning. Advances in Neural Information Processing Systems, 31, 2018.
Baochen Xiong, Xiaoshan Yang, Fan Qi, and Changsheng Xu. A unified framework for modii federated learning. Neurocomputing. 2022.
Guang Yang. Qinghao Yc, and Jun Xia. Unbox the black-box for the medical explainable ai via multi-modal and multi-centre data fusion: A mini-review, showcases and beyond. Information Fusion, 77:29-52, 2022.
I1ker Yildirim. From perception to conception: learning multisensory representations. University of Rochester, 2014.
Yuchen Zhao, Payam Barnaghi, and Hamed Haddadi. Multimodal federated learning. arXiv preprint arXiv:2/09. (00.02//99 2021.
11

---

Published as a conference paper at ICLR 2024
CONTENTS
Introduction
Related Work
3
Notations and Problem Definition
Notations
3.2
Patchwork Learning
Methodology
4.1
Preliminaries: Variational Autoencoder (VAE), Product-of-Exxerts (PoF) and Pareto
Min-max Optimization
2
An Overview of the Realization of CLAP
4.3
Modality VAE and Vertical Pareto Min Max Optimization
Client-Apatation VAH and Horizontal Pareto Min-max Optimization
5
Experiments
5.1
Experiments PolyMNIST
8
5.2
Experiments on More Complicated Dataset Bimodal CelebA
5.3
Experiments Real- world Clinical Dataset eICU
5.4
Ablation Studies
6
Conclusion
Acknowledgments*
10
Theoretical Proofs
13
A.I Proof of Proposition]
13
2 Proof of Proposition
14
B Convergence Analysis
14
Algorithm and Optimization
15
Algorithm
15
Parcto Min-Max Optimization
15
Discussions Limitations
16
Implementation
16
D. Evaluation
16
D.2 More Experiments
17
D.3 Training Details
18
12

---

Published as a conference paper at ICLR 2024
More Ablation Studies
19
More Clients
19
B.2
Different Number of Missing Modalities,
19
Different Latent Vector Dimensionality
20
E,4
Different Level of Statistical Heterogenity
20
Computation Analysis and Devices
20
A THEORETICAL PROOFS
A.1 PROOF OF PROPOSITION
Assumption 3. All modalities form a connected graph in the patchwork.
Firstly, we prove that if a patchwork satisfies Assumption 1 and Assumption could be completed.
3 then the patchwork
Suppose we use S(M,, M,) to denote transformation from M; to Mj. If there are client e owning the two modalities ( (M, M2} c X the transformation f(M, M,) could be optimized by learning from the data.
From Assumption 3 if the patchwork forms a connected graph, each pair of modalities has an undirected path from one modality to the other, i.e.,
Vi,j E 21, .... K), 3p. s.t., M₁ + Mj, where p denotes a path on the graph.
(13)
As each edge denotes an optimized transformation, the path between M, and Mj corresponds to composed transformation f (M, M₁1) fM,, Mes)... [Mam, M2). Therefore, each missing modality could be imputed by the optimized transformations and the patchwork could be completed.
Then we prove that if patchwork could be completed by the optimized transformations, the patchwork satisfies Assumption I and Assumption 3
If the patchwork violates Assumption and there exists client has no data, i.c..
3i E {1, N) X = 0,
(14)
then cf cannot impute its missing modalities and the patchwork cannot be completed. This violates the premise that the patchwork could be completed. Therefore, the patchwork satisfies Assumption [
If the patchwork violates Assumption them, i.e.,
3 there exists two modalities { (M' M²} and no path between
Ji,j 21, K) Ap. s.t., M₁ My-
(15)
Since there is no edge between M, and Mj. there no client that owns the two modalities {M;, M2 If client et has the modality M; and it impute M, from the learned transformations, this means that there exists a path between another observed modality Mg and Mj, i.e.,
AM E X Fp, s.L., M₁ + Mj-
(16)
Meanwhile, d has both modality M, and Me, so there exists a path between Mf and Mj.
3p, s.t., M + M..
(17)
Combining the above two Equations, there exists p from Mi Mj and this violates Eq. 15
Therefore, Assumption holds.
If client d has the modality M, and it can impute M,,
prove Assumption holds similarly.
If client el does not have M; and does not have Mj, d' can impute both modalities using the observed modalities. Then there exists modality Mr E X and there exists path between M2 and M, and path between Mg and My. Therefore, there exists from M, to Mj. This violates Eq. (T5) and Assumption hollds.
13

---

Published as a conference paper at ICLR 2024
A.2 PROOF OF PROPOSITION 2
Firstly, we prove that (I. formulates an appropriate ELBO, i.e. log logp(r') 2 -C (0, Φ x*).
(18)
We prove it follows.
Beo(eloors) loog p(2, x*)
logp(x') = DRL.
(19a)
go (z(arto))
2 ( [oog p(2,
]
(19b)
(0(2))=))))
= E (alaan) log pp)'((=' DEL (go (2)x" (lx(z))
(19c)
(0,
(19d)
Since DEL (go(=|a (1 (1l((((r+) 2 O. the inequality Eq. 19b) holds.
Then we demonstrate that minimizing L (0, o, x² x²) minimizes DEL (go (2(xinn) 1pp (2/x*)) in Eq. 9..
From Eq. 19a, logp(x') is pre-defined. Maximizing (x [log pla, (=/
minimizes
DKL (go ( 1 (>)(((=)>)) In Eq. 19. p(z,at) is agnostic, we use (1 =)p(=)) to approximate
p(z,z'). Therefore. L (0, o, x²) is appropriate approximation of (=( log p(z.x') and maximizing (0, Φ, x²) minimizes DKL (ge (2)x lpp (2|z²))
(=(22))0))
-
B CONVERGENCE ANALYSIS
The work CCu et al. 2021) provides the convergence analysis of Pareto min-max optimization in section Appendix. We present the analysis the following.
The convergence of Pareto min-max max optimization. (More proof details could be found in (Cui et al. 202T).) Recall that Pareto min- max optimization aims achieve a Pareto solution h" that has the lowest loss the maximal value in (h) = obtains such a h" by a two-staged constrained multi-objective optimization.
v (h)]. Pareto min-max optimization
The convergence of the first stage: Suppose we use 0' and 5 to denote the parameters of h and the parameter 5 in the t-th iteration, (Cui et al. 2021 proves that in each iteration, Imaax (ho., 8t) decreases in Lemma in the Appendix.
max (ha++,, 88)
< i max (hor, 8f)
Lunx (hg,, , 08 < mmx (hge, 59) Imax (har+1, 87
< Imax (hor, ,))
As imax (for,, 8*) > 0 and decreases in each iteration, Pareto min-max optimization achieves min-max optimality and the algorithm converges.
The convergence of the second stage: (Cui al. 202 proves that in each iteration, the objective
(hbb) (i 1, ....... decreases or remains unchanged.
4 (hgrr1) & 4 (hor). Vi E [1,,,N}
This means that the objective the second stage + ΣN 4(h)) decreases or remains unchanged without violating the constraints. Finally, Pareto min-max optimization achieves Pareto optimality and the algorithm converges.
The convergence is not affected by the statistical and modality heterogeneity. From the above analysis, the convergence of Pareto min-max optimization is guarantced from the perspective of constrained multi-objective optimization.
14

---

Published as a conference paper at ICLR 2024
Algorithm 1 The training of CLAP
Input: the number of client N, the number of modalities K, epoch Tm, batch size Bm, ini- tialization of the encoder E' (0%) the decoder D (øk the decoder De (0%) where & =
2....... K.
do 2: randomly selects subset of clients S₁
1: for 0
3: the training of modality VAEs
4: for modality MR, k
.... K do 5: for client e E St in parallel do
6: (if Ma is observed) draw mini-batch mi. X calculate the ELBO: (L)
7:
8: then calculate the gradients with respect parameters 0 ox end for
9:
10:
Server aggregates the gradients of selected clients and update the parameters 6³ d by
Pareto min-max optimization; end for
11:
12:
the training of CA-VAEs
for client ct S₁ parallel do draw mini-batch x' ~ X calculate the losses I, (0, o, x²) (el E St) according Eq. then calculate the gradients of losses for parameters 0³ of (k: = 1, 2,,,,..KK K); end for
14:
16:
18:
Server aggregates the gradients of selected clients and update the parameters 6h and of
= 1, ..., K) by Pareto min-max optimization.
19:
end for
20: Output: the learned model 0%, 0% O% (k = 1.2,....
statistical heterogeneity: Pareto min-max optimization is applied to mitigate the statistical hetero- geneity. Even if there severe statistical heterogeneity in local clients, Pareto min-max optimization still converges to Pareto stationary solution, whose gradient is close to zero.
modality heterogeneity: Different modality combinations correspond different objectives. Modal- ity heterogeneity may induce more objectives to be optimized. During the Pareto min-max optimiza- tion, the objectives decrease each iteration and the algorithm still converges after iterations.
C ALGORITHM AND OPTIMIZATION
C.I ALGORITHM
We summarize the algorithm for the training of CLAP in Algorithm For the inference of CLAP, we present the process of imputation in Algorighm During training, observed modalities (X* X') between modalities to rebuild all observed modalities. Note that each client has N modality intersection with other clients. When aggregating the gradients, in each client, only the parameter gradients of encoders and decoders corresponding to visible modalities are transmitted. In the inference, we use the intersection which has the most modalities to infer the missing modalities. This corresponds to Line 5 in Algorithm
the intersection of
C.2 PARETO MIN-MAX OPTIMIZATION
Pareto min- max optimization CCi et al. 2021 aims achieve Pareto solution h² that has the lowest loss on the maximal value in (h) (h), (h), 1 (h)). Pareto min-max optimization
obtains such a h² by two-staged constrained multi-objective optimization.
15

---

Published as a conference paper at ICLR 2024
Algorithm 2 The inference of CLAP
Input: the patchwork be completed, the number of clients N, the number of modalities K, the AM
learned model 0e, @k of (k = ...,.. K).
1: for i 1, 2, .... N do
1,..... K do 2: for k
3: (if m/ observed) calculate the latent representation using the encoder 0h;
4: end for
5: calculates the representation using the maximal intersection modality combination by PoE;
6: rebuild missing modalities using the decoder DC (Ø = 2.,,..., K);
7: end for
8: Output: the completed patchwork.
The first stage: it achieves a min-ma max solution h! by entropy function method, which optimizes a surrogate function defined follows
4(h)
imax (h, = exp
imaz (h, 5) satisfies that
Amar (h) S imax (h, S Imaz (h) + In(N),
where 8 a hyper-parameter. will decrease in each iteration.
The second stage: it continues optimize the obtained solution htt to achieve Pareto optimality with constrained Pareto optimization. The objective be optimized follows:
N
min
S 4(h)), Vi E
In this way, Pareto min- max optimization achieves a Pareto solution, in which each objective cannot be further optimized without degrading others. Mcanwhile, it maintains its min-max optimality achieves the minimal maximum loss value.
C.3 DISCUSSIONS ON LIMITATIONS
This paper focuses on generating missing modalities for local clients. However, privacy and commu- nication concerns arise when learning generative models collaboratively, and CLAP is no exception.
The implementation of CLAP involves collaboratively learning generative VAEs for imputing missing modalities. The transmission of gradient information between the server and clients may result information leakage. To address this privacy concern, differential privacy may be useful to protect privacy from leakage during the training of the generative models.
Privacy.
Communication. Generative models typically have more parameters compared to classification models, which may result in more communication and computation overhead for local clients. In this regard, transferring the most valuable gradient information for model training a potential research direction to reduce communication overhead.
D IMPLEMENTATION
D.1 EVALUATION
In patchwork learning, models learn project the samples with missing modalities into j.jonnt latent representation and reconstruct the complete modalities. We evaluate the performance all methods from two aspects: generation quality and representation quality. Generation quality be assessed by generation coherence and Test Set Log-liked likeliiod. For the eICU dataset, we also report the mean square error (MSE) between generated and input samples. Representation quality can be assessed by
16

---

Published as a conference paper at ICLR 2024
linear classifier accuracy in latent space. The sample labels refer to digit numbers in the PolyMNIST dataset and the MNIST-SVHN-TEN, dataset, 40 binary attributes in the Bimodal CelebA dataset, and mortality in the eICU dataset. The results are averaged over all clients.
Generation Coherence. A classifier trained for each modality to predict the label of samples. During the evaluation of multimodal methods, all the modalities are reconstructed from incompleee modalities. Then we predict labels from the generated modalities and calculate accuracy (for the PolyMNIST dataset and the MNIST-SVHN-TEXT dataset), mean average precision (for the Bimodal CelebA dataset), or AUC (for the eICU dataset) using the true labels.
Test Set Log-likelihood. We generate full modalities for each sample and estimate the test set log-likelihood with 15 importance samples as in (Sutter et al. 2020b).
Linear Classifier Accuracy in Latent Space. Patchwork learning methods project samples into a joint latent representation and total modalities can be reconstructed from the latent representation. The learned latent space of quality should be constructed and contain intact information of all the modalities even if the input modalities are incomplete. Therefore, we perform linear classification in the latent space to predict the sample label. A logistic regression classifier trained with 500 samples in the training set for each modality subset in each client. And it is tested in the latent space of test sets.
D.2 MORE EXPERIMENTS
MNIST-SVHN- TEXT dataset Sutter et al. 2020a) is created based on MNIST-SVHN dataset (Shi et al. 2019), which is composed of image modalities MNIST and SVHN, as well as text modality. In particular, MNIST (Deng 2012) modality a handwritten digit image. SVHN (Netzer et al. 201 modality is an image of digit cropped from street view pictures. Each sample has the same label for all images. The text modality is an English string indicating the digit label. Each MNIST image pairs with 20 SVHN images. The dataset has 20x60,000 training samples and 20 10,000 test samples. For the setting of data heterogeneity, we partition the dataset into 10 clients, and each client contains 6 classes of samples. In each client, there less than one modality be masked.
The performance of all methods on this dataset is shown in Table 6 From Table 6 our method outperforms all baselines and achieves the best coherence. The modeling of the SVHN modality is mediocre as this modality is complex and noisy. CLAP still maintains the best coherence (58.9). As there may exist trade-off between coherence and likeeiiid our method achieves a comparable log-lik -ogllikel with baselines. More results are shown in Appendix.
Table 6: Imputation coherence accuracy (%) of 3 modalities M (MNIST), S (SVHN) and T (Text) on the MNIST-SVHN-TEXT dataset. The ligliioes tested X = (,
Coherence
Likelihood
Model
M
S
T
MVAE
75.5
47.7a1.4
80.420.7
-1971
MMVAE
88.640.6
21.5+2.1
76.821.4
-183842
MoPoE
89.4
54.4
92.1
-1840+1
CLAP
95.3+0.8
58.9
92.3-0.3
-1838
CUB. We also conduct our experiments on CUB datasets. CUB Hwang et al. 20021 contains 11,788 photos of 200 kinds of birds in natural scenes, each annotated with TO fine- grainer captions describing the bird's 's appearance characteristics collected through Amazon Mechanical Turk (AMT). We partition the data into 10 clients. Each client contains 90 different kinds of birds. For the two modalities (image and text), we randomly mask one modality for the randomly selected clients. The results of MSE, linear classification, and likeliikl are shown in Table 7 where I and stand for image and text modalities respectively. As shown in the table, CLAP still outperforms all baselines on this complex dataset.
POISM The log-likelihood results reported in Table 8 which evaluates the generation quality. MVAE has a worse imputation coherence accuracy because it fails to learn the dependencies among modalities. MoPoE achieves better imputation coherence than other baselines as it learns all transformations among modalities. CLAP achieves the best log-likelihoods as it models the required dependencies for imputation compared with baselines.
17

---

Published as a conference paper at ICLR 2024
Table 7: Experimental results on CUB
method
MSE
Latent Classification
Likelihood
I
(1)
(T)
{I.
X
MVAE
0.0298
13.4
1.7
5.3
-6437
-156
-6549
MMVAE
0.0425
2.9
-6650
-117
-6745
MoPoE
0.0241
13.0
3.5
4.9
-6283
-148
-6374
CLAP
0.0233
15.3
3.5
5.7
-6251
-142
-6342
Table 8: The log-likelihoods 5 modalities and the joint log- bog-liielihoods X {Mi 2 = 1,2...53 on PolyMNIST dataset.
Model
M,
M2
M3
M4
Ms
X
MVAE
-1495
-1292+1
-1252+4
-1333
-1431
-684941
MMVAE
-1347=1
+1199
-127444
-1460=1
-134642
-696444
MoPoE
-1412=3
-1237
-1201
-1494
-131216
-6715
CLAP
-1319
-120741
-117713
-1353
-1349
-660314
MNIST-SVHN-TEXT. The results of linear classification accuracy on MNIST-SVHN-TEXT are in
Table g
Table Linear classification accuracy (%) in latent space of some modality subsets on the MNIST- SVHN-TEXT dataset. The average accuracy of all possible subsets is also reported.
Model
{S}
{M, S}
(M
Average
MVAE
32.7+0.2
66.5+1.3
48.8
71.1
MMVAE
79.7+0.5
65.0+0.8
47.2+0.7
79.
MoPoE
79.9 79.9
68.5
49.010.2
80.5
CLAP
79.4
68.7
19.2
80.6
eICU. In the original setting, there are 13 modalities in the eICU dataset, which means the MoPoE can not work due to the vast number (218) of modality subsets. To make the comparison feasible, we discard the last modality and divide the rest 12 modalities into 4 modality groups. The results
shown in the Table 10 In this setting, our method also outperforms all the baselines on all metrics.
D.3 TRAINING DETAILS
We Adam optimizer for training and the training batch size of all experiments is set 256. The architecture of classifiers for the coherence test is the same as the unimodal encoder architecture except for the last layer. The model architecture and training hyperparameters are the same for different methods. When calculating the training loss, the likelihood of reconstructed modalities is weighted according their data dimensions.
Bimodal CelebA. The learning rate is set as 0.0005, 3 is set as 1.0, dimension of latent vectors is set as 64. The models are trained for 200 epochs. The encoders and decoders for both the Image and Text modalities are based on residual convolutional blocks. We use modality-specific latent space as in (Daunhawer et al. (020) improve the generation quality. The modality-spechiee latent vector occupies half of the latent vectors.
PNISMNIST. The learning rate set as 0.001, is set as 0.4, dimension latent vectors is set as 512. The models are trained for 300 epochs. The unimodal encoders are convolutional neural networks and the decoders are based on transposed convolutional layers.
CUB. The learning rate is set 0.0005, 3 is set 0.4, dimension of latent vectors is set 64. The models are trained for 75 epochs. The encoders for both the Image and Text modality are based on convolutional neural networks and the decoders based on transposed convolutional layers.
MNIST-SVHN-TEE The learning rate is set as 0,001, 8 is set as 0.4, dimension of latent vectors set as 20. The models are trained for 150 epochs. The encoder and decoder of the MNIST modality are fully-connected neural networks. For the SVHN and TEXT modality, they are convolutional and transposed convolutional neural networks.
18

---

Published as a conference paper at ICLR 2024
Table 10:
Results eICU dataset with 4 modalities.
MSE
Latent Classification
Likelihood
Method
M1
M2
M3
M4
Average
MVAE
0.0366
0.0426
0.1006
0.0876
40.2
-319
MMVAE
0.0380
0.0440
0.1029
0.0928
38.3
-320
MoPoR 0.0360
0.0426
0.0997
0.0848
41.5
-319
CLAP
0.0360
0.0422
0.0997
0.0847
43.3
-316
eICU. The learning rate is set 0.0005, 3 is set 2.0, dimension of latent vectors is as 32. The models are trained for 150 epochs. The encoders are composed of LSTM and fully connected layers. And the decoders are fully-connected neural networks.
E
MORE ABLATION STUDIES
E. MORE CLIENTS
We conduct experiments the PolyMNIST dataset with more clients (50 and 100). The results shown in Table 15] When we split the dataset into more clients, the statistical heterogeneity gets more complex. Thus the latent classification results and log- likellood decline for all the models and our method still outperforms the baselines.
Table 11: Latent space classification and log- se-likklihhodd results of some modality subsets on the PolyMNIST dataset. The average accuracy of all possible subsets.
Client Number
Latent Classification
Likelihood
Model
MM4
[M4, Ms)
[Mg. M4. Ms)
Average
M2
M4
Mg
MVAE
34.5
31,5
30.7
21
-1668
-1518
-1381
-7168
50
MMVAE
36.7
32.6
32.0
21.3
-1250
-1435
-1372
-7185
MoPoli
CLAP
36.7
40.4
39.9
23.4
-1242
-1465
-1363
-7104
MVAE
37.4
37.6
37.4
22.0
-1295
-1475
-1418
-7327
MMVAE
37.1
35.0
34.0
21.7
-1304
-1470
-1418
-7389
MaPoR
35.8
36.5
36.
21.7
-1306
-1499
-1464
-7385
37.7
38.4
22.3
-1282
-1466
-1406
-7313
E.2 DIFFERENT NUMBER OF MISSING MODALITIES.
As the missing modalities are diverse in local clients, propose to randomly generate the masks on the PolyMNIST dataset. Specifically, we randomly generate different masks with different missing probabilities. We conduct experiments on the PolyMNIST dataset with 3 random masks. From the results in Table I2 our method consistently outperforms other baselines.
Table 12: Imputation coherence accuracy (%), latent space classification and log- likeliiddoood results of some modality subsets on the PolyMNIST dataset.
Coherence
Latent Classification
Likelihood
Model
M₂
M4
M2
(M4)
(M4, Ms}
(M2, M4. Ms}
MVAE
18.1
69.1
18.2
80.3
74.7
77.0
-6851
MMVAE
62.0
63.1
34.6
86.8
79.2
81.3
-6951
mask
MoPoE
69.5
73.1
48.3
79.0
80.5
94.7
-6734
CLAP
69.7
79.2
50.4
88.4
91.9
-6614
MVAE
22.4
70.5
16.1
82.8
71.1
77.5
-6842
MMVAE
64.5
62.8
34.8
87.5
77.7
81.4
-6951
mask
MoPoB
69.8
71.9
45.2
80.3
81.7
95.6
-6768
CLAP
77.1
52.4
88.2
91.5
-6601
MVAE
15.7
69.2
15.2
81.7
74.5
77.4
-6832
MMVAE
63.7
62.5
35.9
84.4
75.3
80.2
-6960
mask
MoPoB
68.4
70.7
46.4
81.3
81.4
95.0
-6751
CLAP
71.5
77.3
53.7
85.2
92.7
94,8
-6634
19

---

Published as a conference paper at ICLR 2024
Besides, mask the different number of modalities the eICU dataset. As the results shown in Table 13 the imputation task more difficult to tackle with more modalities masked.
Table 13: Results on eICU dataset with the different number of modalities masked. MSE and latent classification AUC (%) are averaged over 13 modalities and certain subsets. The log-likelihoods of joint distribution X are reported.
Masked number
MSE
Latent Classification
Likelihood
0.0612
65.4
-321.4
0.0653
62.3
-340.2
0.0667
60.2
-357.6
0.0668
54.2
-382.4
10
0.0705
52.1
-380.7
E.3 DIFFERENT LATENT VECTOR DIMENSIONALITY
We conduct the ablation study using the MNIST-SVHN-TER dataset on the latent vector dimen- sionality. As shown in Table [4] the performance degrades when the latent vector dimensionality is too small (<20). Otherwise, the model performances are similar.
Table 14: Results on MNIST-SVHN-TENS dataset with the different number of latent vector dimensionality.
Coherence
Likelihood
Dimensionality
M
S
X
45.1
19.2
78.9
-1956
5
65.1
47.4
78.9
-1919
10
78.0
53.4
93.1
-1873
20
95.3
58.9
92.3
-1838
40
96.4
60.7
89.3
-147
60
94.4
61.1
91.7
1852
80
95.6
60.6
91.2
1868
100
94.3
61.0
89.3
-1876
E.4 DIFFERENT LEVEL OF STATISTICAL HETEROGENITY
We supplement the ablation study on statistical heterogeneity shown in Table 15 We split the PolyMNIST dataset into 50 clients and each client contains samples from k classes. A smaller leads to greater heterogeneity. As shown in the Table below, the log-likelihoods decrease with the increase of heterogeneity. The latent classification accuracies are larger for smaller k. because smaller k values entail reduction in the number of classes, making classification easier.
Table 15: Results on the PNIST dataset with different statistical heterogeneity indicator k.
Latent Classification
Likelihood
k
{Ma}
(Ma, Mc}
{M, M4, Mc}
Average
M2
M4
Mr
X
31.1
33.0
31.2
19.7
-1261
-1483
-1359
-7104
6
36.7
40.4
39,9
23.4
1242
-165
-1363
-7104
4
43.5
43.9
42,9
26.7
-2667
-148
-1384
-7122
3
50.2
52.1
52.4
30.7
1190
-1487
-1386
-7148
F COMPUTATION ANALYSIS AND DEVICES
As generative model, CLAP has a similar number of parameters with other federated multimodal learning methods, including the baselines PoE, MoE, etc. Therefore, CLAP does not bring many
20

---

Published as a conference paper at ICLR 2024
extra computational and communication costs. As study novel problem, it is hard to compare the computational overhead with other methods used in other problems. We provide the running time comparisons with basclines in Table T6 which demonstrates that CLAP does not have too much additional computational overhead with baselines. MoPoE can not work on the eICU dataset due the vast number (213) of modality subsets. It has exponential complexity with respect to the number of modalities and costs vast number of computing resources. Compared with MVAE and MMVAE, our method aggregates the identified several subsets and does add too much burden on optimization and computation.
Devices In the experiments, we conduct all methods on local Linux server that has two physical CPU chips (Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz) and 32 logical kernels. All methods are implemented using Pytorch framework and models are trained on GeForce RTX 2080 Ti GPUs.
Table 16: Run-time consumption comparisons the eICU dataset
Methods
Run-time consumption (PolyMNIST) Run-time consumption (eICU)
MVAE
261 min
70 min
MMVAE
236 min
61 min
MoPoE
246 min
1440 min
CLAP
279 min
79 min
21


Reviews:
[{"review_id": "NouP52q9zr", "reviewer": "Reviewer_DjdF", "summary": "This paper primarily addresses the task of processing multimodal data in a multi-client context, proposing the CLAP framework. The framework consists of a modality VAE and a client-adaptation VAE. The modality VAE learns a representation for each modality, while the client-adaptation VAE handles dependencies between modalities. Given that data are from multiple clients and not independently and identically distributed, to alleviate the statistical heterogeneity of the learned representations, an encoder is shared and the KL divergence of the representations is balanced by maximizing the maximum KL divergence. The authors demonstrate through experiments that the CLAP framework effectively processes heterogeneous data while protecting user privacy.", "questions": "1. Why is client data non-independent and identically distributed? How is the feature distribution of the same modality dataset across different clients demonstrated?\n2. The author presents a heterogeneous data exchange among multiple medical institutions, but I am not quite clear whether the complexity of the current research algorithm is already suited to the needs of the current research context.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The CLAP framework proposed in this article achieves the imputation of multimodal data when dealing with non-independent and identically distributed data.\n2. By employing a vertical Pareto min-max optimization strategy to balance the KL divergence, CLAP addresses the issue of statistical heterogeneity in client data distributions.\n3. The training of CLAP adheres to the federated learning framework, effectively protecting the privacy of users across multiple clients.", "weaknesses": "1. The framework presented in this paper is complex; is it suitable for practical deployment?\n2. The experimental section includes a limited number of medical datasets, which constrains the assessment of the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sx3kCpGHBf", "reviewer": "Reviewer_scKN", "summary": "The authors introduce a novel problem in machine learning referred to as the \"Checkerboard Learning\" This problem focuses on the issue of missing modalities within local client datasets by employing multimodal learning imputation techniques. To tackle this problem, the authors present a comprehensive framework called \"Collaborative Adaptation\" or CLAP. In essence, CLAP aims to bridge the gap between differing data distributions and unearth interdependencies between local clients through the utilization of two VAEs. A pivotal component of CLAP is the Modality VAE, designed to enhance the consistency of data representation by implementing a shared Modality Encoder across all clients. The Client-Adaptation VAE (CA-VAE) builds upon the Modality VAE by introducing a new decoder to learn the dependencies among different modalities. Empirical experiments demonstrate that CLAP exhibits strong performance across both benchmark and real-world datasets, validating its effectiveness in addressing the checkerboard learning challenge.", "questions": "1. In related work, most of the papers are published before 2022. Are there more recent related works which study the similar learning scenarios?\n\n2. This paper experimentally verifies the effectiveness of the proposed Pareto min-max optimization. Could the authors give more explanation of it? Why the Pareto min-max optimization is helpful for the imputation of the checkerboard?\n\n3. The algorithm is in Appendix. I guess it is because of the page limit. For the real-world application, the implementation is very important. From the C.3 in Appendix, the author discusses the limitations. How to address the privacy issue when in real-world applications?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The proposed method is very practical. In real-world, there are many local clients with multi-modal data. However, the modality types are largely inconsistent, makes it difficult to learn models directly. This paper proposes “checkerboard learning”, which formulates this problem precisely.\n\n2. The analysis of the problem is important, and the proposed method is convincing. This paper analyzes the checkerboard learning deeply and summarizes three challenges of checkerboard learning, e.g., heterogeneity, different modality combinations. To complete the checkerboard, this paper proposes two basic assumptions, which makes it possible to impute the missing modalities by learning from other clients.\n\n3. This paper provides experimental verification of the proposed method. It conducts experiments on MNIST-typed datasets, CelebA CUB and a real-world datasets eICU. The experiments show the superiority of the proposed methods compared with existing methods.\n\n4. In the Appendix, the authors did sufficient ablation studies for the proposed methods, including more clients and the number of missing modalities. It is very helpful to evaluate the proposed method.", "weaknesses": "1.In related work, most of the papers are published before 2022. Are there more recent related works which study the similar learning scenarios?\n\n2.This paper experimentally verifies the effectiveness of the proposed Pareto min-max optimization. Could the authors give more explanation of it? Why the Pareto min-max optimization is helpful for the imputation of the checkerboard?\n\n3.For the real-world application, the implementation is very important. From the C.3 in Appendix, the author discusses the limitations. How to address the privacy issue when in real-world applications?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FytbW9t7Md", "reviewer": "Reviewer_ccS1", "summary": "The authors explore a novel federated multi-modal learning scenario “checkerboard learning”. Different from existing research which mostly assume the presence of complete modalities, the proposed task considers a practical problem where different clients possess various data modalities. The federated clients aim to impute the missing modalities by collaborating with other clients. A client could impute the missing modalities relying on the dependency learned in other clients which contain the aiming modalities. The proposed framework CLAP to tackle this problem consists of modality VAEs and client-adaptation VAEs. The modality VAEs strive to address the statistical heterogeneity problem by balancing the distribution distance among clients. And the client-adaptation VAEs are the additional decoders used for balancing the modality heterogeneity among clients. Experimental results demonstrate the superiority of CLAP.", "questions": "1. This paper studies a useful and significant learning task. From my knowledge, it is the first to investigate checkerboard learning. I want to know whether there are previous related research?\n2. There are modality VAEs and client-adaptation VAEs in the proposed framework. These VAEs are specifically designed to tackle two different challenges: statistical heterogeneity and modality heterogeneity, respectively. One important question is whether they share the same encoders. If they do, it is crucial to understand how they interact and influence each other.\n3. The details behind certain aspects of the proposed method needs more explanation. More clarification is needed regarding why the employment of the Pareto min-max framework is advantageous for addressing heterogeneity, as well as the detailed implementation of this framework.\n4. During the application, the proposed method incorporates an additional set of decoders. Does it entail a significant increase in computational overhead?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The authors propose a very novel task. Existing federated multi-modal learning research mostly assume the same modality combination in all clients. The scenario of different modalities among clients is under-explored. Besides, the proposed task is substantially practical, as collaborative scenarios involving multiple clients with diverse modality combinations are frequently encountered in real-world applications.\n2. In the proposed new task, the missing modalities are not only invisible during testing, but also unavailable during training, making the checkerboard learning a much more challenging and practical problem.\n3. The proposed method is concise and effective. The method is developed based on a comprehensive study of the checkerboard learning problem, wherein the challenges can be concluded into two aspects: statistical heterogeneity and modality combination heterogeneity among clients. The authors employ the Pareto min-max framework to address the aforementioned heterogeneity, which proves to be fundamentally valid.\n4. They conduct extensive experiments with sufficient datasets and compare with various baselines. They also perform experiments on a real-world clinical dataset, which demonstrates the effectiveness of the method in practice.", "weaknesses": "1. This paper studies a useful and significant learning task. From my knowledge, it is the first to investigate checkerboard learning. I want to know whether there is previous related research.\n2. There are modality VAEs and client-adaptation VAEs in the proposed framework. These VAEs are specifically designed to tackle two different challenges: statistical heterogeneity and modality heterogeneity, respectively. One important question is whether they share the same encoders. If they do, it is crucial to understand how they interact and influence each other.\n3. The details behind certain aspects of the proposed method needs more explanation. More clarification is needed regarding why the employment of the Pareto min-max framework is advantageous for addressing heterogeneity, as well as the detailed implementation of this framework.\n4. During the application, the proposed method incorporates an additional set of decoders. Does it entail a significant increase in computational overhead?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OHUw0gRwU6", "reviewer": "Reviewer_QCRo", "summary": "This paper learns the domain of multimodal learning, with a specific focus on the challenge known as \"checkerboard learning.\" In contrast to conventional federated multimodal learning, where all clients typically share the same modalities, checkerboard learning presents a scenario where different clients may possess different modality combinations. The primary objective of this task is to reconstruct or impute the missing modalities for each client without access to the raw data. To address this problem, the authors propose a method named \"Collaborative Adaptation\" (CLAP), which encompasses two distinct training stages within each client: Modality Variational Autoencoder (MVAE) and Client-Adaptation Variational Autoencoder (CA-VAE). The authors provide empirical evidence showcasing the efficacy of their proposed approach on various datasets, underscoring its ability to effectively address the challenges posed by checkerboard learning.", "questions": "Please see the weaknesses above. Besides, I have the following minor problems.\n\n1. In section 4, the author demonstrates the proposed method. However, an algorithm would be more helpful for other to follow this research.\n2. For the assumptions in section 3, it is better to give a justification for its practicality.\n3. In the conclusion, this paper says that it opens a direction for the future research. Could the author discuss more details about it?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Exploring modality differences has great practical significance in real-world applications. In practice, numerous local clients deal with multi-modal data. Yet, the challenge lies in the substantial inconsistency in the types of modalities. This paper considers a practical and challenging problem.\n2. The proposed method is novel and valid for addressing this problem. To impute the incomplete checkerboard, this paper proposes CA-VAE to model the relationships between modalities based on the the assumption that not all modalities are isolated modalities. For the statistical heterogeneity, this paper proposes Pareto Min-Max optimization to balance the performance of all combinations.\n3. This paper offers empirical validation of the proposed methodology through a series of experiments. These experiments encompass MNIST datasets, other benckmark datasets and eICU. The results of these experiments underscore the superior performance of the proposed methods when compared to existing approaches.\n4. This paper is well-written. The figures is very helpful to understand the motivation of the proposed method.", "weaknesses": "1. The proposed method is based on the two assumptions. It seems that these two assumptions are not so strict. Is the effectiveness of the proposed method will be affected by the Assumption 2 (e.g., when there are many missing modalities)?\n2. For the challenge of statistical heterogeneity, the author proposes vertical Pareo min-max optimization to address it. Please explain it why the vertical Pareto min-max optimization could mitigate the statistical heterogeneity.\n3. In this paper, the author did experiments on some datasets. When the data is more completed, is it still useful to complete the missing modalities?\n4. For the statistical heterogeneity, in real-world applications, there are many different types of heterogeneity, how the heterogeneity affects the effectiveness of the proposed method?\n5. For the proposed CA-VAE in section 4.4, it seems sound. However, is there direct experimental verification of the effectiveness of CA-VAE?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)