Multimodal Federated Learning via Contrastive Representation Ensemble
Authors: Qiying Yu
Yang Liu
Yimu Wang
Ke Xu
Jingjing Liu
Venue: ICLR 2023 poster (2023)
Keywords: Federated Learning
Multi-modal Learning
Representation-level Ensemble Knowledge Transfer

Abstract: With the increasing amount of multimedia data on modern mobile systems and IoT infrastructures, harnessing these rich multimodal data without breaching user privacy becomes a critical issue. Federated learning (FL) serves as a privacy-conscious alternative to centralized machine learning. However, existing FL methods extended to multimodal data all rely on model aggregation on single modality level, which restrains the server and clients to have identical model architecture for each modality. This limits the global model in terms of both model complexity and data capacity, not to mention task diversity. In this work, we propose \textit{Contrastive Representation Ensemble and Aggregation for Multimodal FL (CreamFL)}, a multimodal federated learning framework that enables training larger server models from clients with heterogeneous model architectures and data modalities, while only communicating knowledge on public dataset. To achieve better multimodal representation fusion, we design a global-local cross-modal ensemble strategy to aggregate client representations. To mitigate local model drift caused by two unprecedented heterogeneous factors stemming from multimodal discrepancy (\textit{modality gap} and \textit{task gap}), we further propose two inter-modal and intra-modal contrasts to regularize local training, which complements information of the absent modality for uni-modal clients and regularizes local clients to head towards global consensus. Thorough evaluations and ablation studies on image-text retrieval and visual question answering tasks showcase the superiority of CreamFL over state-of-the-art FL methods and its practical value.

Published as a conference paper at ICLR 2023
MULTIMODAL FEDERATED LEARNING VIA CON- TRASTIVE REPRESENTATION ENSEMBLE
Qiying Yu' Yang Liu 14, Yimu Wang', Ke Xu', Jingjing Liu'*
1 Institute for AI Industry Research, Tsinghua University
2 University of Waterloo
Carnegie Mellon University
Shanghai Artificial Intelligence Laboratory yuqy22 2 @mai aaa.ill...... ts ssinghua. edu. cn, tiinto,ppilooeeettttt.nehh www.airurssinnuuu.eeeeoee sesinnhua.edu.cm cn
ABSTRACT
With the increasing amount of multimedia data modern mobile systems and 1oT infrastructures, harnessing these rich multimodal data without breaching user privacy becomes a critical issue. Federated learning (FL) serves as privacy- conscious alternative to centralized machine learning. However, existing FL meth- ods extended to multimodal data all rely on model aggregation on single modality level, which restrains the server and clients to have identical model architecture for each modality. This limits the global model in terms of both model complexity and data capacity, let alone task diversity. In this work, we propose Contrastive Repre- sentation Ensemble and Aggregation for Multimodal TL (CreamFL), multimodal federated learning framework that enables training larger server models from clients with heterogeneous model architectures and data modalities, while only communicating knowledge on public dataset. To achieve better multimodal repre- sentation fusion, we design a global-local cross-modal ensemble strategy to aggre- gate client representations. To mitigate local model drift caused by two unprece- dented heterogeneous factors stemming from multimodal discrepancy (modality gap and task gap), we further propose inter-modal and intra-modal contrasts regularize local training, which complements information of the absent modality for uni-modal clients and regularizes local clients to head towards global consen- sus. Thorough evaluations and ablation studies on image-text retrieval and VQA tasks showcase the superiority of CreamFL. over state-of otate-of-the-art FL. methods.
INTRODUCTION
Federated Learning (FL) (Yang et al., 2019 Li et al. 2020; Kairouz et al.) 2021 Zhao et al.) 2018). decentralized training paradigm that allows multiple parties to collaboratively train models without compromising privacy, has emerged as alternative to centralized machine learning. Most existing FL methods only consider scenarios where the private data from clients belong to the same modality (e.g., image or text). However, with the fast development of mobile technology and IoT infrastruc- tures (Brunete et al., 2021) that harness data from different modalities (e.g. sensory, visual, audio) with privacy constraints, there is an increasing need for advanced FL. algorithms to allow the training of larger and capable model that can absorb heterogeneous private data (across modalities) at edge
and simultaneously handle diverse multimodal tasks (Gan et al. 2022 Chen al. 2020b).
In the past, there has been some early attempts at applying FL to multimodal tasks (Xiong et al. 2022 Zhao et al. 2022 Liu al. 2020), which all adopt the FedAvg (MeMahan al. 2017) framework by using homogeneous models for each modality. In practice, however, edge devices may have limited computational and memory resources, restraining the capacity of the global model to smaller and lighter scales. Moreover, naive aggregation of modallty-dependent models is inadequate in addressing the model drift (Karimireddy et al. 2020) problem between clients.
Recently, few algorithms (Cho et al.) 2022 Cheng et al. 2021) have been proposed to enable larger server model training. For example, FedET (Cho et al. 2022) proposes an ensemble Knowl- edge Distillation (KD) based framework to enable large model at server and relatively small yet
"Corresooningg Authors

---

Published as a conference paper at ICLR 2023
Private
Ing Clieats
MM Clients
Clients
Server
Houeegeoco
Eona-Midaa Ramanal
Daa
Imp Tur Ficoder
Heicrogenesus Archilectue
Beckathh
CNN GRL
ORd
Image Captoting
CSN
Visual Quessius
Model DAt
Public Multimodal Data
Anpoeraag
Two people sitting table with food
Thst
Meld
Clasifa
Tre
Illustration of multimodal FL. A large model at server supports multimodal tasks with public data, and heterogeneous clients edge handle uni- and multi-modal tasks with private data.
Figure 1:
deployable models edge devices. However, they transfer and ensemble knowledge from a bag of client teachers through logit, which is difficult to extend to multimodal setting. Most multimodal tasks (e.g., image/video captioning (Vinyals et al., 2015)) typically operate on fused cross- modality representation level, whereas existing strategies for aggregating logits are no longer applicable.
In this paper. we design novel KD-based multimodal federated learning framework, CreamFL (Contrastive Representation Ensemble and Aggregation for Multimodal FL), which simultaneously leverages uni- and multi-modal data through representation-leve ensemble knowledge transfer. The global model learns from clients via communicating private knowledge on public dataset from diverse client networks without revealing private models and data. CreamFL. transmits low-dimensional representations of public data be- tween server and clients, which are usually contextual and applicable more complex tasks than logits. To effectively aggregate representations transmitted from heterogeneous clients, we propose a global-local cross-modal contrastive aggregation strategy, to 1) filter out drifting outliers by con- trasting local representations to global ones; 2) pick out outstanding candidates that better match their paired partners, by contrasting to representations from another modality.
heterogeneous clients to learn a larger global model,
Morcover, FL with multimodal data brings about two types of model gap: modality gap and task gap. Uni-modal clients trained under a single modality (e.g. image) have never seen data from other modalities (e.g., text) in the training procedure, therefore lacking the capability of recognizing another modality. We call this mutual incompatibility between clients the 'modality gap'. Task gap refers the fact that different clients may be trained for diverse tasks, e.g., uni-modal clients for image classification task and multimodal clients for image-text retrieval task. Both gaps cause unprecedented model drift (Karimireddy al. 2020) problems. To tackle this, introduce two novel contrastive objectives to regularize local training. An inter-modal contrastive objective is designed mitigate the modality gap. by performing cross-modality contrasts using public data in the local training phase, which complements for the information of the absent modality in uni- modal clients. To bridge the task gap, an intra-modal contrastive objective is proposed to contrast local representations to their corresponding global ones in each modality, regularizing models to head towards the global consensus (Li & Wang 2019).
In summary, 1) CreamFL is the first KD-based multimodal FL framework to support heterogeneous modality and model architectures between server and clients, while only communicating private knowledge public dataset without revealing private models and data. Experiments show that CreamFL outperforms other FL systems multimodal setting in terms both model performance and communication cost; 2) CreamFL ensembles representations instead of logits for knowledge transfer between clients and server, with a novel global-local cross-modal aggregation strategy for better representation learning and inter/intra-modal contrastive objectives address model drift; 3) Our framework enables larger model training at server that absorbs modality- knowledge from cosstrce- clients, which is required for complex cross-modee moosl tasks.
2 RELATED WORK
Some works investigated applying KD to FL (Lin et al. 2020; Itabara et al. 2021 Wu et al. 2021).
Li & Wang (2019) allows heterogeneous clients distill know ledge from the aggregated consensus,
2

---

Published as a conference paper at ICLR 2023
1
Representation Appegrtion
Ing Ulents
Locelnng Ieg Rmp
Tul
Server
Loca
Local Rep.
Local
Globail eca Comradive
Sep
MM Cieemss
Local Ieg Ran
Sought vctt
Lock
Glodd Irugand
Lead Tsi
Tor Rop
Image Encoder
Encoder
A
Local Tox Sep.
Claka isa Ras
(1) local Triining Regiitrization
Gllbal Txi
Gloaa trug
Eip.
Local Th: Rox
Global Lus Rep
Global Tei Rep.
Local lin Aep.
Eig Heaarr
Last Tet Pop
Central
Castrll
Server-clae client Communication
Local and Global Exceution
Figure 2: Illustration of CreamFL framework with a large server and heterogeneous clients. ! Clients receive global representations (Global Img/Txt Rep.) from the server and perform regular- ized local training. 2 Clients generate representations of public data (Local Img/Txt Rep.) and transmit them to the server. 3 The server aggregates received local representations with global representations. The server distills knowledge from the aggregated representations.
but does not train server model. FedGKT (He et al., 2020a allows larger server models, while the server makes selection among clients, leading to bad consensus with impeded performance. Cho et al.] (2022) and [Cheng et al. (202T) concurrently propose to train larger server models in FL through ensemble knowledge transter. However, their selective aggregation strategies specif- ically designed for logit (using variance or entropy of logit for weighting) and the learned large server model is limited classification task. We propose transmit representations and design novel ensemble strategy to make the model applicable to complex multimodal tasks.
Several attempts have been made apply FL multimodal tasks. Xiong al. (2022) extended FedAvg to multimodal scenario. Liu et al. (020)) applied FL to leverage multiple datasets from different distributions boost performance, but lacks the communication phase of FL. Zhao et al. (2022) proposed to assign higher weights to multimodal clients in aggregation, but this strategy needs manual tuning for the weights and only works for FI. systems with both uni- and multimodal clients. We design novel tuning-free global-local cross-modal contrastive strategy for aggregation, which has broader applicability to scenarios with only multimodal clients.
Inter-modal contrastive learning is prevailing technique for multimodal self-supervised learn- ing (Jia et al., 2021 Singh et al. 2022, Yuan al. (202T). The contrastive part of most pretraining methods resembles that of CLIP (Radford et al. (021), where aligned and unaligned image-text pairs are treated as positive and negative pairs, respectively. Our method differs from them in two aspects: 1) we creatively apply the concept of contrasting to design "metric" for evaluating the quality of representations, to address the challenge that the representation quality varies significantly between clients due their inevitable heterogeneity; 2) propose use inter-modal contrast regular- ization technique to bridge the modality gap between clients, and restrict global-local discrepancy.
3 FEDERATED MULTIMODAL LEARNING
A key goal of multimodal learning is to unify signals from different modalities into the same vector space, where semantically correlated data across modalities share similar representations. This goal serves as a north star across our design of the multimodal federated learning framework, to guide the global model to better capture intrinsic correlation between multimodal data.
3.1 PROBLEM DEFINITION
We consider heterogeneous FL setting where M multimodal clients, I uni-modal image clients and uni-modal text clients collaboratively train a global model f. w) : R" - Ra through representation--evel ensemble knowledge distillation, where is the model parameter, and d are
3

---

Published as a conference paper at ICLR 2023
the dimensions of input data and extracted features of the input data, respectively. Each image client
E [4] has its own private dataset Ip = { (zH, v/)) } [Zpl
k=='' where o is the k-th training sample of the p-th image client, up is its corresponding label. Similarly, each text client g [T] has its private dataset T4 { (z o7 } Amm The multimodal client E [M]] has its local dataset Mm
TTl
( ) k= where (imm t% is k-thh image-text pair with correlated semantic information. Both [P]
server and clients can access public dataset p { (i(A),(())) I AA=1" where ((0)) is an image-
text pair. We further assume that each client adopts small model fa(:: we) R Rd of its own choice based on its local data, task and resources, which outputs representations of the same dimension as the global model.
As shown in Figure 2 during training, clients first receive global representations of public data and perform multiple local steps of representation learning with inter- and intra-modal regularization (section 3.2 Then, clients generate representations of public data according to their modality and transmit them to the server. To climinate bias between different modalities and clients, the server selectively aggregates representations (section 3.3). Finally, the server performs knowledge distillation from the aggregated representations and transmits its trained representations back to clients. The complete algorithm is provided in Algorithm
3.2 LOCAL TRAINING VIA CONTRASTIVE REGULARIZATION (LCR)
We first focus on mitigating model drift through regularizing local training, by designing contrastive objectives from two perspectives: 1) inter-modal contrast complement the absent modality infor- mation for uni-modal clients and enhance cross-modal interaction; and 2) intra-modal contrast to mitigate non-i.i.d. problem in each modality separately. At the beginning of each communication round, the server sends the representations of public dataset generated by the global model to clients help them regularize local training ('Global Img Rep.' and 'Global Text Rep.' in the '() Local Training Regularization' path of Figure
3.2. INTER-MODAL CONTRAST
Modality gap exists between uni-modal and multimodal clients. For example, image clients have never seen any text information during their training process, while multimodal clients are trained under the presence of both modalities. This gap leads to the discrepancy between image representa- tions generated by image- and multimodal-clients To address this issue, inspired by CLIP (Radford) et al. 2021) that enhances visual representation learning by contrasting to language, we perform an inter-modal contrast in clients' local training to complement information from the absent modality.
We take the regularization image clients as example (similar procedure for text clients). Consid- ering the local training of image client c, at the beginning of each communication round, receives global text representations public data (
global E Rd, .., |P]. For the k-th image ((k), client first generates its representation loocal E Rd. However, current Local may be heavily bi- ased due to: 1) local training data is non-i.i.d. sampled, similar traditional FL; 2) local training is performed under single image modality, lacking not only the language information, but also the cross-modal alignment interaction, unique challenge in multimodal FI..
To address this problem, we resort to global text representations to guide uni-modal image clients head towards the unified multimodal representation space. We assume the global multimodal server has partially learned shared representation space between image and text representations.
enforce the uni-modal image clients to head towards the shared multimodal representation space, while keeping away from ((() other unrelated text data points Rlobal: 1,.... [P]. # contrastive loss:
By encouraging the coupling of local and its paired global data ((k) gglobal:
k. This is achieved by an inter-modal
exp
(k) T local
+
(k) globbll
(b) inter
log
IPI Σ)
(()) |ocall
() Rlobal
)=1
exp
4

---

Published as a conference paper at ICLR 2023
Algorithm 1: CreamFl. algorithm.
Input: Public dataset P. number of communication rounds T, number of clients C, number of local epochs E, server model fa. model fe. dataset D. of the c-th client and fraction of clients p that perform computation in each round.
Output: The final server model w
ServerExecutes:
for 0, T do
glob gob [s(P; w(); Hocal. Tical + []:
generate global representations sets of local representations
S: < random set of max(p- C, clients;
for each client in S; do
b in parallel
send public representations Iglob. Tglob to client C; Ic, & Ciiennnooal t, Iglob. Tglob);
Local, Tocal f- [occl + Le: Tical + Te:
10
end
11
Train server model w' using the public multimodal dataset P:
I,T < Aggregate Local: Tocal with Lglob. Tglob according to Equation 6
12
wi+1 <- w¹ distills knowledge from aggregated I, accroding to Equation
13
14
end
15 ClientLoocalTrainng
4, I glob: T glob):
for epoch = 0, E do (t,1)
regularized local training
(1) +- We 1)c V c(o) (De Iglbb: Tglob; (())))
17
Equation
18
end
Ie, To (P::
generate local representations
return Ie.To;
Through regularization by this contrastive objective, we not only inform the local image client the existence of another modality, but also regularize it towards the shared multimodal representation space to learn better cross-modal interaction.
3.2.2 INTRA-MODAL CONTRAST
The task gap between uni- and multimodal clients arises naturally in multimodal FL. For example, an image client is trained by image classification task while a multimodal client is trained under cross-modal retrieval, which can induce severe model drift because these clients are trained towards different targets. To alleviate such drift, we introduce an intra-modal contrast to regularize local representations towards their global consensus, in each separate modality.
Concretely, image client receives global image representations {(0) 'global- 1...., |P] from the server the beginning of each communication round. We contrast the representation of the k- (Ak) th image locaal to its corresponding global version ((k) gglobal to guide local model towards the global
consensus. Following MOON LLi et al. 2021), we add negative contrast between 2 and the
representation generated by the local model of the last round .(t)) to increase the distance between current local model and the previous one. Different from MOON, our intra-modal contrast uses public data as bridge without operating on private data, thus can be seen as an extension MOON
to more scalable KD-based FL framework. The intra-modal contrastive loss is defined as:
(k) T
(R)
exp
local
"global
(k) intra
log
(K) Jocal
((k) scloball
((k) Local
(k) *pree
exp
+ exp
5

---

Published as a conference paper at ICLR 2023
The final objective of client regularized by inter- and intra-modal contrasts be written as:
Clocal
1 [Iel Σ el y w.) + + [P]
Σ
g(k) iitter
Σ
e
(3)
With a slight abuse symbols for simplicity, ((= V.: We) is the objective the original local training task, y is the coefficient of our proposed regularization terms for mitigating local drift. After local training, clients generate representations of public data according to their own modality (e.g., image client only generates image representations while multimodal clients generate both image and text representations) and transmit them to the server. The server aggregates received representations and then distills knowledge from the aggregated version.
3.3 GLOBAL-LOCAL CONTRASTIVE AGGREGATION (GCA)
Representation aggregation has been underexplored before, either in the area of FL. or in ensemble knowledge distillation. As many representations are highly biased even maliciously attacked, the server has to selectively aggregate them, and the key challenge is to precisely evaluate the quality of representations to decide which one should contribute more during ensemble.
In the context of multimodal learning, we answer this question by designing global-local cross- modal contrastive score for weighting purposes. Generally, local representation that is "close" to its paired partner' global representation generated by server model and resides far away from other unpaired samples, always better captures the semantic information and cross-modal interaction of data. In the meantime, it often causes less local drift because it is benchmarked to the global repre- sentations that contain information from all clients.
3.3. GLOBAL-LOCAL CROSS-MODAL CONTRAST
For the k-th image-text pair ((()) of public dataset P, before aggregating the received local a(,) {(()
representations, the server first generates the global representations of public data gllbbal 'global
E I, ..... IP| by the current server model ('Global Img/Txt Rep.' in the '@) Global Represen- tation Aggregation path of Figure 2 Taking the aggregation of the k-th local image representations .kkcc
Clocal E [C + M] as example, the superscript (k: c) denotes is the k-th image representation from the e-th client. Complying with the golden pursuit of multimodal learning, we assign higher
weight aggregation to the local representation ((fec) that better matches its counterpart's global (i) representation 'global and less approximates other texts globall j * k. We achieve this by the spirit of contrastive learning (He et al. 20206] Chen et al.] (020a),, and compute the score of each local image representation the form of contrastive loss, follows:
( local
(k) global
exp
(k,e) -log
P ==1
(k.e) |ooaal
(i
[ exp
'global
This contrastive-based metric emphasizes representations similar to its paired image & global (bigger nominator) and different from other incorrect pairings {()
(k)
'global (smaller denominator) in aggregation.
We softmax for normalization and aggregate local image representations 2((kac) as: a (k.1), (k.2) a (k.C) softmax(( (R.11), (k:22) st..,,((.C)) (k,,0)
(5)
((R) Σ a (k.c) (kke)
llccl
(6)
3.3.2 KNOWLEDGE ENSEMBLE TRANSFER
After aggregation, the server model distills knowledge from the clients by minimizing the &g distance between the output of the server fi((); w) and the selectively aggregated representation (());
IP]
((((k): w) -240
w = w

---

Published as a conference paper at ICLR 2023
where is the learning rate. Our global-local cross-modal contrastive aggregation strategy benefits model learning from two perspectives: 1) by contrasting local representations to global represen- tations that serve as consensus. the server can filter out outlier representations that drift too far, to mitigate the adverse impact induced by local model drift; 2) by contrasting to the paired data from another modality, our strategy can pick out representations that align better with its paired data, which helps the server model learn better multimodal representations.
4 EXPERIMENTS
To evaluate the proposed FL framework in multimodal setting, approximate a real-life scenario (similar to Figure D where clients provide both uni-modal (images, text) and multimodal data (images tagged with descriptions). These diverse data sources are private, and cannot be disclosed during knowledge communication with the central server. Client models can handle either uni-modal tasks (c.g., image classification, text classification) or cross-modal tasks (e.g.. multimodal retrieval). The goal is to apply federated learning over these heterogeneous clients to train larger modal that can handle multimodal tasks on the server (e.g., multimodal retrieval). Sec. [4. describes the datasets and models used in our experiments to mimic this clients/server setting. Sec. [.2 explains the evaluations on comparing our CreamFL framework with state-of-the-art FL methods. Sec. 4.3
and Sec. provide further ablation studies and qualitative analysis on the model drift problem.
4.1 EXPERIMENTAL SETUP
Datasets The server model is evaluated cross modall retrieval and visual question answering tasks. We randomly choose a subset of MS-COCO (Lin et al. 2014) with 50,000 image-text pairs as public dataset. Details about datasets and evaluation are deferred to Appendix A.2] We distribute Flicker30K (Plummer et al.] 2015) to 15 multimodal clients, CIFARION (Krizhevsky et al., (2099) to 10 unimodal image clients, and AGNEWS (Zhang al. 2015 to 10 unimodal text clients, using Dirichlet distribution (a=0.1) for non-IID data partition (Hsu et al. 2019). We randomly choose 10 from the total 35 clients to participate in training in each communication round.
Baselines We compare CreamFL with state-of-the-art FL approaches including: 1) Fe- dAvg (McMahan et al. 2017). 2) FedIoT (Zhao et al.] 2022). 3) FedMD (Li & Wang 2019), 4) FedET (Cho al. 2022). and 5) FedGEMS (Cheng et al. 2021). To demonstrate the effectiveness of our global-local contrastive aggregation (GCA) and local contrastive regularization (LCR), we further compare CreamFL with its two variants, which exclude the contrastive components (GCA & LCR) and instead use vanilla representation ensemble. We termed it "reamF1." (Cream|LL with- out *C"). With different aggregation strategies, we have two additional baselines: 6) reamFL+Avg, which uses the same settings CreamFL but without LCR and GCA, and the number of local sam- plcs is used for weighting in representation aggregation, same as FedAvg; 7) reamFL+IoT, which is similar the first variant except that higher weights (set 100 following /Zhao et al. 2022 assigned to multimodal clients in representation aggregation, same as FedloT.
All uni-modat algorithms extended to multimodal scenarios by operating each modality sepa- rately, e.g., FedGEMS performs per-modality local distillation and entropy-based selective aggrega- tion. For FedAvg and FedloT, we distribute public data multimodal clients and adopt client model of the same size as our server model for evaluation. For FedMD, we add the same large server model as CreamFL for comparing the server performance, keeping its averaging aggregation and local dis- tillation unchanged. We choose ResNet- (He et al. 2016) and ResNet-18 as the server and client image models, respectively, and BERT (base) (Devlin al. 2018) and GRU (Chung et al. 2014) as the text models. The representation dimension d is 512 for both image and text. We use Adaml optimizer with initial learning rate 0.0002 and cosine learning rate scheduler for server model.
4.2
MAIN RESULTS
Table presents Recall 1, Recall@5 and Recall@ 10 scores of all baselines and our method in multimodal retrieval task on IK and 5K test sets of MS-COCO. As shown, CreamFL framework achieves noticeable performance improvement over all baselines in all settings (on both image-to- text retrieval (i2t) and text-to-image retrieval (12i) tasks).
7

---

Published as a conference paper at ICLR 2023
Table 1: Comparison of CreamFL with baselines image-text retrieval task.
Test Images
Types
Methods
12t 22_R68 R
12t.R@5
i2t.R @ 10
12i @
12.R@5
12i R 10
R@ suum
Model
FedAvg
45.23
76.74
85.59
34.69
71.68
85.40
114.03
Homogeneous
FedIoT
43.31
75.62
86.26
33.94
70.09
84.56
FedMD
48.40
80.24
89.64
38,23
74.44
86.68
128.33
Model
FodET
48.76
80.39
89.73
38.39
74.68
86.76
129.11
Heterogencous
FedGEMS
48.70
80,48
89.62
38.71
74.75
87.01
129.70
(w/ larger
reamFI.+Avg
48.85
80.55
89.93
38.13
74.89
86.79
130.02
server model)
reamFI.+HoT
49.13
80.61
89.69
38.45
74.83
86.74
130.41
CreamFL (ours)
49.66
80.66
90.13
38.94
75.02
87.14
132.88
SK Test Images
Types
Methods
12t R
12: R
R
12i R I
(2i R
12i R 10
Model Homogeneous
FodAvg
20.73
51.53
63.96
13.38
37.88
58.29
FodIoT
20.64
50.81
63.45
13.30
38.49
56.23
FedMD
23.95
53.23
66.29
17.73
44.46
58.47
Model
FedET
24.08
53.42
66.31
17.88
44.59
58.65
Heterngeneous
FedGEMS
24.27
53.56
66.06
18.09
44.58
58.05
(w/ larger
reamFL++Agg
24.79
53.53
66.90
18.25
44.46
58.26
model)
reamFL+ooT
24.63
53.50
66.93
18.20
44.58
58.44
CreamFL (ours)
25.34
53.62
66.95
18.94
44.68
58.82
In comparison FedAvg and FedloT, CreamFL. exhibits superiority both model performance (132.88% v.s. 114.03% & 111.19% on R@ suum) and communication efficiency (Section 4.3). demonstrating the effectiveness and efficiency of the KD-based CreamFL framework. Compared with reamFL+HoT, which assigns higher weights to multimodal clients in aggregation, our method yields consistently better performance (132.88% v.s. 130.41 % @ __sum score). This shows our global-local contrastive aggregation method for multimodal data is more effective. It is also more efficient as multimodal weights are computed automatically and no manual tuning is needed.
Comparing to FedMD, FedET and FedGEMS (algorithms with both aggregation and local reg- ularization components), CreamFL also achieves superior performance (132.88% v.s. 128.33% 129.11% & 129.70% on R@ 1_sum score). This demonstrates the effectiveness of our contrastive aggregation strategy and regularization method. The inferior performance of FedMD indicates that directly taking average of representations uni-modal non-i.i.d scenario, results in much worse performance multimodal settings due to harsher model drift where simple average fails.
ensemble, which may yield satisfying performance in
is also worth noting that comparable performance of FedET and FedGEMS to FedAvg (KD) shows aggregation methods designed for logit with probabilistic properties (FedGEMS uses entropy and FedET uses variance logit for weighting) may no longer have advantage in the context of aggre- gating representation, because representation does not share the same distribution nature logit, revealing the limitation of logit-specific aggregation methods. We further validate the effectiveness
Table 2: VQA evaluation.
Mulimodal
Maliinodal
Image
Image chemi
Image
Image
Methods
Acc.
FedAvg
52.54
FedloT
53.06
FedMID
57.43
FedET
59.90
FedGEMS
60.23
reamFL+Avg
58.64
(a) Clients trained under vanilla reamFL+Mean
(b) Clients trained under CreamFLL
reamFL+IoT
59.64
CreamFL (ours)
62.12
Figure 3: T-SNE visualization (Van & Hinton 2008 model drift.
8

---

Published as a conference paper at ICLR 2023
of CreamFL on VQA task, and present the results in Table 2 CreamFL yields an accuracy gain by 1.89% compared to the best baseline (62.12% v.s. 60.23%)
Table 3: Results on ablation studies.
4.3 ABLATION STUDIES
Methods
R I_sum
Table 3 studies the effect of each CreamFL component. reamfl L+Mean adopts vanilla average as the represen- tation aggregation strategy (all other settings the same as reamF1++Avg). PeamFI+GCA replaces aggregation method with our strategy. I.CR.inter and inCr, re- fer to two regularization techniques 3.2 for local client training. Results show reamFL+GCA not only sur- passes vanilla average by large margin (130.82% v.s. 127.65% but also outperforms all baselines in Table showing the effectiveness of our aggrega- tion strategy. When local training regularization is added, the global performance can be further improved. LCR. inter outperforms LCR. intra by 0.41% % (131.41% 4 v.s. 131.02%) demonstrating that complementing information for the absent modality for uni-modal clients is critical in heterogeneous multimodal setting. Combing cevffintra-modda contrasts to regularize local training yields further performance boost.
reamFL+Mean
127.65
reamFL+Avg reamFL+loT
130.02
130.41
reamFL+GCA
130.82
reamFL+GCA + inR.
131.41
reamFL+GCA + LCR.
131.02
CreamFl. (CCA
132.88
32
64
128
Figure shows the relation between communi- cation cost and model performance. Two fac- tors affect the communication: number of pub- lic data (num) and dimension of representa- tion (dim). We vary each separately to control the communication. Specifically, for the blue lines, we fix dim as 512 and vary num, and vice versa for the orange line where num is fixed to 50k. X-axis is the communication cost in each round, with dim labeled on the upper axis and num the lower. Each data point is the final model performance under the specified setting. Generally, FedAvg and FedIoT exhibit inferior performance to CreamFL with higher commu- nication cost, and the amount of public data (blue solid line) has greater impact on performance than representation dimension (orange solid line). Other baselines also transmit representations on public data and have the same communication cost with CreamFL. Complete analysis is deferred to Appendix A.5
132.5 KR
130-
125.97
127.76
120.32
121.22
120
FodAvg
R&l sum
110.35
110-
FodioT
100
90
(din Dimeraio ESK puls dea)
81.22
OwantF
80
reanFF- -207
10k
20k
30k
40k
50k
Communication cost (higher on right)
Figure Communication v.s. performance.
4.4 QUALITATIVE STUDY ON MODEL DRIFT
Figure 3 visualizes representations of 250 randomly chosen images from COCO, generated by multimodal client (red circle) and 2 different image clients (blue star and green triangle). (a) is vanilla case where aggregation is simple average and no regularization is exerted on local train- ing. We first observe that model drift exists between two modality-identical image clients (blue and green), in line with the observation in Karimireddy et al. (2020). Besides, model drift between uni-modal clients (blue v.s. green) is much smaller compared to the gap between multimodal and uni-modal clients (red v.s. blue+green), confirming our claim that modality gap and task gap cause further model drift problem in multimodal FL. (b) shows CreamFL pulls the representations gener- ated by different clients closer and effectively mitigates such drift.
CONCLUSION
In this paper, we investigate multimodal federated learning and propose new framework CreamFL that enables training larger server models and allows heterogeneous clients, via communicating knowledge public dataset without disclosing private data and models. We transmit representations between server and clients and design a novel global-local cross- modal ensemble strategy with inter- and intra-modal contrastive regularization in local training mitigate model drift.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work was supported by the National Key R&D Program China under Grant No.20222D0160500 Xiaomi Al Innovation Research under grant No.202-422-002. We would also like to thank anonymous reviewers for their insightful comments,
REFERENCES
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics, pp. 2938-2948, PMLR, 2020.
Peva Blanchard, EI Mahdi EI Mhamdi, Rachid Guerraoui, and Julien Stainer, Machine learning with adversaries: Byzantine tolerant gradient descent. Advances in Neural Information Processing Systems, 30, 2017.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint trXiv:2108.072 2021.
Alberto Brunete, Emesto Gambao, Miguel Hernando, and Raquel Cedazo. Smart assistive archi- tecture for the integration iot devices, robotic systems, and multimodal interfaces in healtheare environments. Sensors, (6(2222 2021.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, pp. 104-120. Springer, 2020b.
Sijie Cheng, Jingwen Wu, Yanghua Xiao, and Yang Liu. Fedgems: Federated learning of larger server models via selective knowledge fusion. arXiv preprint 1rXiv:2110.11027, 2021.
Yae Jee Cho, Andre Manoel, Gauri Joshi, Robert Sim, and Dimitrios Dimitriadis. Heterogeneous ensemble knowledge transfer for training large models federated learning. 125Y::2204.17733. 2022.
arXiv preprint
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1
2018.
Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, Jianfeng Gao, et al. Vision-language pre-training: Basics, recent advances, and future trends. Foundations and Trends@ in Computer Graphics and Vision, 14(3-4):163 133-3552,, 2022.
Robin Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client 2017.
level perspective. arXiv preprint arXiv:
Yash Goyal, Tejas Khot, Douglas Summers-Stay. Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Federated learning of large cnns at the edge. Advances in Neural Information Processing Systems, 33: 14068-14080 2020a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
10

---

Published as a conference paper at ICLR 2023
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9733 2020b.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown, Measuring the effects of non-identical data distribution for federated visual classification, arXiv preprint
2019.
Sohei Itahara, Takayuki Nishio, Yusuke Koda, Masahiro Morikura, and Koji Yamamoto. Distillation-basee semi-supervised federated learning for communication-eficicni collaborative training with non-iid private data. IEEE Transactions Mobile Computing, 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904-4916. PMLR, 2021.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning, Foundations and Trends@ in Machine Learning, 14(1-2):1-210, 2021.
Sai Praneeth Karimireddy, Satyen Kale. Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pp. 5132-5143. PMLR, 2020.
Andrej Karpathy and Li Fei-Fei. Deep semaali-emantic alignments for generating image descrip- tions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Daliang Li and Jumpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03500 2019.
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10713-10722, 2021.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedingy of Machine Learning and Sys- lems, 2:429-452 2020.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351-2363, 2020.
Tsung-Yt Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Yujun Lin, Song Han, Huizi Mao, Yu Wang. and William J Dally. Deep gradient compression: Re- ducing the communication bandwidth for distributed training. arXiv preprint arXiv:1712.01888/ 2017.
Fenglin Liu, Xian Wu, Shen Ge, Wei Fan, and Yuexian Zou. Federated learning for vision-and language grounding problems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 11572-11579, 2020.
Yuzhu Mao, Zihao Zhao, Guangfeng Yan, Yang Liu, Tian Lan, Linqi Song, and Wenbo Ding. Communication federated learning with adaptive quantization. ACM Transactions on Intelligent Systems and Technology (TIST), 13(4):1-26. 2022.

---

Published as a conference paper at ICLR 2023
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-cffin efficceent learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282 PMLR, 2017.
Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting unintended feature leakage in collaborative learning. In 2019 IEEE symposium on security and privacy (SP), pp. 691-706. IEEE, 2019.
SkongUk Park and Nojun Kwak. Feature-level ensemble knowledge distillation for aggregating knowledge from multiple networks. In ECAI 2020, pp. 1411-1418. IOS Press, 2020.
Bryan A Plummer, Liwei Wang. Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region- -t--phrase correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp- 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763 PMLR, 2021.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Mar- cus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 15638-15650, 2022.
Laurens Van and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning re- search, 9(11), 2008.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3156-3164, 2015.
Chuhan Wu, Fangzhao Wu, Ruixuan Liu, Lingjuan Lyu, Yongfeng Huang, and Xing Xie. Fedkd: Communication efficient federated learning via knowledge distillation. arXiv preprint arXiv:2108. 13323, 2021.
Baochen Xiong, Xiaoshan Yang, Fan Qi, and Changsheng Xu. A unified framework for multi- federated learning. Nearocomputiing. 480: 100:110-111 2022.
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Trans. Intell. Syst. Technol, 10(2), jan 2019. ISSN 2157-6904. doi: 10.1 10.1145/3298981. URL[httss: /ttps:///do... http://...orr///// 1145/1141999
Dong Yin, Yudong Chen, Ramehandran Kannan, and Peter Bartlet.. Byzantine-robust distributed learning: Towards optimal statistical rates. In International Conference on Machine Learning, pp. 5650-5659. PMLR, 2018
Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta. Multimodal contrastive training for visual representation learning. In Proceedings the the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 6995-7004, 2021.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015.
Yuchen Zhao, Payam Barnaghi, and Hamed Haddadi. Multimodal federated learning on iot data. In 2022 IEEE/ACM Seventh International Conference on Internet-off-Things Design and Implemen- tation (IoTDI), pp. 43-54. IEEE, 2022.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint
2018.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Advances in neural infor- mation processing systems, 32, 2019.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1
REGARDING THE USE OF PUBLIC DATA
Legal and copyright issues about the public data should be carefully taken into account in real-world applications. Obtaining permission is necessary before using public released data for commercial purposes. In general cases, the public data is chosen from publicly released datasets that are widely accessible. Clients normally acknowledge which public dataset the server uses, and can directly download them from public sources. For companies to apply this system, collecting their own public data is also possible. Regarding compliance and users' awareness, real-world applications (such as Google Assistant) often use FI. on end clients who are opt-in [1] Our KD-based FI. framework can be implemented in similar manner.
A.2 DATASETS
A random subset of COCO Lin et al. 2014) with 50,000 image-text pairs is selected as the public multimodal data. CIFAR 100, AG NEWS, and Flicker30k are used as the private datasets for image,
text and multimodal clients, respectively. CIFAR-100 (Krizhevsky al., 2009) consists of 50,000 colored training images in 100 classes, with 500 images each class, AG NEWS (Zhang et al. 2015) contains 120,000 training sentences from 4 classes. Flicker30k (Plummer
al. 2015) contains
31,000 images collected from Flicker, together with 5 captions per image, i.e. 155,000 image-text pairs in total. For moosl retrieval, we follow Karpathy & Fei-Feil (ei-Fei((2015) and report Recall K results on 5K/IK test sets of MS-COCO, which measures the percentage of times a correct item being found among top-K results. For visual question answering, we use VQA v2.0 dataset (Goyal et al. 2017) and report accuracy over the 3,000 most frequent answers.
A.3 DISCUSSIONS ABOUT REPRESENTATION ENSEMBLE DISTILLATION
Representation ensemble distillation has been an under-explored problem. Park & Kwak (2020) looked into this problem and their student network trained to mimic the representations of all teachers after a non-lineat transformation layer, but there is no active selection of teachers. This implausible in the context of FL, because different clients (teachers) are highly heterogeneous in data distribution, and the quality of representations varies a lot (even malicious representations may exist). Thus, selectively aggregating these representations before performing representation-leve distillation is required, and that is why propose aggregation strategy address this challenge.
We believe that representation aggregation ensemble is an important and unsolved problem in FL, especially when the community is embracing big foundation models (Devlin et al. 2018 [omm masani et al. 2021), where the server model in FL. will need to learn foundational universal repre- sentations from heterogeneous data sources and diverse model architectures.
A.4 LARGER DATA CAPACITY
We increase the number of public image-text pairs to 100,000 (also random subset of MSCOCO), to evaluate under larger data capacity. Results in Table[[shov that CreamFL. remains effective the larger-scale setup. It is worth noting that the communication cost will increase linearly proportional to the public data number. Thus, improving model performance under the setting with less amounts of public data is important future direction.
A.5 TRADE-OFF BETWEEN COMMUNICATION AND PERFORMANCE
Trade-off results between communication and performance W.E.t. the number of public data samples for reamFL+loT (the KD version of FedloT), plotted as the blue dashed line in Figure 4 To further investigate the trade-off for other FL. settings which transfer model parameters (such FedAvg and FedloT)) we vary the model architecture of server model from ResNet101 to ResNet50 and ResNet34. Results show that CreamFL (orange and blue solid lines) exhibits better trade-off compared with these frameworks.
13

---

Published as a conference paper at ICLR 2023
Table 4: Results under larger data capacity.
Methods
R@ I_sum
FedGEMS
1.1.85
FedET
151.68
reamFL+Avg 151.83
reamFL+loT
151.57
CreamFL
152.50
A.6 VISUALIZATIONS MODEL DRIFT
Figure visualizes representations of 250 randomly chosen captions from COCO, generated by multimodal client (red) and 2 different text clients (blue and green). Figure (a) is vanilla case where clients are trained under reamFL+ Meaan and clients of (b) are trained under CreamFL.. Similar the observe that model drift exists between two modality-identical identical text clients (blue and green), while this drift is much smaller than the gap between multimodal and uni-modal clients (red blue+green), confirming our claim that modality gap and task gap cause more severe model drift problem in multimodal FL. CreamFL effectively mitigate this drift shown in (b).
observation in Figure g image representations,
Malemrda chert
Malumatai chere
Text dian
Tesreleen
Test dkan
Test clisst
vanilla reamFL
traindd CreamFL
Figure 5: T-SNE visualization of text representations on model drift.
Clian!
Clkmn
Cleur2
Client Ing
Clent Ing
Clian22 Ins
Clien22
(a) Clicnts trained under vanilla L-aan
(b) Clicnts trained under CreamFL
Figure 6: T-SNE visualization of multimodal representations model drift.
Further, representations of 250 randomly chosen image-text pairs from COCO are visualized in Figure 6. generated by two different multimodal clients. In Figure G(a), clients are trained under reamFL+Meat and obvious drift on image representations can be observed (blue points are far away from orange ones). CreamFL effectively mitigate this drift shown in (b) (blue and orange points are pulled together). Both frameworks exhibit less model drift on text representations multimodal clients (red and green points).
14

---

Published as a conference paper at ICLR 2023
A.7 RELATED WORK
Federated Learning McMahan et al. 2017) is a privacy-conseious alternative to centralized ML, allowing multiple clients collaboratively train models without compromising privacy. Existing FL work mainly focuses on: 1) improving server performance under complex non-iid scenarios (Li
al., 2020 Zhao al., 2018 Karimireddy et al. 2020); reducing communication cost from limited wireless bandwidth Lin et al. [2017] Mao et al. 2022]; and 3) protecting privacy (Melis al. 2019 Zhu al. 2019; Geyer al. 2017 ) and enhancing server robustness against malicious clients (Yin et al. 2018 Bagdasaryan et al.) 2020 Blanchard et al., 2017). Numerous aspects need
to taken into account build a practical FL system, our work focus primarily on tackling the heterogeneity in multi-modal scenarios.
15


Reviews:
[{"review_id": "HLxObGdaJC", "reviewer": "Reviewer_U8Yh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes Contrastive Representation Ensemble and Aggregation for Multimodal\nFL (CreamFL) to exploit multimodal data from clients in FL settings in a privacy driven world. CreamFL trains large models from clients with heterogeneous architectures and multiple modalities. To fuse multimodal representations, a global-local cross-model ensemble strategy is proposed. \n", "summary_of_the_review": "Overall, the paper is a decent attempt to address a complex problem however, given the concerns highlighted in the strengths and weaknesses, would like to weight for the response from the authors before changing my scores.", "strength_and_weaknesses": "# Pros:\n\n- The paper is well-written, easy to understand and follow along.\n\n- The more interesting aspect is the way the modality and task gaps are treated through regularized losses and the ensemble strategy, that is the global-local and cross-modality contrastive aggregation.\n\n- The aggregation in a heterogeneous setting, where some clients are uni-modal and some are multimodal is also complex and the proposed approach seem to deal with such complex scenarios.\n\n- Literature is covered adequately.\n\n# Cons:\n\n- Serious concern, use of public data is questionable here from a real world perspective. When the client data is private and public data has licenses in place one has to make sure they comply since the clients are unaware of this public data usage at the server or even worse some of the data seem to get transferred to clients as well, if not mistaken from the explanation in section 4.\n\n- The proposed regularizations in the loss sounds more useful for a multimodal client rather than a unimodal client. \n\n- If you already know a given client is image-modal then you can make use of the global image representations and train on them and communicate only them back, what is the point of using text-modal representations in that case at all. Similarly for clients with text only modality?\n\n- In other words, what is the point of l_{inter} how does pushing the uni-model representations towards multi-modal representations help close the multi-modal gap? This obviously going to degrade the performance on the image tasks (text in text case).\n\n- What is this with and without larger server model in Table 1, tried to find it else where in the text the difference can not identify, needs clarity on this.\n\n- The theme is to improve performance on multimodal test sets, with reamFL+IoT, where you increase the multimodal client weightage, the performance is surprisingly worse than the reamFL+Avg, any reasons?\n\n- At any point of time, be it the server or the client, the technique seem to rely on storing two or more copies of the global and client models, which warrants access to more resources which opposes the motivation in the Introduction that clients are often resource constrained. \n\n- This approach is to train representations in FL in multimodal context, then the details about how the fine-tuning is done and how long it is done and what is the data used for fine-tuning on the down-stream tasks is missing.\n\n- The real ablation that will be interesting to see is how does CreamFL perform when you decrease the uni-modal clients, further you can permute with the number of image and text clients when you decrease the total number of uni-modal clients. Then, what if you mute the l_{inter} on the corresponding uni-modal clients but still use the creamFL framework?\n\n\n- How is the ensemble strategy different from the FedAvg style except that you use L2 for distillation?, in which case, why should this be titled or called an ensemble?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CdyD8CTN5m", "reviewer": "Reviewer_KYph", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a knowledge-distillation-based federated learning framework, CreamFL, for a multimodal setting where clients may hold different combinations of data modalities. The inter-modal and intra-modal contrast losses are used as a regularization term to mitigate the model drifting issue, which is called the modality gap and task gap. The CreamFL utilizes the representations of a public dataset to enhance the aggregation strategy and achieves a state-of-art performance in the multi-modal setting with heterogeneous modality and model architectures.\n", "summary_of_the_review": "This paper connects clients and servers with different target tasks through a public dataset, thereby enabling knowledge distillation to train a larger global model from each client. The extended contrast idea utilizes the global representation and local representation to overcome the novel modality heterogeneity challenge. While there is a lack of well-established experiment design to demonstrate the superiority of the model on communication cost tradeoff and privacy protection. Also, I am concerned about the technique's novelty as well. I would not think it reaches the acceptance bar of this conference.\n", "strength_and_weaknesses": "Strength:\n1. This paper provides insight into the new heterogeneity challenges caused by modality heterogeneity, which is unique for multi-modal federated learning. The authors emphasize on modality gap and task gap between uni-modal clients and multi-modal clients.\n2. By introducing a public dataset in the server, the proposed method further extends the idea of contrastive representation among the local representations to the contrast between the global representation and local representation. The contrastive information is used for local training and global aggregation.\n3. The proposed knowledge ensemble transfer schema utilizes the aggregated local representation to update the global model parameters. This overcomes the challenge that the client owns different model architectures. \nWeakness:\n1. Incomplete experiment design. For the analysis of communication cost and model performance part, the communication cost of both model parameters and transmission of the public dataset for baseline models is expected to be given. As the local models and global model changes, the tradeoff between communication cost and performance is underexplored.\n2. From my understanding, the main technique part is based on KD and ideas from contrastive learning. I got the point that the statement saying these techniques are applied in multimodal FL is interesting. However, the novelty is still the most concerning for me.\n3. For the analysis of model drift, the authors only illustrate the modality gap for the image representation. The analysis of text representation and multi-modal representation is expected. \n4. Lack of analysis of how the composition of the clients will affect the global model performance. \n5. The introduction of the public dataset will endanger the system's security and privacy protection. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "BSZYYfw3F6", "reviewer": "Reviewer_FnzN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes comparative representation integration and aggregation for multimodal FL (CreamFL). In particular, the global-local cross-modal contrastive aggregation strategy is used to achieve better multimodal representation fusion. Meanwhile, regularizing local training is employed to mitigate local model drift caused by modal differences and task differences. CreamFL is the first multimodal FL framework based on knowledge distillation that supports heterogeneous schemas and model architectures between servers and clients, while exchanging private knowledge only on public datasets, without leaking private models and data. And CreamFL can train larger models on the server and absorb modality-diverse knowledge from resource-constrained clients.", "summary_of_the_review": "The article proposes a comparative representation integration and aggregation of multimodal FL based on Knowledge Distillation, which supports heterogeneous schemas and model architectures between servers and clients and can absorb model-diverse knowledge from resource-constrained clients. And the proposed approach allows the exchange of private knowledge on public datasets without disclosing private models and data. The innovative points of the article are novel and practical, the writing is logical and clear, and the experiments are adequate and promising. To further improve the quality of this manuscript, it is hoped that the authors can add some theoretical analysis and comparisons with other image-text multimodal learning algorithms.", "strength_and_weaknesses": "Strengths: The authors propose contrastive representation ensemble and aggregation for multimodal federated learning. The innovative points of the article are novel and practical, the writing is logical and clear with beautiful figures and adequate experiments.\nWeaknesses: \n(1)\tThis paper focuses on image-text multimodal learning, and the authors should fully describe the image-text multimodal learning algorithms and analyze the similarities and differences with the algorithms in this paper, such as the differences between the intra-modal contrast learning and inter-modal contrast learning in other algorithms and this paper, and the differences between other multimodal representation fusion algorithms and this paper.\n(2)\tThe authors point out that existing multimodal FL methods limit the global model in terms of model complexity and data capacity, so it is desired that the authors increase the model complexity and data capacity analysis of the proposed method and compare it numerically with other algorithms. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0NdJTBEONSO", "reviewer": "Reviewer_LDTA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this submission, the authors proposed a heterogeneous federated learning framework to handle the heterogeneous model architectures and data modalities for diverse training clients. Specifically, the proposed CreamFL has a novel cross-modal contrastive feature aggregation strategy and the inter-/intra-model contrastive objective to address the model drift for multi-modal federated learning. The experiments show the advances of the proposed method on efficiency and performance in multi-modal settings for federated learning.", "summary_of_the_review": "Please see the Strength And Weaknesses.", "strength_and_weaknesses": "Weakness: \n\n1. The main results are not quite enough. For instance, the discrepancy between public datasets and private datasets is not taken into account and hence more public/private datasets should be evaluated. What’s more, more practical and complex scenarios could also be tested, e.g., assigning different private datasets for different image/text/multi-modal clients.\n\n2. What are the communication costs of FedMD, FedET and others? Are they more efficient than CreamFL?\n\n3. The ablation results are not complete, e.g., the results of the reamFL+LCR are missing.\n\nStrength:\n\n1. The submission proposed a novel heterogeneous multimodal federated learning framework firstly and achieves a more promising result than prior works. Multi-modal federated learning is a subarea under exploration and this work firstly proposes an attempt to extend K-D distilled-based method to multimodal federated learning.\n\n2. A novel feature aggregation strategy, namely GCA, and local-global contrastive objectives, namely LCR, are designed for ensemble knowledge distillation and addressing the model drift for multi-modal federated learning, which is quite interesting for multimodal federated learning.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster