Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this submission, the authors proposed a heterogeneous federated learning framework to handle the heterogeneous model architectures and data modalities for diverse training clients. Specifically, the proposed CreamFL has a novel cross-modal contrastive feature aggregation strategy and the inter-/intra-model contrastive objective to address the model drift for multi-modal federated learning. The experiments show the advances of the proposed method on efficiency and performance in multi-modal settings for federated learning.
strength_and_weaknesses: Weakness: 

1. The main results are not quite enough. For instance, the discrepancy between public datasets and private datasets is not taken into account and hence more public/private datasets should be evaluated. What’s more, more practical and complex scenarios could also be tested, e.g., assigning different private datasets for different image/text/multi-modal clients.

2. What are the communication costs of FedMD, FedET and others? Are they more efficient than CreamFL?

3. The ablation results are not complete, e.g., the results of the reamFL+LCR are missing.

Strength:

1. The submission proposed a novel heterogeneous multimodal federated learning framework firstly and achieves a more promising result than prior works. Multi-modal federated learning is a subarea under exploration and this work firstly proposes an attempt to extend K-D distilled-based method to multimodal federated learning.

2. A novel feature aggregation strategy, namely GCA, and local-global contrastive objectives, namely LCR, are designed for ensemble knowledge distillation and addressing the model drift for multi-modal federated learning, which is quite interesting for multimodal federated learning.

clarity,_quality,_novelty_and_reproducibility: The proposed method is based on the FedET and K-D based federated learning and the core of the method i.e., LCR and GCA, works fine and quite interesting for multimodal federated learning.
summary_of_the_review: Please see the Strength And Weaknesses.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a knowledge-distillation-based federated learning framework, CreamFL, for a multimodal setting where clients may hold different combinations of data modalities. The inter-modal and intra-modal contrast losses are used as a regularization term to mitigate the model drifting issue, which is called the modality gap and task gap. The CreamFL utilizes the representations of a public dataset to enhance the aggregation strategy and achieves a state-of-art performance in the multi-modal setting with heterogeneous modality and model architectures.

strength_and_weaknesses: Strength:
1. This paper provides insight into the new heterogeneity challenges caused by modality heterogeneity, which is unique for multi-modal federated learning. The authors emphasize on modality gap and task gap between uni-modal clients and multi-modal clients.
2. By introducing a public dataset in the server, the proposed method further extends the idea of contrastive representation among the local representations to the contrast between the global representation and local representation. The contrastive information is used for local training and global aggregation.
3. The proposed knowledge ensemble transfer schema utilizes the aggregated local representation to update the global model parameters. This overcomes the challenge that the client owns different model architectures. 
Weakness:
1. Incomplete experiment design. For the analysis of communication cost and model performance part, the communication cost of both model parameters and transmission of the public dataset for baseline models is expected to be given. As the local models and global model changes, the tradeoff between communication cost and performance is underexplored.
2. From my understanding, the main technique part is based on KD and ideas from contrastive learning. I got the point that the statement saying these techniques are applied in multimodal FL is interesting. However, the novelty is still the most concerning for me.
3. For the analysis of model drift, the authors only illustrate the modality gap for the image representation. The analysis of text representation and multi-modal representation is expected. 
4. Lack of analysis of how the composition of the clients will affect the global model performance. 
5. The introduction of the public dataset will endanger the system's security and privacy protection. 

clarity,_quality,_novelty_and_reproducibility: The structure of the article is clear and easy to understand. The experimental design answers the main considerations of the proposed method.
This article has made improvements and innovations in the methods of the past literature, and applied it to the challenging multimodal federated learning, showing a certain degree of innovation. However, the core techniques are not novel, though I understand the application scenario is interesting.
The author has uploaded their source code on the supplementary materials, which can help others reproduce the work. 

summary_of_the_review: This paper connects clients and servers with different target tasks through a public dataset, thereby enabling knowledge distillation to train a larger global model from each client. The extended contrast idea utilizes the global representation and local representation to overcome the novel modality heterogeneity challenge. While there is a lack of well-established experiment design to demonstrate the superiority of the model on communication cost tradeoff and privacy protection. Also, I am concerned about the technique's novelty as well. I would not think it reaches the acceptance bar of this conference.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes comparative representation integration and aggregation for multimodal FL (CreamFL). In particular, the global-local cross-modal contrastive aggregation strategy is used to achieve better multimodal representation fusion. Meanwhile, regularizing local training is employed to mitigate local model drift caused by modal differences and task differences. CreamFL is the first multimodal FL framework based on knowledge distillation that supports heterogeneous schemas and model architectures between servers and clients, while exchanging private knowledge only on public datasets, without leaking private models and data. And CreamFL can train larger models on the server and absorb modality-diverse knowledge from resource-constrained clients.
strength_and_weaknesses: Strengths: The authors propose contrastive representation ensemble and aggregation for multimodal federated learning. The innovative points of the article are novel and practical, the writing is logical and clear with beautiful figures and adequate experiments.
Weaknesses: 
(1)	This paper focuses on image-text multimodal learning, and the authors should fully describe the image-text multimodal learning algorithms and analyze the similarities and differences with the algorithms in this paper, such as the differences between the intra-modal contrast learning and inter-modal contrast learning in other algorithms and this paper, and the differences between other multimodal representation fusion algorithms and this paper.
(2)	The authors point out that existing multimodal FL methods limit the global model in terms of model complexity and data capacity, so it is desired that the authors increase the model complexity and data capacity analysis of the proposed method and compare it numerically with other algorithms. 

clarity,_quality,_novelty_and_reproducibility: The article has fluent language, clear writing logic, and certain innovation, while the author submitted the code for the implementation of the algorithm.
summary_of_the_review: The article proposes a comparative representation integration and aggregation of multimodal FL based on Knowledge Distillation, which supports heterogeneous schemas and model architectures between servers and clients and can absorb model-diverse knowledge from resource-constrained clients. And the proposed approach allows the exchange of private knowledge on public datasets without disclosing private models and data. The innovative points of the article are novel and practical, the writing is logical and clear, and the experiments are adequate and promising. To further improve the quality of this manuscript, it is hoped that the authors can add some theoretical analysis and comparisons with other image-text multimodal learning algorithms.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes Contrastive Representation Ensemble and Aggregation for Multimodal
FL (CreamFL) to exploit multimodal data from clients in FL settings in a privacy driven world. CreamFL trains large models from clients with heterogeneous architectures and multiple modalities. To fuse multimodal representations, a global-local cross-model ensemble strategy is proposed. 

strength_and_weaknesses: # Pros:

- The paper is well-written, easy to understand and follow along.

- The more interesting aspect is the way the modality and task gaps are treated through regularized losses and the ensemble strategy, that is the global-local and cross-modality contrastive aggregation.

- The aggregation in a heterogeneous setting, where some clients are uni-modal and some are multimodal is also complex and the proposed approach seem to deal with such complex scenarios.

- Literature is covered adequately.

# Cons:

- Serious concern, use of public data is questionable here from a real world perspective. When the client data is private and public data has licenses in place one has to make sure they comply since the clients are unaware of this public data usage at the server or even worse some of the data seem to get transferred to clients as well, if not mistaken from the explanation in section 4.

- The proposed regularizations in the loss sounds more useful for a multimodal client rather than a unimodal client. 

- If you already know a given client is image-modal then you can make use of the global image representations and train on them and communicate only them back, what is the point of using text-modal representations in that case at all. Similarly for clients with text only modality?

- In other words, what is the point of l_{inter} how does pushing the uni-model representations towards multi-modal representations help close the multi-modal gap? This obviously going to degrade the performance on the image tasks (text in text case).

- What is this with and without larger server model in Table 1, tried to find it else where in the text the difference can not identify, needs clarity on this.

- The theme is to improve performance on multimodal test sets, with reamFL+IoT, where you increase the multimodal client weightage, the performance is surprisingly worse than the reamFL+Avg, any reasons?

- At any point of time, be it the server or the client, the technique seem to rely on storing two or more copies of the global and client models, which warrants access to more resources which opposes the motivation in the Introduction that clients are often resource constrained. 

- This approach is to train representations in FL in multimodal context, then the details about how the fine-tuning is done and how long it is done and what is the data used for fine-tuning on the down-stream tasks is missing.

- The real ablation that will be interesting to see is how does CreamFL perform when you decrease the uni-modal clients, further you can permute with the number of image and text clients when you decrease the total number of uni-modal clients. Then, what if you mute the l_{inter} on the corresponding uni-modal clients but still use the creamFL framework?


- How is the ensemble strategy different from the FedAvg style except that you use L2 for distillation?, in which case, why should this be titled or called an ensemble?

clarity,_quality,_novelty_and_reproducibility: Clarity & Quality: Quality of the results are adequate however, a few details can be clarified as highlighted in the weaknesses 

Novelty: The proposed approach is novel enough.

Reproducibility: The presentation and the details provided are reasonable to reproduce however having a bit more clarity will improve the reproducibility.
summary_of_the_review: Overall, the paper is a decent attempt to address a complex problem however, given the concerns highlighted in the strengths and weaknesses, would like to weight for the response from the authors before changing my scores.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a multimodal federated learning framework, named Contrastive Representation Ensemble and Aggregation for Multimodal FL (CreamFL), to handle the heterogeneous model architectures and data modalities. In particular, a global-local cross-model ensemble strategy is proposed to fuse multimodal representations. Experimental results on multiple real-world datasets are reported and discussed. 

Overall, this paper is well written and easy to follow. The authors designed a novel framework to deal with multimodal discrepancy, i.e., modality gap and task gap. In addition, the authors provided a comprehensive overview of related work. The proposed knowledge ensemble transfer schema is able to tackle the challenge that the client owns different model architectures. However, during the discussion phase, there are still some remaining concerns, such as the use of public dataset, which should be further clarified in the final version of this work.  
justification_for_why_not_higher_score: The ideas of using knowledge distillation and contrastive learning for federated learning has been sufficiently explored in recent years.  
justification_for_why_not_lower_score: The paper studies an important problem and presents a new solution for KD-based multi-modal federated learning. The paper is well written, and the extensive experiments demonstrate the advantages of the proposed method over baselines.
Decision: Accept: poster