Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents an algorithm to train dual encoding models in federated learning. The key idea is sharing aggregated encoding statistical information across clients. Experiments were conducted on two benchmark datasets, including CIFAR-100 and Dermatology. The Dermatology dataset consists of de-identified images of skin conditions captured using consumer-grade digital cameras. The results show that the proposed algorithm achieves better utility than some baselines.
strength_and_weaknesses: The problem studied in this paper is interesting. However, the paper has serval major weaknesses as follows:

- The paper leverage privacy concerns to motivate the work using federated learning (no sharing data). However, the proposed approach shares statistical information from encodings, significantly increasing the privacy risk of the client's local data. Note that federated learning does not automatically offer privacy protection since the server can extract clients' local data from the shared information. Therefore, the motivation for the work is unclear.

- The proposed approach is a trivial adaptation of the existing CCO loss used for training dual encoding models in Zbontar et al. (2021). The technical contribution is not significant. What are fundamental questions or challenging problems the paper aims to address?

- Experimental results are unconvincing. The datasets used are small, and the model utility is shallow (<50\% in most cases). The practicability of the proposed approach and setting is unconvincing. Who is going to use models with such utility? In addition, the comparison is unfair. The proposed approach used significantly larger data to train the model than a minimal dataset to train the centralized model. That does not show the advantage of the proposed approach. Why don't we gather all the data to train a usable centralized model instead of sticking with a poorly federated learning model? Again, the privacy concerns are invalid in this setting since the proposed approach does not offer any privacy protection. What is the cost of the proposed method in terms of communication and computation? How do the encodings improve after and before using the proposed approach? The paper does not shed light on understanding the core contribution of the proposed approach. Why FedAvg? There are better aggregations to use. That highlights a poor treatment for critical components, such as privacy, model/data utility, practicability, and usability, of the proposed approach given the learning setting. 
clarity,_quality,_novelty_and_reproducibility: The novelty is limited. The proposed approach is a simple adaptation of the existing work in Zbontar et al. (2021). No code and the Dermatology dataset are provided; therefore, reproducibility is limited. In short, there is room for improvement, and the paper does not advance my knowledge in the field.
summary_of_the_review: This paper has limited contribution. The motivation of the work is unclear; the proposed approach is simple without offering a deep understanding of how it works. Experimental results are unconvincing, and a thorough evaluation and analysis are needed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors propose a domain-aware representation learning method (FedDAR) for the non-iid FL problem. The FedDAR assumes data on clients are from multiple domains and learns a classifier head for each domain. A representation module is shared for all classifier heads and updated by the vanilla FedAvg. The authors also proposed a second-order aggregation method to update domain-aware classifier heads, whose effectiveness is validated by ablation studies. Experiments on both synthetic data and real-world datasets validate the effectiveness of the proposed method.
 

strength_and_weaknesses: Strengths

1. The target problem is meaningful in real-world applications for FL.

 
 Weaknesses

1. This paper’s contribution is incremental.

2. Too much data sharing may lead to security and privacy concern, which is not discussed in the paper. Communication efficiency is another problem;

3. Eq.2 is not bounded between 0 and 1 when local stats are globally aggregated. 

4. One step of FL equivalent to central training will establish for almost all optimization methods, which can not prove the advantageousness of DCCO.

5. The work employed the local loss function of Barlow Twins, but replaced the calculation of C_ij (Eq.2) with correlation coefficient term. How can this replacement lead to better performance? Discussion and ablation experiments are needed.

6. Lack of experiments comparison with other self-supervised FL methods such as [1][2].

[1] Divergence-aware Federated Self-Supervised Learning

[2] Towards Communication-Efficient and Privacy-Preserving Federated Representation Learning

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper’s content organisation could be improved. 

Quality: The paper’s overall quality is below the threshold. 

Novelty: The novelty is fair. But the overall contribution to the FL community is limited.

Originality: The paper is an incremental work. 

summary_of_the_review: Please refer to the comments in Strengths and Weaknesses.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on the problem of training dual encoding models on decentralized datasets, which has few existing works. Moreover, the authors consider a challenging scenario that each client possesses a small and non-IID dataset, where directly utilizing the existing centralized methods decreases efficacy. Hence, the authors propose a novel method, Distributed Cross Correlation Optimization (DCCO), to train dual encoding models on decentralized datasets with federated learning. This paper provides several experimental results and proofs to show the advantages of DCCO.
strength_and_weaknesses: **Strengths:**
1. This paper provides a detailed description of the motivation. The authors discover the limitations of the existing methods and solve the problems under the limited environment that is closer to reality, thus making this work challenging and meaningful. 
2. The details of the proposed approach are well supported by texts and graphs, and the experiment settings are proper for proving the advantages of the approach.

**Weaknesses:**
1. Figure 2 presents the overview of the approach with separate graphs. However, the corresponding relations of the two graphs are vague. Figure 2 can be replaced by a single detailed graph of DCCO and explain the relations in the caption.
2. For the proposed method, the authors should give the algorithm for readers to understand better.
3. We would like to see the influence of other environmental parameters on experimental performance, such as $\lambda$.

clarity,_quality,_novelty_and_reproducibility: The motivation of the problem is clearly described in the introduction, and the proposed approach is easy to understand and well supported by texts and graphs. The paper focuses on a novel scenario of training dual encoding models, which is challenging since the existing methods perform poorly. The experiment settings are detailed, but there is no code in the supplementary materials.
summary_of_the_review: This paper proposes a novel method to solve the problem of training dual encoding models on decentralized datasets and proves its advantages on small and non-IID datasets over existing centralized methods. The details of the problem and the methods are clear and easy to understand for readers. However, there are still some problems that affect the fluency of the paper. The figures and the experiments should be supplemented to help readers grasp the idea and prove the generalizability of the approach. Overall, this work is meaningful in the area of dual encoding model training and can be further studied and optimized.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a distributed implementation of Cross Correlation Optimization (DCCO) loss for contrastive learning, experiments shows the proposed implementation of loss function outperforms some baseline algorithms in the federated learning setting.
strength_and_weaknesses: Strengths is the proposed method outperforms the baseline methods by a large margin in terms of accuracy.

Weaknesses:
1. The limited novelty of the proposed method, it is simply a distributed way to calculate a loss function involving sum and product operations.

2. I may missed some details but I did not find comparison of communication of the proposed algorithm with federated learning baselines, the proposed algorithm does not utilize multi-step local training while FedAvg style algorithms use it, it is important to also compare the amount of communication when achieving the same accuracy, and the number of local steps of FedAvg style algorithms should be well-tuned.


clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow and clear, novelty is limited. With the experimental details provided in the paper, I believe it is producible.
summary_of_the_review: Overall I feel the novelty is limited and some more metrics in experiments should be reported.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a method to perform self supervised learning through dual encoding in an FL environment with many clients with a small number of non-iid data.

Three reviewers suggested reject, particularly on the grounds of limited technical novelty, while one reviewer recommended accept. Although no consensus was reached even during the discussion period, positive reviewer did not strongly support accept.

This AC also carefully reviewed this paper, but agreed with the opinions of the reviewers who argued for rejection. I understand considering the scenario where each client has a small number of data samples, but in the main experiment, the fact that the number of clients is extremely large and each client only has a maximum of 16 samples is not very persuasive. In addition, more reliable verification of privacy leak issues according to additionally shared information is required.
justification_for_why_not_higher_score: limited technical novelty
justification_for_why_not_lower_score: n/a
Decision: Reject