FAACL: Federated Adaptive Asymmetric Clustered Learning

Authors: Zehao Zhang, Guojun Zhang, Hongliang Li, Pascal Poupart
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Federated Learning, Distributed Learning, Clustering

Abstract:
Asymmetric clustering has remained an unexplored problem in Clustered Federated Learning (CFL), diverging from the traditional approach of forming independent, non-interacting clusters. Previous methodologies have been limited to either separating devices with different data quality into distinct clusters or merging all devices into a single cluster, both of which compromise either data utilization or model accuracy. We propose a new federated learning technique where some devices may contribute to the training of the models of other devices, but without enforcing reciprocity, leading to a form of asymmetric clustering.  This is beneficial in a variety of situations including scenarios where it is desirable for a device with high quality data to help train the model of a device with low quality data, but not vice-versa. This method not only enhances data utilization across the devices, but also maintains the integrity of high-quality data. Through a rigorous theoretical analysis and empirical evaluations, we demonstrate that our approach can efficiently find high quality (asymmetric) clusterings for numerous devices, achieving competitive performance metrics on existing CFL benchmarks.

Decision: reject
Reviewer scores: 5, 3, 3

Reviews and discussion:
<thread>
creation date:
1730576381033

thread_type:
official_review

<thread_object>
creator: reviewer_wx7g

<content>
summary:
This paper presents a clustered federated learning method named FAACL, which aims to adaptively cluster devices and dispatch models for each device according to its cluster.

strengths:
The author provides a theoretical analysis of its proposed cluster strategy.

weaknesses:
This paper seems to be an incremental work of CFL. The contribution is limited.

This paper is poorly written. E.g., 1) the author should add a figure of framework and workflow. 2) The expression of lines 186-196 seems to detail the contributions and steps of the proposed method, rather than the components. I advise the author to reorganize the expressions.

The author attempts to utilize CFL-based strategy to address the problem of non-IID data. However, the author lacks comparison the proposed method with SOTA tradition federated learning optimization methods [1-4].

The proposed method cannot adapt to the secure aggregation method. The author should analyze the privacy of the proposed method.

The experiments are mainly based on MNIST and its variants. I would like to know how the method performs on more complex datasets, such as ImageNet. Please note that the experimental results show that the proposed method is not effective enough on the cifar10 dataset and is even inferior to the baseline.

questions:
Please see the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695435164

thread_type:
official_review

<thread_object>
creator: reviewer_7bx5

<content>
summary:
This paper proposes a new federated learning technique where some devices may contribute to the training of the models of other devices, but without enforcing reciprocity, leading to a form of asymmetric clustering. This approach not only enhances data utilization across the devices, but also maintains the integrity of high-quality data.

strengths:
1. The proposed approach tackles the underexplored issue of asymmetric clustering in Clustered Federated Learning, providing a fresh perspective.

2. The experimental results are comprehensive.

weaknesses:
1. The paper presents a clustered federated learning method; however, it lacks an introduction to the implementation of federated learning (FL) and clustered federated learning (CFL). I recommend including details about the specific training processes and objectives of both FL and CFL in the BACKGROUND section to enhance clarity and context.

2. How can the proposed approach be effectively implemented in real-world scenarios? In practical federated learning settings, it seems challenging to determine whether cluster C2 supports cluster C1. Given that devices do not have visibility into each other's data during training, the primary method for assessing support appears to require multiple rounds of training and testing. This approach could lead to considerable resource inefficiencies.

3. The main operations of the method are presented in algorithmic form in the appendix (e.g., Algorithms 5 and 6). However, it may be more beneficial for readers to understand these concepts if they are described in text form within the main body of the paper.

4. In the experimental setup, there is no detailed information on how the data is divided or the rationale for this division. Different data partitioning methods can impact the final performance of federated learning.

questions:
See weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730741288281

thread_type:
official_review

<thread_object>
creator: reviewer_twgm

<content>
summary:
Authors discuss the clustered learning approach in Federated Learning and relax the following constraints:
1. Number of clusters being fixed (Adaptive)
2. Each device contributes to a single cluster (Adaptive) 
3. Model sharing is reciprocal in nature (Asymmetric)

strengths:
S1: Introduced asymmetric clustering 
S2: Source code is shared for reproducibility.

weaknesses:
W1: This comment is in relation to the Section 4.2 and 4.3 -> Some of the related works may need to be studied and probably be included as part of the comparison metrics. [1-2]

W2: How does this handle a newly added device in the Federated Learning? Do we check against all the clusters and find if it supports or not individually or against the clusters only? Does that a performance hit if the clusters are already prepared.  

[1] Vardhan, Harsh, Avishek Ghosh, and Arya Mazumdar. "An Improved Federated Clustering Algorithm with Model-based Clustering." Transactions on Machine Learning Research (2024).

[2] Kim, Heasung, Hyeji Kim, and Gustavo De Veciana. "Clustered Federated Learning via Gradient-based Partitioning." Forty-first International Conference on Machine Learning.

questions:
Q1 :  line 053 - “and all existing techniques assume that each device contributes to a single cluster.” - What does this line mean here? And how do address the same in your paper? 

Q2: Does it have the same privacy guarantees as other models proposed in the same vein such as [1] and others. 

Q3: How does the computational complexity compare to other algorithms? 

Q4: How far off is this clustering from ground truth? Additionally is there a comparison again Oracle? [2] 

Q5: Was there an ablation study performed without merging of clusters and just having asymmetric contribution? 

[1] Sattler, Felix, Klaus-Robert Müller, and Wojciech Samek. "Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints." IEEE transactions on neural networks and learning systems 32.8 (2020): 3710-3722.

[2] Jothimurugesan, Ellango, et al. "Federated learning under distributed concept drift." International Conference on Artificial Intelligence and Statistics. PMLR, 2023.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733296496661

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General comments

comment:
As the discussion period with the authors comes to a close, we'd like to draw the attention of the AC and the reviewers to a few points.  First, given the current low scores of the paper, it will be tempting to simply discard the paper as hopeless.  However, the current scores probably do not take into account the revised version that we submitted and our point-by-point response since there was no feedback from two of the reviewers and only one reviewer posted additional queries on the last day of the discussion period.  Please do read the revised version and each response in the rebuttal before making a decision.  

After receiving the reviews we did ponder about what to do with the paper given the low scores.  Since there was no fundamental concern about FAACL and all the points raised were addressable, we revised the paper and provided a response to each point.  We truly appreciate all the constructive feedback, which really helped to improve the paper.  As far as we are concerned, we believe that we have addressed satisfactorily all points.  We took the feedback seriously and added results for two new baselines as requested.  We also revised several sections of the paper as requested.  All material changes are highlighted in blue in the revised paper.

In case the following is not well understood, it is important to note that ICLR is different from other conferences because of the opportunity to revise the paper as part of the rebuttal period.  In other conferences, feedback that requires changes to a paper cannot be addressed, but with ICLR it is possible.  Again, we believe we have adequately addressed all points, but naturally you shall be the judge of that.  If the paper is still rejected, please make sure to include new feedback that we can act on.  We would like to make sure that whatever we re-submit to the next conference does not end up being the same as the revised version that we already uploaded to ICLR due to a lack of new feedback.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes to create an asymmetric version of clustering in federated learning (training), where the clubbing of different compute nodes may not be bidirectional.

The review scores were pretty low with all recommending rejection. One point that goes against the paper is to further complicate an already involved system while the improvements do not fully compensating for this.

Overall, I recommend rejection.