Kati8/ICLR_2023_2025 huggingface dataset:
authors: Jiaqiang Li
authors: Qiqi Liu
authors: Yaochu Jin
authors: Xiaohu Wu
authors: Zhilong Li
authors: Han Yu
reviews: 
Comments: The paper addresses the challenge of data heterogeneity in federated learning, which can lead to model weight divergence and performance deterioration. This paper proposes a novel personalized federated learning (pFL) method called Federated Learning with Similarity Information Supervision (FedSimSup). The key idea behind FedSimSup is to integrate a local supervisor that refines the personalized model when the broadcast global model is not beneficial to local model performance.
Strengths: 1.	The paper makes an original contribution by proposing a novel hierarchical client clustering approach for federated learning, effectively addressing the challenge of data heterogeneity across clients.
2.	The paper is reasonably clear in structure and presentation, with a logical flow that generally allows readers to follow the authors' approach.
Weaknesses: 1.	While the paper attempts to introduce a hierarchical client clustering method in the field of federated learning, the level of innovation is relatively limited. The concept of client clustering has been explored in existing research, it lacks breakthrough novelty. 

2.	The experimental section lacks sufficient comparison experiments and diverse datasets, making it difficult to convincingly demonstrate the effectiveness of the proposed method across different scenarios. The description of the client clustering strategy in federated learning is detailed, but the depth and breadth of the experiments are insufficient. 

3.	Certain technical details are described ambiguously, particularly in the client clustering process and the complexity analysis of the algorithm. This could make it difficult for readers to fully understand the key points of innovation. 

4.	Figure 2 and Figure 4 are not visually clear, which hinders the reader’s ability to interpret the results effectively. The lack of clear visualizations and detailed performance analysis further weakens the overall quality.

5.	The paper lacks a strong discussion of future research directions, failing to show the potential long-term impact of its approach.
Recommendation: 3
Reviewer confidence: 3
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: This paper a novel personalized federated learning (pFL) method called Federated Learning with Similarity Information Supervision (FedSimSup) to address the challenge of data heterogeneity across clients in federated learning.FedSimSup integrates a local supervisor mechanism and a personalized model to facilitate effective global information aggregation while respecting the unique data distributions of individual clients.
Strengths: 1) The supervisor module demonstrates a certain degree of flexibility, enabling model heterogeneity at a relatively low cost.
2) The paper presents model interpretability in a straightforward and intuitive manner.
Weaknesses: 1) In Section 3.3, what is the rationale behind not aggregating other clients' data for client i? Additionally, Equation (8) describes the calculation of similarity among all clients to create and deploy different models. Is this calculation performed on the server or client side? If on the client side, there may be issues with security and communication overhead; if on the server side, there could be computational overhead for generating different models. The author is requested to provide a detailed response to this issue.

2) The method description is overly concise. For instance, the exact similarity calculation method and how the supervisor guides the model are not elaborated. If the method merely stitches together similarity computation and the supervisor module, the paper may lack sufficient contribution.

3) In Table 1, what does the 3597 value for FEMNIST represent? Additionally, Figure 6 is too small, and the significant fluctuations in the proposed method's performance suggest potential design issues. Could there be an algorithmic problem contributing to this?

4) Numerous recent methods address parameter decoupling, such as FedCP, which extracts both global and local information from data. Its extraction modules are aggregable. Compared to this, the lack of aggregation in the supervisor module here might lead to overfitting. The author is advised to conduct a thorough survey of recent pFL methods and benchmark against newer approaches.

5) If the supervisor module is only conducting gradient descent, would increasing the parameter size of the local model achieve similar results without this module? It is recommended to add ablation experiments and consider this question in further detail.
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 1
reviews: 
Comments: This paper introduces FedSimSup, a novel pFL method that uses a local supervisor to refine the personalized model and ensure effective global information aggregation. FedSimSup also measures client similarity based on label distribution differences to enhance information sharing among similar clients. Experimental results show that FedSimSup outperforms seven state-of-the-art methods on heterogeneous data, supports different model architectures across clients, and offers interpretability.
Strengths: 1. The paper introduces FedSimSup, a new method in personalized federated learning that incorporates a local supervisor to refine personalized models.
2. FedSimSup allows for different model architectures across clients, making it adaptable to various computational capacities and needs.
3. Experimental results show that FedSimSup outperforms seven state-of-the-art federated learning methods on heterogeneous data, demonstrating strong capabilities.
Weaknesses: 1. From Equations (3), (4), and (5), it appears that the client models are training two models simultaneously. This does not seem to follow a clear supervision process like knowledge distillation. Could the authors provide a clearer explanation of how the supervision mechanism operates within this framework?

2. The paper mentions using label distribution differences to compute $s_{ij}$. It would be helpful if the authors could elaborate on the exact method used to evaluate this similarity, ensuring clarity on its computation based on label distributions.

3. Is it possible to observe the effects of the similarity metric $s_{i,j}$ during the training process? Specifically, for a given client $i$, could the authors illustrate how it determines which other clients it tends to select throughout the training?

4. Could the authors provide the memory usage of FedSimSup compared to other baseline methods, particularly when different model architectures are employed on the client side?

5. In Table 3, FedSimSup-T seems to underperform compared to the original method utilizing LeNet-5, despite having more parameters in the Transformer model. Could the authors provide any insights for this case?
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2