Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This study clarifies that the drop in accuracy varies according to the activation function in the FL. Due to the shape of the function, the accuracy of the SOTA activation function drops in the FL setting and HardTanh outperforms other activation functions in most environments.
Furthermore, the authors provide guidelines and benchmark data for choosing activation functions in various FL settings.
strength_and_weaknesses: Overall, the paper is well-written and easy to follow. The experimental results demonstrate that the proposed method is promising. The authors provide a guideline for selecting activation functions in FL, an explanation for the performance degradation and a novel activation function HardTanh.

However, my concern is whether the reason why HardTanh works so well has something to do with poor training. I see that the authors use a learning decay of 0.1 at rounds 50 and 75, which might be beneficial for some activation functions. I suggest that the authors employ multiple
training strategies to observe the result. And more additional experiments lead us to believe that the benefits of the HardTanh activation functions are independent of model training. Moreover, it will be more convincing to compare it with the latest SOTA models.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow.
summary_of_the_review: See Strength And Weaknesses
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In Federated Learning, there is little attention on how different types activation function affect the final accuracy. In this paper, authors discover that good activation function in centralized training is not good in FL settings and vice versa. Authors further perform some analysis on the weight parameters, latent representations and loss landscape resulted from different activation functions. They show that good activation functions in FL are the ones that exclude fewer features and has smoother landscape.  
strength_and_weaknesses: Strength:

1: The overall idea is important and novel. I am not aware of other papers studying the effect of activation function in FL settings. And I am pretty surprised that activation functions can cause significant performance change in FL setting.

2: The experiments seems to be solid and extensive. Authors consider several settings and different algorithms.

Weakness:

1: I am quite confused by the analysis, especially section 4.1. In the first half of 4.1, I quote "This can be summarized simply by saying that the Tanh-like activation functions have low sensitivity to the accuracy drop in the FL aggregation step because they exclude a much smaller number of features than do the recent SOTA activation functions." Up to this point of the section, it seems to be only an intuition instead of explanation. Next, authors measure the CKA similarity and weight similarity between clients. They seem to say ReLU shows higher similarity in both cases compared with Tanh. My question is: if Tanh excludes fewer features in each client, shouldn't it show higher similarity in both CKA (the output features) and weight? From another perspective, as far as I understand, the client drift means each local client learns a significantly different model. Thus, if Tanh somehow alleviates client drift as author claims, shouldn't it result in similar models in all the clients? 

2: I think if there is some theoretical analysis on the activation function and how it affects the convergence, it will really strengthen the paper. 

clarity,_quality,_novelty_and_reproducibility: Clarity and Quality: 

Overall very good. Some confusion in the analysis section.

Novelty:

Novel question to explore as far as I am aware of. Though it might be argued that this paper doesn't propose new algorithms for FL.
summary_of_the_review: I recommend weak accept of this paper. I think the overall question is important and novel. Though more work should be focused on analysis section and maybe some theoratical analysis.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper analyzes the effect of various activation functions on server accuracy in federated learning with heterogeneous clients. It shows that Tanh-based activation functions outperform ReLU-based activation functions in most cases, providing the guidelines for selecting activation functions in various federated learning scenarios. This paper also conducts a few experiments to investigate why Tanh-based activation functions are more robust to client heterogeneity than other activation functions. 
strength_and_weaknesses: ### Strength

- This paper provides an interesting observation that the choice of the activation function plays a critical role in the generalization performance in federate learning, and Tanh-based activation functions outperform Relu-based activation functions, which are widely used in centralized training, in various federated learning scenarios with heterogeneous clients.
- This paper conducts comprehensive experiments that show how each aspect of the federated learning setup (the number of clients, non-IIDness, participation rate, backbone architecture) affects the accuracy according to the different activation functions and provides guidelines for selection activation functions in federated learning.

### Weakness

- The experimental results in Table 3 do not provide convincing evidence demonstrating that Tanh-like activation functions are virtually unaffected by non-IIDness.
    - As illustrated in Table 11, Tanh-like activation functions show a much more significant accuracy drop when the data heterogeneity across clients becomes larger (Dirichlet parameter $\alpha$ gets smaller). This trend becomes more significant when more clients participate per round. While these results support the claim that Tanh-like activation functions are robust to partial participation, but do not support the claim that Tanh-like functions are robust to the non-IIDness of each client.
- The authors claim that the issue of client drift becomes severe since ReLU-based activation functions take fewer features than Tanh-like activation functions. However, Results in Figure 3 and Figure 4 rather show that Tanh-like activation functions have more client drift phenomenon: 1) low CKA similarity between clients, 2) large weight divergence between clients. Note that large weight divergence is known to be a cause of degenerated server accuracy and has been studied in many works [1, 2, 3]. Therefore, these experiments do not support the claim and should provide more convincing evidence.
- Need more additional discussions and experiments:
    - Why are the accuracies of ResNet-based architectures and EfficientNet lower thanConvNet4?
    - 200 rounds of communication seems too short for the convergence of the model. The authors should provide convergence plots or report the results after the model is trained with enough communication rounds.
    - There are no discussions about why large difference in accuracy among ReLU-based activation functions or among Tanh-like activation functions.
    - In Table 4, when $N=20$, why Leaky ReLU performs better with $R=0.3$ than $R=0.4$?
    - ReLU-based activation functions and Tanh-like activation functions may have different optimal learning rates. Authors should compare models trained with each optimal local learning rate.


### Reference
[1] S.P. Karimireddy et al., SCAFFOLD: Stochastic Controlled Averaging for Federated Learning, ICML, 2020.

[2] D.A.E. Acar et al., Federated Learning Based on Dynamic Regularization, ICLR, 2021.

[3] L. Gao et al., FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correction, CVPR, 2022.
clarity,_quality,_novelty_and_reproducibility: The objective of this work is clear, but the experiments are hard to understand and not convincing to support the main contribution.
summary_of_the_review: This paper lacks contribution because (1) the main experiments are difficult to understand, missing many discussions, and do not support few claims, (2) analysis of model behavior does not support the main claim that the Tanh-based activation functions are robust to data heterogeneity of clients, and (3) the paper does not propose any method to handle the issue of existing activation functions in federated learning.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper considers the importance of different activation functions in the performance of the centralized and FL trained models. It tries to empirically show that there are considerable differences between the two settings. 
strength_and_weaknesses: Strength:
- The question that the paper sets out to answer empirically is well-defined.
Weaknesses:
-  For an empirical work, the experiments are very limited and have many issues. 
- No error bars or multiple runs are used
- The number of datasets that are considered are very limited and it is difficult to be certain if the results generalize.
- The parameters of the training algorithms, e.g. lr, local batch size and steps, local lr, momentum are kept fixed and not tuned for different settings. I suspect this is one main reason that the performance of the models are so low and not on par with what they should be (e.g. for larger resnet models with almost iid, the performance should be >70% in my experience for cifar-10). Due to not well-tuned hyper-parameters for each setting, some weird observations happen, e.g. linear model performs better on cifar-100 compared to cifar-10 in table 1. Another example of this is the observation on 3rd paragraph of page 5.
- Only a limited set of heterogeneity is considered (label heterogeneity) and there are no experiments on different local dataset sizes, covariate shifts, ...
- Section 4 is poorly written and it is full of statements without substantial evidence. Unfortunately, the statements in section 4 are mostly vague, difficult to follow and independent of the results that are presented. In figure 3 and 4, it is not clear what is being presented and how does it relate to the other claims. It seems that learnability and heterogeneity are two different and opposing factors based on Figure 3 and 4. But it is not clear why they should both be in the favor of the tanh like activations. 
- Overall, the presented evidence for the claims that the paper makes are not very strong. Moreover, section 4 does not provide a clear picture or intuition on why the claims could be correct either.
clarity,_quality,_novelty_and_reproducibility: Clarity and quality is low. Please see the above.
summary_of_the_review: The experimental results have many flaws and are not convincing. The claims are not well supported by the observations.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this work, the effectiveness of activation functions in various federated learning settings is verified. It is observed that off-the-shelf activation functions that are used in centralized settings exhibit a different performance trend than in federated settings. HardTanh is found to achieve the best accuracy when severe data heterogeneity or low participation rate is present. The authors also provide an analysis to investigate why the representation powers of activation functions are changed in a federated setting by measuring the similarities in terms of weight parameters and representations. Guidelines for selecting activation functions in cross-silo and cross-device settings are also provided. The reviewers thought the paper was easy to follow and thought the numerical experiments are promising. The reviewers however raised a variety of concerns including that: (1) "whether the reason why HardTanh works so well has something to do with poor training", (2) confusion about the analysis of Sec 4.1 (3) limited experiments (4) no error bars (5) limited number of datasets (6) issues with parameter choices. The authors provided a response but this response does not address all of the concerns raised by the reviewers in a satisfactory manner. It is clear the paper requires a substantial review and therefore I recommend rejection.

justification_for_why_not_higher_score: Paper has a lot of flaws and does not merit a higher score
Decision: Reject