Revisiting the Activation Function for Federated Image Classification
Authors: Jaewoo Shin
Taehyeon Kim
Se-Young Yun
Venue: Submitted to ICLR 2023 (2023)
Keywords: Federated Learning
Activation Function

Abstract: Federated learning (FL) has become one of the most popular distributed machine learning paradigms; these paradigms enable training on a large corpus of decentralized data that resides on devices. The recent evolution in FL research is mainly credited to the refinements in training procedures by developing the optimization methods. However, there has been little verification of other technical improvements, especially improvements to the activation functions (e.g., ReLU), that are widely used in the conventional centralized approach (i.e., standard data-centric optimization). In this work, we verify the effectiveness of activation functions in various federated settings. We empirically observe that off-the-shelf activation functions that are used in centralized settings exhibit a totally different performance trend than do federated settings. The experimental results demonstrate that HardTanh achieves the best accuracy when severe data heterogeneity or low participation rate is present. We provide a thorough analysis to investigate why the representation powers of activation functions are changed in a federated setting by measuring the similarities in terms of weight parameters and representations. Lastly, we deliver guidelines for selecting activation functions in both a cross-silo setting (i.e., a number of clients <= 20) and a cross-device setting (i.e., a number of clients >= 100). We believe that our work provides benchmark data and intriguing insights for designing models FL models. 

Under review as a conference paper at ICLR 2023
REVISITING THE ACTIVATION FUNCTION FOR FEDERATED IMAGE CLASSIFICATION
Anonymous authors Paper under double-bling review
ABSTRACT
Federated learning (FL) has become one of the most popular distributed machine learning paradigms; these paradigms enable training on large corpus of decen- tralized data that resides on devices. The recent evolution in FL research is mainly credited the refinements training procedures by developing the optimization methods. However, there has been little verification of other technical improve- ments, especially improvements to the activation functions (e.g.. ReLUU, that are widely used in the conventional centralized approach (i.e., standard data-centric optimization). In this work, we verify the effectiveness of activation functions in various federated settings. We empirically observe that off-thh-sseel activation functions that are used in centralized settings exhibit a totally different perfor- mance trend than do federated settings. The experimental results demonstrate that HardTTan achieves the best accuracy when severe data heterogeneity or low participation rate present. We provide thorough analysis investigate why the representation powers of activation functions are changed in federated setting by measuring the similarities in terms of weight parameters and representations. Lastly, we deliver guidelines for selecting activation functions in both a cross-silo setting (i.e., a number of clients S 20) and a cross-device setting (i.e., num- ber of clients 2 100). We believe that our work provides benchmark data and intriguing insights for designing models FL models. The code is available at 4open. www.scceeeeeeeeeeeee
ACT-160B7
INTRODUCTION
Federated learning (FL) has become common and ubiquitous paradigm for collaborative machine learning techniques (Bonawitz et al.) 2019: Caldas et al., 2018 Kairouz et al.) 2019 [i al.) 2020)] 2019|Shokr & Shmatikov| 2015; McMahan al. [017 Smith al. (017) because it maintains data privacy. Each client (e.g., mobile devices or the entire business) communicates with the central server by transferring their trained model but not the data; all local updates aggregated into global server-side model. Although a centralized method enhances generalization by employing a large amount of training data, the features of the FL. methods appear differ from those of centralized method owing to data non- IIn-es, client resource capability, and model communication (Kairouz) al. 2021 McMahan al. 2017 Zhao et al. 2018).
Most FL studies focus on improving the performance of the global model by applying a new
regularizer in the optimization algorithm. For instance, proximal term is attached to optimize the local update to enhance the method's stability (Acar et al. 2021 Karimireddy et al. 2020 Li et al.,
2021 2020). and hence the local model does not diverge from the global model. Some studies (Hsu et al. 2019 Lin et al., 2020; Wang et al.) [020a Yurochkin et al. 2019) improve the aggregation step of the local models by weight averaging for the server model. When additional public or synthetic datasets are allowed, the server proofreads the weights of the models by utilizing their data
distribution for balancing (Zhao et al. 2018 Jeong et al.) 2018 Goetz& & Tewari 2020, Hao et al.. 2021). Recently, there has been an increasing demand for the personalization of models according to the client. Jiang et al. (2019) and Fallah et al. (2020) attempt to train personalized models for each client with a few rounds of fine-tuning rather than focusing the performance of the server model. In consideration of system heterogeneity (i.e., clients having different computational and communication capabilities), Avdiukhin & Kasiviswanathan (201) mitigate model communication

---

Under review as a conference paper at ICLR 2023
Centralized Settings
FL. Settings
90
01
Lesky RCLL
Sonth
73
Hish
Best
GeLU
63
Best
50
42
-2
Batti
Harefaali
RetU
Leity Retl
Seitt
Getu
Uneat
Tast
Harfaadh
Ritu
Leaky Retui
Swisn
Mon
GetLi
(8) Category of Activation Functions
(b) CIFAR-I 10
(c) CIFAR-100
Figure 1: (a) Plots of different activation functions. (b), (c) Accuracies on CIFAR-10 and CIFAR- according to the different activation functions, respectively. The blue line indicates the performance of models trained on a single central server. The orange line indicates the performance of models trained in an FL environment where 20 clients of the total 100 clients participate in the training per round. We use a model having four convolution layers and one classifier. Here, 'Linear' indicates model without any activation function. In single machine that centralizes the training data, the more up-to-date the activation function used, the better the performance (blue line). In contrast, interestingly, HardTanh (Collobert al. (011)) prints the best server accuracy for both CIFAR-10 and CIFAR- R- 100 under heterogeneous scenarios (orange line). A detailed explanation of the activation functions is provided in Ampppedix
problems by using asynchronous local stochastic gradient descent (SGD), and Horvath et al. 22021) improve accuracy for heterogeneous resource capacity by using different model sizes per client.
Despite the popularity of FI., some options for federated model optimization remain under-explored. Designing FL-familiar training recipes essential to optimizing model performance, but few studies have attempted to design a new recipe instead of using those intended for a centralized setting, Charles et al. (2021) present an empirical analysis of the impact of hyperparameter settings for federated training dynamics from the perspective of a large cohort size. However, FL activation functions (McMahan et al. 2017 Karimireddy et al., 2020; Li et al. 2019) have rarely been studied, although activation functions play a crucial role in facilitating generalization and convergence. We thus raise the seemingly doubtful question: Do activation functions that are popular in centralized settings also produce good optima in FL?
To answer this question, we conduct a pilot experiment to compare performance in centralized settings with performance in FL settings. Figure (b) and (c) show the accuracy of neural networks trained under a centralized setting and an FL setting according to the replacements in the activation function. Surprisingly, neural network with Tanh has better accuracy than ReLU, which is silver bullet in the field of centralized deep learning field. The problems mentioned above lead us to the intriguing question:
Do off- she-the--helf activation functions that are intended for a centralized setting also perform appropriately in the FL setting?
In this work, we answer the question with thorough empirical evaluations: the most recently developed activation functions tend to degrade the performance of the server as the heterogeneity becomes more severe.
Several considerations (e.g., the total number of clients, client participation, non-IIDness) in selecting activation functions may significantly improve the selection of activation function. Combining con- siderations may further boost the model accuracy. We experiment with various activation functions, including functions that are widely-used and rarely-used in the centralized setting, in various environ- ments based CIFAR-10 and CIFAR- 100. The experiments identify an interesting phenomenon in FL. in which applying activation functions like ReLU in stacked convolutional layers demonstrates low accuracy owing the shape of the function. We also provide an analysis of the representation power according to the different activation functions for federated image classification. Our key contributions are summarized as follows:
We provide guidelines for selecting activation functions in FL. FL has the following special considerations: number of clients, participation ratio, and non-IIDness. We provide guide-
2

---

Under review as a conference paper at ICLR 2023
lines for cross-silo settings (i..,, for number of clients v 20) and cross-device settings (i.e., for a number of clients 2 100); the suitability the activation function depends on the situation.
We provide an explanation for the performance degradation (i.e., for the performance differ- ence between centralized settings and FL settings) of activation functions that are preferred in a centralized setting. Specifically, we measure similarities in the weight parameters and representations, and visualize the landscape.
We empirically show that the Hard'Tanh activation function (Collobert et al. 201 leads to a
better optimum than other activation functions such as ReLU (Nair & Hinton 2010). Leaky ReLU (Maas et al. 2013), and GeLU (Hendrycks & Gimpel (016) for a severe non-IID setting, low participation rate, and using large number clients. Additionally, we provide benchmark data for activation functions in FL for various models.
2 RELATED WORK
2.1 ACTIVATION FUNCTIONS IN NEURAL NETWORKS
In deep learning, activation functions are inevitable for learning non-linear latent representations; an input signal is transformed into non-linear output. Recent evolution has introduced an enhancement of representation power and lower computational costs. In the main experiments. the following non-linear activation functions are used: Tanh (LeCun et al.) (2015a). HardTTanh (Collobert et al. 2011
ReLU (Nair& & Hinton 2010), Leaky ReLU (Mass al. (003)). Swish (Ramachandran et al. Mish (Misra (019). and GeLU (Hendrycks & Gimpel 2016))
2017
Activation functions have been devised for the gradient exploding/vanishing problem; where the magnitudes of gradients become either near zero or infinite during backward propagation. A general choice for activation functions has been ReLU, which enable efficient propagation. On the other side, ReLU is not differentiable at zero, and it causes a significant number of dying neurons by forgetting the information during propagation. Recent works (Ramachandran et al. [2017] [Mirra] (2199) have achieved smoother optima in centralized learning by designing self-regularized gradients; whereas those based on FL. settings are badlands.
2.2 FL. METHODS
Federated optimization methods manage handle multiple clients without collecting data, and they use server weights from a central server to coordinate the global model across the network. In particular, these methods aim minimize the following objective function:
N
min/(w) where f(w) = N
(10)
where (() is the loss function based on the client k. N is the total number of clients. At each round, K & N clients are selected from the total number of devices. The selected clients run each local model using SGD for E number of local epochs, and they ultimately aggregate the selected models at the server model.
In the FL environment, the global model can be drifted by optimizing the local clients because the statistical data non-IIDness causes different local optima that are far apart from each other. This called client drift (Karimireddy et al. 2020 Khaled et al. 2019 Reddi et al., 2020), and it indicates the inconsistency between optima. Recently, some works have prevented client drift by designing aggregation methods; Wang et al. (2020b) present a method of normalized averaging that removes objective inconsistency, and Zhang et al. (202T) propose a training algorithm for group knowledge transfer, which allows each client to keep personalized prediction on the server assist the local training of other clients. Federated Averaging (FedAvg (McMahan et al., 2017)) uses the local server for SGD for F. number of epochs. As result, the selected client k's weight is updated as we. To aggregate the local client models at each round, FedAvg sums and averages for the server model parameters formulated as:
1
Σ
W
(2)
K
KES.
3

---

Under review as a conference paper at ICLR 2023
where w/ is the server weight of the t-th round, w.. is the client k's weight after local training using w"=1, and St is the client set. McMahan et al. (2017) empirically show the significance additionally tuning the hyperparameters in FL training. We present that with respect to the architectural and operational side. Additional details of related work are explained in Appendix A
3 EXPERIMENTS
In this section, compare several activation functions. We categorize the activation functions into two groups: (1) ReLU, Leaky ReLU, Swish, Mish, and GeLU as recent state-of-the-ae (SOTA) activation functions that widely used in centralized settings; and (2) Tanh and HardTanh as Tanh-like activation functions that are not widely used in centralized settings.
3.1
EXPERIMENTAL SETUP
Dataset and non-III Settings. Two benchmark datasets are employed: CIFAR-10 and CIFAR- 100 (Krizhevsky et al. 2009). We provide the descriptions of the datasets in Appendix BJ To randomize the non-IlD data, we assume that all client training data use class labels according to an independent categorical distribution of N classes parameterized by the vector q:
2 0, E [1, and Σ =1 [E[L.N]
For the heterogeneous distribution, the Dirichlet distribution (Hsu et al. 2019 Yurochkin et al. 2019), q ~ Dir(a) is used, where is an N-length concentration vector having all elements 0, that is, the prior distribution for N classes controls the heterogeneity of clients.
Models. Our study focuses on compact models that are realistically possible in FL. Therefore, we mainly use simple ConvNet having four convolutional layers and one classifier; ConvNet refers to ConvNet with four convolutional layers. The first convolution layer has 64 kernels, and deeper layers have larger number of kernels (O Shea & Nash 2015). For additional models, which have shortcut and batch normalization layers, use Resnet20, Resnet Resnet 44 (He et al.) 2016), and Mobi eeeet22 (Sandler et al. 2018). The Details of settings and model architectures of ConvNet are provided in Appendix B
Training Details. clients N, the client participation ratio R, and the Dirichlet distribution constant We adapt FedAvg and perform 200 rounds with 5 local epochs using a learning rate 0.01, with a learning decay of 0.1 at the 50th and 75th round, weight decay of 1e-4, and momentum of 0.9, The number of clients available in different FL settings is limited; in a cross-silo setting. a small number of clients available, and large number of clients requested in cross-device. For the cross-silo setting, we use 20 and R = 0.2, which we select 4 clients at each round. For the cross-device setting, use N 100 and R = 0.2, which select 20 clients each round.
this study, we conduct numerical experiments by changing the number of
We mainly demonstrate the training of ConvNet 4 on CIFAR-10 heterogeneously distributed by modifying the the Dirichlet distribution and the client participation rate R. In the captions, we explain each N, R, and value.
3.2 COMPARATIVE EXPERIMENTS ON THE CHANGES IN ACTIVATION FUNCTIONS
Table I shows the result of both centralized and FL settings using CIFAR-I0 10 and CIFAR- 100 the datasets. With the centralized setting, GeLU shows the best performance, and other recent SOTA activation functions surpass the Tanh-like activation functions. However, with the FL setting, the activation functions show significantly different tendency. HardTanh achieves the highest accuracy. Furthermore, the recent SOTA activation functions show lower accuracy than Linear using CIFAR- 100 as the dataset. The activation functions show different accuracy drops; only the recent SOTA activation functions have an accuracy drop near 40, whereas HardTanh and Tanh have 26.21 and 28.42 on 10. As result, we can find that the most popular activation function, ReL.U (as well as recent SOTA activation functions), does not show outstanding performance an FL. setting.
3.3 STRATEGIES FOR SELECTING ACTIVATION FUNCTIONS FL
This section presents the experimental results and guidelines for selecting the activation functions for various FL settings. FL settings have various environmental limitations relative to centralized settings. There have additional components consider, such as the number of clients, non-IIDness, and the participation ratio.
4

---

Under review as a conference paper at ICLR 2023
Table 1: Server accuracy of 4 in centralized and FL settings two datasets (CIFAR-10 100, CIFAR- 100). Centralized settings are used to train one server model using all training data, and FL settings are used train 100 clients with non-IID data. We use R 0.2, and 0.1. For all tables afterwards we bold the highest accuracy except for Linear.
CIFAR-10
CIFAR-100
Activation Function
Centralized Setting
FL setting
Centralized Setting
FL Setting
Linear
73.74+1..
10.00
46.52:+0.31
28.39
Tanh
81. $1.00110.89
52.58
55.39-10.24
30.75
HardTanh
80.64.10.77
54.43
54.31-10.46
31.76
ReLU
87.01110.11
48.37
59.75.10.33
23.99
Leaky ReLU
87.30:10.03
48.34
60.28++0222
24.04
Swish
86.50=00.08
46.16
60.02=00.47
21.55
Mish
86.38+0.229
50.02
60.67=0.03
24.98
GeLU
87.77-10.11
47.46
61.34:10.35
23.26
Table 2: Server accuracy of ConvNet4 with five different number clients (10, 20, 50, 100, 200). We use 0.1 with R 0.2.
Activation Function
N 10
N 20
N 50
N 100
N 200
Tanh
67.14
65.97
55.28
52.58
48.92
HardTTanh
67.90
66.53
54.88
54.43
49.64
ReLU
69.26
63.39
52.88
48.37
41.57
Leaky ReLU
69.26
63.67
53.12
48.34
41.65
Number of Clients. For different FL strategies, the number of clients varies. A cross-silo setting uses fewer than 20 clients, and cross-device setting use more than 100 clients. The total accuracy drop grows as the number of clients rises. [ablle 2 shows the accuracy with different client numbers. As the number of clients increases, the overall drop in accuracy increases. The Tanh-like activation functions surpass the recent SOTA activation functions with larger client numbers (20, 50, 100, 200). Additionally, the accuracy difference between the recent SOTA activation functions and Tanh-like activation functions gets larger client numbers increase. Considering the observations for the number of clients, we hypothesize that as number clients increases, recent SOTA activation functions are increasingly affected and show more significant accuracy drop.
Non-IIDness. With the Dirichlet distribution parameter a, we can control the IID-ness data: larger value of a indicates lower non-iIDness (lower heterogeneity). Table 3 presents the accuracy for different values of a. The overall reduction in accuracy rises as non- Non--llDess does. In most cases, HardTanh shows the highest accuracy. For 20 clients, the accuracy of the recent SOTA activation functions surpasses the Tanh-like activation functions at low non-IiDness. The shape the recent SOTA activation functions with high non-IlDness causes severe accuracy drop, which discuss in /ection 4 For 100 clients, the Tanh-like activation functions are virtually unaffected by non- IDness and outperforms the recent SOTA activation functions.
The low accuracy of the Tanh-like activation functions at a = 0.01 occurs due to tough training settings and the Tanh-like activation functions fail find optimum, such Linear. We conduct an additional experiment for a = 0.01 with learning rate 0,005 to compare the Tanh-like activation functions and recent SOTA activation functions where the Tanh-like activation functions does fail to train. Table 13 in Appendix C shows that Tanh-like activation functions surpass the recent SOTA activation functions without failure of finding optimum.
Participation Ratio. The participation of clients limited in FL. depending on the environment. In a cross-silo setting, high participation may be possible, whereas only limited participation is possible in a the cross-device setting. Table 4) shows the accuracy FL settings for four different values of the participation ratio Rt. As the client participation decreases, the overall accuracy drop increases. For 100 clients, the Tanh-like activation functions achieve the highest accuracy. With 20 clients, however, there is noticeably larger accuracy drop participation decreases for the most recent SOTA activation functions. As a result, as the participation ratio decreases, the accuracy of the Tanh-like activation functions reverses recent SOTA activation functions. With increased

---

Under review as a conference paper at ICLR 2023
Table 3: Server accuracy of ConvNet with four different Dirichlet constant values (0.01, 0.1,
10). We use N = 100 and N = 20 with R = 0.2.
N
20
Activation Function
10
0.1
0.01
10
0.1
0.01
Linear
62.48
62.43
10.00
10.00
66.48
66.48
63.54
10.00
Tanh
64.49
64.14
52.58
29.50
70.22
69.55
65.97
27.64
HardTanh
65,27
65.40
54.43
30.09
70.53
70.01
66.53
28.92
ReLU
57.80
56.23
48.37
34.03
75.59
74.21
63.39
34.70
Leaky ReLU
57.85
56.16
48.34
33.92
75.36
74.23
63.67
35.15
Swish
52.62
51.48
46.16
35.65
72.58
71.58
64.57
36,93
Mish
57.30
55.08
50.02
38.94
73.69
72.95
66.47
37.89
GeLUU 55.59
54.34
47.46
36.09
75.68
74.41
65.62
37.77
Table 4: Server accuracy of ConvNet with four different participation ratios (0.1, 0.2, 0.3, 0.4).
use N 100 and N = with = 0.1.
N 100
N 20
Activation Function
R 0.4
R = 0.3
0.2
= 0.1
R =
R = 0.3
R 0.2
=== 0.1
Lincar
10.00
10.00
10.00
10.00
65.85
66.12
63.54
58.36
Tanh
62.61
61.79
52.58
46.61
69.22
67.75
65.97
62.67
HardTanh
59.45
61.75
54.43
41.90
69.25
68.80
66.53
43.96
ReLU
53.26
50.67
48.37
41.95
71.96
70.17
63.39
54.47
I Leaky ReLU
53.17
50.76
48.34
42.05
66.12
70.22
63.67
$4.46
Swish
51.72
49.45
46.16
40.00
70.36
67.99
64.57
53.74
Mish
56.67
52.80
50.02
43.37
71,30
69.34
66.47
60.38
GeLU 53.35
50.61
47.46
41.50
72.32
70.10
65.62
54.05
client participation, client drift diminishes and the impact of the current SOTA activation functions' accuracy reduction decreases.
Different FL settings components affect accuracy when different activation functions used, according to the observations above. The number of clients the most dominant component, and interacts with the effect of other components when small number of clients is used. Therefore, the Tanh-like activation functions are favorable for a large number of clients, such as in a cross-device setting. For a cross-silo setting, for data with high data non-IlDness, and for low ratio of client participation, the Tanh-like activation functions are preferred. Conversely, the recent SOTA activation functions are preferred for cross-silo setting, for data with low data non- oon-IID and for a high ratio of client participation.
3.4 ADDITIONAL EXPERIMENT
We implement additional experiments with different FL method and models. We choose FedProx as the additional FL method and Resnet20, 3esnet32, Resnet 4, and Mobi eeNetv2 as the additional models. For the additional models, we only use ReI.U and Leaky ReLU from the set of recent SOTA activa- tion functions. For more experimental results, refer to Appendix C
Table 5; Server accuracy of ConvNet using FedProx with the following settings: N 100. R 0.2, and 0.1.
Act. Func.
Acc.
Linear
48.74
Tanh
45.50
HardTanh
49.16
ReI.U
42.19
FedProx. As mentioned in |section studies to im- prove the server model in FL add proximal terms to the local object to enhance the method's stability so that the local model rarely differs from the global model. choose FedProx (Li al. 2020) as an additional method because it is the most basic method that improves the server model by adding proximal term.
Leaky ReLU
42.40
Swish 35.79
Mish
39.83
GeLU 36.00
The details FedProx are shown in Appendix A Tabll S]shows the accuracy for different activation functions using FedProx
learning algorithm. Similar to FedAvg, HardTanh achieves the best accuracy, followed by the other Tanh-like activation functions. Proximal terms in local training do not appear to help prevent accuracy loss.

---

Under review as a conference paper at ICLR 2023
Table 6: Server accuracy of Resnet using four different Dirichlet constants, (10, 1, 0.1, 0.01),
with fixed client participation ratio, R = 0.2, and four different client participation ratios, R (0.4, 0.3, 0.2, 0.1) with fixed Dirichlet constant, a 0.1. We use N 100 with CIFAR-I0 as the dataset.
R
Activation Function
10
0.1
0.01
0.4
R 0.3
R 0.2
R 0.1
Linear
56.93
56.96
45.73
26.70
52.09
49.35
45.73
38.60
Tanh
59.66
57.42
46,58
26.50
52.92
49.69
46.58
38.83
Resnet20
HardTanh
57.54
46.19
26.52
50.16
46.19
39.28
ReLU
56.90
56.95
45.35
25.73
52.54
48.71
45.35
37.53
Leaky ReLU
56.78
56.49
45.18
26.10
52.23
48.38
45.18
37.71
Linear
60.12
56.84
43.01
24.98
49.49
46.15
43.01
37.39
Tanb
60.03
56,81
43.92
25,90
51.03
47.77
43.92
38.24
Resnet32
HardTanh
59,30
$7.39
43,99
25.51
50.75
47.73
43,99
38.34
ReLU
58.52
56.34
43.09
26.66
49,42
46.49
43.09
37.87
Leaky ReLU
58.97
56.16
43.42
26.42
49,40
46.81
43,42
37.33
Linear
57.55
55.98
45.14
27.50
51.25
47.78
45,14
39.56
Tanh
57.94
55,54
46,42
27.23
52.89
50.67
46,42
38.19
Resnet44
HardTanh
58.80
$7.54
45.98
26.46
53,95
45.98
38,88
ReLU
56.89
54.23
44.42
26.06
50.31
46.25
44.42
38.57
Leaky ReLU
57.02
53.76
43.97
26.37
50.40
46.84
43.97
37.35
Resnet. Table 6) shows the result of Resnet20, Resnet 32, and Resnet44 with four different values of a and Rt. The Tanh-like activation functions acheive better accuracy than the recent SOTA activation functions do. Even when employing models with batch normalization and shortcut layers, the Tanh-like activation functions outperform the recent SOTA activation functions. For 0.01 and using Resnet 32 as the model, the Tanh-like activation functions fails to find the optimum and perform poorly.
MobileNetv2. Table 7 shows the result of Mobi leNetv2 with four different values of for
N 100 and 0.1. Tanh-like activation functions surpass recent SOTA activation functions. In particular, the accuracy of the recent SOTA activation functions is lower than Linear. As the partici- pation ratio rises, the recent SOTA activation functions surpass the Tanh-like activation functions on the CIFAR-10 dataset, demonstrating the same result as Table
Table 7: Server accuracy of Mobi 1 leNetv2 with four different participation ratios, R (0.1, 0.2, 0.3, 0.4). We use N 100 with a = 0.1.
CIFAR-10
CIFAR- 100
Activation Function
R 0..4
R 0.3
R 0.2
R 0.1
R = 0.4
R 0.3
R 0.2
R 0.1
Linear
36.74
36.86
35.86
35.50
15.02
14.76
14.23
13.17
Tanh
33.39
33.64
31.59
31.83
19.20
16.99
14.76
11.19
HardTTanh
33.82
32.41
28.91
29.95
18.64
15.91
13.54
9.39
ReLU
36.99
34.21
26.93
20.12
8.16
5.17
Leaky ReLU
35.54
33.07
30.69
20.44
16.92
13.84
10.14
5.67
ANALYSIS
We present thorough investigations of model behavior and the changes in representation during the funnationn local training to answer the question: Do recent SOTA activation functions have a disadvantage in an FL setting?
4.1 INVESTIGATION OF WEIGHT PARAMETERS AND LATENT REPRESENTATION
The shape of the activation function varies accuracy in FL setting. The activation function selects important features to pass through each layer, and the model trained using these features. According to Figure (a), the number of selected features varies according to the shape of the activation function. In a conventional centralized setting, a single model can access all the data and select optimal training features. However, in FL setting, each client only access portion of the data, which is partitioned in the non-lID condition, and each client trains its model to select features that are important to itself. This results in phenomenon known client drift.
7

---

Under review as a conference paper at ICLR 2023
Linear
Tanh
HandTanh
Rel.U
Leaky
Mish
17.5-
Server Accuresy
Server Aurrunrry
$
15.0-
12.5-
15
10.0-
7.5-
65
25-
10
Leyers
(4) Feature Distribution
(b) Centralized Setting Accumey
(c) FI. Setting Accumcy
Figure 2: (a) demonstrates the feature distribution of all class 0 images in the CIFAR-10 test dataset after passing through the first convolution layer and its activation function. After passing the convolution layer and activation function, we flatten the feature values and draw distribution. (b) shows the accuracy of ConvNet centralized setting with different widths and depths. (c) shows the accuracy of ConvNet in federated setting with different widths and depths. For all three figures, we use 100, R 0.2 and a 0.1.
Convi Layer
Conv3 Layer
MaxPool
ReLU
Clene
10
10
1.980
Clin ID
ClientD
Hard Tanh
10 Ci
10
11.0
10
Clent
Figure 3: CKA similarity between 10 client using test images of CIFAR-10. 10. Each client's model is the model before 100th aggregation. We use N 100 with = 1.0 and a = 0.1 for training. We calculate the CKA similarity using features passing through each layer and its activation function.
During FL aggregation step, problem arises where important features for the global optimum cannot be selected due client drift. This phenomenon appears be severe when the recent SOTA activation functions are used. Due to the shape of their activation functions, the excluded features are greater in number than for the Tanh-like activation functions, and addition, when heterogeneity increases, selection failure of features using the recent SOTA activation functions achieves peak, and this phenomenon reaches its pinnacle. This can be summarized simply by saying that the Tanh-like activation functions have low sensitivity to the accuracy drop in the FL aggregation step because they exclude much smaller number of features than do the recent SOTA activation functions. In this aspect, deeper models that achieve greater accuracy in centralized context perform poorly in FL situation because they reject a greater number of features as the model gets deeper. In Figure (c). deeper ConvNets with recent SOTA activation functions in a FL. setting demonstrate significant reduction accuracy, whereas Figure accuracy improvement for deeper Convnets in centralized setting.
accuracy drop occurs. In
(b) demonstrates an
This perspective can be used to explain the empirical results in section A greater number of clients, high data Ion-IIDness, and a low client participation rate all contribute to high level of heterogeneity. In Table 2 as the number of clients increases, in Table 3 with N = 20, as the dirichlet constant decreases, and in Table 4| the participation rate R decreases, heterogeneity increases, and as a result, the recent SOTA activation function demonstrates a significant drop in accuracy and is surpassed by Tanh-like activation functions.
8

---

Under review as a conference paper at ICLR 2023
Cenvl Layer
Tayor
Layer
Tayer
106
700
Geti
120
Clart
Figure 4: Weight difference between 10 client selected in Figure 3 We calculate the weight difference by subtracting each client's weight from the same layer and normalizing it with L2norm.
To observe how the activation function affects ConvNet 4, we perform two additional experimental studies on heterogeneous local models. For simplicity, we look at 10 clients out of total of 100 clients. First, we perform centered kernel alignment (CKA) (Korbblith et al. 2019) measure the similarity of the output features between different clients. Each client trains the server model for 5 epochs with their own non-IID data. In Figure 3 ReLU has higher CKA similarity than does HardTanh in every layer. Because feature selection fails for the server model, as indicated above, ConvNet4 with ReLU has a smaller feature change than does HardTTan, which indicates a lower learning ability. Second, we calculate the weight differential to check how similar the clients weights are to each other. FFiuure reveals that ConvNet4 with ReLU has smaller weight difference each layer than does HardTTanh, and also indicates that ReLU has limited learning ability.
4.2 ANALYSIS OF LANDSCAPE
We visualize the 2-D landscape of Tanh, HardTanh, ReLU, and Leaky ReLU. Figure 5 shows the 2-D landscape of each activation function. The color bar beside each figure shows the loss difference. Tanh and HardTanh have number that is 10, 000 times smaller than ReLU and Leaky ReLU. This emphasizes that Tanh and HardTann have a considerably smoother landscape than ReLU and Leaky ReLU.
Tanh
HardTank
ReLU
Leaky ReLU
Nelurbation
Perturbations
Perturbation X
Rerturbation
Reftrration
Figure 5: Landscape of ConvNet 4 with Tanh, HardTanh, ReLU and Leaky ReLU, We use N = 100,
R 0.1 and 0.01 for training. We draw landscape with 150 levels.
These findings suggest that the sensitivity of the activation functions in the FL aggregation step. where the accuracy drop occurs, is indicated by the shape of the activation functions. The accuracy drop reaches its peak as the number of features excluded by an activation function increases, and it reaches its maximum with severe client drift: a large number of clients, a low client participation ratio, and high data non-IIDness.
CONCLUSION
This study clarifies that the drop in accuracy varies according to the activation function in FL. Our key finding is that the accuracy of the recent SOTA activation functions drops in an FL setting due to the shape of the functions, and HardTanh outperforms other activation functions in most environments. Additionally, we provide guidelines and benchmark data for selecting activation functions in various FL settings.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough. and Venkatesh Saligrama. Federated learning based on dynamic regularization. arXiv preprint arXiv:2111.0466.. 2021.
Dmitrii Avdiukhin and Shiva Kasiviswanathan. Federated learning under arbitrary communication patterns. In International Conference on Machine Learning, pp. 425-435, PMLR, 2021.
Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečný, Stefano Mazzocchi, H Brendan McMahan, et al. Towards federated learning at scale: System design. arXiv preprint arXiv:/902.0/0/ 2019.
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMa- han, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv:/8/2.01009, 2018.
7achary Charles, Zachary Garrett, Zhouyuan Huo, Sergei Shmulyian, and Virginia Smith. On large-cohort training for federated learning. arXiv preprint arXiv:2106.07822 2021.
Djork-Aré Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289, 2015.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from seratch. Journal of machine learning research, 12 (ARTICLEE):2433-25)) 2011.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020.
Stefan Elfwing. Eiji Uchibe, and Kenji Doya. Sigmoid- weighted linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3-11, 2018.
Alireza Fallah, Aryan Mokhtari. and Asuman Ozdaglar. Personalized federated learning: A meta- learning approach. arXiv preprint aXXiv:2002.07944, 2020.
Jack Goetz and Ambuj Tewari. arXiv:2008.00 04489, 2020.
Federated learning via synthetic data.
arXiv preprint
Ian Goodfellow, David Warde-Farkes Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. In International conference on machine learning, pp. 1319-1327. PMLR, 2013.
Richard HR Hahnloser, Rahul Sarpeshkar, Misha Mahowald, Rodney Douglas, and H Sebastian Seung. Digital selection and analogue amplification coexist in cortex-inspired silicon circuit. Nature, 405(6789):947-955 2000.
Weituo Hao, Mostafa El-Khamy, Jungwon Lee, Jianyi Zhang, Kevin J Liang, Changyou Chen, and Lawrence Carin Duke. Towards fair federated learning with zero-shot data augmentation. In Proceedings of the IEEF/CVE Conference Computer Vision and Puttern Recognition, pp.
3310-3319, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-leve performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. 1oXiv:1006.088 2016.
Gaussian error linear units (gelus). arXiv preprint
10

---

Under review as a conference paper at ICLR 2023
Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos Venieris, and Nicholas Lane. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. arXiv preprint arXtv:2102.13451, 2021.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprini urXiv: (tti::/999.063330 06335, 2019.
Kevin Jarrett, Koray Kavukcuoglu, Mare' Aurelio Ranzato, and Yann LeCun. What is the best multi-stage architecture for object recognition? In 2009 IEEE 12th international conference on computer vision, pp. 2146-2155 2153,, IEEE, 2009.
Eunjeong Jeong, Seungeun Oh, Hyesung Kim, Jihong Park, Mehdi Bennis, and Gcong-Lyun Kim. Communication--fffccienn on-device machine learning: Federated distillation and augmentation under non-iid private data. arXiv preprini arXiv: 1....1177 1479, 2018.
Yihan Jiang, Jakub Konečný, Keith Rush, and Sreeram Kannan. Improving federated learning personalization via model agnostic meta learning. arXiv preprint UrXiv:1109..2440 2019.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning. arXiv preprint arXiv: 1912.04977, 2019.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems federated learning. Foundations and Trends@ in Machine Learning. 14(1-2):1-210 2021.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference Machine Learning, pp. 5132-5143. PMLR, 2020.
Ahmed Khaled, Konstantin Mishchenko, and Peter Richtárik. First analysis of local gd on heteroge- neous data. arXiv preprint 1 2019.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalziing neural networks. In Proceedings of the 3t international conference neural information processing systems, pp. 972-981, 2017.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMLR, 2019.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning, nature, 521(7553))436-444 2015a.
Yann LeCun et al. Lenet-5, convolutional neural networks. URL: http://yann. lecun. com/exdb/lenet, 20(5):14, 2015b.
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10713-10722, 2021.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429-450, 2020.
Xiang Li. Kaixuan Huang, Wenhao Yang. Shusen Wang, and Zhihua Zhang, On the convergence of fedavg non-iid data. arXiv preprint 1907.02/000200 2019.

---

Under review as a conference paper at ICLR 2023
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion federated learning. Advances in Neural Information Processing Systems, 33:2351-2363, 2020.
Andrew L Maas, Awni Y Hannun, Andrew Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml, volume 30, pp. 3. Citeseer, 2013.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication-effiee learning of deep networks from decentralized data. In Artificial intelli- gence and statisticy, pp. 1273-1282. PMLR, 2017.
Diganta Misra. Mish: A self regularized non-monotonic activation function. arXiv preprint
arXiv: 08688, 2019.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Ieml, 2010.
Keiron O' Shea and Ryan Nash. An introduction to convolutional neural networks. arXiv preprint OrXiv:1511.084: 2015.
Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00295, 2020.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chier Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018
Reza Shokri and Vitaly Shmatikov. Privacy-preserving deep learning. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pp. 1310-1321, 2015.
Virginia Smith, Chao-Kai Chiang. Maziar Sanjabi, and Ameet Talwalkar. Federated multi-task learning, arXiv preprint arXiv:1705.10467 2017.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. arXiv preprint arXiv:2002.066444 2020a.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611-7622 2020b.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In International Conference on Machine Learning, pp. 1252-7261. PMLR, 2019.
Jie Zhang, Song Guo, Xiaosong Ma, Haozhao Wang, Wenchao Xu, and Feijie Wu. Parameterized knowledge transfer for personalized federated learning, Advances in Neural Information Processing Systems, 34, 2021.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv: 806.022806.00082,, 2018.
12

---

Under review as a conference paper at ICLR 2023
Appendix
A RELATED WORK
A.I
ACTIVATION FUNCTIONS IN NEURAL NETWORKS
In deep neural networks, using activation functions is ubiquitous technique for learning non-linear latent representations; an input signal is transformed into the non-linear output centered on zero. Recent evolution occurs along with the enhancement representation power and the efficiency of computational costs. In the experiment, we use the following non-linear activation functions:
Tanh. hyperbolic tangent function that is smooth zero-centered function. The equation of Tanh is:
Tanh(x) 6 -
e
It is known that it is zero-centered, but computationally expensive and causes vanishing gradient problems neural networks become deeper.
HardTanh. Another variant of Tanh that involves lower computational costs owing to its piece-wise linearity. It is another variant of the hyperbolic tangent function, which represents computationally more efficient form of tanh:
(Tanh((r) = min(1, m-x
ReLU. An wise threshold operation on each input element where values less than zero are to zero otherwise used as is. The equation of ReLU (Hahnloser et al. 2000 Jarrett al. 2009
Nair & Hinton 2010) is:
ReLU(x) = max(0,
ReLU is the abbreviation of Rectified Linear Unit, a modified linear function. When the input value of ReLU is negative, the gradient of its output value is zero; the model does not learn. ReLU has shown
great performance with Convolutional Neural Networks (CNN) (Krizhevskk et al. [012 LeCun et al. 2015b). Since ReLU is computationally cheap, it is still commonly used regardless of numerous attempts to replace it (Maas et al., 2013 Goodfellow et al. 2013 He et al. (2015) Clevert et al. 2015 Klambauer al.
et al. 2018).
Leaky ReLU. variant of ReLU that is designed to prevent the Dying Neuron problem, where gradients will not be zero at any time during training. The equation of Leaky ReLU (Maas et al. 2013) is:
LReLU(z)) max(0.01z,.z)
is ReLU multiplied by a tiny constant on the negative part. Due to the small range, the graphs are drawn almost similarly with ReLU,
Swish. An unbounded function that uses the sigmoid function in its formula. The curve similar to ReLU, but has is smooth and non-monotonic. The equation of Swish (Ramachandran et al. 2017) is:
Swish(ar) = r sigmoid(z)
This function shows better accuracy than ReLU in deep neural networks regardless of batch size.
Mish. A self-regularized non-monotonic activation function. The curve is similar to Swish, but it has a stronger regularization effect and a smoother gradient. The equation of Mish (Misra 2019) is:
Mish(ar) = x tanh(In(1 + e")))
This function has the characteristic of allowing gradients to flow better than the Relu Zero Bound because it allows some negative numbers.
GeLU. A Gaussian distribution-involved smooth variation ReLU. It is often used in NL.P tasks and vision tasks with Vision Transformer (Dosovitskiy et al. 2020 models. The equation of GeLU (Hendrycks & Gimpel, 2016) is:
GeLU(z)= x.
- 1+ tanh - 0.04445557
GeLU is derived by combining the characteristics of dropout, zoneout, and ReLU.
13

---

Under review as a conference paper at ICLR 2023
A.2 ALGORITHMS FEDERATED LEARNING METHODS
We use both FedAvg and FedProx for federated learning methods. AAgoriithm I shows the algorithm of FedAvg and Algorithm shows the algorithm of FedProx. FedProx is similar to FedAvg in that selects selection of clients at each round, performs local training, and then averages client's weight generate a global update. However, the difference between FedAvg and FedProx is shown in line 6. For local training, FedAvg trains each client's 's model using SGD with its local data whereas FedProx, trains each client with additional proximal term, ||w w'll.. Using the proximal term which contributes to the method's stability by efficiently reducing the impact of variable modifications.
Algorithm Federated Averaging (FedAvg)
1: Input: K, T, 7), E, w", N, [1, ,N]
2: for = 0,... T do
3: Server selects subset St randomly which includes number of K devices
4: Server send w² to all selected devices
5: for 0,.... E 1 do
6: Selected device k E S, updates their local weight f+1 using SGD with step-size
7: end for
8: Selected device k E S₁ sends their local weight w back to the server
9: Server aggregates the local weights, w and gets new server weight w K EkESE w/
10: end for
Algorithm 2 FedProx
1: Input: K, 7 u. E, w N, & E [1, ,N]
2: for t= 0, T do
3: Server selects subset Se randomly which includes number of K devices
4: Server send wt to all selected devices
5: i 0,...
do
6: Selected device k E St updates their local weight (w); w minh (K) (w; w²) (k)
I| w 11 with step-size n7
7: end for
8: Selected device l E S, sends their local weight w (+1 back to the server
9: Server aggregates the local weights, w ++1 and gets new server weight w = K ErEs, wi
end for
B IMPLEMENTATION DETAILS
B.1 MODEL ARCHITECTURE
Figure 6 shows ConvNet with five different depth (3,4,5,6, and 7). Each version ConvNet has convolution layer corresponding the number after the model (e.g., ConvNet.3 has three convolution layers and ConvNet has seven convolution layers). The details of each convolution layer is shown in Table 8 ConvNet with varying depth employ convolution layers in the order shown in Table (c.g.. ConvNet 3 use Convl, Conv2, and Conv3 whereas, ConvNet? use Convl, Conv2, Conv3, Conv4, Conv5, Conv6, and Conv7).
B.2 DATASET STATISTICS
We use both CIFAR-I0 10 and CIFAR-100 100 for experiments. As shown in Table 9
CIFAR-10 consists of 60000 images of size 32x32. is divided into 10 classes, and each class consists of 6000 images. Also, each class has 5,000 training images and 1,000 test images. CIFAR-100 also consists of 60000 images with a size of 43. It is classified into 100 classes, and each class consists of 600 images. divided into 20 superclasses, which do in our experiments. Also, each class has 500 training images and 100 test images.
These 100 classes
14

---

Under review as a conference paper at ICLR 2023
Image
3' Joon, 64
3x.i, 54
som, b4
3x35 comy,
5x.5 comy, bi
Sctiattion
acthosion
3x30om, 64
3x3.con, 64
3x3,cm, 64
3x3 con.
3x3 comy 64
actvvaion
activation
acoiation fanction
actission
3x 128
3x3
3x3
activation funct
activation functon
acovzooon function
acticcion function
achiationnnn funciin
actlatioon hactto
activation finction
actinntion funttion
activatin functon
acoia2tio function
acciction function
activation function
3 512
max pani /
max pool
max poel 77
mox pool /2
max port,
Conseer3
Conviett
Condlets
Confleit
Condiet?
Figure 6: The architecture of ConvNet with different number of convolution layers.
Table 8: Setting of each convolution layer in ConvNet. Each version of ConvNet the number convolution layer as much as their version number. (i.e ConvNet.3 uses conv1 through conv3 and ConvNet7 uses convl I through conv7.)
Layer
Number of Input Filter
Number Output Filter
Kernel Size
Padding
Convl
3
64
3x3
Conv2
64
64
3x3
Conv3
64
128
3x3
Conv4
128
128
3x3
Conv5
128
256
3x3
Conv6
256
256
3x3
Conv7
256
512
3x3
Table 9: Dataset statistics.
Data
Train examples
Test examples
Class Number
Task
CIFAR-10
50,000
10,000
10
Image Classification Image Classification
CIFAR-100
50,000
10,000
100
C ADDITIONAL EXPERIMENT RESULT
C.I CONVNET4 RESULT
Experiment results in section fix variables such as N, R, and a. Table 10 shows the result with all combinations of / and with N 20. Table shows the result with all combinations of It and with N = 100. Table 12 shows the result using CIFAR- -000 as the dataset with all combinations of R and with N 100. Comparing Table 10 and Table II
that with bigger client number, the Tanh- like activation function surpasses the recent SOTA activation functions. Additionally, with complex dataset (CIFAR-100) shown in Table 12 the Tanh-like activation functions outperform
15

---

Under review as a conference paper at ICLR 2023
the recent SOTA activation functions with bigger accuracy gap. The low accuracy of the Tanh-like activation functions using CIFAR-10 as dataset with a 0.01 occurs due to severe tough training settings and fails to find the optimum.
To checkout the performance when the Tanh-like activation functions does not fail to find the optimum, additional setting with learning rate 0.005. Table 13 shows the accuracy of ConvNet. with fixed dirichlet constant a 0.01, which the Tanh-like activation functions failed to find optimum Table 10 and Table 11 The Tanh-like activation functions surpass the recent SOTA activation functions in all conditions. In addition, the Tanh-like activation functions surpass the highest accuracy with dirichlet constant a 0.01 of the recent SOTA activation functions presented Table 10 and Table II
Table 10: Server accuracy of ConvNet using four different and four different participation R.
where N = 20. For different dirichlet distribution constant, 0.01 is the most non- setting and 0.01 with participation ratio 0.1, 0.2, 0.3 could not train, leave blank at = 0.01 + 0.1 with participation ratio 0.1, 0.2, 0.3. The most right columns show the accuracy drop non- - increases.
is the most IID setting. Since using Linear
Participation Ratio
Activation Function
10
0.1
0.01
a=
10
a
0.1
a m 0.1 0.01
Linear
Tanh
66.83
66.40
62.67
22.44
0.43
3.73
40.23
HardTanh
0.1
ReLU
69.27
67.40
54.47
27.22
1.87
12.93
27.25
Leaky ReL.U
69.29
67.39
54,46
28.49
12.93
25.97
Swish
66.33
64.56
53.74
30.48
1.77
10.82
23.26
Mish
69.00
67,58
60.38
33,13
1.42
7.20
27.25
GeLU
68.52
66.36
54.05
30.71
12.31
23.34
Lincar
66.48
66.48
63.54
2.94
Tanh
70.22
69.55
65.97
27.64
0.67
3.58
38.33
HardTanh
70.53
70.01
66.53
28.92
0.52
3.48
37.61
0.2
Leaky ReLU
ReLU
75.36
75.59
74.23
74.21
63.39
63.67
35.15
34.70
1,38
1.13
10.82
10.56
28.69
28.52
Swish
72.58
71.58
Mish
73.69
72.95
66.47
37,89
0.74
6.48
28.58
GcLU
Linger
61.15
67.54
66.12
10.00
-6.39
1.42
Tanh
71.95
71.85
67.75
30.01
0.12
4.08
37.74
HardTanh
72.33
71.49
68.80
30.51
0,84
2.69
38.29
0.3
Leaky ReLU
ReLU
78.62
78.65
77.77
78.13
70.22
70.17
41.31
41,00
0.49
7.60
7.91
28.91
29.17
Mish
77.32
76.46
69.34
45.16
0,86
7.12
24.18
GcLU
Lincar
68.65
67, 66
65 85
10.00
0.99
1.81
73.13
HardTanh
73.33
72.19
69.25
29.93
1.14
2.94
39.32
ReLU
Leaky ReLU
80.58
79.82
66.12
45.67
0.76
13.70
20.45
Swish
79.01
77.29
70.36
47.83
6.93
22.53
Mish
79.57
78.59
71.30
$2.16
0.98
7.29
19.14
GeLU
81.67
80.62
72.32
48.66
1.05
8.30
23.66
16

---

Under review as a conference paper at ICLR 2023
Table 11: Server accuracy of ConvNet 4 using four different and four different participation R, where 100. The most right columns show the accuracy drop as non-IIDness increases. Since using Linear at 0.01, 0.1 with participation ratio 0.1, 0.2, 0.3, and 0.4 could not train, leave a 0.1 with participation ratio 0.1, 0.2, 0.3, and 0.4.
blank 0.01 - 0.1,
Participation Ratio
Activation Function
10
0.1
0.01
10 +1
0.1
= 0.1 0.01
Linear
58.24
57.38
Tanh
58.98
57.86
46.61
31.97
1.12
11.25
14.64
HardTanh
ReLU
51.61
49.73
41.95
28.83
1.88
7.78
13.12
0.1
Leaky ReLU
51.60
49.76
7.71
Swish
47.01
46.28
40.00
29.88
0.73
6.28
10.12
Mish
50.00
49.59
43.37
33.13
0.41
6.22
10.24
GeLU
48.64
48.02
41.50
31.61
6.52
9.89
Linear
Tanh
64.49
64.
52.58
29.50
0.35
11.56
23.08
HardTanh
65.27
65.40
0.2
ReLU
$7.80
56.23
48.37
34.03
1.57
7.86
14.34
Leaky Rel.?
$7.85
56.16
48.34
33.92
7.82
14.42
Swish
52.62
$1.48
46.16
35.65
1.14
5.32
10.51
Mish
57.30
55.08
50.02
38.94
2.22
5.06
11.08
GeLU
55.59
54.34
47.46
36.09
6.91
Lincar
64.66
64.4
10.00
10.00
Tanh
67.34
66.57
61.79
31.35
0.77
4.78
30.44
HardTanh
68.35
67,86
61,75
29.93
0.49
6.11
31.82
0.3
Leaky ReLU
ReLU
62.62
62.43
60,34
60.38
50.76
50.67
37.91
38.02
2.05
2.28
9.71
9.58
12.76
12.74
Mish
65.53
62.21
$2.80
42,57
3.32
9.41
10.23
GcLU
Lineer
65.71
65.83
10.00
10.00
-0.12
Tanh
68.83
68.37
29.49
5.76
HardTanh
69.95
69.35
$9.45
27.22
0.60
9.90
32.23
0.4
Leaky ReLU
67.02
64.60
$3.17
41.17
2.42
11.43
12.00
Mish
68.96
67.27
56.67
44.41
1.69
10.60
12.26
GeLU
67.59
63.42
10.07
17

---

Under review as a conference paper at ICLR 2023
Table 12: Server accuracy of ConvNet using CIFAR-I00 as dataset and N 100. The most right columns show the accuracy drop as non-llDness increases. Since using Linear at 0.01 with participation ratio 0.1, 0.2, 0.3, and 0.4 could not train, leave blank at a = 0.01 - 0.1 with participation ratio 0.1, 0.2, 0.3, and 0.4.
Participation Ratio
Activation Function
0
0
a=
0.01
Linear
25.94
25.57
22.58
1.00
0.37
2.99
Tanh
26.56
26.03
21.81
13.47
0.53
4.22
HardTanh
29,04
36.66
23.70
14.66
0.38
4,96
ReLU
21.83
22.67
17.98
-084
4,69
7.01
Leaky ReLU
21.95
22.77
18.12
11.13
-0.82
4.65
6.99
Mish
23.75
23.30
18.42
11,86
0.45
4.88
6.56
GeLU
Lincar
31.78
31.89
28.39
1,00
-0.11
3.50
3.99
HardTanh
35.68
36.08
31.76
21.93
-0.40
4.32
9.83
ReLU
0.2
Leaky ReLU
26.93
27.77
24.04
15.68
-0.84
3.73
8.36
Swish
26.39
26,
4.59
Mish
29.40
28.89
24.89
16.41
0.51
4.00
8.48
GeLU
27.66
26.73
0,93
3.47
Linear
35.50
35.32
31.47
1,00
0,18
3.85
Tanh
HardTanh
39.05
39.40
35.03
24.09
-0.35
4.37
10.94
ReLU
30.24
Leaky ReLU
29:74
30.26
27.45
17.97
-0.52
2.81
9.48
Swish
29.96
29.53
25.34
17.37
0.43
4.19
7.97
Mish
32.33
32.71
29.19
20.16
40,38
3.52
9.03
GcLU
30.24
29.77
27.24
17.89
0.47
2.53
9.35
Lincar
37.48
37.53
33.79
1.00
-0.05
3.74
Tanh
40.01
40.85
37.13
26.35
-0.84
3.72
10.78
HardTanh
41.69
41.73
37.41
27.44
-0.04
4.32
9.97
ReLU
30.75
31.45
30.55
20.51
40.70
0.90
10.04
Leaky ReLU
31.05
31.61
30.79
20.61
-0.56
0.82
10.18
Swish
32.54
32.37
27.61
19.45
0.17
4.76
8.16
GeLU
32.37
32.12
29.27
20.73
0.25
2.85
8.54
Table 13: Server accuracy of ConvNet 4 with four different client participation (0.4, 0.3, 0.2, 0.1) and fixed Dirichlet constant 0.01 for additional setting using learning rate 0.005. We use N 20 and = 100 for the number of clients and CIFAR-10 10 as the dataset.
N
100
N
20
Activation Function
R 0.4
0.3
R
0.1
R 0.4
0.3
R 0.
Lincar
10.00
10.00
10.00
10.00
49.63
10.00
10.00
10.00
Tanh
46.30
43.73
40.85
36.96
51.81
47,59
42.14
33.17
Hard Tanh
47.92
45.06
42.31
38.30
51.86
48.69
42.44
34.06
ReLU
38.02
34.83
30.88
28.66
38.26
35.02
32.39
25.17
Leaky ReLU
38.17
35.15
31.13
28.74
38.53
35.19
32.59
25.32
Swish
36.15
33.72
32.05
28.43
41.93
37.57
33.68
26.32
39.72
36.94
34.67
30.56
46.11
40.88
28.32
GeL.U
37.24
34.57
32.64
28.46
43.80
39.54
35.47
27.15
18

---

Under review as a conference paper at ICLR 2023
C.2 RESNET RESULT
Table 14, Table 15, and Table sooss the result of Resnet20, Resnet32, and Resnet 44 with all combinations of R and with N 100. For all values of R and Resnet20 using HardTanh shows the highest accuracy. According to the existence of shortcut layer's existence, deeper layer can use features that the activation function has excluded via shorteut (He al., 2016), which helps to prevent the recent SOTA activation functions' accuracy drop. As a result, using Resnet 32 and Resnet 4, the recent SOTA activation functions have a small accuracy gap with the Tanh-like activation functions and surpass in some conditions.
Table 14: Server accuracy of Resnet20 using four different and four different participation R, where 100. The most right columns show the accuracy drop as non-IIDness increases.
Participation Ratio
Activation Function
10
0.1
0.01
10
0.1
0.1
0.01
Lincar
49.01
48.
24.63
10.00
13,97
Tanh
50.68
49.20
38.83
23.82
1.48
10.37
15.01
HardTanh
ReLU
48.81
48.12
37.53
23.34
0.69
10.59
14.19
Leaky RcLU
48.63
48.15
10.44
Lincar
56.93
56.96
45.73
26.70
-0.03
11.23
19.03
0.2
HardTanb
$9.60
$7.54
46.19
26.52
2,06
11.35
19.67
ReLU
56.90
56.95
Leaky ReLU
$6.78
56.49
45.18
26.10
0,29
11.31
19.08
Linear
63.17
61.67
49.35
28.25
1.50
12.32
21.07
Tanh
64,63
61.74
49.69
27.90
2.89
12.05
21.79
HardTanh
64.41
61.59
27.86
22.30
ReI.Il
61.99
61.59
48.71
27.58
0.40
12.88
21.13
Leaky ReLU
Lineur
66.24
65,30
$2.09
30.17
0.94
13.21
21.92
Tanh
68.08
65.05
52.92
28.92
3,03
12.13
24.00
0.4
HardTanh
67.90
64.91
53.69
29.51
2.99
11.22
24.18
Leuky ReLU
66.04
64.82
52.23
28.70
1.22
12.59
23.53
Table 15: Server accuracy of Resnet332 using four different a and four different participation R where N = 100. The most right columns show the accuracy drop as non-IlDness increases.
Participation
Lineer
50.06
47.94
37.39
21.95
2.12
10.55
15.44
Tanh
49.55
47.15
21.61
8.91
16.63
0.1
HardTanh
50.04
47.99
38.34
22.37
2,05
9.65
15.97
Leoky ReLU
50.71
48,57
37.33
23.91
2.14
11.24
13.42
Lincar
60.12
56.84
43.01
24.98
3,28
13.83
18.03
Tanh
60.03
56.81
43.92
25.90
3.22
12,89
18.02
HardTanh
ReLU
58.52
56.34
43.09
26.66
2.18
13.25
16.43
Leaky ReLU
Lincar
64.51
62,20
46.15
28.63
2.31
16,05
17.52
Tanh
64.60
47.77
14.19
0.3
HardTanh
63.74
62,20
47.73
28.03
1.54
14.47
19.70
Leaky ReLU
64.20
61.59
46.81
28.49
2.61
14.78
18.32
Lincar
68.45
65,75
49.49
30.28
2,70
16.26
19.21
Tanh
67.75
66.58
51.03
29.41
1.17
15.55
21.62
HardTahh
67.33
66.17
50.75
29.18
15.42
21.57
ReLU
67.52
65.55
49.72
30.18
1.97
15.83
19.54
Leaky ReI.U.
19

---

Under review as a conference paper at ICLR 2023
Table 16: Server accuracy of Resnet44 using four different and four different participation R where N = 100. The most right columns show the accuracy drop as non-IlDness decreases.
Participation Ratio
Activation
Lincar
49.68
47 17.28
39.56
23.26
2.40
7.72
16.30
49.13
8.28
0.1
HardTanh
49.33
47.10
38,88
23.77
2.23
8.22
15.11
Leaky ReLU
48.77
46.27
37.35
23.05
2.50
8.92
14.30
Lincar
57.55
55.98
27.50
1.57
10,84
17.64
Tanh
57.94
55.54
46.42
27.23
2.40
9.12
19.19
ReLU
56.89
$4,23
44.42
26.06
2,66
9.81
18.36
Leaky ReLU
53.76
Linear
63.29
61,20
47.78
30.71
2,09
13.42
17.07
Tanh
64.43
61.27
50.67
29.53
3,16
10.60
21.14
0.3
HardTanh
64.31
61.42
51.23
28.84
2,89
10.19
22.39
Leaky ReLU
62.63
$9.63
46.84
28.27
3.00
12.79
18.57
Lineur
67.30
65.09
51.25
32.67
2,21
13.84
18.58
Tanb
68.14
64.67
52.89
32,37
3.47
11.78
20.52
HardTanh
67.91
65.02
31,90
11.07
22.05
ReI.I
66.74
64,20
50.31
30.51
2.54
13.89
19.80
Leaky ReLU
20


Reviews:
[{"review_id": "iFd66qGRdNo", "reviewer": "Reviewer_H9Ky", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This study clarifies that the drop in accuracy varies according to the activation function in the FL. Due to the shape of the function, the accuracy of the SOTA activation function drops in the FL setting and HardTanh outperforms other activation functions in most environments.\nFurthermore, the authors provide guidelines and benchmark data for choosing activation functions in various FL settings.", "summary_of_the_review": "See Strength And Weaknesses", "strength_and_weaknesses": "Overall, the paper is well-written and easy to follow. The experimental results demonstrate that the proposed method is promising. The authors provide a guideline for selecting activation functions in FL, an explanation for the performance degradation and a novel activation function HardTanh.\n\nHowever, my concern is whether the reason why HardTanh works so well has something to do with poor training. I see that the authors use a learning decay of 0.1 at rounds 50 and 75, which might be beneficial for some activation functions. I suggest that the authors employ multiple\ntraining strategies to observe the result. And more additional experiments lead us to believe that the benefits of the HardTanh activation functions are independent of model training. Moreover, it will be more convincing to compare it with the latest SOTA models.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RMWY2RpD3T", "reviewer": "Reviewer_2hFV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In Federated Learning, there is little attention on how different types activation function affect the final accuracy. In this paper, authors discover that good activation function in centralized training is not good in FL settings and vice versa. Authors further perform some analysis on the weight parameters, latent representations and loss landscape resulted from different activation functions. They show that good activation functions in FL are the ones that exclude fewer features and has smoother landscape.  ", "summary_of_the_review": "I recommend weak accept of this paper. I think the overall question is important and novel. Though more work should be focused on analysis section and maybe some theoratical analysis.", "strength_and_weaknesses": "Strength:\n\n1: The overall idea is important and novel. I am not aware of other papers studying the effect of activation function in FL settings. And I am pretty surprised that activation functions can cause significant performance change in FL setting.\n\n2: The experiments seems to be solid and extensive. Authors consider several settings and different algorithms.\n\nWeakness:\n\n1: I am quite confused by the analysis, especially section 4.1. In the first half of 4.1, I quote \"This can be summarized simply by saying that the Tanh-like activation functions have low sensitivity to the accuracy drop in the FL aggregation step because they exclude a much smaller number of features than do the recent SOTA activation functions.\" Up to this point of the section, it seems to be only an intuition instead of explanation. Next, authors measure the CKA similarity and weight similarity between clients. They seem to say ReLU shows higher similarity in both cases compared with Tanh. My question is: if Tanh excludes fewer features in each client, shouldn't it show higher similarity in both CKA (the output features) and weight? From another perspective, as far as I understand, the client drift means each local client learns a significantly different model. Thus, if Tanh somehow alleviates client drift as author claims, shouldn't it result in similar models in all the clients? \n\n2: I think if there is some theoretical analysis on the activation function and how it affects the convergence, it will really strengthen the paper. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7zk16wcwen", "reviewer": "Reviewer_VoE5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper analyzes the effect of various activation functions on server accuracy in federated learning with heterogeneous clients. It shows that Tanh-based activation functions outperform ReLU-based activation functions in most cases, providing the guidelines for selecting activation functions in various federated learning scenarios. This paper also conducts a few experiments to investigate why Tanh-based activation functions are more robust to client heterogeneity than other activation functions. ", "summary_of_the_review": "This paper lacks contribution because (1) the main experiments are difficult to understand, missing many discussions, and do not support few claims, (2) analysis of model behavior does not support the main claim that the Tanh-based activation functions are robust to data heterogeneity of clients, and (3) the paper does not propose any method to handle the issue of existing activation functions in federated learning.", "strength_and_weaknesses": "### Strength\n\n- This paper provides an interesting observation that the choice of the activation function plays a critical role in the generalization performance in federate learning, and Tanh-based activation functions outperform Relu-based activation functions, which are widely used in centralized training, in various federated learning scenarios with heterogeneous clients.\n- This paper conducts comprehensive experiments that show how each aspect of the federated learning setup (the number of clients, non-IIDness, participation rate, backbone architecture) affects the accuracy according to the different activation functions and provides guidelines for selection activation functions in federated learning.\n\n### Weakness\n\n- The experimental results in Table 3 do not provide convincing evidence demonstrating that Tanh-like activation functions are virtually unaffected by non-IIDness.\n    - As illustrated in Table 11, Tanh-like activation functions show a much more significant accuracy drop when the data heterogeneity across clients becomes larger (Dirichlet parameter $\\alpha$ gets smaller). This trend becomes more significant when more clients participate per round. While these results support the claim that Tanh-like activation functions are robust to partial participation, but do not support the claim that Tanh-like functions are robust to the non-IIDness of each client.\n- The authors claim that the issue of client drift becomes severe since ReLU-based activation functions take fewer features than Tanh-like activation functions. However, Results in Figure 3 and Figure 4 rather show that Tanh-like activation functions have more client drift phenomenon: 1) low CKA similarity between clients, 2) large weight divergence between clients. Note that large weight divergence is known to be a cause of degenerated server accuracy and has been studied in many works [1, 2, 3]. Therefore, these experiments do not support the claim and should provide more convincing evidence.\n- Need more additional discussions and experiments:\n    - Why are the accuracies of ResNet-based architectures and EfficientNet lower thanConvNet4?\n    - 200 rounds of communication seems too short for the convergence of the model. The authors should provide convergence plots or report the results after the model is trained with enough communication rounds.\n    - There are no discussions about why large difference in accuracy among ReLU-based activation functions or among Tanh-like activation functions.\n    - In Table 4, when $N=20$, why Leaky ReLU performs better with $R=0.3$ than $R=0.4$?\n    - ReLU-based activation functions and Tanh-like activation functions may have different optimal learning rates. Authors should compare models trained with each optimal local learning rate.\n\n\n### Reference\n[1] S.P. Karimireddy et al., SCAFFOLD: Stochastic Controlled Averaging for Federated Learning, ICML, 2020.\n\n[2] D.A.E. Acar et al., Federated Learning Based on Dynamic Regularization, ICLR, 2021.\n\n[3] L. Gao et al., FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correction, CVPR, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-zOzWlAW95k", "reviewer": "Reviewer_fb3s", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the importance of different activation functions in the performance of the centralized and FL trained models. It tries to empirically show that there are considerable differences between the two settings. ", "summary_of_the_review": "The experimental results have many flaws and are not convincing. The claims are not well supported by the observations.", "strength_and_weaknesses": "Strength:\n- The question that the paper sets out to answer empirically is well-defined.\nWeaknesses:\n-  For an empirical work, the experiments are very limited and have many issues. \n- No error bars or multiple runs are used\n- The number of datasets that are considered are very limited and it is difficult to be certain if the results generalize.\n- The parameters of the training algorithms, e.g. lr, local batch size and steps, local lr, momentum are kept fixed and not tuned for different settings. I suspect this is one main reason that the performance of the models are so low and not on par with what they should be (e.g. for larger resnet models with almost iid, the performance should be >70% in my experience for cifar-10). Due to not well-tuned hyper-parameters for each setting, some weird observations happen, e.g. linear model performs better on cifar-100 compared to cifar-10 in table 1. Another example of this is the observation on 3rd paragraph of page 5.\n- Only a limited set of heterogeneity is considered (label heterogeneity) and there are no experiments on different local dataset sizes, covariate shifts, ...\n- Section 4 is poorly written and it is full of statements without substantial evidence. Unfortunately, the statements in section 4 are mostly vague, difficult to follow and independent of the results that are presented. In figure 3 and 4, it is not clear what is being presented and how does it relate to the other claims. It seems that learnability and heterogeneity are two different and opposing factors based on Figure 3 and 4. But it is not clear why they should both be in the favor of the tanh like activations. \n- Overall, the presented evidence for the claims that the paper makes are not very strong. Moreover, section 4 does not provide a clear picture or intuition on why the claims could be correct either.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject