Adaptive Model Pruning in Federated Learning through Loss Exploration

Authors: christian  internò
elena  raponi
niki  van stein
thomas  bäck
markus  olhofer
yaochu  jin
barbara  hammer
Venue: ICML_cc_2024_Workshop_WANT
Published: 2024-05-27 18:02:04

Abstract: The rapid proliferation of smart devices coupled with the advent of 6G networks has profoundly reshaped the domain of collaborative machine learning. Alongside growing privacy-security concerns in sensitive fields, these developments have positioned federated learning (FL) as a pivotal technology for decentralized model training. Despite its vast potential, FL encounters challenges such as elevated communication costs, computational constraint, and the complexities of non-IID data distributions. We introduce AutoFLIP, an innovative approach that utilizes a federated loss exploration phase to drive adaptive model pruning. This innovative mechanism automatically identifies and prunes unimportant model parameters by distilling knowledge on model gradients behavior across different non-IID client losses, thereby optimizing computational efficiency and enhancing model performance on resource-constrained scenarios. Extensive experiments across various datasets and FL tasks reveal that AutoFLIP not only efficiently accelerates global convergence but also achieves superior accuracy and robustness compared to traditional methods. On average, AutoFLIP reduces computational overhead by 48.8% and communication costs by 35.5%, while maintaining high accuracy. By significantly reducing these overheads, AutoFLIP offer the way for efficient FL deployment in real-world applications, from healthcare to smart cities.

Main paper: 
 introduction: 
 The proliferation of smart devices at the network edge, coupled with advancements in 6G networks, has created a decentralized setting. Multiple participants store their data locally, Accepted to the Workshop on Advancing Neural Network Training at International Conference on Machine Learning (WANT@ICML 2024).which offers an opportunity for collaborative model training, enhancing robustness and generalization. Distributing the computational load across these devices results in faster training times and lower energy consumption compared to centralized approaches. However, distributed Machine Learning (ML) faces significant challenges. Efficient communication and coordination among participants are crucial, as each device holds only a subset of the data. This requires designing algorithms that minimize data exchange while ensuring high-quality model convergence. Device heterogeneity, including differences in computational power, storage, and bandwidth, further complicates distributed training. Algorithms must adapt to such environments to scale up distributed learning. Privacy and security concerns, along with regulations like the European GDPR (htt, 2016) , add another layer of complexity (Hoffpauir et al., 2023) . With sensitive data distributed across various devices, ensuring the privacy of individual data points becomes essential. For example, medical data stored in hospitals and personal devices is valuable for training diagnostic models but is also subject to strict privacy and security regulations.In this context, Federated Learning (FL) (Zhu et al., 2021) emerges as an effective strategy for training always more complex DL models while preserving the privacy of the data. FL facilitates collaborative model training across multiple devices without exposing local data. A central server, i.e., a global model, coordinates this process by aggregating the updates from locally trained models, which ensures a secure learning environment. Current FL research focuses on enhancing privacy and adapting ML workflows for specific uses, often with predetermined ML model configurations. Tasks related to computer vision may involve well-known neural network (NN) architectures like VGG-16 (Simonyan & Zisserman, 2015) (138 million parameters) or ResNet-50 (He et al., 2015) (25.6 million parameters). However, these complex networks risk overfitting, especially with small training data sizes. FL systems typically expect clients to have high-speed processors and sufficient computational power for local calculations and parameter updates. Yet, many edge devices, such as smartphones, wearables, and sensors, have limited computing and memory capacities, posing a challenge to DL model training systems (Hoffpauir et al., 2023) . Additionally, communicating DL models with millions of parameters presents significant obstacles for FL transmission (Shlezinger et al., 2020; Asad et al., 2023) . Therefore, using FL effectively with edge devices that have limited computational capabilities, while maintaining efficient communication, remains an active research question. FL's effectiveness is further hindered by the prevalence of non-IID data in real-world scenarios (Zhu et al., 2021; Karimireddy et al., 2020) . Non-IID data refers to the unique statistical properties of each client's dataset, reflecting their varied environments. This creates conflicting training goals for local and global models, leading to convergence towards different local optima. As a result, client model updates become biased, impeding global convergence (Zhu et al., 2021; Karimireddy et al., 2020) . These challenges underscore the need for personalized and innovative approaches in FL, particularly in optimizing and compressing models to improve inference time, communication cost, energy efficiency, and complexity, all while maintaining satisfactory accuracy.Our contribution. We introduce a novel automated federated learning approach via informed pruning (AutoFLIP), which uses a novel loss exploration mechanism to automatically prune and compress DL models. In our assumed single-server architecture, each client operates on the same initial deep NN structure that automatically prunes itself at each round, based on the extraction of shared knowledge for an informed model compression. Specifically, by analyzing the variability of weights during a local exploration phase, which provides insights into gradient behaviors on the loss landscapes across clients, and subsequent information aggregation, the Deep Learning (DL) models involved in a FL round are pruned automatically. This strategy allows for dynamically reducing the complexity of the models in FL environments, thereby optimizing performance with limited computational resources at the client level. With our experiments over various datasets, tasks, and realistic non-IID scenarios, we provide strong evidence of the effectiveness of AutoFLIP.Reproducibility. Our code for reproducing the experiments is available on GitHub.1  
background and related work: 
 Pruning in Deep Learning. Following the assumption that a DL model can contain a sub-network that represents the performance of the entire model after being trained, model pruning is a good strategy to reduce computational requirements of resource-constrained devices (Mozer & Smolensky, 1988a; LeCun et al., 1989; Janowsky, 1989) . Most pruning approaches balance accuracy and sparsity during the inference stage by calculating the importance scores of parameters in a well-trained NN and removing those with lower scores. These scores can be derived from weight magnitudes (Janowsky, 1989; Han et al., 2015) , first-order Taylor expansion of the loss function (Mozer & Smolensky, 1988b; Molchanov et al., 2017) , second-order Taylor expansion (LeCun et al., 1989; Hassibi & Stork, 1992; Molchanov et al., 2019) , and other variants (Louizos et al., 2018; Singh & Alistarh, 2020) .Another recent research direction in NN pruning focuses on improving training efficiency, divided into two categories: pruning at initialization and dynamic sparse training. Pruning at initialization involves pruning the original full-size model before training based on connection sensitivity (Lee et al., 2019) , Hessian-gradient product (Wang et al., 2020) , and synaptic flow (Tanaka et al., 2020) . However, since this method does not involve training data, the pruned model may be biased and not specialized for the task. Dynamic sparse training iteratively adjusts the pruned model structure during training while maintaining the desired sparsity (Dettmers & Zettlemoyer, 2019; Evci et al., 2020) . This approach requires memory-intensive operations due to the large search space, making it impractical for resource-constrained devices. Initial attempts to use pruning for deploying deep neural networks on resource-limited devices have utilized pre-trained CNNs in a centralized setting (You et al., 2019; Lin et al., 2020) . However, this approach can lead to reduced data privacy, higher costs, poor adaptation to local conditions, suboptimal performance on diverse data, and latency in real-time applications.Pruning in Federated Learning. The widely accepted FL standard is known as FedAvg (McMahan et al., 2023) . It distributes a global model to clients for local training and aggregates it by averaging their parameters. Empirical studies have shown the robustness of this approach, even when handling non-convex optimization problems (Das et al., 2022) . As a result, it is commonly used as a standard for evaluating newly developed FL protocols. In this study, we will compare the performance of the proposed AutoFLIP method to FedAvg, with different State-of-the-Art (SotA) FL pruning approachs, as tested in (Wu et al., 2023) . In fact, since data remains locally stored and cannot be shared, traditional centralized pruning approaches that rely on access to training data are not feasible in FL.In the context of FL, there has been work focused on dynamic active pruning to increase communication efficiency during training. Liu et al. (2022) ; Zhou et al. (2021) introduced a method where pruning decisions are made dynamically based on the model's real-time performance evaluation, which significantly reduces the data exchanged during training but adds computational complexity to client devices. Jiang et al. (2023) introduced PruneFL, a FL method that incorporates adaptive and distributed parameter pruning. Their approach utilizes an unstructured method that does not take advantage of the collective insights of participating clients to develop a cooperative structured pruning strategy. This is in contrast to the objectives of AutoFLIP, which seeks to harness client-specific knowledge to facilitate a structured approach to pruning. Lin et al. (2022) introduced a novel approach for adaptive per-layer sparsity, however without incorporating any parameter aggregation scheme to reduce the error caused by pruning. This challenge was addressed by Tingting et al. (2023) by moving the pruning process to the global model that works on a computationally more powerful server. The pruned model is distributed to each client, where it undergoes training. Subsequently, each client sends back to the server only the updated parameters, restoring the full structure of the model at the server. Although this study includes various parameter selection criteria from the literature, its pruning method does not incorporate the information gathered during model training. This contrasts with our strategy, AutoFLIP, which leverages such information to enhance the pruning process. Yu et al. (2023a) proposed Resource-aware Federated Foundation Models, focusing on integrating large transformer-based models into FL, with the limitation of not exploring other architectures. Our method, AutoFLIP, diverges by introducing a pruning strategy that avoids the need for continuous evaluation of parameter significance and is applicable across various FL aggregation algorithms and model architectures.training for E epochs using its local dataset D i . The local training aims to minimize the local objective function L(W i ) using stochastic gradient descent (SGD): let W r i be the local model parameters of client i at round r, the update rule is given by:W r+1 i = W r i -η∇L(W r i ),where η is the learning rate. global using a weighted average:W r+1 global = 1 K K i=1 W r+1 i. This iterative process is repeated for R rounds the termination criterion is met. 
preliminaries: 
 We start by defining the notation we use and presenting the standard formulation of FL, and then introduce the problem definition and objective.Notation: We consider a total number of C clients. At each FL round, K clients are chosen and trained on different batches of size B for E epochs. The total number of rounds is R, which represents our termination criterion. For the exploration phase, we denote with C exp the number of clients selected, which, in this study, we take as the totality C of available clients. The exploration lasts for E exp epochs.In the conventional FL setting, each client i (1 ≤ i ≤ K) possesses its own data distribution p i (x, y), where x ∈ R d represents the d-dimensional input vector and y ∈ {1, . . . , M } is the corresponding label from M classes. Each client has a dataset D i with N i data points:D i = {(x (1) i , y (1) i ), . . . , (x (Ni) i , y (Ni) i )}.It is assumed that in a non-IID scenario the data distribution p i (x, y) varies across clients. These data distributions p i (x, y) are sampled from a family E of distributions. The objective is for the clients to collaboratively train a global model with parameters W global , which will perform predictions on new data. The global loss function for a data point (x, y) is denoted by L(W global , x, y), where the global objective function to be minimized is defined as:EQUATIONwith E (xi,yi)∼pi representing the expected loss over the data distribution p i for each client i with parameters W .The optimization process involves several key steps:1. Client Selection: A subset of K clients is selected from the total C clients.The non-IID nature of the data introduces challenges in ensuring the global model effectively generalizes across diverse client data distributions. The variance σ 2 ∆W of the weight updates received by the global model, as discussed by Zhu et al. (2021) , can introduce noise and bias trajectories that potentially slow convergence and affect global model accuracy. Our objective with AutoFLIP is:i) Minimize σ 2∆W through an informed adaptive pruning strategy across clients to mitigate noise and biases into the client trajectory, which could slow the global model convergence. This approach aims to reduce the impact of divergent client learning paths caused by non-IID data.ii) Enhance training and communication efficiency. Pruning reduces the number of parameters that need to be communicated between clients and the server, thus lowering the communication overhead and expediting the overall training process. 
methodology: 
 AutoFLIP is an automatized FL approach that utilizes informed pruning through a federated client loss exploration process. Inspired by the idea of utilizing agents with similar tasks as scouts which explore the conformation of different loss function landscapes from Nikolić et al. ( 2023), AutoFLIP introduces a preliminary step to the FL rounds, which we term federated loss exploration phase. Here, a C exp portion of clients (or the totality C), which inherit their model structure from the global model, explore for a number of E exp exploration epochs their loss landscape using its local dataset D i . Based on this, for each client c exp i , we compute a local guidance matrix G locali , which records how important a certain parameter W i (weight or bias) is in terms of parameter deviation, i.e., change in parameter value after the exploration from its Xavier normal initialization (Kumar, 2017) value, which reflects loss variability. Afterward, we aggregate the information collected locally in a global pruning guidance matrix P G global on the server, which will generate an informed pruning mask to guide the pruning of the client models. The pruning workflow of AutoFLIP is illustrated in Figure 1 . Please note that the initial federated loss exploration, computation of parameter deviations, and definition of local guidance matrices occur only once at the beginning of the FL optimization process as a preliminary procedure. In contrast, the global guidance matrix and subsequent pruning strategy are automatically redefined in each FL round, considering the clients participating in that round.To summarize, the iterative procedure consists of (1) pruning local models using the updated pruning guidance matrix, (2) training the pruned local models, (3) aggregating the locally trained model parameters and (4) evaluating performance and updating the pruning guidance matrix in each FL round.In AutoFLIP, the model initialization phase is augmented by a crucial federated loss exploration phase, allowing clients to explore their loss function landscapes. We envision each client as an explorer delving into different regions of their loss landscape. Through this exploration, they can identify crucial dimensions and those that can be disregarded based on their experience by quantifying gradient variability during the exploration. Subsequently, they transmit this knowledge to the server, which updates a pruning guidance mask P G global . This mask is then shared among participating clients in each FL round to guide the evolution of client model structures within an informed pruning session.To construct the mask P G global , we begin with an initial exploration phase conducted on C exp clients. In this study, we consider C exp = C. In our study we let explore the clients for E exp = 150 epochs, and for each model parameter we evaluate its evolution in the search space during the loss exploration. This evaluation is conducted by calculating the deviation D i,m for the m th parameter of a client model i as the squared difference between the initial (W Initial i,m ) and final (W Final i,m ) parameter values after E exp epochs of exploration:D i,m = (W Initial i,m -W Final i,m ) 2 .(2)Using stochastic gradient descent for exploration, the deviation D i,m in Eq. ( 2) serves as a measure of gradient variability on the loss landscape for parameter m during the preliminary exploration phase before the actual FL procedure. The greater the variation in the parameter space, the faster the improvements in loss: the update rule for a parameter in stochastic gradient descent isW (eexp+1) i,m = W (eexp) i,m -η∇L i W (eexp) i,m ; D i ), where W (eexp) i,m and W (eexp+1) i,mare the values of the parameter m at the exploration epochs e exp and e exp + 1, η is the learning rate, and∇L i W (eexp) i,mis the gradient of the loss function of client i with respect to the parameter m at epoch e exp using its local dataset D i . Given the gradient update rule, the deviation in W i,m from the initial to the final exploration epoch E exp can be approximated toW Final i,m -W Initial i,m ≈ -η Eexp t=1 ∇L i W (t)i,m ; D i . To ensure non-negativity and highlight larger deviations more severely, we take the square of this value. This squared deviation measure D i,m approximates the square of the sum of gradients affecting the parameter evolution, indicating the significance of parameter updates on loss variability during the exploration phase. By squaring the sum of the gradients, we ensure that the deviation measure is always non-negative and that larger deviations are highlighted more severely than smaller ones.The C exp clients compile these deviations into a local matrix G local , whose entries are the deviations for the model parameters. At each FL round, where only K clients are involved, the server aggregates the G local matrices associated to those client to formulate G global through a normalization process: where elements below T p are set to 0 and those above are set to 1:EQUATIONP G global,m = 0 if G global,m < T p 1 otherwise (4)The threshold T p directly determines the compression ratio of the model by setting the proportion of parameters to be pruned. Given their smaller influence, parameters corresponding to 0 are flagged for pruning, whereas those marked with 1 are retained, indicating important search directions within the model parameter space. During each FL round, the K participating clients update P G global by incorporating their G local deviation values derived from the initial loss exploration phase.To select an appropriate T p , consider the desired compression ratio for the model. This ratio reflects the extent to which the model needs to be compressed while maintaining acceptable performance. By carefully selecting T p based on the desired compression ratio and empirical validation, we can achieve a well-balanced model that is both efficient and accurate, tailored to the specific needs of the FL task. G locali = (W Initial i -W Final i ) 2 , ∀i ∈ [1, C exp ] 4: for round r = 1 to R doW (r) global = 1 K K k=1 W (E) k,pruned15: end for=0 only for the parameters with normalized deviations higher than a prescribed threshold T p . Pruning (Lines 8-9). During each round, clients use the pruning mask to compress their models. This happens through element-wise multiplication between their weight matrix and P G global at that FL round. Parameters aligned with a 0 in P G global are pruned; those corresponding to a 1 are kept. FL round with reduced client models (Lines 10-14). The standard algorithm FedAvg (McMahan et al., 2023) is used on the reduced framework. The pruned clients are trained. The server receives the local model updates and, upon aggregation, proceeds to update the global model with the FL aggregation strategy. Once updated, the global model is either ready for the next communication round or deemed ready for deployment if the convergence criteria are satisfied.Referring to (Fraboni et al., 2022; Fallah et al., 2020; Wang et al., 2023; Yin et al., 2024) , we base our convergence guarantees on a federated stochastic aggregation scheme. The authors' assumptions on Lipschitz smoothness, convexity of local loss functions, unbiased gradient estimators, finite client answering times, and specific client aggregation weights form the theoretical backbone of AutoFLIP. These conditions ensure that the learning process remains stable and converges efficiently even in the presence of non-IID data distributions. With AutoFLIP each client experiences the same pruning strategy with P G global , resulting in a substantial decrease in the variance ( σ 2 ∆W ) previously de-fined in Section 3.2, of weight updates for the global model. This uniform pruning strategy minimizes discrepancies in weight adjustments across clients by focusing updates on critical weights identified during the federated loss exploration phase. The reduction in variance helps to alleviate the bias caused by the non-IID setting, as shown in the work of (Zhu et al., 2021) , thus promoting better global convergence.Furthermore, (Yang et al., 2023; Redman et al., 2022; Tukan et al., 2022; Isik et al., 2022) provide a theoretical foundation for which pruned NNs can effectively learn signals.They demonstrate that pruning preserves the signal's magnitude in features and reduces noise, leading to improved generalization. These studies highlight that pruning, when done correctly, does not degrade the model's capacity to learn but rather focuses the learning on more relevant features. By focusing on parameters with significant contributions to the loss function, AutoFLIP ensures that the essential features are retained, thus maintaining or even enhancing model accuracy. As illustrated in Figure 2 for different NN topologies, the parameters in G global with minimal variability during the federated loss exploration phase are pruned, while those exhibiting high deviations, indicative of high gradients and significant contributions to the loss function variability, are retained. Note that higher frequencies are recorded for smaller deviation values, indicating that many parameters, according to our pruning strategy, are non important.AutoFLIP enhances communication efficiency in FL by reducing model sizes transmitted between clients and the server, thus lowering bandwidth requirements for FL rounds. Its selective updating mechanism ensures only essential parameters, those significantly affecting model performance, are communicated. Integrating AutoFLIP into FL systems enhances faster training and inference times, lower energy consumption, and improved model scalability. The appendices D, E and F provide a detailed analysis of how AutoFLIP accelerates inference and improves training efficiency, demonstrating its significant role in lowering computational costs and boosting FL's overall efficiency. 
experiments: 
 Inspired by (Hahn et al., 2022) , we benchmark AutoFLIP across established datasets to evaluate its robustness in various non-IID environments. We explore three distinct partitioning approaches for creating strongly non-IID conditions: a Pathological non-IID scenario, which involves clients using data from two distinct classes, employing MNIST with a six-layer CNN (7, 628, 484 per client, using CIFAR100 with ResNet (23,755,900 parameters), and a LEAF non-IID scenario, which adopts the LEAF benchmark (Caldas et al., 2019) with FEMNIST and Shakespeare datasets. For FEMNIST, a CNN architecture with 13,180,734 parameters is used. For Shakespeare, we consider a two-layer LSTM model with 5,040,000 parameters. Further details on these scenarios are provided in Appendix B.We evaluate AutoFLIP against both FedAvg without any model compression and with SotA FL pruning strategies, incorporating various parameter selection criteria: Random, L1, L2, Similarity, and BN mask, as described in (Wu et al., 2023) . The experimental setup involves C = 20 (for LEAF non-IID scenario we employ C = 730 for Shakespeare and C = 660 for FEMNIST) clients, a batch size B = 350, and a learning rate η = 0.0003 over 200 total rounds R with K = 5 (for LEAF non-IID scenario K = 20) clients selected per round. We incorporate a server momentum of 0.9 and use an SGD optimizer with weight decay. The exploration phase consists of up to E exp = 150 epochs, and the pruning threshold is set to T p = 0.3. Data is divided into 80% for training and 20% for testing, with global model performance assessed by the average prediction accuracy on the test sets. To ensure statistical validity, each experiment is repeated 10 times. We measure the compression rate to evaluate model size reduction and its impact. Experiments were conducted with an Intel Xeon X5680, 128 GB of DDR4 RAM, and an NVIDIA TITAN X GPU.Pathological non-IID) Here, AutoFLIP achieves an average client compression rate of x1.74. At each round, we remove on average 3244298 parameters of the six-layer CNN for each participant client. For the EfficientNet-B3, we obtain an average compression rate of x2.1 with 5677458 deleted parameters. For a fair comparison with the baselines, we ensure that the number of parameters pruned matches the compression ratio of AutoFLIP, quantified as 42% for the six-layer CNN and 52.38% for EfficientNet-B3.The first two subplots in Figure 3 show the evolution of global model accuracy during the FL rounds for the fourlayer CNN with the MNIST dataset and for EfficientNet-B3 with the CIFAR10 dataset. Refer to Appendix C for the evolution of the loss metric. In the case of the MNIST, the early rounds of FL show that AutoFLIP achieves slightly higher accuracy compared to both FedAvg and the other FL pruning strategies, among which RandomPruning emerges as the top performer. This indicates a faster convergence rate for our proposed method. However, the performance of the three baselines soon becomes comparable, with no clear superiority as the FL procedure progresses. We attribute this to the simplicity of the prediction tasks on the MNIST dataset compared to the excessive complexity of the four-layer CNN, which already possesses extremely good prediction capabilities that cannot be further enhanced by pruning. For the CIFAR10 dataset, we do not observe any advantage in using AutoFLIP over the other baselines. Surprisingly, all methods exhibit severe fluctuations in the accuracy convergence profiles up to FL round 100, after which they stabilize and become comparable.Dirichlet-based non-IID) For ResNet, AutoFLIP achieves an average compression rate for the clients of x1.58, with 8,720,520 parameters pruned on average out of 23,755,900 total parameters. Hence, we adjust the percentage of parameters to be pruned to 36.71% for the different baselines.The third subplot in Figure 3 illustrates the evolution of the global model accuracy during the FL rounds for ResNet on CIFAR100. Here, AutoFLIP exhibits a performance enhancement throughout the considered training rounds. At round 200, it achieves an accuracy of 0.987, compared to 0.918 for FedAvg and 0.925 for RandomPruning. This enhancement signifies the robustness of AutoFLIP, showcasing its ability to maintain elevated performance levels when integrated with larger-complex neural networks and What stands out is the initial acceleration in convergence speed observed for AutoFLIP, firmly establishing it as a superior choice overFedAvg, RandomPruning and the other FL baselines. Furthermore, this superiority persists throughout the entire FL training procedure. The final average accuracy values are 0.985 for AutoFLIP, 0.905 for FedAvg, and 0.935 for RandomPruning on the FEMNIST dataset. For the Shakespeare dataset, the values are 0.815, 0.783, and 0.738, respectively. Here, even L1 proves to be competitive, reaching a final accuracy equal to 0.802. However, it demonstrates inferior initial convergence. 
conclusion and limitations: 
 We introduced AutoFLIP, an innovative automated federated learning (FL) approach that employs informed pruning to optimize deep learning (DL) models on clients with limited computational resources. Through extensive experiments across various non-IID scenarios, AutoFLIP has demonstrated its capacity to achieve high model accuracy and significantly reduce computational and communication overheads. It enhances convergence rates in federated settings and shows remarkable adaptability and scalability across diverse DL network architectures and multi-class datasets, particularly as the complexity of tasks increases.Limitations. AutoFLIP shows promise but has limitations. It is primarily tested in the popular efficient singleserver setting, not accounting for multi-server or hierarchical environments with diverse client capabilities and model structure. Our tests also assume standard conditions without data label noise.Future Research Directions. AutoFLIP underscores its potential for future research avenues, such as leveraging loss exploration for guiding complex Neural Architecture Search (NAS) tasks. Enhancements will focus on refining AutoFLIP's dynamic and adaptive pruning to better client personalization. We aim to perform comparison analysis with other strategies from other domain such us like NAS or Client Dropout. Further, the impact on data privacy and defense against adversarial clients during the federated loss exploration phase has to assessed. Research will also explore the extension of AutoFLIP to more complex DL architectures and its integration into real-world applications across various domains such as healthcare and mobile computing.Broader Impact. AutoFLIP enhances sustainability and efficiency in FL, reducing the energy footprint of training deep learning models. Its utility in sensitive sectors like healthcare and finance emphasizes its societal importance. However, deploying AutoFLIP requires careful consideration of ethical issues, including data privacy and biases. Proactive management and regulation are crucial to ensure its positive societal impact and responsible integration into critical fields. 
 
 Appendix: This experimental configuration is delineated by each client possessing data exclusively from two distinct classes within a broader multi-class dataset. Figure 6 illustrates this "pathological" data partitioning scenario within the CIFAR10 dataset across 20 clients. For our experiments, we select the MNIST dataset (Deng, 2012) with a six-layer CNN (7628484 parameters) and the CIFAR10 dataset (Krizhevsky, 2012) with EfficientNet-B3 architecture (10838784 parameters), following the guidelines in (McMahan et al., 2023) and (Tingting et al., 2023) . This advanced experimental setup, as introduced by (Hsu et al., 2019) , utilizes the Dirichlet distribution, modulated by a concentration parameter α. Let p = (p 1 , p 2 , ..., p N ) be the class distribution for a given client, where N is the number of classes. The Dirichlet distribution is defined as p ∼ Dir(α • 1 N ), where "Dir" denotes the Dirichlet distribution, α is the concentration parameter, and 1 N is a N-dimensional vector of ones. In this context, a low value of α, or α → 0, leads to distributions where most of the probability mass is concentrated on a single class, thereby indicating that each client's data is restricted to a single class. Conversely, as α → ∞, p approaches a uniform distribution, ensuring that the samples are evenly split across all clients. Figure 7 illustrates this "Dirichlet-based non-IID" data partitioning scenario within the CIFAR100 dataset across 20 clients, with individual colors denoting separate classes.To address the complexities of larger datasets, we have extended our evaluation to include CIFAR100 (Krizhevsky, 2012) with a α = 100, employing ResNet (23755900 parameters) (He et al., 2015) in alignment with the methodology proposed in (Hahn et al., 2022) .Utilizing the popular LEAF benchmark for FL (Caldas et al., 2019) , we selected the FEMNIST and Shakespeare datasets to simulate closer real-world FL scenarios, with each dataset designed for specific tasks. The FEMNIST dataset is defined for a multi-class classification challenge involving 62 distinct classes. Conversely, the Shakespeare dataset is tailored for a next-character prediction task, requiring models to predict the subsequent character from a sequence of 80 characters, thereby testing the model capabilities in sequential data processing and language modeling. The incorporation of the next-character prediction task allows for a comprehensive assessment of AutoFLIP adaptability and performance across diverse task types and deep neural network architectures, such as Long Short-Term Memory (LSTM) networks.In our experimental setup, we employed the FEMNIST-CNN architecture, as delineated in (Caldas et al., 2019) , for the FEMNIST dataset. For the Shakespeare dataset, we utilized a two-layer (LSTM) (5040000 parameters) model, in accordance with the specifications provided in (McMahan et al., 2023) .https://github.com/ChristianInterno/ AutoFLIP 


Decision: This submission introduces an adaptive pruning strategy for the federated learning setting; the technique involves computing weight importance during the FL exploration phase. Reviewer sentiment for this submission appears positive overall - the idea appears to be novel and has been explained well. However, writing (especially notation) could be improved and the objectives more clearly stated. I recommend acceptance (poster).