Federated Behavioural Planes: Explaining the Evolution of Client Behaviour in Federated Learning

Authors: dario  fenoglio
gabriele  dominici
pietro  barbiero
alberto  tonda
martin  gjoreski
marc  langheinrich
krum  median
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 19:47:08

Abstract: Federated Learning (FL), a privacy-aware approach in distributed deep learning environments, enables many clients to collaboratively train a model without sharing sensitive data, thereby reducing privacy risks. However, enabling human trust and control over FL systems requires understanding the evolving behaviour of clients, whether beneficial or detrimental for the training, which still represents a key challenge in the current literature. To address this challenge, we introduce Federated Behavioural Planes (FBPs), a novel method to analyse, visualise, and explain the dynamics of FL systems, showing how clients behave under two different lenses: predictive performance (error behavioural space) and decision-making processes (counterfactual behavioural space). Our experiments demonstrate that FBPs provide informative trajectories describing the evolving states of clients and their contributions to the global model, thereby enabling the identification of clusters of clients with similar behaviours. Leveraging the patterns identified by FBPs, we propose a robust aggregation technique named Federated Behavioural Shields to detect malicious or noisy client models, thereby enhancing security and surpassing the efficacy of existing state-of-the-art FL defense mechanisms. Our code is publicly available on GitHub.

Main paper: 
 introduction: 
 Federated Learning (FL), a privacy-aware deep learning (DL) approach in distributed environments, is a dynamic system where many clients collaborate to train a model without sharing sensitive data, thus mitigating privacy risks [1, 2] . Analyzing the behaviour of FL systems is crucial to detect anomalies-such as distribution shifts [3, 4] , biased data [5] , or adversarial clients [6] [7] [8] [9] -which may compromise the global model's predictive performance and introduce biases into its decisionmaking process. Various strategies have been developed to detect FL anomalies [10] [11] [12] [13] [14] [15] . However, Various studies have analysed models' behaviour in terms of predictive performance and decisionmaking processes independently. Predictive performance behaviour is primarily investigated in the context of non-linear optimisation using behavioural spaces [16] [17] [18] . This technique allows the visualisation of the predictive diversity of a set of regression models by considering the vector of errors that each model produces on a set of samples. The decision-making process is studied mainly in explainable AI (XAI) research using, for example, counterfactual explanations [19] . Counterfactuals can be used to identify relevant input features used by a model to make predictions, thus describing the position and orientation of the model's decision boundaries. However, these techniques have not yet been applied to FL systems, resulting in a lack of insight into how the behaviour of individual clients affects the overall model's accuracy and decision-making capabilities, leading to inefficiency in the training process.To bridge this gap, we introduce Federated Behavioural Planes (FBPs), a method designed to visualise, explain, and give insights into the dynamics of FL systems. Our key innovation involves the creation of two behavioural planes for FL clients: one to highlight their predictive diversity and another to emphasise their decision-making process diversity via counterfactuals. Building on the client behaviour information provided by FBPs, to show their practical utility, we propose Federated Behavioural Shields, a robust aggregation mechanism that enhances security against malicious or noisy clients by accurately weighting the client models according to their constructive contributions during training. The results of our experiments demonstrate that: (i) counterfactual generators jointly trained with FL systems produce valid and client-specific counterfactual explanations which effectively describe clients' decision-making diversity; (ii) FBPs facilitate the identification of clusters of clients with similar behaviours (e.g., normal vs. outlier clients), allowing for tracking of their trajectories during the entire training; (iii) Federated Behavioural Shields surpasses existing state-ofthe-art defense mechanisms, demonstrating that the information contained in FBPs provides valuable descriptors of client behaviour.2 Background Federated learning. FL systems [20, 3] involve a network of K ∈ N clients, coordinated by a central server, which collaboratively train a DL model. Each client k possesses a local and private dataset characterised by a set of z ∈ N features x (k) ∈ X (k) ⊆ R z and a set of u ∈ N class labels y (k) ∈ Y (k) ⊆ {0, 1} u . In each training round t ∈ N , each client trains a model f : X (k) → Y (k) on local data to maximise the likelihood L(θ (k) | x (k) , y (k) ). Once trained, clients send their local model's parameters θ (k) (t) ∈ R q , q ∈ N to a central server which aggregates these parameters using a permutation-invariant aggregation ⊕ : R q×K → R q (such as the mean or median). The server then sends the aggregated model parameters θ(t + 1) back to the clients to start a new training round:EQUATION(aggregation) θ(tEQUATIONWhile FL is an efficient process to safeguard privacy, the inherent lack of direct control over each individual client makes FL systems particularly vulnerable to various poisoning attacks [6-9, 21, 22, 14, 23-26] . These can be categorised into model and data poisoning attacks. Model poisoning involves altering gradients on compromised devices before transmission to the server [9, 14, 23, 24] , while data poisoning indirectly manipulates gradients by tampering with training datasets on malicious devices [25, 27, 21] .Counterfactual explanations. Counterfactual explanations [19] describe a model's decisionmaking process by identifying minimal and plausible changes to an observed input's features that lead to a desired model prediction. In explainable AI, finding counterfactual explanations is framed as an optimisation problem where the objective is to identify, for each sample x, the nearest data point x ′ such that the classifier f (θ, x ′ ) assigns a desired class label y ′ :EQUATIONAs a result, the variations in the input's features between x and x ′ offer actionable insights into how the model's decisions can be altered, highlighting the most important features.Semantic and behavioural spaces. Semantic spaces [16, 17] represent the semantics of a model by considering the error vector e = [ϵ(f (θ, x i ), y i )] i=1,...,n that a model produces on a set of n ∈ N samples, where ϵ could be the Mean Squared Error, for instance. Given a set of K models, the semantic space contains n-dimensional data points [e 1 , . . . , e K ] ∈ R K×n . Behavioural spaces [18] summarise semantic spaces into lower-dimensional spaces applying a transformation ψ n→m : R n → R m with m ≪ n (typically m = 2 for most applications) i.e., ψ n→m ([e 1 , . . . , e K ]). 
federated behavioural planes: 
 Problem definition: Given an FL system composed of a set of K clients with local data (x (k) , y (k) ) and models f (θ (k) ), we aim to analyse the evolution of the system to understand how clients impact the global model's predictive performance and decision-making over time. In Section 3.1, we introduce what drives our method and formalise the problem. In Section 3.2, we introduce Federated Behavioural Planes (FBPs), describing its components in more detail: the Error Behavioural Plane (Section 3.3) and the Counterfactual Behavioural Plane (Section 3.4). Finally, in Section 3.5, we introduce Federated Behavioural Shields, a new robust aggregation mechanism to enhance security in FL systems, showing a practical application of FBPs.FL is a dynamic process transitioning from a state where all participating clients behave randomly to a state where the behaviour of participating entities becomes coherent (the parameters of client models tend to converge to values which lead to high model accuracy). Thus, FL systems can be effectively analysed through the lens of dynamical systems using tools traditionally employed for such studies, such as differential equations. Defining b(θ (k) ) as the evolving state of the behaviour of the client k at time t, we introduce two primary forces that influence b(θ (k) ): g(θ (k) , x (k) , y (k) ), the local training dynamics, which drives the client towards its local optimum by leveraging information from the local dataset (x (k) , y (k) ); and b k∈K θ (k) -b(θ (k) ), a correction term that periodically aligns b(θ (k) ) with the aggregated state of all clients within the federated system. These dynamics can be encapsulated in the following differential equation 2 , which describes how client behaviours evolve during training and are influenced by internal forces within the FL system:db θ (k) dt = g θ (k) , x (k) , y (k) (1 -δ T ) + b k∈K θ (k) -b θ (k) • δ T (4)Here, δ T is characterised by a periodic Dirac delta function, defined as δ T = ∞ r=0 δ(t -r • T ), which triggers instantaneous adjustments at intervals determined by period T .Instead of finding a general analytical solution for Equation 4 (which is not trivial and requires strong assumptions on its components), we aim to empirically analyse the phase space of a FL system by considering different descriptors of client behaviours. More specifically, we focus on investigating (i) the predictive performance, evaluating how well the model is solving the task, and (ii) the decision-making process, as it contains information on how the model is solving the task. During each round, client behaviours are assessed through their respective models on the server, utilising a server-owned dataset reserved for this evaluation phase. This methodology aligns with existing protocols [9, 14, [28] [29] [30] [31] [32] . Each client behaviour is visualised through a two-dimensional plane: Error Behavioural Plane representing predictive performance and the Counterfactual Behavioural Plane to illustrate decision-making processes, collectively referred to as Federated Behavioural Planes. However, this framework offers a general approach to visualise and monitor different descriptors of the client behaviours simultaneously, which can be customised through specific functions, enabling the creation of additional planes.To comprehensively evaluate each model from the predictive performance point of view, we analyse the errors made by the model on all samples, rather than relying solely on a simpler aggregate metric, such as loss or error [14] . This approach enables a more detailed examination of the differences in the model's performance as observed by Mouret and Clune [18] . Following the methodology proposed by Zhang et al. [33] , we first construct a semantic error space for each model and then map it to a reduced space, called Error Behavioural Plane (EBP). Definition 3.1 (Error Behavioural Plane). Given a model f , parametrised with a set of weights θ (k) (t), related to the client k at round t, a dataset (x (server) , y (server) ), owned by the server, and a dimensionality reduction technique ψ n→2 , the representation e (k) (t) ∈ R 2 in the EBP of the client k is the following:EQUATIONIt is worth noting that two clients, despite having similar accuracy or loss, may receive significantly different representations in the EBP if they produce errors on distinct subsets of samples. In contrast, clients whose trajectories in the EBP converge over time form clusters representing clients whose predictive performance is similar on the same set of samples. However, clusters and trajectories in the EBP do not explain the decision-making process that leads to a prediction. This information could be used to further distinguish different types of clients and can be analysed using counterfactuals.The analysis of a model's decision-making process is the main research objective of explainable AI. Counterfactual explanations represent one of the most effective techniques as they give insights concerning the position and orientation of decision boundaries. Similar counterfactual explanations indicate models having similar decision boundaries, i.e. models taking decisions using a similar decision-making process. To this end, most differentiable counterfactual generators are trained to model the training data distribution [34, 35] , thus potentially providing insights on non identical distributions in clients' data. To produce these explanations, clients' predictive models should be concurrently trained with a counterfactual generator. Additional information on the optimisation objective are provided in Appendix A. EQUATIONEQUATIONCBP produces complementary information to EBP as clients which are similar in the CBP might be far away in the EBP (as discussed in Appendix B.4). Furthermore, as the purpose of CBP is to track clients' behaviour rather than explaining the model decision to a user, counterfactuals can be generated based on predictive models' embeddings, instead of input features, concealing sensitive information.FBPs provide descriptors of client behaviours during training, enabling various applications. Notably, trajectories in behavioural planes converging over time form clusters representing clients with similar predictive performance and decision-making process. This property can be used in practice to identify anomalies, such as malicious clients attempting to compromise FL training. In particular, leveraging this detailed information on client behaviours, we propose Federated Behavioural Shields (FBSs), a new class of robust aggregation strategies designed to enhance security in FL without requiring prior knowledge on the attack. This defensive mechanism generates a behavioural score in round t for a client k, denoted as s (k) (t), which is formulated through the composition of multiple scores s(k) j computed on S behavioural spaces, to guide the aggregation process in creating the next round's global model θ(t + 1), as outlined below:EQUATIONSpecifically, based on the FBPs we previously defined, we can compute these scores as follows:s (k) (t) = s (k) error (t)s (k) cf (t) i∈K s (i) error (t)s (i) cf (t) s (k) error (t) = 1 -min(||e (k) (t)||, 1) s (k) cf (t) = 1 1 K i∈K l (k) i (t)The error score s(k)error (t) measures the distance between the client k and the optimal point at the center of the plane, while the counterfactual score s (k) cf measure the average distance between the distribution of counterfactual generated by a client and all the other clients. In addition, considering that honest clients may occasionally deviate from the norm but generally contribute positively, we introduce a moving average mechanism to track client behaviours (see Appendix A.2 for details).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Justification: The paper does not provide theoretical results.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 The preliminary goal of our experiments is to assess whether counterfactual generators can provide insights in an FL context without compromising the performance of the predictor. We then visualise FBPs to verify that they can reveal information about the behaviour of various clients through the error and counterfatual behavioural planes (EBP and CBP). Lastly, we analyse the effectiveness of Federated Behavioural Shields as a robust aggregation mechanism, demonstrating the utility of the information provided by FBPs. Our experiments aim to answer the following questions:• Counterfactuals in FL: Does the integration of counterfactual generators impact clients' predictive performance? Do counterfactuals have the same quality in FL compared to centralised scenarios? Could counterfactual generators be adapted for each client? Answering these questions is an essential preliminary step to check whether counterfactuals can be used to generate FBPs.• Explaining FL training: Can trajectories in FBPs describe the evolving client behaviours during the training phase? Is it possible to visually identify clusters of clients using FBPs?• Leveraging FBPs information: Do FBPs provide sufficient detail to Federated Behavioural Shields to enhance the security of the FL training process against security attacks?This section describes essential information about the experiments. Further details on model configuration, training setup, and computational cost are presented in Appendices A.2, A.4, and A.6, respectively.In our experiments, we utilise four datasets: a Synthetic dataset (tabular) we designed to have full control on clients' data distributions, and thus test our assumptions; the Breast Cancer Wisconsin [36] (tabular); the Diabetes Health Indicator [37] (tabular); small-MNIST [38] (image); and small-CIFAR-10 [39] (image), reducing its size by 76% to increase task difficulty and highlight client differences in performance. For all the experiments with the small-MNIST dataset, our approach involves generating counterfactuals at a non-interpretable internal representation level of the model instead of the input space. To reflect the most realistic cross-silo scenario [40] , we distribute the training data among various clients such as data are not independent and identically distributed (IID) [41] . This represents the most challenging scenario to detect malicious clients due to the significant variations even among benign clients. Further details on the datasets and non-IID implementation are provided in Appendix A.1. Additional experiments analysing setup characteristics such as window length, local epochs, server validation set size, and differences between non-IID and IID scenarios can be found in Appendix B.Metrics. To determine the efficacy of counterfactuals generated through end-to-end training in FL, we measure several key metrics: task accuracy (↑ -higher is better); counterfactual validity (↑) [19] , which checks if the counterfactuals' labels align with user-provided labels; proximity (↓) [35] , assessing the realism of counterfactuals by their closeness to the training data (distance between the counterfactual and the closest data point in the training set with the same label); and sparsity (↓) [19] , which quantifies the changes made to the input to generate the counterfactuals (number of features changed between the initial sample and the counterfactual). The latter is quantified using Euclidean distance, as counting the number of changes provides less insight on the generated counterfactuals.To evaluate the effectiveness of client-specific adaptation, we analyse the relative change in client proximity between global and client-specific models, expressed as (P global -P local )/P global . Finally, we measure the task accuracy (↑) of the FL system under different attacks and defenses. All metrics are reported as the mean and standard error across five experimental runs with distinct parameter initialization.Baselines. In our experiments, we compare Federated Behavioural Shields with the following state-of-the-art robust aggregation methods: Median [11] , Trimmed-mean [11] , Krum [12] , and RFA [42] . Further information is provided in the Appendix A.7.Federated Attacks. We focus on attacks with realistic assumptions and where additional information outside the typical FL scenario are not available [10, 14, 23] . We test the following attacks: Label-flipping (Data Poisoning) [26, 43] , changes each sample's label to 1 -y (performed in the context of binary classification); Inverted-loss (Data Poisoning) [25] , creates an update that maximises the loss on the local dataset; Crafted-noise (Model Poisoning) [44] , adds noise N (0, β • σ(w t )) to the previous global model w t , where σ(w t ) is the standard deviation of w t and β is a scale factor set to 1.2; and Inverted-gradient (Model Poisoning) [24, 45] , inverts the gradient derived from the server's previous update, misaligning it with the true gradient. Further details are provided in Appendix A.8. 1 ). We compared model accuracy across four settings: two centralised learning (CL) and two FL, using three different datasets under non-IID conditions. For FL experiments, we used the traditional FedAvg approach with two variations: predictor-only and predictor with CF generator. The results indicate that our model, which involves the concurrent training of both the predictor and the counterfactual generator, achieves performance comparable to that of the predictor alone in FL. In the context of our model, both Local CL-where each client trains a model on its local data-and FL, implemented across all clients, comply with privacy standards [2] . For Local CL, we report the average accuracy of models trained independently by each client and evaluated on a common test set. However, only FL reaches the performance levels of the CL scenario, which assumes local access to all client data. This result indicates that incorporating counterfactuals during training does not compromise the predictor's performance, thus supporting their beneficial application without adverse effects on performance.Counterfactuals generated in FL have similar quality to those in CL (Table 1 ). Unlike Local CL, FL leverages information from all clients, thereby producing counterfactuals that more accurately reflect non-IID clients' data. As shown in Table 1 , FL achieves higher validity compared to the Local CL approach, indicating that the FL's counterfactuals better match ground-truth labels. Additionally, FL exhibits lower sparsity, which measures the number of modifications needed to achieve the counterfactuals. Table 1 shows these results using a counterfactual generator we adapted for this scenario (see Appendix A.3), however, similar conclusions are also obtained using out-of-the-box generators [34] (see Appendix B.2). Counterfactual generators can be adapted to specific clients (Figure 2 ). In our study, we explored the impact of client adaptation on counterfactual generation within FL environments, as detailed in Section 4.2. Client-personalisation is key whenever we need to extract client-specific information. This adaptation can be achieved by training (or fine-tuning) a counterfactual generator on local client's data (which naturally happens at each round). The effectiveness of client-specific adaptation can be measured by the relative change in client proximity between global and client-specific models, as shown in Figure 2 . The figure shows a marked reduction in relative proximity across the three datasets under non-IID conditions (up to 70% in the Synthetic dataset). This suggests that client-specific counterfactuals after adaptation are more representative of individual client datasets, providing unique descriptors of client-specific behaviours in the CBP. On the Breast Cancer EBP, Client 6 exhibits a random trajectory around the server trajectory, which indicates a Crafted-noise attack. In small-MNIST, both the CBP and EBP reveal that at each round the trajectory of the attacker consistently moves in the opposite direction to that of the server, which indicates an Inverted-gradient attack. These insights might be useful for FL users as they explain the nature of clients' behaviours, allowing the identification of as various types of attacks. Additional visualizations are available in the Appendix B.3.DatasetQuestion: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Section 3, Appendix A.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
explaining fl training: 
 FBPs allow the identification of clusters of clients (see Figure 3 ). In the Synthetic dataset, CBPs's clusters reflect client-specific data distributions (Client 1,2-3-4,5,6-7,8 and Attacker 9). Clients sharing similar data distributions tend to cluster closely in the CBP, which in the Synthetic dataset is structured with adjacent slices in the feature space (for example, Client 1 is positioned between Client 8 and Client 2). Similarly, in Breast Cancer and small-MNIST, two primary clusters are identifiable in the CBP: honest clients and attackers. This clustering capability is crucial for users aiming to comprehend client characteristics without direct access to data or models' parameters, thereby informing strategic decisions in training a federated model. For instance, one might consider reducing the number of clients with identical data distributions to avoid redundancy and enhance training efficiency. Conversely, identifying distinct clusters in FBPs can be useful to maximise model performance (at the expense of the generalisation ability) by forming a Clustered FL system where independent federated models are trained using a subset of similar clients.FBPs offer detailed insights into client behaviours, enabling the Federated Behavioural Shields to outperform existing state-of-the-art defense mechanisms (Figure 4 ). Our comparative analysis demonstrates that Federated Behavioural Shields generally outperform traditional methods such as Krum, Median, and Trimmed-mean across various datasets including Breast, Diabetes, and small-MNIST. The only exception was in the scenario of the Inverted gradient on the small-MNIST dataset, where Trimmed-mean performed better. Overall, our method enhanced the performance up to 10 percentage points (pp) over Median-the most robust aggregation baseline-when the FL system is under Label-flipping attack and up to 16 pp when the system is not under attack on small-MNIST. The proposed approach surpasses even a FedAvg aggregation in absence of attackers on the Breast Cancer and on the small-MNIST datasets, under both normal and crafted-noise conditions. These results suggest that an aggregation strategy based on predictive performance or decision-making similarities is superior to methods that solely consider sample count. Unlike other baselines, the independence of our method from prior knowledge about attackers establishes it as a robust aggregation tool in both adversarial and non-adversarial settings. Contrary to previous studies [14, 28] , we find that predictive performance alone does not reliably identify malicious clients in non-IID settings, as honest clients may also show consistently low performance, leading to potential misclassification in the aggregation process. Occasionally, this reliance on predictive performance can reduce our method's accuracy compared to scenarios where only counterfactual information is used. For instance, under no-attack conditions, counterfactual information alone often offers more effective behavioural descriptors than combining it with predictive metrics. Further detailed analyses in Appendix B.12 indicate that using both descriptors generally yields better results than relying solely on counterfactual behaviour. Furthermore, visualizations in Appendix B. 6 DiscussionQuestion: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We provide the code Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
related works: 
 Client behaviours in FL have been analysed using integrated visual tools [31, 46, 47] or indirectly through methods such as robust aggregation [10-14, 28, 30, 32, 40, 48-52] and clustering-based aggregation [53] [54] [55] [56] [57] [58] [59] . These studies typically focus on similarities in model or gradient parameters [10-13, 31, 40, 46-52, 55-59] , under the premise that distinct client data distributions manifest as unique model parameters. Nevertheless, these methods might overlook valuable information about how client models behave with the data, particularly in terms of predictive performance. To mitigate this issue, previous studies adopted evaluation methods that utilise a clean validation set on the server to delineate clients' behaviour. Common descriptors used in such cases are straightforward metrics such as accuracy, error rates, and loss [14, 31, 46, 47, 53, 54] . Despite their utility, these metrics might not be able to reveal behavioural patterns, such as those involved in the decision-making processes of models, which may suggest subtle similarities or manipulations. Notably, Wang et al. [46] introduced a post-hoc explainable approach, Grad-CAM [60] , to explain client model behaviours during training. However, Grad-CAM is limited to CNN models and provides primarily qualitative visual insights, which cannot be easily automated and thus still require human intervention. Similarly, SentiNet [32] uses Grad-CAM to detect potentially malicious regions in input images, yet client behaviour assessments continue to rely primarily on evaluating predictive performance on manipulated and unmanipulated images. To the best of our knowledge, this work represents the first systematic attempt to formalise the evolving dynamics of clients in FL, showing how behavioural client trajectories affect the predictive performance and decision-making processes of the global model.The primary constraint of our framework lies in its assumption that the server possesses a minimal validation set for querying client models. While this assumption is common across various methods, it can be mitigated, as demonstrated in Wang et al. [46] , by generating synthetic data points. To this end, our approach might already integrate a potential mechanism as differentiable counterfactual methods can be used to generate synthetic data [61] . Furthermore, as indicated by the promising results in Appendix B.6, utilizing validation-independent descriptors, such as counterfactuals, renders our defense method robust against biased or unfair validation sets. Given the significance of ensuring fairness across clients, developing additional validation-independent descriptors represents a promising direction for future research. Another consideration is the computational overhead introduced by counterfactual generators, which, although minimal compared to other baselines (see Appendix A.6), is higher than that of traditional FedAvg. However, this overhead can be mitigated by using a smaller network for the counterfactual generator, thereby reducing the number of neurons (e.g., 1.8% of predictor parameters) without compromising accuracy.Future work could leverage the extensive information provided by FBPs to explore additional strategies for optimizing the learning process, such as the development of Clustered FL among clusters of clients and the fine-grained categorization of attack types. Incorporating additional behavioural planes may also enhance the specificity of the FBPs explanations. Lastly, since our method allows for the integration of privacy-enhancing techniques such as Local Differential Privacy [62] and Homomorphic Encryption [63] , future studies could analyze their impact on the overall performance and computational efficiency of our system.In this work, we proposed Federated Behavioural Planes, a method to explain the dynamics of FL systems and client behaviours. This innovative method allows to visualise, track, and analyse client behaviours based on specific characteristics. Our focus was twofold: examining predictive performance by analysing prediction errors and investigating the decision-making process through counterfactual generation. The results of our experiments showed that Federated Behavioural Planes enable to track client behaviours over time, cluster similar clients, and identify clients' contributions to the global model with respect to a specific descriptor. Based on Federated Behavioural Planes' information, we introduced a novel robust aggregation mechanism that improves existing state-ofthe-art methods by not requiring prior knowledge of the attacker. This work lays the foundation to explain the evolution of client behaviours, with the potential to enhance reliability and control over FL systems.In our experiments, we employ five distinct datasets:• Synthetic dataset (Tabular): It consists of two features randomly extracted from a range of -5 to +5. As shown in Figure 6 .a, we partition this feature space into K slices, where K equals the number of clients. Each slice corresponds to a specific client. We assign a label of 1 to all data points where x 1 > αx 2 , effectively creating a linear decision boundary that varies with α. For each client, we generally draw 1000 samples. This controlled setup allows us to precisely manipulate and visually understand the data distribution across different clients, as illustrated in Figure 6a .• Breast Cancer Wisconsin (Tabular): It contains data from 569 patients with 30 continuous variables derived from digitised images of a fine needle aspirate of a breast mass. The variables describe characteristics of the cell nuclei present in the image (e.g., radius, area, perimeter), aimed at predicting breast cancer [36] .• Diabetes Health Indicator (Tabular): It comprises data from 70,692 patients, encompassing questionnaire-based variables (e.g., smoking, physical activity, fruit intake) and medical measurements (e.g., BMI, cholesterol levels), with a total of 21 features, to predict the presence of diabetes [37] .• MNIST (Image): It is a comprehensive database of handwritten digits frequently used to benchmark image classification algorithms [38] . Each image is a 28x28 pixel grayscale representation of a digit, ranging from 0 to 9. To increase task difficulty and highlight client differences in performance, our experiments utilise only 10,000 of the available 70,000 images, referred to as small-MNIST. Additionally, we transform these images into color, randomly assigning colors with equal probabilities: red, green, or blue. To explore the decision-making process through counterfactuals, we employed a ResNet-18 architecture to extract 1,000 features from each image.• CIFAR-10 (Image): The CIFAR-10 dataset comprises 60,000 color images categorised into 10 classes, widely used for evaluating object recognition algorithms [39] . Each image has a resolution of 32x32 pixels and is represented in RGB format. Similar to our small-MNIST setup, we reduce the sample size to 10,000 images. Features are extracted using a ResNet-18 architecture, resulting in 1,000-dimensional feature vectors for each image.IID distributions across clients are achieved by randomly selecting samples from each dataset. In contrast, non-IID distributions are created using KMeans clustering [64] to form K clusters of samples within each class-10 for the Synthetic dataset and 5 for others. We then assign the samples of the two nearest clusters from different classes to each client. To ensure each client's distribution is equally represented, we randomly extract 15% of the samples from each client and combine them into the test set to assess the performance of our experiments. We create a clean validation set on the server by extracting 89 samples for Breast Cancer and 250 samples for other datasets. Additionally, we partition each client's dataset locally, allocating 80% for training and 20% for validation.In our work, the model is composed of a predictor, any Deep Neural Network, which in our is a multi-layer perceptron that takes tabular data as input and makes a prediction. It is composed of five layers with hidden dimension equal to 512, 256, 256, 64, respectively. In addition, in our work, the model is also composed of a counterfactual generator, which can be implemented in different ways. The only requirement is that it can be trained concurrently with the main model, in an end-to-end fashion. To generalise our results, we tested the model with two different counterfactual generators.In Section 5, we used a counterfactual generator designed by us, adapting the intuition from Dominici et al. [65] , that explicitly optimises the generation of counterfactual with respect to specific labels using two VAEs, named CFGen and described in Appendix A.3. On the other hand, in Appendix B.2 we also tested VCNet [34] , which could generate counterfactual training a VAE, implementing it according to the original paper. In the context of creating Federated Behavioural Planes, PCA is employed as the dimensionality reduction technique ψ n→2 within the EBP, centering the reduced space around a zero vector, which signifies the absence of errors. For CBP, tSNE is utilised as the dimensionality reduction method ψ K→2 , ensuring the preservation of distances between clients and their respective clusters. Finally, in terms of the implementation of FBSs, at each round t, the score s (k) (t) for each client k is computed using a moving average with a window length of L, as follow:EQUATIONAdditionally, client exclusion (s (k) (t) = 0) is automatically triggered if a client exhibits a score approaching zero (i.e., < 10 -7 ) within the specified window.CFGen is designed to adapt the approach proposed by Dominici et al. [65] to tabular data, eliminating the need for predefined concepts. It is a latent variable model that generates counterfactuals through variational inference. To this end, we have two random variables z and z ′ . These variables represent latent factors of variation whose probability distributions are easier to model and sample compared to those for x and x ′ . We also include dependencies from y to the counterfactual latent distribution z ′ in order to explicitly model the dependency of z ′ on the class labels, resulting in the following overall probabilistic graphical model:x x y x ′ y ′ z z ′(10) This way, the generative distribution factorises as:EQUATIONIn our approach, p(y|x) and p(y ′ |x ′ ) are the task predictor; p(x|z) and p(x ′ |z ′ ) are the same decoder. In practice, we assume that the input x is always observed at test time, making the term and p(x|z) irrelevant. Finally, p(z) is a standard normal prior distribution and p(z ′ |z, x, y) is a learnable normal prior whose mean and variance are parametrised by a pair of neural networks ϕ pµ and ϕ pσ . Amortised inference. CFGen amortise inference needed for training by introducing two approximate Gaussian posteriors q(z|x) and q(z ′ |z, x, y, y ′ ) whose mean and variance are parametrised by a pair of neural networks (ϕ µ , ϕ σ ) ((ϕ µ ′ , ϕ σ ′ ), respectively).Optimization problem. In order to obtain these counterfactuals, it is important to optimise their generation during training. CFGen is trained to maximise the log-likelihood of tuples (x, y, y ′ ), while observing x. Following a variational inference approach, we optimise the evidence lower bound of the log-likelihood, which results in the following objective function to maximise:EQUATIONwhere D KL is the Kullback-Leibler divergence and α = (z, x, y, y ′ ). Moreover, in order to enforce the counterfactuals to be as close as possible to the initial input, we add an additional term to the objective:EQUATIONGradient Descent was employed as the optimisation algorithm, with a batch size equivalent to the dimension of the training dataset. Both the momentum and learning rate were set at 0.9 and 0.01, respectively. For centralised training scenarios, the model was trained over 1,000 epochs. The Flower library was utilised to implement FL [66] . In all federated experiments, except for those evaluating local epochs, we employed 2 local epochs. During the assessment of various defense mechanisms, the number of communication rounds was capped at 200 and the window length for the moving average of 30 rounds. For comparisons with centralised training, 1,000 rounds were used. For client personalization, the generator was trained across 25 local epochs. In each experiments, performance metrics were evaluated on the model that exhibited the lowest aggregated loss during training. The aggregated loss represents the weighted average of client losses, evaluated on each client's local validation set, proportional to the respective number of samples.A.5 Code, licenses and hardwareFor our experiments, we implement all baselines and methods in Python 3. MNIST (GNU license), and CIFAR-10. Our code, along with all the necessary details to reproduce the experiments, is publicly available on GitHub3 under the MIT license. Additionally, we provide pseudo-code for both client-side (Algorithm 2) and server-side (Algorithm 1) implementations of our proposed approach, which includes creating behavioural planes on the server and applying our FBSs. All experiments were conducted on a workstation equipped with an NVIDIA RTX A6000 GPU, two AMD EPYC 7513 32-Core processors, and 512 GB of RAM.Require: Initial model f (θ(0)), number of communication rounds T , number of selected clients per round M , clean validation set (x (server) , y (server) ), set of plane functions S 1: for t = 0, 1, . . . , T -1 doSample M out of N clients for each selected client k in M in parallel do 6:x ′(k) , y (k) = f (θ (k) (t + 1), x (server) ) s k j (t) = plane_f n(x ′(k) , y (k) , x ′(k) , y (server) ) ▷ Our case: Eq.5 and Eq.6-7 This section outlines the computational costs associated with our proposed method, focusing on three primary components: local computation, communication overhead, and server-side computation.Local computation. Our methods (both FBPs and FBSs) integrate a counterfactual generator with the original predictor to analyse decision-making and provide insights into the client's data distribution. For small neural networks, local computation is minimal compared to other costs in the FL framework, such as communication latency and synchronization. In these cases, the counterfactual generator can produce counterfactuals for the model input without impacting training efficiency. As the predictor size increases, as shown with MNIST and CIFAR-10, we can efficiently generate counterfactuals for intermediate layers using a relatively small number of neurons. For instance, in our experiments with small-MNIST and small-CIFAR-10, we used a ResNet-18, which has 12.42M parameters and requires 71.1M GFLOPs for inference with an input RGB image of size 28x28. As shown in Table 2 , the generator with embedding size of 128 contributes only 5.1% of the operations compared to the predictor alone. Furthermore, by reducing the embedding size to 32, we maintained performance while reducing GFLOPs to just 2.7% of the ResNet-18.Communication overhead. Similar to the predictor, the counterfactual generator must be transmitted to the server for evaluation and aggregation, thereby increasing the number of model parameters sent. However, the additional communication overhead is marginal compared to the size of the predictor. As illustrated in Table 2 , the counterfactual generator with an embedding size of 32, which maintains high task performance, consists of only 1.8% of the parameters of the ResNet-18. Consequently, with 32-bit precision, our counterfactual requires 0.92 megabytes (MB) out of approximately 49.68 MB for the predictor.Server-side computation. On the server, our methods involve evaluating client models' performances on a clean validation set and calculating the pair-wise distances of the generated counterfactuals between clients. As shown in Figure 13 , the validation set size can be relatively small, e.g., 250 samples, which can be processed with a single forward pass of the model. Consequently, the models' evaluation on the server, which can also be parallelised, is negligible compared to the computational load of calculating pair-wise distances between client counterfactuals. The primary computational bottleneck on the server is the pair-wise Wasserstein distance between client counterfactuals. To address this, we use the sliced Wasserstein distance implementation, which has a computational complexity of O(m log m), where m is the number of supports, given by m = n_samples × reduction dimension (ψ K→2 ). In our implementation, m = 250 × 2 = 500, and this operation is repeated for each unique pair of clients (i.e., binomial coefficient n 2 ), leading to a computational complexity of O(n 2 • m log m) where n is the number of selected clients.Compared to Krum, which has a complexity of O(n 2 d) with d being the number of model parameters, our pair-wise operation is more efficient for neural networks with more than approximately 4480 parameters, which is likely the case in most practical applications. Overall computational cost. We evaluated the overall computational cost of our defense mechanism in comparison to both the Krum algorithm and the traditional FedAvg algorithm, considering different network sizes and varying numbers of clients. In all experiments, a validation set of 250 samples was used. For tests involving an increasing number of clients, our method incorporated a counterfactual generator utilizing 7.6% of the predictor's parameters (in the worst-case scenario), as described in the original paper. Computational time was measured over 10 training rounds across 3 folds, and we reported the mean and standard error of the time per round. As shown in Figure 5 , our method's computational cost scales more efficiently with both the number of model parameters and the number of clients when compared to Krum. Specifically, our method adds only an additional minute per round for 200 clients compared to FedAvg, while Krum introduces over 15 minutes per round. Importantly, our approach maintains the robustness and security benefits, demonstrating its efficiency and scalability in large-scale FL scenarios.This section details the robust aggregation methods used as baselines in our experiments. These methods are designed to mitigate the effects of poisoning attacks by malicious clients and include:• Median: This approach performs aggregation using the median of the client updates [11] .Compared to the mean, the median is less affected by outliers. Theoretical guarantees on the robustness of Median aggregation are provided in [11] , while empirical evidence demonstrates that it exhibits better robustness than the more sophisticated Krum [14] .• Trimmed-mean: This approach [11] aggregates each dimension of input gradients separately. For each dimension i, it sorts the values of the i th dimension of all gradients, removes the β t largest and smallest values, and averages the remaining values to obtain the aggregate for dimension i. We use a default β t equal to 20% of the number of clients.• Krum: Krum [12] is based on the intuition that malicious gradients must be far from benign gradients to poison the global model. Assuming knowledge of the upper bound on the number of malicious clients m, Krum selects the gradient from the set of K input gradients that is closest to its K -m -2 nearest neighbors in terms of the squared Euclidean norm.• Robust Federated Aggregation (RFA): RFA [42] replaces the standard arithmetic mean aggregation with the geometric median to ensure robustness against poisoned updates. It employs a Weiszfeld-type algorithm to compute the geometric median in a privacypreserving and communication-efficient manner.This section outlines and provides additional details about the federated attack scenarios implemented into our experiments, focusing on both model and data poisoning without prior knowledge of the server aggregation methods. We implemented these attacks assuming 20% of the clients are malicious:• Label-flipping (Data Poisoning): In a multiclass scenario, this attack changes all samples in the dataset with a source class c src into a target class c target [26, 43] . For binary classification, label flipping is implemented as 1 -c source , effectively inverting the class labels. For multiclass classification, we perform targeted label flipping by changing all instances of class 2 (Bird) to class 8 (Ship).• Inverted-loss (Data Poisoning): This attack aims to create an update that maximises the loss and, consequently, causes a significant drop in accuracy [25] . The implemented loss function follows the algorithm outlined in [25] .• Crafted-noise (Model Poisoning): Inspired by the Free-riding attack described in [44] , this attack aims for stealthiness by adding noise to the previous model received from the server, w t . The noise is added as follows:w * = w t + N (0, β • σ(w t )), where σ(w t ) represents the standard deviation of w t and β is a scale factor. We adopted β equal to 1.2, except when we tested the sensitivity values: 0.3, 0.5, 0.8, 1.2, and 1.6.• Inverted-gradient (Model Poisoning): This attack modifies the inner product between the true gradient and the malicious gradient updates sent by the attacker, affecting the alignment with the true gradient [24, 45] . In our implementation, the malicious gradient is inverted:-∇w. Given the lack of information about honest clients, the true gradient is taken from the previous update sent by the server, representing the previous improvement of the global model.For both data poisoning attacks, the attacker uses a local dataset drawn from the same distribution as other clients and with an average number of samples.As depicted in Figure 2 , there is a significant reduction in the relative proximity measure between global and client-specific models across all three datasets on average, indicating a high degree of customization in the client models. Nonetheless, a few clients are still unable to tailor the model effectively to their specific distribution. To explore this phenomenon, we visually represent the generated counterfactuals pre-adaptation in Figure 6 .a, showcasing models with the highest and lowest degrees of client-specific adaptation in Figures 6.c and 6.d, respectively. Notably, the most adapted model corresponds to client 4, whose data is almost perpendicular to the class boundary. In contrast, client 8, whose data distribution lies close to the boundary, struggles to adapt, remaining akin to the pre-adaptation conditions. This could highlight an increased capacity in personalisation in clients whose data is perpendicular to the learnt decision boundary, also indicating that personalization varies among clients, thereby offering more insights on the behaviour of the clients. Furthermore, we analyse the effects of client-specific adaptation under both IID and non-IID conditions. As shown in Figure 7 , in contrast to non-IID, under IID conditions where data distributions are uniform between clients, the enhancements were minimal, indicating an alignment between global and local optima. The lack of customization implies that all client models share a similar decision-making processes, correctly reflecting with their IID condition. However, this setting also facilitates the identification of outliers or anomalous clients, which exhibit a higher degree of customization.To explore the independence of FBPs from the counterfactual generator, we evaluate our methodology with an alternative model named VCNet [34] , maintaining identical experimental conditions. As illustrated in Table 3 , VCNet achieves comparable performance in both Federated Learning (FL) and Centralised Learning (CL) configurations, where data from all clients is placed on a single machine. Similarly to our counterfactual generator, the results demonstrate that the privacy-preserving training Operating under similar privacy constraints, FL enables VCNet to learn from the diverse distributions of client data, thereby surpassing the performance of the Local CL approach, where each client independently trains their own model.To assess VCNet's capacity for adapting the counterfactual distribution to individual clients, we conduct client-specific adaptations starting from the global model achieved through FL. Figure 8 illustrates the relative variation in client-proximity across three datasets: Diabetes, Breast Cancer, and Synthetic. The observed significant reduction in proximity indicates that VCNet can be effectively personalised to each client's distribution, thereby confirming its utility in accurately reflecting client behaviours during the training process. Figure 9 illustrates client trajectories within the FBPs for different scenarios from those presented in Section 5. Specifically, the left side of the figure displays the FBPs for the Synthetic dataset, now including an attacker employing an Inverted Loss attack, while the right side shows the FBPs for the Diabetes dataset under a Data Flip attack scenario. In both cases, the largest clients are subdivided into smaller clients (three and two), forming distinct clusters of clients (2,3,4 and 6,7 for Synthetic; 3,4,5 and 7,8 for Diabetes). These clusters are particularly evident on the CBP, and even within the Synthetic dataset's plane, one can observe clients with similar data distributions (e.g., Client 8's data distribution lies between that of Clients 7 and 1). This observation holds across all clients in the Synthetic dataset, underscoring the ability to highlight information about data distribution similarities among clients without direct access to their data. Moreover, Client 9 (the attacker) is noticeably diverging from the others across all four planes. It is crucial to note the distinct behavioural patterns of the attacker depending on the attack type. In the Inverted Loss scenario, Client 9 moves in the exact opposite direction to the others, converging at the same point in the EBP. Conversely, in the Diabetes dataset with a Data Flip attack, Client 9 simply diverges from the others, each moving towards different minima. This variation highlights the potential to identify the type of attack deployed by the malicious client based on these behavioural information. As previously observed in Figure 6 , EBP and CBP provide orthogonal descriptions of client behaviours. In the Synthetic dataset, CBP effectively identifies clusters of clients with analogous data distributions. In contrast, EBP shows all honest clients moving in the same direction, thereby obscuring individual cluster distinctions but effectively contrasting the movements of honest clients with that of the attacker. Similarly, in the Breast Cancer dataset, the identification of an attack (i.e., Crafted-noise), is more discernible through EBP, which elucidates the direction of the malicious client relative to the server. These findings highlight the distinct yet complementary roles that EBP and CBP play in the analysis of client behaviour across different datasets.To illustrate the complexity of model behaviour and the utility of EBP and CBP, consider the behaviour of three different models with respect to the same input changes. For instance, a data point with features [1, 2] is misclassified as class 1 by Model A and correctly classified as class 0 by Model B. However, both models would change their predictions to class 1 if the features were altered to [2, 3] . In a similar fashion to Model A, a third model, Model C, may also initially misclassify the point as class 1 but would require a change in features to [2, -1] to alter its prediction to class 0, highlighting different sensitivities and prediction dynamics in response to input variations.FBPs provide a detailed representation of each client's behaviour, enabling a deeper understanding of the conditions under which malicious clients deviate from the expected behaviour of benign clients. However, in large-scale FL systems, visual inspection may not always be necessary. Instead, statistical metrics derived from these planes can be automatically extracted and analyzed. This is demonstrated in our FBSs, which extract client-specific scores to mitigate or exclude the contributions of malicious clients during aggregation. To demonstrate the scalability and efficacy of automatic detection, we evaluated our approach using the small-CIFAR-10 dataset and recorded client scores throughout the training process using a 5-fold cross-validation scheme. As shown in Figure 10 , the extracted scores consistently identify and diminish the influence of malicious clients on the global model, as indicated by the 95% confidence interval, which varies based on the type of attack. Notably, as the model converges (with updates between rounds close to zero), more weight is given to the attacker with an inverted gradient, since its model closely resembles the global model from the previous round (plus a negligible inverted update).Our proposed methods rely on a clean validation set on the server to characterize client behaviours. However, in real-world FL scenarios, obtaining a validation set that is entirely fair and representative for each client may not be feasible. This challenge underscores the rationale behind incorporating a multi-plane evaluation in our approach, which assesses client behaviour through two distinct criteria: task performance and counterfactual analysis. Traditional performance metrics in machine learning, such as accuracy, loss, or error, are heavily dependent on the validation set used (i.e., error plane). In contrast, information derived from counterfactuals is more indicative of the learned decision process and provides insight into the training data distribution of each client, even when the validation set is biased (i.e., counterfactual plane). Consequently, an attacker or anomalous client will exhibit distinct behaviours across both planes, particularly showing an unrelated counterfactual distribution compared to other clients. Conversely, an underrepresented client will produce plausible counterfactuals similar to those of other clients, and will only be affected by the unfair validation set in the error plane. To validate the robustness of our defense mechanism under an unfair validation set, we conducted experiments in which data from one client was excluded from the validation set. We then analyzed the behavioural scores assigned by our method. Preliminary results, shown in Figure 11 , indicate that the 95% confidence interval of the scores for the underrepresented client consistently overlaps with those of other honest clients, thereby distinguishing them clearly from malicious clients.In our main study, we focus on the non-IID setting, which is the most prevalent scenario for cross-silo FL in practice [40] . This setting presents considerable challenges due to the substantial differences between updates from honest clients, which can obscure the divergent behaviours of malicious clients [41] . Consequently, we conducted experiments on the Breast Cancer dataset in both IID and non-IID settings using our FBPs for comparative analysis. As depicted in Figure 12 , we initially visualised client trajectories on both behavioural planes and within the counterfactual distance space l (defined in Equation 7). The trajectories illustrate the uniform nature of honest client behaviours on both planes, distinctly highlighting the divergent behaviour of the malicious client. Notably, the visualization of the counterfactual distance space, employed by our FBSs, effectively identifies the malicious client. It indicates a high distance of the attacker from all other clients, leading to a low score during the aggregation process. To further validate our approach, we compared the performance of our FBSs under No-attack, Crafted-noise, Inverted-gradient, Label-flipping, and Inverted-loss attacks in both IID and non-IID settings. Table 4 demonstrates that higher accuracy is achieved in the IID setting under almost all conditions compared to the non-IID setting, underscoring the increased complexity of operating in non-IID environments. Considering the crucial role of the validation set size on the server in the creation of behavioural planes, we initially examined its impact on computational time and accuracy through 10 distinct trials. Specifically, we evaluated our FBSs (without moving average) against a malicious client executing an inverted gradient attack on the Diabetes dataset. As depicted in Figure 13 , computational time increases exponentially with the size of the dataset. However, a dataset containing as few as 250 samples is sufficient to adequately represent the data distribution, achieving performance on par with that observed in larger datasets. Additionally, with fewer than 1000 samples, our method demonstrates greater efficiency compared to the Krum algorithm. B.9 Impact of the window length in the moving averageWe analyse the impact of window length on our FBSs and its two variations, one using only predictive performance (error) and the other focusing solely on decision-making processes (counterfactual). The results presented in Figure 14 depict the average accuracy across various conditions-including No attack, Label-flipping, Inverted-loss, Crafted-noise, and Inverted-gradient-for window lengths ranging from 3 to 30 rounds. While no definitive patterns are clearly evident, on average, a longer window length tends to be advantageous for accurately assessing the behavioural scores of each client during the aggregation process. Noteworthy, the optimal accuracy across both datasets was achieved with a window length of 30 rounds. Particularly, a consistent and slight increase in accuracy is observed in the Diabetes dataset when utilising only the counterfactual information.Local training epochs play a crucial role in FL, particularly in evaluating client behaviours. An increased number of local epochs implies a greater adaptation of the client-model to its local data distribution. For this reason, we assess the performance of our FBSs against data poisoning attacks, including Label-flipping and Inverted-loss, across Breast Cancer, Diabetes, and Synthetic datasets. Metrics are reported as the mean and standard error across ten with distinct parameters' initialization. These attack types were selected because malicious updates are directly influenced by the number of local epochs, similar to updates from honest clients; differently, the behaviour of malicious clients in model poisoning might remain unaffected. As depicted in Figure 15 , the impact of local epochs varies across datasets, with each achieving maximum accuracy at different numbers of epochs. Generally, accuracy increases with the number of local epochs until it reaches a peak, after which it begins to Window Length (L) decline. This trend may be attributed to the fact that initially, increasing the number of local epochs enhances the detectability of anomalies in the behaviour of malicious clients, as their models diverge more quickly from those of honest clients. However, beyond a certain threshold, all client models become highly customd, leading to the potential underweighting of even honest clients' contributions during the aggregation process on the server. This results in the system relying predominantly on a few clients whose models are the most similar to each other.We conducted experiments to evaluate the sensitivity of the proposed method to varying levels of attack intensity. Specifically, we focused on model poisoning attacks by systematically increasing the noise parameter (β) injected by adversaries into the global model prior to transmission to the server (noise = N (0, β • σ(w t ))). The evaluation was performed using 5-fold cross-validation on the Breast Cancer and small-MNIST datasets. As shown in Figure 16 , increasing the attack intensity results in a noticeable decrease in the performance of the global model when using the standard FedAvg approach. In contrast, our method-both when utilizing all feature planes and when restricted to the counterfactual plane-remains stable and unaffected by the attack intensity. Interestingly, a marginal improvement in accuracy is observed as the attack intensity increases, likely because more aggressive modifications render malicious models more degraded and, therefore, easier to detect.We conducted an ablation study to examine the impact of various components on the efficacy of our algorithm. These components include predictive performance (error), decision-making process (counterfactuals), and the application of a moving average. Figure 16 : Impact of attack intensity on FBS and traditional FedAvg in (a) the Breast Cancer dataset and (b) the small-MNIST dataset. Notably, unlike FedAvg, FBS maintains stable accuracy as the attack intensity increases.Label-flipping, and Inverted-loss, comparing different variations of our method against baselines on the Breast Cancer and Diabetes datasets.Notably, relying solely on predictive performance does not yield a statistically significant benefit compared to the strongest baselines, such as RFA on Breast Cancer and Krum on Diabetes. In contrast, counterfactual information provides deep insights into client behaviour, enabling our method to outperform all baselines in identifying malicious clients. Although the improvement is subtle, the combination of descriptors, on average, provides better results than using counterfactuals alone.The integration of a moving average notably enhances the performance of our method, improving from 94. In this section, we provide a detailed numerical breakdown of the results depicted in Figure 4 . This quantitative analysis aims to supplement the visual data presented, offering precise values and statistical insights that underpin the observations and conclusions discussed throughout the paper. Tables 6 for Breast Cancer, 7 for Diabetes, 8 for small-MNIST, and 9 for small-CIFAR-10 present a comprehensive comparison of our FBSs and its streamlined versions using only the CBP or the EBP. These are evaluated against traditional defenses such as Krum, Median, Trimmed-mean, RFA across five scenarios: No attack, Crafted-noise, Inverted-gradient, Label-flipping, and Inverted-loss. This comparison elucidates the effectiveness of our approach under a range of conditions, both adversarial and non-adversarial. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimiser, etc.) necessary to understand the results?Answer: [Yes] Justification: Appendix A Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Appendix A.5Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 9. Code Of Ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We have thoroughly reviewed and adhered to the NeurIPS Code of Ethics Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Section 5 and Section 6 Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimising neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper does not address safeguards for the responsible release of high-risk data or models because our research does not involve such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: see Appendix A.4, A.5.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: Appendix A.4, A.5 and code Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] For simplicity, we omit the dependency on the time variable t in our notation for variables b and θ.https://github.com/dariofenoglio98/CF_FL 
 
 Appendix:  


Decision: The paper introduces a method to track FL clients' dynamic behavior.  There is good agreement among the reviewers on the major strengths of the paper and its novelty.

Reviews: review: paper_summary: paper_summary: This paper introduces a novel method called Federated Behavioural Planes (FBPs) to analyze, visualize, and explain the dynamics of client behavior in Federated Learning (FL) systems. The primary contributions of the paper are as follows:

* 1\. Introduction of Federated Behavioural Planes (FBPs): FBPs consist of two planes:
     * 1.1\. Error Behavioural Plane (EBP): This plane analyzes the predictive performance of client models by visualizing the errors they produce.

     * 1.2\. Counterfactual Behavioural Plane (CBP): This plane examines the decision-making processes of client models through counterfactual explanations, highlighting how decision boundaries are formed.
* 2\. Visualization and Analysis: FBPs provide informative trajectories that describe the evolving states of clients, enabling the identification of clusters of clients with similar behaviors. This helps in understanding both beneficial and detrimental behaviors in FL systems.

* 3\. Federated Behavioural Shields (FBS): Based on the patterns identified by FBPs, the authors propose a robust aggregation technique named Federated Behavioural Shields. This technique enhances security by detecting malicious or noisy client models and surpasses the efficacy of existing state-of-the-art FL defense mechanisms.

* 4\. Experimental Validation: The paper demonstrates through experiments that FBPs can effectively track client behavior, identify client clusters, and improve the security and performance of FL systems. The proposed FBS method outperforms other robust aggregation methods in defending against various types of attacks.

Overall, the paper offers a comprehensive approach to enhance understanding, trust, and control over federated learning systems by introducing a novel method to analyze and secure client behaviors.
strength_weakness: strength_weakness: **Complexity of Methods:**
 - **Computational Overhead**: The concurrent training of counterfactual generators with the main predictive models introduces significant computational overhead. This could be particularly burdensome in real-world federated learning settings where resources are limited. The paper could benefit from a more detailed analysis of the computational costs and potential optimization strategies to mitigate this overhead.
    - **Actionable Insight**: Consider providing a detailed comparison of the computational requirements of the proposed method with baseline methods. Explore possible optimizations or approximations that could reduce the overhead without significantly compromising the performance.

**Real-World Applicability:**
- **Scalability Concerns**: While the experiments are comprehensive, they are conducted on relatively small datasets and a limited number of clients. This raises concerns about the scalability of the proposed methods to larger, real-world federated learning scenarios with many clients and more complex data distributions.
    - **Actionable Insight**: Include a discussion on the scalability of FBPs and FBS. Consider performing a scalability analysis, even if only theoretical, to predict the performance and feasibility of the methods in larger settings. Additionally, simulations or theoretical models could provide insights into expected behavior in large-scale deployments.

**Generalization Across Different Models:**
- **Model-Specific Limitations**: The proposed method may be tailored to specific types of models (e.g., neural networks) and may not generalize well to other types of models used in federated learning (e.g., decision trees, support vector machines).
    - **Actionable Insight**: Discuss the applicability of FBPs and FBS to different types of models. Providing a broader range of experiments that include different model architectures could strengthen the paper. If certain models are not compatible, explain the limitations and potential modifications required for broader applicability.

**Evaluation Metrics:**
- **Limited Evaluation Metrics**: The evaluation primarily focuses on standard metrics like accuracy and robustness against attacks. While these are important, they might not capture all aspects of the system's performance, such as the impact on communication efficiency, latency, and energy consumption.
    - **Actionable Insight**: Introduce additional evaluation metrics that capture the holistic performance of the system, including communication overhead, latency, and energy consumption. This would provide a more comprehensive assessment of the practicality of the proposed methods in real-world applications.

**Interpretability and Usability:**
- **Interpretability for Non-Experts:** The paper, while clear in its technical explanations, may still be challenging for practitioners who are not experts in federated learning or explainable AI. Enhancing the interpretability and usability of the methods for a broader audience could be beneficial.
    - **Actionable Insight:** Provide more intuitive explanations and visualizations of the key concepts and methods. Including case studies or practical examples demonstrating the application of FBPs and FBS in real-world scenarios could make the methods more accessible and easier to understand for non-experts.

**Real-World Validation:**
- **Lack of Real-World Validation**: The experiments are conducted in controlled settings, which may not fully represent the challenges and variability encountered in real-world federated learning deployments.

    - **Actionable Insight**: Discuss potential real-world applications and the expected challenges. If possible, provide preliminary results or insights from deploying the methods in a real-world scenario. Alternatively, outline a detailed plan for future real-world validation studies.

**Conclusion:**
While the paper presents significant contributions to federated learning, addressing these weaknesses could enhance its impact and practicality. By focusing on computational efficiency, scalability, broader model applicability, comprehensive evaluation metrics, interpretability, and real-world validation, the work can move closer to its stated goals and become more robust and applicable to a wider range of scenarios.
questions: questions: **Questions for the Authors:**
* 1\. **Computational Overhead:**
    - Question: How significant is the computational overhead introduced by the concurrent training of counterfactual generators with the main predictive models?
    - Suggestion: Provide a detailed comparison of the computational requirements of your method with baseline methods. Include any potential optimization strategies to mitigate this overhead.

* 2\. **Scalability:**
    - Question: How does the proposed method scale with a larger number of clients and more complex datasets?
    - Suggestion: Include a scalability analysis or discussion that addresses the performance and feasibility of FBPs and FBS in larger, real-world federated learning scenarios. Simulations or theoretical models predicting the system's behavior in large-scale deployments would be helpful.

* 3\. **Applicability to Different Models:**
    - Question: Is the proposed method applicable to different types of models beyond neural networks, such as decision trees or support vector machines?
    - Suggestion: Discuss the generalizability of FBPs and FBS to various model types. If there are limitations, explain the necessary modifications to apply the method to other model architectures.

* 4\. **Evaluation Metrics:**

    - Question: Have you considered additional evaluation metrics that capture communication efficiency, latency, and energy consumption in your analysis?
    - Suggestion: Introduce and discuss additional metrics to provide a comprehensive assessment of the system’s performance in practical applications. This would strengthen the evaluation of the proposed methods.

* 5\. **Interpretability and Usability:**

    - Question: How can the methods be made more interpretable and usable for practitioners who are not experts in federated learning or explainable AI?
    - Suggestion: Provide more intuitive explanations and visualizations. Including case studies or practical examples of FBPs and FBS in real-world scenarios could enhance understanding and applicability for a broader audience.

* 6\. **Real-World Validation:**

    - Question: Have you conducted any preliminary real-world validations of the proposed methods? If not, what are the plans for such validations?
    - Suggestion: Discuss any real-world applications or preliminary results. Outline a detailed plan for future real-world validation studies to demonstrate the practical applicability and effectiveness of your methods.

* 7\. **Comparison with State-of-the-Art:**

    - Question: How does the proposed Federated Behavioural Shields compare with other state-of-the-art defense mechanisms under different attack scenarios?
    - Suggestion: Provide a detailed comparative analysis with other robust aggregation methods under various attack types. Highlight the strengths and potential weaknesses of your approach in comparison to existing techniques.

* 8\. **Visualizations and Trajectories:**

    - Question: Can you provide more detailed examples or visualizations of client trajectories in the Error Behavioural Plane and Counterfactual Behavioural Plane?
    - Suggestion: Include more visual examples and explanations of client trajectories to illustrate how FBPs can be used to identify different client behaviors and detect malicious clients.

* 9\. **Impact of Non-IID Data:**

    - Question: How does the method handle non-IID (non-independent and identically distributed) data across clients, and how robust is it to such scenarios?
    - Suggestion: Discuss the impact of non-IID data distributions on the performance of FBPs and FBS. Provide experimental results or theoretical analysis demonstrating the method’s robustness to non-IID data.

* 10\. **Future Directions:**

    - Question: What are the future directions or potential extensions of your work?
    - Suggestion: Outline possible future research directions or extensions of FBPs and FBS. This could include integrating additional behavioral planes, optimizing computational efficiency, or exploring new applications of the methods.

**Conclusion:**
Addressing these questions and suggestions can provide clarity on the strengths and limitations of the proposed methods, enhance the understanding of their practical applicability, and offer insights into potential improvements. Engaging with these points during the rebuttal phase can lead to a productive discussion and potentially strengthen the overall contribution of the paper.
limitations: limitations: **Assessment of Limitations and Potential Negative Societal Impact:**

Based on the provided content and the NeurIPS checklist guidelines on limitations and broader societal impacts, here is an assessment of how well the authors have addressed these aspects and suggestions for improvement:

**Addressing Limitations:**

- **Identified Limitations:** The paper acknowledges some limitations, such as the computational overhead introduced by the concurrent training of counterfactual generators and the potential scalability issues in larger federated learning deployments.

- **Suggestions for Improvement:**
    - **Computational Efficiency:** Provide a more detailed discussion on the computational requirements and potential optimization strategies. This could include specific techniques to reduce overhead, such as model pruning, quantization, or distributed optimization methods.

    - **Scalability Analysis:** Include more comprehensive scalability experiments or simulations that predict the method's performance in larger, real-world settings. Discuss how the method can be adapted or optimized for large-scale deployments.

**Potential Negative Societal Impact:**
- **Security and Privacy:** The primary focus of the paper is on enhancing security and privacy in federated learning, which is a positive societal impact. However, potential negative impacts, such as the misuse of federated learning systems or unintended biases in the models, should be considered.

- **Suggestions for Improvement:**
    - **Bias and Fairness:** Discuss the potential for unintended biases in federated learning models and how the proposed methods could mitigate or exacerbate these biases. Provide suggestions for ensuring fairness in federated learning deployments.
    - **Misuse of Technology:** Address the potential for misuse of federated learning systems, such as using the technology for surveillance or other harmful purposes. Discuss safeguards and ethical considerations to prevent misuse.

**Constructive Suggestions for Improvement:**
* 1\. **Detailed Discussion on Limitations:**

    - Expand the discussion on identified limitations, providing more details on computational efficiency and scalability. Include theoretical analysis or empirical evidence supporting the claims and potential solutions.
* 2\. **Bias and Fairness:**

    - Add a section discussing the potential for biases in federated learning models. Explain how the proposed methods could impact fairness and provide guidelines or best practices for ensuring equitable outcomes.
* 3\. **Ethical Considerations and Misuse:**

    - Address potential misuse of federated learning technology. Discuss ethical considerations and propose safeguards to prevent the harmful application of the technology. Highlight the importance of transparency and accountability in federated learning deployments.
* 4\. **Societal Impact Statement:**

    - Include a comprehensive societal impact statement that covers both positive and negative aspects. This should highlight the benefits of enhanced security and privacy, as well as address the potential risks and ethical concerns.

**Conclusion:**
While the paper makes significant contributions to enhancing security and privacy in federated learning, it could benefit from a more thorough discussion of limitations and potential negative societal impacts. By addressing these areas, the authors can provide a more balanced view of their work and ensure that it is applied responsibly and ethically.
paper_summary: paper_summary: This paper introduces a novel method called Federated Behavioural Planes (FBPs) for analyzing, visualizing, and explaining the dynamics of Federated Learning (FL) systems. FBPs are consist of Error Behavioural Plane (EBP), reflecting the model’s predictive performance, and Counterfactual Behavioural Plane (CBP), reflecting the decision-making processes. Using insights from FBPs, the paper also proposes a new robust aggregation technique called Federated Behavioural Shields (FBS) to enhance security in FL systems.
strength_weakness: strength_weakness: - The paper heavily relies on visual representations (FBPs) to explain client behavior, which makes it infeasible to scale to large-scale FL systems where the convergence might take more rounds and client selection takes place in each round. These factors will largely make the trajectory on the planes hard to keep track of.

&nbsp;

- The computational overhead is too large. 
  + In each training round, the counterfactuals are required to be computed for the locally updated model from every client. And the differences between the counterfactuals from each pair of clients are also computed. For cross-silo FL settings, this is already a large computational overhead. For cross-device settings, I do not think this process is affordable especially for high dimensional data like image data.
  + Besides, since the paper focuses primarily on detecting anomalies and enhancing security from attackers, the main battlefield of this method should be more on the cross-device setting where the number of clients is large. To this end, the prohibitive computational overhead mitigates the motivation of this work.

&nbsp;

- The method completely relies on a validation set on the server, which may be hard to obtain without introducing any privacy concerns. Even a validation set is available to use, under what circumstances will the validation set be equally fair to all clients so that it is not far from any client’s local data distribution. And if the plane shows anomalies for some client, how to distinguish the reasons from having an unfair validation set and client is the anomaly/attacker?

&nbsp;

- The datasets and the federated settings in the experiments are too simple. And the compared robust aggregation methods are not so recent.
questions: questions: - Is the counterfactual generator learned on the server? Why would a generator whose goal is for better interpretability affect the performance of predictor?
- How is the local CL’s accuracy computed, and why is it so much lower than FL in Table 1?
limitations: limitations: Most limitations of the proposed method (validation set, computational overhead) have been addressed. Please refer to Weaknesses for other limitations of this work. There is no potential societal impact.
paper_summary: paper_summary: The paper introduces Federated Behavioural Planes (FBPs), a method designed to track FL clients' behavior by means of examining the their representations in two behavioural planes with the aid of a server-owned dataset. The Error Behavior Plane (EBP) and Counterfactual Behavioural Plane (CBP) correspond to two 2-D space where each client model's prediction errors and counterfactural distribution are examined. Combining these two representations a new aggregation rule is proposed to fend off malicious clients.
strength_weakness: strength_weakness: - Creating the two plains require the plaintext of all local models, which raises major privacy concerns.
- From Eqs. 6 and 7, I suppose generating the counterfacturals must be very costly.
- Insufficient experimental evidence to support the claim. For example, fig. 3 does not show distinct client behavior clearly. I also expect to see more results such as the behavioural scores of different clients and how sensitive the FBP is to the intensity of attacks.
questions: questions: - Results in Table 1 make me confused whether the whole method is still run in a cannonical FL setting.
- What is the difference between proximity and sparsity?
- How is counterfactural generator trained and why does it impact the task accuracy?
- What purpose does Eq. 4 serve?
limitations: limitations: - A framework overview is missing, which makes it hard to understand how the planes are created throughout the FL iterations.

Comments: title: Thanks for the rebuttal
comment: Thanks the rebuttal. Based on the author's rebuttal, I believe I have a better understanding of the content of the paper. Most of my concerns have been addressed and I have increased my rating.
Thank you for your valuable feedback. Please let us know if you have any further questions or if there are any points that need additional clarification.
Thank you for your valuable feedback. Please let us know if you have any further questions or if there are any points that need additional clarification. We would be grateful if you could consider updating your review in light of our responses.
Thank you for your valuable feedback. Please let us know if you have any further questions or if there are any points that need additional clarification. We would be grateful if you could consider updating your review in light of our responses.