FACT or Fiction: Can Truthful Mechanisms Eliminate Federated Free Riding?

Authors: marco  bornstein
amrit  singh bedi
abdirisak  mohamed
furong  huang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 17:32:18

Abstract: Standard federated learning (FL) approaches are vulnerable to the free-rider dilemma: participating agents can contribute little to nothing yet receive a well-trained aggregated model. While prior mechanisms attempt to solve the free-rider dilemma, none have addressed the issue of truthfulness. In practice, adversarial agents can provide false information to the server in order to cheat its way out of contributing to federated training. In an effort to make free-riding-averse federated mechanisms truthful, and consequently less prone to breaking down in practice, we propose FACT. FACT is the first federated mechanism that: (1) eliminates federated free riding by using a penalty system, (2) ensures agents provide truthful information by creating a competitive environment, and (3) encourages agent participation by offering better performance than training alone. Empirically, FACT avoids free-riding when agents are untruthful, and reduces agent loss by over 4x.

Main paper: 
 introduction: 
 Truth is essential to the functioning of fair systems. In its absence, fraud and corruption lead to negative effects on those left participating. This too is true within machine learning (ML). Adversaries can attack ML algorithms for insidious purposes [3, 7] , and algorithms trained on data that is not well-representative or poisoned can lead to dangerous and unfair results [28, 29, 37] .Issues of truthfulness have begun to bleed into multi-agent collaborative learning systems. Federated Learning (FL), the preeminent collaborative learning framework, is susceptible to the free-rider dilemma: agents can contribute little to nothing during training while still obtaining a well-trained final aggregated model [8, 12, 17] . Furthermore, Karimireddy et al. [12] proves that standard FL algorithms naturally yield catastrophic free-riding equilibria, where most rational agents contribute nothing. It is important to note that we do not consider the possibility of malicious gradient updates from adversarial agents, covered by recent works [6, 23, 25, 27] . We assume that agents share a common goal of reducing loss, and act selfishly rather than maliciously.Recent work has aimed to remedy the free-rider dilemma in FL through contract theory [4, 11, 16, 19] , collaborative fairness [21, 26, 36] , and mechanisms that incentivize contributions [2, 12] . The mechanisms presented in Karimireddy et al. [12] and Bornstein et al. [2] are notable because they do not require (i) carefully optimized contracts (agents will always participate because the mechanism is individually rational) or (ii) alterations to standard FL training (simple FedAvg [22] can be run with no computation of agent reputations). However, these mechanisms assume that agents are honest and truthful with an orchestrating central server. If agents can lie, the proposed mechanisms no longer guarantee elimination of free riding.In our paper, we propose a mechanism, Federated Agent Cost Truthfulness (FACT), that provably eliminates free riding even when agents are untruthful. Unlike standard FL approaches, where agents are incentivized to use little to none of their own data, agents participating in FACT are incentivized to use as much data for federated training as they would on their own. Furthermore, when agents use their locally-optimal amount of data, FACT is individually rational (IR): agents always receive greater benefit participating in FACT than by training alone. These results hold even if agents try to cheat the mechanism by misreporting individual training costs to the central server. FACT ensures that each agent's best strategy is to be truthful with the central server.Summary of Contributions. In summary, the main contributions of our paper are, (No Free Riding): Creating a novel penalization scheme that shifts the catastrophic free-riding optimum back to each agent's local optimum (thereby eliminating free riding). (Truthfulness): Constructing a truthfulness mechanism (competition) that dissuades agents from lying about their individual costs with the central server. (Realistic): Showcasing the efficacy of FACT empirically in eliminating agent free riding under untruthfulness, consequently improving federated training performance. 
related works: 
 Distributed Learning Mechanisms for Agent Contribution. An important line of mechanism research in distributed learning regards mechanisms that incentivize agent contributions for federated training [2, 12, 18, 38, 39] . Some of the first papers in this area were application-based, namely in mobile crowdsensing and smart sensors in vehicles. We detail these works in Appendix A. Zhan et al. [39] is one of the first works to define agent and server utility when data is shared between agents and the server to train a global model (including agent training costs). The server announces a total reward, and agents determine how much data to contribute in order to maximize their reward. A deep reinforcement learning method is used by the server to learn the optimal announced reward.An accuracy-shaping mechanism is proposed in Karimireddy et al. [12] , which also follows a similar data sharing and iid setting. This mechanism does not require the use of contracts or learning of a reward, and guarantees that an agent will receive improved reward when participating in the mechanism than by not participating. In the accuracy-shaping mechanism, the server degrades model performance for an agent if it does not contribute as much data as locally optimal. Bornstein et al. [2] proposes a similar accuracy-shaping mechanism. However, unlike Zhan et al. [39] and Karimireddy et al. [12] , agent utility is modeled in a non-linear and more realistic manner. Additionally, the accuracy-shaping mechanism is generalized to the federated setting (no data sharing) where data can be non-iid. Unlike the majority of the works above, our mechanism seeks not just to incentivize agent contribution, but to eliminate free riders. Moreover, unlike [2, 12] , we do not assume that agents are truthful. Our mechanism ensures that agents contribute what is locally optimal, thereby eliminating free-riding, and punishes agents for lying to the server about their costs.Truthful Distributed Learning Mechanisms. Similar to the agent contribution mechanisms above, many existing truthful distributed learning mechanisms exist within the crowdsensing setting [9, 33, 35, 42] . We detail these works in Appendix A. The more recent works [14, 20, 34] examine truthful mechanisms within traditional FL. Lu et al. [20] explores truthful mechanisms in Vertical FL, where features are split up amongst agents. Specifically, a truthful mechanism is designed (via a novel transfer-payment rule) to maximize social utility, without use of an auction, even when agents can lie about their costs. Within Horizontal FL (the standard approach), Le et al. [14] designs a randomized auction for resource-constrained wireless agents, which achieves approximatetruthfulness in expectation. The server receives bids from agents to perform training, and selects a winning bid (agent) for each uplink sub-channel. The server pays a reward designed to minimize agent social cost, and does not take into account the server's utility. Wu et al. [34] builds on the auction idea in Le et al. [14] , instead using a federated auction bandit which learns to select bidding agents that maximize the server's utility. The mechanism ensures truthfulness and individual rationality. Like the works above, our mechanism ensures truthfulness and individual rationality of participating agents. In contrast, our mechanism setting is within horizontal FL and does not require an auction or bandit method to ensure truthfulness. As such, any willing agent can participate in training (no bids are rejected), an optimal server reward does not need to be learned, and agents do not need to have a private valuation. Finally, our mechanism requires no direct payment from the server (all reward payments come from other agents) and provably eliminates the free-riding dilemma. Due to space constraints, we detail works regarding fairness and agent selection mechanisms in Appendix A. . Models are aggregated by the server and this process is repeated until convergence. Each agent is assumed to have access to the same data distribution D, and thus we assume data is independent and identically distributed (iid). We leave construction of truthful mechanisms in the non-iid setting for future research. Our work aims to bring truthful mechanisms to CFL, as none have existed in either the iid or non-iid settings prior to this work. We now begin by detailing the objective that agents participating in FL seek to minimize. Federated Objective. The goal of FL is to train a global model, leveraging the data and compute power of thousands of agents, that achieves low loss. To start, we utilize the convergence of distributed stochastic gradient descent (D-SGD) [15] to first-order stationary points in non-convex settings,EQUATIONThe parameter ∆ f is the difference between the optimal objective value f * and starting value f (w 0 ). The global objective function f is the uniform sum over all device objective functions f i . There are T total iterations. The step-size is γ and L is the Lipschitz constant, where γL < 2. The effective gradient variance is σ 2 / n j=1 m j . This stems from bounded variance assumptions, formally defined asE∥ 1 M M i=1 ∇f (w; ξ i ) -∇f (w)∥ ≤ σ 2M (for M batches of data ξ).Local Agent Loss. All rational agents seek to minimize their loss. As shown in Equation ( 1), this can be accomplished by using more data, and thus a larger batch size m, as well as more rounds of gradient updates T . However, all agents incur a cost c i for collection, sampling, and compute costs for each data sample m used. The linear cost c i m stems from our cross-agent setting. Within this setting, IoT devices are prevalent and data is not difficult to collect (e.g., IoT sensors). Thus, it makes the most sense that, on average, there are constant costs over time to collect and sample data (e.g., the cost to power a sensor is usually constant on average) [30, 20, 34] .Overall, agents must balance the benefit of using more data to achieve better model performance with the costs of data collection, sampling, and computations. We formulate this tradeoff for each agent by defining a local loss function ℓ i,l (m i ), that depends on the data collected m i by each agent i. Each agent minimizes its loss function, which is a combination of data costs and convergence error terms.When training locally, or alone, an agent loses the benefit of an increased batch size across all agents n j=1 m j . Thus, after removing constants, Equation (1) transforms into our local agent loss,EQUATIONThe optimal amount of data each agent uses for local training can be determined with calculus.Theorem 1 (Optimal Local Data Usage). For an agent i with marginal cost c i , the optimal amount of data m * i,l used for local training is m * i,l := γσ 2 L 2ci .Remark 1. Agents' optimal amount of local data is inversely proportional to its cost c i . Larger costs lead to less data used and vice versa. If costs were zero, agents would use infinite data.The Free-Rider Dilemma: Shifted Optima in Federated Settings. When agents engage in federated training, the effective batch size returns to the sum over all agent data n j=1 m j . By participating in federated training, agents gain the benefit of an increased batch size. As such, each agent i's loss function in the federated setting becomes,EQUATIONWhile slight, this transformation changes the optimal amount of data used by each agent.Theorem 2 (Free-Riding: Optimal Federated Data Usage). For an agent i with marginal cost c i , the optimal amount of data m * i,F used for federated training ism * i,F = γσ 2 L 2ci -m -i .Remark 2. By participating in federated training, each agent is incentivized to use fewer data samples. This explicitly demonstrates the free-rider dilemma mathematically: it may be optimal for an agent to use no data m * i,F = 0 (if m -i ≥ γσ 2 L 2ci ) and still possibly achieve lower cost overall (ifm -i ≥ m * i,l ).Due to space constraints, we leave the proofs of Theorems 1 and 2 in Appendix C. 
eliminating free riding via penalization: 
 The natural equilibrium of traditional FL is free riding: agents use less data than is locally optimal in proportion to how much data other agents use (Theorem 2). This necessitates the construction of a mechanism to restore the equilibrium of FL back to what is locally optimal for each agent. In this section, we detail how penalization can perform this restoration.Penalized Federated Learning (PFL). To disincentivize agents from straying from their locally optimal data usage m * i,l , the central server can use contract theory to ensure each agent i uses its locally optimal amount for a reported cost c i . Namely, when agreeing to participate in FL, each agent i agrees to pay the following free-riding penalty P f r if they do not use m * i,l = γσ 2 L 2ci data samples,EQUATION)The hyperparameter λ i > 0, chosen by the central server, denotes the harshness of the free-riding penalty. The value of λ i is known by each agent i while deciding whether or not to participate in the mechanism. Since the penalty is levied on each agent by the server, it is incorporated into their federated loss function. This results in a new amended federated loss function ℓ i,P F L (m i ) defined as,EQUATIONTheorem 3 (PFL Eliminates Free Riding). For an agent i with marginal cost c i , the optimal amount of data m * i,P F L used for federated training under Equation ( 5) is m * i,P F L := γσ 2 L 2ci .Algorithm 1 PFL: Penalized Federated Learning EQUATION5: Server receives free-rider penalty payment P f r (m i ) from each agent via Equation ( 4) using λ i 6: s i ← m i / n j=1 m j 7: for t = 1, . . . , T do 8:Server distributes w t to all agents 9:for h local steps, each agent i in parallel do 10:w t+1 i ← Agent Update(w t , i, m i , f, h) {AgentUpdate is detailed in Algorithm 3} 11:end for 12:w t+1 ← i s i w t+1 i 13: end for Remark 3. By utilizing the PFL loss in Equation (4), agents participating in FL will use a locally optimal data amount (m * i,P F L = m * i,l ). This eliminates the free-rider dilemma seen in Theorem 2. Agents participating in PFL (Algorithm 1) follow the amended loss shown in Equation ( 5). Careful selection of λ i is required to ensure that PFL is IR.Ensuring Individual Rationality (IR) in PFL. As shown above in Theorem 3, it is optimal for agents participating in the penalized federated scheme (with a loss function shown in Equation ( 5)) to contribute what is locally optimal i.e., m * i,P F L = m * i,l . To ensure individual rationality, it must be true that participating in federated training will produce at least as much reward for an agent than by not participating. The selection of λ i is crucial in accomplishing this task.Lemma 1 (PFL Assurance of IR at Optimum). Let c i be the marginal cost for an agent i, m * := γσ 2 L 2ci = m * i,l = m * i,P F L be the agent's locally optimal data usage, and α ∈ [0, 2) be a server-specified hyper-parameter. In order to ensure that the optimal penalized federated loss is at least lower than the optimal local loss, ℓ i,P F L (m * i ) ≤ ℓ i,l (m * i ), one must select λ i such that,EQUATION)Selection of such λ i results in a loss gap between ℓ i,P F L (m * ) and ℓ i,l (m * ) of,EQUATIONRemark 4. Agents provably receive improved loss ∆ℓ when they use their locally optimal data amount m * i,l participating in PFL (Algorithm 1) versus local training, thereby ensuring IR.Remark 5. The parameters λ i and α control the amount of benefit received by an agent i. A larger value of α, and consequently λ i , results in a larger gap between ℓ i,P F L (m * ) and ℓ i,l (m * ).Inversely, a smaller value of α and λ i results in a smaller gap between local and PFL loss.Due to space constraints, we leave the proofs of Theorem 3 and Lemma 1 in Appendix C. Now, we formally prove that PFL both (i) eliminates agent free riding and (ii) is individually rational.Theorem 4 (Elimination of Federated Free-Riding With Truthful Agents). PFL (Algorithm 1) using λ i from Lemma 1 is IR and eliminates the free-rider dilemma when agents are truthful.Proof. The result of Theorem 3 is that each agent i's optimal strategy within the penalized federated scheme is to use their locally optimal amount of data m * i = γσ 2 L 2ci . Furthermore, Lemma 1 states that using m * i within the penalized federated scheme results in a reward, or improvement over local Algorithm 2 FACT: Federated Agent Cost Truthfulness1: Input: data m = [m 1 , . . . , m n ], marginal costs c = [c 1 , . . . , c n ], h local steps, T iterations, Eepochs, initial parameters w 1 , loss f , step-size γ, and constants (L, σ 2 , α). 2: Output: model w T and monetary reward r i . 3: Server receives contract payment from each agent:∆ℓ i ← α 4 γσ 2 L m-i mi( m)4: Run Penalized Federated Learning: w T ← PFL(m, c, h, T, E, w 1 , f, γ, L, σ 2 , α) 5: Compute each agent reward r i via the cost truthfulness game in Equation (10) training, of∆ℓ i = α 4 γσ 2 L m-i m * i (m * i + m-i). Thus, by combining Theorem 3 and Lemma 1, truthful agents which choose to participate in PFL attain a reshaped and lower loss at the same optimum. Individually rational agents would therefore prefer to participate in PFL (Algorithm 1) over local training due to the reshaped optimum's lower loss. This optimum achieves the same data usage as local training, thereby eliminating the free-rider dilemma.Remark 6 (Truthfulness Concerns). Participating in PFL, rational agents contribute as much as they would locally (Theorem 3) and attain more benefit (Lemma 1). The only issue that remains is truthfulness: agents may lie about their costs in order to contribute less to federated training under the guise of what looks like an "optimal amount" to the server (thereby avoiding large P f r ).While PFL eliminates free riding when agents are truthful, this assumption is not realistic in practice. Untruthful agents can report inflated costs to the server in order to trick it into expecting smaller data usage. In effect, agents can lie about their costs to free ride. In the next section, we propose a mechanism which incentivizes agents to be truthful (i.e., their best strategy is not to lie). 
fact: eliminating free-riding with untruthful agents: 
 When agents are truthful, PFL (Algorithm 1) provably eliminates the free-rider dilemma (Theorem 4). However, an adversarial agent i can still free ride by misreporting, or lying, about their true marginal cost c i . As previously mentioned, we do not consider the situation of malicious gradient updates coming from adversarial agents, covered by recent works [6, 23, 25, 27] . We assume that agents share a common goal of reducing loss, and act selfishly rather than maliciously.As seen in Theorem 2, an adversarial agent can lie their way to a smaller data optimum by inflating their cost:m lie i = γσ 2 L 2(ci+ϵ) < m * i,l .In this case, an adversarial agent will (i) avoid server penalty (since it will be contributing what looks like an optimum from the server's view), (ii) incur smaller true costs c i m lie i < c i m i , and (iii) still reap the benefits of a larger batch size m -i + m lie i . Definition 1 (Truthful Mechanism). A truthful mechanism is one in which no agent can reduce its loss by reporting a cost different from its true cost, regardless of other agents' actions. Overall, each agent's dominant strategy is to report its true cost.To counteract the possibility of an agent being untruthful with the server about its true cost, we propose a truthful mechanism FACT: Federated Agent Cost Truthfulness (Algorithm 2). FACT ensures that each agent i's dominant strategy is to report its true cost c i (thereby satisfying Definition 1).Agents have no knowledge of other agents' costs or the distribution of agent costs f C . In the absence of information, each agent i believes that its cost c i is equally likely to be larger or smaller than any other agent's cost.Our lone assumption about agent knowledge is that agents are unknowledgeable about the distribution of agent costs f C or any other agent's cost: agent costs are private. Assumption 1 ensures that there is no collusion amongst agents. In the absence of knowledge about other agents' costs, it is reasonable for an agent to believe that their private cost is equally likely to be greater or lesser than any other agent's cost (i.e., the median cost). Now that agents can report costs c which differ from their true cost c i , each agent's PFL penalty (Equation 4) and loss (Equation 5) becomes a multivariable function,EQUATION)EQUATIONRegardless of the reported cost c, each agent i will incur a true cost c i m i locally in Equation ( 9).Using a Sandwich to Incentivize Truthfulness. Our mechanism is the first to ensure truthfulness while solving the free-rider dilemma in FL. FACT ensures agent truthfulness by fostering a competition amongst agents. The competition begins at the end of federated training (PFL) by randomly grouping reported agent costs into threes. The rules of this competition, or truthfulness mechanism, are simple: each agent i "wins" the competition, and receives a reward, if its cost c i is sandwiched between the two other agent costs in its group. If there is a tie, the server randomly selects one of the tied agents as the winner. This mechanism is detailed mathematically below,P ct (c) :=    ∆ℓ i -3 n j̸ =i∈[n] ∆ℓ j if C a < c < C b , ∆ℓ i else. C a , C b ∼ f C randomly sampled costs (10)Lemma 2. The probability of an agent "winning" in the truthfulness mechanism in Equation (10) ,given a reported cost c, is υ(c) := 2F C (c)(1 -F C (c)), where F C is the CDF of f C .The crux of the truthfulness competition is that each agent i never receives its own reward ∆ℓ i . Instead, as shown in Equation ( 10), each agent pays ∆ℓ i to the server before training regardless of the mechanism outcome. After training, if the agent "wins" the competition, by having a cost sandwiched between C a and C b , then it receives triple the average of all other agent rewards 3 n j̸ =i∈[n] ∆ℓ j . The average reward is tripled, since two other agents will lose the competition and receive no reward. By making the reward decoupled from an agent's own reward ∆ℓ i , each agent can no longer increase or decrease its reward by altering how much data m i they use for federated training. Now, they can only affect the likelihood of winning the competition by choosing what cost c i to report to the server.Incorporating the truthfulness mechanism into the PFL loss function results in the FACT loss function, Traditional FL is an upper bound on agent loss (if agents did not free ride). FACT improves agent loss over local training by up to a factor of 3x for CIFAR10 and 4x for MNIST.EQUATIONTheorem 5 (Main Theorem). Each agent i's best strategy, when participating in FACT (Equation 11), is to report its true cost and use its locally optimal amount of data(m i , c) * i = ( γσ 2 L 2ci , c i ).Remark 7 (Truthfulness). FACT is a truthful mechanism: each agent's best strategy is to report its true cost c i . Agents cannot reduce their loss by reporting a different cost (satisfying Definition 1).Remark 8 (Elimination of Free Riding). FACT eliminates agent free riding: each agent's best strategy is to use as much data as is locally optimal m * i = γσ 2 L 2ci (Theorem 1). Thus, the improved optimum for PFL (Theorem 3) is maintained even under agent untruthfulness.Remark 9 (Individually Rational). When using their best strategy, agents participating in FACT either (i) receive loss equivalent to local training if they lose the competition or (ii) reduced loss if they win (shown in Equation 10). Thus, FACT is IR as agents using their best strategy never receive worse loss than if they did not participate. In fact, agents receive lower loss in expectation. 
experimental results: 
 Below, we analyze the effectiveness of FACT at (1) enforcing agent truthfulness, (2) eliminating free riding, and (3) reducing agent loss compared to local training. We perform true federated experiments (no simulations) using CIFAR10 [13] and MNIST [5] datasets to train an image classification model.Experimental Setup. Within our experiments, 16 agents train a model individually (locally) as well as in a federated manner. Each agent uses 3,125 and 3,750 data samples each for CIFAR10 and MNIST respectively. We analyze FACT under homogeneous and heterogeneous agent data distributions. For heterogeneous agent data distributions, we use Dirichlet distributions with parameters α = 0.3, 0.6 to determine the label ratio for each agent [10] . We simulate the sandwich truthfulness competition defined in Equation ( 10) by randomly sampling 2,000 synthetic agent costs from a Gaussian distribution with a mean centered at each agent's true cost (standard deviation of one-tenth of the true cost). We perform 100,000 simulation trials and compute mean performance for each agent over all trials. Further experiment details and hyperparameters are found in Appendix B. Lack of Baselines. FACT is the first truthful Horizontal FL method that eliminates free riding while simultaneously enforcing truthfulness without the use of an auction or bandit (thereby allowing all willing agents to participate). As such, we compare the results of FACT with the only available option: local training. We showcase below that FACT reduces agent loss compared to local training while simultaneously avoiding free-riding and truthfulness issues.FACT Enforces Truthfulness. The results of Figure 3 back the theoretical result of Theorem 5, which states that agents maximize their expected reward (i.e., net reduction in loss versus local training) when truthfully reporting their cost. Across both homogeneous (iid) and heterogeneous (n-iid) agent data distributions, agents' net improvement in loss peaks when reporting their true cost (0% added) and monotonically decreases as agents report an inflated or deflated true cost. Thus, the sandwich-style truthfulness competition FACT employs disincentivizes agents from inflating or deflating their true costs. This is important, as agents are no longer incentivized to lie to the server about their true cost in order to free-ride without the free-rider penalty in Equation (4) being imposed.Net improvement in loss grows as data distributions become increasingly non-iid. FL is robust enough to deal with the increasingly non-iid datasets while local training suffers (Figures 5 & 6 ). Therefore, the gap between FL and local training grows, and thus so too does the reward in FACT.FACT Reduces Agent Loss. Agents participating in FACT on average achieve a reduction in total loss compared to training on their own when using the same amount of data. Consequently, participating in FACT is individually rational: agents at worst receive loss equivalent to local training and on average receive improved loss (Figure 2 ). Traditional FL achieves a lower average agent loss than FACT when all agents use as much data as they would locally. However, as shown in Theorem 2, this is unrealistic. Traditional FL emits a catastrophic free-riding optimum. The results of traditional FL in Figure 2 is the upper bound on achievable loss by agents, only attainable if agents refused to free ride. The gap in loss between FACT and the upper bound of traditional FL is the cost of agent untruthfulness and their inclination to free ride. The gap is the direct result of the penalty terms on free riding P f r and truthfulness P ct found in Equation (11) . Average local loss increases as data distributions become increasingly non-iid, while traditional FL remains robust to the distribution shift (Figures 5 & 6 ). Since the agents in FACT who lose the truthfulness competition receive a loss equivalent to local training, the loss of FACT also rises in proportion to the local training loss.FACT Eliminates Free Riding. Within our experiments for CIFAR10 and MNIST, a marginal cost was carefully selected such that it is locally optimal for each agent to use 3,125 and 3,750 data samples for training respectively. As shown in Figure 3 , the free-rider penalty P f r harshly penalizes agents for deviating from this locally-optimal amount. This result accounts for the reduction (gain) in data collection costs for using less (more) data during training. Thus, as proven theoretically in Theorem 3, we confirm that it is suboptimal for an agent i to use either more or less data than is what is locally optimal (for a given reported cost c i ) when participating in FACT. When training an image classifier for diagnosing skin cancer, agents participating in FACT achieve much lower loss (66% less) than if they did not participate (left). Agents maximize their improvement in loss over local training when they are truthful; reporting inflated or deflated costs diminishes improvement in loss (middle). Agents minimize penalties when using their locally optimal amount of data (m * = 801) for training (right).We consider a realistic situation where a consortium of hospitals seek collaboration to train a model, privately in a FL manner, that can diagnose skin cancer. One of the hospitals is smaller and resourceconstrained. It is difficult, but not impossible, for this hospital to collect more data for training. Thus, in the absence of a truthful FL mechanism, the smaller hospital could over-report its collection costs to the server in order to contribute little to no data towards training while still reaping the rewards of a well-trained global model.To test how FACT deals with this scenario, we train an image classification model on the HAM10000 [31] dataset. HAM10000 consists of a collection of dermatoscopic images of pigmented lesions. The goal is to train a model which can perform automated diagnosis of pigmented skin lesions, including melanoma (skin cancer). Our setup is similar to the experimental setup detailed in the previous section. One difference is that we fine-tune a ResNet50 model on HAM10000 that is pre-trained on ImageNet (a realistic approach for a hospital). HAM10000 is an imbalanced dataset, and evenly partitioning 80% of the data amongst 10 devices, as their local training sets, further exacerbates the non-iidness of the data. This too is realistic, as data is often non-iid amongst FL agents. We use the Adam optimizer with a learning rate of 1e-3 and batch size of 128.As expected, FACT provably dissuades any hospital from lying about its cost in order to reduce their data contribution level. Agents are incentivized to be truthful; reduction in loss is maximized when agents truthfully report their costs (see the middle plot in Figure 4 ). As a result, a smaller, resourced-constrained hospital would still contribute its locally optimal amount of data, or face harsh penalties (see the right plot in Figure 4 ). Finally, all participating hospitals are still better off participating in FACT than training alone. Participating agent loss is reduced by nearly 66% compared to local training (see the left plot in Figure 4 ). The effectiveness of FACT to dissuade free-riding in the real-world setting of training a skin cancer classifier across hospitals indicates that it will be successful in other real-world settings. 
conclusion: 
 Traditional FL frameworks are susceptible to free riding: it is often an agent's best strategy to contribute little to nothing during training while still obtaining a well-trained final global model. Furthermore, many FL frameworks do not account for untruthful agents which report inaccurate information to a server in order to skip penalties and collect greater rewards. Our proposed mechanism, FACT, is simultaneously individually rational, truthful, and eliminates free riding. FACT leverages a novel penalization and sandwich mechanism to shift each agent's best strategy to report its true cost with the server (truthfulness) and use as much data as it would on its own (no free riding). Furthermore, agents which participate in FACT will never receive worse loss than by not participating, and receive lower loss on average if they participate (individually rational). Empirically, we find that FACT enforces agent truthfulness while reducing agent loss by upwards of a factor of four.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Our results are run for multiple trials (3). Our results are the average over our multiple trials. Error bars are plotted as detailed in Appendix B.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: This is found in Appendix B.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). On the contrary, we seek to mitigate current risks in distributed learning. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: This can be found in Appendix D.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. 
 
 Appendix: While small, the research area of incentives and mechanism design in FL is growing. There are a few surveys [32, 40, 41] which overview FL mechanisms, current research directions & challenges, and future opportunities. We hone in on a few of these research directions, namely agent contribution and truthful FL mechanisms, as well as fairness and agent selection mechanisms. Due to space constraints, we continue our related works (Section 2) here below.Distributed Learning Mechanisms for Agent Contribution (Continued). Within Zhan et al. [38] , an iterative Nash bargaining solution is proposed where mobile devices bargain with a server over the amount of data to contribute and the payments it should receive. The server dynamically alters the payment price to adjust its gap between desired supply of data and the given demand. Liu et al. [18] designs a reverse-auction approach, with subsidized or "endowment" compensation, that incentivizes non-participating vehicles to reduce roadside server congestion by offloading its data. Both of these works deal with data exchange and do not consider the FL setting, where models are collaboratively trained together. Furthermore, it is not guaranteed in either that agents and the server will both always achieve higher payoff for participating than not (in Liu et al. [18] the server attains smaller payoff by participating).Truthful Distributed Learning Mechanisms (Continued). Xu et al. [35] deals with agents submitted untruthful bids to a server, specifically where an agent can lie about sensing time windows and costs.The proposed mechanism leverages a Vickrey-Clarke-Groves auction, thereby ensuring truthfulness in bid price. Time-window truthfulness is ensured via trusted time stamping. Similarly, the remaining crowdsensing works leverage auctions to ensure agent bid truthfulness. Zhang et al. [42] introduces a double auction mechanism, MobiAuc, for proximity-based mobile crowdsensing. MobiAuc addresses issues of dynamic mobility patterns (device matching) while maintaining agent truthfulness surrounding their bid (e.g., service valuation or arrival time). In Wang et al. [33] , a distributed truthful mechanism is proposed, where agents send bids for their services (usually data) to buyers. Gao et al. [9] designs a reverse auction mechanism that ensures vehicles report their true costs when crowdsensed. The works above do not consider the FL setting, where models are trained through the help of many agents. Thus, these works do not solve the free-rider dilemma, which only exists in FL settings. Furthermore, our mechanism is able to ensure agent truthfulness without the use of an auction.Fairness FL Mechanisms. Recent work within FL has focused on agent fairness [1, 21, 24] . These works seek to fairly allocate model performance to agents depending upon how much they contribute during federated training. Lyu et al. [21] computes a "reputation" metric for each agent (measures the amount of agent contribution). A better reputation leads to improved model performance. Both Blum et al. [1] and Murhekar et al. [24] seek to find agent contribution equilibria that improve the welfare of all agents. Within [1] , the existence and stability of equilibria which avoid free riding and agent envy. In [24] , a mechanism is proposed to alter agent strategies in order to maximize the net utility of all participating agents. Unlike the works above, FACT is a truthful mechanism.There exists recent literature which aims to incentivize high-quality agents to participate in federated training [11, 19, 43] . Agents can be deemed as high quality if, for example, they have valuable data or are very willing to participate in training. In Liu et al. [19] , a two-dimensional contract model is created to consider agent willingness and data quality when designing contract fees. Kang et al. [11] introduces a scheme where agent reputations are calculated (from previous interactions or other server interactions) and leveraged to select reliable agents for federated training. Zhou et al. [43] designs an auction method that incentivizes a diverse set of agents to participate in training. Truthfulness and individual rationality are ensured by properties of the auction that is constructed. FACT does not select agents based on a given quality or willingness criteria. All agents are able to participate if they choose. Instead, FACT solves agent free riding when agents are truthful as well as untruthful. end for 9: end forExperimental Setup. We use 16 agents to train a ResNet18 and a small convolutional neural network (for CIFAR-10 and MNIST respectively). The optimizer we use is Stochastic Gradient Descent for CIFAR-10 and Adam for MNIST. We use the standard FedAvg algorithm to perform federated training. We ran all experiments on a cluster of 2-4 GPUs, with the 16 CPUs (agents) pinned to a GPU. We use GeForce GTX 1080 Ti GPUs (11GB of memory) and the CPUs used are Xeon 4216. We provide a table of the hyperparameters not listed in Section 6 below. Proof. Each agent has a local loss function as shown in Equation ( 2). Taking the derivative of ℓ i,l (m) with respect to the data contribution amount m yields,EQUATION)Theorem 2 (Free-Riding: Optimal Federated Data Usage). For an agent i with marginal cost c i , the optimal amount of data m * i,F used for federated training ism * i,F = γσ 2 L 2ci -m -i .Proof. Each agent has a federated loss function as shown in Equation (3) . Taking the derivative of ℓ i,F (m) with respect to the data contribution amount m yields,EQUATIONEQUATIONTheorem 3 (PFL Eliminates Free Riding). For an agent i with marginal cost c i , the optimal amount of data m * i,P F L used for federated training under Equation ( 4) is m * i,P F L := γσ 2 L 2ci .Proof. Each agent has a penalized federated loss function as shown in Equation ( 5). Taking the derivative of ℓ i,P F L (m) with respect to the data contribution amount m yields,EQUATIONDue to the convexity of Equation ( 5), as each piece of the utility function is convex, there is a single minimum which is carefully constructed to be at m * = γσ 2 L 2ci ,EQUATION)Lemma 1 (Assurance of IR at Optimum). Let c i be the marginal cost for an agent i, m * := γσ 2 L 2ci = m * i,l = m * i,P F L be the agent's locally optimal data usage, and α ∈ [0, 2) be a serverspecified hyper-parameter. In order to ensure that the optimal penalized federated loss is at least lower than the optimal local loss, ℓ i,P F L (m * i ) ≤ ℓ i,l (m * i ), one must select λ i such that,EQUATION)Selection of such λ i results in a loss gap between ℓ i,P F L (m * ) and ℓ i,l (m * ) of,EQUATION)Proof. We can determine the range of λ i values which ensure IR by plugging in m * i,l = m * i,P F L = m * into Equations ( 2) and ( 5) and finding the difference,EQUATION)To ensure the Equation ( 19) is greater than or equal to 0, we must select λ i as the following,EQUATION)Equation (20) shows that λ i must be non-negative (all values are positive and the squared term can be zero at a minimum). One can think of λ i as the parameter controlling the benefit received by an agent i. A larger value of λ i will result in a larger gap between ℓ i,l (m * ) and ℓ i,P F L (m * ) (Equation ( 19)). Inversely, a smaller value of λ i will result in a smaller gap between local and federated training utility. Therefore, we fully define λ i with a user or server specified hyperparameter α ∈ [0, 2),EQUATION)Plugging this term into Equation ( 19) yields the following utility gap between ℓ i,l (m * ) and ℓ i,P F L (m * ),EQUATION)Theorem 4 (Elimination of Federated Free-Riding With Truthful Agents). PFL (Algorithm 1) using λ i from Lemma 1 is IR and eliminates the free-rider dilemma when agents are truthful.Proof Reproduced from Section 4. The result of Theorem 3 is that each agent i's optimal strategy within the penalized federated scheme is to use their locally optimal amount of data m * i = γσ 2 L 2ci . Furthermore, Lemma 1 states that using m * i within the penalized federated scheme results in a reward, or improvement over local training, of∆ℓ i = α 4 γσ 2 L m-i m * i (m * i + m-i). Thus, by combining Theorem 3 and Lemma 1, truthful agents which choose to participate in PFL attain a reshaped and lower loss at the same optimum. Individually rational agents would therefore prefer to participate in PFL (Algorithm 1) over local training due to the reshaped optimum's lower loss. This optimum achieves the same data usage as local training, thereby eliminating the free-rider dilemma.Lemma 2. The probability of an agent "winning" in the truthfulness mechanism in Equation (10) ,given a reported cost c, is υ(c) := 2F C (c)(1 -F C (c)), where F C is the CDF of f C .Proof. The probability that an agent's cost c is sandwiched in between two other randomly sampled agents' costs c u , c v is equal to,EQUATIONLet F C be the cumulative distribution function (CDF) of the agent cost distribution f C . The probability that a random cost, let's say c u , is smaller than c is equal to F C (c). The probability that a random cost, let's say c v , is larger than c is 1 -F C (c). Since the costs are random, and the above situation could be flipped (i.e., c v is smaller than c with probability F C (c) and c u is larger than c with probability 1 -F C (c)), the two probabilities in Equation ( 23) are equivalent. Thus, due to symmetry,P (c u ≤ c ≤ c v ) = P (c v ≤ c ≤ c u ). Therefore, we can rewrite the equation above as,EQUATIONTheorem 5 (Main Theorem). Each agent i's best strategy, when participating in FACT (Equation 11), is to report its true cost and use its locally optimal amount of data (m i , c) * i = ( γσ 2 L 2ci , c i ).Proof. Agents participating in FACT follow the loss function shown in Equation (11) ,EQUATIONTaking the expectation of the equation above over the distribution of agent costs f C , and using the results of Lemma 2, yields,E f C [ℓ i,F act (m i , c)] = γσ 2 L 2(m i + m -i ) + c i m i + P f r (m i , c) + E f C [P ct (c)], = γσ 2 L 2(m i + m -i ) + c i m i + P f r (m i , c) + ∆ℓ i - 3υ(c) n j̸ =i∈[n]∆ℓ j (26) Taking the partial derivative with respect to m i and setting it equal to zero results in,EQUATIONFrom Theorem 3, we know that this results in m * i = γσ 2 L 2c . Now, when taking the partial derivative with respect to c we can plug in m * i (to ensure the critical point is evaluated as zero), ∂ ∂cEQUATIONBy Lemma 1 and specifically Equation ( 7), we find that the first four terms above are equivalent to the local agent loss at its local optimum. Therefore, we can rewrite the equation above as, ∂ ∂cEQUATIONThe final line follows from Lemma 2. Since F C (c) is a CDF, its range is[0, 1]. The function υ(c) = 2F C (c)(1 -F C (c)) achieves is global maximum when F C (c) = 1/2 (its critical point).Therefore, the value of c which satisfies Equation ( 29) is one such that F C (c) = 1/2. By Assumption 1, we know that the median value is assumed by each agent i as c i , since each agent believes its cost is equally likely to be larger or smaller than any other agent's cost. Thus, F C (c i ) = 1/2 and c * = c i . Plugging this optimal value of c back into the optimal value for m i leads to the following optimum:(m i , c) * = ( γσ 2 L 2ci , c i ).Current federated systems are inhibited by agent free riding. The result is an unfair system. Some agents perform the bulk of training, while others sit idle. In the end, all the agents receive the same model performance. While there are current methods which eliminate agent free riding, they do not take into account agent truthfulness. Many agents can still provide false information to the server in order to free ride. The impact of FACT is that it provides an easily implementable mechanism which can make federated training more robust to free riding. Using FACT, agents are incentivized to no longer free ride even if they can lie about their training costs.Our work faces limitations when agents and the central server act maliciously. On the agent side, we assume that agents are rational and do not collude with one another. Our future research direction is to prove that equilibria for FACT exist when agents are boundedly rational or colluding. On the server side, we assume that the central server acts honestly. In settings where the server cannot be trusted, new incentives or avenues must be built in order to ensure server honesty. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: This paper introduces Federated Agent Cost Truthfulness (FACT) to address the issue of free-riding in federated learning. The key contributions include introducing a penalization scheme that incentivizes agents to provide truthful information, thereby eliminating free-riding and improving the performance of federated training. The proposed mechanism ensures that agents contribute as much data as they would in local training, thus aligning individual incentives with collective goals. Empirically, FACT effectively avoids free-riding even when agents are untruthful and significantly reduces agent loss. Some strengths of the paper include that: the method part of the paper has clear logic, and the contract theory is used to effectively prevent free-rider attacks in federated learning;
almost all the proposed theories or definitions have been proven, so sufficient theoretical evidence has been provided to demonstrate the effectiveness of FACT.; the problem solved is very critical, and it is the first study of free-rider attacks in federated learning under active deception conditions. In the rebuttal discussion, the authors had provided relatively concrete content to resolve many of reviewers’ concerns. There are still few concerns related to experiment parts in the discussion. After reading the reviews and the rebuttal, and going through the paper, my own reading is that the effectiveness of FACT mechanism has been demonstrated generally by the way of proving theories. Therefore, I suggest to accept this paper.

Reviews: review: paper_summary: paper_summary: The paper "FACT or Fiction: Can Truthful Mechanisms Eliminate Federated Free Riding?" proposes a mechanism called FACT (Federated Agent Cost Truthfulness) to address the issue of free-riding in Federated Learning (FL). The key contributions include introducing a penalization scheme that incentivizes agents to provide truthful information, thereby eliminating free-riding and improving the performance of federated training. The proposed mechanism ensures that agents contribute as much data as they would in local training, thus aligning individual incentives with collective goals.
strength_weakness: strength_weakness: Theoretical Reasoning: The paper does not explicitly define the game type and solution concept used, and lacks comparison with alternative abstractions. Recent critiques on the practicality of truthful mechanisms in complex environments are not addressed.

Practical Application: The focus on client-side truthfulness overlooks server-side challenges. Implementing the sandwich mechanism on the server side could increase costs and disincentivize its adoption. Clients might not trust the server to implement the mechanism, leading to untruthful behavior.

Experimental Validation: The experiments are limited to CIFAR-10 and MNIST datasets, and do not provide a broad validation of the mechanism's applicability and robustness across different scenarios.
questions: questions: Can you explicitly define the type of game and solution concept used in the proposed mechanism? How does it compare with alternative abstractions in the literature?

How do you address recent critiques on the practicality of truthful mechanisms, that is abundant literature havs found that agents are not telling the true in truthful mechanisms, not even in the one with the strongest solution concept dominant strategies, especially in complex and distributed environments like federated learning?

What are the incentives for the server to implement the sandwich mechanism, given the potential increase in implementation costs? How do you ensure clients trust the server to follow through with the mechanism?

Can you provide more extensive experimental validation, possibly with different datasets and practical scenarios, to demonstrate the robustness and applicability of the proposed mechanism?
limitations: limitations: The paper acknowledges some limitations but does not sufficiently address the practical challenges and potential negative impacts of the proposed mechanism. Specifically:

Server-Side Incentives: The cost and incentive structure for the server to implement the mechanism is not thoroughly explored. This is crucial for practical adoption.

Broader Applicability: The experiments are limited in scope, and further validation with diverse datasets and scenarios is needed.

Trust and Adoption: The paper does not discuss how to build trust among clients that the server will implement the mechanism fairly, which is essential for the success of the proposed approach.
paper_summary: paper_summary: This paper introduces Federated Agent Cost Truthfulness (FACT) to tackle the issues of free riding and dishonesty in federated learning.  A penalty system is proposed to eliminate federated free riding. Meanwhile, a competitive environment is proposed to incentive agents provide truthful information.
strength_weakness: strength_weakness: 1.	Are there any real-world examples that illustrate the necessity of addressing free-riding and dishonesty issues in federated learning?
2.	The paper states that "All rational agents seek to minimize their loss," but a more intuitive understanding is that rational agents seek to maximize their gains. There is a slight discrepancy between these two statements. How can this discrepancy be understood?
3.	Generally speaking, the payoff of a game is determined by the actions of all participants. Why, in Equation 2, can the payoff of agent i be calculated based solely on its own action mi ?
4.	Please elaborate more clearly on the logical relationship between Equation 2 and Equation 3.
5.	It would be better if the intuitive principles behind the formula design could be provided, such as for Equation 4.
6.	In Assumption 1, the assumption of "No Collusion" is made. Is this a simplified assumption, or is it also quite consistent with real-world federated learning scenarios?
7.	What are the limitations of the mechanism proposed in this paper?
questions: questions: What if the individuals are boundedly rational?
limitations: limitations: NA
paper_summary: paper_summary: This paper introduces Federated Agent Cost Truthfulness (FACT), a novel mechanism that addresses the free-rider dilemma in federated learning. FACT ensures truthful reporting by agents and eliminates free-riding through a penalty system, competitive environment, and by encouraging agent participation with better performance than training alone. Empirically, FACT effectively avoids free-riding even when agents are untruthful and significantly reduces agent loss.
strength_weakness: strength_weakness: 1. Eq 10, 26, and 27 lack punctuation.

2. Line 447, 452, 459, 475, 485, 496, and 514 are extra blank lines.

3. It may be better to add one figure in the Introduction to show the proposed issues.

4. The experimental part is not sufficient to thoroughly prove the effectiveness of the FACT mechanism. For example, it is obvious that the introduction of the FACT mechanism will bring additional communication overhead, but this part is completely missing in the experimental part.

5. The experimental part lacks organization. For example, I think the overall accuracy of the model prediction after using the FACT mechanism should be shown first, and then the various parts of FACT should be ablated, such as removing the Sandwich module, Penalized module, etc. Finally, some hyperparameters or other visualization experiments should be shown. At present, I am very confused about the results of the experimental part.
questions: questions: 1. The core contribution of the paper is to prevent free-rider attacks by participants in federated learning when they are able to cheat. And the author claimed FACT is the first truthful Horizontal FL method that eliminates free riding while simultaneously enforcing truthfulness. However, other federated learning algorithms also can prevent free-rider attacks, even though the authors claim that they cannot prevent participants from actively cheating. Wouldn't it be helpful to prove the effectiveness of the FACT if the paper add comparative experimental results of several related algorithms? 

2. Is it a limitation that the parties cannot collude? If the parties collude, can FACT also effectively protect FL?

3. What is the additional computational overhead of FACT compared to traditional FL and local training?

4. According to Figure 2, why does using FACT to protect the FL process cause performance degradation? I noticed that the loss of FACT is higher than the loss of traditional FL.

5. I think the loss is not enough to reflect the performance of the model. When the model is overfitted, the loss increases but the accuracy still decreases. At the very least, the paper should show the accuracy of the final prediction effect (necessary) and add some ablation and hyperparameter experiments (after accept). I suggest you refer to these papers to show your experimental results if you still have the intermediate records of the experimental results.

[1] Fraboni, Yann, et al. Free-rider attacks on model aggregation in federated learning. International Conference on Artificial Intelligence and Statistics. PMLR, 2021.

[2] Lin, Jierui, et al. "Free-riders in federated learning: Attacks and defenses." arXiv preprint arXiv:1911.12560 (2019).
limitations: limitations: Based on the checklist, the author said "We provide the assumptions used for our work, as well as explain them, in Sections 3 and 5." However, I find it difficult to understand what the author wants to express in terms of specific limitations. I suggest writing it clearly at the end of the paper or in a separate section.

Comments: title: Reviewer dpi3 Follow-Up
Reviewer hdyq Follow-Up
Response to Reviewer k1Mz
comment: Thank you again for your insightful review! We wanted to follow-up to ensure that our rebuttal has indeed addressed all concerns listed within your original review. If not, we would be happy to continue discussion before the end of the discussion period tomorrow.
Thank you for your response! We wanted to follow-up to ensure that we have addressed all of your concerns. If not, we would be happy to continue discussion.
Thanks for the author's responses. After reading them and other reviewer's comments, I would like to keep my score.
Thank you for your responsiveness and thoughtful discussion. We are happy to hear that you value our theory, and that our paper warrants acceptance!
Thank you for your detailed explanation. My concerns almost have been solved. Although the experiment part of this paper has some limitation, the theory part is perfect. The logic is clear, there are no unknown symbols, and the relevant formulas have been proved accordingly. I think 6 is a proper score. I will raise my score.
Thank you for your quick reply and continued discussion.

---

> **Question 1:** I am curious about how much additional time FACT will add after training is completed. Although this is not part of the training phase, it still represents an additional time cost.

**Response to Question 1:**

The additional time that FACT adds after training is negligible. Here are the full steps that the server takes post-training:

1. Randomly group agents into threes. 
2. Determine which agent has the middle reported cost within each group (any tiebreakers are randomly decided by the server).
3. Send rewards as prescribed in Equation (10) to the winning agent within each group.

These three steps combined took no longer than five seconds within each of our experiments.

> **Question 2:** Do you mean you didn't have time to complete the ablation experiments? Why not present them in a table?

**Response to Question 2:**

Yes. Due to running time-consuming additional experiments on real world datasets ([shown in our global rebuttal](https://openreview.net/forum?id=JiRGxrqHh0&noteId=w65DtMxzEP)) within a short time period, we were unable to complete the ablation experiments. We are committed to adding these ablation studies within our final version.

> **Question 3:** I think the provided references can adopted as a baseline for comparison.

**Response to Question 3:**

The first provided reference, [1], is a paper detailing free rider attacks. Our paper proposes a novel mechanistic defense to disincentivize agents from free riding. Thus, [1] is not a baseline we can compare against.

The second reference, [2], details both free riding attacks and defenses. The defenses proposed are not mechanisms, they are anomaly detection methods. As such, [2] does not disincentivize agents from free riding. The proposed methods simply allow a server to detect when free riding is occurring (thereby not solving the larger underlying issue that agents are *incentivized* to free ride). **FACT provably dissuades agents from free-riding, even when agents can be untruthful, eliminating the need to have an anomaly detection system.** Thus, [2] is not a baseline we can compare against as it is a detection method and not a mechanism.

> **Question 4:** Do you mean that if the parties do not free-ride, FACT will cause a decline in FL performance?

**Response to Question 4:**

We believe there may be some confusion surrounding FACT's performance. Below, we take this opportunity to clarify the confusion. 

FACT does not cause a decline in FL model accuracy. The resultant model performs as well as the FL method utilized (since FACT does not alter training). However, in order to penalize and eliminate untruthfulness and free riding, FACT requires a contract fee from all participating agents. This fee is regarded as a loss for each agent, as shown in Equations (10) & (11). Fortunately, winning agents recoup a reward much larger than the contract fee (also shown in Equations (10) & (11)). Even with contract fees, agents provably achieve lower loss by participating in FACT than not (*i.e.,* FACT is Individually Rational).
Thank you for your response. I have a few more questions:

1. RW3: I am curious about how much additional time FACT will add after training is completed. Although this is not part of the training phase, it still represents an additional time cost.

2. RW5: Do you mean you didn't have time to complete the ablation experiments? Why not present them in a table?

3. RQ1: I think the provided references can adopted as a baseline for comparison.

4. RQ4: Do you mean that if the parties do not free-ride, FACT will cause a decline in FL performance?