SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Florian E. Dorner
MPI for Intelligent Systems, Tübingen ETH Zurich
florian.dorner@tuebingen.mpg.de
Georgi Pashaliev
Sofia High School of Mathematics
Nikola Konstantinov
INSAIT, Sofia University nikola.konstantinov@insait.ai
Martin Vechev
martin.vechev@inf.ethz.ch
Collaborative learning techniques have the potential to enable training machine learning models that are superior to models trained on a single entity's data. However, in many cases, potential participants in such collaborative schemes are competitors on a downstream task, such as firms that each aim to attract customers by providing the best recommendations. This can incentivize dishonest updates that damage other participants' models, potentially undermining the benefits of collaboration. In this work, we formulate a game that models such interactions and study two learning tasks within this framework: single-round mean estimation and multi-round SGD on strongly-convex objectives. For a natural class of player actions, we show that rational clients are incentivized to strongly manipulate their updates, preventing learning. We then propose mechanisms that incentivize honest communication and ensure learning quality comparable to full cooperation. Lastly, we empirically demonstrate the effectiveness of our incentive scheme on a standard non-convex federated learning benchmark. Our work shows that explicitly modeling the incentives and actions of dishonest clients, rather than assuming them malicious, can enable strong robustness guarantees for collaborative learning.
Recent years have seen an increased interest in designing methods for collaborative learning, where multiple participants contribute data and train a model jointly. The premise is that the participants will then be able to obtain a better model than if they were learning in isolation. Most prominently, federated learning (FL) (Kairouz et al., 2021) provides a method for training models in a distributed manner, allowing data to stay with institutions, while still harvesting the benefits of collaboration.
An underlying premise for the success of collaborative learning schemes is that the participants contribute data (or gradient updates) relevant to the learning task at hand. However, when participants are in competition on some downstream task, they may have an incentive to sabotage other participants' models. For instance, firms that are competing on the same market can often improve their machine learning models by having access to their competitors's data, but at the same time will likely benefit from a gap between the quality of their models and those of other firms.
These conflicting incentives raise a concern that collaborative learning may be vulnerable to strategic updates from participants. Previous work has empirically demonstrated that irrelevant or malicious updates can negatively impact collaborative learning (Tolpegin et al., 2020; Kairouz et al., 2021). In
particular, if a subset of participants is modeled as fully-malicious (Byzantine) agents, that collude in a worst-case manner, it is known that optimal convergence rates contain a leading-order term that is based on the fraction of Byzantine agents and is irreducible as the number of players increases (Yin et al., 2018; Alistarh et al., 2018). This suggests that collaborative learning in the presence of strategic behavior may often not provide asymptotic benefits over learning with one's own data.
Contributions In this work, we study collaborative learning in the presence of strategic behavior by explicitly modeling players' competitive incentives. We consider a game between multiple players that exchange updates via a central server, where players' rewards increase both when they obtain a good model for themselves, and when other players' models perform poorly.
We study two important instantiations of this collaborative learning game: mean estimation with a very general action space and strongly-convex stochastic optimization with attacks that add gradient noise. We show that players are often incentivized to strongly manipulate their estimates, rendering collaborative learning useless. To remedy this, we suggest mechanisms inspired by peer prediction (Miller et al., 2005), that penalize cheating players using side payments. Our results on stochastic optimization rely on a novel recursive bound for the squared norm of differences in SGD-iterates between a clean trajectory and a strategically corrupted trajectory. Meanwhile, we show that side payments can be avoided in the mean estimation case, using a novel communication protocol in which the server sends noisy estimates back to players that are suspected of cheating. Our mechanisms are solely based on observable player behaviour, and recover near-optimal convergence rates at equilibrium. Furthermore, expected payments cancel out when all players are honest, so that players are incentivized to participate in the training, rather than use their own data only, despite the penalties.
Finally, we conduct experiments on the FeMNIST and Twitter datasets from LEAF (Caldas et al., 2018) and demonstrate that our mechanisms can incentivize honesty for realistic non-convex problems
Game theory and collaborative learning Many works that study connections between FL and game theory focus on clients' incentives to participate in the training process at all (see Tu et al. (2021) for a recent survey, and Gradwohl & Tennenholtz (2022) for an analysis of how this relates to competition). Similarly, Karimireddy et al. (2022) study incentives for free-riding, i.e. joining collaborative learning without spending resources to contribute data. In contrast, our setting covers clients that strategically manipulate their updates in order to damage other participants' models.
Another line of work studies FL as a coalitional game theory problem, in which the players need to deal with potential issues of between-client heterogeneity by deciding with whom to collaborate Donahue & Kleinberg (2021a,b). Optimal behavior under data heterogeneity is also studied by Chayti et al. (2021), while Gupta et al. (2022) studies invariant risk minimization games in which FL participants learn predictors invariant across client distributions. In contrast, we study FL as a non-cooperative game and seek to address strategic manipulation, rather than data heterogeneity.
Robustness in federated learning The robustness of FL to noisy or corrupted updates from clients has received substantial recent interest, see Shejwalkar et al. (2022) for a recent overview. One line of work studies federated learning in the context of various data corruption models, e.g. noise or bias towards protected groups (Fang & Ye, 2022; Abay et al., 2020). In contrast, we study strategic manipulation by clients as the source of data corruption.
Clients attacking the training are typically modeled as Byzantine (Blanchard et al., 2017; Yin et al., 2018; Alistarh et al., 2018), adversarially seeking to sabotage training by deviating from the FL protocol in a worst-case manner. The goal of Byzantine-robust learning is to achieve guarantees in that setting. Similar models have been studied in a statistical context, where data is stored at a single location and so communication is not a concern (Qiao & Valiant, 2018; Konstantinov et al., 2020). In contrast to these works, we model manipulation as a consequence of competitive incentives rather than maliciousness and analyze client behavior using game theory (Osborne & Rubinstein, 1994). Rather than focussing on robustness to manipulations, we aim to prevent manipulation altogether.
Peer prediction mechanisms Our mechanisms for inducing honesty are closely related to peer prediction that aims to incentivize honest ratings on online platforms. In their seminal paper,
Miller et al. (2005) suggest paying raters based on how much their rating helps to predict other raters' ratings. While they require a common prior shared by all raters and the mechanism designer, Witkowski & Parkes (2012) extend their results relaxing that assumption. Closely related to our work, Cai et al. (2015) suggest to incentivize crowdworkers to produce accurate labels by paying them more, the better their label gets predicted by a model estimate from other workers' data. Meanwhile, Waggoner & Chen (2014) prove that peer prediction elicits common knowledge, rather than truth from participants. Lastly, while Karger et al. (2021) find that peer prediction can elicit subjective forecasts of similar accuracy as scoring based on the ground truth, Gao et al. (2014) demonstrate that human raters can end up with dishonest strategies despite the existence of honest equilibria.
3 Competitive federated learning
In this section, we present our framework on a high level and explain how it models competitive behaviour in FL. In this generality, however, it is impossible to analyze the problem quantitatively. We therefore define specific instantiations of the framework, which we study for the rest of the paper.
3.1 General framework
Overview Throughout the paper, we assume that there are  $N \geq 2$  players, who each have a private dataset. The players participate in an FL-like procedure, which takes place over multiple rounds and requires them to send messages with information relevant to update a centrally trained model at every step. In our setup, players act strategically and competitive pressures might incentivize them to try to corrupt other players' models. This is done by manipulating updates sent to the central model, while simultaneously keeping track of an unmanipulated private (presumably more accurate) model.
To model the participants' strategic interactions, we need to define a game by specifying their action spaces and rewards. To this end, one needs to specify: their attack strategies, that describe whether and how they will corrupt their messages to the server; their defense strategies, which describes how they postprocess the server's updates to defend themselves against others' manipulations; and their rewards, in a way that reflects the quality of learning and the competition between them. Given these components, we are interested in the Nash equilibria of the corresponding game, in order to understand strategic behavior in FL and how it affects the quality of the players' models.
Formal setup We denote the samples of each player  $i$  by  $x^{i} = \{x_{1}^{i},\ldots ,x_{n}^{i}\}$ , where  $x\in \mathcal{X}$ , and assume that  $\{x_j^i:i\in [N],j\in [n]\}$  are not necessarily independent samples from an unknown distribution  $\mathcal{D}\in \mathcal{P}(\mathcal{X})$ . For simplicity, we assume that all players have an equal number of samples  $n$ . Players communicate via an FL-like protocol, through which a central server model  $\theta^s\in \mathbb{R}^d$  is updated. The intended goal of this procedure is to find a value for  $\theta^s$  that minimizes a loss function  $f_{\mathcal{D}}(\theta)$ . Note that because  $\mathcal{D}$  is unknown to the players, they can benefit from honest collaboration.
The protocol consists of  $T$  rounds and the central model is initialized at some  $\theta_1^s \in \mathbb{R}^d$ . At time  $t = 1,2,\ldots,T$  the server sends the model  $\theta_t^s$  to all participants. Each agent  $i$  is then meant to send an update  $g_t(\theta_t^s,x^i)$  to the server, for some function  $g_t: \mathbb{R}^d \times \mathcal{X}^n \to \mathbb{R}^d$ . For example, in the standard FedSGD setting,  $f_{\mathcal{D}}(\theta) = \mathbb{E}_{x \in \mathcal{D}}[f_x(\theta)]$  for functions  $f_x: \mathbb{R}^d \to \mathbb{R}$  and  $g = \frac{1}{n}\sum_j\nabla f_{x_j^i}(\theta_t^s)$  serves as an estimate of the gradient of  $f$  evaluated at the data of player  $i$ . Each player  $i$  then sends a message  $m_t^i \in \mathbb{R}^d$  to the server (which may or may not be equal to  $g_t(\theta_t^s,x^i)$ ). Finally, the server computes a new model  $\theta_{t+1}^s = \mathrm{Agg}(\theta_t^i,m_t^1,m_t^2,\dots,m_t^N)$ , via an aggregating function  $\mathrm{Agg}: \mathbb{R}^d \times \mathbb{R}^{N\times d} \to \mathbb{R}^d$  (for example, in FedSGD this will be a gradient update computed as  $\bar{m}_t = \frac{1}{N}\sum_{i=1}^{N}m_t^i$ ).
Players' strategies At every step, the players take two decisions: how to attack by sending a manipulated estimate to the server, and how to defend themselves from unreliable estimates when updating their locally tracked model based on information received from the server. Formally, we assume that each player  $i \in [N]$  chooses (potentially randomized) functions  $a_1^i, a_2^i, \ldots, a_T^i$  and  $d_1^i, \ldots, d_T^i$ , that describe their behavior for the attack and defense stages at every time step. These functions are chosen from two respective sets of possible actions  $\mathcal{A}$  and  $\mathcal{D}$ . The tuple of chosen actions  $p^i = (a_1^i, \ldots, a_T^i, d_1^i, \ldots, d_T^i) \in \mathcal{A}^T \times \mathcal{D}^T$  represents the player's global strategy.
In the most general case, the attacks and defenses of each player  $i$  may take into account all information available to the player, throughout the history of the optimization process. At time  $t$  this
include the models  $\theta_1^s,\dots ,\theta_t^s$  received by the server; the local models  $\theta_1^i,\ldots ,\theta_t^i$  the player kept at previous iterations; the attack strategies  $a_1^i,\ldots ,a_t^i$  used up to time  $t$  (e.g. to correct for one's own faulty estimate at time  $t$ ); as well as additional randomness  $\xi_1^i,\dots ,\xi_t^i$  sampled at each round.
Players' rewards Each player aims to obtain a final model  $\theta_{T + 1}^{i}$  that approximately minimizes  $f_{\mathcal{D}}(\theta_{T + 1}^{i})$ . Crucially, their reward also depends on other players' models. Specifically, we assume that each player  $i$  has a reward function  $\mathcal{R}^i:\mathbb{R}^{N\times d}\times \mathcal{P}(\mathcal{X})\to \mathbb{R}$  and receives the reward
$$
r ^ {i} = \mathcal {R} ^ {i} \left(\theta_ {T + 1} ^ {1}, \theta_ {T + 1} ^ {2}, \dots , \theta_ {T + 1} ^ {N}, \mathcal {D}\right). \tag {1}
$$
Note that the messages  $m^i$  sent by players and thus  $\theta_{T+1}^i$  and each player's reward depend not only on players' strategies, but also on the particular realization of their samples  $x^i$ . We thus focus on expected rewards, averaging out the effects of particular realizations of players' samples and randomness in their strategies. We study the vector of expected rewards  $\mathbb{E}(r^1, \ldots, r^N)$  and its dependence on the strategy profile, that is on the distribution of strategies  $p = (p^1, p^2, \ldots, p^N)$  chosen by each player.
Assumptions on players' behaviour To analyze players' behaviour, we make two assumptions, giving rise to a classic game-theoretic setup. The first is that players seek to maximize their expected reward, as defined above, i.e. players are rational. In addition, since the reward of each player depends on the actions of the other players, players account for the actions of the others, which means that their behavior is strategic. A natural solution concept in this context is the Nash equilibrium (Nash, 1951). This describes a strategy profile in which no player can improve their reward by unilaterally changing their strategy. In our case, this classic notion translates to the following definition.
Definition 1. Let  $p = (p^1, p^2, \ldots, p^N) \in \left(\mathcal{P}(\mathcal{A}^T \times \mathcal{D}^T)\right)^N$  be a strategy profile. Then  $p$  is a (mixed) Nash equilibrium if:
$$
\forall p ^ {*} \in \mathcal {P} (\mathcal {A} ^ {T} \times \mathcal {D} ^ {T}) \text {a n d} \forall i \in [ N ]: \mathbb {E} \left(r ^ {i} (p ^ {1}, \dots , p ^ {i}, \dots , p ^ {N})\right) \geq \mathbb {E} \left(r ^ {i} (p ^ {1}, \dots , p ^ {*}, \dots , p ^ {N})\right),
$$
where the expectation is taken with respect to the randomness of the data and the players' strategies.
3.2 Specific instantiations: mean estimation and stochastic gradient descent
We study two specific cases of the game, each modeling a fundamental learning problem. The first is single-round mean estimation (Sections 4 and 5), which correspond to the general setup with  $T = 1$ ,  $f_{\mathcal{D}}(\theta) = \| \theta - \mu \|^2$ , where  $\mu = \mathbb{E}_{X \sim \mathcal{D}}(X)$ , and the updates  $g$  being the sample means of the players. The second is multi-round stochastic optimization of strongly-convex functions  $f_{\mathcal{D}}(\theta)$  via SGD (Section 6), in which case the updates  $g$  are stochastic gradient estimates based on players' data. In the corresponding sections, we define natural rewards that model the competing incentives.
Strategy spaces To describe the attack strategies, in both cases we model attacks that send a noisy update  $g_{t}(\theta_{t}^{s},x^{i}) + \alpha_{t}^{i}\xi_{t}^{i}$ , for normalized zero-mean noise  $\xi_t^i$  and an attack parameter  $\alpha_{t}^{i}\in \mathbb{R}$ , to the server. Up to a certain magnitude of  $\alpha_{t}^{i}$ , these attacks have a natural interpretation as the act of hiding a random subset of a player's data. In addition, the  $\alpha_{t}^{i}$  parameters have a natural interpretation as the of aggressiveness of the player. In the case of mean estimation, we are also able to analyze much more general attack strategies that can adjust  $\alpha_{t}^{i}$  based on the players' samples  $x^{i}$  and additionally allow for adding a directed bias to the communicated messages.
For the defense strategies in the mean estimation case, we consider a defense strategy that corrects the mean estimate received from the server for the player's own manipulation. The player then computes a weighted average of the result and the their local mean. The weighting parameter  $\beta^i$  used then has a natural interpretation as the cautiousness of the player. In the SGD case we directly provide mechanisms that incentivize honesty at the attack stage, making potential defenses redundant.
4 A single-round version of the game: competitive mean estimation
In this section we analyze a single-round version of the game, in which players aim to estimate the mean of a random variable  $X \in \mathcal{P}(\mathbb{R}^d)$ . Specifically, we consider the game defined in Section 3, in the case of  $T = 1$  rounds. Players sample from a distribution  $\mathcal{D} \in \mathcal{P}(\mathbb{R}^d)$  by first independently sampling a random mean  $\mu_i \sim D_\mu$  and "variance"  $\sigma_i^2 \coloneqq \mathbb{E}\|X^i - \mu_i\|^2 \sim D_\sigma$  (this models potential
heterogeneity between clients), and then receiving (conditionally) independent samples from a random variable  $X^i$  with mean  $\mu_i$  and "variance"  $\sigma_i^2$ . We call  $\mu = \mathbb{E}\mu_i$ ,  $\sigma^2 = \mathbb{E}\sigma_i^2$  and  $\sigma_{\star}^{2} = \mathbb{E}\| \mu -\mu_{i}\|^{2}$ . We assume that players do not know the distributions  $D_{\mu}$  and  $D_{\sigma}$ .
Each player wants to estimate the global mean  $\mu$  as well as possible. During the single communication round, the players are meant to communicate the mean of their samples:  $g_{1}(\theta_{1}^{s},x^{i}) = \frac{1}{n}\sum_{j = 1}^{n}x_{j}^{i}$ . Instead, they send messages  $m_{1}^{i}$ . The server aggregates the received messages by averaging them, so that  $\theta_2^s = \frac{1}{N}\sum_{i = 1}^N m_1^i$ . The players then receive the value of  $\theta_2^s$  from the server and use their defense strategy to arrive at a final estimate  $\theta_2^i = d(\theta_2^s,x^i)$  of the mean. For simplicity of notation, we ignore the dependence of all values on the time  $t = 1$  in the rest of this and the next section.
Reward functions To model competitive incentives, the reward of each player needs to increase as their own estimate of the mean becomes better, and as the estimates of other players become worse. Therefore, a natural reward function is:
$$
\mathcal {R} ^ {i} \left(\theta^ {1}, \dots , \theta^ {N}, \mu\right) = \frac {\sum_ {j \neq i} \| \theta^ {j} - \mu \| ^ {2}}{N - 1} - \lambda_ {i} \| \theta^ {i} - \mu \| ^ {2}, \tag {2}
$$
for some  $\lambda_i \geq 0$ . The value of  $\lambda_i$  quantifies to what extent player  $i$  prioritizes the quality of their own estimate over damaging the estimates of the other players.
Attack strategies We assume that players choose what estimates to communicate by deciding how to perturb the empirical mean of their data. Specifically, each player  $i$  selects parameters  $\alpha^i (x^i)\geq 0,b^i (x^i)\in \mathbb{R}^d$  based on their sample, in a potentially randomized manner, and communicates:
$$
m ^ {i} = \bar {x} ^ {i} + \alpha^ {i} \left(x ^ {i}\right) \xi^ {i} + b ^ {i} \left(x ^ {i}\right), \tag {3}
$$
where  $\bar{x}^i = \frac{1}{n}\sum_{j=1}^{n}x_j^i$ ,  $\mathbb{E}[\xi^i] = 0$  and  $\mathbb{E}\|\xi^i\|^2 = 1$ . Here  $\bar{x}^i$  is the standard empirical mean of  $x^i$ , while  $\alpha^i$  represents the magnitude of the noise player  $i$  adds to the estimate and  $b^i(x^i)$  is an additional bias term. Note that the case of  $b^i(x^i) = 0$  recovers the data-hiding attack discussed in Section 3.2.
In order to prevent "non-general" strategies, such as simply setting  $m^i = \mu$ , that cannot be analyzed properly as their success depends on the true parameter  $\mu$ , we assume that players do not base their strategies on guesses about  $\mu$  beyond the information they obtained from  $\bar{x}^i$ . Formally, we assume
$$
\mathbb {E} <   \bar {x} ^ {j} - \mu , b ^ {j} \left(x ^ {j}\right) > = 0. \tag {4}
$$
This prevents  $b^{i}$  from linearly encoding additional knowledge about  $\mu$  and for example holds whenever  $b^{j}(x^{j})$  is independent of the residuals  $\bar{x}^{j} - \mu$ . We also assume that the noise variables  $\xi^{i}$  are independent of each other and all  $x_{j}^{k}$  and  $\alpha^{k}(x^{k})$ , but make no further distributional assumptions about  $\xi^{i}$ . Indeed, all of our theorems will hold regardless of any additional assumptions about  $\xi^{i}$ .
We denote this set of attack strategies as  $\mathcal{A}$ . Each element in  $\mathcal{A}$  can be uniquely identified via the distribution of the noise  $\xi^i$  and the functions  $\alpha^i(x^i)$  and  $b^i(x^i)$ . As  $\alpha = b = 0$  can be interpreted as covering the fully collaborative case, while  $\alpha, b \to \infty$  covers the fully malicious case, the (adaptive) parameters  $\alpha, b$  have a natural interpretation as measures of the aggressiveness of a player.
We also note that these attacks are very general:  $m^i(x^i) - \bar{x}^i$  can always be written as the sum of a deterministic component  $\hat{b}(x^i)$  and zero mean noise  $\hat{\xi}(x^i)$ , such that (4) and the fixed distribution of  $\xi^i$  are the only assumptions separating us from the most general possible set of attacks strategies.
Defense strategies In the defense stage each player uses the received estimate  $\theta^s = \bar{m} = \frac{1}{N}\sum_{i=1}^{N} m^i$  and their local data  $x^i$  to compute a final estimate of the unknown mean. Two extreme approaches for player  $i$  are being fully cautious and using their local mean  $\bar{x}^i$  only, or fully trusting other players and computing the average of all sent updates, corrected for their own manipulation, that is  $\bar{m}^i = \frac{1}{N} (N\theta^s - m^i + \bar{x}^i)$ . We consider defense strategies that take a weighted average of these two extremes: Each player  $i$  chooses a parameter  $\beta^i \in [0,1]$  and constructs a final estimate
$$
\theta^ {i} = (1 - \beta^ {i}) \bar {m} ^ {i} + \beta^ {i} \bar {x} ^ {i}. \tag {5}
$$
Denote the described set of defenses, as  $\mathcal{D}$ . Each element in  $\mathcal{D}$  is uniquely identified via the corresponding parameter  $\beta$  with  $\beta = 0$  and  $\beta = 1$  covering the extreme cases from above. Since  $\beta$  can be used to interpolate between these two extremes, it can be seen as a measure of cautiousness.