Reward Poisoning on Federated Reinforcement Learning
Authors: Evelyn Ma
S. Rasoul Etesami
Praneet Rathi
Venue: Accepted by TMLR ()

Abstract: Federated learning (FL) has become a popular tool for solving traditional Reinforcement Learning (RL) tasks. The multi-agent structure addresses the major concern of data-hungry in traditional RL, while the federated mechanism protects the data privacy of individual agents. Despite the advantage FL brings to RL, Federated Reinforcement Learning (FRL) is inherently susceptible to poisoning, as both FL and RL are vulnerable to such training-time attacks; however, the vulnerability of FRL has not been well-studied before. In this work, we propose a general framework to characterize FRL poisoning as an optimization problem and design a poisoning protocol that can be applied to policy-based FRL. Our framework is versatile, catering to FRL scenarios employing both policy-gradient local RL and actor-critic local RL. In the context of actor-critic configurations, we conduct training for a pair of critics, one private and one public, aimed at maximizing the potency of poisoning. We provably show that our method can strictly hurt the global objective. We verify the effectiveness of our poisoning approach through comprehensive experiments, supported by mainstream RL algorithms, across various RL OpenAI Gym environments covering a wide range of difficulty levels. Within these experiments, we assess our proposed attack by comparing it to various baselines, including standard, poisoned, and robust FRL methods. The results demonstrate the power of the proposed protocol in effectively poisoning FRL systems – It consistently diminishes performance across diverse environments, proving to be more effective than baseline methods. Our work provides new insights into the training-time vulnerability of FL in RL and poses new challenges for designing secure FRL algorithms.

Reward Poisoning on Federated Reinforcement Learning
Evelyn Ma
pingm@illknois.et edu
Deportment of Industrial and Systems Engineering University of Mlinois Urbana-Chann Chanp---n
S. Rasoul Etesami
etesamil @illiniis edu
Department of Industrial and Systems Engineering, Coordinated Science Lab University of Ilinois Urbana- Chana-a-aaaaiinn
Praneet Rathi
prothi3@illinois.. edu
Deportment of Computer Science
University of Iliinis Urbena- Chanpaaghaaaiin
Abstract
Federated learning (FL) has become popular tool for solving traditional Reinforcement Learning (RL) tasks, particularly when individual agents need to collaborate duc to low sample efficiency but are concerned about data privacy. The multi-agent structure addresses the major concern of data-hungry in traditional RI., while the federated mechanism protects the data privacy individual agents. Despite the advantage FL brings to RL, Federated Reinforcement Learning (FRL) is inherently susceptible to poisoning, as both FL and RL vulnerable to such training-time attacks; however, the vulnerability of FRL has not been well-studied before. In this work, we propose general framework to characterize FRL poisoning optimization problem and design poisoning protocol that can be applied to policy-basee based FRL. Our framework is versatile, catering to FRL scenarios employing both policy-gradient local RL and actor-critic local RL. In the context of actor-critic configurations, we conduct training for pair of critics, one private and one public, aimed at maximizing the potency of poisoning. We provably show that our method can strictly hurt the global objective. We verify the effectiveness of our poisoning approach through comprehensive experiments, supported by mainstream RL algorithms, across various RL OpenAI Gym environments covering wide range of difficulty levels. Within these experiments, assess our proposed attack by comparing it to various baselines, including standard, poisoned, and robust FRL methods. The results demonstrate the power of the proposed protocol in effectively poisoning FRL systems - It consistently diminishes performance across diverse environments, proving be more effective than baseline methods. Our work provides new insights into the training-time vulnerability of FL in RL and poses new challenges for designing secure FRL algorithms.
1 Introduction
Reinforcement Learning (R.) has gained popularity recent years as paradigm for solving complex
sequential decision-making problems and has been applied to a wide range of real-world problems, including
game playing Siiverr et al.) 2016 Vinyals et al.) 2019). autonomous driving (Yurtsever al. 2020). network
security (Xiao et al. 2018 design of materials (Govindarajan et al.) 2024 Zhou et al. 2019 Ghugare et al. 2023), circuit design RRoy et al. 2021), and optimization (Chen et al. 2022). In RL, the agent's goal is to learn an optimal policy that maximizes the long-term cumulative rewards, which is done by repeatedly interacting with a stochastic enviromment, taking actions, and receiving feedback in the form of rewards. However, despite the impressive performance of RI. algorithms, they are notoriously known
data-hungry, often suffering from poor sample efficiency (Dulac-Arnold et al. 2021 Schwarzer al.) 2020). One traditional solution to this challenge is Parallel R.. Kretchmar 20022) which adopts multiple parallel RT. agents that.

---

sample data from the environment and share it with central server, as seen in practical implementations such as game-playing MMnih et al. 2016 Berner et al. 2019). However, transferring raw data may not be feasible: on the one hand, it can cause significant communication costs in applications, such as the Internet of Things (IoT) (Wang et al. 2020); On the other hand, it is not suitable for privacy-sensitive industrics, such as clinical decision support (Liu el al. 2020).
Federated Reinforcement Learning (FRL) has been proposed in order to address the limitations of traditional parallel RL, inspired by the recent success of Federated Learning (FL). FRL allows multiple agents to solve the same RL task collaboratively without sharing their sensitive raw data, thus addressing the drawbacks of heavy overhcad and privacy violation in traditional Parallel RL. On the communication side, the communication efficiency of FRL is improved by the ability of FL to perform multiple local model updates during each communication round. On the privacy side, FRL enables data privacy protection by only commuicating model updates, but not raw data, to central server. With advantages of addressing communication efficiency and protecting privacy, FRL practically appealing to a wide range of applications, including IoT (Wang et al. 2020p, autonomous driving (Liang et al., 2023), robotics (Liu el. al. 2019), etc. The success of FRL
applications has spurred theoretical and methodological advancements. For instance, Mnih et al. (016
Min ct al. (2023b a) investigate FRL under asynchronous communication, while Xie & Song (202bb) Zhu &
Gong 2023); Xie & Song (0023a); Mai et al. (2023): Fan et al. (2023) delve into FRL with heterogeneity in
local environments. Khodadadian et al. (2022): ) Zheng et al. 2023) provides insights into linear speed-up
convergence rates, Woo et al. (2023) proposes an algorithm with sample complexity linearly scaling with the
number of agents, and Zhao et al. (2023): Liu al. (2023) analyze FRL with personalized local objectives.
Poisoning in FRL is practical and harmful. The inherent nature of FL and RL amplifies the susceptibility to poisoning (training-time attacks) when combined into FRI. On the RL side, individual RL agents are designed to dynamically learn from raw feedback from the environment, making the learning system vulnerable to data corruption during training time (Zhang et al. 2020 Banihashem et al. 2022). As an example, a chatbot could be misled by a small group of Twitter users, resulting in the generation of misogynistic and racist remarks (Nell 2016). Similarly, recommendation systems can be manipulated by a minimal number of fake clicks, reviews, or comments, leading to products being inaccurately ranked higher. Besides, RL in material design (Govindarajan et al., 2024 Zhou et al. 2019 Ghugare et al. 2023) is susceptible to poisoned material data, and RL in circuit design (Roy et al. 2021 can be compromised by misleading prefix graphs and actions. On the FL. side, the impact of poisoning is exacerbated in FT. frameworks compared to single-agent RL. The lack of transparency in local training within FL naturally exposes the FRL system to adversarial attacks by
malicious agents (Fang et al. 2020 Bagdasaryan et al. 2020 Bhagoji et al.. 2019 So et al., 2020
Although practical and harmful, poisoning in FRI. has yet to be explored. In this work, our goal is to systematically study the vulnerability of FRL systems when facing adversarial attacks that attempt to mislead the trained policy.
In the remainder of this introduction, we discuss the challenges of directly applying prior RL poisoning methods to FRL in Section 1.1 review related work on poisoning in Section 1.2 and summarize our contributions in
Section 1.3
1.1 Challenges of applying prior RL poisoning to FRL
Applying existing RL poisoning techniques to FRL presents several challenges. Many prior methods, such as those by Zhang et al. (2200) and (Rakhsha et al. 2020 rely on unrealistic assumptions, including the attacker having complete knowledge of the MDP enviroument, which is often impractical. Additionally, some approaches, such as TrojDRL (Panagiota et al.) 2020), target RL agents' actions rather than rewarding schemes, making them incompatible with our framework. Furthermore, the effectiveness of certain mechanisms, such as VA2C-P (Sun et al. 2020), diminishes in federated settings due to small local training steps, which lead to frequent reinitialization of the adversarial critic and impaired learning of the poisoned actor's value function. Nevertheless, we have included the federated version of VA2C-P as a baseline, and our experimental results highlight the advantages of our approach compared to this fedorated extension of RL poisoning.
2

---

1.2 Related Work on Poisoning
Here, we provide comprehensive literature comparison regarding poisoning in various contexts of FT., RL, Multi-Agent RL, Distributed RL, and FRI.
Poisoning in FL has been analyzed in different settings. Selected representative studies (Tolpegin et al. 2020
Bhagoji et al. 2019 Fang et al. 2020 Jodayree et al. 2023b strategically poison FL by either compromising the integrity of the training dataset or manipulating submitted local model paramcters. However, these studies are under the context. of Supervised Learning, which is substantially different from RL in the sense of the availability of future data, the knowledge of the dynamics of the environment, etc (Sun et al., 2020). Thus, the existing works on FL poisoning are not directly applicable to FRI. poisoning. In contrast, our work focuses on the poisoning designed specifically for FRL, taking into account the unique characteristies of RL.
Poisoning in RL, referring to committing attacks during the training process (Zhang et al. 2020), can be categorized into two types: weak attacks that only poison data (e.g., rewards and states) and strong attacks that can manipulate actions in addition to data (Panagiota et al. 2020). In this study, we focus on weak attacks, also known as environmental poisoning, as they allow easier access for the attacker and, therefore, are more likely to occur in real-world scenarios. [Rakhsha et al. (2020) formulated optimization frameworks to characterize RL poisoning, but they have limitations, such as requiring knowledge of the underlying MDP and focusing on targeted poisoning. Our proposed framework, however, considers both targeted and untargeted poisoning under the realistic assumption that the attacker does not have access to the MDP dynamics. Sun el al. (220)) designs a vulnerability-wwaee poisoning method for RL. Their algorithm focuses on manipulating the actor model, which cannot be directly applied to FRL as the critic model is also communicated among agents and the server. In contrast, our proposed poisoning mechanism specifically designed for FRL and focuses on manipulating the critic model by training a pair of public and private critics.
Poisoning in Multi-Ageni Reinforcement Learning (MARL) is typically different from poisoning in FRL. Existing literature studying the robustness poisoning of MARL more or less violates the security code in FL. In MARL, multiple agents' actions jointly determine the next state of the environment and the corresponding
rewards for each agent, thus exposing data and environment between agents (Wu et al. 2023 Liu & Tai 2023:
Mohammadi et al. 2023 Li et al. 2023). ) On the contrary, our work strictly adheres to the privacy-oriented settings in FRL, letting the agents independently engage in exploration and data collection solely within their local environments, thus ensuring that no data or environment is exposed to other agents.
Poisoning in Distributed RL, similar poisoning in MARL, can violate the privacy code within the FRL context. Previous studies poisoning or robustness in Distributed RL can expose data to the central server. For instance, Chen et al. (2023) illustrates that local agents are tasked with exploring the environment and collecting data for the central server without conducting any local training. This arrangement grants the server access to all data, thereby compromising the privacy safeguards in FRL. In contrast, our work allows local agents to conduct local training and prohibits data leakage to the central server. We further discuss differences between FRL and Distributed RL in Appendix E.1
Poisoning in FRL is an almost unxxploeed territory, except for very limited existing studies. Fan et al. 2021 Jordan et al. (202 provide fault-tolerance guarantee for FRI. However, their frameworks require the central server to access local environments, which is a form of privacy leakage. In contrast, our work prohibits the central server from accessing any local tasks; Anwar & Raychowdhury/ (0221) restricts local agents from multiple local updates its algorithm, which is not only against the nature of RT. exploration but also severcly expensive in the communication costs applications. In contrast, our poisoning method is uniquely failored to accommodate multiple local steps, aligning with realistic settings of RL scenarios. [Zhang et al. (2022); Jordan et al. (2024) present FRL methods resilient against Byzantine failure. However, Byzantine failure is considered relatively naive and weak compared with real-world attacks that can be meticulously designed. In contrast, our work addresses the gap in studying strategic malicious attacks in FRL.
Finally, our work is also related to the literature of RL in strategic settings such stochastic games. In stochastic games, the RL agents (players) are often selfish and optimize their own objectives, and the goal to see whether their collective behavior results in any equilibrium outcome (Ning & Xic 20224] Altabaa et al.] 2023). However, in our poisoning FRL framework, the goal is to optimize an objective function through central
3

---

server whose computations are poisoned by malicious RL agents. Thus, although a group of distributed agents perform RL in both settings, their objectives and learning tasks are very different.
1.3 Overview and Contributions
We provide an overview of the remaining content in this work as follows. To address the vulnerability of FRL to adversarial attacks, we start by proposing a theoretical framework (Sections 2 and ( that characterizes the problem of environment poisoning in FRL as an optimization problem. We assume the presence of a suspicious agent within the federated system, which can be either malicious client or compromised victim. This high-risk agent is trained by perturbed observations through reward manipulation. However, the attacker does not have prior knowledge of the underlying Markov Decision Process (MDP) of the environment and can only learn it through observations. As mentioned previously, this type of attack is practical and can be easily implemented in real-world scenarios, such as buying an loT device to participate in an FRL system and providing false signals its sensors.
To assoss this risk, design novel poisoning mechanism (Section 4 that targets FRI with policy-based local training. Our protocol is designed to target not only general policy-based methods but also the Actor- Critic setting, wherein the allacker trains a set of both public and privale critics. The privale critic calculates optimized poisoned rewards, while the public critic manipulates the coordinator's 's model during the training process. Notably, our poisoning protocol operates without requiring knowledge of the MDP of the environment and remains consistent with the multiple local steps setting of FRI. Furthermore, we offer a theoretical guarantee for our attack mechanism (Section 5), demonstrating that our approach can lead a substantial drop in the performance of the FRL system. We establish theoretical result that our attack is effective even in the most challenging scenario for the attacker
Our method is evaluated through extensive experiments on OpenGYM environments (Brockman et al., 2016), which represent standard RL tasks across various difficulty levels such as CartPole, InvertedPendulun, LunarLander, Hopper, Walker2d, and HalfCheetah. These experiments employ different FRL backbone models. The findings conclusively illustrate that, through our attack mechanism, a malicious agent can effectively poison the entire FRL system.
Contributions. Our findings highlight the vulnerability of the FRL framework to local poisoning attacks, emphasizing the need for awareness of system risks during training.
Innovative FRL Poisoning Method. Our method addresses the challenges of prior RL. poisoning in two key aspects: (a) Adherence to federated privacy code. Our method strictly avoids accessing the environment's MDP, a code often violated by previous RL poisoning.
(b) Leveraging historical federrated rounds. Our double-critic mechanism leverages historical data. from prior federated rounds, overcoming the reduced poisoning effectiveness of prior RL attacks applied to federated contexts.
Thcoretical Analysis. We provide thcoretical evidence demonstrating the impact of our method in poisoning the system. Our main theoretical findings imply that the extent the objective's decrease scales with the square of the attack budget. Moreover, a larger learning rate and a smaller system size increase the system's 's vulnerability to our attack.
Empirical Validation. We validate the effectiveness of our poisoning protocol through extensive experiments largeling mainstream RL algorithms like VPG and PPO. These experiments encompass OpenGYM environments with varying difficulty levels, as detailed in Section 6 This evaluation includes comparisons with various baseline models and assessments of different (targeted and non- targeted) poisoning types.
In summary, our work indicates that when FL is applied to RL training, the systematic security risk can make FRL suscoptible to poisoning and threats applications. Consequently, the trained policy of FRL may not be entirely reliable and requires a more robust and secure protocol. Our findings highlight the potential risks of FRL under adversarial attacks and inspire future research toward developing more robust algorithms.
4

---

2 Preliminaries and Notations
In this section, we overview some background and notations that are necessary for introducing the concept of poisoning in FRI. We consider single-task FRL, where number of agents work together to achieve common task. As such, all agents are trained on the same MDP. We consider the ubiquitous on-policy training setting (Singh et al. 2000).
MDP and RL. An MDP is a discrete stochastic control process for decision-making (Puterman 1990 that
is defined by a tuple M = (8, P. where S is a state space, A is an action space, r()) : x A + R is a reward function, P()) : S x A x + [0, 1] is a state transition probability function, and y E (0, 1) is a discount factor. Given an MDP, the goal of RL is to find an optimal policy, x(-) - DA: where Ax is the set of all probability distributions over the action space A, which maximizes the expected accumulated discounted reward. As is common in the literature (Agarwal et al. 2021), often represent policy by its parametrization 0 (e.g., tabular parametrization or neural network weight parametrization) During the process, at each step f, the decision maker begins in some state St, selects an action a according to the policy (s,), receives a reward r(s;, and transitions to the next state St41 with probability (($t+1|$,, at). This decision-making and interaction process continues until the MDP terminates.
Federated Reinforcement Learning (FRL). An FRL system consists of n agents and a central server. The agents perform local training for L steps and then send their updated policies to the central server. The server performs aggregation to create central policy, which is then broadcast back to all the agents. This process is repeated for T rounds, and the broadcast policy is used to initialize the next round of local training. More specifically, at each round < T, at the start of local step q S. L, denote the policy of each agent by its policy parameter 00:--1. Following this policy, the agent interacts with its environment, gathoring sequences of states, actions, and rewards: O% = (s)) (i)
P.g.
RBO) = ((s,, .2,, (a1, a a2....); (2,, ())
Based on (4) the local policy is typically updated using 00 arg maxe J(0 0 0%,),, where is some objective function. After completing sleps of local training, all agents update their local policies to (0%) (etn)) which are then submitted to the server Subsequently, the server forms a new global policy using (%) == A099 ((0%) ( where A is an aggregation algorithm. The server broadcasts the updated global policy 0% to all agents, after which each agent updates its local policy as 0pt %% and the system proceeds to the next round p ++1 1. This process repeats until the maximum foderated round T.
(i)
3 Problem Formulation
We propose a theoretical framework to conceptualize the challenge of reward poisoning attacks to FRI as a sequential optimization problem as Problem P Problem P is defined in terms of the notations and concepts introduced in Section 2 such the local policies, global policy, and aggregation algorithms. The remainder of this section is organized as follows. In Section 3.1 we elaborate on the rationale for implementing
local reward poisoning. In Sections 3.2 and 3.3 we provide a detailed explanation of the objective function,
3.4 we present the knowledge possessed by each involved party, ensuring aligmment with established security protocols in FI..
feasible domain, and constraints in Problem (P) Finally, in Section
3.1 Local Reward Poisoning
We consider a threat model targeting FRL systems through reward poisoning. This model assumes the presence of an untrustworthy participant in the system, categorized as either an internal malicious agent a victim contaminated by external attackers (a. common scenario in RL poisoning (Zhang et al. 2020 ). We consider the rewards of the suspicious agents get poisoned during local training, following establishen hed icactices in RI. poisoning (Huang & Zhu, 2019 Zhang et al. 2020
(typical in FL poisoning Tolpegin et al. 2020
Rakhsha et al.) 2021 Banihashem et al. 2022). For clarity, we designate the compromised agent as agent n. At each round and step q, the attacker may manipulate the reward sequence of P.g. transforming it (2)
We distinguish the parameters related the server by index
5

---

into P. attacker.
(a)- Throughout this paper, will use the notation ? to indicate variables that are poisoned by the
3.2 Objective Function and Feasible Domain
Objective Function. In the optimization Problem the objective LA represents the loss of the attacker, which can characterize both unlargeted and largeled poisoning settings. In the case of untargeted poisoning, CA the benefit of the server, typically represented by the long-tern cumulated reward. In the case of targeted poisoning, LA is a policy matrix distance, measuring the difference between the learned policy 0% and some targeted policy 0¹. This captures the attacker's s goal to manipulate the global model to align with a specifically targeted policy.
The objective LA is minimized over manipulated rewards R. which is subject to the constraint that R should closely align with the ground-truth rewards R (Eq. 5). Although R may initially seem to behave as random variable, it essential to note that in Problem we characterize the process such that once R is obtained and observed, it is treated as deterministic variable. Subsequently, the attacker refers to this observed deterministic variable R to optimize its manipulatod rewards R
Feasible Domain.
arg min LA O (RP.9) 1SASL) (n) 1SPST)
(P)
R
s.t.
(t)
0 Vi <
(1)
00. arg max J0, 00 0 100-1) Vi <
(2)
0
P arg max [0, 0 Ö
(3)
(n)
(n)
0 99 (102)
(1) he(m))
(4)
D R P.g R
(8) Se.
(5)
3.3 Constraints
The constraints in optimization
characterize the process of poisoning in FRL, including local initialization
(Eq. D. local train (Eq. 2 allack with limiled budgel (Eq. 5), and global aggregation (Eq. 4 A concise summary is provided in Table and further details are explained below.
Local train. Constraints and 3 outline the local update of each agent at local step q in round In Constraint each clean agent uses its current policy, denoted by 0%--- to roll out an observation sequence 1,--1 Then, based on this observation, the agent updates its policy from 00.9-1 to 0 by maximizing an objective function J()), defined by the agent's specific RL algorithm. Constraint 3 characterizes the update for the suspicious agent's 's update, akin to 2 with the distinction that the local update for suspicious agent is based on the manipulated rewards R.
Attack budget. Constraint 5 captures the budget constraint for the attacker, where D(..)) represents the distance between the perturbed (poisoned) observations and the clean observations, which restricted by a cost budget c. The consideration of the attack budget is crucial. On the one hand, a limited budget constitutes standard assumption in RL reward poisoning Huang & Zhu (2019); Rakhsha et al.) (2021) On the other hand, our framework strives to encompass scenarios of both internal malicious clients and victim clients susceptible to external poisoning From a practical standpoint, an internal malicious agent may seek to avoid detection, necessitating caution in its manipulation, while an external attacker is likely to be concerned about the costs associated with their attacks. Therefore, the inclusion of an attack budget is imperative for both scenarios.
Global aggregation and local initialization. Constraint models the aggregation step of the central server by aggregation algorithm Aaa9 This algorithm processes the models sent from agents, denoted by

---

Table 1: Constraints of Problem P
PARTY
CONSTRAINTS
ACENT (1), inn
.Q.
LOCAL INITIALIZATION
(CLEAN AGENTS)
EQ.
LOCAL TRAINING
AGENT (n)
EQ.
LOCAL INTIIALIZATION
(SUSPICIOUS AGENTS)
EQ.
3
LOCAL TRAINING
EQ.
ATTACK BUDGET
COORDINATOR
EQ.
AGCREGATION
(0%),. inputs and produces the new global model (0) as an output. Constraint stipulates that at the start of local training, all agents initialize their individual policies as the global model.
3.4 Information Structure
We present the knowledge of each party in the FRL system to guarantee strict adherence to privacy protection in FL. Table the knowledge of the three parties involved in FL, namely, the coordinator, the attacker, and the agents, and clarifies the knowledge of each party to guarantee the data privacy expected from FL.
The coordinator only has knowledge of the submitted models and the global policy, while the agents only have knowledge of their local data, policy, and the broadcast global model. The attacker only has knowledge of its own observations, manipulations, model, and the broadcast global policy. This ensures that the agents' private data is kept confidential.
Table 2: Knowledge of the parties in a poisoned FRL.
COORDINATOR AGENT (0.. < a AGENT (xi
Cal-)
J(]
V
OP
OP4
R 1
We refer Appendix A for a specification of this framework the case of Proximal Policy Optimization (Schulman et al. 2017), which provides an illustrative example, demonstrating how our framework characterizes poisoning in FRL for local actor-critic algorithms.
4 Method
In this section, we propose practical local reward poisoning methods for FRL. We consider two scenarios when the local RL training in the FRL system uses actor-critic methods (Algorithm 2 and policy gradient methods (Algorithm 3 We note that our reward poisoning methods can be employed for both targeted and untargeted reward poisoning, as detailed in Section 3.2, by employing the appropriate corresponding objective. As before, we use 7 to denote the suspicious agent's parameters communicated with the server.

---

4.1 Reward Poisoning for Actor-Critic-based FRL
Actor-Critic mechanism. In actor-critic algorithms (Peters & Schaal 2008), we have a policy parameterized by A alongside a corresponding value function Vs(s), determined by policy O and state 8. Besides the policy model, each agent incorporates a critic model (-) parameterized by w, providing estimation of the policy- related value function Vo(s), denoted by V(s) (s).. To simplify notations, by some abuse of notation, we refer to the critic model dw by its paramet rization w in subsequent analysis. The estimated value function ((s) generated by the critic w further gives an estimation of the Q-function as Q(s, a) = r(s, + + V(s'), where r(s, a) is the observed reward, and s' the next observed state. The critic model updates itself by minimizing the ceepoora-ddiffeemn error between its model estimation and ground truth observation (Tesauro et al. 1995). The policy parameter 0 is updated by 0 arg maxg J(0 0', O9), where J(.) is some objective function specified by the actor-critic algorithm, is the exploration step, and Of is the observed trajectory. In the following, we describe the poisoning mechanism in detail and summarize it in Algorithm 2 which uses Algorithm ! as subroutine.
Algorithm Poisoned Local Train for Actor-Critic-baseed FRL
Input: current actor 0 [(n] 00.0-
current private critic PP:4-1, current public critic 0
(n)
2: Output: updated actor 0"4
(n): updated private critic (n) updated public critic [ (n)
3: Agent n uses current actor 0 to obtain ground-truth observation Op
P.-9-1
4: computes true objective (nn using ground-truth observation (n) and private critic w(x)) wp---
5: poisons rewards as R.: using true objective J. and budget e by Eq.
6: obtains poisoned objective (n) with poisoned observation 0 and public critic 23.-1
updates poisoned actor to 0 gog
(n) with poisoned objective Jr...
updates public critic to a with poisoned observation 0 (n)
9: updates private critic to
p-g with true observation P.g (n)
Public and private critics. The term "public critic" implies that the agent uses this critic for communication
with the server (Line 23), while "private critic" signifies that the agent retains this critic solely for local
computation purposes (Line 4.. To manipulate the global model by training (Line S and submitting (Line
23
poisoned public critic, the attacker also undergoes training for an unpoisoned private critic (Line
This approach allows the attacker to possess knowledge of true estimation of the value function (Line to make optimal poisoning decisions (Line 5
We denote the pair of public and private critics respectively as aP:3 and w.g. The private critic w is trained using ground- truth rewards R (1) (Line 9h, and then the attacker harnesses the private critic w th to
obtain poisoned rewards R... (Line R
P... is then utilized to train the public critic w (Line (xx)
80. Due
to the different goals and training methods of the pair of critics, their initialization also differs. At the
beginning of new round of local training, the agent initializes its public critic with the broadeast global
model (i.e., ap.0 w in Line 10), while inherits its private critic from the end of last round of training (i.e., wP0 (m)
L (m)) (m)
1,4 in Line 100.
Reward poisoning and actor update. The attacker aims to minimize the objective function of the policy model (the actor) by poisoning its local rewards. When deciding how to poison the rewards, the attacker should use the unpoisoned objective J given by the ground-truth rewards R and private critic (Line 4 to obtain a true estimation and make a right poisoning decision (Line 5). Concretely, the attacker minimizes the original objective J with respect to R: for each round p < T' and local step q S L, the corrupted agent n interacts with the environment and obtains the ground-truth observation (m)- The attacker computes the unpoisoned objective JP& using true rewards R.4.g and private critic wP. (Line 4. Then, the attacker (x)
8

---

Algorithm Reward Poisoning for Actor-Critic-bsse FRL
Input:
max federated rounds T', max local episodes L, number of agents n, poisoning budget e, aggregation
algorithm A actor-critic objective function J.
Output: server's actor model (%) and critic model w%)
3: Initialize the server's actor model (0) and critic model w (0)
4: for to 7 do
5: for = to n do
if i * then
Agent i initializes local actor and critic:
0%
w
(1)
8: else
Agent initializes local actor p(m) 0p-1
(0)
initializes local privale critic as w.. end if
w and local public critic as 6.D (n) (n)
(n) 6 wpp
10:
(0)
11:
12:
for q = do
13:
if # then
Agent uses local actor 02:9-1 obtain observation 0. (i)
14:
15:
computes objective [() with observation Ogg and local critic w P-2-
(8)
16:
updates local actor to 0p:9 with (i) and updates local critic to w p.g with 0
17:
else
18:
Agent runs Algorithm PPisooed Local Train for Actor Crioo-Critic-bseed FRL). end if
19:
20:
end for
21:
end for
22:
Central server aggregates global actor 0% A
L
P..L an1y (n- (m));
Central server aggregates global critic w = A p.L
SpLL) (n- (m))
23:
(0)
24:
end for
poisons the reward according to
J
(m)
R.g + R.4 - c
(6)
which guarantees that the manipulation power is within the attack budget After the attacker has generated poisoned rewards R. R.g. (Line 50, the actor is updated according to the poisoned objective Iing (Line) 70.
4.2 Reward Poisoning for BaiicyGraadient-aasee FRL
In Policy Gradient (PG) algorithms (Silver al. 2014 agents do not require a critic model. Therefore, have adapted the poisoning method described in Section 4.1 that uses Oetor-Critic for agents' local RL algorithm. The overall procedure is outlined in Algorithm 3 The attacker's goal is still to minimize the objective function / of the policy model. Since there is no critic, the agent directly caleulates J from the observed O (Lines 14 and 15). and uses this information to decide how to poison the rewards to R by Eqq 6
(Line 166. The policy is then updated (Line 18) with the poisoned objective . calculated by the poisoned rewards (Line 17
5 A Theoretical Performance Guarantee
In this section, prove that poisoning method strictly decrease the global objective under certain assumptions. We begin by considering the following assumptions, which is standard for thcoretical analysis
in the existing FRL literature (Jordan el al. 2024 Zhang et al. 2022 Fan el al. 2021

---

Algorithm Reward Poisoning for Policy Gradient-based FRL
max federated rounds T', max local episodes L, number of agents n, poisoning budget e, aggregation algorithm A policy gradient objective function J.
Input:
Output: server's policy O/)--
3: Initialize the server's policy 600)-
for p = to T do for to do
for q = 1 to L do if i then Initialize local policy 0% - 1
7:
8:
Interact with environment and obtain Og- Compute Jeh with Opg
(1)
10:
Update 0 with ja. (6)
11:
(
12:
else
Initialize local policy 0 0 (n) (0)
13:
14:
Interact with enviromment and obtain clean observation 0 0.
Compute clean objective ((o) with the clean observation 0.4
15:
(x)
Poison reward as P.q using the clean objective Jo and the clean observation 0 by Eq. 6
16:
Obtain poisoned objective Jii with poisoned observation 0%
17:
Update local policy (m)) with the poisoned objective Jin end if end for
18:
19:
20:
21:
end for
22: 0% A"99 (OP.L (1)
(2)1... ... P (1- oual)
23:
end for
Assumption (Single-step local training). We assume that for each round, all local agents only apply single-step local training, i.e., L 1.
Clean local reward sequence function. Let us denote the dimensions and r by de and dr, respectively. Given a policy 0, define T(o(-) : Rdo + Rd as the function of the reward sequence generated by the agent interacting with the local environment i under policy O. Under Assumption observed by agent during round p by rl.. Then, we have (0) r()) (O))')
[ denote the reward sequence
Clean local objective function. With slight abuse of notations, let J(0:r) denote the local RL objective lunction given the policy and the reward sequence r, where we have J Rddoxd, - R. According the mechanism of FRL, we further define the objective of local agent given an initialized policy 0 (0) J(0; (() (e)())))))))))))))))))))))))(( (f)). If denote the objective of local agent at the start of round p by Jod we have
Rg -(0 (1 (0 (0%)1)) 2(0%)):r10))
:
Poisoned local reward sequence and local objective. Under Assumption [ agent n is poisoned.
According to our algorithms/ 2 and we have ((n) ((m) -V, so Jay), where E(v) TT]l for any vector
v. Therefore, the poisoned objective is given by J(. = 1O (PP)))
Local policy update function. Under Assumption 1 each agent performs only a one-step local update. Denote the update rate as >g IR. Since only agent n is poisoned, we have
0% =0 1 + Ao V 00 (0)
Vop (m)
10

---

Assumption 2 (Central Aggregation). Suppose the server updates global policy by the conventional FedAVG sellings, e.e 0% = 1 à Lielm] (() for clean braining. In particular, when agent is poisoned, we have 80 -1 0 + (-))
In fact, the aggregation method described in Assumption 2 aligns with common practices found in establishod FRL literature (Xie & Song 2023b: Zhu & Gong 2023
Global objective function. Under the conventional setting of FL, given model 0, the global objective is defined as (o) (0) == 2.
x -== (1) (MeMahan et al.) 2017 In FRI, at the end of round p, the clean
federated objective is given by Roy = ( When agent is poisoned, we have (o) J(o (o)).
Assumption 3 (Objective smoothness). We assume that Ro is differentiable with respect to r and 0 almost everywhere, and Ro is Lm-shhoth with respect to ((n)
Assumption 3 is well-founded, aligning with prevalent local RL objectives. We provide two illustrative examples: a) Discounted Cumulaatve Reward (Kaelbling et al. 1996): In this case, we have Jg : F
is the discount vector. Here, y (0, 1) is discount factor, and d, represents the cardinality of (c)- b) Average Reward (Kaelbling et al. 1996 In this case, the local objective is expressed
y: where 7 = (, 10,,1" 22 ,..., )dx) ;..,;))))
P/ == (1.r where di denotes the cardinality of a and I (1, 1,., 1) E Rd, stands for all-one vector. We note that the primary purpose of Assumption 3 is facilitate our theoretical analysis. However, in our numerical experiments (Section 6.1), we demonstrate the effectiveness of our proposed attack even if this assumption fails to hold (e.g,, when the Siales/actions/eeee rewards are discrete).
Assumption 4 (Worst-Case). We assume the allacker is only able to allack in the latest round.
Remark 5. Assumption 4 is designed to encapsulate the most challenging scenario for the allacker, where manipulation restricted solely to the latest round. The proven guarantee under this assumption serves as indicator of the effectiveness of our attack method in more general and lenient scenarios, wherein the allacker has the abilily to poison the system over mullipple rounds. Furthermore, our experiments showcase. the effectiveness of the attack when deployed over multiple rounds (refer to Section 6).
We are ready to state our main theoretical result, whose proof is deferred to Appendix A.
Thcorem 6. Let Assumptions 1 2
and hold. Suppose that all agents are updated cleanly at the first p 1 rounds, and at round p, agent is poisoned. Let L+ nh,
2d where B is scalar defined
0=0
B = (o -Vrr) ((V,J(,))
0"
Then, for B > and < 6+, we have Pio S Joy a, where a E (0,
Remark 7. Theorem 0 asserts that the gap a strictly positive, and its upper bound increases proportionally with è4, which is the square of the upper limit of the attack budget. With small poison budget c, con guarantee that the poisoned global objective J% strictly smaller than the clean global objective J and a higher allack budgel can indicale a greater decrease in the global objective. Similarly, a larger local learning rate 0e or fewer agents can increase c+, indicating a stronger objective decrease a. In Appendir A we not only eaplore a practical scenario ensuring B but also hightight that a higher learning rate Xg enhances the likelihood of B >>
6 Numerical Experiments
In this section, we conduct a series experiments to evaluate the effectiveness our poisoning method on the FRL system. Our results show that the proposed poisoning method can effectively reduce the mean episode reward of the server in the FRL system. Additionally, our poisoning protocol does not require knowledge of the MDP of the environment and is consistent. with the multiple local steps setting of FRL.
11

---

6.1 Environments
Nature of FRL: Following existing FRL literatures Jordan al. 2024 Zhang al. 2022 we implement experiments on various OpenAl Gym environments [Brockman al, 2016] with increasing complexity, including CartPole, Inverted Pendulum, Hopper, Lunar Lander, and Half Cheetah, all of which are standard RL task enviromments. The selection of these datasets reflects the inherent nature of FRL, where the focus is on solving RL tasks, and FT. serves as a versatile toolbox.
It worth noting that envirouments such as Inverled Pendulum, Hopper, Half Cheelah Teature continuous reward spaces, aligning with our theoretical analysis (Assumption Additionally, we incorporate environments such Cartpole and Lanar Landar, providing discrete reward spaces, to demonstrate the applicability of our attack in diverse scenarios of reward space.
6.2 Backbone Model
We explain backbone framework by describing both its local RL model and its global FL mechanism.
Local RL models. Our local training backbones cover both the Policy Gradient method (corresponding to Algorithm 3) and the Actor-Critic method (corresponding to Algorithm 2) maintaining consistency with our attack methods in Section
4 For the Policy Gradient backbone, we opt for the conventional Vanilla Policy Gradient (VPG) (Sutton et al. 1999). For the Actor-Critee Criti backbone, we choose a widely applicable model, Proximal Policy Optimization (PPO) (Schulman et al.J 2017). In Appendix E.2 we discuss broader RL backbones and justify that VPG and PPO maintain accessibility and practicality while delivering competitive performance.
Federated mechanism. We adhere to a prevalent practice observed in existing FRL literature (ie & Song, 2023b Zhu & Gong] 2023), where the central server aggregates the models submitted by local agents by taking the average at the end each communication round and then broadcasts the new global model, which is used by the local agents as initialization for the next. round of local training. This federated mechanism aligns with the settings in our theoretical analysis (Assumption 2
6.3 Baselines
We evaluate our methods against diverse baselines, including standard FRL, poisoned FRI, and robust FRL.
Standard FRL. In this setting, agents undergo regular training without any poisoning, adhering strietly to the conventional rules of local updates and federated communications. The baseline comprises both VPG-based standard FRI. (Algorithm 5) and PPO-based standard FRI. (Algorithm) 4j. We refer to Appendix C for a detailed description of those algorithms.
Poisoned FRL. To the best of our knowledge, no existing FRL method designed for poisoning matches the criteria of a standard and reasonable FRI. setting, encompassing multiple local steps and data security code (refer to Section [.2]. Therefore, we establish baselines for poisoned FRL: 1) the federated version of prior RL poisoning methods designed for single-agent systems, and 2) a randomized attack. Further details are provided below:
1. Federated version of prior RL poisoning. As one of our baselines, we include the federated extension of VA2C-P (Snn et al. 2020). prior RL poisoning method. This choice is due to VA2C-P's compatibility with our reward-manipulaa the attacker does not have access to the environment's MDP. In contrast, other prior RL poisoning methods either present compatibility issues or violate this knowledge limitation, as discussed in Section 1.1 the challenges of applying prior RL poisoning to FRL.
scheme and its adherence to the knowledge limitation that
2. Randomized Poisoning. In this attack, Eq.
G is replaced by
R
p.g ()
R P-g E X,
(7)
(j)
12

---

where is vector with the same cardinality as R, denoted as |x] |B Each index-wise value x follows a uniform distribution, such that x= (x1 x X where x; ~ UO,,1) 1) for all I [R]. Here, U (a, b) represents the uniform distribution in the range from to b. The detailed algorithm is provided in Appendix
C.2
Robust FRL. To our knowledge, no existing robust FRL method aligns with the criteria of a standard and reasonable FRL sctting (refer to Scction 1.2p. Therefore, we adopt a standard robust mechanism inspired by FL, where the aggregation executed by re-weighting with the clients' reliability (Fu et al. 2019 Tahmasebian
et al.
2022
Wan & Chen 2021 The detailed defense algorithm is outlined in Algorithm
6.4 Experimental Settings
Evaluation matrix. The evaluation metrics can vary based on the type of poisoning. For untaryeted poisoning, we evaluate the performance of these methods by measuring the mean-episode reward of the central model, which is calculated based on 100 test episodes at the end of each federated round. For largeled poisoning, we measure the similarity between learned policy and targeted policy. Moreover, for discrete action space, we calculate the proportion of target actions among all actions, while for continuous action space, we collect 1-scaled distance. Under both measurements, higher value indicates a closer learned policy to the target policy. By default, our assumed poisoning type is untargeted unless stated otherwise.
Attack Budget. We have two configurations for the attack budget: a) Small Budgel. We set 1 to simulate a small-budget scenario where the attacker is cautious and cost-conssious. We choose e = 1 as it represents the maximum possible value gained by one action move in the CartPole and Inverted Pendulum environments. is important to note that e = is generally much smaller than the maximum possible reward per action in the other environments (i.e., 100 for Lunar Lander). b) Large Budget. For a large-budget. scenario where the attacker is bold and almost indifferent to costs, we use e 100. In instances without specific notation, refer to small-budget attack.
Hyper-parameters. For both VPG and PPO settings, we let the malicious agent attack the reward with a budget of = 1. There are 200 total communication rounds, and all agents run 5 local steps in each commuicatiin round. The number of poisoned agents is set to one it is not specifically mentioned. For all experiments, we average the results over random seeds ?
6.5 Empirical Results
We present experimental results that demonstrate the effectiveness of our proposed poisoning method. We compare its performance with various baselines outlined in Section
6.3 including standard FRL (Section 6.5.1)
poisoned FRL (Section 6.5.2), and robust FRL (Section 6.5.2 The Experiments cover diverse backbone models (Section (.2) and environments (Section 6.1 consistently demonstrating superior performance. In addition to evaluating the effectiveness of untargeted poisoning, we further validate our method through targeted poisoning (Section (6.5.)). Furthermore, we demonstrate the adaptability of our approach in scenarios involving multiple suspicious agents (Section 6.5.3). The results affirm that our method is both practical and detrimental for real-world applications. To enhance clarity, we have smoothed the plots by preserving values from the first 5 rounds and subsequently applying moving average with a window size of 10 rounds.
6.5.1 Comparison with Standard FRL
We conducted experiments to poison standard VPG- FRL (Fig. 1 and PPO-based FRL (Fig. 2). The backbones for these experiments were constructed as outlined in Section 6.2 utilizing Algorithms and
We see that across scveral environments, single attacker successfully poisons a system of up to to 4 agents using our method. We only present the results of systems with maximum sizes that our method can poison. For instance, in the first plot of Fig. the system size is four, indicating that our method can poison VPG-based FRL with
Poison standard VPG-based FRL. We present our results Fig.
We refer to Appendix D.1.1 for additional settings.
13

---

CartPole-so
Hompeer222
PowettePendullm--
Walker
Hean Reward Per Epcood
Mean Newordd hor Eskocce
Moan Nawward per tasksoea
Moatt Raoward her Esskssca
the
Ceen
Ceenn Rangom PIFEDI
Oean
Clean
RanSom WAC PIFED
Random VAZC FIFEDI
Random
MUZC FIFED)
Ours
Oura
Communccation Bounds
Communccation Bounds
Communcaiion Rounds
Communicaioo Bounds
Figure 1: Poison VPG-based FRL. The red line, labeled poison, shows the performance of our adversarial attack, while the green line, labeled clean, represents the performance of the standard FRL system. The blue line, labeled rand, denotes a random poisoning basclinc. The yellow line, labeled VA2C-P(Fed), illustrates the federated version of VA2C-P, existing poisoning method for single-agent RL. We specifically highlight the results for the largest system sizes that our method can successfully attack: 4 for CartPole and Hopper, and 3 for Inverted Pendulum and Walker.
Co
Hopper
InvertosPendllum-22
Nean Reward Per Epliode
Moaan Rowardd tppiscde
Mitar Rewword Phe tyshecdee
Clean
Claa
Clean
Dean
Random
Randon
Random
Fandor
WaCAFFE))
Out
Ours
Ours
Communication Rounds
Communication Rounds
Communication Rounds
Communicarion Rounds
Figure 2: Poison PPO-based FRL. The annotation is the same as Fig. [ We specifically showcase results for the largest system size that our mothod can successfully attack. The maximum system size is 3 for Inverted Pendulum and for the others.
the number of agents equaling 1,2,3.4. However, our method would fail when the system size is 5. Across environments of varying difficulty levels, we consistently observe a significant performance gap between clean training (the green line) and training under our poisoning attack (the red linc) as the number of training rounds increases. This proves the capability of our VPG attack method in poisoning federated systems.
Poison standard PPO-based FRL. Similar to the VPG-based FRL case, report those systems with maximum possible size that method poison. Attacking a PPO-based FRL system (Fig. 2 lends itself to a much different dynamic because the separate value parameters allow all agents to learn more effectively, and thus, we expect both clean and malicious agents to be more successful in their opposite goals. We observe a consistent performance gap emerging more quickly, with fewer federated rounds, compared to the VPG-based FRL system (Fig D] This indicates that our method is particularly effective in PPO-based FRL systems, benefiting from the double-critic protocol specifically designed for actor-critic backbones.
6.5.2 Comparison with Poisoned FRL and Robust FRL
Poisoned FRL. We conduct experiments on random attacks against standard VPG-basod FRI and PPO- based FRL. We outline two poisoning baselines, a federated version of prior RL poisoning method (VA2C-P) and randomized attack in Section 6.3 with detailed algorithms provided in Section C.2 For VPG-based FRL, we observe that our method (Fig. I the red line) significantly reduces rewards of various environments compared to both the federated version of VA2C-P poisoning (Fig. | the yellow line) and the random attack (Fig. blue line). For PPO-based FRL, both the federated version of VA2C-P (Fig. 2 the yellow line) and the random attack (Fig. 2 the blue line) perform very poorly in poisoning, as they are overwhelmed by the strength of the clean agents, leading to rewards that remain close to clean training (Fig. [ the green linc) over time, considering performance variance. In contrast, our poisoning (Fig. 2 the red line) significantly harms the system performance, outperforming the poisoning baselines.
14

---

Experiments are conducted following settings in Section 6.4 and D.1.1 Results are presented in Fig. 3 Tt is noteworthy that, although the defense mechanism can successfully mitigate the harm of our poisoning in relatively simple environments (CartPole, Hopper, Walker2d), for complicated environments (Half Cheetah), which are closer to real-world scenarios, the defense mechanism fails against our poisoning. Thus, our results consolidate that this kind of poisoning harmful in real-world applications.
Robust FRL. The selection of the robust FRL baseline is detailed in Section
6.3
CartPole
Hopppe
Waiber2d
HaitCheetah
Moan Raavard her Estood
Clesn
219
Delerse
30
303
200
-30
-23
Communication Rounds
Communication Rounds
Communcction Rounds
Communisttion Rounds
Figure Poison Robust FRL. The red dashed line, labeled Poison, depicts the performance of our adversarial attack in standard FRL system. The green dashed line, labeled as Clean, represents the standard FRL system's 's performance. The blue dashed line, labeled Defense, showcases the performance of our attack in a robust FRL system.
6.5.3 Additional Results
All additional results deferred to Appendix D.2 including:
(a) Multiple Poisoned Agents. Fig 5 illustrraes the method's applicability scenarios with multiple
poisoned agents. The attack performance remains stable, given a consistent ratio of poisoned agents to unpoisoned agents.
(b) Targeted Poisoning. Fig demonstrates the effectiveness of our proposed method in targeted poisoning secnarios.
(c) High-Budget Poisoning. Fig showcases that, with a sufficiently high attack budget, our proposed method empowers single poisoned agent to impact an FRL system considerable size (e.g., a system 100 agents).
7 Broader Impact
The development of our poisoning method for FRL has significant implications for both rescarch and practical applications. By highlighting the vulnerabilities inherent in FRL franeworks, our work raises critical security concerns in sensitive domains such as healthcare and finance, while guiding the design of robust policies to mitigate these risks. Additionally, our research paves the way for further exploration of adversarial poisoning and defensive strategies in FRL, emphasizing the need for resilient algorithms capable of withstanding malicious manipulation during the training process.
Moreover, our findings initiate important ethical discussions about the responsible use of federated learning technologies. As these systems become more integrated into critical reinforcement learning applications, it is essential to establish guidelines that ensure their deployment is both secure and beneficial. By highlighting potential poisoning risks, we encourage the adoption of sccurity measures that not only protect data privacy but also enhance public Irust in FRL systems.
8 Conclusions
In this work, we have proposed novel method for poisoning FRL under both general policy- and actor-critic algorithms, which provably decrease the system's global objective. Our method is evaluated
15

---

through extensive experiments various OpenGYM environments using popular RL models, and the results demonstrate that our method is effective in poisoning FRI. systems. Our work highlights the potential risks of FRL and inspires future research to design more robust algorithms to protect FRL against poisoning attacks.
Acknowledgments
This work is supported by the Air Force Office of Scientific Research under award number FA9550-23-1-0107 and the NSF CAREER Award under Grant No. EPCN-1944408.
References
Alekh Agarwal, Sham M Kakade, Jason Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Mach. Learn. Res., 22(98):1-76, 2021.
Awni Altabaa, Bora Yongacoglu, and Serdar Yüksel. Decentralized multi-agent reinforcement learning for continuous-space stochastic games. In 2023 American Control Conference (ACC), pp. 72-77. IEEE, 2023.
Aqeel Anwar and Arijit Raychowdhury. Multi-task federated reinforcement learning with adversaries. arXiv preprint orXiv:2103.00 78, 2021.
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics, pp. 2938 2948. PMLR, 2020.
Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning Proceedings 1995, pp. 30-37. Elsevier, 1995.
Kiarash Banihashem, Adish Singla, Jiarui Gan, and Goran Radanovic. Admissible policy teaching through reward design. In Proceedings of the AAAI Conference Artificial Intelligence, volune 36, pp. 6037 6045, 2022.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dkebiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement. learning. arx preprint arXiv:1912.000 06680, 2019.
Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Analyzing federated learning through an adversarial lens. In International Conference on Machine Learning, pp. 634-643. PMLR, 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint ar NTT::1666.0154000 2016.
Tianlong Chen, Xiaohan Chen, Wuyang Chen, Howard Heaton, Jialin Liu, Zhangyang Wang, and Wotao Yin. Learning to optimize: A primer and a benchmark. Journal of Machine Learning Research, 23(189):1 59, 2022.
Yiding Chen, Xuezhou Zhang, Kaiqing Zhang, Mengdi Wang, and Xiaojin Zhu. Byzantinc-robust online and oflline distributed reinforcement learning. In International Conference on Arlificial Intelligence and Statistics, pp. 3230 3269. PMLR, 2023.
Gabriel Aulac-Arnold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 110(9):2419 2468, 2021.
Flint Xiaofeng Fan, Yining Ma, Zhongxiang Dai, Cheston Tan, Bryan Kian Hsiang Low, and Roger Watten- hofer. Fedhql: Federated heterogeneous q-learning. arXiv preprint Xii:22301.1115, 2023.
Xiaofeng Fan, Yining Ma, Zhongxiang Dai, Wei Jing, Cheston Tan, and Bryan Kian Hsiang Low. Fault- tolerant federated reinforcement learning with theoretical guarantee. Advances in Neural Information Processing Systems, 34, 2021.
16

---

Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Gong. Local model poisoning attacks to {Byzantine- Robust} federated learning. In 29th USENIX Securily Symposium (USENIX Securily 20), pp. 1605-16000000 1622, 2020.
Shuhao Fu, Chulin Xie, Bo Li, and Qifeng Chen, Attack-resistant federated learning with residual-based reweighting. arXiv preprint WrXi::112.111422 2019.
Raj Ghugare, Santiago Miret, Adriana Hugessen, Mariano Phielipp, and Glen Berseth. Searching for high-value molecules using reinforcement learning and transformers. arXiv preprint arXiv:2310.0220 02900, 2023.
Prashant Govindarajan, Santiago Miret, Jarrid Rector-Brooks, Mariano Phielipp, Janarthanan Rajendran, and Sarath Chandar. Learning conditional policies for crystal design using offline reinforcement learning. Digital Discovery, 3(4)7669 785, 2024.
Yunhan Huang and Quanyan Zhu. Deceptive reinforcement learning under adversarial manipulations on cost signals. In Decision and Game Theory for Security: 10th International Conference, GameSec 2019, Stockholm, Sweden, October 30 30-November 1, 2019, Proceedings 10, pp. 217-237. Springer, 2019.
Mahdee Jodayrce, Wenbo He, and Dr. Ryszard Janicki. Preventing image data poisoning attacks in federated machine learning by an encrypted verification key. Procedia Compuler Science, 225:2723 2732, 2023a. ISSN 1877-0509, doi: https://doi.orgg /ttp:///ioi.rrr/11111001//////// 10.1016/j http://jpposs.222 procs.2023..0 20.2..0..... URL https: //www..ccieceeirecctt 27th International Conference on Knowledge Based and Intelligent Information and Engineering Sytems (KES 2023).
Mahdee Jodayree, Wenbo He, and Ryszard Janicki. Preventing text data poisoning attacks in federated machine learning by an encrypted verification key. In Andrea Campagner, Oliver Urs Lenz, Shuyin Xia, Dominik Slęzak, Jaroslaw Was, and JingTao Yao (eds.), Rough Sets, pp. 612 626, Cham, 2023b. Springer Nature Switzerland. ISBN 978-3-031-50959-9.
Philip Jordan, Florian Grötschla, Flint Xiaofeng Fan, and Roger Waltenhofer. Decentralized federated policy gradient with byzantine fault-tolerance and provably fast convergence. arXiv preprint arXiv:441.003489 2024.
Leslie Pack Kaelbling, Michael I, Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237-285, 1996.
Sajad Khodadadian, Pranay Sharma, Gauri Joshi, and Siva Theja Maguluri. Federatod reinforcement learning: Linear speedup under markovian sampling. In International Conference on Machine Learning, pp. 10997-11057. PMLR, 2022.
R. Matthew Kretchmar. Parallel reinforcement learning. In The 6th World Conference on Systemics, Cybernetics, and Informatics. Citeseer, 2002.
Simin Li, Jun Guo, Jingqiao Xiu, Xini Yu, Jiakai Wang, Aishan Liu, Yaodong Yang, and Xianglong Liu. Byzantinc robust coopcrative multi-agent reinforcement learning as bayesian game. arXiv preprint 2rxiv:2305. 100005...855 12872, 2023.
Xinle Liang, Yang Liu, Tianjian Chen, Ming Liu, and Qiang Yang. Federated transfer reinforcement learning for autonomous driving. In Federated and Transfer Learning, pp- 357-371. Springer, 2023.
Boyi Liu, Lujia Wang, and Ming Liu. Lifelong lederated reinforcement. learning: a. learning architecture for navigation in cloud robotic systems. IEEE Robotics and Automation Letters, 4(4):4555 4562, 2019.
Guanlin Liu and Lifeng Lai. Efficient adversarial attacks on online multi-agent reinforcement learning, ar preprint ar xrx::22377.07677 2023.
Siqi Liu, Kay Choong Soe, Kee Yuan Ngiam, Leo Anthony Coli, Xingzhi Sun, Mengling Feng, et al. Reinforcement learning for clinical decision support in critical care: comprehensive review. Journal of medical Internet. research, 22(7):e18477, 2020.
17

---

Wentao Liu, Xiaolong Xu, Jintao Wu, and Jielin Jiang. Federated meta reinforcement learning for personalized tasks. Tsinghua Science and Technology, 29(3):911-99) 926, 2023.
Weiming Mai, Jiangchao Yao, Gong Chen, Ya Zhang, Yiu-Ming Cheung, and Bo Han. Server-client collaborative distillation for federated reinforcement learning. ACM Transactions on Knowledge Discovery from Data, 18(1):1 22, 2023.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication- efficient. learning of deep networks from decentralized data. In Arlificial intelligence and slatistics, pp- PMLR, 2017.
Yifei Min, Jiafan He, Tianhao Wang, and Quanquan Gu. Cooperative multi-agent reinforcement learning: asynchronous communication and linear function approximation. In International Conference Machine Learning, pp. 24785-24811. PMLR, 2023a.
Yifei Min, Jiafan He, Tianhao Wang, and Quanquan Gu. Multi-agent reinforcement learning: Asynchronous communication and linear function approximation. orXiv preprint 2xXf:23355004400 2023b.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukeuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928 1937. PMLR, 2016.
Mohammad Mohammadi, Jonathan Nöther, Debmalya Mandal, Adish Singla, and Goran Radanovie. Implicit poisoning allacks in two-agent. reinforcement learning: Adversarial policies for training-time allacks. arxiv preprint Xrx:22300 13851, 2023.
Gina Neff. Talking to bots: Symbiotic agency and the case of tay. International Journal of Communication, 2016.
Zepeng Ning and Lihua Xic. A survey on multi-agent reinforcement learning and its application. Journal of Aulomation and Intelligence, 2024. ISSN 2949-8554. doi: https://bbi.orr//0.10000
100./ooog///1111111111 10116/......2222.2..000.. URL com coom/science/ean htt.:///ww.................aa///
Kiourti Panagiota, Wardega Kacper, Susmit Jha, and Li Wenchao. Trojdrl: Trojan attacks on deep reinforcement learning agents. in proc. 57th acm/ieee design automation conference (dac), 2020, march 2020. In Proc. 57th ACM/IEEE Design Automation Conference (DAC), 2020, 2020.
Jan Peters and Stefan Schaal. Natural actor-critic. Neurocomputing, 71(7-9):1180 1190, 2008.
Martin Puterman. Markov decision processcs. Handbooks in operations research and management science,
2:331 434, 1990.
Amin Rakhsha, Goran Radanovic, Rati Devidze, Xiaojin Zhu, and Adish Singla. Policy teaching via environment poisoning: Training-time adversarial attacks against. reinforcement learning. In International Conference on Machine Learning, pp. 7974 7984. PMLR, 2020.
Amin Rakhsha, Xuezhon Zhang, Xiaojin Zhu, and Adish Singla. Reward poisoning in reinforcement learning: Attacks against unknown learners in unknown environments. ar preprint 2Xiu:2102.08492 2021.
Rajarshi Roy, Jonathan Raiman, Neel Kant, Tyas Elkin, Robert Kirby, Michael Siu, Stuart Oberman, Saad Godil, and Bryan Catanzaro. Prefixrl: Optimization of parallel prefix circuits using deep reinforcement learning. In 2021 58th ACM/TEEF Design Automation Conference (DAC), pp. 853-858. TEEE, 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. aixiv preprint 707.00777 2017.
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with peedicredic representations. arXiv preprint xxiv:200770502. 2020.
18

---

David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference machine learning, pp. 387-395. PMLR, 2014.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Toannis Antonoglou, Veda Pannecrshelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):454-489, 2016.
Satinder Singh, Tommi Jaakkola, Michael L Littman, and Csaba Szepesvári. Convergence results for single-step on-policy reinforcement-leannne algorithms. Machine learning, 38(3):287-32) -08, 2000.
Jinhyun So, Başak Güler, and A Salman Avestimehr. Byziintne-resilient secure federated learning. IEEE
Journal on Selected Areas in Communications, 39(7):2168-218 2020.
Yanchao Sun, Da Huo, and Furong Huang. awwe poisoning mechanism for online rl with unknown dynamics. arXiv preprint arXiv:2009.00774, 2020.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement. learning with function approximation. Advances in neural information processing systems, 12, 1999.
Farnaz Tahmasebian, Jian Lou, and Li Xiong. Robustfed: a truth inference approach for robust feder- ated learning. In Proceedings of the 31st ACM International Conference on Information & Knowledge Monagement, pp. 1868 1877, 2022.
Gerald Tesauro et al. Temporal difference learning and td-gammon. Communications of the ACM, 38(3):
58 58-68, 1995.
Vale Tolpegin, Stacey Truex Mehmet Emre Gursoy, and Ling Liu. Data poisoning attacks against federated learning systems. In European Symposium on Research in Computer Security, pp. 480 501. Springer, 2020.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaöl Mathicu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent. reinforcement learning. Nature, 575(7782):350 354-3544---0--- 2019.
Ching Pui Wan and Qifeng Chen. Robust federated learning with attack-adaptive aggregation. arXie preprint w:Xi:2202.052571 2021.
Xiaofei Wang, Chenyang Wang, Xiuhna Li, Victor CM Loung, and Tarik Taleb. Foderated deep reinforcement learning for internet of things with decentralized cooperative edge caching. IEEE Internet of Things Journal, 7(10):9441-9455) 2020.
Jiin Woo, Gauri Joshi, and Yuejie Chi. The blessing of heterogeneity in federated q-learning: Linear speedup and beyond. arXiv preprint arXiv:2305. 10697, 2023.
Young Wu, Jeremy McMahan, Xiaojin Zhu, and Qiaomin Xie. Reward poisoning attacks on offline multi-agent reinforcement learning. In Proceedings of the aaai conference artificial intelligence, volume 37, pp. 2023.
Liang Xiao, Xiaoyue Wan, Canhuang Dai, Xiaojiang Du, Xiang Chen, and Mohsen Guizani. Security in mobile edge caching with reinforcement learning. IEEE Wireless Communications, 25 122, 2018.
Zhijie Xie and SH Song. Client selection for federated policy optimization with environment heterogeneity. arXiv preprint arXi:2200. 10978, 2023a.
Zhijie Xie and Shenghui Song. Fedkl: Tackling data heterogeneity in federated reinforcement learning by
penalizing kl divergence. IEEE Journal on Selected Areas in Communications, (142):227212 2023b.
Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and Kazuya Takeda. A survey of autonomous driving: Common practices and emerging technologies. IEEE access, 8:58443-588 58469, 2020.
19

---

Mingyue Zhang, Zhi Jin, Jian Hou, and Renwei Luo. Resilient mechanism against byzantine failure for distributed deep reinforcement learning. In 2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE), pp. 378 389. IEEE, 2022.
Xuezhou Zhang, Yuzhe Ma, Adish Singla, and Xiaojin Zhu. Adaptive reward-poisonie attacks against reinforcement learning. In International Conference Machine Learning, pp. 11225 11234. PMLR, 2020.
Fangyuan Zhao, Xuebin Ren, Shusen Yang, Peng Zhao, Rui Zhang, and Xinxin Xu. Federated multi-objective reinforcement learning. Information Sciences, 624:811-833 832, 2023.
Zhong Zheng, Fengyu Gao, Lingzhou Xue, and Jing Yang. Federated q-learning: Linear regret speedup with low communication cost. arXiv preprint arXiv:2312. 1e011225500033 15023, 2023.
Zhenpeng Zhou, Steven Kearnes, Li Li, Richard N Zare, and Patrick Riley. Optimization of molecules via deep reinforcement learning. Scientific reports, 9(1):10752, 2019.
Ye Zhu and Xiaowen Gong. Distributed policy gradient with hetcrogencous computations for federated reinforcement learning. In 2023 57th Anreual Conference on Information Sciences and Systems (CISS), pp. 1-6. IEFE, 2023.
A Proof of Theorem 6
Theorem (Theorem restated.. Let Assumptions 1 2 and 3 hold. Suppose that all agents are updated
cleanly at the first p rounds, and at round p, agent n poisoned. Let c
2
where B is defined nl,
8 B Vo. (oo (V.oJ((,r)) ((V..((0)))
0
Then, for B O and c+, we have Jr 1oy a, where E [0, 1.
Proof. Since 1oy is smosohhh with respect to [n): we have
Poy S jrp
+ V E (oj)
ml
I
+ ワ م
r 12
(8)
2 (n)
Moreover, we can write
P0o T V
J 1o (
(n) n
vE(m]
= V Vo %
Σ
O0
Vr %
(9)
vE(m)
and
00
V (m)
0 (1)
(n)
n
1E(m)
1
V (n)
0
(m)
1
V (n)
00 (0)
+ \o V
pp-1J
(( (.))
gu-i J(0
)
(10)
(n)
20

---

Substituting Eq. 10
into Eq.
9
we obtain
Ao
(V
Pool"
Eivog
oo (0) (0))
[Vrr v
r m.))
(11)
(m)
n²
(0)
(0)
kE(m]
On the other hand, we have
p
r (n)
-.V((Vr J (mm))
(12)
(n)
If we combine Eq.
12 with Eq.
11
we get
c. Aa
V
P
r
Σ
Ve
1
0
(1)) I1
1V ry Vor 1J)
(0) [ ε(V
re Sny).
(13)
(m)
n
iE|n]
Substituting Eq.
13
into Eq.
and using the fact that
|iP (n)
[(n)ll' ===
we gel
xe
B 4r E
Po
(14)
n
As the hight-hand side of Eq. 14 is quadratic function with respect to e, we get that when B 0 and
0 O<E < 2doB it holds that (0) Ko) indicating a strict decrease in the objective of FRI being poisoned nL,
compared with clean training. In particular, the smallest bound in Eq. 14 is achieved when Ae al, implying
Ro S J
B (0) 2
J
(0)
A case where B 0. Suppose r(0;r)= 7 r, where the reward sequence and y is the discount factor vector. This objective corresponds with the typical accumulated discounted reward setting. Suppose that (0) is differentiable function. Recall that
0= (0)
B = Vo (0)
Define
Bi = Vo (()
0P
B2 = Vr,oJ(0,1)
0-0
Bg = ((),(()))))
(0)
We have B= BT B2 B3. To intuitively understand B, we simplify the reward |r] = 1, where denotes cardinality, and correspondingly y is simplified to scalar: ạ = 1. Since
1 Σ
1
(0))
J (0)) = J(PP (0) T ( (%) (0
r (0 )
1=1
vE(m
have
:= = Vo-Joo)
- Σ
0 )) (0
0--0"
n
iE|m]
Since
J(OP-) (0) (r)))) = J( ((m)) (0%))))))) = (mm)(0p)) (0%-) = (n)
21

---

have
0=0mm (0)
B2 (0,0)(0,r)
V. %)
(0 (0))
[mm (0) (r/m)) (0))
r=r (n)
(
B3 1.
Therefore,
(mn (0
(0% (0)
B =
Σ
(0 (0
(
(E(M]
(m) (0)
Suppose that agents interact with the same environment, that is r()) (0) r(0), Vi E [n]. Then, (0) = 0,
get
0 1 Xor 10)
and hence,
(0 )0") (0)
r' r(00))))
( (0) + r "00 1o . "0 (0))))
1((0%))))
(1 1 ((0%) (0%)
which is quadratic function w.r.t. 7000) Therefore, in this case we always have > 0, as long as +oo). In particular, a higher rate Ag makes easier to achieve a positive B.
y" (0)
Proximal Policy Optimization (PPo)-Specific Framework
this appendix, we focus on specific local RL algorithm, Proximal Policy Optimization (PPO) (Schulman et al., 2017), for the individual agents in FRL and propose corresponding framework for poisoning. By specifying the local RL algorithm PPO, and accordingly propose targeted solution in Section 4 to poisoning PPO-specifie FRL. In Section B
able to tailor the problem formulation in Problem P
introduce PPO preliminaries to specify the general variable in Problem P Then we discuss the PPO-specific
problem formulation for poisoning FRL in Section B This will allow us to take into account the specific charactoristics of the PPO algorithm when defining the problem.
PPO Preliminaries
PPO is a popular Actor-Critic algorithm that uses a clipped surrogate objective. For agent at federated round a ap..
and local episode 9, denote the pair of state and action at its t-th step rollout as P.g.t (s..
(i)
and denote the V-function and Q-function defined by Bellman Equation (Baird 1995 as V(-) and Q(:,),
respectively. Then the advantage function is AP Qs p.g.t p.g.) V(s
denote estimation. Denote PPO's critic model as Φ(- (W))) where w is the model's weights. As with all typical actor-critic algorithms, the critic is a Value neural network to help estimate the V-value the actor so as to further calculate the actor's objective. In PPO, the actor's objective is a clipped advantage (Eqq. 15), where the advantage is estimated by the critic and observation, which can be written in the form of AP
PPO's critic. Let us use
A(wP. Og..)) The critic model updates itself by minimizing the lemporrl-diiffrence error Tesaauro al. 1995 between the estimated and observed V-value. We denote the critic's objective by 8P.4.
p.g.t
22

---

PPO's actor. Denote the actor model (T(-|s, 0), where 0 is the model weight and is some given state.
To specify the general problem P to the PPO case, the clean agent's objective J() (Eq. 2 should be the PPO surrogate objective
gag. (6) Ee [mi (To Ali) (i)
(15)
(() and the observation that the actor samples. Here, (0) clip(1(0) function parameterized by 77.
where [()) (1 (s)) (00 ( (a P.0.
(0) ni
g.g.t is estimated based both PPO's critic (3) 0 and A()
p. ,1 17. 1),, where clip(-) is clipping
PPO-specific FRL poisoning.
As an actor-critic algorithum, when we fit single-agent PPO into a federated framework, we assume that besides the actor model, the critic model should also be updated from individual agents the server, then aggregated by the server and finally broadcast to the local agents at each foderated round p. Denote the aggregated actor 0% and the aggregated critic as (o)) To specify the server's aggregation function A (Eq. 4 we take a conventional paradigm FL:
O 9999 (0m-1 (0)
li=1)
(16)
Σ
O%. (x)
0 +
0P.)
(17)
where A aggregates the local models by adding the averaged local model update to the server's
model Bhagojji et al. 2019 Bagdasaryan al. 2020). By substituting 0%.0 0%- 409a is equivalent to
assigning the averaged local model as the server's s model (Baagoi et al. 2019
We set the poison cost as D(R RR-9) = RP Ra.g
12, and thereby propose the PPO-specific Problem as:
arg min LA oz
p.q 15 (0)
(P-PPO)
(0)
1SPST)
s.t. VI < p < T, 1 15 5 L,
(18)
00.0 --- Vi < n
(19)
(6) (0)
000
( ( 1i S n,
(20)
0 000 arg max j Vi < n,
(21)
(6)
P- arg min P.4 Vi < n, (8)
(22)
(8)
PP- arg max P.9 P.U
(23)
(n)
2.0 arg min
P.0
(24)
8P
L /n.
(25)
(0)
im1
R
p.g 2
(26)
In Problem P-PPO denotes the poisoned variables induced by ÔP4, The constraints interpretation is similar to that in Section 3.3 except that all the equations related to characterize the initialization and local training for the critic, while those related f are for the actor. The constraints are summarized in Table g
23

---

Table 3: Constraints of Problem P-PPO
PARTY
CONSTRAINTS
INTERPRETATION
AGENT i, i
EQ.
19
LOCAL ACTOR INTTIALIZATION
(CLEAN AGENTS)
EQ.
21
LOCAL ACTOR TRAIN
EQ.
20
LOCAL CRITIC INITIALIZATION
EQ.
22
LOCAL CRITIC TRAIN
AGENT
Eq. 19
LOCAL ACTOR INITIALIZATION
(ATTACKER)
EQ. 23
LOCAL ACTOR
EQ. 20
LOCAL CRITIC INIIIALIZATIN
EQ. 24
LOCAL CRITIC TRAIN
EQ.
26
ATTACK BUDGET
COORDINATOR
Eq.
(5
AGCREGATION
C Baseline Algorithms
In this appendix, we provide the baseline algorithms described in Section 6.3
C.1 Standard FRL
Standard Setor-Critic-ban FRL is given in Algorithm and the standard Policy-Gradient-. based FRL is given Algorithm
Algorithm Standard Actor- Siloo-Crttic-baeee FRL
Input: max federated rounds T, max local episodes L, number of agents n, aggregation algorithm A actor-critic objective function J.
Output: server's actor model 0%" and critic model (())
3: Initialize the server's actor model (0) and critic model w (0)-
4: for to T do
5: for = to n do Initialize local actor 0
00-1
(6)
(0)
Initialize local critic P.0
--1
(i)
(0)
8: for y = to do
Interact with environment and obtain OP-g
Compute JPg with Ogg and P.g-1
10:
(i)
11: Updale 0..0 with Jn.9"
Update P.g with 0P.9 (8)
12:
13:
end for
14: end for
15: 0
A49s OR (1)
P.L
(0)
()- (m)
16: w (0) aas PiL p.L
(1) 1n- ((b))
17:
end for
24

---

Algorithm Standard Policy-Gradient-basee FRL
Input:
max federated rounds T, max local episodes L, number of agents n, aggregation algorithm A policy gradient objective function J.
Output: server's policy (0)-
3: Initialize the server's policy ((0)-
for p = to 7 do
for 1 to do for q 1 to L do Initialize local policy 0% + 0%)
Interact with enviromment and obtain Op P.g Compute fog with o Update 0 with (()) 11: end for
8:
10:
12: end for
13:
P
An9y
A499 (AP.T. P.L (1) 1 (n-1)) (m)
(0)
14:
end for
C.2 Random Poisoning
Below give random poisoning algorithms mentioned in Baselines (Section 6.3
Random Poisoning for Actor-Critic-based FRL. The algorithm is implemented almost the same Algorithm except that Line should be replaced with "Poison Reward as R()) and budget by Eq.
P.g using true objective J(n
Random Poisoning for Policy-Gradient-bassee FRL. The algorithm is implemented almost the same Algorithm except that Line 16 should be replaced with "Poison Reward as R (() using true objective Jon and budget by Eq.
C.3 Robust FRL
To mitigate the risk of FRL being exposed to malicious agents, below we describe defense mechanism against FRL (Algorithm (0) that inherits from conventional FL defense mechanisms, where the aggregation is implemented by re-weighting with the clients' reliability. To adapt this robust aggregation protocol from FL to FRL, we let the clients' local RL performance be a reflection of their reliability. The aggregation process incorporates re-ieithting mechanism based on the clients' reliability, as discussed prior work (Fu) et al. 2019 Tahmasebian et al. 2022 Wan & Chen 2021). In adapting this robust aggregation protocol from FL to FRL, we leverage the local RL performance of clients as indicator of their reliability. To be more precise, the re-weighting is accomplished by assigning credits to each agent's policy according to its performance. To that end, the central server runs tests each policy it receives from the agents and records the observations denoted by P.test The server then calculates the average reward P ph.estt for each policy by averaging the rewards in the sequence (ri)efesst dividing them by the sum of all averaged rewards, resulting in set of normalized weights d:. These weights are used to weight the average aggregation of the policies:
pp.eest Finally, the server normalizes the average rewards by
0% Σ
+
Σ
(27)
iEM
w (0)
Σ
W.I. (0)
(28)
iEM
M
The defense mechanism is outlined in Algorithm as the aggregation algorithm _
0 This protocol can integrated into Algorithm and
25

---

end for Algorithm FRL Delense Aggregation
Input: Submitted local actors {0)) rign: Jigh: Submitted local critics (w)()) (sph)i Jisn-
2: Output: Aggregated actor and critic (a)> 4000
3: for i= 1 to do
4: Server obtains Oofert by g
5: Gets mean reward p.eett from Ofje
(i)
6: end for
7: for i to n do
8: Server normalizes the credit of
10: Server obtains 0 and w%) by Eq. 27 and (0)
28
D Experiments
D.1 Additional Settings
D.1.1 Settings for Main Experiments
Additional general settings. We measure the attack cost by the & distance between the poisoned reward and the ground-truth observed reward. During cach local training step, the maximum number steps before termination 300 unless restricted by the environment. The learning rate is set to 0.001, and the discount. parameter is set to 7 0.99.
Additional settings for robust FRL. For each model uploaded to the server, ran it for 10 episodes and collected the mean reward per episode. We then used the normalized rewards the weights of aggregation. We structure the system with only 2 agents for all environments to ensure that the suspicious agent possesses the maximum possible power to poison the system since a smaller number of agents corresponds to a more potent attack.
D.1.2 Settings for Additional Experiments
General settings. Since VPG is more challenging to attack (refer to Section 6.5.1b. in Section D.2 our focus is on attacking VPG-basod FRL.
Targeted poisoning. For simplicity, we choose the target policy to be a single action, whether in discrete or continuous space. We ensure the chosen environments encompass both discrete and continuous action spaces, featuring diverse cardinalitics and dimensions for the action space. Concretely, we the target policies to be 0 (0, 1} for CartPole, 0 E {0, 2, for Lunar Lander, E [-3,3] for Inverted Pendulum, and 0 1- for Half Cheetah. We choose environments of different action space: CartPole (two discrete actions), LunarLander (four discrete actions), InvertedPendulum (one continuous dimension). HalfCheetah (six continuous dimensions).
Multi-agent attack. We opt for the CartPole environment, a relatively simple task, to balance between manageable computation costs and a clear understanding how a fixed proportion operates in multi-agent attack scenario. In Section [.5.T. we have determined that the maximum size of the system one agent can effectively attack is 4 for the CartPole environment and VPG-based FRL system. Therefore, we initiate the system size from and accordingly, the number of poisoned agents begins at 1. Subscquently, we incrementally enlarge the system size and the corresponding number of poisoned agents while maintaining constant proportion between the number of poisoned agents and the system size.
D.2 Results of Experiments
All additional results obtained from experiments following settings in Section 6.4 and D.1.2
26

---

Large-budget Attack. We evaluated the performance of our poisoning with large budget in Figure (4 The results show that with a larger budget, our method is able to attack much larger systems compared with the constraints that appeared in the case of a small budget, i.e., we can attack to 100 agents.
Carthole
bnvertadhenduk..m
Hopper
HallChestah
Moin Roward her Eptios
40-
Potson
100
20 na 20 20 208
130
20
40
200 250
20 22 20 20 105 203
400
Figure 4: High-Budget Poisoning. The red dashed line, labeled as Poison, depicts the performance of our adversarial attack in a standard FRL system. The green dashed line, labeled as Clean, represents the standard FRL system's performance.
Multi-agent poisoning. We have shown that the proportion of malicious agents required to poison a system is consistent regardless of the size of the system. We found that when we increase the system size from agents to 100 agents, fixed proportion of attackers can always poison the system successfully. This result is depicted in Figure
maticious/ 26
maticious/ 32
maicious
maticiuu:
Hoan Rosaad for Esau0de
200
200
383
Peiso Onn
Peison
Reton Oeen
50
Pelson Oren
Clan
Communisation Rounds
Communisation Rounds
Communccation Rounds
Communisation Rounds
Figure 5: Multi-Agent Poisoning. The annotation the same as Fig.
Targeted attack. Our poisoning works well for targeted attack, shown in Figure 16
CartPole-v0
LinarLanderv22
InvertedRenuulmaa
HaitCheetahv-22
claa posor
deon
Cleen poison
dean polscn
cor
310
Communication Rounds
Communi-atin Rounds
Communcation Rounds
Communication Rounds
Figure Targeted attack. The green dashed line, labeled as poison, depicts the performance of our adversarial attack in a standard FRL system. The red dashed line, labeled as clean, represents the standard FRL system's performance.
27

---

E
Additional Background
E.1
Differences between distributed RL and FRL
Distributed RL and FRL are both advanced paradigms in the field of machine learning, but they differ significantly in their architecture and application. Distributed RL involves distributing the computation of single RL algorithm across multiple machines to accelerate learning and handle larger datasets. This approach focuses on parallclizing the training process improve efficiency and scalability. On the other hand, FRL is designed to enable multiple independent agents collaboratively (rain RL model without sharing their local data. In FRI, each agent trains local model using its data and periodically shares model updates (rather than raw data) with central server, which aggregates these updates to improve the global model. This method preserves data privacy and is particularly useful in scenarios where data cannot be centralized due to privacy concerns or regulatory restrictions. Thus, while distributed RL aims at enhancing computational efficiency, FRL emphasizes data privacy and decentralized training, making suitable for applications where data sharing is limited.
E.2 Different RL algorithms
Reinforcement learning encompasses a variety algorithms designed to enable agents to learn optimal behaviors through interactions with their environment. Among these, actor-critic algorithms, such as Advantage actor-Critic (A2C), Asynchronous Advantage Actor-Critic (A3C), and more advancod methods like Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO), are notable for their efficiency in handling high-dimensional action spaces. PPO, recognized for its robustness and simplicity, prevents large updates that can destabilize training, balancing exploration and exploitation effectively. We chose to focus on PPO and VPG due to their practical balance between performance and computational efficiency. While more advanced algorithms like Deep Deterministic Policy Gradient (DDPG) or Soft Actor-Critic (SAC) offer better performance in certain scenarios, they come with increased complexity and computational costs. We focus on VPG and PPO ensure the approach remains accessible and practical while achieving competitive performance.
28


Reviews:
[{"review_id": "mGpnMvcRgD", "reviewer": "Reviewer_jXvH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n(1) The authors study an interesting and important problem, i.e., the poisoning attack in FRL, which should be paid more attention in the real-world scenarios given the vulnerability of FL and RL as well as their composition. Specifically the authors are among the first attempts to show the high risk of the potential poisoning to FRL.\n\n(2) The authors design a novel poisoning attack, which is generally effective under different FRL models, such as actor-critic-based FRL and policy-based FRL. Besides, the authors provide a principled proof to show the effect of the proposed reward poisoning in sabotaging the system, specifically by degrading the global objective.\n\n(3) Through extensive experiments under the OpenGYM environments with varying difficulty levels, the authors shows the consistent effectiveness by comparing with various baseline models and assessments of different (targeted and nontargeted) poisoning types in the mainstream RL algorithms like VPG and PPO.\n\nWeakness:\n\nDespite promise, there are still some concerns that should be carefully addressed, which I summarized as follows.\n\n(1) Some claims in the manuscript is not very proper. For example, the first sentence in the abstract \"Federated learning (FL) has become a popular tool for solving traditional Reinforcement Learning (RL) tasks.\" FL is not a popular tool to RL when there is no constrain on the data governess. Therefore, the authors should point out the application range of FL regarding RL. The beginning of the third paragraph in the introduction \"Poisoning in FRL is practical and harmful. The inherent nature of FL and RL amplifies the susceptibility to poisoning (training-time attacks) when combined into FRL.\" Some evidence can be given to show the potential amplifying effect w.r.t. the poisoning. Or you can give more intuitive explanation to convince the readers about your point.\n\n(2)  The authors can further explain the rationality about Theorem 6, as the even the global objective can be smaller under some poisoning budget, the optimization eventually decides the convergence of the model. It will be more helpful to enrich the discussion about the theorem so that both the implication and limitation are covered. \n\n(3) The experiments can be enhanced, as many factors like the ranging number of the participants in FRL (not the poisoning agents), the different configurations of one environment to show the consistency etc, has not comprehensively considered.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission introduced a novel reward poisoning mechanism for FRL under both general policy-based and actor-critic algorithms. The authors theoretically demonstrate the importance of poisoning attack w.r.t. the risks during real-world scenarios, and highlight the problem that draws the attention of the community. By conducting a range of experiments on various OpenGYM environments using popular RL models, the authors show that the proposed protocol is effective in poisoning FRL systems and inspires future research to design more robust algorithms to protect FRL against poisoning attacks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weakness, which should be taken into consideration to improve the submission. In detail,\n\n(1) the writing on some critical claims should be improved to make them more proper and are well supported with some facts or discussion.\n\n(2) The experiments can be enhanced by considering some varying factors in FRL.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lT9GgNwehB", "reviewer": "Reviewer_BvyU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n- This work is looking into a new problem of poisoning attacks for FRL\n- Theoretical analysis is provided to guarantee the attack method can decrease the global objective. \n\n## Weakness\n- The contribution seems limited. It is unclear what the challenges are for applying poisoning attacks to FRL compared to poisoning attacks for RL. \n- For baselines in the evaluation section, why are the many existing cited RL poisoning methods not compared in this work, you can apply those methods for a single agent in the RL setting. It is hard to see the performance advantage of your proposed method when not compared to those baselines.\n- In the evaluation section, it is unclear how the method performs as you have different numbers of agents. The number of agents for each case is also not clearly stated (only mentioned 3 to 4 agents for the VPG-base FRL case). The evaluation results for poison VPG-base FRL seem to have similar effects as the random attack. \n- The attack method involves a public critic and a private critic which seems to require intervention in the training processes and weight upload, so it's not only just modifying the rewards. Is that a realistic threat model?\n- I suggest improving the writing of the formulation section to make it simpler for readers to understand your work. For instance, 1) you define O as the sequence of state, action, and reward in section 2, and refer to them as observations in section 3.3. This can be easily confused with the \"observation\" in partial observable RL problems. 2)when you use the notation of i<n (equation 2) and i!=n (table 1 & 2), is there a reason why they have to be different? if not, it's better to keep them consistent.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work looks into a new area of problem in poisoning attacks for federated reinforcement learning. This framework is designed for policy-based FRL and can be applied to both policy-gradient local RL and actor-critic local RL scenarios. \n\nThe authors provided theoretical analysis and conducted experiments to evaluate the poisoning attack method.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Clarify the challenges of the problem and the contribution of the work\n- Add the RL poisoning work as baselines and compare the performance (see above)\n- clarify the evaluation section in terms of the number of agents used, and evaluate how the number of agent and random selection of agents affect the performance of the proposed method\n- clarify the threat model\n- improve writing (see above)", "broader_impact_concerns": "This is not provided. Since the work focuses on the security of AI, I suggest adding a section on this.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KL6iUOkcOK", "reviewer": "Reviewer_tQxM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n* The paper proposes a new method for poisoning FRL agents that can be relevant to enhancing the robustness of FRL.\n* The paper provides through details on the algorithmic descriptions.\n* The paper provides a useful review of poisoning methods in related RL scenarios.\n\n**Weaknesses:**\n* The paper dedicates a lot of space to describing preliminaries and some background that can distract from the main message of the paper. I recommend moving some of the extraneous description into the Appendix and moving relevant experimental results and descriptions from Appendix C and Appendix D into the main paper.\n* Some of the empirical results don’t fully support the authors’ claims (e.g., Figure 1 – walker) and the paper currently offers no explanation for those.\n* There is no discussion of limitations, future work and/or potential poisoning mitigation strategies.\n* Discussion of related work and methods could be strengthened.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new training-time attack for federated reinforcement learning. The starts by introducing applications of reinforcement learning, as well as federated reinforcement learning (FRL) before discussing related work on poisoning for different reinforcement learning contexts. Next, the paper outlines contributions, which mostly centers on a new poisoning protocol for FRL, a theoretical analysis of that protocol and experimental evaluation of the protocol.\n\nIn Section 2, the paper introduces preliminaries and notations followed by the problem formulation in Section 3. In Section 4, the paper introduces the proposed poisoning protocols methods, including reward poisoning for actor-critic based FRL and policy gradient based FRL with through algorithmic details. Section 5 provides a theoretical analysis of the proposed method and Section 6 provides empirical results across multiple RL environments. Many of the empirical results show that the poisoning methods generally decreases performance of the FRL agent.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I request addressing the weakness identified above, including:\n\n* A more thorough discussion of the experimental results, especially when the attack appears to make little difference when considering the variance of the results.\n* Adding more relevant results and description into the main paper, such as those found in Appendix C and Appendix D.\n* Provide a discussion of different RL algorithms, especially actor-critic algorithms, and why the authors chose to focus on PPO and VGP instead of more advanced algorithms.\n* A more thorough discussion on the differences between distributed RL and FRL\n* A discussion on why policy gradient does not have local gradient poisoning\n* An expansion of the introduction including the application of RL to different design problems, such as design of materials [1][2][3], circuit design [4] and optimization [5] and how the poisoning aspects relate to those cases as well as the cases already identified (e.g., IoT).\n\n[1] Govindarajan, Prashant, et al. \"Learning Conditional Policies for Crystal Design Using Offline Reinforcement Learning.\" Digital Discovery (2024).\n\n[2] Zhou, Zhenpeng, et al. \"Optimization of molecules via deep reinforcement learning.\" Scientific reports 9.1 (2019): 10752.\n\n[3] Ghugare, Raj, et al. \"Searching for High-Value Molecules Using Reinforcement Learning and Transformers.\" arXiv preprint arXiv:2310.02902 (2023).\n\n[4] Roy, Rajarshi, et al. \"PrefixRL: Optimization of parallel prefix circuits using deep reinforcement learning.\" 2021 58th ACM/IEEE Design Automation Conference (DAC). IEEE, 2021.\n\n[5] Chen, Tianlong, et al. \"Learning to optimize: A primer and a benchmark.\" Journal of Machine Learning Research 23.189 (2022): 1-59.", "broader_impact_concerns": "The paper would be strengthened by including a broader impact section that relates to potential misuse of the poisoning method and how to mitigate it.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper ntroduced a novel reward poisoning mechanism for FRL under both general policy-based and actor-critic algorithms. The authors theoretically demonstrate the importance of poisoning attack w.r.t. the risks during real-world scenarios, and highlight the problem that draws the attention of the community. By conducting a range of experiments on various OpenGYM environments using popular RL models, the authors show that the proposed protocol is effective in poisoning FRL systems and inspires future research to design more robust algorithms to protect FRL against poisoning attacks.

The authors study an interesting and important problem, i.e., the poisoning attack in FRL, which should be paid more attention in the real-world scenarios given the vulnerability of FL and RL as well as their composition.The authors design a novel poisoning attack, which is generally effective under different FRL models, such as actor-critic-based FRL and policy-based FRL. Through extensive experiments under the OpenGYM environments with varying difficulty levels, the authors shows the consistent effectiveness by comparing with various baseline models and assessments of different (targeted and nontargeted) poisoning types in the mainstream RL algorithms like VPG and PPO. However, the contribution seems a bit limited. It is unclear what the challenges are for applying poisoning attacks to FRL compared to poisoning attacks for RL. Therefore, it is better to clarify the challenges of the problem and the contribution of the work, and add the RL poisoning work as baselines and compare the performance. Based on three qualified reviews, this paper can be accepted with minor revision and the authors are encouraged to merge the comments into their update versions.
Decision: Accept (poster)