Value-Based Membership Inference Attack on Actor-Critic Reinforcement Learning
Authors: Yunhao Yang
ufuk topcu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Privacy
Membership Inference Attack
Value Function
Actor-Critic
Reinforcement Learning

Abstract: In actor-critic reinforcement learning (RL), the so-called actor and critic, respectively, compute candidate policies and a value function that evaluates the candidate policies. Such RL algorithms may be vulnerable to membership inference attacks (MIAs), a privacy attack that infers the data membership, i.e., whether a specific data record belongs to the training dataset. We investigate the vulnerability of value function in actor-critic to MIAs. We develop \textit{CriticAttack}, a new MIA that targets black-box RL agents by examining the correlation between the expected reward and the value function. We empirically show that \textit{CriticAttack} can correctly infer approximately 90\% of the training data membership, i.e., it achieves 90\% attack accuracy. Such accuracy is far beyond the 50\% random guessing accuracy, indicating a severe privacy vulnerability of the value function. To defend against \textit{CriticAttack}, we design a method called \textit{CriticDefense} that inserts uniform noise to the value function. \textit{CriticDefense} can reduce the attack accuracy to 60\% without significantly affecting the agent’s performance.

Under review as conference paper at ICLR 2023
VALUE-BASED MEMBERSHIP INFERENCE ATTACK ON ACTOR-CRITIC REINFORCEMENT LEARNING
Anonymous authors authorss Paper under double-blind review
ABSTRACT
In actor-critic reinforcement learning (RL), the so-called actor and critic, re- spectively, compute candidate policies and value function that evaluates the candidate policies. Such RL. algorithms may be vulnerable to membership in- ference attacks (MIAs), a privacy attack that infers the data membership, i.e., whether specific data record belongs the training dataset. We investigate the vulnerability of value function in actor-critic to MIAs. We develop CriticAr- tack, a new MIA that targets black-bor -bx RL agents by examining the correlation between the expected reward and the value function. We empirically show that CriticAttack can correctly infer approximately 90% of the training data mem- bership, i.e., it achieves 90% attack accuracy. Such accuracy is far beyond the 50% random guessing accuracy, indicating severe privacy vulnerability of the value function. To defend against CriticAttack, we design method called Crit- icDefense that inserts uniform noise to the value function. CriticDefense can reduce the attack accuracy to 60% without significantly affecting the agent's performance.
INTRODUCTION
Membership inference attacks (MIAs) pose privacy vulnerabilities in reinforcement learning (RL) algorithms (Gomrokchi et al. 2020 Such attacks may make inferences about the train- ing environments wheeemeeeetttthhtereeeeeeee whether a particular environment has been used in training by observing the outcomes of an RL algorithm. For example, |an et al. 2019 ; Wang et al. (2019); Chen et al. (20T)) show that MIAs can infer users' vehicle routes or room layouts.
Most, if not all, existing methods for MIA suffer from high computational complexity or make un- realistic assumptions. For example, the methods in Pan et al. (2019) and Yang et al. 2002 rely on observing the learned policies. Both methods are computationally inefficient because they need to learn separate policies for each environment the attacker wants to infer. The methods Gom- rokchi et al. [2021 2020) do not require learning additional policies for different environments, but assume that the attacker has full access to the RL algorithm, including the states, transitions, actions, and rewards which the algorithm relies.
We propose a new black-box MIA called CriticAttack that alleviates the computational burden and relaxes the unrealistic assumptions made in the existing works. CriticAttack trains one set of policies for all environments, as opposed to training one set of policies per environment (e.g., Yang et al. (2021). It makes inferences only based the values generated by the value function and the expected rewards, in contrast to the states, transitions, actions, and rewards required by the existing work (e.g./Goomrol Gomrokchi et (2021).
We empirically show that CriticAttack can achieve 90% accuracy in inferring environments from the MiniGrid library (Chevalier- Cchiselie--ooiserttt et al. 2018 We perform the MIA on a state-of-the-an actor-critic RI. algorithm (Schulman et al.) 2017 7). The actor-critic algorithm trains two compo- nents: an actor and a critic. The actor generates policies that determine an RL agent's actions. The critic learns value function that evaluates the policies by predicting the expected rewards, also known as rewards to-go. The actor and the critic typically memorize their training environ- ments [Harnoja et al. 2018 Raichuk al. 2021) Hence, expect high correlation between the values and the expected rewards from a training environment. On multiple RL tasks, CriticAr- tack achieves 90% attack accuracy, significantly higher than the 50% random guessing accuracy.

---

Under review as conference paper at ICLR 2023
Such high attack aceuracy is an indication of the severe privacy vulnerability of the value func- tion.
We then turn attention to defending against CriticAttack. We design a simple and efficient defense method called CriticDefense that concentrates on the value function. It inserts uniform noise to the value function to reduce the correlation between the values and the rewards-to- go. However, inserting noise introduces a trade-off between the attack accuracy and the agent's performance, e-g,, measured by the cumulative reward that the agent obtains. CriticDefense can reduce the attack accuracy from 90% to 60% while degrading no more than 10% of the agent's performance.
Furthermore, we provide empirical evidence to show that the correlation between the values and the rewards-to-go is the primary source of privacy vulnerability. Due the exploitation feature of RI., agents tend to choose the states experienced during training. The value function can accurately predict rewards- to-go on experienced states. Hence the correlation computed from training environment is significantly higher than that from a test environment. The high correlation in the training environment leads to high attack accuracy.
The optimized value function plays key role in transfer learning and the teacher-student frame- work. Many knolw transfer learning algorithms for actor-critic require the source agents to release their optimized value functions (Xu et al. 2020 Zhang & Whiteson, 2019 Takano et al. 2010). In the seacher-student framework, the student agents learn the optimal policies from the teacher's policies and value functions (Kurenkov et al. 2019 Therefore, it is essential to consider the privacy implications of the value function.
2 RELATED WORK
Pan et al. 2019 and Yang et al. 2021 develop MIA methods for deep RL that collect policies
or actions for inference. While CriticAttack collects values from the value function and the cu-
mulative reward for membership inference. Gomrokchi et al. [202T] and Gomrokchi et al. (2020) introduce two MIA methods to infer the roll-out trajectories in off- off-policy RI. algorithms, which
learn the optimal policy independently of the agent's actions. In contrast, CriticAttack works for on-policy RL algorithms, which optimize policies that determine what actions to take. From the defense perspective, several works (Garcelon et al. 2021 Lebensold et al. 2019b Liao et al. 2021 Balle et al. 2016b Chen et al. 2021 enforce differential privacy to the RL algorithm, which can protect against MIAs. Compared to the differential privacy mechanisms, we design CriticDefense for protecting the value function specifically. CriticDefense provides robust protection against attacks on the value function; however, it has limited ability to protect other components in the algorithm and does not achieve differential privacy.
3 PRELIMINARY
Reinforcement Learning (RL) is area of machine learning where train agent or set of agents by interacting with a set of environments. The agent observes a state from the environ- ment, then takes action based its policy л,, and receives reward from the environment that evaluates this action.
We formally define the environment Markov decision process (MDP) E A,P,,,,,, where S and A are the sets of states and actions, P: 5x A - is the state transition function, /:S- [0, IJ the initial distribution of the states, and R: S is the reward function. We consider set of environments as the training dataset the agent that may face privacy threats.
Actor-Critic Konda Tsitsiklis 1999) is one of the kate-of-the-aa RL algorithms that trains two components: actor and critic. The actor with parameters 0 takes the current state repre- sentation and all possible actions as input and then generates a policy πg. The critic (*((s) with parameters & learns value function, which takes the current state observation as input and out- puts a value that evaluates the actions leading to the current state. We present the details of the actor-critic algorithm in the Appendix.
2

---

Under review conference paper at ICLR 2023
In the training stage, run the agent in the set of environments to collect set of trajectories. Each trajectory consists of a sequence of tuples (state S,, action a, reward Ft, new state $1+1) with respect to timestamp = I,... T. We then estimate the advantage AG to compute the param- eters' gradient in both the actor and the critic. Most actor-critic algorithms use one of the three advantage estimation methods in Equation
The value function evaluates the current state and past actions leading to the current state by estimating the reward-1o-Pe -oogo. Reward-to-go is the expected cumulative reward the agent can get if starting from the current state: P Ik= /k-- Note that rix k> is an expected reward the agent will likely to get at timestamp k. the training stage, we compute the reward-to to-go at timestamp r by giving the values and the advantage estimation method &:
TD advantage: F, A7 7 + (5p),
A = vV (S
V (s)).
N-1 NN,
N-step advantage: F,,N A²,N + (s)),
y IIkIl + V7 (S (sc).
(1)
k=
Generalized advantage: Fi, A,,, (st),
Ma = = (yA) Ax
We train the critic estimate the reward-to-go at given state s,, and update the parameters accordingly to minimize the value loss:
L(()) Σ (s)
Membership Inference Attack (MIA) is one of the well-known privacy attacks that be ap- plied to machine learning models to infer whether a selected data record belongs to the training dataset of the given model. The shadow model framework (Shokri et al.) 2017 is the standard approach to MIAs on machine learning models, where the shadow models mimic the behavior of the targer model. Since the training datasets of the shadow models known, the attacker can learn to infer whether data record is used in training the shadow model. We then apply the trained attacker infer the target model. We denote the percentage of the correctly inferred data records as the attack accuracy.
4 ATTACK METHOD
We design an environment-bact MIA on actor-critic algorithms named CriticAttack: In environment-bass based (user-based) MIA, the attacker infers about an environment, as opposed to trajectory-basee bbased (sample-bas -baee) MIA, where the attacker infers about a single trajectory. Criti- cAttack determines whether the agent has been trained under particular environment based on the observation of the values and rewards-to-go.
4.1 ASSUMPTION
target agent is the agent trained by set of private environments that the attacker wants to infer. In this work, we perform CriticAstack on the target agent whose policies and value functions are composed of neural networks and optimized by the actor-critic algorithm.
The information of the target agent includes the well- trained parameters of the actor-network and the Priticnetwwrrk the specifications of the two networks such as the number of layers and the activation function, the training algorithm with hyper-pare parameters, loss functions, the gradi- ent history, and feedback from the environments.
The attacker typically does not have full access to the information of the target agent. Based on the attacker's access, we can categorize MIAs into two groups: black-box attack and white-bon attack (Hu et al. 2021 The black-box attacker only has access to the inputs and outputs of the neural networks, the actor- netor-network and the critic network in this case. contrast, the white- box attacker has full access the parameters of the neural networks, loss functions, and gradients. However, several black-box MIAs (Shokri et al. 2017 Sablayrolles et al. 2019) can also access the network specifications, training algorithm, and hyper- hyper-parameters.
3

---

Under review conference paper at ICLR 2023
To distinguish black-box and white- attacks, consider the access the networks' param- eters and gradient history as the borderlines between the two types of attacks. The attacker is a black-box attacker if it has access to neither networks' parameters nor gradient history.
In this work, we assume the attacker only has access to the inputs and outputs of the actor and critic networks, the training algorithm (including the advantage estimation method), hyper- parameters, and rewards from the environments. Therefore, CriticAttack falls into the category of blx attack.
4.2 CRITICATTACK
CriticAttack follows the shadow model framework (Shokri al. 2017). Since we do not have access to the target agent's training environments, we train a set of shadow agents with known environments mimic the behavior of the target agent. The attacker learns how the shadow agents behave differently in visited and new environments.
We assume there is a public universal data distribution that all the environments, regardless of whether they are used during training, are drawn from this distribution. So, the attacker can ob- tain similar datasets to train the shadow agents. Once the attacker learns to differentiate whether environment has been used in training the shadow agents,
apply it to the target agent.
Training the attacker takes the following three steps:
First, obtain a set of environments from the data source and evenly partition the environ- ments into two groups: training environments and validation environments. We train each shadow agent using the training environments until its performance is less than 5% different from the target agent. We measure the performance by the average rewards in the validation environments. We then repeat this step to construct multiple shadow agents.
Second, we run each shadow agent on its training and validation environments to collect trajec- tories. For each environment E, we collect corresponding trajectory set Sg that contains critic trajectories, which is defined in Definition 4.1
Definition 4.1. critic trajectory T consists sequence of (value, reward to-go) tuples:
F. = (s). t0..,,
where is the trajectory length. The critic trajectory can break up into a value trajectory T" and reward-to- to--too--oooo go trajectory T":
(V) t=) T}, (F T).
Note that we need to compute the rewards-to-ge given the rewards and the value trajectory. We trace the rewards [f1...., [r] and the value trajectory T from T to 0 compute the reward- to-go trajectory T" and get the critic trajectory T using Algorithm 1 After obtaining the critic trajectories, we label each critic trajectory set Sg as 'in' E belongs the training environments and 'out' otherwise. By repeating the second step, we get critic trajectory set and its label for every environment the training and validation dataset. These critic trajectories and labels form supervised learning dataset for the attacker.
Third, we train a binary classifier that takes set of critic trajectories Sg as input and determines the corresponding environment E 'in' or 'out' of the training environments. We design two ar- chitectures for the binary classifier: the logistic regression classifier and the deep neural network classifier.
Logistic Regression on Correlation Score (LR) focuses on the correlation between values and rewards-to-go. Suppose we have collected N sets of critic trajectories from the shadow agents, where each set contains n trajectories. For each environment E, we extract the value trajecto- ries T. and the reward-to-ge trajectories T from the trajectory set Sg, where i I,..., and number of trajectories Sg- Then, we compute the average correlation between the value trajectories and reward-to-ge trajectories following Equation
4

---

Under review conference paper at ICLR 2023
Algorithm I: REWARD-TO-GE ESTIMATION
Input: value trajectory 7", rewards r, discount factor y, estimation method &, hyper-parameters N,A
Output: critic trajectory T
A *T1NNN, t (ST-1), 0T
for f = T to 0 do
if & is TD advantage then
A I.N. = TI + y (5++1) - 1 (
/* refer to Equation
end
& is N-step advantage then
A T(+1 V (5²) +V (81+1);
/* refer to Equation
Ar+N = F++ & (8++N=1)++ + (8++N+2)
A A,,.A =Y (+IN, A, - A
/+ Proof: see Appendix=
end
if & is Generalized advantage then
A I.N.A = T1 V (8++1) -V (s,) + Y.A. A (+1NA
/* refer to Equation 1
end
F, A T+L,NA + V (s()
end
TM ([Vt,, Fr) 0,..,,
coulTY, TD an ECTT -T - -MT))) PE - Σ N F= OreOT r
In the training stage, we compute a correlation score pe for each environment E, form (corre- lation, label) tuple, and mark the label 1 to represent 'in' and 0 to represent 'out.' We then fit the logistic regression classifier with all the (correlation, label) tuples.
In the inference stage, we compute the average correlation score Pr for the target agent on given environment Eyal and use the logistic regression classifier to predict Eval belongs to the training dataset of the target agent.
Deep Neural Network (DNN) takes the concatenation e of the value trajectory and the corre- sponding reward-to-Pe trajectory as input and performs binary classification:
as 'in',
(T. e T/
as 'out'.
In the training stage, assign a label 'in' or 'out' to each critic trajectory depending whether is collected from a training environment. We then train the neural network with these labeled trajectories.
In the inference stage, we run the target agent on a given environment to obtain n trajectories. We apply the trained neural network to predict each trajectory and take the majority vote as the prediction for the given environment.
DEFENSE METHOD
In practice, the best protection is to conceal the value function from users. However, RL agents that allow users to fine-tune or allow to be the teacher in transfer learning must release their value functions. In such scenarios, we introduce a defense method named CriticDefense specifically against CriticAttack. We consider the correlation between the values and the rewards-to go as one the primary factors impacting the attacker's decision. Therefore, we develop CriticDefense modify the correlations and examine if it can effectively reduce the attack accuracy.
5

---

Under review conference paper at ICLR 2023
5.1 CRITICDEFENSE
CriticDefense inserts uniform noise into a certain percentage of the values during training. Prac- tically, we replace the value function &b in the actor-critic algorithm with the following:
V (s,) (V (s,) (oo.se) (4) 38) 1,,1,
(4)
where A is hyper-parameter to define the noise percentage, u is sampled from a standard uni- form distribution: - U(0,1) 1), % is the modulo operation. CriticDefense adds negligible noise to a small proportion the values as A approaches 0 and adds large noise to almost all the values if Ø approaches 1.
Adding uniform noise to the values indirectly adds noise to the rewards-to soaaaddsst--ggo to-go since we compute rewards-to-g go based the values according to Algorithm CriticAttack makes inference only based on the values and rewards-to-gee adding noise to both components is the most straight- forward approach protect against such attack.
In RL. algorithmss, values are strictly between and 1, so we use modulo to guarantee this. Com- pared clipping the noisy values, modulo one ensures the noisy values do not exceed the upper limit and allows a more significant change, which may further break the correlation between the values and rewards-to-g to-go. For instance, the original value of 0.95 with a noise of 0.1 will result in an invalid value of 1.05. Value clipping clips the noisy value to 1 and makes it valid. In con- trast, modulo results in a new value of 0.05. The modulo will be triggered more constantly as æ approaches 1.
5.2 COMPATIBILITY
CriticDefense is designed to protect the value function against CriticAttack; so it has limited effi- cacy protecting the policies and mitigating overfitting, We integrate two other methods with CriticDefense to strengthen the protection of other components of the actor-critic algorithm, such as policies. CriticDefense compatible with the two methods below, which means they do not interfere with each other and introduce extra performance loss.
Regularization is prominent approach to prevent overfitting by lowering the complexity of the neural networks during training (Kukačka et al.] 2017). Many works have demonstrated that regularization reduces MIA accuracy by mitigating overfitting [Ving et al. 2020 Nasr et al. 2018]) B
Kaya et al. 2020 We consider applying L2 regularization in the actor- critic algorithm: add
regularization loss with a regularization rate a to the original value loss L (Φ).
Value Clipping is another approach to prevent the value function from being over-adapted to the newly added training environment and losing the information from previous environments (Schulman et al. 2017). Existing work has shown its effectiveness in protecting against MIA in RL policies (Yang et al. 2021). It clips the norm the value loss to Eclip to restrict the step size of updating the parameters.
EMPIRICAL ANALYSIS
In this section, we perform two sets of MIA experiments. the first set of experiments, we pro- vide empirical evidence to show that CriticAttack can correctly infer above 90% of the training environments. In the second set experiments, we demonstrate the vulnerability factor of Crit- icAttack and show that CriticDefense reduce the attack accuracy to 60% while maintaining the agent's performance.
In both the attack and defense section, we clip the values to Eciip = 0.2 and enforce L2- regularization with a 0.01 to mitigate overfitting while training the agents. We apply the two methods in both sections to show that: 1) CriticAttack works well even if the mitigation methods are applied, and 2) the attack accuracy drops due to CriticDefense rather than the two mitigation methods.

---

Under review conference paper at ICLR 2023
Figure 1: RL tasks from left to right:
Door-Key, Lava-Crossing, Four-Rooee Rooms.
Environment Setup In all the experiments, we use the MiniGrid toolkit et al. 2018) as the underlying testbed. We choose four tasks listed in Figure wherein the agent learns to reach target destination without bumping into obstacles. Once the agent reaches the destination or fixed number of steps, the environment resets to new map while the task remains the same. We control the map's layout by fixing random seeds due the one-to one-to-one cor- respondence between seeds and maps. We use the maps simulate room hoom/wrreoous layouts in world settings and reveal their privacy. Assuming that each new map represents a floor map of private property, we perform MIA infer whether the agent has visited given floor map in its training, which violates the privacy of the private properties.
Experiment Setup We have presented three different advantage estimation methods in Equa- tion I so perform three subsets of experiments for the three estimation methods separately. We use the Proximal Policy Optimization (PPO) algorithm [Schulman et al. 2017 for deploying the three advantage estimations: TD advantage, N-step advantage (N- Step), and Generalized ad- vantage (GAE). We apply the PPO algorithm on 40 unique maps to train each target agent (we also show the MIA results targets with training maps in the Appendix). We implement the PPO algorithm using the RL-Starter- Files library (Willems) 2018 with default unless specified below.
6.1 ATTACK
Following CriticArtack the Methodology section, we train five shadow agents for each target agent. Due the assumption that the attacker does not have access the training data size, we use 20 distinct maps to train each shadow agent until it converges to the same reward as the target agent.
Then, we apply each shadow agent to 40 distinct maps- in which 20 maps are used to train this shadow agent- collect 25 critic trajectories from each map. So, we can collect 200 trajectory sets with 5,000 critic trajectories from 5 shadow agents and use them for training the attacker. Once we finish training the attacker, we apply the target agent to infer the training maps of the target agent.
have proposed two architectures for the attacker: L.R and DNN. present the MIA results of the two architectures in Table and show some visualized examples in Figure The LR attacker can achieve approximately 90% accuracies by only finding a correlation threshold. The DNN attacker can get close to 95% accuracies; however, it is computationally inefficient compared to the LR. In summary, both attackers demonstrate the severe vulnerability the value function by showing such high attack accuracies.
Estimattio((Attakee)
Multi-Rooms
Door-Key
Lava-Crossing
Four-Roomt
TD Advantage (LR)
90.5 1.6
89.2 + 1.3
91.8 +: 0.9
93.1 : 0.8
TD Advantage (DNN)
9..2
92.9 1.2
93.8 1.2
96.2
N-Step (LR)
1.4
1..7
90,6+
91.5 1.1
N-Step (DNN)
94.2+
91.4 1.4
93.2
9.6
GAB (LR)
89.7
89.7 1.6
90.2
92.5 1.0
Table 1: MIA accuracies (mean standard deviation) across five repetitions. 97.1 + 0.6
GAE (DNN)
95.7 + 1.3
94.1 1 1.3
94.2
7

---

Under review conference paper at ICLR 2023
Value
Reward-to-go
'in', no defense
"out', no defense
'in', CriticDefense
'out', CriticDefense
value
0.5
0.5
0.5
0.5
0
20
40
60
0
20
40
60
20
40
60
20
40
60
step
step
step
step
No Defense
CriticDefense
0.55
probability
0.6
0.5
0.4
0.45
predict in, actual in
0.8
0.4
0.6
predict out, actual out
predict out, actual in
40
40
predict in, actual out
votes
20
2.
0
0
xXX
x
0.6
0.8
1
0.2
0.4
0.6
correlation
correlation
Figure 2: MIA PPO algorithm with GAE trained for the multi-Rooms task. The figures in the first row show the value and rewards-to-ge trajectories. The second and third rows show the MIA results using LR and DNN, respectively.
6.2 DEFENSE
now investigate the effects CriticDefense and compare with the well-known differential privacy mechanism DP-SGD. We assume the attacker knows the defense methods applied to the target model, so we also deploy the defense methods with the same parameters while training the shadow models. Note that we are not trying to indicate that the CriticDefense outperforms DP-SGD. CriticDefense is more suitable for protecting the value function against CriticAttack, but SP- is potentially more effective in protecting the policies.
Privacy-Perfon Performance Trade-off 3igure the defense results on GAE on the solti-Rooms task, while the MIA settings are identical to Section 6.1 We observe that both of the defense methods can reduce attack accuracies. CriticDefense can reduce the attack accuracy to approx- imately 60% with less than 10% performance loss, measured by rewards. We compute the per- formance loss using the cumulative reward of the unprotected agent rT and the cumulative re-
n- ward the protected agent r'y: Lper Fr CriticDefense can reduce the attack accuracies to around 60% with approximately 10% of performance losses in the other tasks, regardless of the estimation methods. Figure c attack results before and after applying CriticDefense and shows how CriticDefense reduces the attack accuracy. We also present the numerical results in Table in the Appendix.
In contrast, the DP-SGD algorithm can reduce the MIA accuracy to approximately 50% with over 60% performance loss. Additionally, the actor-critic algorithm with DP-SGD requires signifi- cantly larger number of steps to convergence than CriticDefense. means the computation of training protected agent is ten times more expensive than training an unprotected one.
8

---

Under review conference paper at ICLR 2023
Training
Validation
LR
DNN
Total Steps
Reward
Correlation
MIA Accuracy
100
8
0.6
6
0.8
0.8
4
0.6
0.4
2
0.4
0.2
0.6
0.2
10
10
10 2
10
10 2
10
10
2
10
R
R
R
R
-106
8
0.6
6
0.8
0.8
0.4
4
0.6
2
0.4
0.2
0.6
0.2
0
1100
[T)
10-2
10
10
10
2
10-1
10
10
10
10
10
10
100
0
0
0
0
Figure 3: Protections against MIA. The first row shows the results of CriticDefense, and the second row shows the results of DP-SGD. The columns from left to right present the required number of steps to convergence, final rewards upon convergence, the average correlation between values and rewards-to- and attack accuracies. In DP-SGD, we use the noise variance as the x-axis. We define the differential privacy budget 6, and plot a vs. in the Appendix.
6.3 VULNERABILITY FACTOR
Due the exploitation feature of RI, when we place an agent in its training environment, the agent tends to take the 'best' trajectory that it has experienced during training. Therefore, the value function can accurately predict the rewards-to-go causing the correlations from training environments to be significantly higher than correlations from validation environments. Hence we can achieve above 85% attack accuracy simply by using a logistic regression classifier to find the threshold between the training and validation correlations.
CriticDefense significantty reduces the training correlation and shrinks the gap between the train- ing correlation and validation correlation. We can observe that the attack accuracies decreas- ing as the training and validation correlations approach each other. Therefore, we conclude that the correlation between values and rewards-to-ge is the primary source of privacy vulnerability to CriticAttack.
Since CriticDefenss can reduce the correlation by only adding small amount of noise (e.g., 9 0.3), it is sufficient to protect the value function against CriticAttack while maintaining the agent's performance. We also present specific examples support our claims in the Appendix.
CONCLUSION
In this work, we introduce an effective and efficient black-box membership inference attack named CriticAttack that concentrates on the value function of the actor-critic algorithm. empirically demonstrate the high vulnerability of the value function of the actor-criti algorithm to MIAS by showing approximately 90% attack accuracies. Therefore, RL services should provide users with the least possible access to the value function. We then design corresponding de- fense method called CriticDefense, which can significantly reduce the attack accuracies of Crit- icAttack without hurting the target agent's performance. A limitation of the current work is that CriticAttack only works for actor- critic algorithms. We can generalize this MIA to other reinforce- ment learning algorithms consisting of value functions future direction.

---

Under review as conference paper at ICLR 2023
REFERENCES
Martin Abadi, Andy Chu, lan Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, New York, NY, USA, 2016. ACM.
Borja Balle, Maziar Gomrokchi, and Doina Preuuar Differentially private policy evaluation, 2016a.
Borja Balle, Maziar Gomrokchi, and Doina Precup. Differentially private policy evaluation. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd Interna- tional Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Re- search, pp. 2138, New York, New York, USA, 20-22 Jun 2016b. PMLR. URL https: //proceedings... .ml..iinns..m...m...mmmmmm.m wateas//48/aaleee6/ html
Xin Chen, Tao Zhang, Sheng Shen, Tianqing Zhu, and Ping Xiong. An optimized differential pri- vacy scheme with reinforcement learning in vanet. Computer Security, 110:102446, 2021.
Maxime Chevalier- Boisvert, Lucas Willems, and Suman Pal. Minimalistic for openai gym. https://ggthh:i com/ www.//////mme/eeeeee /ttp//awxxa///////////m////t//tt catmeeebgggmmmmiiiiii gam-gyiiigiiiii 2018.
environment
Evrard Garcelon, Vianney Perchet, Ciara Fike-Burke, and Matteo Pirotta. Local differential pri- vacy for regret minimization in reinforcement learning. In NeurlPS, 2021.
Maziar Gomrokchi, Spencer Main, and Donis Precup. Privattack: A membership inference attack framework against deep reinforcement learning agents, 2020.
Maziar Gomrokchi, Susan Amin, Hossein Aboutalebi, Alexander Wong, and Doina Precup. Where did you learn that from? surprising effectiveness of membership inference attacks against tem- porally correlated data in deep reinforcement learning, 2021.
Tuomas Haarnoja, Aurick Zhou, P. Abbeel, and Sergey Levine. Soft actor critic: Off- policy maxi- mum entropy deep reinforcement learning with a stochastic actor. ICML, 2018.
Hongsheng Hu, Zoran Salcie, Gillian Dobbie, and Xuyun Zhang. Membership inference attacks machine learning: A survey, 2021.
Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. On the effectiveness of regularization against membership inference attacks. ArXiv, abs/2006.05336, 2020.
Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information process- ing systems, 12, 1999.
Jan Kukačka, Vladimir Golkov, and Daniel Cremers. Regularization for deep learning: A taxon- omy, 2017.
Andrey Kurenkov, Ajay Mandlekar, Roberto Martin Martin, Silvio Savarese, and Animesh Garg. Ac-teach: A bayesian actor-critic method for policy learning with an ensemble of suboptimal teachers. In CoRL, 2019.
Jonathan Lebensold, William Hamilton, Borja Balle, and Doina Precup. Actor critic with differ- entially private critic, 2019a.
Jonathan Lebensold, William Hamilton, Borja Balle, and Doina Precup. Actor critic with differ- entially private critic. arXiv preprint arXiv:1910.05877 2019b.
Chonghua Liao, Jiafan He, and Quanquan Gu. Locally differentially private reinforcement learn- ing for linear mixture markov decision processes. ArXiv, abs/2110.10133,, 2021.
Milad Nasr, R. Shokri, and Amir Houmansadr. Machine learning with membership privacy using adversarial regularization. Proceedings of the 2018 ACM SIGSAC Conference Computer and Communications Security, 2018
Xinlei Pan, Weiyao Wang, Xiaoshuai Zhang, Bo Li, Jinfeng Yi, and Dawn Song. How you act tells a lot: Privacy-leaking attack deep reinforcement learning. In AAMAS, pp. pp.368-33 368-376, 2019.
10

---

Under review as conference paper at ICLR 2023
Anton Raichuk, Piotr Stanczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, L'oonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, and Sylvain Gelly. What mat- ters for on-policy deep actor- actor-critic methods? large-scale study. In ICLR, 2021.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. White- box vs black-box: Bayes optimal strategies for membership inference. In ICML, Cambridge, MA, USA, 2019. MIT Press.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference tacks against machine learning models. 2017 JEEE Symposium on Security and Privacy (SP), 1, 5 20177 doi 10. 10.1109/sp.2017..2
URL httL: http::://.....o.rg//1ooooo.. 1001100.1199/11 1109/51090000 /tw..//.... 2017.
Toshiaki Takano, Haruhiko Takase, Hiroharu Kawanaka, and Shinji Tsuruoka. Effective reuse method for transfer learning in actor-critic. In SCIS & ISIS SCIS & ISIS 2010, pp. 137-141. Japan Society for Fuzzy Theory and Intelligent Informatics, 2010.
Weihang Wang, Minghui Min, Liang Xiao, Ye Chen, and Huaiyu Dai. Protecting semantic trajec- tory privacy for vanet with reinforcement learning, ICC 2019 IEEE International Confer-
ence on Communications (ICC), pp. 1-5, 2019.
Lucas
Willems.
Rl-starter-fiic files.
https:///gthhub.com/ccsss/ 1emms
--ttrteerrflleee 2018
Zhiyuan Xu, Dejun Yang. Jian Tang, Yinan Tang. Tongtong Yuan, Yanzhi Wang, and Guoliang Xue. An actor- actor-critic-l based transfer learning framework for experience-drive driven networking. IEERIACM Transactions on Networking, 29(1):360-371, 2020.
Yunhao Yang, Parham Gohari, and Ufuk Topcu. On the privacy risks of deploying neural networks in machine learning models, 2021.
Zuobin Ying, Yun Zhang, and Ximeng Liu. Privacy-preservey in defending against membership inference attacks. Proceedings of the 2020 Workshop on Privacy-Preeserving Machine Learning in Practice 2020.
Shangtong Zhang and Shimon Whiteson. Dac: The double actor-critic architecture for learning options. Advances in Neural Information Processing Systems, 32, 2019.
11

---

Under review conference paper at ICLR 2023
A APPENDIX
A.1 ACTOR-CRITIC ALGORITHM
Algorithm 2: ACTOR-CRITIC
Input: policy xg, value function @ environment E (,,,,,,,,,,, discount factor y,
learning rate a, episode r
Output: optimized policy no, optimized value function V + for k= r do
Initialize first state timestamp 1 0
while s; is not terminal do
P(s,, T,, R(s)); /+ take action ai observe the next state 8,+1 and the reward Ty*/ Compute (8,)
d; ~ [p)
compute advantage A7 and reward-to-go Fr: /+ varied by estimation methods, stated in Equation 0 0 aV (oggoggg(a)s)) Φ = Φ av ( (s,) 2;)2;
* update parameters in actorx/
/x update parameters in critic
&
= ++1
end
end
A.2
CRITICATTACK PIPELINE
Agent
Prjjectry
Shadon Agent
X
Parton
Drvisemee 1et
Seop det "aaerbey Buat for Beals Bnseamment
is Than Sharkee Agent
Tapetay
tol Labal
X
(hajectory Label
Sabeced Enarnneett
Ataccer
kieanaa MAA antte Earget Agent
Thaldday Sat. "OuT Label
Step
Figure CriticAttack Pipeline.
A.3 PROOF OF THE N-STEP ADVANTAGE IN ALGORITHM
Theorem A.I. In the N-Step Advantage, given the (I+ 13th advantage AT AT then the 7t advan- tage is
1 y Y (rI#)- V (8r) +y (s++1)
(5)
y ( (5)+
Proof. We have shown the N-Step Advantage in the Background section:
A
Σ
Y Tt=k+1 + y V (S++N=1) -V. (8y)
k=
(6)
Fral + + -y TON
+ +y V (s -V (s)).
12

---

Under review conference paper at ICLR 2023
The (t 11 advantage is
N-1
AT
Σ y T r+k++ +++tk++++ +y (T -V (5p (8(+1)
(7)
yr++3 +... +g Tr+N+1
+ +y V/ (Sr+N+2) (Sc+1).
Then, compute
YA 1+IN = yTr+2 + 2 + + T++N+1
(8)
+yN- (s -VV (5++1),
hence
A - F(+ - TIN (51)
(9)
-V x (S++N+2) + v (s
AN = YA (1141- V2 +y (S(+1)
(10)
A.4
COMPARISON BETWEEN EXISTING MIAS
Attacker
Multi-Rooms
Door-Key
Lava-Crossing 7..4
Your-Roome
Pan et al. 2019
82.3 1.9
7..
87.2 1.6
Yang al. [2021
93.9 1.2
93.5
91.6
96.1 16.1
CriticAttack (LR)
89.7
89.7
90.2
92.5 1.0
CriticAttack (DNN)
9.3
9.3
94.2
97.1 0.6
Table MIA accuracies (mean standard deviation) across five repetitions. We use GAE for all the results.
A.5 MORE EXAMPLES OF CRITICATTACK
We present more examples of CriticAttack all three advantage estimation methods in Figure We also show the value trajectories and reward-to-go trajectories collected from a selected environment which is 'in' agent 1's training dataset but 'out' of agent 2's training dataset. We can observe the value and reward-to-go trajectories generated by agent are highly correlated compared to agent 2.
5
We also perform CriticArtack on target agents whose training data sizes are varied. We attack the target agents trained using 10, 20...., and 100 environments and observe how the training data size affects the attack accuracy. We train the shadow models using identical numbers of training environments as the target model. We present the results in Figure
We can observe negative correlation between the training data size and the attack accuracy. Increasing the training data size will improve the RL agent's generalization power, reducing the attack accuracy. However, the impact of training data size to attack accuracy is insignificant. can only reduce the attack accuracy by approximately 5% by doubling the training data size. Simultaneously, the training time significantly increases while we have more training data.
A.6 DIFFERENTIAL PRIVACY
Differential Privacy is a technique for describing the overall patterns of the dataset while conceal- ing the individual information. It guarantees that two adjacent sets of objects have indistin- guishable impact on the overall outcomes by adding a proper amount of statistical noise to the raw data.
13

---

Under review conference paper at ICLR 2023
TD Advantage
N-Step Advantage
GAE
0.8
value
0.6
0.4
0.5
0.5
0.2
0
0
0
20
40
60
20
40
60
20
40
60
Value
Reward-to-ge
0.6
value
0.4
0.4
0.5
0.2
0.2
0
0
20
40
60
0
20
40
60
0
20
40
60
step
step
step
predict in, actual in
predict out, actual out predict in, actual out
predict out, actual in
probability
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.6
0.8
1
0.4
0.6
0.8
0.6
0.8
40
40
40
votes
20
20
20
0
0
0400
0.4
0.6
0.8
0.4
0.6
0.8
0.6
0.8
correlation
correlation
correlation
Figure 5: The first and second rows show the value and reward-to-ga trajectories collected from selected environment Eo, generated by two separate agents: agent and agent Note that Eg the training dataset agent but out of the training dataset of agent 2. The third and fourth rows show CriticArtack results where the attacker uses logistic regression classifier and deep neural network, respectively.
logistic regression
mumberr of steps
0.92
MIA Accuracy
0.9
0.88
20
40
60
80
100
20
40
80
100
waining
size
Figure 6: the left figure MIA results on target agents with varied numbers of training environ- ments. These target agents trained by GAE advantage for the Multi-Room task. The right figure shows the number of steps required for convergence.
14

---

Under review conference paper at ICLR 2023
DefinitionA.I. Let be a given function. Define two datasets D and D', both in the input domain of f, adjacent if the number of entries in which the two datasets hold different values at most one.
After defining the adjacent dataset, we can define (c, (c.,))-diffeeette privacy as the following: Definition A.2 ((c,))-DIS DIFFFERDEEE PRIVAcY). Let e be a positive real number and M be a ran- domized algorithm that takes dataset input. Let e be the image of M. The algorithm M is e-differentially private if, for all adjacent datasets Dj and D2, and all R S
P(M(Di) R] exp(c)))|M(D))) E R] + 6,
(11)
where captures the probability that c-differential privacy fails. If 8 then say M is an C-DIFFERENTIAL PRIVACY.
DP-LSL [Balle al. 2016a Lebensold et 2019a is pretraining protection mechanism that construct a differentially private value function (critic network). It achieves differential privacy by adding Gaussian noise the critic's parameters before running the actor-critic algorithm.
In the initialization phase, we construct differentially private critic in the actor- citorrcritic algorithm by adding Gaussian noise the critic's parameters. DP-LSL refers the process of adding noise, which takes the differential privacy budget and ö as input.
Then, we apply CriticAttack the actor-critic algorithm with the differentially private critic. We present the MIA results in Table 3 and show that the DP-LSL has negligible effect on protect- ing against CriticAttack. Note that we obtain the results by performing MIAs on the actor-critic algorithm with GAE.
The work [Lebensold al. 2019a empirically shows that DP-ISI. can achieve differential privacy with minimal loss in performance. However, our experiments demonstrate that DP-LSL. does not affect protecting against CriticAttack. We present the results on Table
e
6
LR MIA
DNN MIA
Final Reward
10
2e-5
9.1
91.1
0.973 + 0.012
1
2e-5
9.
90.6 1.8
0.967 + 0.009 +
0.1
2e-5
1..
91.
Table 3: MIA accuracy and performance loss under DP-LSL. We report the (mean standard deviation) tuple across five repetitions.
DP-SGD (Abadi al. 2016 is a standard approach that helps deep learning models satisfy dif- ferential privacy. DP-SGD modifies the stochastic gradient descent in the training algorithm to enforce differential privacy to the algorithm itself. DP-SGD modifies the stochastic gradient de- scent in the training algorithm of the deep learning model to enforce differential privacy to the algorithm itself.
During the training procedure, DP-SGD first clips the gradients computed over the training data; then applies the Gaussian mechanism to add statistical noise drawn from a defined Gaussian distribution to the gradients; finally updates the model with the noisy gradients. Let 0 be the parameters of the deep learning model; the DP-SGD works as the following:
Σ
0j+1 0; B
Mgauss
(12)
where 0 is the parameters of the deep learning model at iteration i; a, are the learning rate and the batch size of the training algorithm; g(x, C) the clipping function defined by
gxx C) x -in
C
[xll
C, are the clipping value and the noise standard deviation of the DP-SGD algorithm.
We define the Gaussian mechanism M Mgauss f(x) n, where . (0,00
15

---

Under review conference paper at ICLR 2023
Estimation(Attacker)
Multi-Rooms
Door-Key 52.1(6.7)
Lava-Crossing
Four-Rooms
TD Advantage (LR)
55.6(8.9)
53.8(11.2)
56.3(7.5)
TD Advantage (DNN)
62.4(8.9)
64.6(6.7)
58.1(11.2)
65.2(7.5)
N-Step (LR)
56.3(9.9)
50.8(7.5)
58.1(11.9)
54.2(8.8)
N-Step (DNN)
63.6(9.9)
59.7(7.5)
64.1(11.9)
58.4(8.8)
GAE (LR)
53.7(8.4)
51.5(7.2)
55.9(9.7)
52.6(8.1)
GAE (DNNN)
64.4(8.4)
60.8(7.2)
65.7(9.7)
62.5(8.1)
Table 4: MIA accuracies (corresponding performance loss in percentage) across five repetitions.
DP-SGD Privacy Budget We apply the DP-SGD algorithm to enforce differential privacy to actor-critic reinforcement learning. We set the gradient clipping value Celip 0.2, epoch = 4, batch size & = 256, privacy offset le 1ee-4 4, and sample size 1000. We show the privacy budget at each noise variance in Figure
10a
10
epsilon
10a
10
10
10
10a
sigma
Figure 7: DP-SGD privacy budget at each noise level.
A.7 EXAMPLE OF DEFENSE AN ENVIRONMENT
Wpapll run several agents on the same environment Ep to collect value and reward-to-pe trajec- tories. The first agent is unprotected and has experienced En in its training phase, which means E, is 'in' the training dataset of the first agent. The second, third, and fourth agents are agents protected by DP-SGD with different noise variances during training. Ep is 'in' the training dataset of these three agents. The fifth agent is unprotected and has experienced Eg during training. We present five sets of trajectories in Figure respectively.
The correlations between the value and reward-to- to-go trajectories in the five rows 0.88, 0.83, 0.71, 0.32, and 0.26.
We observe that smaller amount of noise does not affect the correlation between the rewards- to-go and values, as shown in the second and third rows in Figure g Instead, a small amount of noise only changes the trajectories' smoothness. We must increase the noise variance to reduce the correlation, as in the third and fourth rows in Figure 8, However, we already significantly degrade the agent's performance by introducing a large noise.
A.8 MORE DEFENSE RESULTS
16

---

Under review as conference paper at ICLR 2023
Value
Reward-to -o-go go
CriticDefense
DP-SGD
value
0.5
20
40
20
90 0.1, PT 0.96, PF 0.74
0.04, ry 0.85, P 0.78
value
0.5
0.5
20
40
60
20
40
60
9 0.3, FT 0.93, PE
0.61
0 0.16, rT 0.63, pg 0.69
value
0.5
0.5
20
40
60
20
40
60
90 0.5, = 0.85, PE 0.37
0.64, Fr 0.32, PE 0.39
valui
0.5
20
40
20
Figure 8: Each figure shows the value and reward-to-go trajectories from selected environment under various conditions.
17


Reviews:
[{"review_id": "hdHzQzHnJ_", "reviewer": "Reviewer_Xm55", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the vulnerability of actor-critic reinforcement learning in the context of membership inference attacks. The attacker aims to make inferences about the training environments based on the outcomes of an RL algorithm. From the attack perspective, this paper proposes CriticAttack, which examines the correlation between the expected reward and the value function. CriticAttack only requires knowledge about the value function and expected rewards, which requires less information than prior work. From the defense perspective, this paper proposes CriticAttack, which inserts uniform noise to the value function to reduce correlation. Both CriticAttack and CriticDefense are evaluated empirically.", "summary_of_the_review": "This is an interesting paper, but some steps in the main methods are not well-supported. ", "strength_and_weaknesses": "Strength:\n1. This paper relaxes the assumptions in existing work by providing a black-box attack;\n2. The idea of examing the correlation between expected rewards and value function is interesting;\n3. Both attack and defense are studied.\n\nWeakness:\n1. there are not enough explanations for some steps in the proposed methods:\na. In CriticAttack: why performance on validation environments is used as stopping criteria?\nb. In CriticAttack: when does the correlation technique work? Do we need to assume the target agent is trained with a set of similar environments and the other environments are quite different?\nc. In CriticDefense: why do we use uniform noise instead of zero-mean noise (e.g. Gaussian)?\nd. In CriticDefense: why do we use modulo instead of clipping to make sure the values are between 0 and 1? I understand modulo allows a more significant change, but I also think modulo could result in a worse policy.\n\n2. There is no comparison in terms of performance between CriticAttack and existing methods. More experiments are needed to understand whether the 90% accuracy is a good accuracy. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "3x1ZCU1trZ", "reviewer": "Reviewer_JgPB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript proposes a binary classifier that determines if an environment is used in the training of an agent, by inspecting $n$ trajectories that are generated from that environment. The manuscript claims this to be an attack toward a value function of an RL agent. The manuscript subsequently proposes to perturb the value function so as to mitigate the capability of making the aforementioned binary classification, and proclaims this to be a defense method. Both the classifier and the prevention of the classifier are evaluated in an empirical manner.", "summary_of_the_review": "The manuscript has spent a decent effort on trying to figure out an attack method towards RL but I do not see either the setting or the method making sense.", "strength_and_weaknesses": "Pros:\n\n1. Investigations on RL algorithms' vulnerability and corresponding attack and defense methods seem relevant. \n\nCons:\n\n1. It is not very realistic to assume the release of the value function. The manuscript provided a reason that releasing the value function could be for the customer's fine-tuning, which is kind of a stretch. In practice it is the converse: The critic could be discarded once one finishes the training. The actor is the only component that is released.\n\n2. A binary prediction of whether a set of trajectories are used in the training does not sound to be anywhere an attack. Of course some trajectories could carry sensitive information, but most of the trajectories wouldn't. The manuscript does not provide a characterization on how much sensitive information each trajectory would carry and how the prediction works in terms of sensitive information on could obtain. The so-called 90% accuracy also doesn't make much sense under this view.\n\n3. Based on 2, the so-called defense also doesn't make sense, because this defense only protects one from this specific attack (which is not necessarily an attack). To be specific, adding some noise towards the value function only prevents one from figuring things out from the value function, but one could still possibly figure things out from the induced policy. The relevance of this defense is questionable.\n\n4. The presentation of this manuscript is weird: Several proclaimed names are given without introducing the specific methods; The attack algorithm is introduced even before introducing the attack's goal; The algorithm is more like a list of existing policy evaluation methods; The manuscript is filled up by equations but I don't see them informative or even needed; Notations like % and $\\oplus$ are used without definition; Weird usage of mathcal; etc.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1WpU6ra9Uq", "reviewer": "Reviewer_x7wn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies how to launch the value-based membership inference attack on actor-critic reinforcement learning. Such attacks may make inferences about the training environments—whether a particular environment has been used in training—by observing the outcomes of a reinforcement learning algorithm.\nThey develop CriticAttack, a new membership inference attack that targets black-box RL agents by examining the correlation between the expected reward and the value function. They empirically show that CriticAttack can correctly infer approximately 90% of the training data membership by using the MiniGrid toolkit. To defend against CriticAttack, they designed a method called CriticDefense that inserts uniform noise into the value function. CriticDefense can reduce the attack accuracy to 60% while degrading no more than 10% of the agent’s performance.", "summary_of_the_review": "This paper proposes a new attack with a simple defense method to mitigate such an attack, which suffers some limitation on the novelty, the significance of the work, and experimental evaluation.", "strength_and_weaknesses": "This paper proposes a new attack with a simple defense method to mitigate such an attack, which is evaluated by experiments. The main evidence of this paper is experimental evaluation, but all experiments are done on a single task (i.e., reach a target destination without bumping into obstacles) on MiniGrid toolkit. Let alone the proposed attack only works for actor-critic algorithms, and it is questionable if the proposed attack can attack other tasks.\n\nI am not sure if the proposed attack is realistic. If the training datasets are known, why is it interesting to infer whether a data record is used in training the shadow model?\nIf the training datasets are known, we can train the target model. Why do we need to infer the target model from the trained attacker?\n\nAbout environment-based MIA and trajectory-based MIA, which\none is easier? This paper focuses on environment-based MiA, but I think trajectory-based MIA is harder because an environment could have many trajectories. In addition, why do we care about inferring the environment when the information about the environment of the model is usually public?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject