Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This manuscript proposes a binary classifier that determines if an environment is used in the training of an agent, by inspecting $n$ trajectories that are generated from that environment. The manuscript claims this to be an attack toward a value function of an RL agent. The manuscript subsequently proposes to perturb the value function so as to mitigate the capability of making the aforementioned binary classification, and proclaims this to be a defense method. Both the classifier and the prevention of the classifier are evaluated in an empirical manner.
strength_and_weaknesses: Pros:

1. Investigations on RL algorithms' vulnerability and corresponding attack and defense methods seem relevant. 

Cons:

1. It is not very realistic to assume the release of the value function. The manuscript provided a reason that releasing the value function could be for the customer's fine-tuning, which is kind of a stretch. In practice it is the converse: The critic could be discarded once one finishes the training. The actor is the only component that is released.

2. A binary prediction of whether a set of trajectories are used in the training does not sound to be anywhere an attack. Of course some trajectories could carry sensitive information, but most of the trajectories wouldn't. The manuscript does not provide a characterization on how much sensitive information each trajectory would carry and how the prediction works in terms of sensitive information on could obtain. The so-called 90% accuracy also doesn't make much sense under this view.

3. Based on 2, the so-called defense also doesn't make sense, because this defense only protects one from this specific attack (which is not necessarily an attack). To be specific, adding some noise towards the value function only prevents one from figuring things out from the value function, but one could still possibly figure things out from the induced policy. The relevance of this defense is questionable.

4. The presentation of this manuscript is weird: Several proclaimed names are given without introducing the specific methods; The attack algorithm is introduced even before introducing the attack's goal; The algorithm is more like a list of existing policy evaluation methods; The manuscript is filled up by equations but I don't see them informative or even needed; Notations like % and $\oplus$ are used without definition; Weird usage of mathcal; etc.
clarity,_quality,_novelty_and_reproducibility: See the last part.
summary_of_the_review: The manuscript has spent a decent effort on trying to figure out an attack method towards RL but I do not see either the setting or the method making sense.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies how to launch the value-based membership inference attack on actor-critic reinforcement learning. Such attacks may make inferences about the training environments—whether a particular environment has been used in training—by observing the outcomes of a reinforcement learning algorithm.
They develop CriticAttack, a new membership inference attack that targets black-box RL agents by examining the correlation between the expected reward and the value function. They empirically show that CriticAttack can correctly infer approximately 90% of the training data membership by using the MiniGrid toolkit. To defend against CriticAttack, they designed a method called CriticDefense that inserts uniform noise into the value function. CriticDefense can reduce the attack accuracy to 60% while degrading no more than 10% of the agent’s performance.
strength_and_weaknesses: This paper proposes a new attack with a simple defense method to mitigate such an attack, which is evaluated by experiments. The main evidence of this paper is experimental evaluation, but all experiments are done on a single task (i.e., reach a target destination without bumping into obstacles) on MiniGrid toolkit. Let alone the proposed attack only works for actor-critic algorithms, and it is questionable if the proposed attack can attack other tasks.

I am not sure if the proposed attack is realistic. If the training datasets are known, why is it interesting to infer whether a data record is used in training the shadow model?
If the training datasets are known, we can train the target model. Why do we need to infer the target model from the trained attacker?

About environment-based MIA and trajectory-based MIA, which
one is easier? This paper focuses on environment-based MiA, but I think trajectory-based MIA is harder because an environment could have many trajectories. In addition, why do we care about inferring the environment when the information about the environment of the model is usually public?


clarity,_quality,_novelty_and_reproducibility: As mentioned by the authors, CriticAttack collects values from the value function and the cumulative reward for membership inference, and Gomrokchi et al. (2021) and Gomrokchi et al. (2020) introduce two membership inference attack methods to infer the roll-out trajectories in off-policy RL algorithms, which learn the optimal policy independently of the agent’s actions. I am not sure how hard to transfer from a trajectory-based membership inference attack to an environment-based based membership inference attack proposed in this paper.
summary_of_the_review: This paper proposes a new attack with a simple defense method to mitigate such an attack, which suffers some limitation on the novelty, the significance of the work, and experimental evaluation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the vulnerability of actor-critic reinforcement learning in the context of membership inference attacks. The attacker aims to make inferences about the training environments based on the outcomes of an RL algorithm. From the attack perspective, this paper proposes CriticAttack, which examines the correlation between the expected reward and the value function. CriticAttack only requires knowledge about the value function and expected rewards, which requires less information than prior work. From the defense perspective, this paper proposes CriticAttack, which inserts uniform noise to the value function to reduce correlation. Both CriticAttack and CriticDefense are evaluated empirically.
strength_and_weaknesses: Strength:
1. This paper relaxes the assumptions in existing work by providing a black-box attack;
2. The idea of examing the correlation between expected rewards and value function is interesting;
3. Both attack and defense are studied.

Weakness:
1. there are not enough explanations for some steps in the proposed methods:
a. In CriticAttack: why performance on validation environments is used as stopping criteria?
b. In CriticAttack: when does the correlation technique work? Do we need to assume the target agent is trained with a set of similar environments and the other environments are quite different?
c. In CriticDefense: why do we use uniform noise instead of zero-mean noise (e.g. Gaussian)?
d. In CriticDefense: why do we use modulo instead of clipping to make sure the values are between 0 and 1? I understand modulo allows a more significant change, but I also think modulo could result in a worse policy.

2. There is no comparison in terms of performance between CriticAttack and existing methods. More experiments are needed to understand whether the 90% accuracy is a good accuracy. 
clarity,_quality,_novelty_and_reproducibility: Quality: fair (refer to the weaknesses above)
Clarity: fair (the presentation of CriticAttack can be improved by using both description and pseudocode)
Originality: good
summary_of_the_review: This is an interesting paper, but some steps in the main methods are not well-supported.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the problem of Membership Inference Attacks (MIAs) on actor-critic RL methods. The paper introduces an empirical attack algorithm that can infer the membership of a given trajectory with good accuracy, given that the value function is released to the attack. As is pointed out by reviewer JgPB, empirically, it doesn't really make sense for the system designer to release such information in the first place. Overall, this paper seems to be half-baked and not ready for publication.
justification_for_why_not_higher_score: NA
justification_for_why_not_lower_score: NA
Decision: Reject