Improving Defense Mechanisms for Subgraph-Structure Membership Inference Attacks

Authors: Peizhi Niu, Chao Pan, Siheng Chen, Olgica Milenkovic
Primary area: learning on graphs and other geometries & topologies
Keywords: Graph neural network, subgraph-substructure membership inference attack, 2-stage training

Abstract:
Graph neural networks (GNNs) are of significant importance in diverse real-world applications since they leverage powerful graph learning techniques to solve problems pertaining to social network mining and medical data analysis. Despite their practical relevance, GNNs remain vulnerable to adversarial attacks such as membership inference attacks (MIAs) which pose privacy risks by revealing whether specific data records were part of the training set of the model. While most existing research has focused on designing defense mechanisms for known node-level MIAs, and in particular, for determining if a certain node was used during training, only limited attention has been paid to subgraph-structure MIA (SMIA) problems. SMIA methods seek to infer whether a set of nodes forms a particular target structure of interest (such as a graph motif, e.g., clique or multi-hop path) in the training graph. The main contributions of our work are three-fold. The first is a novel robust defense mechanism for GNNs against SMIA attacks. It  combines an alternating train-test schedule with a flattening strategy to mitigate the attacks. The second contribution is a new end-to-end SMIA attack model that outperforms existing attacks by using multiset functions to generate learnable embeddings for collections of nodes. Extensive simulations reveal that the new attack model outperforms prior state-of-the-art attack models on GNNs by 12.31% across four datasets when no defense mechanism is present. With the new defense mechanism, one can achieve an average decrease of 14.30% in the attack AUROC and an 10.05% improvement in target model utility compared to classical defenses, even when using the improved attack scheme. The third contribution is a study that shows that our defense mechanism extends to node-level MIAs as well, offering similar improvements in attack resistance and utility.

Decision: reject
Reviewer scores: 3, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1730179147239

thread_type:
official_review

<thread_object>
creator: reviewer_pglu

<content>
summary:
The paper focuses on the defense strategies for graph neural networks against subgraph-structure membership inference attacks (SMIAs).Specically, the paper defend against the state-of-the-art SMIAs using flattening. The authors also strengthen the current attacks. Extensive results demonstrate that the proposed defense method can decrease the attack performance.

strengths:
- Trendy topic
- Well-written paper

weaknesses:
- Unclear motivation
- Performance is not that good
- Lack of adaptive attacks

questions:
The paper proposed novel defenses and attacks on the domain of subgraph-structure membership inference attacks against graph neural networks. The authors first introduce the limitations of the previous state-of-the-art attacks. Then, the authors introduced the novel defense methods to defend against the attacks. Finally, the authors also improved the attacks. Experiences are conducted to demonstrate the effectiveness of the proposed defenses and attacks. The paper is well-written. However, I have several comments regarding motivation and performance.

- The authors proposed both defenses and attacks. The defenses can also mitigate the proposed new attacks. Therefore, I am wondering what is the motivation of the proposed attacks. The authors are supposed to explain why the proposed new attacks are important and what conclusions can be achieved by the new attacks.
- The defense method does not work that well. The attack success rate is still high enough to pose a threat. Therefore, the authors are suggested to discuss why the attacks can still achieve moderate performance.
- Instead of introducing new attacks, I recommend that the authors consider adaptive attacks against the proposed defenses. i.e., what can the adversary do if they know the defense methods?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730700358207

thread_type:
official_review

<thread_object>
creator: reviewer_8qxu

<content>
summary:
This paper discusses subgraph structure membership inference attacks (SMIA) on GNNs, which is an important yet unattended problem, and presents a novel defense mechanism. Specifically, for the attack method, the paper introduces an end-to-end SMIA attack model based on multiset functions, which seems to be a powerful attacker due to the order-invariant property. For the defense method, the paper designs a two-stage training strategy and a flattening technique to alter the posterior distributions and enhance defense capabilities.

strengths:
- The paper is well written and technically sound. 
- The attention on defenses on SMIA should be valued.
- The experiment are conducted thoroughly.

weaknesses:
I have the following concerns about the settings of attacker and the effectiveness of the defense methods.

questions:
*  For SMIA, the paper assumes that the attacker can access sufficient posterior distributions for an effective attack. However, it limits the range of application of this paper as that attackers might not be always available for obtaining such posterior information.

* In the defense, TSD uses soft labels to change the loss distribution, assuming that this label smoothing works well in all situations. However, for some tasks, using pseudolabels instead of actual labels might lower the model’s performance or cause unexpected biases.

* TSD assumes that it can remain effective across various types of GNNs and datasets. I am curious whether TSD can maintain the same level of performance on heterogeneous (not heterophilic) graphs is questionable.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730898960904

thread_type:
official_review

<thread_object>
creator: reviewer_uyli

<content>
summary:
This paper proposes a defense strategy against SMIA attacks. To defend sub-graph and label inference from GNNs, this paper devises a two-stage defense method (TSD) consisting of a flattening strategy and test set adjustment strategy.  The flatten strategy aims to introduce noises to the label distribution and the test set adjustment will reduce differences in outputs between the training set and the test set.

strengths:
1. Sufficient empirical comparisons. Authors present lots of comparisons with other defense baselines to demonstrate advantages of TSD.
2. Clear writing and easy-to-follow.

weaknesses:
1. Significant performance deterioration. Especially in Table3, after TSD defense, clean performance deteriorates from 80\% to 77\%. Performance deterioration problem is a significant problem in studies of defending against MIA attacks. I suggest authors also focus on tackling this problem.
2. Unclear motivation. The two-stage method is described clearly, but the motivation is unclear. For instance, Flatten strategy aims to introduce noises to the label distribution, but why could this help defend SMIA? In the second stage, does the test set adjustment aim to reduce differences in outputs between the training set and test set to defend against SMIA? If so, almost all defense motivation follows this motivation. Hence, I suggest more description or demonstration of motivations.

questions:
Please refer to weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731442026398

thread_type:
official_review

<thread_object>
creator: reviewer_or7l

<content>
summary:
In this paper, the authors explore the robustness of graph neural networks (GNNs) against membership inference attack (MIA), specifically subgraph-structure membership inference attack (SMIA) where the foal is to infer whether a set of nodes form a particular structure e.g., a clique or multi-hop path in the training graph. The authors propose an SMIA attack which uses multiset function to improve the impact of the attack. To combat the attack, they propose a two-stage defense strategy. In the first stage, the model is trained on the training set, with some one-hot labels converted to soft labels to add ambiguity to selected samples. In the second stage, this model is further refined on the test set, where pseudo-labels are generated based on the model’s predictions after the first stage of training. Experimental results are included to support the claims.

strengths:
1. The paper is well written.
2. Substantial experimental results are included in the paper.

weaknesses:
1. The methodology section of the paper is lacking. The authors include comparison with other attacks and defenses in the methodology section which is not recommended. This section of the paper should emphasize on the proposed solution and the novelty. It is not clear as to why the new attack is more effective. Some explanations should be provided.
 
2. Novelty of the paper is lacking. Flattening is used by other defenses too [1].

3. For the proposed defense solution, the authors predict the pseudo-labels for the test nodes which is then used for training the model further. What happens when we have a dynamic graph where new nodes are added? Will the defense be still effective?

4. No adaptive attack is developed against the proposed defense solution to show its effectiveness. 

[1] RelaxLoss: Defending Membership Inference Attacks without Losing Utility

questions:
1. The problem formulation is not very clear. If I have the adjacency matrix of the graph available, why do I need to create a model to predict if the nodes form a particular structure or not? What setup is used for the experiments here - transductive or inductive?

2. Why small scale datasets are used for the SMIA attack? More statistical information about the dataset (structure you are trying to find) should be included to show the motivation of the work.

3. What kind of adaptive attack can be developed against the proposed solution?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732786992923

thread_type:
official_comment

<thread_object>
creator: area_chair_ql1w

<content>
comment:
I would like to encourage the reviewers to engage with the author's replies if they have not already done so. At the very least, please 
acknowledge that you have read the rebuttal.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies subgraph-structure membership inference attacks (SMIA). it is not clear to me how relevant are SMIA in real-world settings, and I find some of the scenarios used for motivation unconvincing, nonetheless, this problem is interesting from a technical point of view. The paper proposes a two-stage defence where the first model is trained on the training set with flattening (introduced in prior work), then used to generate pseudo-labels for a second model trained on test set. The authors also propose a SMIA attack model using multisets (Deep Sets). Some reviewers found the motivation and methodology description unclear. In the initial submission the authors did not introduce any adaptive attacks against the proposed defence. In the rebuttal the authors provided additional experiments trying to address the issue with adaptive attacks, however, these attacks are not very convincing. For example the claim "the best strategy for the attacker is to train the shadow model in a standard manner without applying TSD." in the first scenario needs to be better substantiated. 

In future iterations I suggest the authors to better motivate the two-stage approach and consider additional improvements to reduce the accuracy drop. More importantly, I suggest the authors to spend more time on developing stronger adaptive attacks given that the defences is heuristic. These attack should be the main focus (part of the main paper rather than the appendix). Experiments on larger-scale datasets to validate scalability would also straighten the paper.