Title: Toward Efficient Inference Attacks: Shadow Model Sharing via Mixture-of-Experts
Venue: NeurIPS 2025
Accepted: yes
Keywords: shadow model technique

data privacy

membership inference attacks
Abstract: Machine learning models are often vulnerable to inference attacks that expose sensitive information from their training data. Shadow model technique is commonly employed in such attacks, like membership inference. However, the need for a large number of shadow models leads to high computational costs, limiting their practical applicability. Such inefficiency mainly stems from the independent training and use of these shadow models. To address this issue, we present a novel shadow pool training framework SHAPOOL, which constructs multiple shared models and trains them jointly within a single process. In particular, we leverage the Mixture-of-Experts mechanism as the shadow pool to interconnect individual models, enabling them to share some sub-networks and thereby improving efficiency. To ensure the shared models closely resemble independent models and serve as effective substitutes, we introduce three novel modules: path-choice routing, pathway regularization, and pathway alignment. These modules guarantee random data allocation for pathway learning, promote diversity among shared models, and maintain consistency with target models. We evaluate SHAPOOL in the context of various membership inference attacks and show that it significantly reduces the computational cost of shadow model construction while maintaining comparable attack performance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the computational inefficiency in shadow model construction for inference attacks (e.g., membership inference) against machine learning models. The authors propose SHAPOOL, a framework leveraging the Mixture-of-Experts (MoE) mechanism to train multiple “shared” shadow models within a single process. Experiments across multiple datasets and settings demonstrate a significant reduction in computational costs, as well as competitive attack performance, compared to traditional approaches.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The proposed issue, computational cost of training shadow models, is a pressing bottleneck in the privacy attack community, especially for attacks such as LiRA.\n- Adapting MoE to address the issue is novel.\n- Comprehensive experiments across multiple settings demonstrate the effectiveness of the proposed method. \n\n\n### Weaknesses\n- Some parts seem confusing to follow.\n- Experiments could be further enhanced.\n\nQUESTIONS:\n- I'm a bit confused by the description of the method in Module 3: Pathway Alignment. The aim is to better mimic the behaviors of the target model. Does this process require access to confidence scores from the target model, align results only between experts within the MoE, or simply fine-tune certain experts using additional subsets of the training data?\n\n- The introduction discusses three existing methods for efficient shadow model training: QMIA, RMIA, and SNAP. But the experimental section only provides a direct comparison with RMIA. Is there any reason why QMIA and SNAP were not included in the evaluation?\n\n- I understand that the authors focus on three specific factors in Section 3 to maintain consistency with the subsequent challenges and method design. I wonder if there were additional aspects that could impact MIA performance. Exploring or discussing the potential impact of these additional factors would be more intriguing. \n\n- It's better to provide ROC curves for MIA result comparisons.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel method of training shadow models for inference attacks that is more computationally efficient and in specific settings outperforms the baseline method of training shadow models independently.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. Their method is novel and interesting.\n2. The improvements to computational cost appear significant.\n\n**Weaknesses**  \n_1. Unsubstantiated Claims_  \nThere are some claims that are left entirely unsubstantiated in the paper. Specifically, the authors make the claim that their method is \"more generalizable\" than prior work. However, I do not fully understand the claim that prior work is \"coupled with specific algorithms\" and is not cited in the Introduction. What do the authors mean by this and where does this claim come from? Furthermore in the Evaluation section under \"Extension to Other Inference Attacks\", the authors claim that Table 3 shows that their method generalizes across different attacks. But I do not see how Table 3 shows that since the attack they compare against (RMIA) is also a membership inference attack. I was expecting the authors to compare their SHAPOOL method with LiRA and RMIA across different attack types, e.g., property inference and model inversion in order to make this claim.\n\n_2. Additional Experiments_  \nI am not entirely convinced that the authors' main claims are fully substantiated either. First, Tables 1 and 2 are missing TPR@FPR=0.1%, even though the authors had indicated this was a metric they were interested in studying. Furthermore, I cannot find full ROC curves for any of the experiments and this is not discussed in the experiments. Specifically, I would like to see whether there are regions of FPRs that their method underperforms prior work. Lastly, Tables 1 and 2 only really compare one configuration for the constrained and un-constrained settings, each. I do not think this is enough for the authors to claim that the computational cost gains are always justified. Here, I think it's important for the authors to vary the number of shadow models and shadow pools for LiRA and SHAPOOL, respectively and compare the two based on the computational cost. Basically, the analogue of Figure 3 from the arXiv version of the RMIA paper.\n\n_3. Unfair comparisons_  \nI am not really sure why the authors are comparing their method against LiRA under the constrained scenario. Shouldn't they compare against RMIA since it is already known that RMIA beats LiRA in resource-constrained settings? Also it seems odd that Table 3 uses 64 shadow models for RMIA when 4/8 is supposedly enough. Again going back to point 2. this I believe further motivates the authors to do a comparison between SHAPOOL, LiRA, and RMIA across different number of models/computational costs.  \n\n_4. Writing Improvements_  \nJust some minor points, \"Bhattacharya coefficient\", \"CINC10\" are missing citations. Hardware details (i.e., number and type of GPUs) also seem missing from the Experimental Details section.\n\nQUESTIONS:\n1. Could you explain what \"Extension to Other Inference Attacks\" means and what you mean by prior work is \"coupled with specific algorithms\"?\n\n2. How does the full ROC-curve comparison between your attack and LiRA & RMIA look like for the experiments you have already run?\n\n3. Does your attack perform as well as RMIA in the low reference model (<= 4) setting?\n\n4. Does your attack outperform (in terms of computational time and attack performance) LiRA across a range of fixed compute budgets?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SHAPOOL, a shared shadow model training framework based on MoE, to reduce the computational cost of inference attacks through joint training with subnetwork sharing. The paper designs three main modules Pathway-choice Routing,  Pathway Regularization, and  Pathway Alignment to address the real-world challenges of MoE, and experiments show that the framework is able to have a low computational overhead against membership inference attacks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n\n- Although the use of MoE to enhance the performance of shadowing models is rather intuitive, it is also a way to bring new research perspectives to existing studies.\n- The article has a logical narrative, relatively adequate experiments, and open source code.\n\nWeaknesses：\n\n- Many of the statements in the main text reflect that the paper focuses on inference attacks rather than specifically on MIA, but only MIA is used in terms of both framework design and experimental evaluation. Therefore, the paper lacks focus in some of the narratives and lacks experimental validation of other inference attacks such as FIA (in fact, other inference attack methods may not require the use of shadow models, which also leads to a conflict with the concept of inference attack referred to in the article).\n- In addition to the intuitive introduction of the MoE approach, the design of the three core modules is actually rather intuitive, so I am pessimistic about the real contribution of these approaches.\n- An obvious flaw in the experimental design is that it only compares BASELINE and lacks comparisons with other shadow model building methods. As far as I know this part of the work is very extensive, so did the authors forget to compare with these or was the performance limited?\n- The experiment results are missing statistical data (e.g., variance, etc.).\n- The data font sizes in Figures 3 and 6 are too small and can be adjusted subsequently.\n\nQUESTIONS:\n- Why did the performance comparison in the experiment only compare BASELINE and not the other  related methods?\n- How many times has the experiment been run and how stable is it?\n- Considering that the construction of shadow models is actually very dependent on training data, is the performance of the model maintained when only a small amount of data is available?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper aims to tackle the problem of efficient membership inference attacks—specifically, how to reduce the number of shadow models required for such attacks. To this end, the paper proposes a novel shadow pool training framework, SHAPOOL, which constructs multiple shared models and trains them jointly within a single process. In addition, the authors introduce three novel modules—path-choice routing, pathway regularization, and pathway alignment—to ensure that the shared models closely resemble independently trained models and can serve as effective substitutes.

STRENGTHS AND WEAKNESSES:
Strengths:
- The studied problem is interesting and important to the community.
- The motivations for the studied problem and the proposed solution are both very sufficient.
- The writing and the structure of the paper are good.
- My expertise is not in the field. However, the paper provides necessary background information for me to understand the problem setting.
- The proposed method makes sense in general.

Weaknesses:
- Experiments: The existing experiments are conducted Cifar10, Cifar100, and CINIC10. Could you also provide some experiments on datasets of large image size, e.g., ImageNet-Subset?

QUESTIONS:
See Weakness above.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-02