Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors proposed a Metric-based and a feature-based membership inference attack for multi-modal machine learning. The authors evaluated the  proposed attacks under unrestricted, data-only, and constrained settings on various datasets and architectures to show the effectiveness of the attacks. The authors also evaluated the effect of two mechanisms (preventing overfitting and using DP) to mitigate MIA.
strengths_and_weaknesses: Strength:
The authors proposed the first MIA against multi-modal ML.
The overall presentation of the paper is well organized and easy to read. 

Weakness:
The authors propose preventing overfitting (data augmentation + l2 reg) as a mechanism to mitigate MIA. These two techniques are commonly used in ML training, and I believe the attack should be evaluated on non-overfit target models. 

questions: 1. For Fig 8, shouldn’t the ROC be plotted in log-scale as Carlini etal. [73] suggested? Also, can data augmentation be used to improve the attack as in [73]?
2. For the unrestricted setting, does the shadow training dataset overlap with the target training dataset? Are there any differences in the attack success rate if the shadow training data does or does not overlap with the target training dataset? 

limitations: 1. In the paper, the authors mentioned the attack is not as effective when the target model is trained on MS-COCO dataset (the attack success rate is only slightly above 50%) and the reason might be that ROUGE and BLEU scores cannot capture the semantics well. Have you considered other scoring methods? 
2. The authors only evaluated the proposed on image captioning tasks. It would be nice if the authors can show that the attack is successful on other multimodal applications.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes an approach for inferring the membership status in the multi-modal data, called M^4I, which aims to study the privacy leakage problem in the multi-modal learning setting. Specifically, the authors introduce two attack methods for this purpose, including a metric-based and a feature-based. Authors test the proposed methods on only a task, i.e., image captioning. The experimental results show the advantage of the proposed method.
strengths_and_weaknesses: Strengths:
1. First work to explore the membership inference problem in a multimodal setting. 
2. Proposing a meaningful setting for the work.
3. Examining performance on four real-world datasets for one task.

Weaknesses:
1. The proposed models contain limited technical contributions.  For example, 
1) The metric-based model only employs two existing similarity metrics on the texts as a feature for classification. Meanwhile, the feature-based model adopts an existing pre-trained model and a two-layer neural network for classification. 
2) The reviewer is expected to see some theoretical results that can guarantee the discrimination errors. 

2. Although the paper claims that it is under the multimodality setting, it is strange that it misses the image modality information in the metric-based model.

3. The paper only evaluates the model in one task.  It is doubtful whether it can be extended to more tasks for evaluating the generalization ability.  Therefore, it is hard to draw a conclusion that this method can perform well on other multimodal tasks.

4. The experiment section lacks a detailed description and complete comparison.  For example, 
1) In Section 3.2 and the main figures, the authors mention that the feature-based model employs a pre-trained image encoder and a text encoder. However, it does not tell the exact pre-trained model in the experiment section.
2) The reviewer also doubts how the effect of the pre-trained models.  The paper lacks sufficient experimental comparisons on different pre-trained models. 
3) The experiments should add more intuitive baselines to demonstrate the effectiveness of the proposed methods.

questions: 1. Will different encoders yield different results?
2. How does the structure of the feature-based model affect the results? 
3. How does your model perform on other state-of-the-art image caption models?
4. Can the paper present a theory to guarantee the discrimination errors?

limitations: Yes
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: 
The paper studies the privacy leakage of multi-modal models, proposing a membership inference attack against multi-modal models. Two attack methods are introduced, the metric-based M4I and the feature-based M4I. In metric-based M4I, the adversary can score the data and use a threshold or a binary classifier to distinguish between the scores of member data and non-member data; while in feature-based M4I, a pre-trained shadow multi-modal feature extractor is used to conduct data inference attack. The experimental results show that both attack methods can achieve strong performances.
strengths_and_weaknesses: 
+ The topic related to multi-modal membership inference attacks is interesting and the privacy leakage of multi-modal models is an important security issue. As far as I know, this is the first research so far focusing on the membership inference of multi-modal models. Considering that the multi-modal model is one of the most rapidly rising technology in recent years, and its highly sensitive application scenarios, eg, medical, voices, or videos, the privacy risk in this domain is critical to the security community.

+ The proposed attack methods are practical and the experimental results indicate a strong performance. I appreciate the design of different scenarios (unrestricted, data-only, and constrained scenarios), which provide a thorough evaluation of the proposed attack methods. Both the metric-based and feature-based M4I are evaluated under different scenarios using different datasets. The performance is compared with the random guess, which is still reasonable,  considering that the multi-modal membership inference is a relatively new research area.

+ Mitigation of the proposed attack has been evaluated and discussed

It is always good to see the evaluation and discussion related to the mitigation of a proposed attack. Three countermeasures have been involved in the experiments. I agree that, although both metric-based and feature-based attacks are affected by the privacy-enhanced training, the trade-off can not be ignored; while the overfitting results could be a potential solution. It would be better to also discuss the trade-off of applying the defence by preventing overfitting.

I cannot find any major weaknesses that would keep me from accepting the paper but there are my two cents to further improve the paper.

- It would be nice if the three scenarios are explained and marked in Figures 4 and 6. The current figures blend the results of three scenarios in heat maps, which is good, but it could be hard for readers to compare the attack performance.


questions: Please mark the three scenarios explicitly in Figures 4 & 6.

limitations: See my advices above.
review_title: null

Meta-review: metareview: This work studies membership inference attacks for multimodal models. It proposes a few different attacks under different assumptions on the attack model, and evaluates them empirically.
The reviewers found the problem interesting and the paper well-written. The paper is a welcome addition to the literature on membership inference attacks and should be of interest to this conference. I would encourage the reviewers to address the feedback from the reviewers in the final version.  I recommend acceptance.
confidence: Less certain
Decision: Accept