reviews: 
Comments: This work tackles document-level inference for VQA models that have been trained on text as well as visual-documents, where additional processing steps like OCR and repeated instances of documents make the problem significantly different from standard MI under language models. The authors propose attacks that are free of any auxiliary data, for both black-box and white-box access models.  The proposed attacks are evaluated on multiple datasets, outperforming both existing and newly-designed baseline inference methods.
Strengths: - The work tackles document VQA systems, which post a lot more challenges than simple text-based language models. Understanding leakage in this scenario would be helpful.
- The attack methodology is interesting and straightforward, and the authors experiment with multiple ways of "fine-tuning" the target model to measure scores used for membership inference.
- Results are presented very well, and overall the figures/tables are very neat, with a very detailed Appendix.
Weaknesses: I think the works make a fair contribution, but I do have some concerns (major and minor listed below and under questions) that the authors could answer to help me better understand some of their design choices.

- The evaluation design is completely disconnected from how MI attacks are usually evaluated. Currently, thresholds are computed based on average values in $D_\text{test}$. In MI (or similar evaluations), the standard protocol is to sweep over the non-member's values and based on the FPR (false positive rate) dictated by a certain threshold, compute the TPR (true positive rate). From there, multiple statistics (such as AUC for the TPR-FPR curve, or TPR at a specific FPR) can be computed. I would request the authors to re-evaluate their methods using this approach. I also do not know why the authors have constructed their own version of baselines- there are existing attacks (as one example, see Min-k%++)

- Appendix E.1: "... have assumed complete knowledge of the original training questions" This is a very strong assumption! This implies that the adversary also knows the exact questions associated with the target document (and by extension the answers). This was not mentioned clearly anywhere in the main paper. The fact that performance drops when this is relaxed is not surprising to me, and demonstrates that the majority of performance reported in the paper is originating from this strong (and obfuscated) assumption. The relaxed setting here (where exact questions are not known) should be the **default** setting in the main paper- after all, true "document" inference would not entail one knowing which *exact* questions and answers were present in the training data


## Minor comments

- L31: "...fuels a significant number of operations daily" - source?
- L49-50: "...utilize a dual representation..." - please cite relevant works
- L102-102: "The literature indicates that...." - this is not the case. While recent work [1] does suggest that parameter access may be necessary for optimal membership inference, the cited papers here (and most empirical results) indeed conclude that additional access to parameters does not help with membership inference. 
- L114: "...larger scale models"- please see [2] that indeed tackles inferring the use of certain text documents for LLMs.
- L186: "adversary lacks access to an auxiliary data". Membership-inference evaluation by design needs "non-members" to compute thresholds for membership classification (and as recent work [3] has shown, you cannot just use any non-member data especially when it comes to language). What the authors probably mean here is that there is not *enough* auxiliary data, and should clarify this.
- L191: "...would be prohibitively expensive" - this would be a problem for a resource-constrained adversary, but an adversary that is malicious (in the security sense) will not be limited like this. An adversary can always use public/pretrained models as a starting point.
- L210: "...types of documents" - define knowledge of "type" of documents/questions
- L447: "Table 1 (right)" - there is only one table. Please fix reference (similarly for Table 1 (left), etc.)

### References
- [1] Suri, Anshuman, et al. "Do Parameters Reveal More than Loss for Membership Inference?" HiLD, ICML (2024).
- [2] Maini, Pratyush, et al. "LLM Dataset Inference: Did you train on my dataset?" arXiv:2406.06443 (2024).
- [3] Duan, Michael, et al. "Do membership inference attacks work on large language models?" arXiv:2402.07841 (2024).
Recommendation: 6
Reviewer confidence: 5
Soundness/correctness: 3
Clarity: 3