Title: ImageSentinel: Protecting Visual Datasets from Unauthorized Retrieval-Augmented Image Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: RAG

watermarking

copyright protection
Abstract: The widespread adoption of Retrieval-Augmented Image Generation (RAIG) has raised significant concerns about the unauthorized use of private image datasets. While these systems have shown remarkable capabilities in enhancing generation quality through reference images, protecting visual datasets from unauthorized use in such systems remains a challenging problem. Traditional digital watermarking approaches face limitations in RAIG systems, as the complex feature extraction and recombination processes fail to preserve watermark signals during generation. To address these challenges, we propose ImageSentinel, a novel framework for protecting visual datasets in RAIG. Our framework synthesizes sentinel images that maintain visual consistency with the original dataset. These sentinels enable protection verification through randomly generated character sequences that serve as retrieval keys. To ensure seamless integration, we leverage vision-language models to generate the sentinel images. Experimental results demonstrate that ImageSentinel effectively detects unauthorized dataset usage while preserving generation quality for authorized applications.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to detect unauthorized usage of private image datasets in a Retrieval-Augmented Image Generation (RAIG) system. Authors propose to include some digitally crafted images referred as “image sentinels” into the dataset to enable the detection of unauthorized usage. Authors synthesize these sentinels using reference images from the dataset and explicitly condition the image synthesizer to embed a random character sequence (key) into the generated images. These generated images are added to the private dataset as image sentinels. When queried the black-box RAIG system with the keys i.e., specific random character sequences, the detection system compares the RAIG generated images with the image sentinels to find the similarity and raise the detection flag if similarity score exceeds a threshold. Experiments show that the proposed approach reaches 100% detection rate with 10-20 queries to the RAIG system and outperforms watermarking methods Ward-HiDDeN and Ward-FIN.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n-\tA simple but shown to be an effective approach to detect unauthorized image of private image datasets.\n\n-\tNo modifications are made to the original images in the dataset, hence authorized usage still provides relevant results.\n\n-\tThese sentinels make the detection to be triggered on demand i.e., reveal only when explicitly prompted with the specific keys (random character sequence).\n\n-\tIt is a simple and straightforward pipeline to generate the image sentinels leveraging existing vision-language foundation models, and find image similarity using vision foundation models at the time of detection.\n\n-\tResults show that the detection rate can be reached to 100% with 10-20 queries, outperforming watermarking based methods.\n\n-\tPaper is clearly written, and it is easy to read and understand.\n\nWeakness:\nDespite the strengths mentioned above, authors have not explored any possible detection evading techniques. I find that is the major weakness of the paper. A RAIG provider could easily incorporate some simple techniques to remove the image sentinels from the dataset:\n\n-\tIdentifying duplicate images in the dataset, and finding representative images that are unique based on the type, description, characteristics or even with perceptual similarity using DINO and filtering the duplicates or over-represented content.\n\n-\tAnother approach could be leveraging OCR models to detect the text from images and filtering out images with random character sequences or masking such image regions.\n\nQUESTIONS:\nI suggest authors to consider the points mentioned in the weakness section to thoroughly evaluate the robustness of the proposed detection mechanism. Currently my rating stands at borderline reject. However if my concerns are adequately addressed during rebuttal, I am willing to reconsider my rating.\n\nLIMITATIONS:\nAuthors have not explored possible detection evading mechanism to counter their proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a method to protect image datasets in the context of retrieval augmented generation. It works by generating “sentinel images” and including them in the protected dataset. They are images that incorporate a random-character key in the image, associated with a particular image semantics. To detect the use of a protected dataset, one has to prompt the model with the specific key and analyze the semantic similarity between the sentinel and the generated image (done with DINO). The paper validates the approach on SOTA generative models (GPT-4o, SD-XL, OmniGen) and thorough experiments.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n(1) The paper addresses an important and currently underexplored problem: protecting visual datasets from unauthorized use in Retrieval-Augmented Image Generation (RAIG).\n\n(2) The paper is well-written and easy to understand. It uses good illustrations, like prompts in boxes, which helps in the explanation.\n\n(3) The main idea of using sentinel images to get a clear detection signal is simple and effective. This makes it easy to tell when protected data has been used by a RAIG model.\n\n(4) The experiments are done carefully and thoroughly:  \n    (a) The code is clean, and seems easy to check and run (although I did not).  \n    (b) The authors used adapted WARD as a baseline, which, even if different from their method, provides a good comparison point and shows effort to be rigorous.  \n    (c) The method is tested with several up-to-date generative models (like GPT-4o, SD-XL, and Omnigen), showing its wide applicability.  \n    (d) Extra results are included in the supplementary materials, adding more proof for their findings.  \n\n## Weaknesses\n\n(1) The use of DINO for semantic similarity in detection limits how strong the statistical proof can be. For example, evaluating only TPR at FPR=0.01 might not be enough for legal situations, unlike methods such as the adapted WARD-Hidden, which can offer more statistical tests based on the matching its between binary messages.\n\n(2) Some parts of the experimental setup are not fully clear in the paper:  \n    (a) The exact number of sentinel images used in the experiments is not explicitly stated, which makes it hard to understand the scale of the protection and how stealthy it is.  \n    (b) How TPR at 1% FPR is calculated is not completely clear. The paper mentions \"100 trials with 5 queries\" but doesn't fully explain what a \"trial\" is or how the \"negatives\" (without the protected dataset) are created for evaluation.  \n\n(3) The paper does not discuss how people might try to get around the protection.  \n    (a) For example, if a generative model provider sees unusual images with random characters in their database, they might simply remove them.  \n    (b) They could also prevent users from using prompts with random characters. The paper would benefit from discussing these potential challenges and how the method might handle them.\n\n---\nNote for the ratings:\n- Quality: the experiments are done with care, but the idea of sentinel images in itself leads to strong limitations (no theoretical guarantees on the detection, ease of protection removal\n\nQUESTIONS:\n(1) The paper mentions a total of 10,000 images, with 2,800 images designated as the private dataset. Could the authors clarify the precise number or proportion of these 2,800 private images that are converted into or replaced by sentinel images? If a significant portion, or indeed all, of the 2,800 images serve as sentinels, what are the implications for the probability of inadvertently observing a sentinel image when inspecting the dataset, and what is the potential impact on detection efficiency or overhead?\n\n(2) When the generative model operates on a protected dataset but without the specific retrieval key included in the prompt, has it been observed that the model inadvertently generates the embedded random character sequences within the output images?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new approach to detect the unauthorized use of private image datasets in RAIG systems. The method involves augmenting a private dataset with synthesized \"sentinel\" images that are visually consistent with the dataset but also contain a unique, embedded textual key. A dataset owner can then probe a suspect RAIG system by providing these keys as prompts. High visual similarity between the generated output and the original sentinel image is used as evidence of dataset misuse. The work demonstrates high detection rates across several modern RAIG systems with minimal impact on authorized generation quality.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- This paper studies a pressing problem when a retrieval-augmented pipeline is commonly adopted by GenAI.\n- The experimental setup and ablation studies are comprehensive, and the writing is clear.\n\n### Weaknesses\n\n- The evaluation lacks a defense against an adaptive adversary, a standard requirement for security claims. The entire defense hinges on the stealthiness of sentinels, yet it assumes a passive adversary. A motivated attacker aware of this strategy could easily employ countermeasures, such as using OCR to filter out any images containing conspicuous, random text strings before ingestion. The necessity of evaluating against such adaptive attacks is a well-established principle in the security community (e.g., [Tramèr et al., 2020], [Athalye et al., 2018]). Without this, the claimed robustness is unsubstantiated.\n\n- Some baselines are not competitive, comprising known-to-fail \"strawmen\" while ignoring the state-of-the-art. The paper compares its method against traditional image watermarking (HiDDeN, FIN). However, if I understand it correctly, recent work [37] has already underscored that such watermarks are not robust to the complex transformations of modern generative models. These baseline watermark methods resulting in near-random performance (AUC $\\approx$ 0.5) are therefore expected and do not constitute a meaningful benchmark. In my opinion, the paper avoids comparison with more relevant and challenging baselines like the membership attacks for RAG that the paper cites in its own related work (e.g., [25]).\n\n- The evaluation of the detection mechanism is a bit ambiguous. The core claim is that the key triggers the retrieval of the sentinel. However, the evaluation fails to disentangle this from a more trivial explanation: that the RAIG's powerful generator (e.g., GPT-4o) simply renders the key from the prompt, leading to an image that is naturally similar to the sentinel. The \"triggering rate\" in Table 2, which is key to this claim, is not clearly defined for a black-box setting, and this ambiguity undermines the interpretation of the core results.\n\n- My final primary concern is about the assumption of the proposed defense. Based on my understanding, the defense only works if the adversary's generator is powerful enough to faithfully render the key; if a weaker generator is used, the detection mechanism will likely fail.\n\n### References\n- [Tramèr et al., 2020] Florian Tramèr et al. On Adaptive Attacks to Adversarial Example Defenses. Proc. of NeurIPS, 2020.\n- [Athalye et al., 2018] Anish Athalye et al. Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples. Proc. of ICML, 2018.\n\nQUESTIONS:\n- Can you discuss the robustness of ImageSentinel to an adaptive adversary who actively filters the dataset for suspicious images (e.g., via OCR)?\n- Can you propose an experiment to decisively prove that the high similarity scores you observe are a result of retrieval and not just the direct, non-retrieval-based rendering of the key by the generator?\n- How do you propose this method remains reliable against adversaries using models where this capability is weak or absent?\n\nLIMITATIONS:\nThe authors include a \"Limitations\" section, which is appreciated. However, it omits several critical limitations raised in the weaknesses section, such as the vulnerability to adaptive adversaries.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a framework (ImageSentinel) to protect image datasets from unauthorised use in RAIG systems. It inserts sentinel images containing random character sequences as triggers for detecting unauthorised dataset use. The authors demonstrate their framework by showing reliable detection of dataset misuse across multiple RAIG systems in a black-box setting.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper addresses an important and timely challenge, namely the unathorised use of image (datasets), specifically with RAIG systems. Here, existing watermarking approaches fall short, making this an important contribution to the topic of RAIG image retrieval and copyright in AI. I appreciate that the authors show how their approach can work without disrupting overall RAIG for general usage. The experiments and evaluation appear quite comprehensive, they test across multiple RAIG systems and relevant ablations.\n\nThere are a limitations with regards to the real-world testing of the models, namely evaluations are performed on synthetic datasets, the threat from proprietary or fine-tuned RAIG systems in the wild is not assessed. In a similar vein, but rather as a note than a criticism of the method overall, the approach here relies on black-box querying, which requires repeated queries to the suspected RAIG system, which may not be feasible in a practical setting. \n\nFurther, the authors could extend their evaluation of the methodology by providing adverserial analysis, specifically w.r.t. the detectability of sentinel images; malicious users could potentially filter or recognise sentinel patterns.\n\nQUESTIONS:\n* Have you tested whether an adversary with access to the protected dataset could filter or detect the sentinel images before integrating them into a RAIG system?\n* How would your approach perform if the RAIG system filtered or sanitized the random character prompts before retrieval (e.g., by stripping non-semantic input)? How much would the detection rate drop when such preprocessing is applied to the prompts?\n* Can you clarify how many queries and how much compute/time is required for reliable detection? What are the limits of detection with fewer queries?\n\nLIMITATIONS:\nAs described above", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}