ICML 2025

STAMP Your Content:
Proving Dataset Membership via Watermarked Rephrasings
Saksham Rastogi 1 Pratyush Maini 2 3 Danish Pruthi 1
Abstract
Given how large parts of publicly available text
are crawled to pretrain large language mod-
els (LLMs), data creators increasingly worry
about the inclusion of their proprietary data for
model training without attribution or licensing.
Their concerns are also shared by benchmark cu-
rators whose test-sets might be compromised. In
this paper, we present STAMP, a framework for de-
tecting dataset membership—i.e., determining the
inclusion of a dataset in the pretraining corpora of
LLMs. Given an original piece of content, our pro-
posal involves first generating multiple rephrases,
each embedding a watermark with a unique se-
cret key. One version is to be released publicly,
while others are to be kept private. Subsequently,
creators can compare model likelihoods between
public and private versions using paired statis-
tical tests to prove membership. We show that
our framework can successfully detect contami-
nation across four benchmarks which appear only
once in the training data and constitute less than
0.001% of the total tokens, outperforming sev-
eral contamination detection and dataset inference
baselines. We verify that STAMP preserves both
the semantic meaning and utility of the original
data. We apply STAMP to two real-world scenar-
ios to confirm the inclusion of paper abstracts and
blog articles in the pretraining corpora.
1. Introduction
To train large language models, much of the available text
from the internet is crawled, allegedly including copyrighted
material such as news articles and blogs (Grynbaum & Mac,
1Indian
Institute
of
Science
2Carnegie
Mellon
Uni-
versity
3DatologyAI.
Correspondence
to:
Saksham
Ras-
togi <iitdsaksham@gmail.com>, Pratyush Maini <pratyush-
maini.@cmu.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author...