Machine Text Detectors are Membership Inference Attacks
Authors: name: Ryuto Koike, name: Liam Dugan, name: Masahiro Kaneko, name: Chris Callison-Burch, name: Naoaki Okazaki

Although membership inference attacks (MIAs) and machine-generated text
detection target different goals, identifying training samples and synthetic
texts, their methods often exploit similar signals based on a language model's
probability distribution. Despite this shared methodological foundation, the
two tasks have been independently studied, which may lead to conclusions that
overlook stronger methods and valuable insights developed in the other task. In
this work, we theoretically and empirically investigate the transferability,
i.e., how well a method originally developed for one task performs on the
other, between MIAs and machine text detection. For our theoretical
contribution, we prove that the metric that achieves the asymptotically highest
performance on both tasks is the same. We unify a large proportion of the
existing literature in the context of this optimal metric and hypothesize that
the accuracy with which a given method approximates this metric is directly
correlated with its transferability. Our large-scale empirical experiments,
including 7 state-of-the-art MIA methods and 5 state-of-the-art machine text
detectors across 13 domains and 10 generators, demonstrate very strong rank
correlation (rho > 0.6) in cross-task performance. We notably find that
Binoculars, originally designed for machine text detection, achieves
state-of-the-art performance on MIA benchmarks as well, demonstrating the
practical impact of the transferability. Our findings highlight the need for
greater cross-task awareness and collaboration between the two research
communities. To facilitate cross-task developments and fair evaluations, we
introduce MINT, a unified evaluation suite for MIAs and machine-generated text
detection, with implementation of 15 recent methods from both tasks.