Mitigating Paraphrase Attacks on Machine-Text Detection via Paraphrase Inversion
Authors: Rafael Alberto Rivera Soto, Barry Y. Chen, Nicholas Andrews
Accepted at: acl25

Mitigating Paraphrase Attacks on Machine-Text Detectors via Paraphrase Inversion

Rafael Rivera Soto1,2, Barry Chen1, and Nicholas Andrews2

1Lawrence Livermore National Laboratory
2Johns Hopkins University
1rafaelriverasoto@jhu.edu, chen52@llnl.gov, noa@jhu.edu

Abstract

High-quality paraphrases are easy to produce using instruction-tuned language models or specialized paraphrasing models. Although this capability has a variety of benign applications, paraphrasing attacks—paraphrases applied to machine-generated texts—are known to significantly degrade the performance of machine-text detectors. This motivates us to consider the novel problem of paraphrase inversion, where, given paraphrased text, the objective is to recover an approximation of the original text. The closer the approximation is to the original text, the better machine-text detectors will perform. We propose an approach which frames the problem as translation from paraphrased text back to the original text, which requires examples of texts and corresponding paraphrases to train the inversion model. Fortunately, such training data can easily be generated, given a corpus of original texts and one or more paraphrasing models. We find that language models such as GPT-4 and Llama-3 exhibit biases when paraphrasing which an inversion model can learn with a modest amount of data. Perhaps surprisingly, we also find that such models generalize well, including to paraphrase models unseen at training time. Finally, we show that when combined with a paraphrased-text detector, our inversion models provide an effective defense against paraphrasing attacks, and overall our approach yields an average improvement of +22% AUROC across seven machine-text detectors and three different domains.

1 Introduction

Recent developments in the capabilities of large language models (LLMs) such as GPT-4 (OpenAI et al., 2024) have resulted in their widespread use by a variety of users. Although most users act responsibly, there is growing concern about abuses of LLMs, such as for plagiarism, spam, or spreading misinformation (Weidinger et al., 2022; Hazell, 2023). To minimize the abuse of these systems, several machine-text detection systems have been proposed, including Binoculars (Hans et al., 2024), FastDetectGPT (Bao et al., 2024), and watermarking-based algorithms (Kirchenbauer et al., 2023; Kuditipudi et al., 2024). However, these systems often fail to detect text that has been paraphrased by another model (Krishna et al., 2020; Sadasivan et al., 2025), leaving a critical gap in current detection systems.

To tackle this issue, a recent study has proposed jointly training a paraphraser and a machine-text detector with an adversarial objective: the paraphraser generates text to evade detection, while the detector identifies paraphrased text (Hu et al., 2023). Another study has proposed that LLM API providers cache their generations, enabling retrieval over a semantic space, where candidates with high similarity to previous generations are marked as paraphrases (Krishna et al., 2023). Unfortunately, both approaches lack generality, as they depend on training a specialized detector, or having access to all model generations. A more desirable defense would be detector agnostic, improving the performance of any detector.

Ideally, if the original tokens of a paraphrased text could be recovered, machine-text detectors would perform well, eliminating the need for any specialized solutions. Therefore, we propose the novel task of paraphrase inversion, where the objective is to recover the original text from a paraphrased one. This approach has the added benefit of being detector agnostic. Given the space of possible paraphrases and the stochastic sampling procedures commonly used, inverting paraphrased text is challenging. Nonetheless, there is evidence that LLMs exhibit consistent biases even when the instruction implicitly or explicitly requests diversity in the responses (Zhang et al., 2024b; Wu et al., 2025).

Even if paraphrase inversion is possible, we must know when to apply it, making paraphrase detec-
Machine-generated   Detected as Machine? ✔️
Our experiments demonstrate that the proposed technique achieves state-of-the-art results in segmenting brain tumours from MRI scans, demonstrating its effectiveness and potential impact in clinical applications.

Paraphrase   Detected as Machine? ✗
We show that the proposed method is able to achieve the best results in separating brain tumours from MR images, thereby demonstrating its effectiveness and its practical application.

Paraphrase Inversion   Detected as Machine? ✔️
We demonstrate that our proposed technique achieves state-of-the-art results in segmenting brain tumours from MRI scans, highlighting its effectiveness and potential applicability in clinical settings.

BLEU(original, paraphrase) = 14
BLEU(original, inverse) = 57

Figure 1: Paraphrasing defeats machine-text detection system. Our proposed defense (§3) consists of two steps: (1) detecting whether text is a paraphrase, and (2) if so: (2) inverting the paraphrase back to the original text. This pipeline improves the AUROC of 7 machine-text detectors across three domains by an average of +22% AUROC (Table 1).

tion a necessary step. Detecting text as having undergone LLM paraphrasing differs from detecting it as machine-generated, as the original text may have been human-written, in which case large portions of the original document may be copies of the human-written original. In cases where the original text is human-written, a machine-text detector should classify it as such, for example in cases where an LLM is used as a writing assistant.

To address these concerns, we propose paraphrase detection and paraphrase inversion as a pipeline to improve the performance of any machine-text detector in scenarios where texts may have been paraphrased (Figure 1). Our main contributions are as follows:

• We introduce the task of paraphrase inversion (§3), where the goal is to recover the original text from a paraphrased one. We formalize the task and provide a comprehensive analysis of its challenges. We find that inverting human-written text is significantly harder than inverting paraphrases of machine-generated text, which is to be expected given that human-written text exhibits higher entropy under LLM distributions (Gehrmann et al., 2019).

• We explore two paraphrase detection schemes: (1) a simple neural classifier trained to detect paraphrased text and (2) an approach that leverages our paraphrase inversion model directly without requiring an additional model (§3.3).

• We combine paraphrase detection and paraphrase inversion into a single pipeline that improves the detection rate of seven machine-text detectors across three domains (§5.2) by an average of +22% AUROC.

Reproducibility The dataset, method implementations, model checkpoints, and experimental scripts, will be released along with the paper.1

2 Related Work

Paraphrasing A number of paraphrase corpora have been released over the years which has enabled the development of paraphrase detection and generation models (Dolan and Brockett, 2005; Ganitkevitch et al., 2013; Wieting and Gimpel, 2018; Zhang et al., 2019; Krishna et al., 2020). Paraphrases have been shown to degrade the performance of machine-text detectors, including those based upon watermarking (Krishna et al., 2023; Sadasivan et al., 2025). In response to this, several defenses have been proposed, including jointly training a paraphraser and a detector in an adversarial setting (Hu et al., 2023), building specialized detectors for both the paraphrasing model and the language model (Soto et al., 2024), and retrieval over a database of semantically similar generations produced by the model in the past (Krishna et al., 2023). Paraphrases have also been shown to be an effective attack against authorship verification systems (Potthast et al., 2016; Wang et al., 2023), allowing bad actors to conceal their identity. To our knowledge, our approach is the first attempt at inverting the paraphrases, both in general and in the context of defending against paraphrasing attacks on machine-text detection.

Embedding inversion Several lines of work, both in computer vision (Mahendran and Vedaldi, 2015; Teterwak et al., 2021; Dosovitskiy and Brox, 2016) and natural language processing (Song and Raghunathan, 2020; Li et al., 2023; Morris et al., 2023) have explored whether embeddings can be inverted back to their inputs. Prior work has shown

1Code for all experiments available https://github.com/rriveral849/inversion
that it is possible to recover 92% of 32-token text inputs given semantic embeddings (Morris et al., 2024). Moreover, even when the text is isn’t recovered with high-fidelity, sensitive attributes such as the authorship are recoverable (Song and Raghu-nathan, 2020). In computer vision, even when an inversion model is applied to an adversarially robust classifier, enough local and global detail remains, making the inversion confusable with the original image, highlighting the difficulty of safeguarding sensitive attributes (Teterwak et al., 2021). Inverting embeddings is significantly easier than inverting paraphrases, as embeddings encode rich features of their inputs in continuous latent-space, in contrast to the discrete space of paraphrased tokens.

Language model inversion (Morris et al., 2024)
The objective here is to recover the prompt that generated a particular output. Language model inversion techniques such as logit2text (Morris et al., 2024) require knowledge of the LLM that generated the output and access to the next-token probability distribution, making it difficult to apply in practice. Another approach more closely related to ours is output2prompt (Zhang et al., 2024a), which trains an encoder-decoder architecture to generate the prompt given multiple outputs. However, output2prompt requires upwards of 16 outputs per prompt to successfully match the performance of logit2text, and only handles prompts up to 64 tokens long. In contrast to these methods, we focus exclusively on inverting LLM-generated paraphrases given a single example cleaned of all obvious generation artifacts such as “note: I changed...”, thereby removing all telltale signs of what the original text might’ve been2. Therefore, the paraphrase inversion problem considered in this paper is more challenging than related problems posed in prior work.

3 Methods

3.1 Overview
Given a text sample \( y_i \), we first detect whether it is a paraphrase using one of our detection schemes. If it is classified as a paraphrase, we apply our paraphrase inversion model to recover the original text \( \hat{x} \sim p(. \mid y_i) \). This sample is then run through a machine-text detector.

Paraphrase inversion The task of reconstructing the original source text given paraphrased text. The difficulty of this task hinges in large part on assumptions regarding the paraphrasing model. We assume access to one or more paraphrasing models from which we can generate new paraphrases \( \{y_i\}_{i=1}^N \) given a corpus of \( N \) source documents \( \{x_i\}_{i=1}^N \). While access to the paraphrasing models in principle affords the possibility of producing an arbitrary amount of training data, in practice the paraphraser may be associated with non-trivial inference costs (e.g., GPT-4). Moreover, even if the paraphrasing model is known, the decoding parameters such as temperature may not be.3 Therefore, a key question is whether paraphrase inversion models generalize to unseen paraphrasers, which we consider in §6.3.

Paraphrase detection The goal is to identify whether a given text is the output of an LLM paraphraser, regardless of whether original text was human-written or machine-generated. Paraphrase detection is crucial for machine-text detection in the wild, where determining when to apply a paraphrase inversion model is necessary. We emphasize that detecting text as a paraphrase is not the same as identifying text as machine-generated, as the original text may have been human-written. In cases where the original text is human-written, a machine-text detector should classify it as such. This highlights the need of applying a paraphrase inversion model to ensure correct detection. However, such a pipeline raises the risk of propagation of errors, and we should therefore carefully consider the cost of such errors.
1. A false positive occurs when a non-paraphrased text is misidentified as paraphrased. To minimize the impact of such errors, a robust paraphrase inversion model should make minimal changes to the text in such cases. We find that our models make significantly fewer changes to non-paraphrased documents (§3.3), and that this can in fact be used as a way to distinguish between paraphrased and non-paraphrased text.
2. A false negative occurs when a paraphrased text is missed by the detector and we fail to apply the inversion model. In this case, the machine-text detector is applied to the unmodified paraphrased text, which if the original text was machine-generated, is likely to result in

2More details can be found in Appendix G
3We investigate the impact of varying sampling the temperature during training and inference in Appendix C.
falsely predicting that it is human written.
Given the above considerations, the paraphrase text detector should aim for high recall at the cost of potentially lower precision.

3.2 End-to-end paraphrase inversion

Training objective The inversion models considered in this paper are fine-tuned using the standard supervised text-to-text objective, fitting an autoregressive conditional language model \( p_\theta(y_i \mid x_i) \) on the basis of observed pairs of texts and their paraphrases \((x_i, y_i)\). Our datasets are described in §4.1. We parameterize all our inversion models using Mistral-7B\footnote{mistralai/Mistral-7B-Instruct-v0.3}, training it with the hyper-parameters shown in Appendix F. We use teacher forcing during training, conditioning on the the true observed tokens.

Inference However good the paraphrasing model, there may be considerable uncertainty in the distribution over the original text. Therefore we sample several inversions and use a scoring function to select a single sample which scores highest.

Choice of score A number of criteria could be optimized to help select a single inversion likely to be close to the original text. For example, inversions should retain the meaning of the paraphrased text, and so the score could include a measure of semantic similarity. Furthermore, inspired by the finding of Soto et al. (2024) that machine-generated text is *stylistically* distinct from human-written text, we posit that the inversion should be stylistically distinct from the paraphrased text, as this would indicate a return to the original machine or human style. In preliminary experiments, we found that the paraphrasing model consistently preserved meaning in generated samples, and so to avoid introducing additional hyper-parameters and computational expense, we focus on stylistic distinctness. Specifically, we compute a stylistic embedding of the samples and original text to compute a stylistic distance for each candidate inversion, and select the inversion which is *furthest*—the most stylistically distinct.

3.3 Detecting Paraphrases

Neural paraphrase detector In the simplest case, we train a paraphrase detector \( d_\phi(. \mid y_i) \) using the standard binary-cross-entropy classification loss. In addition to the standard loss, we optimize the model for a paraphrased token prediction task, where the goal is to determine whether each token in a document is copied from the original text or paraphrased. We include this loss to help the model capture the biases that paraphrasers introduce when rewording text. We optimize the binary-cross-entropy for each token, corresponding to independent classification decisions. Our model is initialized from RoBERTa-large\footnote{FacebookAI/roberta-large} (Liu et al., 2019), with a multi-layer-perceptron (MLP) head that predicts whether each token was copied from the original text or paraphrased.

Edit-based paraphrase detector Rather than training a neural classifier, we determine whether a sample \( y_i \) is a paraphrase based on how many edits our paraphrase inversion model makes. Intuitively, if the paraphrase inversion model captures LLM paraphrasing biases, it should make *fewer* edits when “inverting” a human-written text than when inverting a paraphrase. Indeed, we find that this is the case in Figure 2. This observation motivates the following paraphrase detection scheme. Given two Gaussian distributions \( g_h \) and \( g_m \), where \( g_h \) is fit on edit distances of human-text inversions and their originals and \( g_m \) on those from paraphrases of human- and machine-text and their inversions, we detect whether a sample \( y_i \) is a paraphrase by calculating whether \( y_i \) is more probable under \( g_m \) than \( g_h \). This is equivalent to applying a likelihood-ratio test with a threshold of 1. In practice, because we have \( N \) inversions per sample, we take the majority

![Edit distances between the original text and its inversion when the machine-paraphrase inversion model is applied to human-text and paraphrases of human- or machine-text. The inversion model edits human-written significantly less.](page_1012_186_377_285.png)

Figure 2: Edit distances between the original text and its inversion when the machine-paraphrase inversion model is applied to human-text and paraphrases of human- or machine-text. The inversion model edits human-written significantly less.
vote of all such predictions.

4 Experimental Procedure

4.1 Datasets
We evaluate our approach on three domains: Reddit, ArXiv, and MovieReviews. We use Reddit specifically to test the feasibility of paraphrase inversion, while all three domains are used to evaluate our pipeline for defending machine-text detectors against paraphrase attacks. The validation sets of each domain are used to train our edit-based paraphrase detector introduced in §3.3, while the training sets are used to train our paraphrase inversion models and our neural paraphrase detector. The ArXiv and MovieReviews datasets are subsampled from the RAID (Dugan et al., 2024) dataset, a machine-text detection benchmark which contains paraphrases of machine-text using DIPPER (Krishna et al., 2023). We refer to these two datasets as RAID-ArXiv and RAID-MovieReviews. The details of how RAID was subsampled can be found in Appendix B. Here, we discuss how we generate human-text paraphrases and machine-text paraphrases for the Reddit domain, as well as the construction of the Reddit machine-detection dataset.

Human-text paraphrases We use the Reddit Million User Dataset (MUD), which contains comments from over 1 million Reddit users over a wide variety of topics (Khan et al., 2021). We subsample the dataset according to the procedure in Appendix A. Once subsampled, we generate the paraphrases of human-text by prompting Mistral-7B6 (Jiang et al., 2023), Phi-3B7 (Abdin et al., 2024), and Llama-3.1-8B8 (Dubey et al., 2024). We clean all obvious LLM-generated artifacts such as This rephrased passage condenses, note: I changed..., and ensure that all paraphrases have a semantic similarity of at least 0.7 under SBERT9 (Reimers and Gurevych, 2019).

Machine-text paraphrases To generate paraphrases of machine-text, we first prompt one of the three LLM at random to produce a response to each human-written comment, then we follow the same paraphrasing procedure described above.

Machine-text detection We combine the test set of both our human-text paraphrase and machine-text paraphrase datasets to create a new set composed of 500 samples in each category: human text, paraphrases of human text, and paraphrases of machine text.

4.2 Metrics
To measure how well the inverted text recovers the true tokens, we make use of BLEU (Papineni et al., 2002), a measure of n-gram overlap. Recovering the original tokens may be difficult, if not impossible. As such, we posit that the inverted text should be close both in style and semantics to the original. We measure the stylistic similarity by embedding the inversion and the original using LUAR (Rivera-Soto et al., 2021)10, a model that captures the stylistic features of text; we report the stylistic similarity as the cosine similarity between the embeddings. For semantic similarity, we use SBERT (Reimers and Gurevych, 2019) to embed the texts and report the cosine similarity between them. To test the performance of the machine-text detectors, we report the area under the curve (AUC) of the receiver operating curve (ROC), here denoted as AUROC. The ROC curve captures the trade-off between false positive and true positive rates across all decision thresholds; the AUROC summarizes this performance as a single scalar. To measure the performance of the paraphrase detectors, we use the F1 score which measures the harmonic mean between the precision and recall achieved by the detector.

4.3 Baselines
For comparison, we prompt GPT-4 to invert the paraphrases. We report the prompts used in §D.2. Additionally, we compare our inversion model to output2prompt (Zhang et al., 2024a), training it on the same dataset. For machine-text detection, we avail of many popular detectors. We use Rank (Gehrmann et al., 2019), LogRank (Solaiman et al., 2019), Entropy (Ippolito et al., 2020), OpenAI’s detector (Solaiman et al., 2019), RADAR (Hu et al., 2023), FastDetectGPT (Bao et al., 2024), and Binoculars (Hans et al., 2024).

5 Main Results
This section present results for our motivating application of defending against paraphrasing attacks

6mistralai/Mistral-7B-Instruct-v0.3
7microsoft/Phi-3-mini-4k-instruct
8meta-llama/Meta-Llama-3-8B-Instruct
9sentence-transformers/all-mpnet-base-v2
10riveral849/LUAR-CRUD
<table>
  <tr>
    <th rowspan="2">Detector</th>
    <th colspan="3">AUROC</th>
  </tr>
  <tr>
    <th>Baseline</th>
    <th>Inversion+Edit-based</th>
    <th>Inversion+Neural</th>
  </tr>
  <tr>
    <th colspan="4">Reddit</th>
  </tr>
  <tr>
    <td>OpenAI (2019)</td>
    <td>0.56</td>
    <td>0.77</td>
    <td><b>0.79</b></td>
  </tr>
  <tr>
    <td>Rank (2019)</td>
    <td>0.56</td>
    <td>0.66</td>
    <td><b>0.68</b></td>
  </tr>
  <tr>
    <td>LogRank (2019)</td>
    <td>0.58</td>
    <td>0.74</td>
    <td><b>0.77</b></td>
  </tr>
  <tr>
    <td>Entropy (2020)</td>
    <td>0.51</td>
    <td><b>0.59</b></td>
    <td><b>0.59</b></td>
  </tr>
  <tr>
    <td>RADAR (2023)</td>
    <td>0.62</td>
    <td>0.66</td>
    <td><b>0.70</b></td>
  </tr>
  <tr>
    <td>FastDetectGPT (2024)</td>
    <td>0.66</td>
    <td>0.80</td>
    <td><b>0.84</b></td>
  </tr>
  <tr>
    <td>Binoculars (2024)</td>
    <td>0.77</td>
    <td>0.84</td>
    <td><b>0.89</b></td>
  </tr>
  <tr>
    <th colspan="4">RAID-ArXiv</th>
  </tr>
  <tr>
    <td>OpenAI (2019)</td>
    <td><b>0.81</b></td>
    <td>0.79</td>
    <td>0.77</td>
  </tr>
  <tr>
    <td>Rank (2019)</td>
    <td>0.71</td>
    <td>0.69</td>
    <td><b>0.79</b></td>
  </tr>
  <tr>
    <td>LogRank (2019)</td>
    <td>0.75</td>
    <td>0.72</td>
    <td><b>0.91</b></td>
  </tr>
  <tr>
    <td>Entropy (2020)</td>
    <td>0.39</td>
    <td>0.42</td>
    <td>0.62</td>
  </tr>
  <tr>
    <td>RADAR (2023)</td>
    <td><b>0.99</b></td>
    <td>0.98</td>
    <td><b>0.99</b></td>
  </tr>
  <tr>
    <td>FastDetectGPT (2024)</td>
    <td>0.83</td>
    <td>0.78</td>
    <td><b>0.91</b></td>
  </tr>
  <tr>
    <td>Binoculars (2024)</td>
    <td>0.92</td>
    <td>0.86</td>
    <td><b>0.98</b></td>
  </tr>
  <tr>
    <th colspan="4">RAID-MovieReviews</th>
  </tr>
  <tr>
    <td>OpenAI (2019)</td>
    <td>0.82</td>
    <td>0.77</td>
    <td><b>0.83</b></td>
  </tr>
  <tr>
    <td>Rank (2019)</td>
    <td>0.60</td>
    <td>0.76</td>
    <td><b>0.84</b></td>
  </tr>
  <tr>
    <td>LogRank (2019)</td>
    <td>0.66</td>
    <td>0.84</td>
    <td><b>0.91</b></td>
  </tr>
  <tr>
    <td>Entropy (2020)</td>
    <td>0.39</td>
    <td>0.63</td>
    <td><b>0.71</b></td>
  </tr>
  <tr>
    <td>RADAR (2023)</td>
    <td>0.92</td>
    <td>0.92</td>
    <td><b>0.95</b></td>
  </tr>
  <tr>
    <td>FastDetectGPT (2024)</td>
    <td>0.74</td>
    <td>0.80</td>
    <td><b>0.89</b></td>
  </tr>
  <tr>
    <td>Binoculars (2024)</td>
    <td>0.91</td>
    <td>0.92</td>
    <td><b>0.96</b></td>
  </tr>
</table>

Table 1: Machine-text detection performance on a dataset of human-text, paraphrases of human-text, and paraphrases of machine-text. Applying our inversion model to all samples detected as paraphrases using our paraphrase detection schemes (\S3.3), we observe significant improvements in detection performance.

<table>
  <tr>
    <th>Dataset</th>
    <th>Edit-based</th>
    <th>Neural</th>
  </tr>
  <tr>
    <td>Reddit</td>
    <td>0.79</td>
    <td><b>0.94</b></td>
  </tr>
  <tr>
    <td>RAID-ArXiv</td>
    <td>0.52</td>
    <td><b>0.67</b></td>
  </tr>
  <tr>
    <td>RAID-Reviews</td>
    <td><b>0.79</b></td>
    <td>0.72</td>
  </tr>
</table>

Table 2: F1 scores for the proposed paraphrased detection schemes (\S3.3).

for machine-text detection. Next, in \S6, we perform further analysis of individual components of our approach, including the feasibility of paraphrase inversion as a stand-alone task, considering both inversions of paraphrased machine-generated (\S6.1) and inversions paraphrased human-written documents (\S6.2).

5.1 Paraphrase detection

We evaluate the proposed paraphrased detection schemes described in \S3.3. We train the methods in all three domains, and report results in Table 2. We find that the neural detector outperforms the edit-based detector across two out of three of the domains. Moreover, the edit-based detector performs poorly in RAID-ArXiv, the most challenging domain, which in turn harms the performance of machine-text detectors in this setting (\S5.2).

5.2 Machine-Text Detection

We consider the scenario where human- or machine-text may have been paraphrased by an LLM. In this scenario, it would be desirable to label paraphrases of human-text as human-written and paraphrases of machine-text as machine-generated. We train and evaluate our defense pipeline on all three domains separately. We run our paraphrase detection schemes on the held-out test set, inverting each sample detected as a paraphrase 100 times, and picking the inversion that is the farthest away from the input-text in LUAR space, ensuring that the style is dissimilar from paraphrasing style. We report the AUROC of 7 popular machine-text detectors in Table 1, and make the following observations: (1) Our defense, with the neural paraphrase detector improves the performance of 7 machine-text detectors across 3 domains. The only exception is OpenAI’s detector on the RAID-ArXiv dataset. (2) RADAR, a detector designed to be robust against paraphrase attacks, also benefits. Indeed, in the worst case, RADAR’s performance remains unchanged (RADAR-ArXiv), but in other domains, we observe notable improvements. This highlights that our defense can be combined with other existing defenses. (3) The
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Type</th>
    <th colspan="3">Machine-written Text</th>
    <th colspan="3">Human-written Text</th>
  </tr>
  <tr>
    <th>Style (↑)</th>
    <th>Meaning (↑)</th>
    <th>BLEU (↑)</th>
    <th>Style (↑)</th>
    <th>Meaning (↑)</th>
    <th>BLEU (↑)</th>
  </tr>
  <tr>
    <td colspan="8"><b>Paraphrases</b></td>
  </tr>
  <tr>
    <td colspan="8">-</td>
    <td>0.80</td>
    <td>0.88</td>
    <td>0.17</td>
    <td>0.51</td>
    <td>0.82</td>
    <td>0.08</td>
  </tr>
  <tr>
    <td colspan="8"><b>Baselines</b></td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>Single</td>
    <td>0.80</td>
    <td>0.85</td>
    <td>0.20</td>
    <td>0.50</td>
    <td>0.80</td>
    <td>0.07</td>
  </tr>
  <tr>
    <td></td>
    <td>Max</td>
    <td>0.86</td>
    <td>0.90</td>
    <td>0.33</td>
    <td>0.56</td>
    <td>0.84</td>
    <td>0.11</td>
  </tr>
  <tr>
    <td></td>
    <td>Mean</td>
    <td>0.80</td>
    <td>0.87</td>
    <td>0.21</td>
    <td>0.50</td>
    <td>0.80</td>
    <td>0.07</td>
  </tr>
  <tr>
    <td>out2prompt</td>
    <td>Single</td>
    <td>0.48</td>
    <td>0.17</td>
    <td>0.00</td>
    <td>0.39</td>
    <td>0.10</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td></td>
    <td>Max</td>
    <td>0.71</td>
    <td>0.40</td>
    <td>0.04</td>
    <td>0.53</td>
    <td>0.32</td>
    <td>0.02</td>
  </tr>
  <tr>
    <td></td>
    <td>Mean</td>
    <td>0.48</td>
    <td>0.17</td>
    <td>0.00</td>
    <td>0.39</td>
    <td>0.09</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td colspan="8"><b>Ours</b></td>
  </tr>
  <tr>
    <td>Inversion</td>
    <td>Single</td>
    <td>0.84</td>
    <td>0.90</td>
    <td>0.34</td>
    <td>0.54</td>
    <td>0.81</td>
    <td>0.13</td>
  </tr>
  <tr>
    <td></td>
    <td>Max</td>
    <td><b>0.91</b></td>
    <td><b>0.95</b></td>
    <td><b>0.51</b></td>
    <td><b>0.70</b></td>
    <td><b>0.90</b></td>
    <td><b>0.25</b></td>
  </tr>
  <tr>
    <td></td>
    <td>Mean</td>
    <td>0.84</td>
    <td>0.90</td>
    <td>0.35</td>
    <td>0.54</td>
    <td>0.81</td>
    <td>0.12</td>
  </tr>
</table>

Table 3: Results of inverting paraphrases of machine-written text (left three columns) and paraphrases of human-written text (right three columns). We generate 100 inversions per sample and report the metrics achieved by a single inversion, by the best inversion (max), and the average across all inversions. Our proposed inversion model outperforms all baselines.

<table>
  <tr>
    <th>Detector</th>
    <th colspan="2">AUROC</th>
  </tr>
  <tr>
    <th></th>
    <th>Baseline</th>
    <th>Inversion</th>
  </tr>
  <tr>
    <td colspan="3"><b>Train - RAID-MovieReviews, Eval - RAID-ArXiv</b></td>
  </tr>
  <tr>
    <td>OpenAI (<b>2019</b>)</td>
    <td>0.81</td>
    <td>0.84</td>
  </tr>
  <tr>
    <td>Rank (<b>2019</b>)</td>
    <td>0.71</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>LogRank (<b>2019</b>)</td>
    <td>0.75</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>Entropy (<b>2020</b>)</td>
    <td>0.39</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>RADAR (<b>2023</b>)</td>
    <td><b>0.99</b></td>
    <td>0.99</td>
  </tr>
  <tr>
    <td>FastDetectGPT (<b>2024</b>)</td>
    <td>0.83</td>
    <td>0.90</td>
  </tr>
  <tr>
    <td>Binoculars (<b>2024</b>)</td>
    <td>0.92</td>
    <td>0.96</td>
  </tr>
  <tr>
    <td colspan="3"><b>Train - RAID-ArXiv, Eval - RAID-MovieReviews</b></td>
  </tr>
  <tr>
    <td>OpenAI (<b>2019</b>)</td>
    <td><b>0.82</b></td>
    <td>0.82</td>
  </tr>
  <tr>
    <td>Rank (<b>2019</b>)</td>
    <td>0.60</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>LogRank (<b>2019</b>)</td>
    <td>0.66</td>
    <td>0.90</td>
  </tr>
  <tr>
    <td>Entropy (<b>2020</b>)</td>
    <td>0.39</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>RADAR (<b>2023</b>)</td>
    <td>0.92</td>
    <td>0.94</td>
  </tr>
  <tr>
    <td>FastDetectGPT (<b>2024</b>)</td>
    <td>0.74</td>
    <td>0.87</td>
  </tr>
  <tr>
    <td>Binoculars (<b>2024</b>)</td>
    <td>0.91</td>
    <td>0.95</td>
  </tr>
</table>

Table 4: Machine-text detection performance on a dataset of human-text, paraphrases of human-text, and paraphrases of machine-text. We find that when our pipeline generalizes even when trained on one domain, and evaluated on another (e.g. RAID-ArXiv → RAID-MovieReviews).

edit-based paraphrase detector is not robust across all domains. Although the edit-based paraphrase detector improves performance on the Reddit and RAID-MovieReviews datasets, it reduces performance on RAID-ArXiv. This decline is due to the many mis-classifications in that domain. However, overall we observe an average improvement of +22% AUROC averaged across all detectors and domains.

5.3 Generalizing across domains

Do the paraphrase detection and paraphrase inversion models generalize from one dataset to another? We apply the pipeline using the neural paraphrase detector and inversion model trained on RAID-ArXiv to RAID-MovieReviews, and vice versa, showing our results in Table 4. We find that our pipeline improves results across all detectors even under these conditions, suggesting that paraphrasers exhibit similar biases regardless of what domain they’re applied to.

6 Further Analysis

6.1 Inverting paraphrases of machine-generated text

In this section, we explore the extent to which paraphrases of machine-generated text can be inverted to their original tokens. We expect this task to be easier than inverting paraphrases of human-written text, as human-written tokens exhibit high entropy under LLM distributions (Gehrmann et al., 2019). We train and evaluate all models on Reddit, generating 100 inversions per sample on the held-out test set and report metrics in Table 3. We observe that our model recovers significant portions of the original text, with the best-scoring inversions achieving an average BLEU score of 51, with semantic and stylistic similarities of 0.95 and 0.91, respectively.

6.2 Inverting paraphrases of human-written text

We now turn to the more difficult problem of inverting paraphrases of human-written text. We train and evaluate all models on Reddit, generating 100 inversions per sample on the held-out test set and report metrics in Table 3. We highlight some key observations: (1) Inverting paraphrases
<table>
  <tr>
    <th>Method</th>
    <th>Type</th>
    <th>Style Sim. (↑)</th>
    <th>Semantic Sim. (↑)</th>
    <th>BLEU (↑)</th>
  </tr>
  <tr>
    <td>Paraphrases</td>
    <td>-</td>
    <td>0.61</td>
    <td>0.90</td>
    <td>0.21</td>
  </tr>
  <tr>
    <td>Inversion</td>
    <td>Single</td>
    <td>0.62</td>
    <td>0.88</td>
    <td>0.26</td>
  </tr>
  <tr>
    <td></td>
    <td>Maximum</td>
    <td><b>0.77</b></td>
    <td><b>0.94</b></td>
    <td><b>0.41</b></td>
  </tr>
  <tr>
    <td></td>
    <td>Average</td>
    <td>0.62</td>
    <td>0.88</td>
    <td>0.26</td>
  </tr>
</table>

Table 5: Inverting GPT-4 paraphrases of human-text, an LLM unseen by the inversion model during training time. We generate 100 inversions per sample, and report the metrics achieved by a single inversion, by the best inversion (maximum), and the average across all inversions.

<table>
  <tr>
    <th>Model</th>
    <th>BLEU</th>
  </tr>
  <tr>
    <td>Phi-3</td>
    <td>0.08</td>
  </tr>
  <tr>
    <td>Mistral-7b</td>
    <td>0.11</td>
  </tr>
  <tr>
    <td>Llama-3-8B</td>
    <td>0.08</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>0.23</td>
  </tr>
</table>

Table 6: LLMs prompted to invert their own paraphrases both with, and without in-context examples. Generated 100 inversions per sample, best BLEU score per sample shown. Note that GPT-4 paraphrases already begin with a BLEU score of 0.21 (Table 5).

of human-written text is harder than paraphrases of machine-generated text, with the best scoring inversions achieving an average BLEU score of 25, which is half of that achieved when inverting paraphrases of machine-written text (\S6.1). One reason paraphrases of human-written text may be harder to invert is that human-written words lie in the low-probability (i.e., low-rank) region of an LLM’s predicted distribution, as observed by Gehrmann et al. (2019). (2) output2prompt does not recover significant portions of the original-text, we attribute this to its requirement of observing multiple outputs per prompt, and to the fact that the model has much lower capacity than ours (T5-base vs Mistral-7B).

6.3 Can inversion models invert a novel paraphraser?

To answer this question, we prompt GPT-4, an unseen LLM during training time, to paraphrase the human-written Reddit test set. We use our inversion model trained on Reddit to invert each paraphrase 100 times, and report the metrics in Table 5. Surprisingly, we find that GPT-4 is easier to invert than the models seen during training, with our model achieving a BLEU score of 41. We attribute this to GPT-4 paraphrases retaining more of the original text, with its paraphrases achieving a BLEU score of 21 in contrast to the BLEU score of 8 achieved by the LLMs used for training (Table 3).

6.4 Can an LLM invert its own paraphrases?

We prompt each LLM that generated a paraphrase in our Reddit dataset to invert its own paraphrase. We generate 100 inversions, and report the average maximum BLEU score achieved in Table 6. Overall, we find when prompted, state-of-the-art LLMs, including GPT-4, are unable to invert their own paraphrase. This implies that even if some parametric knowledge encodes the paraphrasing process, the LLM is not able to recover the original text given a paraphrase, further motivating our approach of training paraphrase inversion models.

7 Conclusion

Summary of findings  In this paper, we presented the first detector-agnostic defense against paraphrase attacks. This defense relies on the novel task of paraphrase inversion, where the goal is to recover the original tokens of paraphrased text. Furthermore, we proposed two paraphrase detection schemes: one based upon a neural-classifier and another that relies on the number of edits our inversion model makes. When combined with one of the proposed paraphrase detectors, our pipeline improves the results of 7 machine-text detectors across 3 domains by an average of +22% AUROC. We attribute the effectiveness of our defense to the stylistic similarity of the inverted paraphrases to the original text, which is sufficient for machine text detectors to accurately classify the inverted text. Furthermore, we show that when our defense is trained on one domain, it generalizes to another, suggesting that paraphrasers exhibit consistent biases that can be exploited both for detecting paraphrased text and for learning to invert them.

Limitations

The number of paraphrases we use to train our inversion models is limited by our compute budget. We expect that training on additional LLM-generated paraphrases will improve all the results
reported in the paper; as such, the results reported here should be viewed as a lower bound on achievable performance. Our compute budget also precluded experimenting with larger local models such as Llama-3 70B; however, we do include results with GPT-4 which is of comparable or greater quality.

Acknowledgements

Part of this work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DEAC52-07NA27344. This work was supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract #D2022-2205150003. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

References

Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. Preprint, arXiv:2404.14219.

Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. 2024. Fast-detectgpt: Efficient zero-shot detection of machine-generated text via conditional probability curvature. In ICLR.

Bill Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Third international workshop on paraphrasing (IWP2005).

Alexey Dosovitskiy and Thomas Brox. 2016. Inverting visual representations with convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4829–4837.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Liam Dugan, Alyssa Hwang, Filip Trhlík, Andrew Zhu, Josh Magnus Ludan, Hainiu Xu, Daphne Ippolito, and Chris Callison-Burch. 2024. RAID: A shared benchmark for robust evaluation of machine-generated text detectors. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12463–12492, Bangkok, Thailand. Association for Computational Linguistics.

Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. Ppdb: The paraphrase database. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 758–764.

Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. 2019. Gltr: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 111–116.

Abhimanyu Hans, Avi Schwarzschild, Valeriiia Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2024. Spotting lms with binoculars: Zero-shot detection of machine-generated text. In ICML.

Julian Hazell. 2023. Spear phishing with large language models. Preprint, arXiv:2305.06972.

Xiaomeng Hu, Pin-Yu Chen, and Tsung-Yi Ho. 2023. Radar: Robust ai-text detection via adversarial learning. Advances in neural information processing systems, 36:15077–15095.

Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. 2020. Automatic detection of generated text is easiest when humans are fooled. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1808–1822, Online. Association for Computational Linguistics.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825.

Aleem Khan, Elizabeth Fleming, Noah Schofield, Marcus Bishop, and Nicholas Andrews. 2021. A deep metric learning approach to account linking. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5275–5287, Online. Association for Computational Linguistics.

John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. A watermark for large language models. In International Conference on Machine Learning, pages 17061–17084. PMLR.
Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. 2023. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense. Advances in Neural Information Processing Systems, 36:27469–27500.

Kalpesh Krishna, John Wieting, and Mohit Iyyer. 2020. Reformulating unsupervised style transfer as paraphrase generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 737–762.

Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. 2024. Robust distortion-free watermarks for language models. Transactions on Machine Learning Research.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.

Haoran Li, Mingshi Xu, and Yangqiu Song. 2023. Sentence embedding leaks more information than you expect: Generative embedding inversion attack to recover the whole sentence. In Findings of the Association for Computational Linguistics: ACL 2023, pages 14022–14040, Toronto, Canada. Association for Computational Linguistics.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. Preprint, arXiv:1907.11692.

Aravindh Mahendran and Andrea Vedaldi. 2015. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5188–5196.

John Morris, Volodymyr Kuleshov, Vitaly Shmatikov, and Alexander Rush. 2023. Text embeddings reveal (almost) as much as text. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12448–12460, Singapore. Association for Computational Linguistics.

John Xavier Morris, Wenting Zhao, Justin T Chiu, Vitaly Shmatikov, and Alexander M Rush. 2024. Language model inversion. In The Twelfth International Conference on Learning Representations.

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, et al. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

Martin Potthast, Matthias Hagen, and Benno Stein. 2016. Author obfuscation: Attacking the state of the art in authorship verification. In Conference and Labs of the Evaluation Forum.

Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.

Rafael A. Rivera-Soto, Olivia Elizabeth Miano, Juanita Ordonez, Barry Y. Chen, Aleem Khan, Marcus Bishop, and Nicholas Andrews. 2021. Learning universal authorship representations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 913–919, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Vinu Sankar Sadasivan, Aounou Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. 2025. Can ai-generated text be reliably detected? Preprint, arXiv:2303.11156.

Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, Miles McCain, Alex Newhouse, Jason Blazakis, Kris McGuffie, and Jasmine Wang. 2019. Release strategies and the social impacts of language models. Preprint, arXiv:1908.09203.

Congzheng Song and Ananth Raghunathan. 2020. Information leakage in embedding models. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, CCS ’20, page 377–390, New York, NY, USA. Association for Computing Machinery.

Rafael Alberto Rivera Soto, Kailin Koch, Aleem Khan, Barry Y. Chen, Marcus Bishop, and Nicholas Andrews. 2024. Few-shot detection of machine-generated text using style representations. In The Twelfth International Conference on Learning Representations.

Piotr Teterwak, Chiyuan Zhang, Dilip Krishnan, and Michael C. Mozer. 2021. Understanding invariance via feedforward inversion of discriminatively trained classifiers. Preprint, arXiv:2103.07470.

Andrew Wang, Cristina Aggazzotti, Rebecca Kotula, Rafael Rivera Soto, Marcus Bishop, and Nicholas Andrews. 2023. Can authorship representation learning capture stylistic features? Transactions of the Association for Computational Linguistics, 11:1416–1431.
Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, Zac Kenton, Will Hawkins, Tom Stepleton, Abeba Birhané, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Sean Legassick, Geoffrey Irving, and Jason Gabriel. 2022. Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’22, page 214–229, New York, NY, USA. Association for Computing Machinery.

John Wieting and Kevin Gimpel. 2018. ParaNMT-50M: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 451–462, Melbourne, Australia. Association for Computational Linguistics.

Fan Wu, Emily Black, and Varun Chandrasekaran. 2025. Generative monoculture in large language models. In The Thirteenth International Conference on Learning Representations.

Collin Zhang, John Xavier Morris, and Vitaly Shmatikov. 2024a. Extracting prompts by inverting LLM outputs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14753–14777, Miami, Florida, USA. Association for Computational Linguistics.

Yiming Zhang, Avi Schwarzschild, Nicholas Carlini, J Zico Kolter, and Daphne Ippolito. 2024b. Forcing diffuse distributions out of language models. In First Conference on Language Modeling.

Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1298–1308, Minneapolis, Minnesota. Association for Computational Linguistics.

A Subsampling the Reddit Dataset

We subsample the dataset to authors who post in r/politics and r/PoliticalDiscussion, keeping comments composed of at least 64 tokens but no more than 128 tokens according to the LUAR tokenizer. Furthermore, we remove authors with less than 10 comments, and randomly sample 10 comments from all others, ensuring that no author is over-represented.

To learn to invert paraphrases, we must observe a diverse set of source documents and corresponding paraphrases. However, a random sample of documents may not provide broad enough coverage of writing styles. For example, when we prompt GPT-4 to generate a paraphrase of "HELLO WORLD", it produces "Greetings, Universe!", removing the capital letters. Without observing authors who write only with capital letters during training, it would be impossible for the inversion model to invert the paraphrase. As such, we split authors into training, validation, and testing splits by sampling authors evenly across the stylistic space. We use LUAR (Rivera-Soto et al., 2021), an embedding that captures stylistic features, to embed each author’s posts into a single stylistic embedding. Then, we cluster the dataset using K-Means, setting \( K = 100 \). Finally, we take 80% of the authors from each cluster for training, 10% for validation, and randomly sample 100 authors (2, 449 posts) of those remaining for testing.

B Creating Datasets from RAID

In contrast to our Reddit dataset, the RAID (Dugan et al., 2024) benchmark doesn’t contain author-labels. Therefore, sampling authors evenly across stylistic space as in §4.1 is not possible. RAID contains paraphrases of machine-text using DIPPER, but lacks paraphrases of human-text. To address this, we paraphrase all human-text within ArXiv and MovieReviews with DIPPER, using the same hyper-parameters as the creators of RAID (60 lexical diversity, 0 order diversity, 512 max-tokens). We pair up the machine-text with their corresponding paraphrases, randomly sampling 80% of these pairs for training, 10% for validation, and 10% for testing. Furthermore, ensure that the validation sets contain an equal number of machine-text and paraphrases of machine-text, augmenting them with an equal number of the human-paraphrases we generated. We follow the same procedure for test set, while additionally mensuring that we have exactly 500 samples for each category: human-text, paraphrases of human-text, and paraphrases of machine-text. The validation sets are used to train the edit-based detector discussed in §3.3, while the training sets are used to train both our paraphrase inversion and paraphrase detection models.

C Ablations

How does varying the sampling procedure impact paraphrase inversion? In Table 7 we show the effect that the decoding temperature has in the quality of the inversions generated by our untar-
geted inversion model. We generate 100 inversions for every paraphrase in our test dataset, and report metrics using the “max” scoring strategy discussed in §3. We observe that temperature plays an important role in the quality of the inversions, with values too low or too high significantly degrading the quality of the inversions. As the temperature increases, the entropy of the distribution approximates that of a uniform distribution, thereby diffusing the style of the inversions. Conversely, as the temperature decreases, the inversion model becomes over-confident in its predictive distribution, thereby not exploring neighboring tokens and styles.

<table>
  <tr>
    <th>Temperature</th>
    <th>Style Sim.</th>
    <th>BLEU</th>
  </tr>
  <tr>
    <td>0.3</td>
    <td>0.67</td>
    <td>0.23</td>
  </tr>
  <tr>
    <td>0.5</td>
    <td>0.69</td>
    <td>0.24</td>
  </tr>
  <tr>
    <td>0.6</td>
    <td>0.70</td>
    <td>0.25</td>
  </tr>
  <tr>
    <td>0.7</td>
    <td>0.70</td>
    <td>0.25</td>
  </tr>
  <tr>
    <td>0.8</td>
    <td>0.71</td>
    <td>0.24</td>
  </tr>
  <tr>
    <td>0.9</td>
    <td>0.71</td>
    <td>0.23</td>
  </tr>
  <tr>
    <td>1.5</td>
    <td>0.55</td>
    <td>0.06</td>
  </tr>
</table>

Table 7: Effect of the temperature in the quality of the untargeted inversions.

<table>
  <tr>
    <th>Training Temperature</th>
    <th>Style Sim.</th>
    <th>BLEU</th>
  </tr>
  <tr>
    <td>0.3</td>
    <td>0.71</td>
    <td>0.26</td>
  </tr>
  <tr>
    <td>0.5</td>
    <td>0.70</td>
    <td>0.25</td>
  </tr>
  <tr>
    <td>0.7</td>
    <td>0.70</td>
    <td>0.25</td>
  </tr>
</table>

Table 8: Effect of training on a paraphrase dataset generated with different temperature values.

Are paraphrases generated with lower temperature values easier to invert? To answer this question, we re-generate our human-text paraphrase data with lower temperature values, training and testing the untargeted inversion model in matched temperature conditions. We report the results in Table 8. We observe that, as the temperature decreases, the similarity metrics improve. We attribute this to the LLMs becoming over-confident in their predictive-distribution, thereby generating less diverse data which in turn is easier to invert.

D Prompts

D.1 Paraphrasing

When paraphrasing with an instruction-tuned LLM, we use the following prompt:

Prompt:
Rephrase the following passage:
<PASSAGE>
Only output the rephrased-passage, do not include any other details.
Rephrased passage:

We also clean out all obvious generation artifacts, keeping only the paraphrased text.

D.2 Inversion

D.2.1 Inversion

Prompt:

 The following passage is a mix of human and machine text, recover the original human text: {generation}

\n####\nOutput:
{original}

D.3 Prompting Inversion

Prompt:
The following passage is a mix of human and machine text, recover the original human text:

D.4 Generating Reddit Responses

Prompt:
Write a response to the following Reddit comment: comment

E Dataset Statistics

We show the statistics of the Reddit, RAID-ArXiv, and RAID-MovieReviews in Table 9.

F Training Hyper-Parameters

We train all our inversion models with the hyperparameters shown in Table 10. We train all our models on 4 NVIDIA-A100 GPUs. Each model took at most 10 hours to train.

Most of the compute was spent generating the inversions necessary to run all the experiments, which are in the ballpark of 1M total generations. We used VLLM (Kwon et al., 2023) to speed up the inference time. We estimate an upper bound of around 150 GPU hours to run all experiments.
<table>
  <tr>
    <th>Split</th>
    <th>Number of Examples</th>
  </tr>
  <tr>
    <td colspan="2"><b>Reddit Human-Paraphrase</b></td>
  </tr>
  <tr>
    <td>Train</td>
    <td>204260</td>
  </tr>
  <tr>
    <td>Valid</td>
    <td>24549</td>
  </tr>
  <tr>
    <td>Test</td>
    <td>2449</td>
  </tr>
  <tr>
    <td colspan="2"><b>Reddit Machine-Paraphrase</b></td>
  </tr>
  <tr>
    <td>Train</td>
    <td>239710</td>
  </tr>
  <tr>
    <td>Valid</td>
    <td>28883</td>
  </tr>
  <tr>
    <td>Test</td>
    <td>2854</td>
  </tr>
  <tr>
    <td colspan="2"><b>Reddit Machine-Text Detection</b></td>
  </tr>
  <tr>
    <td>Test</td>
    <td>1500</td>
  </tr>
  <tr>
    <td colspan="2"><b>RAID-ArXiv</b></td>
  </tr>
  <tr>
    <td>Train</td>
    <td>48035</td>
  </tr>
  <tr>
    <td>Valid</td>
    <td>3798</td>
  </tr>
  <tr>
    <td>Test</td>
    <td>1500</td>
  </tr>
  <tr>
    <td colspan="2"><b>RAID-MovieReviews</b></td>
  </tr>
  <tr>
    <td>Train</td>
    <td>25649</td>
  </tr>
  <tr>
    <td>Valid</td>
    <td>1329</td>
  </tr>
  <tr>
    <td>Test</td>
    <td>1500</td>
  </tr>
</table>

Table 9: Statistics of the Reddit, RAID-ArXiv, and RAID-MovieReviews datasets.

<table>
  <tr>
    <th>Hyper-Parameter</th>
    <th>Value.</th>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>2e^{-5}</td>
  </tr>
  <tr>
    <td>Number of Epochs</td>
    <td>4</td>
  </tr>
  <tr>
    <td>LoRA-R</td>
    <td>32</td>
  </tr>
  <tr>
    <td>LoRA-\alpha</td>
    <td>64</td>
  </tr>
  <tr>
    <td>LoRA-Dropout</td>
    <td>0.1</td>
  </tr>
</table>

Table 10: Training Hyper-parameters.

G   Cleaning Obvious Machine-Generated Artifacts

When creating our dataset with the paraphrase prompt in Appendix D, we noticed that the LLMs prompted oftentimes introduced obvious generation artifacts that could be used to easily invert the paraphrases. To ensure that our inversion models don’t rely on any shortcuts, we manually inspected the dataset to find the obvious artifacts and wrote the following code to remove them from the text: https://github.com/rrivera1849/inversion/blob/main/src/utils_clean_gen.py

Reviewer paper summaries: The paper addresses paraphrase attacks on machine-generated text by introducing two novel pipelines that detect and “reverse” paraphrases back to their original form. The first pipeline uses a dedicated classifier to determine if a text is paraphrased; if so, their fine-tuned Mistral-7B inversion model reconstructs an approximation of the original text. The second approach relies solely on the inversion model, using the number of edits it makes as an indicator of whether the text has been paraphrased or not. To evaluate these methods, the authors compile a dataset spanning three domains (Reddit, ArXiv, and MovieReviews), each containing human-written text, paraphrases of human text, and paraphrases of machine-generated text (created using the DIPPER paraphrase model and by prompting Mistral, Llama 3.1 and Phi LLM model to generate paraphrases). Applying the new “inversion” step to paraphrased text before running it through standard machine-generated text detectors yields an AUROC improvement of up to 22%.

This paper proposes a defense method for paraphrase attacks to enhance the robustness of machine-generated text (MGT) detection by using an inversion-paraphrase pipeline. The pipeline consists of three modules: paraphrase detection -> paraphrase inversion -> text detection.
Summary of strengths: 1. This paper presents an interesting approach to combat paraphrasing attacks, using an inversion model to restore the original text. 

2. The paper presents multiple methods to detect paraphrase attacks and also uses multiple machine-generated text detectors to ensure generalizability.

- The approach presents an interesting and novel idea.

- Comprehensive experimental analyses on paraphrase detection and inversion, providing insights into their interaction.
Summary of weaknesses: 1. The paper lacks comparison between human and machine-generated text – or an explanation to clarify why this is not done. All detectors are analyzed on a mixed dataset of human and machine-generated text (Table 1). It would be interesting to know if there is a difference between them. Table 3 shows that there are differences but doesn't show the scores from the machine-generated text detectors. 

2. The paper lacks a clear description of what exactly the AUROC score represents, making it difficult to understand some of the tables. 

3. The creation of the dataset is hard to understand, and important parts required to understand the dataset are only available in the appendix. The authors explain in detail how the Reddit dataset was built, but the methods for creating the ArXiv and MovieReviews datasets are only in the appendix. Since these two datasets differ in how they were constructed, key details should be included in the main text to ensure full reproducibility. 

4. They present AUROC and F1 improvements but never clarify the practical relevance of these metrics for the context investigated. The authors should briefly explain why these scores matter and what different values imply for real-world detection performance. This would provide more depth to the discussions presented. 
5. For the Reddit dataset the paraphrases are generated using different LLM models (Llama, Mistral, Phi) prompted to create a paraphrase. For the Arxiv and MovieReview datasets the authors used the DIPPER paraphrase model. The paper does not test if there are differences in how well the inversion model can recreate the original text based on the used model. They only test how well the same LLM can invert the result, but here they also don't test dipper as this is a specialized model trained only for paraphrasing and not inversion. 

6. There are no details as to why Reddit paraphrases used LLMs and Arxiv and MoveReview used DIPPER. Is there a reason for such? Without details on this, it is hard to compare the obtained results. In particular, the quality of generated paraphrases regarding how hard they are to detect.

While the overall motivation is intriguing and generally justified through experiments, the concept of paraphrase inversion raises some concerns. Specifically, the authors mention in Section 6.4 that a certain LLM, when tested on its own generated texts, performs poorly in inversion. However, when switching to GPT-4 (as shown in Table 3), performance significantly improves (BLEU score increases from 0.11 for Mistral to 0.21 for GPT-4). It remains unclear what the inversion results would be if GPT-4 were used to invert GPT-4-generated paraphrases directly. Clarifying this would strengthen the analysis.

The necessity of the paraphrase inversion module is not clearly justified. As illustrated in Appendix D.3, the inversion prompt appears to request another paraphrase with outputs closer to human-like text. This raises the question of whether a simpler alternative—such as directly weighting the outputs of the paraphrase detector and the MGT detector (e.g., \theta \times score_p + (1−\theta) \times score_d)—might achieve comparable or superior performance without the added complexity of inversion.
Comments, suggestions and typos: 1. Page 4: MLP was introduced as an acronym but never used afterwards 

2. Add axis labels to Figure 2 

3. Use a more colorblind friendly color pallet for Figure 2 

4. Provide more decimal places in Table 8 

5. Inconsistent use of bleu score as a decimal number and integer in chapter 6.1, 6.3 and table 3 

6. Figures/Tables need clearer descriptions of what can be seen. Table 2 shows F1 scores, but this is never mentioned in the text and does not clearly describe what it means in this context. Overall, Table 2 is only shortly mentioned in Table 2 but not properly analyzed. 

7. It would be interesting to see if this inversion process also helps with identifying plagiarism in scientific work that was covered by using LLMs to paraphrase the text. 

8. Could you separately analyze performance on human vs. machine paraphrases? Having distinct scores for each subset would clarify whether your method behaves differently in the two categories. 

9. The overall research direction of the paper is very interesting, but the current version of the paper lacks some details, as described above. With these comments and weaknesses addressed a better score is possible.

see weakness
Overall assessment scores: 3.5

3
Metareview score: 4
Metareview: This study focuses on paraphrasing attacks, which challenge machine-text detectors by paraphrasing generated text. To address this issue, the study proposes paraphrase inversion, a method that translates paraphrased text back into its original form. Although conceptually simple, this approach effectively improves the overall performance when combined with various machine-text detectors.
Summary of reasons to publish: 1. This approach is interesting and novel.
2. The experimental analyses are comprehensive.