PaLD: Detection of Text Partially Written by Large Language Models

Authors: Eric Lei, Hsiang Hsu, Chun-Fu Chen
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: LLM text detection, mixed-text detection, distribution shift

Abstract:
Advances in large language models (LLM) have produced text that appears increasingly human-like and difficult to detect with the human eye. In order to mitigate the impact of misusing LLM-generated texts, e.g., copyright infringement, fair student assessment, fraud, and other societally harmful LLM usage, a line of work on detecting human and LLM-written text has been explored. While recent work has focused on classifying entire text samples (e.g., paragraphs) as human or LLM-written, this paper investigates a more realistic setting of mixed-text, where the text's individual segments (e.g., sentences) could each be written by either a human or an LLM. A text encountered in practical usage cannot generally be assumed to be fully human or fully LLM-written; simply predicting whether it is human or LLM-written is insufficient as it does not provide the user with full context on its origins, such as the amount of LLM-written text, or locating the LLM-written parts. Therefore, we study two relevant problems in the mixed-text setting: (i) estimating the percentage of a text that was LLM-written, and (ii) determining which segments were LLM-written. To this end, we propose Partial-LLM Detector (PaLD), a black-box method that leverages the scores of text classifiers. Experimentally, we demonstrate the effectiveness of PaLD compared to baseline methods that build on existing LLM text detectors.

Decision: accept (poster)
Reviewer scores: 6, 5, 8, 6

Reviews and discussion:
<thread>
creation date:
1730166302035

thread_type:
official_review

<thread_object>
creator: reviewer_qntj

<content>
summary:
This paper explores methods for detecting human and LLM-generated text in a mixed-text environment. A detector called PaLD is proposed, which is used for estimating the proportion of LLM-generated content in a text and determining which passages are written by LLMs.

strengths:
The paper tackles the mixed text problem through the PaLD method, which is an innovation in current LLM detection techniques. Above all, this paper has a complete structure and provides relatively comprehensive experiments.

weaknesses:
1. The effectiveness of this method seems contingent upon accurate segmentation of text into LLM and human-written parts. It would be helpful to explore how it performs under less ideal segmentation conditions or how it handles errors in segmentation. 

2. The article mentions the performance of cross-domain detection, but just two types are involved here, which seems to be insufficient to validate the generalization ability of the model.

questions:
1. The paper used quantitative T-score and KDE to analyze or differentiate text. It is suggested that the authors provide more detailed basis for parameter setting, such as the choice of bandwidth for KDE and how to determine the threshold of T-score.

2.  Authors provide performance metrics of the algorithm such as accuracy and coverage-accuracy trade-offs, but lacks the analysis of misclassifications (false positives and false negatives). It is recommended to explore or discuss in detail the possible causes of misclassification.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730586082471

thread_type:
official_review

<thread_object>
creator: reviewer_1fyb

<content>
summary:
This work challenges traditional binary views of machine-generated text detection for classifying entire text samples as either human or LLM-written, and instead studies detecting texts partially written by LLMs. The experiments consist of two main parts: a) estimating the percentage of LLM-written texts; b) identifying LLM-written segments.

strengths:
1. The problem raised by this paper is very interesting and worth studying. This makes this paper have great academic impacts.
2. This paper overall is well-written. The presentation and structure are clear and concise, with well-crafted figures and only a few typos.
3. In-depth theoretical formulation and justification of the statistical method chosen.

weaknesses:
1. Computational complexity: As the authors have acknowledged, scaling this method over many segments could be challenging. This poses limits to its applicability. In my opinion, this is a relatively minor issue as this partial detection problem is inherently harder, and we should encourage solutions to hard problems. This should be left for future works to work on.

2. Need more robustness analysis: My main criticism is that this method lacks more sensitivity analysis regarding to segment lengths and various forms of perturbation attacks. How would it perform when there are adversarial attacks trying to undermine the detection is another main problem we should consider. I understand the paper has limited scope and cannot study everything, but I would assign a more affirmative rating had the paper dedicated a section for robustness analysis.

3. This method also requires training with regards to the specialized domain, which makes it less generalizable. Although the authors have studied cross-domain performance, the results are less convincing as only two domains have been studied, and maybe domains with more distinct writing styles have more drastic performance difference. I point it out not because I want to ask the authors to do additional experiments on new domains (although you are welcomed to!), but it’s just that I think it is that the nature of the methods makes training necessary, and this weakness cannot be easily avoided.

4. A minor point: I would suggest to have the related work section review every existing work studying partially LLM-written text detection (I’ve searched for some and did not find too many), and reduce the part on watermarking methods as they are less relevant to this paper. A literature review in this understudied area would be very helpful!

questions:
No further questions. Overall, I like the paper and would like to give positive ratings. Although there are imperfections, I think this work is valuable, memorable, and should be encouraged.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730628019980

thread_type:
official_review

<thread_object>
creator: reviewer_3hmg

<content>
summary:
The paper introduces Partial-LLM Detector (PaLD), a method designed to estimate what percentage of a text is generated by an LLM and identify which specific segments are machine-generated. This approach uses black-box methods leveraging scores from text classifiers to achieve these tasks. It offers a more nuanced detection capability compared to previous methods that only classify texts as either human or machine-generated in entirety.

strengths:
1. This method allows for a more detailed understanding of the text composition by identifying LLM-generated segments.
2. The proposed method can handle texts of mixed origin, reflecting more realistic scenarios than those handled by traditional binary classifiers.
3. Provides statistically robust methods to back its detections, including confidence intervals and likelihood measures.

weaknesses:
1. Some of the concepts and mathematical notations need further explanation for readers' better understanding.
2. The experimental setup needs improvement. Currently, experiments are conducted on only two specific datasets, so additional validation is required to assess generalizability. Moreover, the proposed method appears to rely on prior knowledge about distributional information, which could further limit its generalization ability.

questions:
1. Line 93, could you please define the term 'robust' in a human-written-/LLM-text detection problem (e.g., what types of changes, uncertainties, and unexpected perturbations in input data are considered)?
2. Lines 90-100, Bayesian framework, MAP estimation, mixture Gaussian models, and greedy policy algorithm have already been widely used in text detection or related NLP problems, please provide some references to them.
3. Line 112, '..., and x \in \chi ...' \chi needs to be defined before the first time use it (e.g., is it a set or list).
4. Line 154, how do you determine the number of decomposed segments in real applications? Is it equal to the number of sentences in a paragraph?
5. Lines 190-193, are \eta and \delta different from each segment Xi? Or they are calculated over the entire mixed-text paragraph?
6. Lines 205-206, is \epsilon a pre-defined hyperparameter? Since it determines the range of p, will the optimal \epsilon depend on the distributions P_llm and P_human, and how to choose the optimal \epsilon?
7. The example in Fig. 6 only consists of 3 sentences. If the text has more sentences, will the T-score of all the combinations be computed? Or just 2-sentence combinations will be considered?
8. Lines 324-326, a greedy algorithm is employed to approximately solve the problem and reduce complexity. Could you briefly discuss the approximation errors introduced by this algorithm and whether or not they impact detection accuracy?
9. In section 4.2, the cross-domain detection experiment is particularly intriguing, and I can somewhat understand that this test is generally more challenging compared to in-domain detection. Could you please briefly describe whether the WP and Yelp data are correlated (or consist of some similar information) before we can safely reach the conclusion?
10. The experiments are conducted on only two specific datasets, so additional validation is required to assess generalizability. I am also curious about the performance of detection in more challenging cases, such as humans mimicking the LLM style or LLM imitating highly stylized human writing. These can be mentioned in future works or current limitations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730690481235

thread_type:
official_review

<thread_object>
creator: reviewer_duty

<content>
summary:
The authors introduce PALD-PE and PALD-TI, methods that estimate the percentage of LLM-generated content in a piece of text, and an approach to identifying the LLM-generated segments, respectively. PALD-PE provides MAP estimates of the percentage of LLM-generated text, and using gaussian kernel density estimation, they provide intervals that cover the ground-truth percentage. PALD-TI runs a combinatorial search over all possible segments using a score from a text-classifier to identify the LLM-generated segments. Furthermore, a greedy version of the PALD-TI approach is introduced. The authors evaluate their approach on the Writing Prompts and Yelp Dataset.

strengths:
•	It is one of the first papers to address the more realistic mixed-text scenario and proposes to estimate the amount of text that is machine-generated, as well as to identify the segments that are machine-generated. This is a departure from the usual machine-or-human detection. 

•	The paper includes rigorous ablations of the PALD-PE method, which greatly led credence to the observations that methods with a higher quantile slope were more useful in mix-text scenarios.

weaknesses:
o	The fine-tuned RoBERTa baselines were trained to identify between fully human and fully-LLM texts that are larger than a single sentence in length, but then tested on individual sentences. This introduces a distribution mismatch during testing that might make the baselines weaker than they really should be. 

o	For DetectGPT, and FastDetectGPT, the threshold should’ve been derived from a validation set to maximize their performance at the segment level identification task.  

o	In Table 3, PALD TI relies on a supervised model, whereas DetectGPT and FastDetectGPT do not so it’s unfair to compare one against the other. It would be good to have a zero-shot version of PALD. It would’ve been interesting to test them against a version of PALD-TI which relies upon an unsupervised T-score. 

o	The datasets and testing conditions considered are not sufficient. In particular, the work could’ve also been easily evaluated on the RoFT dataset (https://arxiv.org/abs/2212.12672), and it could’ve also been built off of RAID (https://arxiv.org/abs/2405.07940) and the M4 dataset (https://github.com/mbzuai-nlp/M4), both of which include more realistic testing domains. 

o	Only GPT-4o was used when infilling. More LLMs should’ve been considered, including non-instruction-tuned LLMs.

•	The PALD-TI approach relies heavily on a T-Score from literature such as a trained RoBERTa model, FastDetectGPT, DetectGPT, etc. It uses them to perform a combinatorial search of 2^n – 2. A more efficient method for search that is not just greedy would make the method much stronger.

questions:
•	Line 191, wrong notation? X_i instead of X?

•	Typo 297: performnace -> performance

•	What was the reason for having different fractions set for the testing split than for the training split? 

•	Any insights as to why the quantile slope of DetectGPT is smaller, but the segment accuracy is stronger than that of GhostBuster and FastDetectGPT?

•	Were PALD-TI and PALD-PE evaluated on cases where there were no machine segments? How about in cases where there are only machine segments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732168164879

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Initial Common Response to Reviewers

comment:
We appreciate the reviewers taking the time to read our paper and provide helpful comments and feedback. We thank the reviewers for recognizing the impactfulness of PaLD, as it is the first work to initiate study of and provide solutions for the more realistic mixed-text LLM detection setting. In this common response, we first address the concern about more experiments and domains made by multiple reviewers. To address the reviewers' individual concerns, we have responded to each reviewer's comments below with a point-by-point reply, and hope they help alleviate the reviewer's concerns. The manuscript will be updated soon to reflect suggested changes.

**New experiments and domains**: A common feedback from all reviewers was to provide more experimental results on datasets or domains aside from the WritingPrompt (WP) and Yelp datasets. As suggested by reviewer DUtY, we are currently evaluating our method on the RoFT and RAID datasets, and plan to update this thread and the manuscript with results once they are complete. The RoFT dataset contains texts from domains such as recipes, New York Times articles, and short stories. The mixed-texts are generated by prompting GPT2-XL to complete $10-k$ sentences given the first $k$ human-written sentences as an input prompt. The RAID dataset contains domains such as paper abstracts and Wikipedia articles. RAID does contains mixed texts, so we apply the same mask-and-fill approach described in the manuscript (cf. Appendix A.1).

**Complexity of PaLD-TI and greedy algorithm**: Another concern brought up by several reviewers is about the computational efficiency of PaLD-TI. As mentioned in the paper, it is a combinatorial problem (actually NP-hard) with complexity exponential in the number of segments $n$. The greedy algorithm already significantly reduces runtime for texts with $n$ segments, from $O(2^n)$ to $O(n^2)$, as the greedy method requires $O(n)$ complexity per iteration. Additionally, greedy has no hyperpameter choices, is straightforward to implement, and unlike many other approximate methods, is amenable to parallel batch computation, as computing $f(S \cup e) - f(S)$ over all $e \in A$ in Alg. 1 can be batched together. As experimentally verified in Table 3, PaLD-TI with greedy already outperforms all other baselines by at least 9\% in terms of segment accuracy. Determining if other methods may achieve a better performance-complexity tradeoff compared to the greedy algorithm could be research left for future work, as the current paper focuses on formulations for solving the text identification problem itself. Regarding performance guarantees, another potential benefit of greedy is that it returns a $(1-1/e)$ approximation of the optimal when the objective is submodular. Additionally, solving PaLD-TI exactly is reasonably efficient for texts of up to 10 segments, which is what is considered in this paper. To give concrete running times on a single A10 GPU, on average, running PaLD-TI with exact solver takes $\approx 30$ seconds for a paragraph-level text of 10 segments (around 200 words), whereas running PaLD-TI with greedy solver takes $\approx 2.1$ seconds; this is with the RoBERTa-based $T$-scores. If a text with many segments is encountered in practice, a more practical method may be to divide the text into chunks, and run greedy/exact on those chunks. For a chunk size $b$ on $n$ segments, solving PaLD-TI exactly would have complexity $O((n/b) 2^b)$, and with greedy $O((n/b) b^2)$. We plan to update the manuscript with these discussions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732504580377

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Second Common Response to Reviewers

comment:
Again, we thank all the reviewers for their time in providing us with helpful feedback and interactions. We have now **updated the manuscript**, with edited content highlighted in blue. 

## Experiments on new domains and LLMs

Regarding the common feedback from all reviewers to provide more experimental results on additional domains, we have added the following:

- Additional results on mixed-texts written by a different LLM (Claude-3.5-Sonnet), in Sec. 4.2, Table 6 (reviewer DuTY)
- Additional experiments on the RAID and RoFT datasets, a total of 5 new domains, shown in Sec. 4.2, Tables 5 and 6. These datasets are used to both further support the cross-domain generalization performance, as well as cross-LLM generalization performance (all reviewers)


We summarize the results here. 

**Fixed-LLM**: We first keep the LLM that generates the mixed-texts as GPT-4o, and evaluate PaLD when the data domain during test time shifts. For PaLD-PE, the likelihood fitting phase is performed on the WritingPrompts (WP) dataset, and the point estimate or predictive interval is estimated on the testing dataset of a different domain;
similarly, the baselines are trained on WP and evaluated on a different testing domain. The testing domain datasets we evaluate on include Yelp and RAID. For RAID, we use the Abstracts and Wiki domains. For text identification, PaLD-TI does not require training, but may use a $T$-score that is fitted to a dataset (e.g., RoBERTa-LN trained on WP). The FastDetectGPT baseline (applied segment-wise) has threshold set on WP, and RoBERTa (segment-wise) is trained on WP. The results are shown here:

Cross-domain fixed-LLM results for percentage estimation (mean absolute error) and text identification (segment-wise accuracy). Training dataset: WP; LLM: GPT-4o.

| Test Dataset | PaLD-PE | RoBERTa-Reg | RoBERTa-QuantileReg | PaLD-TI | FastDetect-GPT-Seg | RoBERTa-Seg |
|--------------|---------|-------------|--------------------|---------|--------------------|--------------|
| Yelp | **0.147** | 0.211 | 0.231 | **0.801** | 0.626 | 0.592 |
| Abstracts | **0.151** | 0.306 | 0.223 | **0.718** | 0.693 | 0.656 |
| Wiki | **0.224** | 0.313 | 0.238 | 0.623 | 0.636 | **0.735** |

PaLD-PE remains the best-performing method compared to the baselines. This demonstrates that PaLD-PE has a superior ability to generalize across domains,
We see that PaLD-TI is superior to baselines on Yelp and Abstracts but not Wiki, where RoBERTa-Seg achieves high segment-wise accuracy.

**Cross-LLM**: Then, we evaluate the case when the LLM that generated the mixed-text may also be different at test time. We use a similar setup to the previous section. In particular, we evaluated PaLD and baselines trained/fitted on WP (GPT-4o-written) with the testing domains set as WP, Yelp, and RAID (Abstracts, Wiki) generated using Claude-3.5-Sonnet. We also evaluate RoFT, which are mixed texts where the first $k$ sentences are human, and remaining $10-k$ are GPT2-written; we use the Short Stories (SS), Recipes, and New York Times (NYT) domains. Note all are cross-domain and cross-LLM except for WP, which is only cross-LLM.

| LLM | Test Dataset | PaLD-PE | RoBERTa-Reg | RoBERTa-QuantileReg | PaLD-TI | FastDetect-GPT-Seg | RoBERTa-Seg |
|-------|--------------|---------|-------------|--------------------|---------|--------------------|--------------|
| Claude | WP | **0.114** | 0.184 | 0.138 | **0.751** | 0.531 | 0.675 |
| Claude | Yelp | **0.128** | 0.193 | 0.144 | **0.731** | 0.628 | 0.654 |
| Claude | Abstracts | **0.115** | 0.204 | 0.145 | **0.710** | 0.633 | 0.651 |
| Claude | Wiki | **0.174** | 0.245 | 0.185 | 0.602 | 0.634 | **0.683** |
| GPT-2 | SS | **0.227** | 0.236 | **0.227** | 0.533 | **0.610** | 0.525 |
| GPT-2 | Recipes | **0.166** | 0.209 | 0.186 | **0.548** | 0.520 | 0.505 |
| GPT-2 | NYT | **0.236** | 0.246 | 0.239 | 0.477 | **0.613** | 0.544 |

On Claude, PaLD-PE outperforms all baselines despite the domain and LLM shift. PaLD-TI is superior on all domains except Wiki, where RoBERTa-Seg again achieves high accuracy. For GPT-2 (i.e., RoFT data), PaLD-PE performs similar or better. For the TI task, both PaLD-TI and RoBERTa-Seg result in a significant performance decrease compared to FastDetectGPT; this may be due to poor generalization to GPT-2 texts of the RoBERTa-based classifiers that is used for both RoBERTa-Seg and the $T$-score for PaLD-TI. The better generalization to Claude from GPT-4o compared to GPT-2 may be expected, as the parameter count of GPT-2 is orders of magnitude lower compared to Claude and GPT-4o.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces Partial-LLM Detector (PaLD), a method designed to address the mixed-text scenario, where texts are partially generated by LLMs and partially written by humans. The proposed methods, PALD-PE and PALD-TI, estimate the percentage of LLM-generated content and identify specific LLM-generated segments, respectively. The authors evaluate their methods on the Writing Prompts and Yelp datasets and propose improvements such as a greedy combinatorial search to enhance scalability.

This paper makes a great contribution to the field by tackling the mixed-text problem, an important yet underexplored area in LLM detection. The methods are innovative, the experiments are comprehensive, and the results are promising. While there are areas for improvement, such as dataset diversity and computational scalability, the strengths of the paper outweigh its weaknesses. I recommend acceptance with minor revisions to address these concerns in future iterations.