Online Detecting LLM-Generated Texts via Sequential Hypothesis Testing by Betting

Authors: Can Chen, Jun-Kun Wang
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models (LLMs), Machine-Generated Text, Text Generation Detection, Sequential Hypothesis Testing, Online Optimization

Abstract:
Developing algorithms to differentiate between machine-generated texts and human-written texts has garnered substantial attention in recent years. Existing methods in this direction typically concern an offline setting where a dataset containing a mix of real and machine-generated texts is given upfront, and the task is to determine whether each sample in the dataset is from a large language model (LLM) or a human. However, in many practical scenarios, sources such as news websites, social media accounts, or on other forums publish content in a streaming fashion. Therefore, in this online scenario, how to quickly and accurately determine whether the source is an LLM with strong statistical guarantees is crucial for these media or platforms to function effectively and prevent the spread of misinformation and other potential misuse of LLMs. To tackle the problem of online detection, we develop an algorithm based on the techniques of sequential hypothesis testing by betting that not only builds upon and complements existing offline detection techniques but also enjoys statistical guarantees, which include a controlled false positive rate and the expected time to correctly identify a source as an LLM. Experiments were conducted to demonstrate the effectiveness of our method.

Decision: reject
Reviewer scores: 8, 6, 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1730214378605

thread_type:
official_review

<thread_object>
creator: reviewer_ndvj

<content>
summary:
This paper proposes to detect LLM-generated texts in an online streaming scenario, addressing the gap in current detection methods that primarily operate offline. The authors introduce an algorithm based on sequential hypothesis testing through a betting mechanism, allowing for real-time, statistically robust detection with controlled false-positive rates and bounded detection times. Experimental results demonstrate that the method can effectively distinguish LLM-generated text sequences from human-written texts, even under practical constraints.

strengths:
* The paper explores the detection of generated text in a streaming context, which is timely and might be useful for real-world applications such as monitoring social media or news platforms.
* The authors build on the theory of sequential hypothesis testing by betting, establishing clear statistical guarantees. This foundation provides rigor, including control over type-I errors and expected detection times, which is valuable for applications requiring reliable, real-time decisions.
* The authors test their approach using different scoring models, source models, and baselines, showing a commendable effort in evaluating robustness and sensitivity to various parameters.

weaknesses:
* The scope of the paper’s application could be better defined. For example, is the goal to identify whether the content in a stream is entirely machine-generated or to ascertain the overall nature of the source (e.g., a bot account)? If the latter, the assumption that all content from a source is generated by a single LLM may not hold in practice, where mixed content (mixed LLMs or mixed human and LLM-generated text) is common. How the method would handle mixed content or bot accounts using various LLMs is unclear.
* The experiments rely on a single dataset and use a relatively small amount of data, which weakens the persuasiveness of the results. Furthermore, the generated texts tested in this study are produced by limited models; the evaluation would be more compelling if it included results from more advanced models, such as LLaMA 3, Mistral, or GPT-4.
* The writing can be polished for clarity and completeness. Many of the experimental result tables are placed in the appendix, which could have been included in the main text given there is remaining space.

questions:
* How does the method handle situations where the source content may contain a mixture of human-written and machine-generated text?
* Could the sequential hypothesis testing technique adapt dynamically to model drift or evolution in LLMs over time?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730725741644

thread_type:
official_review

<thread_object>
creator: reviewer_87z9

<content>
summary:
This paper proposes a novel method for the online detection of text generated by large language models (LLMs), employing sequential hypothesis testing based on betting techniques. The approach aims to provide robust statistical guarantees, balancing type-I error control with efficient detection times. By leveraging sequential hypothesis testing, the method ensures accurate identification of LLM-generated text while minimizing false positives.

strengths:
1. The application of sequential hypothesis testing with betting is innovative and well-suited to the online detection problem, distinguishing this approach from conventional offline methods.
2. The method includes strict statistical guarantees for false positive rates and expected detection times, which is critical for applications requiring high reliability.
3. The paper demonstrates solid technical grounding, and a wealth of theoretical knowledge, and is well-written.

weaknesses:
1. The abstract lacks clarity, particularly in the phrasing "publish content in a streaming fashion." The motivation should emphasize the need for continuous online detection due to the constant emergence of text, which differs from offline detection. This ongoing process allows for enhanced detection performance through hypothesis testing.
2. Based on this motivation, the baseline experimental setup appears problematic. It should compare your method with current offline approaches (e.g., detection without leveraging continuous information) rather than with other hypothesis tests, to better evaluate effectiveness and identify advantages and disadvantages.
3. The scoring model has not been optimized for this task. While experiments on text domain matching were conducted, no solutions were proposed. The experiments are somewhat limited; the focus should be on the modules and parameter settings of sequential hypothesis testing compared to offline methods. Ablation studies should emphasize this point.

questions:
1. Why introduce a new dataset? Are the existing datasets inadequate? Is 500 samples insufficient for credibility? Why generate only one new dataset instead of two or three? How can we ensure that the series of input texts y_x come from the same source?
2. The examples provided at the beginning are unconvincing; they primarily address account and regulatory issues. Why not present more compelling examples that better reflect your motivation for the revision?
3. For hyperparameters such as d_* and epsilon, is there an optimal selection method? What are the trade-offs involved?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731031001596

thread_type:
official_review

<thread_object>
creator: reviewer_p6pn

<content>
summary:
The authors study the problem of inferring whether a source of texts is human or an LLM in an online setting, i.e, in which texts are observed as a continuous stream, as in social media and online forums.

The paper employs a sequential hypothesis testing framework, allowing the judgment about the likelihood of a source being a LLM to be made dynamically rather than waiting for the entire dataset. The framework is fed with state-of-the-art offline approaches to detect LLM-generated text, and at each step the "wealth" of the system is updated based on the probability of a text being generated by an LLM. This accumulated wealth serves as evidence to reject the null hypothesis (that the text was written by a human) and declare the source as an LLM. 

The paper discusses the statistical guarantees of the algorithm, including the controlled false positive rate and the average time to correctly detect an LLM. Additionally, experimental results are presented, demonstrating the effectiveness of the proposed method compared to baseline methods.

strengths:
The paper is well-written paper, easy to understand, the method is defined clearly. The experimental results, considering the absolute numbers,  seem to outperform the employed baselines. The method is intuitive and the weight assignment strategy may help the interpretability of the results. The originality is a bit limited, since it basically adapts sequential hypothesis testing for detecting LLM-generated texts. The significance is still rather limited since they use a dataset that they constructed and it is not published or employed in other works.

weaknesses:
First of all, I believe that authors should better characterize how online detection differs from the offline detection from an algorithmic perspective. I understand that latency and throughput became main requirements, but it is not clear to me, with respect to content handling how it differs from the offline problem, for which there are a large number of works published. Interestingly enough, there are no computational performance assessments in the paper.

The evaluation seems to me as the main weakness to the manuscript. There seems to be several design and experimental decisions that are quite adhoc. For instance, the authors mention that they used 10 score functions in total. How did these functions chosen? Why? Similar issues arise regarding the dataset, which were constructed by the authors and it is not detailed how. For instance, they mention that they generate fake news considering the first 30 tokens of each real news. Is there a validation that the generated fake news are really fake? To what extent?  I also missed more actual baselines that have been employed on the task or similar ones. 

Although the authors provided a quite detailed description of the experimental setup, it is not clear to what extent it is reproducible. For instance, in section 4.1 the authors mention that they constructed a dataset containing both real and fake news, but both the construction process and the characteristics of the resulting dataset are not provided.

questions:
I miss a contextualization of the state of the art of LLM-generated text detection – are those algorithms effective?

It seems that an important characteristic of the problem tackled by the authors is that the paper tries to classify sources instead of individual documents – hence, the signal and confidence is dramatically higher than when you have to judge a single document. Consider highlighting this and take a look at https://arxiv.org/html/2406.07016v1.

We do need both validation and characterization of the dataset used in the experiments.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731038480282

thread_type:
official_review

<thread_object>
creator: reviewer_xdtt

<content>
summary:
The paper addresses the challenge of detecting large language model (LLM)-generated text in an online, streaming context. Traditional methods focus on offline detection, but the authors propose a sequential hypothesis testing algorithm that enables real-time detection with statistical guarantees, such as controlling the false positive rate and expected detection time. This method leverages online optimization and betting strategies to discern between human and LLM-generated content across platforms like social media and news websites.

strengths:
S1. The problem of detecting LLM-generated text in an online, streaming context is important.

S2. The problem statement and the rationale behind the method are clear.

S3. The algorithm ensures a controlled false positive rate and has bounded detection time, which is crucial for reliability.

S4. The evaluation is very comprehensive. The paper tests the method with multiple score functions, datasets, and LLM configurations.

weaknesses:
1. The method relies heavily on existing score functions from offline detection methods, potentially limiting its adaptability to new types of LLM-generated content. Certain score functions perform significantly better than others, indicating that the algorithm's efficacy may vary depending on the selected score function and dataset.

2. The experiments primarily focus on LLM-generated news about the Olympics and a pre-collected set of human-written news articles from XSum. This setup is limited in scope and may not reflect the diversity of content types that an LLM could generate. Would additional real-world validations in social media or low-quality content improve confidence in the algorithm’s effectiveness across various online platforms? (This comment doesn't lower my rating for this article, but rather my expectations for follow-up work)

3. The algorithm requires estimations for parameters like $\epsilon$ and $d$ before the main process begins. This estimation is based on initial samples, which introduces the risk of inaccurate settings, especially if the initial sample size is small. Can further optimization in estimating parameters improve consistency across different datasets and score functions?

questions:
Please see Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731076002092

thread_type:
official_review

<thread_object>
creator: reviewer_y933

<content>
summary:
In this paper, the authors present an online sequential hypothesis test for flagging LLM texts. At each time step, the algorithm updates an accumulated reward. 

The reward increases whenever an algorithm scores higher for an LLM text, indicating that we have evidence to reject the null. When this is not the case, the authors control the reward so that it is never negative via an Online Newton algorithm. This is simple because the score functions are bounded (commonly between 0 and 1).

Proofs that the algorithm works are adapted from the literature on sequential hypothesis tests. I point out that this is now my area of expertise, but they seem reasonable. Overall, being one of the first approaches of this kind, the algorithm naturally achieves good results (the baseline is a quite simple approach from 10 years ago).

strengths:
- Based on the authors' text, this is the first (if not the first) work to pose this problem. The algorithm is relatively easy to understand, although the presentation could improve slightly (see weaknesses) the first.

- The authors experiment with several different LLM detectors, showing how their method is easily adaptable to several LLM detection models.

- An extensive discussion of the models used and details of the method are presented in the appendixes. I did not double-check these; I only skimmed them, but they are welcomed.

weaknesses:
Please provide an intuition for Line 11 earlier in the text. This is a crucial step of the algorithm, and I searched for the term "dt" on the pdf to better understand it. Ultimately, it is pretty intuitive since the score function is bounded. Nevertheless, with the explanation being brief and after the algorithm, it took me some time to understand it.

The paper does not discuss in detail why this approach is necessary, which is a significant weakness. Online testing may be employed, but is it the correct approach? For instance, how can I use this when evaluating student essays? I have only one essay per student. Maybe we could accumulate evidence that the account is a bot on a social media page. However, has this not already been done in batches? 

The point above is not a weakness of the method but of the use case. Please clarify why this method is needed. Some examples of online forums are present, but were these attacks sequential? Were they in batches?

questions:
See weaknesses. In particular, please answer the questions on the use case.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732494355372

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revisions and Responses to Reviewers

comment:
We thank the reviewers for taking their time to read our paper and providing constructive feedback. We would like to take the opportunity to summarize the updates that we have made during this rebuttal period.

A couple of reviewers commented that our experiments were conducted using a single dataset and/or using a new dataset rather than existing ones. We have clarified that additional experimental results for online testing are provided in Appendix H of our paper, where samples of LLM-generated texts and human-written texts (from datasets such as XSum, WritingPrompts, and PubMed) are from existing sources in the literature [1], with the LLM-generated texts produced by models such as GPT-3, ChatGPT, and GPT-4. We created a new dataset of Olympic 2024 news because we would like to validate the effectiveness of our approach in a more realistic scenario. Our codes and the new dataset have been uploaded for the reader to reproduce the results. As suggested by Reviewer p6Pn, we also have described the process of constructing the dataset and has included it in the updated version.

Second, to address the concern/question by Reviewer XdTt about specifying the parameters in our algorithm, we have added a new experimental result, where we double the number of rounds in the initial stage of estimating the parameter $d_t$ and $\epsilon$ (from $10$ to $20$). The results (Figure 5~(b) on page 10) indicate that using a larger sample size for estimation could potentially yield better outcomes.  We also emphasize that if prior knowledge is available, estimating the required parameters is unnecessary. For instance, if one already knows the output range of an off-the-shelf score function and the discrepancy in scores written by different humans, the estimation is not required. This corresponds to the oracle scenario in the experiments, which could be realistic in certain cases. 
Additionally, we have included two more new experiments (Figure 6 on page 10), as suggested by Reviewer NdvJ, where we consider (1) tackling the case when the sequence of texts are generated by a few LLMs instead of a single one, (2) when the sequence contains a mix of human-written texts and LLM-generated texts, rather than exclusively one or the other. The new experimental results demonstrate our method effectively handles these scenarios as well.

In terms of writing/presentation, we have updated our paper accordingly based on the reviewers' feedback. As suggested by Reviewer y933 and Reviewer p6Pn, we have updated and highlighted in the paper the the online testing that our paper concerns (i.e., 
texts are observed sequentially, and one has to maintain an any-time valid level test while controlling the time to declare the source are LLMs when they are) can be particularly useful when one seeks substantial savings in both data collection and time without compromising the reliability of their statistical testing, which might be challenging for approaches that based on collecting data in batch (or offline) to achieve. To the best of our knowledge, this is a new scenario, whereas related works on offline detection focus on classifying a given set of texts without emphasizing such kind of statistical guarantees and may not be directly applicable to online detection. The tasks (i.e., online vs. offline) are fundamentally different, and therefore the corresponding methods might not be directly comparable. Specifically, we highlight in our response to Reviewer 87z9 that many state-of-the-art offline detection methods aim to determine whether a text is from an LLM by comparing its evaluated text score to a pre-specified threshold. A straightforward way to adapt an offline detector to our scenario is to declare the source as LLM (and stop) at the point where the detector predicts that a sample is written by an LLM. However, this approach will have a false positive rate of 1 when the number of rounds becomes sufficiently large, unless the offline detector never makes an error in recognizing human texts.

[1] Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. Fast-detectgpt: Efficient zero-shot detection of machine-generated text via conditional probability curvature. arXiv preprint arXiv:2310.05130, 2023.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper tackles the problem of detecting whether a text is either written by a large language model or a human in an online setting. To this end, it reduces the problem to a sequential hypothesis testing problem and adapts state of the art techniques. In terms of strengths, the reviewers appreciate the broad problem of detecting text generated by LLMs is certainly timely and important and the theoretical guarantees of the proposed methodology. In terms of weaknesses, the reviewers highlight insufficient motivation and contextualization with the state of the art and several issues regarding the experimentents. In terms of overall recommendation, the reviewers differ significantly and, as a result, I did read the manuscript to better decide about my recommendation. Based on the reviews and my own reading, my recommendation is to reject the paper. The two main reasons are: 

(i) the actual application is not very well motivated, as highlighted by several reviewers, and the rebuttal by the authors did not persuade me otherwise. In that context, the authors do not sufficiently engage with related work on detection of LLM generated text based on, e.g., watermarking nor with related work on sequential hypothesis testing.

(ii) both in the main and the supplementary, it is unclear/difficult to realize to what extent the authors claim the methodology/technique to be a contribution or to be a straightforward adaptation of previous work on sequential hypothesis testing. In that context, one of the main differences with previous work seems to be the assumption on the bound for g_t, however, the authors fail to explain why the assumption from previous work does not hold in their case and instead needs to be related both in their manuscript and the rebuttal. Further, if the authors indeed claim the methodology advances the state of the art in sequential hypothesis testing, then, the framing of the manuscript is misleading.