Short-PHD: Detecting Short LLM-generated Text with Topological Data Analysis After Off-topic Content Insertion
Authors: Dongjun Wei
Minjia Mao
Xiao Fang
Michael Chau
Venue: COLM 2025 (2025)
Keywords: large language model
zero-shot detection
short text
topological data analysis
TL;DR: We present Short-PHD, a zero-shot LLM-generated text detection method tailored for short texts.

Abstract: The malicious usage of large language models (LLMs) has motivated the detection of LLM-generated texts. Previous work in topological data analysis shows that the persistent homology dimension (PHD) of text embeddings can serve as a more robust and promising score than other zero-shot methods. However, effectively detecting short LLM-generated texts remains a challenge. This paper presents Short-PHD, a zero-shot LLM-generated text detection method tailored for short texts. Short-PHD stabilizes the estimation of the previous PHD method for short texts by inserting off-topic content before the given input text and identifies LLM-generated text based on an established detection threshold. Experimental results on both public and generated datasets demonstrate that Short-PHD outperforms existing zero-shot methods in short LLM-generated text detection. The implementation codes of this study are available online.

Published as a conference paper at COLM 2025
Short-PHD: Detecting Short LLM-generated Text with Topo- logical Data Analysis After Off-topic Content Insertion
Dongjun Wei', Minjia Mao², Xiao Fang* Michael Chau
The University of Hong Kong, Hong Kong, China
2 University of Delaware, DE 19716, USA dongjun@connect. hku.. hk, {mjmao, xfang @udel leaideeeeddu edu, mchau@business. hku,
Abstract
The malicious usage of large language models (LLMs) has motivated the de- tection of L.L -egeerredtedd texts. Previous work in topological data analysis shows that the persistent homology dimension (PHD) of text embeddings can serve as more robust and promising score than other zero-shot meth- ods. However, effectively detecting short LLM-generated texts remains a challenge. This paper presents Short-PHD, a zero-shot LLM-generated text detection method tailored for short texts. Short-PHD stabilizes the estimation of the previous PHD method for short texts by inserting off- topic content before the given input text and identifies L.L.-generated text based on an established detection threshold. Experimental results on both public and generated datasets demonstrate that Short-PHD outperforms existing zero-shot methods in short LLM-generated text detection. The implementation codes of this study are available online.
1
Introduction
Table 1: Detection performance (AUC) of PHD and Short-PHD for identifying short LLM- generated texts (50 tokens). Results are averaged from Table
Method
OPT/OPT-im//GPT-NNox
GPT-3.5/GPT--4
PHD (Tulchinskii et al., 2024)
0.632
0.653
0.793
0.830
Short-PHD (Ours)
(Improve 1 25.47%)
(Improve 1 27.11% %)
With the rapid development of Large Language Models (LLMs) (Guo et al., 2025), detecting LLM-generated content has become a crucial issue in controlling malicious content on the internet (Lin et al., 2022; Liu et al., 2024; Fang et al., 2024). In this work, we focus zero-shot methods for detecting LLM-generated texts. A typical zero-shot method leverages an LLM scoring model to compute the conditional probability of each token given previous tokens and analyzes the statistical properties of these probabilities (Ippolito et al., 2020; Su al., 2023; Mitchell al., 2023). For example, DetectGPT (Mitchell et al., 2023) and Fast-DetectGITT (Bao et al., 2024) focus on conditional probability curvature, which evaluates the direction of perplexity change after perturbing random tokens. However, to ensure reliable performance, these methods often require 'white-box' setting in which the generative LLM can serve as the scoring model during detection. For example, DetectGPT achieves detection AUC of only 0.6 when using GPT-J as the scoring model to detect GPT-2-generated text (Mitchell al., 2023).
To overcome the 'white-box' challenge, apart from probability-basel methods, Tulchinskii et al. (2024) propose utilizing the Persistent Homology Dimension (PHD) from topological
"Dongjun Wei and Minjia Mao contribute equally. Corresponding to Minjia Mao. hhttps://githin ctt.uub..oo/////// www/dwwwew/eeeeeeeccotttteeeeeee 9 tHD
1

---

Published as a conference paper at COLM 2025
Textx
Increased Sampling
Regretsion local Expacted regrassion
fogt Call
is human-written t
The American Civil War (April 12, 1861 - May 26, 1865) was civil war the United States between the Union ("the North") and the Confederacy ("the South "),
some ocacs
is LLM-gannrrated di
PHD
log %C
Off-topic Content Insertion (OCI) before Text
Increased Sampling
Regrassion denaity
LLMs often produce text that is well-structured and free from obvious errrrs, sometimes appearing overly polished. The language may be somewhat generic, lacking the nuance, style, personal touch that human writer might include. The American Civil War (April 12, 1861 May 26, 1865) was civil war in the United States between Union ("the North") ") and the Confederacy ("the South") "),
Yes
x human-written
d >>
x LLM-generrted
+
No
Short-PHD
log R1
Figure 1: PHD (Tulchinskii et al., 2024) v.s. Short-PHD. n; is the number of sampled points in the point cloud W. W; is the sampled subset of points in W with size ni, E(W;) is computed by the minimal spanning tree (MST).
data analysis to identify LLM-generated texts. Specifically, texts are first embedded using pretrained LLM such RoBERTa (Liu et al., 2019). Next, the PHD (Kališnik, 2019), a concept of topological data analysis (Chazal & Michel, 2021), is computed for these embeddings. From a topological perspective, PHD measures the connectedness of the embedding, and a higher PHD represents a less connected data structure. LLM-generatee texts are found to exhibit lower PHD values than human-written text due their naturally higher connectedness in the embedding space. PHD outperforms other zero-shot methods, such as DetectGPT, in detecting LLM-generated texts acToss various models and domains, and also demonstrates greater robustness against adversarial attacks (Tulchinskii et al., 2024).
However, similar to other zero-shot methods, the PHD method can only detect text that is sufficiently long, and Tulchinskii et al. (2024) assume LLMs are fluent enough to generate at least 200 tokens, which limits their applicability for detecting short texts. In this study, we first analyze the scalability challenges of the current PHD method. Specifically, based on the increased sampling technique used to estimate PHDs, the PHD method encounters phenomenon named local density peaks, which can make estimation unstable (either too high too low). These local density peaks are particularly problematic for short texts, as they lack sufficient variability to sample out of the peak.
To facilitate the PHD method to overcome the local density peak issue in short texts, we propose hypothesis involving an off-topic content insertion (OCI) technique. Specifically, we hypothesize that inserting off-topic content unrelated to the input text can stabilize and increase the PHD score. The underlying rationale is that by adding extra content, the text becomes longer, helping the PHD estimation avoid local density peaks in increased sampling. Furthermore, since PHD measures the connectedness of embeddings, limiting the inserted content to off-topic (i.e., irrelevant) content ensures that the PHD increases for all inputs because text embeddings are less connected.
Based on OCI, we present Short-PHD, a zero-shot method derived from the original PHD method, specifically tailored to detect short LLM-generated texts. A comparison between Short-PHD and PHD is illustrated in Figure 1. Specifically, given a short human-written text and short LLM-generated text, Short-PHD first inserts the same off-topic content into each text. Then, it applies increased sampling to compute their respective PHD scores. According to the hypothesis on OCI, introducing OCI stabilizes and increases the PHD values of both
2

---

Published as a conference paper at COLM 2025
human-written and LLM-generated texts. To preserve the effectiveness of detection, further hypothesize that the gap between hurttewrittten and LLM-generated PHD values remains the same in expectation after inserting the same off-topic content. Consequently, Short-PHDD can establish a threshold of PHD to identify L.M-generated texts.
We summarize our main contributions follows: 1) We identify the local density problem in estimating PHDs of short texts in the previous PHD method and propose that OCI could alleviate this issue. 2) We propose novel Short-PHD method based on OCI to detect short LlM-ggnerated texts. 3) Experimental results on both public and generated datasets indicate that Short-PHD achieves state-of-the-ai performance in detecting short LLM-generated texts compared to the previous PHD method and other baselines.
Preliminary
The persistent homology dimension (PHD) analyzes the data in high dimensional space about the birth and death of its persistence cycles (Tulchinskii et al., 2024). Empirical evidence shows that LLM-generated texts exhibit lower PHD values than human-written texts. Formally, let W C Rd be d-dimensional topological space, and let W = [wi E R*) be discrete point cloud sampled from W, i.e., W C W C R In our work, we represent text as a point cloud in a high-dimensional space using embeddings from pretrained LLM such as LLaMA (Dubey et al., 2024). Specifically, given a text sequence of и tokens, each token is mapped to d-dimensional embedding vector, forming a set of points in Rd. For example, with d 768, the text forms point cloud in R768 where each point corresponds to a token embedding.
2.1 Mathematics Preliminary
We provide basic concepts of homology groups in Appendix A.1. Informally speaking, the idea of homology groups is to find non-trivial topological features in a topological space. These features include connected components, holes, and cavities that exist in the space. Next, we introduce the mathematical concepts of persistent homology in Appendix A.2. To sum up, different from homology groups that analyze topological space, persistent homology belongs to data analysis that analyzes data points (or a point cloud). First, from the point cloud W, the Vietoris-Rips (VR) complexes VR(W) (Bakhtin et al., 2019) are constructed with a filtrating parameter r. By changing the filtrating parameter r, persistent homology focuses on the birth and death of cycles (fbirth death) in the complex. The persistent homology dimension (PHD) is defined based on the lifetime of these cycles. We refer readers to Appendix A.2 and previous work (Birdal al., 2021) for more details.
2.2 Estimating Persistence Homology Dimension
Following previous work (Tulchinskii et al., 2024), we focus on PHg, which measures the O-persistent homology. The PHg corresponds to the minimal spanning tree (Tulchinskii et al., 2024) and Equation A1 (in Appendix A.2) can be calculated as
(Q
Σ le|
(1)
eEMST(W)
where MST(W) denotes the minimal spanning tree (MST) from point set W, is the edge in the tree, and le] is the length of e. The exact number of PHg is hard to compute even for small point cloud, but it could be approximated through increased sampling. The increased sampling is conducted by sampling different numbers of data points in the point cloud and estimating their MST. Specifically, if points w0 in cloud are independently
distributed, then with probability approaching one, (W) Cn شي 1988). Therefore, in practice, by increasing the number n; of sampled points in subset W; {W1, (wW,WW2.- W2,... .., Ww;,, we can find its minimal spanning tree and compute (W,). Since know that log (W;)) (1 - -)) log n + log C, we can fit linear regression to compute
=+ holds as n + (Steele,
3

---

Published as a conference paper at COLM 2025
d if we fix (usually E [0.5, We write the PHD calculation d PHD(W) to simplify our notation.
Problems in Estimating PHD for Short Texts (Local Density Peak), Although estimating PHD with increased sampling is sample-eficciet, it introduces additional challenges for short texts, their point cloud contains only small number of points. When the number of points is small, there is a higher probability that these points are too close (or distinct) to each other. This could cause log Eg (W;) to become unstable due to the edge lengths of the MST becoming extremely small (or large). We refer to this issue as local density peak (Tulchinskii et al., 2024). Meanwhile, as previously discussed, the value of PHD is based on the slope of the linear regression between log R;; and log (WW). Therefore, an unstable log Eg (W;) caused by local density peaks could affect the accuracy of the PHD estimation process. An example is shown in Figure 1. Specifically, the regression contains three sampled subsets through increased sampling, which contain 10, 20, and 40 points (before logarithm), where log 20 experiences a local density peak. The peak will lead to higher slope and thereby lead to higher PHD value d. In practice, the increased sampling process is conducted multiple times to reduce the influence of local density peaks (Birdal et al., 2021). However, if we only have few points to sample, the local density peak cannot be easily avoided by the randomness in sampling.
3 Method: Short-PHD
In this section, we first introduce a technique named off-topic insertion (OCT) which can stabilize the PHD estimation for short texts. We then present Short-PHD, an OCT-based method for detecting short LLM-generated texts. Finally, we provide empirical evidence to validate the effectiveness of these two techniques.
3.1 OCI: Off-topic Content Insertion
As discussed in Section 2.2, compared to long texts, the PHD method more likely to encounter local density peaks when estimating short texts, leading to unstable results that may be either too high or too low. To address such an issue, we propose a hypothesis using off-topic insertion as follows:
Hypothesis (OCI). Off-topic content insertion (OCI) will slabilize and increase the PHD values of short texts.
The rationale behind this hypothesis is twofold. First, by adding additional content, the text becomes longer, helping the PHD estimation avoid local density peaks and achieve more stable results. Second, since PHD measures the connectedness of the embedding, adding completely off-topic text reduces this connectedness, thereby increasing the PHD for any given input text. For example, the original text is a Wikipedia page on the American Civil War, the off-topic content could be tip for detecting LLM -LM--eeerated texts. In practice, we insert a fixed text sequence before each text sample. We apply OCT in distinguishing short human-written and LLM-ggnnrated texts below.
3.2 Short-PHD: Detecting Short LLM-generated Text
We assume that the PHD distributions of short human-written and LLM-generated texts follow independent normal distributions, i.e., d N(R", (o")²) and dm N N(r" (2"")), where the superscripts It and signify human and machine, respectively. Following Tulchinskii et al. (2024), human-written texts exhibit larger PHDs than LLM-generated texts. Therefore, we can use PHD to detect LLM-generated text by measuring the probability that human-written text has higher PHD than LLM-generated text, i.e., P(d > d Specifically, let Z d d Then, the probability of dh being greater than d is given by P(d" d = P(Z > 0). Based on our assumption, d² and d are independent, then Z follows a normal distribution ~ N(H² - n², (o²)² + (o"")2). Thus, the standardized Z follows standard normal distribution. The probability of correctly classifying text is
4

---

Published as a conference paper at COLM 2025
Algorithm 1 Short-PHD
Input: a text sequence X, an off-topic content set Tx, an pretrained LLM PM, PHD estimator PHD(), and detection threshold e.
2: Initialize a score list S
3: for each off-topic content x, E Tx do
4: X; + t-nocat((x; X)
5: Compute the embedding of X,, W; < (x)(
6: Estimate the PHD of W,, d, + PH:(W;)
7: Record di in S
8: end for
9: Score s mS)
10: ifs > then
return human-written
12: else
return LLM-generated
14: end if
given by
P(Z 0) = "(1² pp" (0 + (om²2
(2)
where d is the cumulative distribution function (CDF) of a standard normal distribution.
Since & is monotonically increasing, a larger mean difference (1² =² and lower variance
((o")2+ + (o"")3) increase the probability of successfully detecting LLM-generated text.
When the text is short, the estimation of PHD becomes unstable, and the variance ((o*)2 (a"))2 increases, making detection ineffective. Therefore, in Short-PHD, we apply OCI to stabilize the estimation process for detection. Similarly to the previous PHD method analysis, we assume that each PHD distribution after OCI follows independent normal distributions: d!) ~ NN*,, (o and 4TT ~ N NIrT,, (2) where denotes OCI. According to Hypothesis 1 on OCI, the variance of PHD decreases (of) < o" and oTIi < o""), while the PHD values for both human-written and LLM-generated texts increase. We approximate this increase as follows μ ~ n" + t and n x n² + em where en and e represent the increases in PHD for human-written and LLM-generated texts, respectively. In Short-PHD, we further make the following assumption on the increase.
Hypothesis 2 (Short-PHD), The gap befween human-writter and LLM LLM--gnnrraed PHD values remains the same in expectation after inserting the same off-topic content.
This implies that n² nr ~ n/ j4"", which means the PHD scores of human-written and LLM-generated texts increase by the same value ex ~ e" Therefore, in Short-PHD, when the gap between human-written and LLM-generated texts remains and the variances are decreased, we have
(3) meaning that L.M-ggenrrted texts can be detected with higher probability using Short- PHD compared to PHD. We present Short-PHD in Figure 1 and Algorithm In practice, Short-PHD employs OCI multiple times with different content and computes the average.
P(a" d < P(d) > d'"),
3.3 Empirical Validation
We conduct an empirical validation to test the proposed hypotheses on OCI and Short-PHD using the same dataset from the original PHD paper (Tulchinskii al., 2024). Specifically, we choose two subsets from Tulchinskii et al. (2024): Reddit WritingPromptss (WP) stories (Fan et al., 2018) and Wikipedia pages (Guo et al., 2020), both of which contain GPT-3.5 --T-3.5-generrted and human-written texts. We leverage LLaMA-3-SB (Dubey et al., 2024) to compute the embedding LLaMA(X) text sequence X and calculate the PHD detection score using increased sampling. We then apply OCI before the input text and calculate the PHD
5

---

Published as a conference paper at COLM 2025
Table 2: Average number of tokens in each dataset and statistics of detection score distribu- tions in each dataset with PHD and Short-PHD. Bolded standard deviation (std) implies large std reduction by Short-PHD.
Dataset
WP human
WP GPT-3.5
Wikipedia human
Wikipedia GPT-3.5
Token number
153
276
461
165
PHD mean
10.17
8.84
10.70
8.93
PHD std.
1.49
0.79
1.01
1.54
Short-PHD mean
10.26
9.23
10.81
9.23
Short-PHD std
0.79
0.67
0.85
1.00
WP-GPP-3.5
Wiki-GPT-3.5 (PHD)
(PHD)
WP-Human
WixHHuman (PHD)
WP-GPT-3.5
Wiki-GPT-3.5
(Short-PHID
(Short-PHi)
WP-Human (SDD
Wiki-Human
(Short PHD)
100
11
12
13
14
15
10
12
13
14
15
Figure 2: Boxplots of score distributions on human-written and GPT-3.5 --r-e.--generated Wikipedia and WritingPromppe (WP) data.
score for the modified text. Specifically, we choose off-topic content related to tips for detecting LLM-generated texts, which is generally unrelated to the text being detected. The off-topic content is shown in Figure 1. We provide the average number tokens each subset and the statistics of PHD scores in Table 2. Additionally, we present the distributions
of PHD scores using boxplots in Figure 2.
Validation of Hypothesis (OCI). From Table 2, we first observe that the average token number of human-written WP and GPT-3..-geenratee
Wikipedia is significantly smaller than that of human-written Wikipedia and -PT-3.5-generated WP. Meanwhile, we also find that subsets with shorter texts have larger standard deviation than those with longer texts. Similarly, as depicted in Figure 2, demonstrated that human-written WP and GPT-3.5- generated Wikipedia have wider (blue) boxes compared to other blue boxes, indicating larger variances. Therefore, we can validate that the PHD estimations are less stable for both short human-written and short LLM-generated texts than for long texts.
Next, as shown in Table 2, compared to the standard deviation PHD, applying OCI significantly reduces the standard deviation of PHD values for both short human-writter WP and GPT-3.5-generated Wikipedia. The same results can be concluded from Figure 2, where the yellow boxes (Short-PHD) of human-written WP and GPT-3.5-generater Wikipedia are narrower than the blue boxes of these two subsets. In summary, we conclude that OCT stabilizes the estimation of PHD for short texts. Second, we observe that the mean PHD values for all subsets simultaneously increase. Therefore, we also conclude that applying OCI increases the PHD values for both short human-written and LLM-generated texts.
Validation of Hypothesis 2 (Short-PHD). As depicted in Figure 2, the gap between human- written and LLM--enerated text after applying OCI is largely maintained compared to the PHD method, without dropping significantly. For example, in the WP dataset, the gap for PHD is 1.33 (10.17 8.84), while the gap for Short-PHD is 1 1.03 (10.26 9.23). Similarly, in the Wikipedia dataset, the gap for PHD is 1.77 (10.70 - 8.93), whereas for Short-PHD, it 1.58 (10.81 - 9.23). Therefore, the Short-PHD method could establish detection threshold like the PHD method to identify LLM-generated texts. Furthermore, by plugging in the mean and standard deviation values, we can explicitly compute the value of the CDF. For the PHD method, the probabilities of successful detection are approximately Φ 10.17 8,84
V1.49 V1.49- (0.792 (0.79) 0.78 and (0.96) 0.83 for the WP and Wikipedia datasets, respectively. After applying Short-PHD, the probabilities increase to (00998) z 0.84 and

---

Published as a conference paper at COLM 2025
((1200) z 0.88 for the WP and Wikipedia datasets, respectively. This indicates that the detection performance of Short-PHD is improved. However, note that a PHD distribution does not strictly follow normal distribution. These statistical results should only be considered an approximation.
4 Experiments
In this section, we evaluate our Short-PHD method using both public and generated datasets. We benchmark our method against various zero-shot and supervised detection approaches. Additionally, we conduct multiple sensitivity analyses for Short-PHD and evaluate its robustness against several detection attacks.
4.1 Experimental Setup
Public Dataset Setting. We first evaluate the Short-PHD method the original dataset used in the PHD method (Tulchinskii et al., 2024), referred to as GPTID. The dataset consists of two domains of human-written text: Wiki40b (Guo al., 2020) from Wikipedia and WritingProoppe from Reddit's stories subsector (Fan et al., 2018). The paper claims another StackExchange subset, but it is not publicly available. Text generation is performed using GPT-2 XL (Radford et al., 2019), OPT-13B (Zhang et al., 2022), and GPT-3.5. Detailed statistics are summarized in Table A1.
Generated Dataset Setting. We then construct a more comprehensive dataset in two addi- tional domains using more diverse and advanced LLMs to further evaluate the performance of our method. Specifically, we select Wikihow (Koupaee & Wang, 2018) for articles and quizzes and Eli5 (Fan et al., 2019) for question answering, We employ OPT-13B (Zhang et al., 2022), OPT-imm--1.3B (lyer et al., 2022), GPT-NeoX (Black et al., 2022), GPT-335-turbo, and GPT-4o for text generation. For each domain, we randomly sample 500 human-written texts and prompt each LLM to generate text. Detailed generation settings provided in Appendix B.1.
Baselines. For zero-shot methods, mainly compare Short-PHD method with other state-of-the-art approaches, including PHD (Tulchinskii et al., 2024), DetectGPT (Mitchell et al., 2023), and DetectLLM (Su et al., 2023). Additionally, we evaluate Short-PHD against methods using heuristic scores, including Perplexity (mean log probabilities) and LogRank (average log of ranks). For supervised methods, we incorporate a fine-tuned RoBERTa model trained on GPT-2 outputs from OpenAl (Solaiman et al., 2019). We also implemented more recent baselines, including Fast-DetectGPT (Bao et al., 2024) and DNA-GPT (Yang et al., 2024) in Appendix B.
Implementation Details. We truncate human-written and LLM-generated texts to 50, 100, and 150 tokens to evaluate detection performance for short texts. For the Short-PHD method, we perform OCI 12 times using different off-topic content and utilize LLaMA-3-8B (Dubey et al., 2024) to compute the embedding of each insertion (i.e., the 5th line in Algorithm 1). For a fair comparison, we also leverage LLaMA-3-8b as the scoring model for all zero-shot baselines. The RoBERTa model is directly taken from OpenAl's public version without further modification. More implementation details are provided in Appendix B.1.
4.2
Main Results
We first conduct an empirical evaluation of our method compared to PHD using the public GPTID dataset employed by PHD (without truncation) terms of AUC and TPR05%FPR as shown in Table 3. We observe that Short-PHD outperforms PHD across all four subsets in terms of both metrics. Next, we demonstrate the AUC scores on the generated dataset in Table 4 for Short-PHD and baselines. As reported, our Short-PHD method outperforms other baselines in almost all datasets across five LLMs and two domains, corroborating its effectiveness. Specifically, we observe that detecting short LLM-generated text is generally a challenging task for existing zero-shot methods. For example, DetectGPT fails to detect LLM-generated ELI5 answers, achieving average AUC of only 0.24. This suggests
7

---

Published as a conference paper at COLM 2025
Table 3: Results on the GPTID dataset. The table reports the AUC and TPR#5%FPR (T@F) of PHD and Short-PHD. For results of other baselines, see Tulchinskii et al. (2024).
Wikipedia OPT
WritingPromppe
GPT-2
GPT-3.5
GPT-3.5
Method
AUC
T@F
AUC
T@F
AUC
T@F
AUC
T@F
PHD
0.825
0.388
0.821
0.423
0.847
0.549
0.808
0.088
Short-PHD
0.860
0.445
0.854
0.446
0.891
0.555
0.845
0.329
Improve(t)
4.242
14.691
4.019
5.437
5.195
1.093
4.579
273.864
Table 4: Results on the Generated Dataset. The table reports AUC of baselines and our
method on texts (50 tokens). The improvement is computed between PHD and Short-PHD.
Wikihow
Eli5
Method
OPT OPT-iml
NeoX GPT-3.5 GPT-4o OPT OPT-iml
NeoX GPT-3.5 GPT-4o
RoBERTa
0.66
0.77
0.66
0.65
0.66
0.75
0.78
0.77
0.80
0.80
Perplexity
0.47
0.44
0.47
0.62
0.63
0.46
0.37
0.46
0.70
0.73
LogRank
0.46
0.43
0.47
0.58
0.59
0.42
0.33
0.42
0.62
0.64
DetectGPT
0.56
0.53
0.56
0.62
0.59
0.22
0.16
0.21
0.27
0.27
DetectLLM
0.44
0.43
0.41
0.45
0.47
0.37
0.30
0.34
0.36
0.36
PHD
0.62
0.60
0.66
0.66
0.63
0.64
0.65
0.62
0.66
0.66
Short-PHD
0.78
0.73
0.77
0.80
0.79
0.84
0.82
0.82
0.87
0.86
Improve(1)
25.81
21.67
16.67
21.21
25.40
31.25
26.15
32.26
31.82
30.30
that probability alone is not a reliable metric for detecting short texts. In contrast, PHD (Tulchinskii et al., 2024) attains average AUC of approximately 0.65. Similarly, our method achieves an average AUC score of over 0.8, surpassing even the supervised RoBERTa detector. We thus conclude that PHD captures more meaningful information for short text detection through text embeddings. Furthermore, we present box plots of PHD distributions computed by PHD and Short-PHD in Figure 3 for texts generated by GPT-4o. The results for all LLMs are provided in Figure A1 in the appendix. From these figures, we observe that OCI used by Short-PHD increases PHD's value and stabilizes PHD estimations by reducing variance. Meanwhile, the gap between human-written and LLM-generated texts remains largely unchanged, making Short-PHD effective for detection. As result, Table shows that our method outperforms PHD by more than 16% across all subsets.
Wikihow GPT-4e
Elis GPT (PHD]
[PHD]
WildhowHHuman [PHD]
Human (PHDI
Wikihow-GPT-4o (Short-PHDi
BIS--GPT-0 (Short-PHD)
Wikhhw Human (Short PHD]
Ell5 Human (Short PHD]
12
Figure Box plots of PHD and Short-PHD distributions for texts generated by GPT-4o.
4.3 Sensitivity Analysis
We then conduct a series of sensitivity analyses regarding text length, OCI, white-box detection, and detection attacks to further evaluate Short-PHD.
Sensitivity Analysis on Text Length. We first evaluate the detection performance on texts with fixed length of 50, 100, and 150 tokens in Figures 4a and 4b. As depicted, for texts with a fixed length of 50 tokens, Short-PHD significantly outperforms PHD in terms of AUC. When the text becomes longer, at 100 and 150 tokens, the gap continues to narrow,
8

---

Published as a conference paper at COLM 2025
1.00
1.00
0.95
0.95
0.90
0.90
0.85
0.85
0.80
0.75
0.75
0.70
0.70
0.65
PHD
0.65
PHD
0.60
Short-PHD
0.60
Short-PHD
50
100
150
50
100
150
(a) AUC scores w.r.t text length in Wikihow domain.
(b) AUC main.
text length in Eli5 do-
0.90
RoBERTa
PPL
0.85
Wikihow
LogRank
DetectLLM
DetectGPT
0.80
PHD
Short-PHD
RoBERTa
PPL
0.70
Eli5
LogRank
DetectLLM M
Wikihow
DetectGPT
0.65
PHD
Eli5
Short-PHD
0.60
6
10
11
0.2
0.4
0.6
1.0
(c) AUC scores w.r.t. the number of off-topic content pieces.
(d) AUC scores in white-box (LLaMA) detec-
tion.
RoBERTa
RoBERTa
PPL
PPL
Eli5 Wikihow
LogRank
Wikihow
LogRank
DetectL
DetectL
Get
DetectGPT
PHD
PHD
Short-PHD
Short-PHD
RoBERTa
RoBERTa
PPL
PPL
LogRank
Eli5
LogRank
DetectLLM
DetectLLM MM
DetectGPT
DetectGPT
PHD
PHD
Short-PHi
Short-PHD
0.0
0.2
0.6
0.8
0.4
0.8
1.0
(e) AUC scores under decoherence attack.
(f) AUC scores under paraphrasing attack.
Figure 4: Sensitivity Analysis.
but Short-PHD still surpasses PHD. The reason is that the text length increases, the PHD method encounters fewer local density peak problems. To sum up, our proposed Short-PHD highly effective for short texts of around 50 tokens and remains superior the original PHD method for longer texts.
Sensitivity Analysis on the Number of Off-topic Content Pieces. We adjust the number of content pieces in the off-topic content set in Algorithm 1 and analyze its impact. As depicted in Figure 4c, by simply inserting a piece of off-topic content, the average AUC score reaches around 0.75. The performance of Short-PHD stabilizes when at least content pieces are provided. This demonstrates that by inserting multiple off-topic content pieces, the performance of Short-PHD becomes more stable and effective.

---

Published as a conference paper at COLM 2025
Table 5: Sensitivity Analysis on the Topics of the Inserted Content.
Wikihow
Eli5
OPT OPT-iml NeoX GPT-3.5 GPT-4o OPT OPT-iml NeoX GPT-3.5 GPT--oo
On-topic 0.66
0.67
0.73
0.60
0.60
0.60
0.60
0.60
0.56
0.55
Off-topic 0.78
0.73
0.77
0.80
0.79
0.84
0.82
0.82
0.87
0.86
Sensitivity Analysis on White-box Detection. We then evaluate the performance of each method in a white-box setting. Specifically, since we use LLaMA as the scoring model for all cases, generate WikiHow and ELI5 articles using LLaMA with 50 tokens and then evaluate the detection performance. As shown in Figure 4d, probability-based methods such PPL and DetectGPT achieve the best performance. This is because the probability calculation is more accurate in a white-box scenario. However, the Short-PHD method still achieves AUC score above 0.9, outperforming PHD and baselines like RoBERTa and DetectLLM, demonstrating its effectiveness in the white-box setting.
Sensitivity Analysis on the Topics of the Inserted Content. We then evaluate how the relevance of the inserted topic affects the detection performance of the proposed method, considering both off-topic and on-topic content. For off-topic content, we follow the same setup as in the main experiment. For on-topic content, we apply simple heuristic approach by inserting the same content as the detected text. This ensures that the inserted content is on the same topic as the detected text. The results shown in Table 5 demonstrate that inserting off-topic content yields higher detection scores across all generators and domains compared to inserting the same detected text. It provides some initial evidence that simply increasing the length of the detected text does not improve detection.
Robustness against Detection Attacks. Lastly, we evaluate the robustness of each method against the decoherence attack (Bao et al., 2024) and the paraphrasing attack (Krishna et al., 2023) (see Appendix B.2 for more details) in Figure 4e and Figure 4f. As shown in both figures, we find Short-PHD is robust against these attacks, which is consistent with previous work, which shows PHD is more robust compared to other baselines (Tulchinskii et al., 2024). Similar to Tulchinskii et al. (2024), we observe that the detection AUCs are almost the same before and after attacks, demonstrating the robustness of Short-PHHD.
Conclusions and Future Work
In this study, we propose Short-PHD, novel zero-shot LLM-generated text detection method tailored for short texts. Experimental results demonstrate that Short-PHD achieves superior detection performance for short texts compared to PHD and other baselines. Meanwhile, Short-PHD is robust against various detection attacks.
We acknowledge several limitations of our study. First, the original PHD method is em- ployed in different languages, but only test Short-PHD in English. Second, utilize only persistent homology in 0 dimensions. Whether high-dimensional topological structures of embeddings can be used for detection remains to be explored. Third, note that PHD can be viewed as a feature of the text. Future work can take PHD into account when developing training-based methods.
Related Work
Existing methods for detecting L.M-gennrated texts can be divided into two categories: supervised detectors and zero-shot methods (Wu et al., 2023). In this work, we focus zero-shot detection methods, which require no training and exhibit better generalizability in identifying LLM-generated text across various models and domains (Bao et al., 2024). We refer readers to Appendix C for a detailed review of each stream of literature.
10

---

Published as a conference paper at COLM 2025
References
Aaron Adcock, Erik Carlsson, and Gunnar Carlsson. The ring of algebraic functions persistence bar codes. Homology, Homotopy and Applications, 18(1)3381-02, 2016.
Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Marc' Aurelio Ranzato, and Arthur Szlam. Real or fake? learning to discriminate machine from human generated text. arv preprint ar Xiv::996.00357/ 2019.
Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. Fast-detectgpt: Efficient zero-shot detection of machine-generated text via conditional probability curva- ture. In The Twelffh International Conference on Learning Representations, 2024.
Ulrich Bauer. Ripser: efficient computation of vietoris-rips persistence barcodes. Journal of Applied and Computational Topology, 5(3):391-423, 2021.
Paras Bhatt and Anthony Rios. Detecting bot-generated text by characterizing linguistic accommodation in human-bot interactions. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3235-3247, 2021.
Tolga Birdal, Aaron Lou, Leonidas J Guibas, and Umut Simsekli. Intrinsic dimension, persistent homology and generalization in neural networks. Advances Neural Information Processing Systems, 14:6776-67888 2021.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-20b: An open- source autoregressive language model. arXiv preprint ar arXin:2204.0677 000:2220..0 2022.
Zixuan Cang, Lin Mu, Kedi Wu, Kristopher Opron, Kelin Xia, and Guo-Wei Wei. A topolog- ical approach for protein classification. Computational and Mathematical Biophysics, 3(1), 2015.
Frédéric Chazal and Bertrand Michel, An introduction to topological data analysis: funda- mental and practical aspects for data scientists. Frontiers in artificial intelligence, 4:667963, 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv preprint ar Xix:22077.21783 2024.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889-898, 2018.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Eli5: Long form question answering. ar Xiv preprint ar axXiv:1907.09190, 2019.
Xiao Fang, Shangkun Che, Minjia Mao, Hongzhe Zhang, Ming Zhao, and Xiaohang Zhao. Bias of ai-generated content: an examination of news produced by large language models. Scientific Reports, 14(1):2224, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, ar arXiv preprint ar xxxi:25011.1244 2025.
Mandy Guo, Zihang Dai, Denny Vrandečić, and Rami Al-Rfou. Wiki-40b: Multilingual
language model dataset. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 2440-2452. 2020.
Daphne Ippolito, Daniel Duckworth, and Douglas Eck. Automatic detection of generated text is easiest when humans are fooled. In Proceedings of the 58th Amnual Meeting of the Association for Computational Linguistics, pp. 1808-1822, 2020.
11

---

Published as a conference paper at COLM 2025
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint ar Xit:2212.12011 2022.
Sara Kališnik. Tropical coordinates on the space of persistence barcodes. Foundations of Computational Mathematics, 19(1):101-129, 2019.
Mahnaz Koupaee and William Yang Wang. Wikihow: A large scale text summarization dataset. Xiv preprint arXiv:18100 0.00099 2018.
Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit lyyer. Para- phrasing evades detectors of ai-generated text, but retrieval is an effective defense. Ad- vances in Neural Information Processing Systems, 36:27469-2277 27500,, 2023.
Yafu Li, Qintong Li, Leyang Cui, Wei Bi, Zhilin Wang, Longyue Wang, Linyi Yang, Shuming Shi, and Yue Zhang. Mage: Machine-generated text detection in the wild. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 36-53, 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume Long Papers), pp. 3214-3252, 2022.
Aiwei Liu, Qiang Sheng, and Xuming Hu. Preventing and detecting misinformation generated by large language models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 3001-3004, 2024.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv e-prints, Pp. arXiv-1907, 2019.
2019.
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. Detectgpt: Zero-shot machine-gen -echhie-ggeertedd text detection using probability curvature. In International Conference on Machine Learning, pp. 21950-249996 PMLR, 2023.
Chi Seng Pun, Si Xian Lee, and Kelin Xia. Persistent-homolle Persistent-homollgg--beeed machine learning: survey and a comparative study. Artificial Intelligence Review, 5567))5169-55213, 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8);9, 2019.
Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can ai-generated text be reliably detected? arXiv preprint 1xx::2303331156 2023.
Umut Simsekli, Ozan Sener, George Deligianmidis, and Murat A Erdogdu. Hausdorff dimen- sion, heavy tails, and generalization in neural networks. Advances in Neural Information Processing Systems, 33:5138 5151, 2020.
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert- Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. Release strategies and the social impacts language models. arXio preprint arXiv:1908.09200 2019.
J Michael Steele. Growth rates of euclidean minimal spanning trees with power weighted edges. The Annals of Probability, 16(4):1767-1787, 1988.
Jinyan Su, Terry Yue Zhuo, Di Wang, and Preslav Nakov. Detectllm: Leveraging log
rank information for zero-shot detection of machine-generated text. arXiv preprint arXio:2306.05540, 2023.
Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva, Daniil Chemnavskii, Sergey Nikolenko, Evgeny Burnaev, Serguei Barannikov, and Irina Piontkovskaya. Intrinsic dimension estimation for robust detection of ai-generated texts. Advances Neural Information Processing Systems, 36, 2024.
12

---

Published as a conference paper at COLM 2025
Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Derek F Wong, and Lidia S Chao. A survey on llm-gemerrted text detection: Necessity, methods, and future directions. arXiv preprint ar #rii:2310.14724, 2023.
Xianjun Yang, Wei Cheng, Yue Wu, Linda Ruth Petzold, William Yang Wang, and Haifeng Chen. Dna-gpt: Divergent n-gram analysis for training-free detection of gpt-generated text. In ICLR, 2024.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roes- ner, and Yejin Choi. Defending against neural fake news. Advances in neural information processing systems,, 32, 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. ar Xiv preprint ar Xix::2205.00166, 2022.
13

---

Published as a conference paper at COLM 2025
Appendix A
Mathematics Preliminary
A.1 Homology Group
Informally speaking, the idea of homology groups is to find non-trivial topological features in a topological space. These features include connected components, holes, and cavities that exist in the space.
Concretely, if W is a simplicial complex, for k = ..., proper boundary operator (group homomorphism) can be defined as dx CxV)) CR-(WW), Vk, where Cx(W) is the free abelian group generated by oriented k-simplices in W. Next, we can form a chain of
C++1 (W) 3-1 C+(M) - (x-1(W) .... Co(M), with de o dR-1 = O. In this work, we consider finite W because the number of tokens LLMs can embed has maximum limit in practice (e.g., 768 for LLaMA).
group homomorphism:
Concretely, the k-th homology group HxW is defined as the quotient group (WV = ((W)/B/()) where (W) ker(a;) is the cycle group and By (W) im + is the boundary group. The rank of the k-th homology group is topological invariance, known as the Betti number Py rank H2). Informally and geometrically speaking, the homology group characterizes the 'closed but non-boundary' structure.
A.2 Persistent Homology Dimension
Persistent Homology. The idea of persistent homology is to construct complexes from a point cloud. Denote (x(x) as the ball with center x and radius 8 in Rd. A widely-used Vietoris-Rips (VR) complex is defined as follows (Bakhtin et al., 2019).
Definition 1 (VR complex). Given a set of points W and a radius r, subset Q C W is constructed as an element of the VR complex using the intersection of pairwise balls: W, = (W) = (0 c W Vx, x' E (x) n (x())) + 0}.
The persistent homology defined based on filtrating certain complexes constructed from set of points. Note that the filtrating parameter of VR complexes is the radius r. Each persistent homology PH₃, k 0, is a detailed analysis of homology in dimension k. Persistent homology focuses the birth and death of cycles along with the filtrating parameter. Concretely, the k-persistent homology PH₃ keeps track of the lifespan of each k-persistence cycle у:- The lifespan is pair of birth and death thresholds for each cycle (Ibinty ddath)) k-persistence cycle Yi, denote (1pirth (Tr),, 'eath (7:)) as its own lifespan features. Specifically, for VR complexes, PH₂ records the lifespan of k-persistence cycle as the radius increases.
For example, geometrically speaking, a O-persistence cycle tracks the evolution of connected components in the topological space. The birth of each cycle is always 0 because each point starts as its own connected component unless there are identically the same points. By increasing the filtration parameter (radius), the death of the cycle happens when two connected components merge. The PHo is related to many geometrical meanings such as the maximal number of disjoint balls (Simsekli et al., 2020; Birdal et al., 2021) and minimal spanning tree (Bauer, 2021; Tulchinskii al., 2024), which makes it easy to calculate.
Persistent Homology Dimension. There are many approaches to extract features from the birth and death thresholds (Pun et al., 2022). For example, previous work measures statistics of birth and death thresholds as features, including the first/secom hand/Hi /third largest Betti numbers (Cang et al., 2015), min-plus and max-plus coordinates (Adcock et al., 2016; Kališnik, 2019). In this work, we follow previous work (Birdal et al., 2021) and define the a-weighted k-th homology lifetime sum as follows
(WW)=
Σ
[(T)))
(A1)
(VR(WW
14

---

Published as a conference paper at COLM 2025
where I(>:) death (7i) fbirth (7:) is the lifetime of a cycle y in the k-th persistent homol- ogy of the VR complex PH, (VR(W)) Next, the k-PHD is defined as
dimfu inf(d]
C;3C 0, A finite W' C W],
(A2)
where W is a finite subset of W. Since W is finite in this work, W" can be any subset of W. By estimating the lifetime of cycles, PHD is suitable estimator that combines both local and global features of the dataset.
Appendix B Experiments
Table A1: Statistics of the GPTID dataset.
Domain
LLM
Number of Average human pairs
Average LLM tokens
tokens
Wikipedia
GP'T-2
2557
465
272
Wikipedia
OPT
2475
463
260
Wikipedia
GPT-3.5
2317
461
165
WritingPrompts
GPT-3355
2128
153
276
Table A2: The off-topic content set used in Algorithm 1.
No. Off-topic Content
1
Identifying text generated by large language models (LLMs) like GPT can be challenging, but there are some strategies and telitale signs that can help. Here some suggestions: Repetition and Redundancy and Overly Polished Generic Language.
2
LLM--gnerated text may contain repetitive phrases or ideas. Although models are improving, they might still circle buck to the same puints or restate them in slightly different ways.
3 LLMs often produce text that is well-structured and free from obvious errors, sometimes appearing overly polished. The language may also be somewhat generic, lacking the nuance, style, or personal touch that human writer might include.
The tone style L-M-generated might fluctuate unexpectedly. For instance, paragraph may shift from formal to informal language or from being highly detailed to overly vague.
5
While LLMs are generally accurate, they can make mistakes that are unusual for humans, such as misinterpreting context, creating fictional facts, or making nonsensical statements.
6
L.MSs can generate text that seems insightful but lacks true depth. The text may superficially ad a topic without providing meaningful analysis understanding.
The phrasing in --generated text might sound slightly off, either too formal too casual, or use uncommon word combinations that human writer might not choose.
I.I Ms might draw connections between ideas that don't Ion'tlogicallyyy logically follow from each other. This result in text that feels disjointed or where the flow of argument is nut clear.
9
Some LLMs may overuse certain phrases sentence structures, especially ones that were prominent in their training data.
10
Use Al detection tools designed to identify LLM-generated content. These tools analyze patterss in the text to predict whether was written by human or machine.
11 LLMs may provide inconsistent answers when asked the same question in different ways. A human writer is more likely maintain consistent opinions facts across different cuntexts.
12 Using combination of these strategies can help you more reliably identify LLM-generated text, although worth noting that as Al models improve, the distinction between human and machine--geeeratee text is becoonnng increasingly subtle.
B.1 Experimental Details
Datasets. The summary statistics for the GPTID dataset are presented in Table A1. Specifi- cally, we select Wikihow (Koupaee & Wang, 2018) for articles and quizzes and Eli5 (Fan et al., 2019) for question answering. We employ OPT-13B (Zhang et al., 2022), OPT-iml-1.3B (Iyer et al., 2022), GPT-NeoX (Black et al., 2022), GPT--.5-turbo, and GPT-4o for text generation.
15

---

Published as a conference paper at COLM 2025
Table A3: Examples of Prompts Used in the Generated Dataset.
Domain
Prompt
Wikihow
Please, generate short wikihow article in 100 words or less from title 'How to Get Rid of Adcholoes' and headline 'Downloud and install AdwCleaner, Download and install Malwarebyte Antimalware, Download and install Spyhot Free Edition. Reboot your computer Safe Mode, Run AdwCleaner, Ensure that all the results are checked, ..., Reboot your Mac!
Wikihow
Please, generate a short wikihow article in 100 words or less from title 'How to Make No Sew Curtains2" and headline 'Measure your windows decide how large your want each curtain panel to be, Choose your fabric, then wash, dry, and iron it, if necessary. to get rid of any shrinking and wrinkles... Hang your curtains."
Wikihow
Please, generate a short wikihow article in 100 words or less from title 'How to Account for Cost of Goods Sold' and headline 'Determine the beginning inventory value. This should always be ending inventory value from the previous reporting period. Add the value of all inventory purchases, .... Record the journal entry you are using perpetual inventory method.'
E1S
will ask you question. For this question, provide me short answer in 100 words or less in formal academic and scientific writing voice. Question: How Precisely Are Satellites put orbit? the meter?
Els5
will ask you question. For this question, provide me short answer 100 words or less expert confident voice. Question: Why the ocean sumetimes really dark, like in the north Atlantic yet in some places turquoise colour like the Caribbean South Pacific?
Eli5
will ask you question. For this question, pruvide me short answer in 100 wurds less child's voice. The child has limited vocabulary and limited knowledge so they may make grammatical as well as factual mistakes. Question: Why did the Soviets accept the Finnish peace agreement instead of installing the puppet government they had prepared, despite having great cost) crushed the Finnish army?
For each domain, we randomly sample 500 generate text. The examples of prompts used in Wikiohw and Eli5 to generate the dataset are shown in Table A3. We maintain the default configuration, such as temperature, in each LLM during generation.
texts and prompt each LLM to
Implementation Details. All experiments were conducted on a single NVIDIA H100 GPU. The off-topic content set used in Algorithm 1 is listed in Table A2. Following (Tulchinskii et al., 2024), we set the a to 1.0 to calculate the PHD dimension. For supervised methods, we use the public version of the OpenAl detector from Hugging Face. For zero-shot methods, we use the public implementations of Perplexity, LogRank, DetectGPT, and DetectLLM. For dataset generation, we apply Transformers' and take the weights of OPT-13B, OPT-1.3B, and GPT-NeoX from Hugging Face.5 For GPT-3.5-turbe and GPT-4, we utilize the API from OPe
Attacks. The decoherence attack randomly flips two adjacent tokens when a sentence exceeds 20 words (Bao et al., 2024). The paraphrasing attack employs DIPPER (Krishna et al., 2023) to rewrite each text. We set the lexical and order parameters as 60 and O = 60.
B.2 Additional Experimental Results
Main Results. We present the box plots of PHD distributions for text generated by each LLM in each domain in Figure A1. Specifically, we show the PHD distributions computed by both PHD and Short-PHD.
Additionally, we implemented two more recent baselines, which are Fast-DetectGPT (Bao et al., 2024) and DNA-GPT (Yang et al., 2024), in our comparison. We conducted experiments on black-box detection in the tables below for texts with 50 tokens each. The setting is kept the same as in the previous experiment in Table 4. As shown in Table A5, Short-PHD consistently outperforms or matches the performance of the recent baselines. Specifically,
hhttps://huggiiggaace
hhttos////opi.orr//////tttttttts $nttps://hugiingffccee Phttps://platformmopeee
16

---

Published as a conference paper at COLM 2025
Wikthew-De
(PHD)
Wkthow-OPT (Short-PHD)
EI15-PPT (Short PHD)
Wikhoo
(Short-PHD)
(Short-PH))
10
16
18
10
14
16
20
Wikhow-OFF.iml
IIS-OPt
Wikihow-Human (PHD)
Wikhow-OFFimm (HD)
OPTimi
Wikmhow-HHman
(Short-PHD)
(Short PHD)
2
10
16
18
10
12
14
16
18
20
Wikxhow-NeoX
Ell NeoX-iml (PHD)
(PHD)
Wikhhw-Human (PHD)
EIS Human (PHD)
Wikhow NeoX iml
FII--Nex (Short PHD)
(Short-PHD)
Eli--human (Short.PHi)
(Short-PHD)
2
10
12
14
16
18
20
10
12
14
16
20
Wikhow-GFFa335
CII--GPT-3.5
(PHD)
(PHD)
Wikhho-Human (PHD)
Eli--tumen (PHD)
Wichow-GF...5 (Short
BI5--GFT-3.5 (Short PHD]
Wilhho-Humma
ElI--huaan (Short
(Short
10
16
18
10
14
16
20
Wikihow-GP1-do
Eli GP1-4 (PHD)
(PHD)
Wicchow-Human
EIS-Human
(PHD)
(PHD)
Wikhhoe-PP--0
B15--PPT-O (Short PHD)
IShort PHO)
Klldhow-HHman ISh
EIG- EIG-Human (Short-PHID)
Figure A1: Box plots of PHD distributions by PHD and Short-PHI. Table A4: Sensitivity Analysis.
Sensitivity Analysis the Number of Off- (b) Sensitivity Analysis on White-box Detec-
topic Content Pieces.
tion.
Number
Wikihow
Eli5
Method
Wikihow
Eli5
1 0.745
0.785
Roberta
0.730
0.812
2
0.753
0.812
Perplexity
0.924
0.959
4
0.759
0.827
LogRank
0.897
0.921
6
0.763
0.840
DetectGPT
0.911
0.512
8
0.768
0.844
DetectLLM
0.478
0.431
10
0.771
0.847
PHD
0.602
0.684
12 (default)
0.773
0.849
Short-PHD
0.760
0.880
our method surpasses DNA-GPT significantly across different models and domains. Short- PHD also achieves an overall improvement in performance compared to Fast-DetectGPT for short text detection. These findings further highlight the effectiveness short-PHD.
Sensitivity Analysis. We present the detailed results when changing the number of pieces of off-topic content in Table A4a. Meanwhile, we show the detailed white-box detection performance (AUC) in Table A4b. We find that for white-box detection, simple perplexity
17

---

Published as a conference paper at COLM 2025
Table A5: Results on the Generated Dataset. The table reports AUC of baselines and our method on texts (50 tokens). For space concerns, Fast-D refers to Fast-DetectGPT.
Wikihow
Eli5
Method
OPT OPT-iml
NeoX GPT-3.5 GPT-4o OPT OPT-iml NeoX GPT-3.5 GPTT-4o
Fast-D
0.73
0.74
0.74
0.77
0.76
0.82
0.81
0.80
0.83
0.84
DNA-GPT
0.57
0.55
0.58
0.64
0.64
0.55
0.50
0.55
0.73
0.74
Short-PHD
0.78
0.73
0.77
0.80
0.79
0.84
0.82
0.82
0.87
0.86
or log rank metric can achieve a satisfactory result because the probability is computed precisely.
Appendix Related Work
Existing methods for detecting LLM-generated text can be categorized into supervised and zero-shot methods (Wu et al., 2023). Early supervised methods were trained on extracted linguistic or semantic features (Bhatt & Rios, 2021; Sadasivan et al., 2023; Li et al., 2024). Current supervised methods focus training classifiers to distinguish the neural represen- tations of LLMM-generaed text from human-writter text (Liu et al., 2019; Zellers et al., 2019).
Research has shown that supervised methods easily overfit to the extracted features or texts in the training dataset (Bakhtin et al., 2019), making them prone to overfitting to the training distribution of specific LLMs and domains. Therefore, we focus on zero-shot detec- tion methods, which require no training and exhibit better generalizability in identifying LLM-generated text across various models and domains (Bao et al., 2024).
Zero-shot methods focus on extracting certain features from large language models and set threshold differentiate between LLM-generate and human-written text. For example, one early detection method established threshold based on the average log probability of text, which serves as strong baseline for detecting LLM-generated text (Solaiman al., 2019). Recent works, such as DetectGPT (Mitchell et al., 2023) and Fast-DetectGP((Bao et al., 2024), focus on conditional probability curvature, which evaluates the direction of perplexity change after perturbing random tokens. In this work, we investigate novel detection score called persistent homology dimension (PHD) (Birdal et al., 2021; Tulchinskii et al., 2024), which shows that LLM-generated text tends to have lower PHD value than human-written text. However, prior studies on PHD face challenges with short texts due to local density peak issues. We address this challenge by proposing novel method called off-topic content insertion.
18


Reviews:
[{"review_id": "ZtmoqXHnOj", "reviewer": "Reviewer_9J1C", "summary": "This paper addresses the challenge of detecting short LLM-generated texts by proposing Short-PHD, an extension of the Persistent Homology Dimension (PHD) method. The major innovation is the off-topic content insertion (OCI) technology, which stabilizes PHD estimation for short texts by prepending irrelevant content. \n\nThe authors claimed that OCI mitigates local density peaks in PHD estimation, thus improving detection robustness. Experiments on public and generated datasets show that Short-PHD outperforms zero-shot baselines (e.g., PHD, DetectGPT) and even supervised methods like RoBERTa, achieving a 25–30% improvement in AUC for 50-token texts.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Novelty: The OCI technique is creative and well-motivated, addressing a clear gap in PHD-based detection for short texts. \n\nTechnical soundness: The method is grounded in topological data analysis, with rigorous mathematical equations,  and a clear algorithmic workflow (Algorithm 1). The two hypotheses for OCI and Short-PHD are also empirically validated.\n\nGood performance: Short-PHD achieves significant improvements over PHD and some other baselines, and maintains robustness across domains and against the adversarial paraphrasing attacks.\n\nThe paper is well-structured.", "reasons_to_reject": "There could be a broader coverage of baseline methods.\n\n1. Although Fast-DetectGPT (Bao et al, 2024) is cited in addition to DetectGPT (Mitchell et al., 2023), its performance on your dataset is not  added to Table 4. According to the original Fast-DetectGPT paper, it should have better overall detection acc for most cases. Just wondering why it is not reported.\n\n2. Here are some recent detection methods that may be considered as a comparison detection:\n- FourierGPT, Xu et al., 2024: A psycholinguistics-inspired method that use spectral features of surprisal for detection, which works particularly well on short text (out-performing Fast-DetectGPT on PubMed)\n- GPT-who, Venkatraman et al., 2024: Also a psycholinguistics-inspired method that utilizes UID (uniform information density) features, which has also out-performed DetectGPT on some datasets.\n- Han et al., 2024, Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text\n- Yang et al., 2023, DNA-GPT, which provides more interpretability at n-gram levels.", "questions_to_authors": "What is the time and storage complexities of Short-PHD (and PHD), and how does it compare to likelihood-based methods that requires one-time or multiple times of LLM inference.", "extra_scores": {}}, {"review_id": "hGCpG90F7c", "reviewer": "Reviewer_hvbd", "summary": "Tulchinskii, et al. 2024 introduced a method for detecting LLM-generated text based on the persistent homology dimension (PHD) of contextual embeddings. This paper introduces an extremely simple modification to that method, showing that detection performance can be substantially improved by simply appending a paragraph of text before the input, especially for shorter input texts. The paper argues that this approach works because PHD estimators are unstable given only a small number of inputs, leading to high variance. Adding dummy text before the input results in a larger number of vectors to be input to the PHD estimator, which stabilizes its result.\n\nThe paper also provides comparisons to a number of standard zero-shot baselines, and also provides some ablations on paraphrasing and decoherence attacks", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem of detecting LLM-generated text is an important one. The method in this paper is extremely simple to implement and provides clear gains over a well-known baseline (PHD).\n\n2. The method in this paper provides large improvements to the detection of very short texts, which is an important but understudied problem in the LLM detection space. However, it is worth noting that high-accuracy detection may not be possible for short texts [1].\n\n[1] Chakraborty, S., Bedi, A. S., Zhu, S., An, B., Manocha, D., & Huang, F. (2023). On the possibilities of ai-generated text detection. arXiv preprint arXiv:2304.04736.", "reasons_to_reject": "1. My main concern is that the baselines used in this paper are excessively weak, so while the method is clearly better than PHD, it's hard to determine how useful it is in real-world settings. In particular, DetectGPT with mismatched scoring and target models is a very weak method (cf. Figure 6 of the DetectGPT paper), and the RoBERTa-based classifier is trained on the outputs of GPT-2, which is substantially weaker than the target models. I'd consider implementing some stronger baselines, e.g., those in [2] or [3]. Of course, different methods have different tradeoffs, but I would recommend clearly delineating those when comparing models \n\n2. Nitpicking here, but the paper makes the claim that \"since PHD measures the connectedness of embeddings, limiting the inserted content to off-topic (i.e., irrelevant) content ensures that the PHD increases for all inputs because text embeddings are less connected,\" but I'm not sure this claim is actually supported by the results. Do you have any evidence that the inserted text needs to be off-topic? Or will any prefix text do just as well?\n\n[2] Hans, A., Schwarzschild, A., Cherepanova, V., Kazemi, H., Saha, A., Goldblum, M., ... & Goldstein, T. (2024). Spotting LLMs with binoculars: Zero-shot detection of machine-generated text. arXiv preprint arXiv:2401.12070. [3] Verma, V., Fleisig, E., Tomlin, N., & Klein, D. (2024, June). Ghostbuster: Detecting Text Ghostwritten by Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (pp. 1702-1717).", "questions_to_authors": "1. For figure 4(d), did you retrain the RoBERTa model on LLaMA? My guess is probably not, and the section on sensitivity to white-box detection should be updated to reflect that if so", "extra_scores": {}}, {"review_id": "9jf5N4yirz", "reviewer": "Reviewer_nqRY", "summary": "The paper proposes Short-PHD, a new zero-shot method for detecting short LLM-generated texts using Persistent Homology Dimension (PHD). It identifies a key limitation of the original PHD method (instability in estimating PHD for short texts due to “local density peaks”) and addresses this by inserting off-topic content (OCI) before PHD computation. The authors provide both theoretical justification and empirical evaluation, demonstrating that Short-PHD improves detection accuracy and robustness over prior zero-shot and supervised methods, especially for short texts.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "**Novelty:** The idea of stabilizing PHD via off-topic content insertion is new and well-motivated.\n\n**Empirical Evidence:** The paper provides experimental results on multiple datasets and LLMs, showing that Short-PHD generally outperforms compared baselines for short text detection. The sensitivity analyses on text length and the number of off-topic content pieces provide further support for the method's design choices.\n\n**Robustness Evaluation:** The evaluation of Short-PHD against decoherence and paraphrasing attacks is a valuable contribution, suggesting the method’s potential practical utility in adversarial settings.", "reasons_to_reject": "**Limited Theoretical Depth:** The mathematical justification for why OCI maintains the gap between human and machine PHD values is largely empirical and heuristic rather than formal.\n\n**Motivation for Off-Topic Content Choice:** While the paper states that the off-topic content is generally unrelated and provides the specific content used, a more detailed justification for the choice of this particular off-topic content, and a discussion of whether the nature of the off-topic content impacts the results, would strengthen the method’s explanation. Furthermore, repeated insertion of the same set of fixed contents may create patterns detectable by adversaries or confound downstream NLP tasks.\n\n**Writing Quality:** The overall writing is understandable, but there are opportunities for improvement in clarity, conciseness, and flow. For example, making the main body more self-contained.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose an extension of the persistent homology dimension (PHD) method for machine-generated text detection for the case of short texts. This is an important issue, as most detectors struggle with short texts. The newly proposed method Short-PHD has strong performance, even though it is an extremely simple modification of the original PHD method: add some extra off-topic text in order to stabilize PHD.

The additional results and the discussion are promising. Overall, this is good work, but the method is quite simple. I find this work borderline, leaning positive.

I am an expert, actively working on this topic, and I am very well familiar with related work.
Decision: Accept