Detecting LLM-Generated Korean Text through Linguistic Feature Analysis
Authors: Shinwoo Park, Shubin Kim, Do-Kyung Kim, Yo-Sub Han
Accepted at: acl25

KatFishNet: Detecting LLM-Generated Korean Text through Linguistic Feature Analysis

Shinwoo Park    Shubin Kim    Do-Kyung Kim    Yo-Sub Han*
Yonsei University, Seoul, Republic of Korea
pshkhh@yonsei.ac.kr, shubs919@yonsei.ac.kr, kdky95@yonsei.ac.kr, emmous@yonsei.ac.kr

Abstract

The rapid advancement of large language models (LLMs) increases the difficulty of distinguishing between human-written and LLM-generated text. Detecting LLM-generated text is crucial for upholding academic integrity, preventing plagiarism, protecting copyrights, and ensuring ethical research practices. Most prior studies on detecting LLM-generated text focus primarily on English text. However, languages with distinct morphological and syntactic characteristics require specialized detection approaches. Their unique structures and usage patterns hinder the direct application of methods primarily designed for English. Among such languages, we focus on Korean, which has relatively flexible spacing rules, a rich morphological system, and less frequent comma usage compared to English. We introduce KatFish, the first benchmark dataset for detecting LLM-generated Korean text. The dataset consists of text written by humans and generated by four LLMs across three genres. By examining spacing patterns, part-of-speech diversity, and comma usage, we illuminate the linguistic differences between human-written and LLM-generated Korean text. Building on these observations, we propose KatFishNet, a detection method specifically designed for the Korean language. KatFishNet achieves an average of 19.78% higher AUC-ROC compared to the best-performing existing detection method. Our code and data are available at https://github.com/Shinwoo-Park/katfishnet.

1 Introduction

The rise of LLMs has led to significant advancements in various writing tasks (Brown et al., 2020; Gómez-Rodríguez and Williams, 2023; Xiao et al., 2024). However, their ability to generate coherent texts also raises concerns about potential misuse, such as spreading misinformation (Pan et al., 2023; Wang et al., 2024a) and facilitating academic dishonesty (Zellers et al., 2019; Perkins, 2023). Consequently, detecting LLM-generated text is paramount for safeguarding academic integrity, preventing plagiarism, protecting copyrights, and upholding research ethics (Guo et al., 2023; Su et al., 2023b; Wu et al., 2023; Orenstrakh et al., 2024).

Despite its significance, research on Korean text has been limited. The limitations in research on detecting LLM-generated Korean text include the lack of suitable benchmarks and the challenges arising from the unique linguistic characteristics of the Korean language (Park et al., 2020; Kim et al., 2022; Yoon et al., 2023; Choi et al., 2024; Kim et al., 2024). We present the first dataset for detecting LLM-generated Korean text, along with a detection method that utilizes Korean linguistic features. Figure 1 provides an illustration of our proposed dataset and detection approach.

We present KatFish (Korean LLM-generated text Benchmark For identifying Authorship), a

<table>
  <tr>
    <th colspan="2">Benchmark and Detection for LLM-Generated Korean Text</th>
  </tr>
  <tr>
    <th>LLMs</th>
    <th>[Types of Writing]</th>
    <th>[Need for Detection]</th>
  </tr>
  <tr>
    <td rowspan="4">Essay, Persuasive<br>Poetry, Literary<br>Paper Abstract: Academic<br>Concise summary of research.</td>
    <td>Expresses individual opinions.<br>Free-form with poetic license.<br>Academic<br>Concise summary of research.</td>
    <td>Academic Integrity<br>Plagiarism & Copyright<br>Research Ethics</td>
  </tr>
  <tr>
    <th colspan="2">[Key Linguistic Features]</th>
  </tr>
  <tr>
    <td colspan="2">1. Word Spacing<br>2. POS Combinations<br>3. Punctuation</td>
  </tr>
  <tr>
    <th>Genres</th>
    <th>Essay<br>Poetry<br>Paper Abstract</th>
    <th>KatFishNet</th>
  </tr>
  <tr>
    <td colspan="3">KatFish<br>A Dataset of Korean<br>Human-Written Text<br>LLM-Generated Text<br>Detection Model<br>Using Korean<br>Linguistic Features</td>
  </tr>
</table>

Figure 1: An illustration of our KatFish dataset (Sec. 2) and the detection method KatFishNet (Sec. 3).
dataset developed specifically for detecting Korean text generated by LLMs. KatFish includes text samples from argumentative essays, poetry, and research paper abstracts, produced by both humans and four LLMs. Korean has distinct morphological and syntactic features, including flexible spacing rules and a rich system of postpositions and verb endings. These characteristics indicate that detection strategies designed for English may be less effective for Korean, highlighting the need for a language-specific approach. We explore the linguistic differences between human-written and LLM-generated Korean text by analyzing three key aspects: 1) word spacing; 2) part-of-speech combinations; and 3) punctuation. Our analysis uncovers distinct feature differences, revealing patterns that can be exploited for effective detection. Based on these findings, we propose KatFishNet, a machine learning-based detection method that incorporates the linguistic characteristics of the Korean language.

2 Dataset Construction: KatFish

The KatFish consists of three types of text:

• Essay: Argumentative essays aim to persuade readers of a specific viewpoint or claim. They include a thesis statement, a logically organized structure with supporting evidence and counterarguments, and a concise conclusion.

• Poetry: Poetry is a creative form of writing that focuses on expressing emotions and artistic ideas. It often features metaphor, symbolism, and rhythm, breaking traditional linguistic norms to achieve distinctive artistic effects.

• Paper Abstract: Paper abstracts are concise summaries of academic research. They use precise language and technical terminology to clearly communicate the purpose, methodology, and key findings of a study.

We select these three genres to build a dataset that captures a diverse range of real-life scenarios and linguistic features while addressing practical challenges in detecting LLM-generated Korean text. Each genre highlights the importance of accurate detection in real-world applications. For a detailed rationale behind our genre selection, see Appendix A.

Linguistic and Structural Diversity Essays typically follow a logical and coherent structure, incorporating argumentation and supporting evidence. Poetry stands out with its use of metaphor, rhythm, and stylistic innovation, often pushing the boundaries of traditional linguistic norms. In contrast, paper abstracts are compact and information-dense, marked by the frequent use of technical terms and discipline-specific language.

Practical Importance of Detection Detecting LLM-generated text in these genres is critical due to potential misuse. Essays generated by LLMs could facilitate academic dishonesty, eroding the value of original thinking. LLM-generated poetry raises concerns about plagiarism and the loss of authenticity in creative expression. Similarly, LLM-generated paper abstracts could threaten the integrity of academic research by introducing inaccuracies.

2.1 Human-Written Text Collection

We collect human-written Korean text from different sources depending on the type of writing.

Essay We collect writings from the essay corpus provided by AIHub\(^1\), which includes argumentative essays written by elementary, middle, and high school students. The collected essays cover a total of 11 topics: 4 topics for elementary school students, 4 topics for middle school students, and 3 topics for high school students. There are no overlapping topics among the essays written by elementary, middle, and high school students. The statistics of the essays written by humans included in our final dataset are as follows: 1) Elementary school: 69 essays; 2) Middle school: 78 essays; and 3) High school: 34 essays. Descriptions of each essay topic are provided in Table 5.

Poetry We collect free verse poems from the poetry corpus provided by the National Institute of Korean Language\(^2\). The collected poems are written by individuals under 10, those aged 10 to 19, those in their 20s and 30s, and those aged 40 and above. The statistics of the poems are as follows:

• Individuals under 10: 19 poems
• Individuals aged 10 to 19: 116 poems
• Individuals in their 20s and 30s: 44 poems
• Individuals aged 40 and above: 10 poems

\(^1\)https://aihub.or.kr/
\(^2\)https://kcorpus.korean.go.kr/index/goMain.do
Paper Abstract  We randomly select 100 papers related to language engineering from those published by the Korean Institute of Information Scientists and Engineers3 in 2016–2018.

2.2 LLM-Generated Text Collection

We generate LLM-generated Korean text using two commercial LLMs and two open-source LLMs. Specifically, we use the following LLMs: 1) GPT-4o: GPT-4o is a commercial LLM capable of understanding and processing all forms of input, including text, images, and speech. 2) Solar: Solar is a commercial LLM developed by Upstage, a Korean AI startup. 3) Qwen2 72B: Qwen2 is an open-source LLM developed by Alibaba, capable of understanding and processing around 30 languages, including Korean. 4) Llama3.1 70B: Llama3.1 is an open-source LLM developed by Meta, showing outstanding performance across various tasks. Table 6 shows the prompts used for text generation.

Essay  When generating essays using LLMs, we design instructions based on education levels, essay topics, and prompts. The same essay prompts used by human writers serve as inputs for LLMs. LLMs receive instructions to write essays following the given topic and prompt while maintaining a writing style suitable for the specified education level. This approach helps minimize the influence of writing proficiency differences across education levels when distinguishing between human-written and LLM-generated essays.

Poetry  When generating poems with LLMs, we provide the model with a human-written poem along with the age group of the poet and instruct it to create a new poem that matches the style and content suitable for that age group. The model takes the full human-written poem as input and generates a new poem based on it, mimicking a realistic scenario where a person may draws inspiration from existing works to produce something original. Additionally, the model composes poems tailored to a given age group, which helps reduce the impact of age-related differences in writing style when distinguishing between human-written and LLM-generated poems.

Paper Abstract  A paper abstract summarizes the overall content of a study and highlights its key contributions. Therefore, we have the LLM read the entire paper excluding the abstract and generate a new abstract from the remaining content.

Data Cleaning  We perform a manual analysis of the LLM-generated text and remove those that fall into the following three categories: 1) texts that do not follow the instructions and simply output the given prompt; 2) texts that repeatedly produce meaningless content (e.g., AI assistant); 3) text generated in languages other than Korean.

<table>
  <tr>
    <th></th>
    <th>Essay</th>
    <th>Poetry</th>
    <th>Paper Abstract</th>
    <th>Total</th>
  </tr>
  <tr>
    <td># Human</td>
    <td>181</td>
    <td>189</td>
    <td>100</td>
    <td>470</td>
  </tr>
  <tr>
    <td># GPT-4o</td>
    <td>181</td>
    <td>189</td>
    <td>100</td>
    <td>470</td>
  </tr>
  <tr>
    <td># Solar</td>
    <td>140</td>
    <td>189</td>
    <td>100</td>
    <td>429</td>
  </tr>
  <tr>
    <td># Qwen2</td>
    <td>181</td>
    <td>189</td>
    <td>17</td>
    <td>387</td>
  </tr>
  <tr>
    <td># Llama3.1</td>
    <td>88</td>
    <td>189</td>
    <td>61</td>
    <td>338</td>
  </tr>
  <tr>
    <td>Total</td>
    <td>771</td>
    <td>945</td>
    <td>378</td>
    <td>2,094</td>
  </tr>
</table>

Table 1: Data statistics of the KatFish dataset.

Dataset Statistic  Table 1 presents the data statistics of the KatFish dataset. The KatFish includes 470 human-written Korean text and 1,624 LLM-generated Korean text. Each text undergoes a careful manual review to ensure it does not contain any sensitive personal information. We demonstrate that KatFish provides a sufficiently large benchmark for the task of distinguishing between human-written and LLM-generated Korean text. In comparison, recent studies by Mitchell et al. (2023) and Su et al. (2023a) conduct experiments on similar tasks involving human-written and LLM-generated English texts using 150 to 500 examples.

Table 7 presents the mean and standard deviation of Eojeol counts in texts by each author for each genre. An Eojeol is the smallest unit in a Korean sentence, separated by spaces.

3 Detection Method: KatFishNet

We compare and analyze the linguistic features of human-written Korean text and LLM-generated Korean text, and design KatFishNet based on these findings. Specifically, we focus on spacing patterns, part-of-speech n-gram diversity, and comma usage patterns. These are closely related to writing habits, grammatical structures, and textual coherence.

3.1 Word Spacing Patterns

Unlike English, Korean has many exceptions and flexibilities in its spacing rules, making it one of the most variable and challenging aspects of writing. This makes it a valuable feature for examining stylistic and grammatical differences. In line with
the Korean orthography word spacing guidelines4, we examine bound nouns (BN) and auxiliary predicate elements (VX), which are the key part-of-speech (POS) categories related to word spacing. We also investigate Eojeol POS Diversity and Unspaced VX Diversity in Appendix E. We use Bareun POS tagger.

Bound nouns must be used in conjunction with another word but require word spacing as they function as nouns. Their dependent nature leads to frequent mistakes, accounting for 70% of total word spacing errors (Shin et al., 2015). We examine two metrics: Numerical Determiner–Bound Noun (MMN-BN) Space Ratio and Bound Noun (BN) Space Ratio. The MMN-BN Space Ratio measures the frequency of word spacing between a numeral determiner (MMN) and a bound noun (BN). BN Space Ratio quantifies the frequency of word spacing before a BN. To focus on cases with greater variation, we eliminate trivial cases where spaces may be omitted.

Auxiliary predicate elements attach to the main predicate to complement its meaning. In principle, VXs should have a space preceding them, but the guidelines allow flexibility in exceptional cases. When used correctly, omitting the space may enhance readability. VX Space Ratio measures the frequency of word spacing before a VX, excluding the specific case of "-이/가/-의" (ENDING) + "지" (VX), where spacing is strictly prohibited. This metric indicates how strictly the author adheres to the principle while also considering flexibility in applying exceptions.

Figure 2 reveals that in essays and poetry, human-written text exhibits a lower space ratio across all metrics. Notably, LLM-generated essays display a highly consistent BN Space Ratio, with a standard deviation of 0.02. While LLMs rigidly enforce spacing rules, humans often omit spaces, influenced by various stylistic and grammatical factors. These factors include prioritizing readability and convenience over adherence to principles, poetic license, and a lack of understanding of spacing rules.

The differences are the most evident in essays and the least pronounced in paper abstracts. This illustrates that spacing behavior is influenced by context-dependent stylistic tendencies. In domains where humans adhere to highly structured formats and conventions, word spacing patterns may be less significant. However, they remain useful in domains with a wider range of authors and writing styles.

Finding. LLMs strictly follow spacing rules, while human writers omit spaces due to stylistic and grammatical factors.

3.2 Part-of-Speech N-gram Diversity

We analyze POS n-gram diversity to examine structural differences between human-written and LLM-generated Korean text. Using the Kkma POS tagger (Park and Cho, 2014), we extract POS sequences from each text and compute the POS N-gram Diversity Score by dividing the number of unique POS n-grams by the total number of POS n-grams in the text. After calculating the average diversity score for all human-written and LLM-generated text, we compare the results to identify differences. We consider n-grams ranging from unigrams (1-gram) to pentagrams (5-gram) to capture linguistic patterns at different levels. Unigrams reflect basic lexical choices, while higher-order n-grams capture more complex syntactic structures and dependen-

![Comparison of space occurrence ratios between MMN+BN, prior to BN, and prior to VX.](page_98_120_1256_370.png)

Figure 2: Comparison of space occurrence ratios between MMN+BN, prior to BN, and prior to VX.
cies. By analyzing diversity across these varying n-gram lengths, we aim to gain a comprehensive understanding of how humans and LLMs construct text differently.

![Bar chart comparing POS n-gram diversity between human-written and LLM-generated Korean essays.](page_184_370_495_263.png)

Figure 3: Comparison of POS n-gram diversity between human-written and LLM-generated Korean essays.

Figure 3 compares the average POS n-gram diversity scores between human-written and LLM-generated essays. We include the analysis results for poetry and paper abstracts in the Appendix F due to space limitations. Human-written essays exhibit a higher diversity score than LLM-generated essays. The analysis results highlight that humans use a wider range of grammatical structures and construct sentences more flexibly than LLMs. Since LLMs generate text by selecting the most probable word combinations based on training data, they tend to repeat commonly used structures more frequently.

Finding. Humans tend to use a more diverse range of POS combinations in their writing compared to LLMs.

3.3 Comma Usage Patterns

In Korean, commas help improve readability and clarify meaning within sentences. We analyze differences in comma usage to compare how humans and LLMs structure sentences and manage their flow. Specifically, we investigate the proportion of commas within sentences, their placement, structural changes in sentences, and linguistic diversity around commas. We compute the following five metrics: 1) Comma Inclusion Rate: The proportion of sentences containing at least one comma out of all sentences in a text. 2) Average Comma Usage Rate: The number of commas in a sentence divided by the total number of morphemes in that sentence. 3) Average Relative Position of Comma: The position of each comma (counting the number of morphemes before it) divided by the total number of morphemes in the sentence. If multiple commas appear in a sentence, the average relative position is calculated. 4) Average Segment Length: The average length of sentence segments split by commas. 5) POS Diversity Score Before and After Comma: The diversity of part-of-speech pairs appearing before and after a comma. This score is calculated by dividing the number of unique POS pairs by the total number of POS pairs. We segment each text into individual sentences, compute these metrics for each sentence, and use the average values of these sentence-level metrics as the representative values for the entire text. Appendix G further explores the specific POS patterns surrounding commas.

<table>
  <tr>
    <th>Genre</th>
    <th>Metric</th>
    <th>Human</th>
    <th>LLM</th>
  </tr>
  <tr>
    <td rowspan="4">Essay</td>
    <td>Inclusion Rate (%)</td>
    <td>26.31</td>
    <td><b>61.03</b></td>
  </tr>
  <tr>
    <td>Usage Rate (%)</td>
    <td>1.13</td>
    <td><b>2.56</b></td>
  </tr>
  <tr>
    <td>Avg. Relative Position</td>
    <td>0.09</td>
    <td><b>0.18</b></td>
  </tr>
  <tr>
    <td>Avg. Segment Length</td>
    <td>4.35</td>
    <td><b>8.56</b></td>
  </tr>
  <tr>
    <td rowspan="4">Poetry</td>
    <td>POS Diversity Score</td>
    <td>24.38</td>
    <td><b>59.39</b></td>
  </tr>
  <tr>
    <td>Inclusion Rate (%)</td>
    <td>27.01</td>
    <td><b>42.90</b></td>
  </tr>
  <tr>
    <td>Usage Rate (%)</td>
    <td>2.61</td>
    <td><b>4.84</b></td>
  </tr>
  <tr>
    <td>Avg. Relative Position</td>
    <td>0.14</td>
    <td><b>0.28</b></td>
  </tr>
  <tr>
    <td rowspan="4">Paper</td>
    <td>Avg. Segment Length</td>
    <td>1.96</td>
    <td><b>2.13</b></td>
  </tr>
  <tr>
    <td>POS Diversity Score</td>
    <td>23.13</td>
    <td><b>23.86</b></td>
  </tr>
  <tr>
    <td>Inclusion Rate (%)</td>
    <td>47.48</td>
    <td><b>65.21</b></td>
  </tr>
  <tr>
    <td>Usage Rate (%)</td>
    <td>1.73</td>
    <td><b>2.40</b></td>
  </tr>
  <tr>
    <td rowspan="4">Abstract</td>
    <td>Avg. Relative Position</td>
    <td>0.20</td>
    <td><b>0.25</b></td>
  </tr>
  <tr>
    <td>Avg. Segment Length</td>
    <td>9.07</td>
    <td><b>11.55</b></td>
  </tr>
  <tr>
    <td>POS Diversity Score</td>
    <td>42.85</td>
    <td><b>61.95</b></td>
  </tr>
</table>

Table 2: Comparison of comma usage patterns between human-written and LLM-generated Korean text.

Table 2 presents the analysis results. 1) LLMs include commas in more sentences and uses them more frequently: LLM-generated text contains a higher proportion of sentences with commas compared to human-written text. The frequency of comma usage within sentences is also higher in LLM-generated text, leading to more frequent segmentation. 2) LLMs tend to place commas later in a sentence than humans: While both humans and LLMs often place commas near the beginning of a sentence, analysis shows that LLMs tend to insert them slightly later. This can be attributed to the fact that LLMs are trained on large multilingual datasets, particularly those with substantial amounts of English, where comma usage patterns can differ from those commonly found in Korean. Additionally, in LLM-generated text, the segments of sentences separated by commas tend to be relatively longer compared to human-written text. 3)
LLM-generated text shows greater diversity in part-of-speech combinations around commas: A higher diversity of POS combinations before and after commas indicates that LLMs apply a wider range of grammatical patterns when constructing sentences. Poetry shows almost no difference between human and LLM-generated text. This similarity likely arises because poetry naturally consists of shorter and simpler sentence structures.

Finding. LLMs use commas more often, place them later, and show greater POS diversity than humans.

3.4 Design of KatFishNet

Our detection method, KatFishNet, is a machine learning-based model that leverages Korean linguistic features. Table 8 in Appendix H presents the Korean linguistic features used by KatFishNet. KatFishNet offers several advantages: First, it enables analysis of which features play a crucial role in detecting LLM-generated text, making the model highly interpretable. Second, compared to deep learning-based models such as Transformer-based approaches, it does not require training text embeddings or large-scale datasets, allowing for a lightweight and efficient detection system. Lastly, traditional machine learning models like logistic regression, random forests, and support vector machines provide a practical and efficient solution by enabling training on CPUs, avoiding the costs associated with GPU resources.

We construct input feature vectors based on quantitative values obtained from word spacing, POS combinations, or punctuation analysis, and perform machine learning on these vectors. In Section 5, we investigate which types of quantitative metrics are most effective in detecting LLM-generated Korean text by comparing the performance of models trained with different feature sets.

4 Experimental Settings

4.1 Task Definition

We address a binary classification task where, given a Korean text, the goal is to classify whether the text is generated by a human or an LLM. We evaluate performance using the Area Under the Receiver Operating Characteristic curve (AUC-ROC). AUC-ROC measures the area under the curve that plots the True Positive Rate against the False Positive Rate at different threshold levels. This metric measures overall detection performance across all potential classification thresholds.

4.2 Baselines

We select the following five types of baselines.

Confidence-based Methods Confidence-based methods leverage a pre-trained language model such as GPT-2 to analyze text and extract distinctive features, such as the rank or entropy of each word based on its preceding context. We employ the following methods: 1) Log-Likelihood (Solaiman et al., 2019): A language model calculates the log probability of each word in a given text, and the average of these values serves as the score. Higher average log probabilities indicate a greater likelihood that the text was generated by an LLM. 2) Entropy (Gehrmann et al., 2019): The entropy of each word is measured based on its preceding context, and the average entropy across the text is used as the score. LLM-generated text typically have lower entropy values. 3) Log-Rank (Mitchell et al., 2023): The absolute rank of each word is determined based on the preceding context, and the text-level score is obtained by averaging the logarithm of these values. 4) LRR (Log-Likelihood Log-Rank Ratio) (Su et al., 2023a): The LRR is computed by dividing log-likelihood values by log-rank values. Generally, LLM-generated text tend to have higher LRR values than human-written text.

Perturbation-based Methods Perturbation-based methods evaluate changes in the log probability of a model when making slight modifications to the original text. These methods use a pre-trained language model such as T5 to generate multiple perturbed versions of the text. By calculating log probabilities for both the original and perturbed text, they determine whether a text is machine-generated. We use the following methods: 1) DetectGPT (Mitchell et al., 2023): DetectGPT measures how the log probability of a model changes when making small modifications to the original text. This method is based on the idea that text generated by an LLM typically reaches a local optimum of the model log probability function. As a result, minor alterations to LLM-generated text tend to lower the log probability compared to the original version. 2) NPR (Normalized Log-Rank Perturbation) (Su et al., 2023a): NPR examines how the log-rank score responds to slight perturbations. When small modifications are applied, the log-rank
score of LLM-generated text increases more than that of human-written text.

LLM Paraphrasing This method determines whether a given text is human-written or LLM-generated by paraphrasing the original text using an LLM and measuring the similarity between the paraphrased and original versions. The method operates on the intuition that LLM-generated text undergo fewer changes because they align more closely with the generation patterns of LLMs (Zhu et al., 2023). In other words, if the similarity between the original and paraphrased text is high, the original text is considered to be LLM-generated. We measure this similarity using BARTScore (Yuan et al., 2021). We use Exaone 3.5 (Research et al., 2024) 32B, an LLM released by the Korean company LG AI Research, to perform paraphrasing.

LLM Prompting We provide a Korean text to an LLM and ask it to output 1 if the text is LLM-generated and 0 if it is human-written. For this baseline, we use Exaone 3.5.

Fine-tuning We fine-tune the encoder of a pre-trained language model using the KatFish dataset. Specifically, we build on a RoBERTa-base model initially trained on the HC3 dataset (Guo et al., 2023), which consists of English and Chinese text written by both humans and ChatGPT. By further training this model with the KatFish dataset, we enhance its ability to distinguish between human-written and LLM-generated Korean text.

The implementation details of KatFishNet and the baselines are provided in the Appendix I.

4.3 OOD Evaluation: Unseen LLMs

We perform out-of-distribution (OOD) evaluation to assess how well the detection methods generalize. Specifically, we test whether the model can accurately distinguish between human-written and LLM-generated Korean text even when faced with texts from an unseen LLM. This evaluation design is essential given the frequent emergence of LLMs with distinct text generation patterns. If a detection system relies only on data from familiar LLMs, it may struggle to maintain performance when confronted with a previously unseen model. By testing on LLMs not encountered during training, we can better approximate real-world conditions and gain deeper insights into how the detection system adapts without retraining for every new LLM.

We split the human-written text into an 8:2 ratio, using 80% of it along with text generated by GPT-4o—one of the most representative LLMs—to create the training dataset. For evaluation, we use text generated by Solar, Qwen2, and Llama3.1. Specifically, we construct three separate test sets by combining the text from each of these LLMs with the remaining 20% of human-written text. The detection methods requiring training include our proposed approach along with the fine-tuning baseline. These methods undergo training on the training dataset and are evaluated using the test sets. Despite the training process, the models are always tested on text generated by unseen LLMs. This ensures that all detection methods operate in a zero-shot classification setting.

5 Experimental Results

Table 3 presents the experimental results. We analyze the results from two perspectives: 1) Which baseline method performs best? 2) Which type of linguistic features contributes most to performance?

Best Performing Baseline Method The experimental results show that among the baseline methods, LLM paraphrasing achieves the highest performance for essays and abstracts, while DetectGPT performs best for poetry. In terms of average performance across the three text genres, LLM paraphrasing outperforms the other baselines. This may be because LLM paraphrasing directly exploits the characteristics of LLM-generated text. The results provide experimental support for the hypothesis that when an LLM modifies text, it introduces fewer changes to LLM-generated text than to human-written text.

Most Effective Linguistic Features We use logistic regression as the backbone model for KatFishNet and provide additional experimental results with random forest and support vector machine models in Appendix J. The results show that KatFishNet achieves the highest performance when leveraging comma usage patterns, compared to spacing patterns and POS n-gram diversity. KatFishNet with comma usage patterns outperforms all other methods across all three text genres. It achieves a 16.74% performance improvement over LLM paraphrasing for essays, a 10.72% improvement over DetectGPT for poetry, and a 31.90% improvement over LLM paraphrasing for paper abstracts. Meanwhile, KatFishNet with POS
<table>
  <tr>
    <th>Genre</th>
    <th colspan="2">Detection Methods</th>
    <th>→ Solar</th>
    <th>→ Owen2</th>
    <th>→ Llama3.1</th>
    <th>Average</th>
  </tr>
  <tr>
    <td rowspan="6">Essay</td>
    <td>Confidence</td>
    <td>Log-Likelihood</td>
    <td>83.84</td>
    <td>23.89</td>
    <td>66.20</td>
    <td>57.97</td>
  </tr>
  <tr>
    <td></td>
    <td>Entropy</td>
    <td>31.25</td>
    <td>84.53</td>
    <td>44.12</td>
    <td>53.30</td>
  </tr>
  <tr>
    <td></td>
    <td>Log-Rank</td>
    <td>78.84</td>
    <td>20.66</td>
    <td>61.92</td>
    <td>53.80</td>
  </tr>
  <tr>
    <td></td>
    <td>LRR</td>
    <td>45.08</td>
    <td>80.56</td>
    <td>53.15</td>
    <td>59.59</td>
  </tr>
  <tr>
    <td>Perturbation</td>
    <td>DetectGPT</td>
    <td>52.78</td>
    <td>37.45</td>
    <td>47.18</td>
    <td>45.80</td>
  </tr>
  <tr>
    <td></td>
    <td>NPR</td>
    <td>55.22</td>
    <td>19.90</td>
    <td>44.71</td>
    <td>39.94</td>
  </tr>
  <tr>
    <td>LLM Paraphrasing</td>
    <td>Exaone 3.5</td>
    <td>92.08</td>
    <td>79.74</td>
    <td>72.00</td>
    <td>81.27</td>
  </tr>
  <tr>
    <td>LLM Prompting</td>
    <td>Exaone 3.5</td>
    <td>50.42</td>
    <td>49.74</td>
    <td>50.07</td>
    <td>50.07</td>
  </tr>
  <tr>
    <td>Fine-tuning</td>
    <td>RoBERTa</td>
    <td>66.77</td>
    <td>66.65</td>
    <td>64.37</td>
    <td>65.93</td>
  </tr>
  <tr>
    <td rowspan="3">KatFishNet (Ours)</td>
    <td>Word Spacing</td>
    <td>86.00</td>
    <td>80.63</td>
    <td>71.91</td>
    <td>79.51</td>
  </tr>
  <tr>
    <td>POS Combinations</td>
    <td>92.26</td>
    <td>83.10</td>
    <td>73.63</td>
    <td>82.99</td>
  </tr>
  <tr>
    <td>Punctuation</td>
    <td>97.57</td>
    <td>94.63</td>
    <td>92.45</td>
    <td><b>94.88</b></td>
  </tr>
  <tr>
    <td rowspan="6">Poetry</td>
    <td>Confidence</td>
    <td>Log-Likelihood</td>
    <td>77.06</td>
    <td>47.34</td>
    <td>59.99</td>
    <td>61.46</td>
  </tr>
  <tr>
    <td></td>
    <td>Entropy</td>
    <td>30.90</td>
    <td>68.28</td>
    <td>47.68</td>
    <td>48.95</td>
  </tr>
  <tr>
    <td></td>
    <td>Log-Rank</td>
    <td>75.76</td>
    <td>45.67</td>
    <td>60.54</td>
    <td>60.65</td>
  </tr>
  <tr>
    <td></td>
    <td>LRR</td>
    <td>34.40</td>
    <td>55.86</td>
    <td>39.79</td>
    <td>43.35</td>
  </tr>
  <tr>
    <td>Perturbation</td>
    <td>DetectGPT</td>
    <td>67.04</td>
    <td>64.00</td>
    <td>67.02</td>
    <td>66.02</td>
  </tr>
  <tr>
    <td></td>
    <td>NPR</td>
    <td>63.75</td>
    <td>41.21</td>
    <td>62.92</td>
    <td>55.96</td>
  </tr>
  <tr>
    <td>LLM Paraphrasing</td>
    <td>Exaone 3.5</td>
    <td>71.32</td>
    <td>58.79</td>
    <td>61.51</td>
    <td>63.87</td>
  </tr>
  <tr>
    <td>LLM Prompting</td>
    <td>Exaone 3.5</td>
    <td>50.53</td>
    <td>50.16</td>
    <td>49.42</td>
    <td>50.03</td>
  </tr>
  <tr>
    <td>Fine-tuning</td>
    <td>RoBERTa</td>
    <td>60.35</td>
    <td>69.61</td>
    <td>55.96</td>
    <td>61.97</td>
  </tr>
  <tr>
    <td rowspan="3">KatFishNet (Ours)</td>
    <td>Word Spacing</td>
    <td>71.85</td>
    <td>65.56</td>
    <td>43.81</td>
    <td>60.40</td>
  </tr>
  <tr>
    <td>POS Combinations</td>
    <td>39.41</td>
    <td>79.17</td>
    <td>53.32</td>
    <td>57.30</td>
  </tr>
  <tr>
    <td>Punctuation</td>
    <td>62.65</td>
    <td>93.45</td>
    <td>63.22</td>
    <td><b>73.10</b></td>
  </tr>
  <tr>
    <td rowspan="6">Paper Abstract</td>
    <td>Confidence</td>
    <td>Log-Likelihood</td>
    <td>58.52</td>
    <td>42.41</td>
    <td>47.86</td>
    <td>49.59</td>
  </tr>
  <tr>
    <td></td>
    <td>Entropy</td>
    <td>36.13</td>
    <td>72.64</td>
    <td>51.85</td>
    <td>53.54</td>
  </tr>
  <tr>
    <td></td>
    <td>Log-Rank</td>
    <td>57.08</td>
    <td>45.05</td>
    <td>47.57</td>
    <td>49.90</td>
  </tr>
  <tr>
    <td></td>
    <td>LRR</td>
    <td>49.39</td>
    <td>47.82</td>
    <td>54.80</td>
    <td>50.67</td>
  </tr>
  <tr>
    <td>Perturbation</td>
    <td>DetectGPT</td>
    <td>55.81</td>
    <td>51.70</td>
    <td>51.11</td>
    <td>52.87</td>
  </tr>
  <tr>
    <td></td>
    <td>NPR</td>
    <td>63.14</td>
    <td>46.76</td>
    <td>60.98</td>
    <td>56.96</td>
  </tr>
  <tr>
    <td>LLM Paraphrasing</td>
    <td>Exaone 3.5</td>
    <td>70.80</td>
    <td>36.47</td>
    <td>64.72</td>
    <td>57.33</td>
  </tr>
  <tr>
    <td>LLM Prompting</td>
    <td>Exaone 3.5</td>
    <td>48.60</td>
    <td>46.41</td>
    <td>47.18</td>
    <td>47.39</td>
  </tr>
  <tr>
    <td>Fine-tuning</td>
    <td>RoBERTa</td>
    <td>50.70</td>
    <td>49.73</td>
    <td>50.02</td>
    <td>50.15</td>
  </tr>
  <tr>
    <td rowspan="3">KatFishNet (Ours)</td>
    <td>Word Spacing</td>
    <td>57.73</td>
    <td>66.91</td>
    <td>49.36</td>
    <td>58.00</td>
  </tr>
  <tr>
    <td>POS Combinations</td>
    <td>47.47</td>
    <td>70.05</td>
    <td>42.47</td>
    <td>53.33</td>
  </tr>
  <tr>
    <td>Punctuation</td>
    <td>78.99</td>
    <td>77.47</td>
    <td>70.41</td>
    <td><b>75.62</b></td>
  </tr>
</table>

Table 3: Performance of detecting LLM-generated Korean text. We report the average performance (AUC-ROC) over five experiments. We separately report the performance of the detection model for the task of distinguishing between human-written text and text generated by a specific LLM. For example, → Solar indicates that the detection model is evaluated on the task of classifying human-written text and text generated by Solar.

n-gram diversity as features ranks second among all methods for essay, while KatFishNet with spacing patterns as features ranks second for abstract. We hypothesize that comma usage patterns are more useful for detecting LLM-generated Korean text than spacing patterns or POS n-gram diversity, as LLMs tend to have more difficulty learning comma usage than word spacing or POS combinations. Spacing follows relatively clear patterns in training data, and POS sequences can be learned as probabilistic patterns. In contrast, comma usage reflects contextual and stylistic factors, making it highly variable depending on the intent of the writer.

We analyze the detection performance of KatFishNet in Appendix K by combining three categories of linguistic features using an ensemble approach.

Finding. Comma usage patterns serve as a key feature for distinguishing between human-written and LLM-generated Korean text.

6 Related Work

Zellers et al. (2019) developed the GROVER dataset, which includes human-written and AI-generated news articles to support research on detecting machine-generated disinformation. Similarly, Fagni et al. (2021) created TweepFake, a dataset of tweets authored by both humans and bots, facilitating studies on social media content authenticity. Guo et al. (2023) introduced the HC3 dataset, which contains questions and answers generated by both human experts and ChatGPT in English and Chinese. Wang et al. (2024c) advanced this direction by introducing the M4 dataset, a multi-generator, multi-domain, and multilingual corpus.
Wang et al. (2024b) introduced M4GT-Bench, a multilingual, multi-domain corpus covering various LLMs and proposes more nuanced tasks, such as pinpointing the specific LLM behind the generated text (multi-way detection) and identifying human–machine boundary points in mixed-content texts. Similarly, Macko et al. (2023) developed MULTITuDE, a large-scale multilingual dataset spanning eleven languages, to examine the cross-lingual and cross-generator generalization capabilities of detectors, comparing multilingual detectors to monolingual detectors. By systematically varying the training and evaluation languages, MULTITuDE demonstrates that English-focused models often do not seamlessly transfer to other languages without explicit multilingual training.

Although M4GT-Bench and MULTITuDE both address multilingual machine-generated text detection, neither benchmark focuses on language-specific approaches. In M4GT-Bench, the monolingual setup is trained exclusively on English data, while the multilingual setup uses a mixture of languages excluding the target language. As a result, there is no training that captures the morphological or syntactic characteristics of any single language in particular. MULTITuDE trains a separate model for each language and applies the same detection method across all of them. Its primary goal is to explore language transfer—that is, to examine how a model trained on certain languages generalizes to others—rather than to enhance performance for individual languages.

In contrast, our work highlights Korean-specific features—such as flexible spacing rules, part-of-speech n-gram patterns, and punctuation usage—in the design of the detection method. By focusing on language-specific characteristics, we demonstrate how linguistically tailored strategies can enhance the performance of LLM-generated text detection.

7 Conclusion

We address the challenge of distinguishing between human-written and LLM-generated Korean text by introducing KatFish, the first benchmark dataset for this task. Building on this foundation, we propose KatFishNet, a detection method that leverages linguistic features of the Korean language, including word spacing, POS combinations, and punctuation. Experimental results show that KatFishNet, particularly its use of comma usage patterns, sets a new state-of-the-art in detection performance. Notably, the strong performance of KatFishNet demonstrates the effectiveness of designing language-specific detection methods. Our research demonstrates the potential of designing detection methods based on linguistic features, providing a foundation for developing similar approaches for other languages in the future.

Limitations

This study has several limitations that should be acknowledged. First, the scope of the KatFish benchmark is limited to three specific genres: essays, poetry, and paper abstracts. While these genres provide a useful foundation, they do not encompass all text types where LLM-generated content could present risks, such as news articles, social media posts, and legal documents. Expanding the dataset to include a more diverse range of text types would enhance the generalizability of the findings.

Second, this study focuses on distinguishing between fully human-written and fully LLM-generated text. However, real-world scenarios often involve hybrid content, where human- and LLM-generated text are interwoven. Future research should investigate detection methods capable of effectively handling such mixed cases.

Finally, enhancing the performance of detection methods that rely on linguistic features requires advancements in Korean morphological analysis. Current morphological analyzers still face challenges, which can impact the accuracy and reliability of extracted features. Further improvements to these tools could improve linguistic feature extraction and detection performance.

Ethical Considerations

We ensure that the data collection process for KatFish respects privacy and intellectual property rights by using publicly available texts and generating AI content within ethical guidelines.

Acknowledgments

This research was supported by the National Research Foundation of Korea (NRF) grants funded by the Korean government (MSIT) (RS-2025-02222626 and RS-2025-00562134), and by the AI Graduate School Program (RS-2020-II201361).
References

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems.

ChangSu Choi, Yongbin Jeong, Seoyoon Park, Inho Won, HyeonSeok Lim, SangMin Kim, Yejee Kang, Chanhyuk Yoon, Jaewan Park, Yiseul Lee, HyeJin Lee, Younggyun Hahm, Hansae Kim, and Kyung-Tae Lim. 2024. Optimizing language augmentation for multilingual large language models: A case study on Korean. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).

Tiziano Fagni, Fabrizio Falchi, Margherita Gambini, Antonio Martella, and Maurizio Tesconi. 2021. Tweet-Fake: About Detecting Deepfake Tweets. Plos one.

Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. 2019. GLTR: Statistical Detection and Visualization of Generated Text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations.

Carlos Gómez-Rodríguez and Paul Williams. 2023. A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing. In Findings of the Association for Computational Linguistics: EMNLP 2023.

Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection. arXiv preprint arXiv:2301.07597.

Yuxuan Guo, Zhiliang Tian, Yiping Song, Tianlun Liu, Liang Ding, and Dongsheng Li. 2024. Context-aware watermark with semantic balanced green-red lists for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP).

Xinlei He, Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. 2023. MGTBench: Benchmarking Machine-Generated Text Detection. arXiv preprint arXiv:2303.14822.

HS Heaps. 1978. Information Retrieval: Computational and Theoretical Aspects.

Hyeondey Kim, Seonhoon Kim, Inho Kang, Nojun Kwak, and Pascale Fung. 2022. Korean language modeling via syntactic guide. In Proceedings of the Thirteenth Language Resources and Evaluation Conference (LREC 2022).

Jong Myoung Kim, Young-Jun Lee, Yong-Jin Han, Ho-Jin Choi, and Sangkeun Jung. 2024. Does incomplete syntax influence korean language model? focusing on word order and case markers. In First Conference on Language Modeling.

John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. A watermark for large language models. In International Conference on Machine Learning (ICML).

Dominik Macko, Robert Moro, Adaku Uchendu, Jason Lucas, Michiharu Yamashita, Matúš Pikuliak, Ivan Srba, Thai Le, Dongwon Lee, Jakub Simko, and Maria Bielikova. 2023. MULTITuDE: Large-scale multilingual machine-generated text detection benchmark. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.

Nilofar Mireshghallah, Justus Mattern, Sicun Gao, Reza Shokri, and Taylor Berg-Kirkpatrick. 2024. Smaller Language Models are Better Zero-shot Machine-Generated Text Detectors. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers).

Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. 2023. DetectGPT: Zero-shot Machine-Generated Text Detection using Probability Curvature. In International Conference on Machine Learning.

Michael Sheinman Orenstrakh, Oscar Karnalim, Carlos Anibal Suarez, and Michael Liut. 2024. Detecting LLM-Generated Text in Computing Education: Comparative Study for ChatGPT Cases. In 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC). IEEE.

Yikang Pan, Liangming Pan, Wenhui Chen, Preslav Nakov, Min-Yen Kan, and William Wang. 2023. On the Risk of Misinformation Pollution with Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2023.

Eunjeong L. Park and Sungzoon Cho. 2014. KoNLPy: Korean Natural Language Processing in Python. In Proceedings of the 26th Annual Conference on Human and Cognitive Language Technology, Chuncheon, Korea.

Kyubyong Park, Joohong Lee, Seongbo Jang, and Da-woon Jung. 2020. An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing.

Mike Perkins. 2023. Academic Integrity considerations of AI Large Language Models in the post-pandemic era: ChatGPT and beyond. Journal of University Teaching and Learning Practice.
LG Research, Soyoung An, Kyunghoon Bae, Eunbi Choi, Kibong Choi, Stanley Jungkyu Choi, Seokhee Hong, Junwon Hwang, Hyojin Jeon, Gerrard Jeong-won Jo, et al. 2024. Exaone 3.5: Series of large language models for real-world use cases. arXiv preprint arXiv:2412.04862.

Hocheol Shin, Buyeon Kim, and Kyubum Lee. 2015. A study on the hangeul orthography error status. Grammar Education, 23:63–94.

Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. 2019. Release Strategies and the Social Impacts of Language Models. arXiv preprint arXiv:1908.09203.

Jinyan Su, Terry Zhuo, Di Wang, and Preslav Nakov. 2023a. DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text. In Findings of the Association for Computational Linguistics: EMNLP 2023.

Zhenpeng Su, Xing Wu, Wei Zhou, Guangyuan Ma, and Songlin Hu. 2023b. HC3 Plus: A Semantic-Invariant Human ChatGPT Comparison Corpus. arXiv preprint arXiv:2309.02731.

Lionel Z Wang, Yiming Ma, Renfei Gao, Beichen Guo, Zhoran Li, Han Zhu, Wenqi Fan, Zexin Lu, and Ka Chung Ng. 2024a. MegaFake: A Theory-Driven Dataset of Fake News Generated by Large Language Models. arXiv preprint arXiv:2408.11871.

Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Osama Mohammed Afzal, Tarek Mahmoud, Giovanni Puccetti, Thomas Arnold, Alham Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. 2024b. M4GT-bench: Evaluation benchmark for black-box machine-generated text detection. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Toru Sasaki, Thomas Arnold, Alham Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. 2024c. M4: Multi-generator, Multi-domain, and Multilingual Black-Box Machine-Generated Text Detection. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamas work in English? on the latent language of multilingual transformers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024).

Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Derek F Wong, and Lidia S Chao. 2023. A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions. arXiv preprint arXiv:2310.14724.

Changrong Xiao, Wenxing Ma, Qingping Song, Sean Xin Xu, Kunpeng Zhang, Yufang Wang, and Qi Fu. 2024. Human-AI Collaborative Essay Scoring: A Dual-Process Framework with LLMs. arXiv preprint arXiv:2401.06431.

Soyoung Yoon, Sungjoon Park, Gyuwan Kim, Junhee Cho, Kihyo Park, Gyu Tae Kim, Minjoon Seo, and Alice Oh. 2023. Towards Standardizing Korean Grammatical Error Correction: Datasets and Annotation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023).

Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. BARTScore: Evaluating Generated Text as Text Generation. In Advances in Neural Information Processing Systems (NeurIPS).

Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending Against Neural Fake News. In Advances in Neural Information Processing Systems.

Biru Zhu, Lifan Yuan, Ganqu Cui, Yangyi Chen, Chong Fu, Bingxiang He, Yangdong Deng, Zhiyuan Liu, Maosong Sun, and Ming Gu. 2023. Beat LLMs at Their Own Game: Zero-Shot LLM-Generated Text Detection via Querying ChatGPT. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.

George Kingsley Zipf. 2016. Human Behavior and the Principle of Least Effort: An Introduction to Human Ecology. Ravenio books.
A  Genre Selection Criteria and Rationale

Practicality-Driven Domain Selection: Targeting Directly Problematic Use Cases  Our selection of text types is driven by a focus on practical use cases where the use of LLMs for text generation is directly problematic. This contrasts with other genres where LLM use is either legitimate or not explicitly prohibited, but may still lead to indirect issues. We prioritize three genres that directly relate to violations of academic integrity, copyright, and research ethics—major issues that are of immediate concern in real-world applications of LLM-generated text.

Ensuring Broad Coverage in Writing Types  Texts are classified into different types by their purpose, writing style, and usage. To ensure the broad applicability of our findings, we meticulously choose three genres—student-written argumentative essays, free-verse poetry, and research paper abstracts—to represent major facets of writing. Table 4 shows how each genre in KatFish contributes to this coverage.

<table>
  <tr>
    <th>Category</th>
    <th>Type</th>
    <th>Genres</th>
  </tr>
  <tr>
    <td rowspan="3">Purpose</td>
    <td>Expressive Writing</td>
    <td>Poetry</td>
  </tr>
  <tr>
    <td>Informative Writing</td>
    <td>Paper Abstract</td>
  </tr>
  <tr>
    <td>Persuasive Writing</td>
    <td>Essay</td>
  </tr>
  <tr>
    <td rowspan="4">Writing Style</td>
    <td>Descriptive</td>
    <td>Poetry</td>
  </tr>
  <tr>
    <td>Narrative</td>
    <td>Poetry, Essay</td>
  </tr>
  <tr>
    <td>Expository</td>
    <td>Paper Abstract</td>
  </tr>
  <tr>
    <td>Persuasive</td>
    <td>Essay</td>
  </tr>
  <tr>
    <td rowspan="2">Usage</td>
    <td>Literary Writing</td>
    <td>Poetry</td>
  </tr>
  <tr>
    <td>Academic Writing</td>
    <td>Essay, Paper Abstract</td>
  </tr>
</table>

Table 4: Summary of genre characteristics in KatFish, illustrating their complementary roles across purpose, writing style, and usage.

We choose free-verse poetry, a less conventional representative for literary text, to investigate how less conventional writing affects LLM-generated text detection. This allows us to gain insights that cannot be obtained from more structured forms.

B  Essay Topics for Argumentative Writing Across Education Levels

Table 5 shows the essay topics used for writing argumentative essays at each education level.

<table>
  <tr>
    <th>Education Levels</th>
    <th>Essay Topic</th>
  </tr>
  <tr>
    <td rowspan="3">Elementary</td>
    <td>다문화 가족을 대하는 본인의 자세 (Your Attitude Towards Multicultural Families)</td>
  </tr>
  <tr>
    <td>폭력 예방 방법 (Ways to Prevent Violence)</td>
  </tr>
  <tr>
    <td>과학의 발전에 대한 본인의 생각 (Your Thoughts on the Advancement of Science)</td>
  </tr>
  <tr>
    <td rowspan="6">Middle</td>
    <td>미디어 발전과 사용방법 (Advancement of Media and its Usage Methods)</td>
  </tr>
  <tr>
    <td>SNS상의 문제에 대한 본인의 생각 (Your Thoughts on the Issues Related to Social Media)</td>
  </tr>
  <tr>
    <td>e스포츠에 대한 본인의 생각 (Your Thoughts on Esports)</td>
  </tr>
  <tr>
    <td>전통과 악습에 대한 본인의 생각 (Your Thoughts on Tradition and Bad Practices)</td>
  </tr>
  <tr>
    <td>생물학적으로 다른 남녀에 대한 본인의 생각 (Your Thoughts on Biologically Different Males and Females)</td>
  </tr>
  <tr>
    <td>사회적 차별에 대한 본인의 생각 (Your Thoughts on Racism)</td>
  </tr>
  <tr>
    <td rowspan="3">High</td>
    <td>지적 재산권에 대한 본인의 생각 (Your Thoughts on Intellectual Property)</td>
  </tr>
  <tr>
    <td>평가에 대한 본인의 생각 (Your Thoughts on the Evaluation)</td>
  </tr>
</table>

Table 5: Essay topics used for argumentative writing at different education levels.
C  Prompt for Korean Text Generation via LLMs

<table>
  <tr>
    <th>Korean</th>
    <th>Prompt Template</th>
  </tr>
  <tr>
    <td>Essay</td>
    <td>너는 과제로 주장을 작성해야 하는 [EDUCATION LEVEL] 학생이야.<br>주어진 주제와 질문에 맞춰 에세이를 작성해줘. 한국어로만 작성해줘. 에세이만 출력해.<br>주제: [TOPIC] 질문: [ESSAY PROMPT]</td>
  </tr>
  <tr>
    <td>Poetry</td>
    <td>너는 [AGE GROUP] 지인이야. 주어진 시를 읽고 내용을 파악해줘.<br>그 후 너의 스타일로 너의 나이대에 맞는 새로운 시를 작성해줘.<br>한국어로만 작성해줘. 새로운 시만 출력해. 주어진 시: [POEM]</td>
  </tr>
  <tr>
    <td>Paper Abstract</td>
    <td>조목을 제외한 논문을 출력해. 이 논문의 조목을 작성해줘.<br>한국어로만 작성해줘. [PAPER]</td>
  </tr>
</table>

<table>
  <tr>
    <th>English</th>
    <th>Prompt Template</th>
  </tr>
  <tr>
    <td>Essay</td>
    <td>You are a [EDUCATION LEVEL] student who has to write an argumentative essay as an assignment.<br>Write an essay according to the given topic and question. Write only in Korean. Output the essay only.<br>Topic: [TOPIC] Question: [ESSAY PROMPT]</td>
  </tr>
  <tr>
    <td>Poetry</td>
    <td>You are a poet in your [AGE GROUP]. Read the given poem and understand its content.<br>Then, write a new poem in your style that suits your age group.<br>Write only in Korean. Output the new poem only. Given poem: [POEM]</td>
  </tr>
  <tr>
    <td>Paper Abstract</td>
    <td>I’ll give you a paper without the abstract. Write an abstract for this paper.<br>Write only in Korean. [PAPER]</td>
  </tr>
</table>

Table 6: Prompt templates used for building the KatFish benchmark. The upper table shows the original Korean prompt templates used for data generation, and the lower table displays the translated English prompt templates.

D  Data Statistics: Eojeol Counts

Table 7 presents the mean and standard deviation of the number of Eojeols in text written by each author for each text type. An Eojeol is the smallest unit in a Korean sentence, separated by spaces, and may consist of a single morpheme or a combination of multiple morphemes.

<table>
  <tr>
    <th></th>
    <th>Essay</th>
    <th>Poetry</th>
    <th>Paper Abstract</th>
  </tr>
  <tr>
    <td>Human</td>
    <td>152_{\pm54}</td>
    <td>52_{\pm52}</td>
    <td>91_{\pm27}</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>268_{\pm38}</td>
    <td>54_{\pm24}</td>
    <td>120_{\pm26}</td>
  </tr>
  <tr>
    <td>Solar</td>
    <td>175_{\pm56}</td>
    <td>40_{\pm19}</td>
    <td>94_{\pm55}</td>
  </tr>
  <tr>
    <td>Qwen2</td>
    <td>179_{\pm44}</td>
    <td>62_{\pm27}</td>
    <td>88_{\pm38}</td>
  </tr>
  <tr>
    <td>Llama3.1</td>
    <td>211_{\pm28}</td>
    <td>77_{\pm30}</td>
    <td>133_{\pm33}</td>
  </tr>
</table>

Table 7: The average and standard deviation of Eojeol counts for each text type by author.
E  Word Spacing Analysis

Alongside the three metrics related to word spacing in Section 3.1, we also analyze Eojeol POS Diversity and Unspaced VX Diversity. These metrics offer stylistic and grammatical distinctions in Korean text generation. Eojeol POS Diversity is computed by dividing the number of unique POS sequences by the total number of Eojeols in the text, where each sequence is defined at the Eojeol level. Eojeol POS diversity captures how diverse the syntactic structures are within individual Eojeols, reflecting differences in linguistic complexity and variability between human-written and LLM-generated text. Unspaced VX Diversity is computed by dividing the number of unique unspaced auxiliary verb stems by the total number of unspaced auxiliary verbs. We exclude the case where spacing is explicitly allowed. This analysis provides insights into the tendency of humans and LLMs to make consistent or varied spacing choices in VX-related word spacing.

Figure 4 shows that Eojeol VX Diversity is consistently higher for human-written text across all three genres, similar to the POS n-gram diversity results. Figure 5 illustrates that Unspaced VX Diversity results vary by genre, with humans scoring lower for essays and LLMs scoring lower for paper abstracts. This shows that genre-specific stylistic tendencies influence the spacing behavior of both humans and LLMs, reflecting variations in writing conventions and levels of formality.

![Bar chart comparing Eojeol POS Diversity between human-written and LLM-generated Korean text across Essay, Poetry, and Paper Abstract.](page_276_682_900_324.png)

Figure 4: Comparison of Eojeol POS Diversity between human-written and LLM-generated Korean text.

![Bar chart comparing Unspaced VX Diversity between human-written and LLM-generated Korean text across Essay, Poetry, and Paper Abstract.](page_276_1042_900_324.png)

Figure 5: Comparison of Unspaced VX Diversity between human-written and LLM-generated Korean text.
F  Part-of-Speech N-gram Diversity

Figure 6 compares the average POS n-gram diversity scores between human-written and LLM-generated Korean text. Excluding POS 4-gram and 5-gram in poetry, human-written text shows a higher diversity score than LLM-generated text. Poetry has shorter length and simpler structure compared to essays and paper abstracts, which can reduce the difference in POS n-gram diversity between human-written and LLM-generated textx for 4-gram and 5-gram sequences.

![Bar charts comparing POS n-gram diversity between human-written and LLM-generated Korean text for Essay, Poetry, and Paper Abstract](page_184_370_1082_312.png)

(a) Essay        (b) Poetry        (c) Paper Abstract

Figure 6: Comparison of POS n-gram diversity between human-written and LLM-generated Korean text.

G  POS-Comma Analysis

Exploring when humans and LLMs use commas can give valuable information on stylistic differences between human-written and LLM-generated text. Section 3.3 extensively discusses how humans and LLMs differ in their quantitative use of commas. In this section, we investigate the differences at the lexical level.

G.1  Before Comma

Figure 7 illustrates the ratio of each POS followed by a comma, calculated as the number of times a given POS appears before a comma divided by the total occurrences of that POS in the text. Endings and affixes are morpheme-level tags rather than POS tags. However, following the tagged output of the Bareun tagger, we include them in our analysis as a individual categories. Predicates and interjections are excluded due to their sparsity.

Use of Commas After Endings  While the frequency of a comma after a POS is consistently higher for LLM-generated text, the difference is particularly notable in endings. The ratios of a connective ending followed by a comma are 19.83%, 15.57%, and 28.01% for LLM-generated essays, poetry, and paper abstracts, respectively, while they are 4.10%, 4.68%, and 13.27% for human-written text. This discrepancy indicates that LLMs systematically overuse commas with connective endings, diverging from common patterns in human-written Korean text.

Use of Commas After Modifiers  The Korean orthography guidelines state that it is natural not to use a comma after conjunctive adverbs such as 그리고 ('and' in function), 그러나, 그런데 ('however'), and 그러므로 ('therefore'), as the functions of conjunctive adverbs and commas overlap. Unlike English, which often requires a comma in such cases, Korean does not, but its use remains a matter of stylistic preference. The tendency of LLMs to insert unnecessary commas after these modifiers suggests an influence of English punctuation norms on their output.

Influence of Multilingual Training  The higher frequency of commas following connective endings and modifiers in LLM-generated Korean text may be influenced by the biases ingrained in multilingual language models. Prior research highlights how multilingual models exhibit preferences shaped by dominant training languages (Wendler et al., 2024). Given that LLMs are trained on multilingual data with English-dominated
corpora, they may internalize English punctuation conventions, where commas frequently appear before conjunctions. This could lead LLMs to insert commas after connective endings more often than native Korean writers.

![Histogram showing the percentage of each part of speech that appears before a comma, measured by the number of times a given POS is followed by a comma relative to its total occurrences in the text.](page_184_232_1079_670.png)

Figure 7: Histogram showing the percentage of each part of speech that appears before a comma, measured by the number of times a given POS is followed by a comma relative to its total occurrences in the text.

G.2 Before and After Comma

Figure 8 shows the normalized POS pair frequencies, calculated as the proportion of each POS pair relative to the total number of comma uses. Sparse cases have been omitted. Both humans and LLMs primarily use commas in noun-noun and ending-noun pairs. Especially in essays, however, human-written texts exhibit a more concentrated pattern, with a few POS pairs showing high frequencies. In contrast, LLM-generated text tends to display a more evenly distributed pattern across different POS pairs. This suggests that humans use commas more selectively, whereas LLMs apply them more broadly.
(a) Essay

<table>
  <tr>
    <th></th>
    <th>adjective</th>
    <th>adverb</th>
    <th>determiner</th>
    <th>noun</th>
    <th>pronoun</th>
  </tr>
  <tr>
    <th>Before Human</th>
    <td>3.97</td>
    <td>7.73</td>
    <td>4.42</td>
    <td>30.68</td>
    <td>2.87</td>
  </tr>
  <tr>
    <th>Before LLM</th>
    <td>5.51</td>
    <td>3.81</td>
    <td>5.39</td>
    <td>30.99</td>
    <td>12.58</td>
  </tr>
  <tr>
    <th>After Human</th>
    <td>0.00</td>
    <td>1.99</td>
    <td>0.00</td>
    <td>38.19</td>
    <td>0.00</td>
  </tr>
  <tr>
    <th>After LLM</th>
    <td>0.00</td>
    <td>2.81</td>
    <td>0.00</td>
    <td>15.36</td>
    <td>0.00</td>
  </tr>
  <tr>
    <th>Before Human</th>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>4.64</td>
    <td>0.00</td>
  </tr>
  <tr>
    <th>Before LLM</th>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>11.24</td>
    <td>0.00</td>
  </tr>
  <tr>
    <th>After Human</th>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>4.35</td>
    <td>0.00</td>
  </tr>
  <tr>
    <th>After LLM</th>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>2.84</td>
    <td>0.00</td>
  </tr>
  <tr>
    <th>Before Human</th>
    <td>7.83</td>
    <td>13.04</td>
    <td>4.35</td>
    <td>24.35</td>
    <td>16.52</td>
  </tr>
  <tr>
    <th>Before LLM</th>
    <td>8.17</td>
    <td>9.99</td>
    <td>6.47</td>
    <td>19.18</td>
    <td>18.96</td>
  </tr>
  <tr>
    <th>After Human</th>
    <td>0.00</td>
    <td>2.61</td>
    <td>0.00</td>
    <td>14.78</td>
    <td>0.00</td>
  </tr>
  <tr>
    <th>After LLM</th>
    <td>0.00</td>
    <td>7.15</td>
    <td>0.00</td>
    <td>19.86</td>
    <td>0.00</td>
  </tr>
  <tr>
    <th>Before Human</th>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>12.17</td>
    <td>0.00</td>
  </tr>
  <tr>
    <th>Before LLM</th>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>7.38</td>
    <td>0.00</td>
  </tr>
</table>

(b) Poetry

<table>
  <tr>
    <th></th>
    <th>adjective</th>
    <th>adverb</th>
    <th>determiner</th>
    <th>noun</th>
    <th>pronoun</th>
  </tr>
  <tr>
    <th>Before Human</th>
    <td>2.71</td>
    <td>1.81</td>
    <td>5.88</td>
    <td>30.77</td>
    <td>4.52</td>
  </tr>
  <tr>
    <th>Before LLM</th>
    <td>2.33</td>
    <td>3.62</td>
    <td>4.67</td>
    <td>36.20</td>
    <td>8.21</td>
  </tr>
  <tr>
    <th>After Human</th>
    <td>0.00</td>
    <td>1.36</td>
    <td>0.00</td>
    <td>43.89</td>
    <td>0.00</td>
  </tr>
  <tr>
    <th>After LLM</th>
    <td>0.00</td>
    <td>1.45</td>
    <td>0.00</td>
    <td>31.38</td>
    <td>0.00</td>
  </tr>
  <tr>
    <th>Before Human</th>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>4.98</td>
    <td>0.00</td>
  </tr>
  <tr>
    <th>Before LLM</th>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>4.83</td>
    <td>0.00</td>
  </tr>
  <tr>
    <th>After Human</th>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>4.07</td>
    <td>0.00</td>
  </tr>
  <tr>
    <th>After LLM</th>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>7.32</td>
    <td>0.00</td>
  </tr>
</table>

(c) Paper Abstract

Figure 8: Normalized POS pair frequencies around a comma, expressed as the percentage of each POS pair relative to the total number of comma uses.
H  Summary of Linguistic Features

This section presents a summary of the linguistic features employed in KatFishNet. The feature set falls into three primary categories: word spacing, POS combinations, and punctuation. Table 8 outlines each feature with a brief description.

<table>
  <tr>
    <th>Category</th>
    <th>Feature</th>
    <th>Description</th>
  </tr>
  <tr>
    <td rowspan="3">Word Spacing</td>
    <td>MMN-BN Space Ratio</td>
    <td>The frequency of word spacing between a MMN and a BN.</td>
  </tr>
  <tr>
    <td>BN Space Ratio</td>
    <td>The frequency of word spacing before a BN.</td>
  </tr>
  <tr>
    <td>VX Space Ratio</td>
    <td>The frequency of word spacing before a VX, excluding the specific case of "-♂"-"♂" (ENDING) + "♂" (VX), where spacing is strictly prohibited.</td>
  </tr>
  <tr>
    <td>POS Combinations</td>
    <td>N-gram Diversity</td>
    <td>The syntactic variability in a text by computing the ratio of unique POS n-grams to the total number of POS n-grams.</td>
  </tr>
  <tr>
    <td rowspan="5">Punctuation</td>
    <td>Comma Inclusion Rate</td>
    <td>The proportion of sentences containing at least one comma out of all sentences in a text.</td>
  </tr>
  <tr>
    <td>Comma Usage Rate</td>
    <td>The number of commas in a sentence divided by the total number of morphemes in that sentence.</td>
  </tr>
  <tr>
    <td>Relative Position</td>
    <td>The position of each comma (counting the number of morphemes before it) divided by the total number of morphemes in the sentence.</td>
  </tr>
  <tr>
    <td>Segment Length</td>
    <td>The average length of sentence segments split by commas.</td>
  </tr>
  <tr>
    <td>POS Diversity Score</td>
    <td>The diversity of part-of-speech pairs appearing before and after a comma.</td>
  </tr>
</table>

Table 8: Overview of linguistic features used by KatFishNet across three categories.

I  Implementation Details

When building the KatFish dataset, we access GPT-4o and Solar through their official APIs and use Qwen2 and Llama3.1 via Ollama\footnote{https://ollama.com/search}. We implement the proposed detection method, KatFishNet, using machine learning models provided by Scikit-learn. For the fine-tuning baseline, we use the chatgpt-detector-roberta\footnote{https://huggingface.co/Hello-SimpleAI/chatgpt-detector-roberta} model released by the HC3 dataset authors on HuggingFace as the base model and fine-tune it for five epochs using the KatFish dataset. When reproducing confidence-based and perturbation-based baselines, we use the implementations provided by MGTBench (He et al., 2023). For LLM paraphrasing and LLM prompting baselines, we access Exaone 3.5 through Ollama. We conduct experiments on a server equipped with an NVIDIA RTX A6000 with 48GB of memory.
J  Performance of KatFishNet with Different Machine Learning Models

Table 9 presents the performance of KatFishNet with different base machine learning models. Below is a brief overview of the models incorporated in KatFishNet:

• Logistic Regression: A statistical model used when the dependent variable is categorical. It applies a sigmoid function to the input values to compute probabilities, and the regression coefficients help interpret how each feature influences the outcome.

• Random Forest: An ensemble method that constructs multiple decision trees and combines their predictions to produce the final output. Each tree is trained on randomly selected subsets of the data and features, thus enhancing the diversity and generalization capability of the model.

• Support Vector Machine: A classification algorithm that seeks an optimal decision boundary to separate data points. It learns by maximizing the margin between the decision boundary and the closest data points. This approach is effective in high-dimensional spaces and can handle complex data structures well.

We find that KatFishNet, which leverages comma usage patterns, achieves the highest performance in detecting LLM-generated Korean text, regardless of the type of machine learning model used as the backbone. Looking ahead, we also plan to develop an expanded version of KatFishNet based on ensemble or voting methods that integrate predictions from a diverse set of machine learning models. By understanding and leveraging the unique strengths of each model, we anticipate that our approach will achieve even stronger detection performance and robustness.

<table>
  <tr>
    <th rowspan="2">Genre</th>
    <th colspan="4">KatFishNet</th>
    <th rowspan="2">Average</th>
  </tr>
  <tr>
    <th>→ Solar</th>
    <th>→ Qwen2</th>
    <th>→ Llama3.1</th>
    <th>Average</th>
  </tr>
  <tr>
    <td rowspan="3">Essay</td>
    <td>Word Spacing<br>Logistic Regression</td>
    <td>86.00</td>
    <td>80.63</td>
    <td>71.91</td>
    <td>79.51</td>
  </tr>
  <tr>
    <td>Word Spacing<br>Random Forest</td>
    <td>81.56</td>
    <td>75.53</td>
    <td>68.35</td>
    <td>75.14</td>
  </tr>
  <tr>
    <td>Word Spacing<br>Support Vector Machine</td>
    <td>82.46</td>
    <td>78.29</td>
    <td>69.08</td>
    <td>76.61</td>
  </tr>
  <tr>
    <td rowspan="3">Essay</td>
    <td>POS Combinations<br>Logistic Regression</td>
    <td>92.26</td>
    <td>83.10</td>
    <td>73.63</td>
    <td>82.99</td>
  </tr>
  <tr>
    <td>POS Combinations<br>Random Forest</td>
    <td>91.85</td>
    <td>80.08</td>
    <td>75.08</td>
    <td>82.33</td>
  </tr>
  <tr>
    <td>POS Combinations<br>Support Vector Machine</td>
    <td>89.03</td>
    <td>73.60</td>
    <td>76.02</td>
    <td>79.55</td>
  </tr>
  <tr>
    <td rowspan="3">Essay</td>
    <td>Punctuation<br>Logistic Regression</td>
    <td>97.57</td>
    <td>94.63</td>
    <td>92.45</td>
    <td>94.88</td>
  </tr>
  <tr>
    <td>Punctuation<br>Random Forest</td>
    <td>96.07</td>
    <td>95.12</td>
    <td>90.29</td>
    <td>93.82</td>
  </tr>
  <tr>
    <td>Punctuation<br>Support Vector Machine</td>
    <td>96.26</td>
    <td>95.49</td>
    <td>91.33</td>
    <td>94.36</td>
  </tr>
  <tr>
    <td rowspan="3">Poetry</td>
    <td>Word Spacing<br>Logistic Regression</td>
    <td>71.85</td>
    <td>65.56</td>
    <td>43.81</td>
    <td>60.40</td>
  </tr>
  <tr>
    <td>Word Spacing<br>Random Forest</td>
    <td>59.24</td>
    <td>63.04</td>
    <td>52.47</td>
    <td>58.25</td>
  </tr>
  <tr>
    <td>Word Spacing<br>Support Vector Machine</td>
    <td>67.33</td>
    <td>69.79</td>
    <td>47.96</td>
    <td>61.69</td>
  </tr>
  <tr>
    <td rowspan="3">Poetry</td>
    <td>POS Combinations<br>Logistic Regression</td>
    <td>39.41</td>
    <td>79.17</td>
    <td>53.32</td>
    <td>57.30</td>
  </tr>
  <tr>
    <td>POS Combinations<br>Random Forest</td>
    <td>45.99</td>
    <td>69.27</td>
    <td>55.52</td>
    <td>56.92</td>
  </tr>
  <tr>
    <td>POS Combinations<br>Support Vector Machine</td>
    <td>43.66</td>
    <td>75.88</td>
    <td>56.53</td>
    <td>58.69</td>
  </tr>
  <tr>
    <td rowspan="3">Poetry</td>
    <td>Punctuation<br>Logistic Regression</td>
    <td>62.65</td>
    <td>93.45</td>
    <td>63.22</td>
    <td>73.10</td>
  </tr>
  <tr>
    <td>Punctuation<br>Random Forest</td>
    <td>63.21</td>
    <td>90.63</td>
    <td>60.91</td>
    <td>71.58</td>
  </tr>
  <tr>
    <td>Punctuation<br>Support Vector Machine</td>
    <td>60.76</td>
    <td>87.92</td>
    <td>60.65</td>
    <td>69.77</td>
  </tr>
  <tr>
    <td rowspan="3">Paper Abstract</td>
    <td>Word Spacing<br>Logistic Regression</td>
    <td>57.73</td>
    <td>66.91</td>
    <td>49.36</td>
    <td>58.00</td>
  </tr>
  <tr>
    <td>Word Spacing<br>Random Forest</td>
    <td>51.60</td>
    <td>49.67</td>
    <td>43.70</td>
    <td>48.32</td>
  </tr>
  <tr>
    <td>Word Spacing<br>Support Vector Machine</td>
    <td>55.86</td>
    <td>63.38</td>
    <td>46.99</td>
    <td>55.41</td>
  </tr>
  <tr>
    <td rowspan="3">Paper Abstract</td>
    <td>POS Combinations<br>Logistic Regression</td>
    <td>47.47</td>
    <td>70.05</td>
    <td>42.47</td>
    <td>53.33</td>
  </tr>
  <tr>
    <td>POS Combinations<br>Random Forest</td>
    <td>54.05</td>
    <td>55.29</td>
    <td>43.92</td>
    <td>51.08</td>
  </tr>
  <tr>
    <td>POS Combinations<br>Support Vector Machine</td>
    <td>52.62</td>
    <td>60.82</td>
    <td>49.37</td>
    <td>54.27</td>
  </tr>
  <tr>
    <td rowspan="3">Paper Abstract</td>
    <td>Punctuation<br>Logistic Regression</td>
    <td>78.99</td>
    <td>77.47</td>
    <td>70.41</td>
    <td>75.62</td>
  </tr>
  <tr>
    <td>Punctuation<br>Random Forest</td>
    <td>75.51</td>
    <td>77.32</td>
    <td>67.61</td>
    <td>73.48</td>
  </tr>
  <tr>
    <td>Punctuation<br>Support Vector Machine</td>
    <td>79.51</td>
    <td>76.35</td>
    <td>69.59</td>
    <td>75.15</td>
  </tr>
</table>

Table 9: Performance of detecting LLM-generated Korean text. We report the average performance (AUC-ROC) over five experiments.
K  Ensemble of Linguistic Features

We conduct an additional experiment to explore the effectiveness of combining word spacing, POS combinations, and punctuation features for LLM-generated Korean text detection. Specifically, we apply an ensemble approach by averaging the predicted probabilities (ranging from 0 to 1) from separate logistic regression models. A value of 1 indicates that the text is predicted to be LLM-generated. Tables 10, 11, and 12 show the experimental results.

<table>
  <tr>
    <th>Feature Combination</th>
    <th>→ Solar</th>
    <th>→ Qwen2</th>
    <th>→ Llama3.1</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>Word Spacing</td>
    <td>86.00</td>
    <td>80.63</td>
    <td>71.91</td>
    <td>79.51</td>
  </tr>
  <tr>
    <td>POS Combinations</td>
    <td>92.26</td>
    <td>83.10</td>
    <td>73.63</td>
    <td>82.99</td>
  </tr>
  <tr>
    <td>Punctuation</td>
    <td>97.57</td>
    <td>94.63</td>
    <td>92.45</td>
    <td>94.88</td>
  </tr>
  <tr>
    <td>All</td>
    <td>97.67</td>
    <td>93.93</td>
    <td>88.05</td>
    <td>93.21</td>
  </tr>
  <tr>
    <td>POS + Punctuation</td>
    <td>98.97</td>
    <td>95.09</td>
    <td>90.58</td>
    <td>94.88</td>
  </tr>
  <tr>
    <td>POS + Spacing</td>
    <td>90.29</td>
    <td>83.65</td>
    <td>73.73</td>
    <td>82.55</td>
  </tr>
  <tr>
    <td>Punctuation + Spacing</td>
    <td>97.89</td>
    <td>94.74</td>
    <td>90.35</td>
    <td>94.32</td>
  </tr>
</table>

Table 10: Performance on LLM-generated essay detection using individual and ensemble combinations of linguistic features. The ensemble method averages outputs from separate logistic regression classifiers.

<table>
  <tr>
    <th>Feature Combination</th>
    <th>→ Solar</th>
    <th>→ Qwen2</th>
    <th>→ Llama3.1</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>Word Spacing</td>
    <td>71.85</td>
    <td>65.56</td>
    <td>43.81</td>
    <td>60.40</td>
  </tr>
  <tr>
    <td>POS Combinations</td>
    <td>39.41</td>
    <td>79.17</td>
    <td>53.32</td>
    <td>57.30</td>
  </tr>
  <tr>
    <td>Punctuation</td>
    <td>62.65</td>
    <td>93.45</td>
    <td>63.22</td>
    <td>73.10</td>
  </tr>
  <tr>
    <td>All</td>
    <td>63.66</td>
    <td>96.04</td>
    <td>59.94</td>
    <td>73.21</td>
  </tr>
  <tr>
    <td>POS + Punctuation</td>
    <td>55.30</td>
    <td>94.83</td>
    <td>63.40</td>
    <td>71.17</td>
  </tr>
  <tr>
    <td>POS + Spacing</td>
    <td>52.09</td>
    <td>80.27</td>
    <td>48.69</td>
    <td>60.35</td>
  </tr>
  <tr>
    <td>Punctuation + Spacing</td>
    <td>73.78</td>
    <td>95.41</td>
    <td>58.64</td>
    <td>75.94</td>
  </tr>
</table>

Table 11: Performance on LLM-generated poetry detection using individual and ensemble combinations of linguistic features. The ensemble method averages outputs from separate logistic regression classifiers.

<table>
  <tr>
    <th>Feature Combination</th>
    <th>→ Solar</th>
    <th>→ Qwen2</th>
    <th>→ Llama3.1</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>Word Spacing</td>
    <td>57.73</td>
    <td>66.91</td>
    <td>49.36</td>
    <td>58.00</td>
  </tr>
  <tr>
    <td>POS Combinations</td>
    <td>47.47</td>
    <td>70.05</td>
    <td>42.47</td>
    <td>53.33</td>
  </tr>
  <tr>
    <td>Punctuation</td>
    <td>78.99</td>
    <td>77.47</td>
    <td>70.41</td>
    <td>75.62</td>
  </tr>
  <tr>
    <td>All</td>
    <td>68.93</td>
    <td>80.00</td>
    <td>57.01</td>
    <td>68.64</td>
  </tr>
  <tr>
    <td>POS + Punctuation</td>
    <td>71.70</td>
    <td>82.82</td>
    <td>60.93</td>
    <td>71.81</td>
  </tr>
  <tr>
    <td>POS + Spacing</td>
    <td>53.22</td>
    <td>70.76</td>
    <td>44.98</td>
    <td>56.32</td>
  </tr>
  <tr>
    <td>Punctuation + Spacing</td>
    <td>77.81</td>
    <td>79.70</td>
    <td>65.63</td>
    <td>74.38</td>
  </tr>
</table>

Table 12: Performance on LLM-generated paper abstract detection using individual and ensemble combinations of linguistic features. The ensemble method averages outputs from separate logistic regression classifiers.

In most cases, punctuation alone achieves detection performance that is either higher or comparable to that of ensemble models. This reinforces the strong effectiveness of punctuation-related features in detecting LLM-generated Korean text. Combining POS and spacing leads to the lowest performance among the ensembles, suggesting that these features alone do not capture the high-impact signals that punctuation features provide. Building on these findings, we plan to explore ensemble or voting approaches using multiple machine learning models, with a particular emphasis on punctuation features.
L  Methods for Detection Machine-Generated Text

This section serves as supplementary material to Section 6, providing an overview of methodologies for detecting LLM-generated text. Methods for detecting machine-generated text can be broadly grouped into three main categories:

Watermarking  Watermarking methods (Kirchenbauer et al., 2023; Guo et al., 2024) detect machine-generated text by embedding recognizable patterns during the text generation process. These methods intervene in the text generation process of LLMs by dividing the vocabulary into green and red lists and prioritizing the generation of tokens from the green list. If the proportion of green tokens in a generated text exceeds a certain threshold, the text is classified as machine-generated.

White-Box Detection  White-box detection methods detect machine-generated text by analyzing the log probability (Mitchell et al., 2023) or log rank (Su et al., 2023a) of the text. These methods rely on full or partial access to the text generator, which makes them difficult to use with commercial LLMs that restrict such access. Recently, Mireshghallah et al. (2024) proposed the use of surrogate models (e.g., GPT-2) to approximate log probability or log rank.

Black-Box Detection  Black-box detection methods require only the target text and do not rely on access to the text generator. For example, Zhu et al. (2023) introduced a paraphrasing-based detection approach using LLMs. Their method is based on the hypothesis that machine-generated text aligns more closely with the generation logic and statistical patterns learned by LLMs than human-written text. Accordingly, they proposed identifying a text as machine-generated if its paraphrased version closely resembles the original.

M  Detection Results From the LLMs Used to Generate the KatFish Dataset

We conduct an additional experiment to investigate whether the LLM that generated a given text can reliably distinguish between its own outputs and human-written texts. Table 13 presents the performance of the LLM-generated Korean text detection task using the LLMs employed in the creation of the KatFish dataset. The results suggest that even LLMs struggle to identify their own generated text, highlighting the limitations of LLM prompting baseline.

<table>
  <tr>
    <th>Genre</th>
    <th colspan="4">Detection LLMs</th>
    <th>Average</th>
  </tr>
  <tr>
    <td rowspan="4">Essay</td>
    <td>GPT-4o</td>
    <td>57.75</td>
    <td>49.70</td>
    <td>46.26</td>
    <td>51.23</td>
  </tr>
  <tr>
    <td>Solar</td>
    <td>46.50</td>
    <td>48.11</td>
    <td>47.38</td>
    <td>47.33</td>
  </tr>
  <tr>
    <td>Qwen2</td>
    <td>62.04</td>
    <td>61.03</td>
    <td>54.79</td>
    <td>59.28</td>
  </tr>
  <tr>
    <td>Llama3.1</td>
    <td>52.64</td>
    <td>37.95</td>
    <td>45.11</td>
    <td>45.23</td>
  </tr>
  <tr>
    <td rowspan="4">Poetry</td>
    <td>GPT-4o</td>
    <td>51.70</td>
    <td>52.28</td>
    <td>50.06</td>
    <td>51.34</td>
  </tr>
  <tr>
    <td>Solar</td>
    <td>49.99</td>
    <td>51.15</td>
    <td>44.34</td>
    <td>48.49</td>
  </tr>
  <tr>
    <td>Qwen2</td>
    <td>47.17</td>
    <td>49.91</td>
    <td>45.79</td>
    <td>47.62</td>
  </tr>
  <tr>
    <td>Llama3.1</td>
    <td>64.93</td>
    <td>48.58</td>
    <td>50.60</td>
    <td>54.70</td>
  </tr>
  <tr>
    <td rowspan="4">Paper Abstract</td>
    <td>GPT-4o</td>
    <td>48.20</td>
    <td>47.70</td>
    <td>47.77</td>
    <td>47.89</td>
  </tr>
  <tr>
    <td>Solar</td>
    <td>51.49</td>
    <td>52.32</td>
    <td>53.05</td>
    <td>52.28</td>
  </tr>
  <tr>
    <td>Qwen2</td>
    <td>50.00</td>
    <td>50.50</td>
    <td>49.52</td>
    <td>50.00</td>
  </tr>
  <tr>
    <td>Llama3.1</td>
    <td>49.80</td>
    <td>49.32</td>
    <td>49.68</td>
    <td>49.60</td>
  </tr>
</table>

Table 13: Performance Comparison of Different LLMs in Detecting LLM-Generated Korean Text. We report the average performance (AUC-ROC) over five experiments. We separately report the performance of the detection model for the task of distinguishing between human-written text and text generated by a specific LLM.
N  Detection with Knowledge-Integrated Few-Shot Prompting

We present an additional experiment that incorporates linguistic insights into the prompt for LLM-generated text detection. The experiment focuses on comma usage patterns, identified as the most effective feature in our main analysis.

N.1  Experimental Setup

We follow the equivalent dataset and experimental settings described in Section 4.3. We use Exaone 3.5, a Korean-focused LLM for this additional experiment. We compare the new experimental results with the simpler prompting approach from Table 3.

N.2  Prompt Design

Zero-shot Learning

Distinguish between Korean text written by a human and Korean text generated by an LLM.
Provide the basis for your decision under "Reasoning:".
Indicate the probability that the given text was generated by an LLM on a scale from 0 to 1 under "Answer:".

Korean texts written by LLMs tend to use commas more frequently than those written by humans.
Additionally, LLMs are more likely to place commas toward the latter part of a sentence.
They also tend to use a wider variety of part-of-speech combinations around commas.
Use these characteristics to help you distinguish the text.

Text to be classified: {t}
Reasoning:
Answer:

Few-shot Learning

Distinguish between Korean text written by a human and Korean text generated by an LLM.
Provide the basis for your decision under "Reasoning:".
Indicate the probability that the given text was generated by an LLM on a scale from 0 to 1 under "Answer:".

Korean texts written by LLMs tend to use commas more frequently than those written by humans.
Additionally, LLMs are more likely to place commas toward the latter part of a sentence.
They also tend to use a wider variety of part-of-speech combinations around commas.
Use these characteristics to help you distinguish the text.

Here are examples of Korean texts written by a human and by an LLM:
Human-written Korean text 1: {h1}
Human-written Korean text 2: {h2}
LLM-generated Korean text 1: {g1}
LLM-generated Korean text 2: {g2}
Text to be classified: {t}
Reasoning:
Answer:

N.3  Three Prompting Methods

We compare three prompting methods:
• No Knowledge: The default LLM prompting approach as in Table 3 (label: LLM Prompting Exaone 3.5), where the prompt simply asks whether the text is human-written or LLM-generated, without providing specific linguistic insights.

• Zero-shot Learning: The new prompt that provides linguistic insights about commas, without example texts of human/LLM writing.

• Few-shot Learning: The new prompt that provides linguistic insights about commas and two examples each of GPT4o-generated and human-written Korean texts.

N.4 Findings

Table 14 shows the experimental results:

• For poetry and paper abstracts, prompts that provide linguistic guidance about comma usage improve detection performance over the baseline. While Zero-shot yields limited benefit in essays, the Few-shot prompt leads to a noticeable improvement.

• Providing both linguistic insights and representative examples further enhances detection accuracy in nearly all cases.

<table>
  <tr>
    <th>Genre</th>
    <th>Prompting</th>
    <th>Solar</th>
    <th>Qwen2</th>
    <th>Llama3.1</th>
    <th>Average</th>
  </tr>
  <tr>
    <td rowspan="3">Essay</td>
    <td>No Knowledge</td>
    <td>50.42</td>
    <td>49.74</td>
    <td>50.07</td>
    <td>50.07</td>
  </tr>
  <tr>
    <td>Zero-shot</td>
    <td>51.89</td>
    <td>47.98</td>
    <td>46.39</td>
    <td>48.75</td>
  </tr>
  <tr>
    <td>Few-shot</td>
    <td>56.10</td>
    <td>51.70</td>
    <td>47.92</td>
    <td><b>51.90</b></td>
  </tr>
  <tr>
    <td rowspan="3">Poetry</td>
    <td>No Knowledge</td>
    <td>50.53</td>
    <td>50.16</td>
    <td>49.42</td>
    <td>50.03</td>
  </tr>
  <tr>
    <td>Zero-shot</td>
    <td>62.51</td>
    <td>66.03</td>
    <td>50.06</td>
    <td>59.53</td>
  </tr>
  <tr>
    <td>Few-shot</td>
    <td>67.16</td>
    <td>66.75</td>
    <td>57.35</td>
    <td><b>63.75</b></td>
  </tr>
  <tr>
    <td rowspan="3">Paper Abstract</td>
    <td>No Knowledge</td>
    <td>48.60</td>
    <td>46.41</td>
    <td>47.18</td>
    <td>47.39</td>
  </tr>
  <tr>
    <td>Zero-shot</td>
    <td>52.80</td>
    <td>64.26</td>
    <td>62.25</td>
    <td>59.77</td>
  </tr>
  <tr>
    <td>Few-shot</td>
    <td>53.81</td>
    <td>66.61</td>
    <td>59.91</td>
    <td><b>60.11</b></td>
  </tr>
</table>

Table 14: Comparison of detection performance across three LLM prompting strategies.

These results highlight the usefulness of incorporating linguistic insights—specifically, the differences in comma usage—into the prompt. We believe the findings signify the importance of detailed linguistic analysis, which can be cataloged and utilized in LLM context-learning approaches. We plan to explore this approach in more detail in our future work. For example, we may integrate chain-of-thought or other advanced prompting techniques with our linguistic analysis to further improve LLM-generated text detection.
O  Robustness of KatFishNet Against Human-Style Imitation via LLM Prompting

We investigate the robustness of KatFishNet in scenarios where LLMs are prompted to imitate human writing style. We experiment to see how effectively KatFishNet can detect text that has been deliberately revised to appear more human-like, focusing on comma-related linguistic features.

O.1  Experimental Setup
We follow the experimental settings described in Section 4.3. Specifically:

• We select 20% of the human-written texts and all texts generated by Solar for our data.

• We then feed each Solar-generated text back into Solar, instructing it to revise the text in such a way that it avoids LLM-like comma usage patterns (e.g., frequent comma placement, commas toward the end of sentences, diverse POS combinations around commas). Essentially, we ask the model to fix the text to make it more closely resemble human writing.

• We combine these revised Solar texts with the 20% human-written texts to form a new test dataset.

• We evaluate KatFishNet (using comma-related features) on this revised dataset.

O.2  Prompt Design
We create prompts for essays, poetry, and paper abstracts, which all share the same structure except for specifying which text type is being revised. Below is the essay-specific prompt example:

Essay Prompt
I'm going to give you a Korean essay written by an LLM.
Please revise it to make it sound like it was written by a human.
Provide the revised essay under "Revised Essay:".

Essays written by LLMs tend to use commas more frequently than those written by humans.
They also tend to place commas toward the latter part of a sentence and use a wider variety of part-of-speech combinations around commas.
Keep this in mind as you revise the essay to make it more human-like.

Essay to revise: {t}
Revised Essay:

O.3  Experimental Results
Table 15 compares Original (the results from Table 3, under the “→ Solar” column using Punctuation-based features) with Revised (detection performance on the newly revised texts).

<table>
  <tr>
    <th>Genre</th>
    <th>Original</th>
    <th>Revised</th>
  </tr>
  <tr>
    <td>Essay</td>
    <td>97.57</td>
    <td>90.83</td>
  </tr>
  <tr>
    <td>Poetry</td>
    <td>62.65</td>
    <td>75.91</td>
  </tr>
  <tr>
    <td>Paper Abstract</td>
    <td>78.99</td>
    <td>80.69</td>
  </tr>
</table>

Table 15: KatFishNet performance on Solar-generated texts before and after human-style revision.

Observations:

• Essays: There is a performance drop from 97.57% to 90.83%, yet the score remains above 90. This suggests that while the LLM manages some level of style manipulation, KatFishNet still maintains strong detection performance.
• Poetry & Paper Abstracts: Interestingly, we observe a performance increase (62.65→75.91 for poetry; 78.99→80.69 for abstracts). This suggests that instructing the LLM to alter comma usage may yield text that is even more easily flagged by the punctuation-based features of KatFishNet. This may be because the LLM performs human-like revisions based on an internal conception of human-written text, which does not fully align with actual human writing patterns.

These findings indicate that comma-related linguistic features are not trivially shallow. While commas may seem trivial, their usage reflects deeper, inherent characteristics of LLMs. Without a nuanced understanding of Korean stylistic conventions, LLMs struggle to convincingly mimic human-like text. Their multilingual nature also contributes to this challenge. This aligns with our hypothesis that LLMs may still have difficulty fully capturing the stylistic intricacies of Korean writing.

O.4 Further Justification
In our paper, we demonstrate that comma-related features are exceptionally useful for distinguishing between LLM-generated and human-written Korean texts. In particular:

• As discussed in Section 3.3, LLMs are trained on extensive multilingual datasets—often with a heavy emphasis on English. Since English comma usage differs considerably from Korean conventions, LLMs tend to adopt comma usage patterns that do not align with those commonly found in native Korean writing.

• As discussed in Section 5, comma usage reflects contextual and stylistic factors, making it highly variable depending on the intent of the writer. Consequently, it is challenging for LLMs to precisely mimic human writers.

• As detailed in Appendix G, we conduct an extensive analysis of the differences in part-of-speech patterns surrounding commas between human-written and LLM-generated texts.

Together, these results strongly support our hypothesis that comma-related features go beyond shallow surface-level analysis and capture critical stylistic and syntactic nuances inherent to Korean writing.

O.5 Future Directions
Adversarial attacks that aim to deceive LLM-generated text detectors represent an intriguing avenue for future exploration. We plan to explore prompt-engineered adversarial attacks, leveraging the linguistic distinctions found between human- and LLM-generated Korean texts, as part of our subsequent research.
P Morphological Analysis

We conduct a morphological analysis of Korean text written by humans and LLMs to analyze the frequency of different parts of speech. We perform the analysis using the HanNanum POS tagger, provided by the KoNLPy. Table 16 shows the English names of each POS along with their corresponding Korean names. In Korean, words are classified into five word types (nominal, predicate, modifier, interjection, relation word) based on their function and into nine parts of speech based on their meaning (noun, pronoun, numeral, verb, adjective, determiner, adverb, interjection, particle).

<table>
  <tr>
    <th>English</th>
    <th>Korean</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>Nominal</td>
    <td>체언</td>
    <td>A nominal is a part of speech that includes nouns, pronouns, and numeral classifiers, functioning as the subject or object in a sentence.</td>
  </tr>
  <tr>
    <td>Predicate</td>
    <td>용언</td>
    <td>A predicate is a part of speech that includes verbs and adjectives, expressing actions, states, or descriptions while serving as the main component of the predicate in a sentence.</td>
  </tr>
  <tr>
    <td>Modifier</td>
    <td>수식언</td>
    <td>A modifier is a part of speech that includes adverbs and determiners, providing additional information about other words, such as nouns or verbs, by describing or qualifying them.</td>
  </tr>
  <tr>
    <td>Interjection</td>
    <td>독립언</td>
    <td>An interjection is a part of speech that expresses emotions, exclamations, or commands independently within a sentence, often without a grammatical connection to other words.</td>
  </tr>
  <tr>
    <td>Relation Word</td>
    <td>관계언</td>
    <td>A relation word, or particle, is a part of speech that links nouns, pronouns, or phrases to other words in a sentence, indicating grammatical relations such as subject, object, or possession.</td>
  </tr>
  <tr>
    <td>Ending</td>
    <td>어미</td>
    <td>An ending is a part of speech attached to the stem of a verb or adjective, modifying its meaning, tense, mood, or form, and determining the sentence’s grammatical structure.</td>
  </tr>
  <tr>
    <td>Affix</td>
    <td>접사</td>
    <td>An affix is a part of speech that attaches to a root word, altering its meaning or grammatical function, and includes prefixes, suffixes, infixes, and circumfixes.</td>
  </tr>
  <tr>
    <td>Symbol</td>
    <td>기호</td>
    <td>A symbol is a part of speech that includes non-alphabetic characters such as punctuation marks, numbers, and special signs, used to convey meaning or structure in writing.</td>
  </tr>
  <tr>
    <td>Foreign Language</td>
    <td>외국어</td>
    <td>Foreign language refers to words or parts borrowed from other languages, used within a text to convey specific meanings, often retaining their original form and pronunciation.</td>
  </tr>
</table>

Table 16: English names of each part of speech along with their corresponding Korean names.

Figure 9 visualizes the distribution of POS usage in human and LLM-written Korean text.

P.1 Higher Usage of Endings and Predicates in Human-written text

Endings and predicates significantly contribute to the diversity of sentence structure and expression. Humans naturally use a variety of sentence endings and predicates to connect sentences fluidly, express emotions, or reinforce arguments. These elements are essential for shaping the flow and rhythm of writing.

In contrast, LLM-generated text show relatively lower usage of endings and predicates. This indicates that LLMs often create simpler and more formulaic sentence structures. In creative or emotionally rich writing, they tend to struggle with replicating the nuanced flow of sentences that human writers achieve. Guided by statistical patterns rather than the natural rhythm of language, LLMs frequently produce sentences that lack the dynamic quality characteristic of human writing.

P.2 Higher Usage of Nominals in LLM-generated text

LLMs often rely heavily on nouns, focusing on generating sentences that are technical or primarily aimed at delivering information. Nouns play a key role in clearly conveying topics or concepts, and LLMs frequently build sentences around them, resulting in expressions that are direct and concise.

In contrast, human writers use nouns less frequently than LLMs, opting for a more balanced mix of verbs, adjectives, and adverbs to add depth and nuance to their writing. Human writing typically exhibits greater flexibility and complexity in sentence structures, reflecting context and emotion more effectively.

P.3 Overall Analysis

Human Writing is More Dynamic in Expression and Structure The abundant use of endings and predicates makes human writing appear more natural, capturing the flow of emotions and logic. This is because humans tend to connect sentences organically and emphasize a variety of expressions.

LLM-generated text are Optimized for Information Delivery The higher frequency of nouns indicates that LLMs focus on clarity and brevity, often structuring sentences in a more repetitive manner.
Figure 9: Histogram visualizing the distribution of part-of-speech usage in human-written and LLM-generated text.
Q  Lexical Diversity Analysis

We use Zipf’s Law (Zipf, 2016) and Heap’s Law (Heaps, 1978) to compare and analyze the lexical diversity between human-written text and text generated by LLMs. Zipf’s Law is an empirical rule that describes the frequency distribution of words in natural language texts. Zipf’s Law illustrates the relationship between word frequency and rank, showing that frequently used words appear much more often than less common words. Heap’s Law describes the relationship between the size of a text and the number of unique words (vocabulary size) it contains. Heap’s Law examines how the number of unique words increases with the total word count, indicating how often new words are introduced as the text lengthens. Figure 10 shows Zipf’s Law and Heap’s Law for Korean text written by humans and LLMs.

First, as seen in Zipf’s Law, text written by humans show a steep decline in the frequency of commonly used words, indicating that a wide variety of words are used. In contrast, text written by LLMs show a higher concentration of frequently used words, with noticeable repetition of certain vocabulary. This reflects the tendency of LLMs to repeatedly use words that frequently appear within learned patterns. Next, as observed in Heap’s Law, text written by humans demonstrate a consistent increase in the number of unique words as the total word count rises, showcasing lexical diversity. On the other hand, text written by LLMs show a slower increase in unique words compared to humans, indicating relatively lower lexical diversity. This suggests that LLMs tend to use vocabulary repetitively. In conclusion, LLMs exhibit lower lexical diversity when writing texts compared to humans.

Finding. LLMs tend to use a narrower range of vocabulary and frequently repeat specific word patterns when writing, unlike humans who typically show greater lexical diversity.

![Comparison of Zipf’s Law and Heap’s Law between Korean text written by humans and those generated by LLMs.](page_340_654_1067_496.png)

Figure 10: Comparison of Zipf’s Law and Heap’s Law between Korean text written by humans and those generated by LLMs.
R  Expert Evaluation of Human-Written vs. LLM-Generated Korean Text

R.1  Expert Evaluation Settings

Data Preparation for Expert Evaluation  We randomly select a total of 75 texts written by humans and four different LLMs. First, we randomly select text written by humans. For essays, we select two writings for each education level. For poetry, we select one poem each written by individuals under the age of 10 and over 40, and two poems each by individuals in their 10s and 20s & 30s. For paper abstracts, we select three different papers. We select essays and research papers written by LLMs that are on the same topics as those written by humans, and for research abstracts, we select abstracts from the same papers. Overall, we select a total of 30 essays, 30 poems, and 15 paper abstracts from humans and four LLMs.

Evaluator  We request qualitative analysis for expert evaluation from three native Korean speakers specializing in Korean literature or Korean language education. Among them, one is a university student majoring in Korean literature, and the other two are current high school Korean language teachers.

Evaluation Rubric  We design an evaluation rubric to assess text written by humans and generated by LLMs from multiple perspectives. For essays and paper abstracts, we categorize the evaluation criteria into Language, Organization, and Content, creating specific subcategories for each. In poetry, we evaluate based on Poetic Diction, Organization, Content, and Creativity, also detailing subcategories for these main categories. Each detailed evaluation item is rated on a 3-point scale to standardize the assessment process across different text types. This structured approach allows us to systematically compare the qualitative aspects of human and LLM-written Korean text. We have our rubric reviewed by an independent expert, a high school Korean language teacher specializing in Korean language education, alongside the evaluators involved in the expert evaluation. The evaluation rubrics we designed can be found in Figure 11.

Essay
• Language focuses on grammatical accuracy and semantic clarity, emphasizing the importance of clear communication in essays, which is crucial for conveying arguments effectively.
• Organization evaluates the logical structure, which is essential for maintaining a coherent flow of ideas and ensuring the reader can follow the argument easily.
• Content addresses the purpose of essays, which is to present arguments. Criteria like argument clarity, use of evidence, comprehension, and extension of ideas beyond the given passage are central because essays are often judged on their analytical depth and ability to engage with the topic.

Poetry
• Poetic Diction examines imagery and poetic devices, which are fundamental in poetry for evoking emotions and creating depth.
• Organization looks at the completeness of the poetic structure, acknowledging that form and structure are as vital as content in poetry.
• Content emphasizes emotion, sensitivity, and thematic clarity, highlighting the role of poetry in conveying complex emotions and abstract themes.
• Creativity evaluates the originality of content, diction, and organization, recognizing the reliance of poetry on creative expression.

Paper Abstract
• Language focuses on grammatical accuracy and sentence cohesion.
• Organization evaluates the abstract structure, ensuring that it follows the conventional format that guides readers through the research purpose, methods, and findings.
• Content targets the clarity of the research topic, purpose, and results, ensuring that abstracts succinctly summarize the essential components of the study.
<table>
  <tr>
    <th>Criteria</th>
    <th>3 (Excellent)</th>
    <th>2 (Average)</th>
    <th>1 (Poor)</th>
  </tr>
  <tr>
    <td>표현 Language</td>
    <td>문법적 정확성 Grammatical Accuracy<br>문법 오류가 거의 없다.<br>There are almost no grammatical errors.</td>
    <td>문장이 명확하고 이해하기 쉽다.<br>The sentences are clear and easy to understand.</td>
    <td>문장의 문법 오류가 있으나 전체적으로 이해하기는 어렵다.<br>There are some grammatical errors, but they do not interfere with overall understanding.</td>
  </tr>
  <tr>
    <td></td>
    <td>문장의 명확성 Semantic Clarity<br>문장이 명확하고, 의도가 효과적으로 전달된다.<br>The sentences are clear, and the intent is effectively conveyed.</td>
    <td>문장이 대체로 명확하나 일부 부분만이 불명확하다.<br>Sentences are generally clear, but some parts lack clarity.</td>
    <td>문장이 모호하거나 명확하지 않거나 일부 부분만이 불명확하다.<br>Some sentences are ambiguous, or the intent is unclear. Unnecessary or verbose expressions are used.</td>
  </tr>
  <tr>
    <td>구성 Organization</td>
    <td>구조적 체계성 Logical Structure<br>구조는 명확하며, 전체적으로 일관된 구조를 가지고 있다.<br>(예: 서론 - 본론 - 결론)<br>The structure is logically and is coherent throughout the whole essay (e.g., introduction - body - conclusion).</td>
    <td>구조가 대체로 논리적이나 일부 부분에서 일관성이 부족하다.<br>The essay is generally structured logically, but some parts lack coherence.</td>
    <td>구조가 논리적이지 않거나 주요 내용을 이해하기 어렵다.<br>The essay lacks a logical structure, making it difficult to follow and understand the main points.</td>
  </tr>
  <tr>
    <td></td>
    <td>주장의 명확성 및 일관성 Clear and Consistent Argument<br>주장이 명확히 제시되고, 전개에 따라 일관성 있게 유지된다.<br>The argument is clearly conveyed and maintained consistently throughout the essay.</td>
    <td>주장이 대체로 명확하나 일부 부분에서 일관성이 부족하다.<br>The argument is generally clear but in unclear or inconsistent places.</td>
    <td>주장이 명확하지 않거나 일관성이 부족하다.<br>The argument is unclear or lacks consistency.</td>
  </tr>
  <tr>
    <td>내용 Content</td>
    <td>증거의 적절성 Usage of Relevant Supporting Evidence<br>증거가 논리적으로 및 밝게 제시된다.<br>The evidence effectively supports the argument and is logical.</td>
    <td>증거가 대체로 명확한 것으로 보이나 일부 부분에서 부족하다.<br>The evidence generally supports the argument but is somewhat limited in some places.</td>
    <td>증거가 부족하거나 증거를 제시하지 못한다.<br>The evidence is insufficient or fails to support the argument.</td>
  </tr>
  <tr>
    <td></td>
    <td>자신의 사고의 확장성 Novelty Beyond the Given Passage<br>자신의 사고가 자신의 생각과 관점을 넘어서, 새로운 사고나 창의적인 아이디어가 담겨있다.<br>The author's thinking goes beyond the information provided in the passage.</td>
    <td>자신의 사고가 제한적으로 정보에만 의존하고, 정보를 바탕으로 한 단순한 해석에 머무른다.<br>The essay relies solely on the information presented in the passage.</td>
    <td>자신의 사고가 제한적으로 정보에만 의존하고, 정보를 바탕으로 한 단순한 해석에 머무른다.<br>The essay relies solely on the information presented in the passage.</td>
  </tr>
  <tr>
    <td></td>
    <td>자신의 답변에 대한 이해와 적절한 답변이 제시되었다.<br>The given passage is thoroughly understood, and an appropriate response is provided.</td>
    <td>자신의 답변에 대한 이해가 부족하거나, 답변이 부적절하거나 답변이 제시되지 않았다.<br>The passage is insufficiently understood, and the response is inappropriate or missing.</td>
    <td>자신의 답변에 대한 이해가 부족하거나, 답변이 부적절하거나 답변이 제시되지 않았다.<br>The passage is insufficiently understood, and the response is inappropriate or missing.</td>
  </tr>
</table>

(a) Essay

<table>
  <tr>
    <th>Criteria</th>
    <th>3 (Excellent)</th>
    <th>2 (Average)</th>
    <th>1 (Poor)</th>
  </tr>
  <tr>
    <td>시어 Poetic Diction</td>
    <td>시어가 매우 명확하게 드러나며, 독자에게 강한 인상을 주고, 독자가 강한 인상을 받는다.<br>The imagery is very vivid and leaves a strong impression on the reader.</td>
    <td>시어가 대체로 드러나지만, 일부 부분에서 이미지가 약하다.<br>The imagery is generally present, but some parts are weak.</td>
    <td>시어가 대체로 드러나지 않거나, 독자가 약간의 이미지를 받는다.<br>The imagery is too weak and does not reach the reader.</td>
  </tr>
  <tr>
    <td></td>
    <td>시적 장치의 활용 Use of Poetic Devices<br>시적 장치가 매우 효과적으로 사용되었다.<br>Poetic devices are used very effectively.</td>
    <td>시적 장치가 사용되었으나, 효과가 제한적이다.<br>Poetic devices are used, but their effect is limited.</td>
    <td>시적 장치가 사용되지 않았다.<br>Poetic devices are barely used.</td>
  </tr>
  <tr>
    <td>구성 Organization</td>
    <td>시구조의 합리도 Poetic Structure<br>시구조가 대체로 일관되어 있으며 주제를 효과적으로 전달한다.<br>The poem is well structured and leveraged effectively to convey the theme.</td>
    <td>시구조가 대체로 잘 구성되어 있으나, 주제 전달에 한계가 있다.<br>The poem's structure is generally well-organized, but there is room for improvement.</td>
    <td>시구조가 합리적이지 않거나 주제 전달에 한계가 있다.<br>The poem is poorly structured, hindering the delivery of the theme.</td>
  </tr>
  <tr>
    <td>내용 Content</td>
    <td>정서 및 감정성 Emotion and Sensitivity<br>시의 감정적 감각이 강하게 드러난다.<br>The poem deeply expresses emotion and sensitivity.</td>
    <td>정서와 감정성이 대체로 드러나지만 감정이 부족하거나 깊이가 부족하다.<br>Emotion and sensitivity are generally expressed but lack depth.</td>
    <td>정서와 감정성이 제대로 드러나지 않거나 감정이 부족하거나 깊이가 부족하다.<br>Emotions and sensitivity are not well expressed or are not felt at all.</td>
  </tr>
  <tr>
    <td></td>
    <td>주제의 명확성 Clarity of the Theme<br>주제가 명확하게 드러나며, 전개에 따라 일관성 있게 유지된다.<br>The theme is clearly expressed, and it is consistent throughout the poem.</td>
    <td>주제가 대체로 명확하나, 일부 부분에서 명확하지 않거나 일관성이 부족하다.<br>The theme is generally clear, but it is unclear in some parts.</td>
    <td>주제가 명확하지 않거나, 일관성이 부족하다.<br>The theme is unclear or lacks consistency.</td>
  </tr>
  <tr>
    <td>창의성 Creativity<br>(시의, 구조, 내용)</td>
    <td>시의 구조, 또는 내용에서 독창적이고, 독특한 생각이나 창의적인 아이디어가 담겨있다.<br>It includes some creative elements in its structure, organization, or content but conveys the theme in a generally predictable or conventional way.</td>
    <td>시의 구조, 또는 내용에서 일부 창의적인 생각이나 창의적인 아이디어가 담겨있다.<br>It includes some creative elements in its structure, organization, or content but conveys the theme in a generally predictable or conventional way.</td>
    <td>시의 구조, 또는 내용에서 독창적이고, 독특한 생각이나 창의적인 아이디어가 담겨있다.<br>It includes some creative elements in its structure, organization, or content and conveys the theme in a common or cliché manner.</td>
  </tr>
</table>

(b) Poetry

<table>
  <tr>
    <th>Criteria</th>
    <th>3 (Excellent)</th>
    <th>2 (Average)</th>
    <th>1 (Poor)</th>
  </tr>
  <tr>
    <td>표현 Language</td>
    <td>문법적 정확성 Grammatical Accuracy<br>문법 오류가 거의 없다.<br>There are almost no grammatical errors.</td>
    <td>문장이 문법 오류가 있으나 전체적으로 이해하기는 어렵다.<br>There are some grammatical errors, but they do not interfere with overall understanding.</td>
    <td>문법 오류가 자주 나타나 이해하기 어려운 경우가 많다.<br>Grammatical errors appear frequently, making it difficult to understand certain parts.</td>
  </tr>
  <tr>
    <td></td>
    <td>문장의 기획성 Sentence Cohesion<br>문장이 명확하고 이해하기 쉽다.<br>The sentences are clear and easy to understand.</td>
    <td>문장이 명확하지 않거나 이해하기 어렵다.<br>Some sentences are ambiguous or hard to understand.</td>
    <td>전반적으로 문장이 복잡하고 이해하기 어렵다.<br>Overall, the sentences are complex and difficult to understand.</td>
  </tr>
  <tr>
    <td>구성 Organization</td>
    <td>초록 구조의 체계성 Abstract Structure<br>초록이 명확한 구조를 가지고 있다.<br>- 제목만은 연구 주제나 연구 가이드라인을 명확히 제시한다.<br>- 연구의 배경, 제한, 연구 방법 등이 명확하게 제시되어 있다.<br>The abstract has a clear structure.<br>(e.g., research background - limitations of the study - methodology - contributions and findings)</td>
    <td>초록의 구조가 명확하지 않거나 연구 내용이 이해하기 어렵다.<br>The structure of the abstract is somewhat vague or the research content is difficult to understand.</td>
    <td>초록의 구조가 적절치 않다.<br>The structure of the abstract is inappropriate.</td>
  </tr>
  <tr>
    <td>내용 Content</td>
    <td>연구 주제 및 목적의 명확성 Clarity of the Research Topic and Purpose<br>연구 주제와 목적에 명확한 의미가 있으며 이해하기 쉽다.<br>The research topic and purpose are clearly defined and easy to understand.</td>
    <td>연구 주제와 목적에 대해 약간의 혼동이 있거나 이해하기 어렵다.<br>The research topic and purpose are somewhat ambiguous or hard to understand.</td>
    <td>연구 주제와 목적에 대해 명확하지 않거나 이해하기 어렵다.<br>The research topic and purpose are unclear or it is very unclear.</td>
  </tr>
  <tr>
    <td></td>
    <td>연구 결과 요약 Summary of the Research Results<br>연구 결과가 명확하게 요약되어 있다.<br>The research results are clearly summarized.</td>
    <td>연구 결과가 명확하지 않거나 요약이 부족하다.<br>The summary of the research results is somewhat insufficient or unclear.</td>
    <td>연구 결과 요약이 명확하지 않거나 요약이 부족하다.<br>The summary of the research results is insufficient or unclear.</td>
  </tr>
</table>

(c) Paper Abstract

Figure 11: Evaluation rubrics for expert evaluation.
Guidelines for Evaluation   We provide our evaluation rubric along with the texts to evaluators with the following guidelines: 1) Essay: We provide the topic of the essay, the education level of the author, and the essay prompt along with the text to be evaluated. 2) Poetry: For poetry, we include the age group of the author along with the text being evaluated. 3) Paper Abstract: We provide the title of the research paper and the full content of the paper along with the abstract to be evaluated. These guidelines ensure that evaluators have all necessary context to accurately assess the texts across different categories and criteria. Figures 12 and 13 respectively show an example essay for expert evaluation, and its English translation. We ensure that evaluators do not make biased assessments by not providing information on whether each evaluated text is written by a human or generated by LLMs.

3. 에세이 (주장하는 글)

[주제1]

<table>
  <tr>
    <th>주제</th>
    <th>#1 다문화 가족을 대하는 본인의 자세</th>
    <th>학교</th>
    <th>초등학교</th>
  </tr>
  <tr>
    <td rowspan="2">지문/질문</td>
    <td colspan="3">다문화 가족은 아버지 혹은 어머니가 외국 국적의 사람일 경우를 말합니다. 국가 정책을 통해 다문화 가정에 대한 이해 교육을 펼쳐 국민들의 생각을 변화시키려 하고 있지만, 여전히 사회나 학교 안에서 차별과 편견이 남아있습니다. 생김새와 문화가 한국인과는 다르다는 이유로 다문화 가정인 아이들이 사람들이 보는 시선에 의해 상처받는 경우가 많습니다. 다문화 가족의 아이들이 차별 혹은 편견 받지 않고 우리와 동일한 한국인으로 인정받고 존중하는 사회를 만들기 위해서는 어떠한 노력이 필요할까요? 여러분이 생각하기에 가장 먼저 실천되어야 하는 것이 무엇인지 작성해주시고 그 이유에 관해 설명해주세요..</td>
  </tr>
  <tr>
    <td colspan="3"></td>
  </tr>
  <tr>
    <td>고유번호</td>
    <td colspan="3">본문</td>
  </tr>
  <tr>
    <td>에세이_1_1</td>
    <td colspan="3">우리의 사회 안에서, 차별과 편견이 여전히 존재하고 있습니다. 특히, 생김새나 문화가 조금 다른 다문화 가정의 아이들이 이로 인해 상처받는 경우가 많습니다. 모든 사람이 동등한 인간이라는 것을 인정하고 존중하는 세상을 만들어야 합니다. 그러나, 그런 사회를 만들기 위해서 가장 먼저 실천되어야 할 일은 바로 교육입니다. 첫째로, 학교교육에 다문화 이해의 중요성을 강조해야 합니다. 우리 아이들에게는 생김새나 국적, 인종 등의 차이가 존재한다는 사실 자체를 알려주는 것뿐 아니라, 그 차이에 대해 공포감이나 편견을 가지지 말아야 합니다. 다양한 배경과 문화 속에서 성장하는 사람들에 대한 이해와 존중의 중요성을 가르쳐야 합니다. 둘째로, 사회 전반적인 다문화 인식 개선 교육도 필요합니다. 특히, 매체를 통해 다문화 가정이 보여지는 방식을 바꾸어야 합니다. 외모나 문화적 차이에 주목하지 말고 그들의 인간성을 강조하는 내용으로 변화시켜야 합니다. 이를 통해 국민들에게 다문화 가족 역시 우리 사회의 중요한 구성원임을 알리는 교육이 필요합니다. 셋째로, 법률적인 보호와 지원도 중요합니다. 차별이나 편견에 대해 강력하게 처벌하고, 다문화 가정의 아이들이 편견 없이 공평한 기회를 누릴 수 있도록 제도적 보완이 필요합니다. 마지막으로, 우리 모두 스스로의 행동을 바꾸어야 합니다. 우리는 이미 우리의 사회에서 다양한 차이가 존재한다는 사실에 익숙해져야 합니다. 다른 사람들과 다르다는 이유로 그들을 판단하거나 편견을 가질 대신, 이해하고 존중하는 자세를 보여줘야 합니다. 우리 모두 함께 노력하여 우리 주변의 다문화 가족들이 차별이나 편견 받지 않고 동일한 인간으로 인정받고 존중받는 세상을 만들어가야 합니다. 이 모든 것이 이루어진다면, 우리는 진정으로 공평하고 다채로운 사회를 경험하게 될 것입니다.</td>
  </tr>
</table>

Figure 12: Example essay for expert evaluation.
3. Essay (Argumentative Essay)

[Topic 1]

<table>
  <tr>
    <th>Topic</th>
    <th>#1 Your Attitude Towards Multicultural Families</th>
    <th>Education Level</th>
    <th>Elementary School</th>
  </tr>
  <tr>
    <td rowspan="2">Passage/Question</td>
    <td colspan="3">A multicultural family is a family where the father or mother holds a foreign nationality. Although efforts are being made to change public perception through national policies that promote education on multicultural families, discrimination and prejudice still persist in society and schools. Many children from multicultural families are often hurt by the way others look at them simply because their appearance or culture differs from that of Koreans. What can we do to create a society where children from multicultural families are treated equally as Koreans, free from discrimination or prejudice? Please write down what you think should be implemented first, and explain the reason for your choice.</td>
  </tr>
</table>

<table>
  <tr>
    <th>ID</th>
    <th>Text</th>
  </tr>
  <tr>
    <td>Essay_1_1</td>
    <td>Discrimination and prejudice still exist in our society. Children from multicultural families are often hurt simply because they look or act differently. We must create a world that recognizes and respects every person as an equal human being. However, the first step toward building such a society is education. First, schools need to stress the importance of understanding multiculturalism. Children should be taught not only that differences in appearance, nationality, and race exist but also that these differences should not be feared or met with prejudice. It’s crucial to teach the value of understanding and respecting people from diverse backgrounds and cultures. Second, there must be a broader effort to improve multicultural awareness across society. This includes changing how the media portrays multicultural families. Instead of focusing on physical or cultural differences, the emphasis should be on their humanity. This shift in perspective will help convey to the public that multicultural families are valuable members of our society. Second, it is necessary to improve multicultural awareness throughout society. In particular, the way multicultural families are portrayed in the media must change. Rather than focusing on differences in appearance or culture, the emphasis should be on their humanity. Through this approach, it is crucial to educate the public that multicultural families are also vital members of our</td>
  </tr>
</table>

Figure 13: Example essay for expert evaluation (English translation).
R.2 Expert Evaluation Results

Figure 14 presents the expert evaluation results for essays written by humans and those generated by LLMs. We compare the results for text written by humans, commercial LLMs (GPT-4o and Solar), and open-source LLMs (Qwen2 and Llama3.1). We report the average evaluation results of the three human evaluators. Our analysis reveals that essays generated by LLMs receive better evaluations than those written by humans across seven metrics. Notably, essays by LLMs are distinctly rated higher in Grammatical Accuracy compared to those written by humans. This is likely because LLMs are trained on extensive data, enabling them to learn grammatical rules and thereby generate texts with fewer grammatical errors. Overall, human-written essays score well in Novelty Beyond the Given Passage and Comprehension of the Given Passage. One reason is that humans tend to weave in creative elements that go beyond the given context. However, human-written essays receive lower ratings in these two metrics compared to LLM-generated essays, which is an interesting result showing that LLMs not only properly understand and write about the given passage but also have the capability to utilize knowledge beyond the information provided in the passage.

The evaluators provide descriptive comments for each sample. The three main characteristics identified in LLM-generated essays are the excessive use of commas, repetition, and linguistic maturity. LLMs overuse commas, especially after adverbial case markers and connective endings. LLMs frequently repeat expressions or present paraphrased versions of the same sentence. Furthermore, LLM-generated text often exceed the expected writing level for the given educational stage, exhibiting complex vocabulary, advanced logic, and intricate sentence structures.

![Bar chart comparing expert evaluation scores for human, commercial LLMs, and open-source LLMs across seven metrics](page_355_682_1042_377.png)

Figure 14: Results of expert evaluations for essays.
Figures 15 and 16 display the results of expert evaluations for poetry and paper abstracts, respectively. Our analysis indicates that for poetry, commercial LLMs generally receive higher ratings than open-source LLMs, and the evaluations for poetry written by humans and commercial LLMs are quite similar. For paper abstracts, those generated by commercial LLMs receive the highest ratings, while abstracts written by humans and open-source LLMs are rated similarly. Overall, while LLM-generated essays consistently outperform human-written ones across all metrics, the differences in evaluations for poetry and paper abstracts are less pronounced.

![Bar and line chart showing expert evaluation scores for poetry, comparing Human, Commercial LLMs, and Open Source LLMs across six categories: Vivid Imagery, Use of Poetic Devices, Poetic Structure, Emotion and Sensitivity, Clarity of the Theme, and Creativity.](page_246_370_1017_384.png)

Figure 15: Results of expert evaluations for poetry.

![Bar and line chart showing expert evaluation scores for paper abstracts, comparing Human, Commercial LLMs, and Open Source LLMs across five categories: Grammatical Accuracy, Sentence Cohesion, Abstract Structure, Clarity of the Research Topic and Purpose, and Summary of the Research Results.](page_246_900_1017_384.png)

Figure 16: Results of expert evaluations for paper abstracts.

Reviewer paper summaries: This paper focuses on detecting machine-generated texts in Korean. This paper first points out some Korean characters and then proposes a dataset named KatFish, which is the first dataset for detecting LLM-generated Korean text. Meanwhile, the authors propose KatFishNet to identify LLM-generated Korean text.

This study proposes a detection method specifically designed for LLM-generated Korean text, filling a gap in existing research. The authors constructed the KatFish dataset, which includes three types of text: argumentative essays, poetry, and research abstracts. Based on word spacing patterns, POS n-grams, and punctuation usage features, they developed KatFishNet as a detection model. Experimental results demonstrate that this method achieves high detection accuracy across various LLM-generated texts, with punctuation features contributing the most significantly. Furthermore, the study evaluates the generalization ability of KatFishNet on unseen LLM-generated texts, indicating its adaptability to different generative models.

Existing research on machine-generated text (MGT) detection primarily focuses on English text. This paper investigates MGT detection in Korean, emphasizing its unique structural and linguistic characteristics, specifically word-spacing patterns, part-of-speech (POS) patterns, and comma usage. The authors collected human-written data from essays, poetry, and paper abstracts, analyzing these patterns to identify distinguishing features. Based on their quantitative findings, they constructed input feature vectors and applied logistic regression, with additional experiments using random forests and SVMs (reported in the appendix). Their approach was compared against several baseline methods, including confidence-based techniques, perturbation-based methods, LLM paraphrasing, LLM prompting, and fine-tuning. The proposed method demonstrated a significant improvement in AUC-ROC performance over these baselines.
Summary of strengths: - This paper focuses on an important research question -- machine-generated text detection. Meanwhile, this paper focuses on Korean, which is not explored before.
- The authors propose a novel dataset and design a detecting model, which could promote the development of this field.
- Before proposing the detecting model, the authors have proven the differences between human-written and LLM-generated texts. Meanwhile, the authors conclude the findings, which makes it easy to follow.
- The experiment baselines and settings are comprehensive.

1.For the first time, a detection method specifically targeting LLM-generated Korean text is proposed, filling a gap in existing research.

2.The authors constructed the KatFish dataset, which is representative and valuable for experimentation.

3.In terms of experiments, the authors conducted a thorough and comprehensive analysis, fully evaluating the impact of different linguistic features and comparing the performance of various detection methods.

- The paper is clearly written, with well-explained methods, datasets, processing steps, and experimental details.
- It presents a detailed analysis of three key linguistic patterns: word-spacing patterns, part-of-speech (POS) N-gram diversity, and comma usage. The study effectively compares these patterns between human- and machine-generated text.
- The approach demonstrates a promising way to enhance MGT detection in a specific language by integrating linguistic features, leveraging insights from a deep analysis of human and machine texts, and achieving significant performance improvements.
- The paper incorporates an out-of-distribution (OOD) evaluation, testing detection methods on LLMs not encountered during training, which improves generalization ability.
Summary of weaknesses: This paper does not have major weaknesses. I only have a minor question:
- Which machine learning model do the authors employ? The authors could provide the details of the machine learning model. Meanwhile, the author could experiment with more machine learning models to conduct an ablation study.

1.The paper lacks a detailed framework diagram of the methodology, making it less intuitive for readers to understand the workflow of KatFishNet.

2.The KatFish dataset in this paper includes only three types of text: argumentative essays, poetry, and research abstracts. Since texts in other domains may exhibit different linguistic characteristics, the applicability of this method to those domains remains to be further explored.

3.The methodology of this paper primarily relies on shallow linguistic feature analysis (such as word spacing, POS combinations, and punctuation) to detect LLM-generated text. If an LLM is guided by a carefully designed prompt to adjust its writing style, making its linguistic features more closely resemble human writing, would this weaken the effectiveness of the detection model?

4.The related work section is relatively insufficient, and some sentences in the paper are overly lengthy, affecting readability and the overall flow of the text.

- Ambiguity in the selection of text types: The distinction between essays and paper abstracts is somewhat unclear. While essays are described as following a logical and coherent structure, and abstracts as compact and information-dense (L104-L110), both share structural similarities. Since the need for MGT detection exists across various scenarios (Figure 1), including academic integrity and plagiarism detection, the rationale for selecting these specific genres could be further justified. Expanding the dataset to include other text types, such as stories, news articles, and company/government reports, would improve domain diversity and strengthen the study’s applicability.
- Limited discussion on related work: The paper would benefit from a more comprehensive discussion of prior research on MGT detection and the linguistic analysis of structure and usage patterns. For MGT, as far as I know, existing works exist like M4GT-Bench: Evaluation Benchmark for Black-Box Machine-Generated Text Detection (Wang et al., ACL 2024), MULTITuDE: Large-Scale Multilingual Machine-Generated Text Detection Benchmark (Macko et al., EMNLP 2023) could be discussed.
- Dataset limitations: The dataset size is relatively small, which may impact the robustness of the conclusions. Additionally, as mentioned earlier, increasing domain diversity would enhance the study’s generalizability and practical relevance.
Comments, suggestions and typos: See the weakness above.

1.A detailed flowchart of KatFishNet can be created to visually illustrate its working mechanism for better comprehension.

2.The KatFish dataset can be further expanded to include a wider range of domain-specific data and undergo validation for its applicability.

3.It would be valuable to explore whether style imitation through prompts affects detection performance.

4.The writing of the paper can be refined to improve its clarity, coherence, and readability.

- L246-251: Consider placing the full term immediately after the abbreviation for clarity. Also, is MMN intended to refer to a numeral determiner? If so, clarifying this would be helpful.
- L406: Minor issue—“AUROC” is acceptable, but “AUC-ROC” is the more commonly used term.
- L541: Given the strong contextual learning abilities of LLMs, it would be valuable to explore integrating this knowledge into the model and prompting an LLM to complete the same task could provide insights into whether larger, decoder-only models can mitigate these linguistic patterns and improve MGT detection in Korean.
- Data selection: The rationale behind the chosen text types is somewhat unclear. Providing stronger justification for selecting these specific data types, and expanding the dataset if feasible, would enhance the study’s robustness.
Overall assessment scores: 4

3.5

3.5
Metareview score: 3.5
Metareview: This paper introduces KatFish, the first benchmark dataset specifically designed for detecting LLM-generated Korean text, and proposes KatFishNet, a lightweight and interpretable detection model leveraging Korean-specific linguistic features such as word spacing, part-of-speech (POS) n-gram diversity, and comma usage. The paper addresses the under-explored challenge of machine-generated text (MGT) detection in morphologically rich and syntactically distinct languages like Korean, where methods developed for English do not directly apply. Reviewers generally appreciated the novelty and importance of this work. The proposed method is sound, with detailed linguistic analyses and a well-motivated set of detection features.

However, some reviewers also raised concerns regarding the limited domain coverage of the dataset (restricted to essays, poetry, and paper abstracts) and the need for deeper engagement with related work on multilingual MGT detection. However, the authors’ rebuttal provided thoughtful justifications for their genre selection, emphasizing the practical motivation and complementary diversity of the three genres. They also promised to integrate further literature and consider broader domains in future work.
Summary of reasons to publish: 1. This paper introduces the first dedicated dataset and method for Korean MGT detection, addressing a major gap in the current literature dominated by English-centric approaches.

2.  Empirical results across different LLMs shows the effectiveness of the proposed detection method KatFishNet , especially in zero-shot and OOD settings.