M4GT-Bench: Evaluation Benchmark for Black-Box Machine-Generated Text Detection (2024)

Abstract: The advent of Large Language Models (LLMs) has brought an unprecedented surge in machinegenerated text (MGT) across diverse channels. This raises legitimate concerns about its potential misuse and societal implications. The need to identify and differentiate such content from genuine human-generated text is critical in combating disinformation, preserving the integrity of education and scientific fields, and maintaining trust in communication. In this work, we address this problem by introducing a new benchmark based on a multilingual, multidomain, and multi-generator corpus of MGTs — M4GT-Bench. The benchmark is compiled of three tasks: (1) mono-lingual and multi-lingual binary MGT detection; (2) multi-way detection where one need to identify, which particular model generated the text; and (3) mixed humanmachine text detection, where a word boundary delimiting MGT from human-written content should be determined. On the developed benchmark, we have tested several MGT detection baselines and also conducted an evaluation of human performance. We see that obtaining good performance in MGT detection usually requires an access to the training data from the same domain and generators. The benchmark is available at https://github. com/mbzuai-nlp/M4GT-Bench.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3964–3992 August 11-16, 2024 ©2024 Association for Computational Linguistics
1 Introduction
The advent of Large Language Models (LLMs) such as ChatGPT1 and GPT-4 marks a transformative era in text generation. These models are able to generate a coherent text that is very similar to human-written content. They are also easily accessible and becoming more and more widespread. While such tools significantly boost productivity across various fields such as journalism, social media, education, and academic writing, they also
1We refer to GPT-3.5-Turbo throughout this paper.
introduce unprecedented avenues for misuse and, consequently, pose negative societal implications.
LLMs can generate immense amounts of deceptive fake news cluttering the information space. In social media, they have the potential to automatically generate fake accounts and increase their influence in social media communities without much effort from their owners. In the academic context, these technologies may advance beyond mere replication of existing content, potentially diminishing the effectiveness of established plagiarism detection methodologies and, consequently, posing a threat to the fundamental tenets of scholarly integrity. As a result, it could impact academic writing by leading to an increase in publications of automatically generated papers lacking scientific merit in respected journals and conferences. Finally, the scientific community has started reporting on cases of potentially automatically generated reviews in some well-known conferences.
Studies have shown that humans perform only marginally better than random chance in distinguishing human-written from machine-generated texts (MGTs) (Wang et al., 2023). Therefore, automatic MGT detection becomes essential in tackling misinformation, preserving the integrity of digital platforms and the scientific community, and ensuring trust in communications.
Previous work typically frames MGT detection as binary classification (Zellers et al., 2019; Mitchell et al., 2023; Bao et al., 2023), focusing primarily on English. Moreover, the majority of studies also overlook the fact that the content can be a mixture of an MGT and human-written text. These limitations make the previous experimental setups significantly different from practice.
In this work, we address these issues by presenting a new benchmark for MGT detection that consists of three tasks, which were not previously
3964
explored in this exact formulation, each shedding light on different facets of this challenge.
• Task 1: Binary Human-Written vs. Machine-Generated Text Classification. The objective is to categorize a given text as either human-written or machine-generated. This task is similar to the problem formulations in the previous work. However, our dataset offers annotated resources of new domains and generators in multiple languages. Task 1 sets two distinct tracks, one focusing exclusively on English sources (monolingual), and the other embracing a multilingual scope and introducing greater diversity compared to the previous work.
• Task 2: Multi-Way Machine-Generated Text’s Generator Detection. This task involves identifying the specific generator responsible for producing a given text. The text may be either human-written or crafted by a LLM. A different perspective to look at Task 2 is that we are attributing authorship to a specific generator (Munir et al., 2021).
• Task 3: Human-Written to MachineGenerated Text Change Point Detection. In this task, the goal is to precisely recognize the boundary within the mixed text, where the transition occurs from human-written to machine-generated content.
Our contributions are summarized below:
• We construct a diverse public multilingual MGT detection benchmark involving nine languages, six domains, nine LLM generators and three different tasks.
• We introduce a novel task formulation for MGT detection, where human-machine mixed text is explored. It is the first attempt to evaluate the ability of automatic approaches on detecting a boundary between human-written and LLM-generated texts.
• On the developed benchmark, we have tested several strong MGT detection baselines and also conducted an evaluation of human performance. We see that obtaining good performance in MGT detection usually requires an access to the training data from the same domain and generators.
2 Related Work
Binary Detection The task of detecting MGTs has traditionally been formulated as a binary classification problem (Zellers et al., 2019; Gehrmann et al., 2019a; Solaiman et al., 2019; Ippolito et al., 2019). MGT detection can be broadly categorized into two main types: supervised and unsupervised methods. Supervised approaches (Wang et al., 2023; Uchendu et al., 2021; Zellers et al., 2019; Zhong et al., 2020; Liu et al., 2022) rely on annotated datasets to train classifiers. In contrast, unsupervised methods leverage white-box features such as likelihood and log-rank (Solaiman et al., 2019; Ippolito et al., 2019; Mitchell et al., 2023; Su et al., 2023; He et al., 2023; Hans et al., 2024), or employ watermarking techniques (Kirchenbauer et al., 2023; Zhao et al., 2023b,a) to identify machinegenerated text. Here, we focus on supervised approaches.
Wang et al. (2023) evaluate several several supervised detectors, including RoBERTa (Liu et al., 2019), XLM-R (Conneau et al., 2019), logistic regression classifier with GLTR features (Gehrmann et al., 2019b), stylistic features (Li et al., 2014), and NELA (Horne et al., 2019) features. Similar investigation of supervised methods is also done by Guo et al. (2023); Hu et al. (2023); Xiong et al. (2024).
Multi-Class Detection There is a growing interest for more fine-grained classification that can not only identify the nature of the texts (i.e., whether it is machine-generated or human written), but also its specific source (i.e., which LLM generates it?). The fine-grained multi-class classification problem is closely related to the authorship attribution (Uchendu et al., 2020; Venkatraman et al., 2023; Rivera Soto et al., 2024). Munir et al. (2021) find that texts generated by LLMs contain distinguishable signals that can be used to attribute the source of texts. Uchendu et al. (2020) investigate three authorship attribution problems: (1) whether two texts are produced by the same generator, (2) whether a text is generated by a machine or a human, and (3) which LLM generated the text. Venkatraman et al. (2023) examine if the principle that humans prefer to spread information evenly can help capturing unique signatures of LLMs and human authors. Similarly, Rivera Soto et al. (2024) leverage representations of writing styles.
Authorship Obfuscation Work on authorship obfuscation extends beyond the binary/multi-class classification perspective by addressing the adversarial context of co-authorship between human and machine (Macko et al., 2024). Several works show that MGT detection methods are susceptible to authorship obfuscation attacks such as paraphrasing, back-translation, and human-machine collaboration mixture (Crothers et al., 2022; Krishna et al., 2023; Shi et al., 2023; Koike et al., 2023). Gao et al. (2024) introduces a dataset with a mixture of machine and human written texts using operations such as polishing, completing (Xie et al., 2023), rewriting (Shu et al., 2023), adding natural noise (Wang et al., 2021), and adapting (Gero et al., 2022). Kumarage et al. (2023) use stylometric signals to quantify changes in tweets and detect if and when AI starts to generate tweets. Dugan et al. (2020, 2023) investigate human ability to detect the boundary, where machine generated continuations are from LLMs. However, they use currently outdated LLMs such as GPT2-XL. In this work, we suggest the task of automatic detection and localization of change points from human-written segments to machine-generated segments. Our task is more challenging than previous works as we used state-of-the-art LLMs for generating continuations
3 Datasets and Metrics
This section describes the datasets for each task and the details of their creation. The data statistics is shown in Table 1, 2 and 3.
3.1 Human vs. Machine
The task 1 aims to distinguish human-written from machine-generated text — a binary classification task. Based on the M4 dataset (Wang et al., 2023), given a domain, we up-sampled human text to match the total number of machine-generated text
to avoid data imbalance. After upsampling, the dataset consists of 65,177 human-written texts and 73,288 machine-generated texts as shown in Table 1. We additionally generated texts using GPT-4 for each domain to evaluate the detectors’ generalization ability on unseen and strong generators.
For the multilingual track, we integrate new languages (German and Italian), and news for Arabic by generators of ChatGPT and Jais-30B, as the statistics shown in Table 2.
Data Cleaning Simple artifacts in human-written texts, such as multiple newlines and bullet points, could assist detectors. Their presence in the training data may discourage detectors from learning more generalized signals. Most importantly, these artifacts primarily originate from data crawling (multiple newlines at the beginning and the end of text) and conversion (such as PDF to text).
To recover the original format of human-written plain text, we performed minimal cleaning on the upsampled human-written texts: (i) in a WikiHow text, we removed multiple commas at the beginning of a new line (e.g. “„„„, we believe that ...”) and repeating newlines (“\n\n\n\n text begin \n\n\n\n”); (ii) in Wikipedia articles, we removed references (e.g., [1], [2]), URLs, multiple newlines, as well as paragraphs whose length was less than 50 characters; and (iii) in arXiv abstracts, we removed newlines stemming from PDF conversion.
Metrics In addition to accuracy, we also use precision, recall, and F1-score with respect to MGT.
3.2 Multi-way Detection
Task 2 is to determine which generator model produces the considered text. This is motivated by the Intelligence Protection (IP) of LLMs generations. For example, it can be used to detect whether model A generations are extensively used to train another
commercial model B by distillation (this is forbidden by OpenAI). If many generations from B are detected as model A generations, one might suspect that model B used a large amount of generations of model A in the training process.
In general, identifying the specific model used to produce a copyrighted material is important for legal and copyright reasoning. Additionally, in cases where the generated material is harmful, misleading, or illegal, pinpointing the exact LLM responsible is essential for addressing ethical concerns and legal obligations.
Different from task 1, we use the parallel subset for task 2 without the upsampling of human-written texts, including six generators: ChatGPT, davinci-003, GPT-4, Cohere, Dolly-v2, and BLOOMz. A new domain OUTFOX is collected to evaluate the classifier’s domain generalization in student essays.
Metrics F1-score, precision, recall in terms of seven individual labels representing different generators and human-written texts are used. We also provide F1-macro and accuracy.
3.3 Boundary Identification
Task 3 aims to imitate the real-world LLM usage situation, where not the full text is generated by machine, but partially written by human and partially generated by a machine. To simplify the task, we formulate it as first written by a human and then continued by a machine, and the task is to detect the single boundary of change.
We consider two common misuse scenarios of human-machine mixed text — continuing to write academic paper reviews and student essays. Given the partial human-written essay with the corre-
sponding problem statement and the partial humanwritten paper reviews with the title and abstract, we complete them by generating text based on GPT and LLaMA-2 series using prompts as shown in Figure 2. The proportion of words that are humanwritten range from 0 to 50%. The goal of a detection model is to identify the boundary position from which the text is generated by machines.
Mixed texts are specifically generated using ChatGPT, GPT-4, and LLaMA-2 (7B, 13B and 70B). We generate 5,676 and 1,000 examples for two domains respectively using each generator (full statistics is presented in Table 3).
How realistic is the task setting? Providing the beginning of a text and asking an LLM to generate the rest is a common use case. At the same time, we believe that MGTs seriously modified by humans might not represent a malicious use case of LLMs, as LLMs are legitimate writing tools. Depending on the amount of human intervention, one could argue that this is not anymore machine-generated, but machine-human collaboration. Therefore, we seek balance between the complexity of the task and its practicality. Moreover, detecting texts that contain multiple changing points is an interesting research direction for future work, but it would bring new challenges with analyzing shorter spans. Extremely, if a human added an extra word in a machine-generated sentence, it would be much harder for blackbox detectors to identify this word.
Metrics Mean Absolute Error (MAE) is used to evaluate the boundary detection model’s performance. It measures the average absolute difference between the predicted position index and the actual changing point.
4 Human Evaluation
We investigate the complexity of differentiating the language model used for creating machinegenerated texts for human readers. Considering that annotators may lose patience to learn and summarize the patterns of seven distinct generators, we control the number of unique models they could identify and simplify to four-class classification with five demonstration examples (five-shot) as practice to learn before the formal annotation.
Data for Human Evaluation There are 35 unique domain-generator combinations — five domains × seven generators (six LLMs plus humanwritten texts) for task 2 data. With the concern of annotators’ workload, we sample 4 examples for each combination, in total of 140 examples, split into four groups as Figure 1.
Without consideration of out-of-domain classification, we follow the best-performing setup: see all, classify all. If each annotator needs to check all domains, they need to learn 20 demonstration examples, which is a time consuming and laborious undertaking. Let us consider that each annotator annotates 3 domains. They would need to check 4 models × 3 domains × 4 examples = 48, which is
in a more reasonable range. Five demonstration examples are randomly se-
lected, to ensure that four classes are all included. Annotator background. Group 1, 2, 3, and 4 are annotated by native Italian, Chinese, English, and Russian speakers, respectively; they are either postdoc or PhD students in NLP.
Results Intuitively, it would be challenging to distinguish between generators based on 1-shot per class. Annotators may make decisions depending on the observations that generations by Cohere tend to be repetitive, Dolly-v2 has high-quality and coherent texts, while BLOOMz texts are more likely to contain a bunch of numbers or a bunch of “YES” or “NO” characters repetitively at the end of the texts, compared to other models. For GPT series, annotators distinguish them depending on their quality level assumption that GPT-4 > ChatGPT > davinci-003. They spot human-written texts based on formatting patterns (e.g., initial double newlines, initial space, completely missing new lines in the paragraph), typos, inconsistencies within texts, or specific references and URLs.
As shown in Table 4, the best distinction accuracy for humans is 27%, and the average accuracy
21% is less than random guess. This implies that it is extremely difficult for humans to learn patterns from five demonstration examples and then distinguish generations of different LLM generators (see the F1-score for each separate class in Appendix B).
5 Experiments
We conduct an experimental evaluation with several common baselines representing a diverse set of detection techniques: standard fine-tuning of an encoder-based Transformer model, elaborated feature engineering for the similar task, ranking-based features which form patterns for different LLMs. For classification tasks 1 and 2, we use five baseline classifiers suggested by Wang et al. (2023). They include fine-tuned RoBERTa and XLM-R classifiers, logistic regression with GLTR features (LR-GLTR) (Gehrmann et al., 2019a), SVM with Stylistic (Stylistic-SVM), and SVM with NELA features (NELA-SVM) (Horne et al., 2019). For the boundary detection task 3, we applied two detectors based on semantic features.
GLTR features These features are based on the assumption that in order to generate fluent and natural-looking text, most LLM decoding strategies sample high-probability tokens from the head of the distribution (Gehrmann et al., 2019a). Thus, word ranking information of an LM can be used to distinguish machine-generated texts from humanwritten. Following Wang et al. (2023), we select two categories of these features: (1) the number of tokens in the top-10, top-100, top-1000, and 1000+ ranks from the LM predicted probability distributions (4 features); and (2) the Frac(p) distribution over 10 bins ranging from 0.0 to 1.0 (10 features). Frac(p) describes the fraction of probability for the actual word divided by the maximum probability of any word at this position. We train a logistic regression model based on these 14 features to perform the binary classification task with the parameter of the maximum iteration = 1,000.
Stylistic and NELA features We use stylistic features including (1) character-based features, e.g., the number of characters, letters, special characters, etc., (2) syntactic features, e.g., a number of punctuation and function words, (3) structural features, e.g., a total number of sentences, and (4) word-based features, e.g., a total number of words, average word length, average sentence length, etc.
NEws LAndscape (NELA) features (Horne et al., 2019) involves six aspects: (1) style: the style and the structure of the article; (2) complexity: how complex the writing is; (3) bias: overall bias and subjectivity; (4) affect: sentiment and emotional patterns; (5) moral based on the Moral Foundation Theory (Graham et al., 2012), and (6) event: time and location. An SVM classifier is applied to perform the binary distinction.
5.1 Monolingual Human vs. Machine
For Task 1, we simulate a practical scenario where a detector have to deal with texts generated from a new LLM while it is trained on the outputs from
a limited number of other generators. A practical detector should be able to generalize over different generators and be robust on unseen inputs.
Experimental Setup We conduct experiments in two setups. (1) In the first setup, we select an LLM and combine its generations across all domains with the upsampled human-written texts to create a test set. Texts produced by other LLMs and humans are divided into training and development sets with a 4:1 ratio. (2) In the second setup, we take the data from all domains and generators and randomly split it into train and test sets (row All in Table 5). We train detectors with five different seeds for all experiments and present the mean values of the metrics.
Results and Analysis Classification results of five detectors are presented in Table 5. We analyze the results to answer three research questions: (1) For a given detector, on which unseen generator does it generalize best (highlighting F1-score and accuracy)? (2) Given a setting that training without the data of the generator to test, which detector performs the best (underlined Acc). (3) On average across both settings, which detector is the most accurate and has the best generalization capabilities? Which detector has higher recall than precision?
RoBERTa, XLM-R, and LR based on stylistic features give the best F1-score and accuracy on the setting where the classifier is trained with generations of all generators and tested across all generators. GLTR features perform the best in the setting where the detector is trained on data without generations from Cohere, and tested over Human vs. Cohere data, while NELA has the highest accuracy in distinguishing human vs. GPT-4 when it has not seen any data from GPT-4.
The results for RoBERTa and XLM-R suggest that such detectors encountered notable challenges when confronted with text from new generators not encompassed in their training data, exhibiting the large accuracy gap between the setting of All and others. This problem is especially conspicuous in the case of BLOOMz for all detectors (All vs. BLOOMz), followed by Dolly-v2, davinci-003, ChatGPT, GPT-4, and Cohere. XLMR and GLTR demonstrate better generalization performance (underlines numbers) over (davinci-003, Dolly-v2, and BLOOMz) and (ChatGPT, GPT-4, and Cohere) respectively.
XLM-R is the best on average. It has higher
accuracy and better generalization ability. LR with NELA features is the worst. Recall of RoBERTa, XLM-R, and GLTR is higher than precision, while it is the opposite for stylistic and NELA features. In the scenario of detecting misuse, we expect higher recall to avoid less misses of potentially malice.
Overall, detectors show poor generalization performance when testing on unseen generators. This is well-known due to exposure bias. The generations by BLOOMz are significantly different from other generators, leading to the result that when training on the data combination of other generators and testing on BLOOMz, the performance is extremely low for all detectors. The detector based on NELA features overall performs the worst, which demonstrates that the features used for fact verification are not suitable to serve as signals to distinguish human vs. machine text.
5.2 Multilingual Human vs. Machine Similar to monolingual setups, we evaluate detectors generalization on new languages (the language to test) and train on the combination of other languages. In the setting of All in Table 6, we merge data of all languages, and randomly split into train, validation and test sets.
Results and Analysis We categorize the performance of the detector (XLM-R) across ten configurations into three tiers based on the F1 score. (1) F1<50: Bulgarian, Russian and Indonesian; (2) 60<F1<85: English, German, Italian, Chinese; and (3) F1>90: Arabic, Urdu, All.
For the first level, Bulgarian and Russian are both Slavic languages, and Indonesian is a member of the Malayo-Polynesian branch of the language
family, training on other distantly-unrelated languages (Latin, Chinese and Arabic family) would naturally result in low performance. For the second level, Chinese is a high-resource language, so the detector could benefit from the learned patterns during XLM-R pretraining. For German and Italian, English training data will keep the same family languages in a medium performance, vice versa.
High recall and low precision with respect to MGT across all testing languages implies that the majority of examples are recognized as machinegenerated text. This leads to lower accuracy on human-written text than that on MGT (unweighted average macro-F1 < MGT F1).
Overall, the detector often predicts texts as MGT. when dealing with low-resource languages or training on language families that are distantly related, the accuracy tends to be moderate.
5.3 Multi-way Detection
Experimental Setup For task 2, the classifiers are trained to distinguish multiple generators (seven-label classification), including human, davinci-003, ChatGPT, GPT-4, Cohere, Dolly-v2, BLOOMz. We measure the capabilities of detectors to generalize on a new domain that was not presented during training. The detectors are trained on the whole dataset except the domain used for testing. For this task, we calculate accuracy, macro F1-score, and the class-wise F1-scores.
We experiment with six baselines: fine-tuned RoBERTa and XLM-R, logistic regression (one-vsrest) and SVM (one-vs-one) with GLTR features, and SVM (one-vs-one) with stylistic and NELA.
Results and Analysis We try to answer three questions: (1) given a detector, which domain is the hardest to predict; (2) which generator is the most challenging one to distinguish; and (3) which detector performs the best on task 2.
As results in Table 7, Transformer-based detectors show poor performance in this task when facing with text from new domains not included in their training data. Taking RoBERTa as an example, low accuracy is presented when testing on any unseen domains, such as arXiv and Wikipedia with accuracy of 36 and 52. Only when training over all domains and testing over all domains, accuracy reaches 97. The detectors exhibit significant challenges in generalizing to unfamiliar content, reflecting a notable weakness in their ability to handle previously unseen domains. The domain of
arXiv is the hardest for most detectors to identify if they are trained without arXiv examples.
For most domains, davinci-003 is the most challenging generator to distinguish, followed by ChatGPT. BLOOMz appears to be the easiest one, for all classifiers (see class-wise F1-scores in Appendix C). Even the worst detector NELA-SVM can identify BLOOMz with >90% F1-score. This may result from the fact that the generative distri-
bution of BLOOMz is significantly different from other generators, while the distribution between davinci-003 and ChatGPT are more similar to each other, making it hard to make distinctions.
RoBERTa is the best in this task followed by XLM-R. Based on the GLTR features, SVM onevs-one mode is better than logistic regression onevs-rest mode. Under the same SVM, GLTR features are more helpful than stylistic and NELA.
5.4 Boundary Identification Experimental Setup We perform evaluation for task 3 in two settings: (1) train on PeerRead and test on PeerRead (same or different generators); and (2) train on PeerRead and test on a new domain with multiple generators (test column in Table 3 with 11,123 examples).
As a baseline, we use a sequence tagger based on Longformer (Beltagy et al., 2020) considering the long context (> 1,024 tokens) and DeBERTav3 (He et al., 2021), where the position of the first word that is predicted as machine-generated is our prediction. The label was predicted at the token level: human-written words are tagged as 0 and machine-generated words are tagged as 1. We remap the tagged tokens back to the word level and calculate the MAE. We run experiments three times with different random seeds and calculate the mean and standard deviation.
Results and Analysis In Table 8, when training on generations from LLaMA-2 PeerRead and testing on ChatGPT PeerRead and vice versa, even within the same domain, predicted boundary shows large gap with the gold label. This reveals that it is much harder for boundary detection models to predict changing points for unseen generators.
Testing on an unseen domain OUTFOX with mixture of generators including ChatGPT, GPT-4 and LLaMA-2 series, results in worse MAE, i.e., column all test. Taking the setting of All as an example, MAE changes from <5 (column
PeerRead LLaMA-2-7B* and PeerRead ChatGPT) to MAE>21 and >15 (column all test) respectively for Longformer and Deterta-v3. Particularly for Longformer trained with LLaMA-2 data, MAE is greater than 53 when testing on all test, demonstrating challenges on unseen domains and generators, consistent with findings in task 1 and 2.
6 Conclusion and Future Direction
In this work, we extend M4 into a new benchmark M4GT-Bench for MGT detection, providing a wide coverage of nine languages, six domains, and nine state-of-the-art generators, including GPT-4 and LLaMA-2 series. We formulate three tasks, with a novel formulation of detecting the changing point from human-written to machine-generated. We observe that human readers are unable to consistently differentiate between generators and operate merely at the level of making random guesses.
For all tasks, we find that the detector usually suffers a severe penalty in their performance when they are challenged with unseen domains or generators. Some generators such as BLOOMz and domains like arXiv appear harder than others. Overall, Transformer-based detectors such as RoBERTa and XLM-R are usually the best on average.
In future work, we plan to develop a demonstration system assisting human to detect MGT, especially in some high-stake domains such as clinical, financial and legal domains, and distinguish the authorship of different LLMs, promoting the protection of intelligence property. In addition to text, we would extend to other modalities detection including speech, image and videos. From the perspective of adversarial attacks such as paraphrasing, it is worthwhile to explore mixed texts with multiple changing points.
Limitations
While we provide strong baselines and analyze their performance in various evaluation settings, certain limitations remain for future work. Firstly, the nature of supervised multi-way classification of different LLMs disables the distinction of unseen generators, and tends to fall into random guess once the distribution of training and test data differs. Also, it is vulnerable to language style attack such as paraphrasing in different tones, back-translation and other textual adversarial attacks. Methods based on watermark and few-shot in-content learning are more promising for this task.
Secondly, for all tasks, current black-box approaches may be less effective and may demonstrate poor generalization for unseen domains, generators, and languages, and this suggests the need to study more general methods to improve the detection and the potential misuse of LLMs.
Thirdly, we simplify the task of boundary identification by assuming that the presented text is a mixture of first human-written segments and then machine-generated segments with only one changing point. Practical scenarios are more complicated. We should first detect whether it is a mixed text, and further identify all changing points.
Ethics and Broader Impact
We discuss some potential ethical concerns about the present work.
Data Collection and Licenses We used preexisting corpora: M4 and OUTFOX that have been publicly released and approved for research purposes, with clear dataset licenses. Data licenses for multilingual dataset are listed in Table 2.
Security Implication The M4GT-Bench is intended for the development of a robust MGT detection system to identify and mitigate misuse, such as blocking the spread of automated misinformation campaigns, safeguarding individuals and institutions from potential financial losses. For domains of journalism, academia, and legal processes, the authenticity of information is paramount MGT detection ensures the integrity of content in these fields, thereby preserving trust. Also, users become more aware of LLMs capabilities, a growing skepticism towards digital content happens. Effective MGT detection can alleviate these concerns, ensuring users can trust LLM generations.
References Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi
Yang, and Yue Zhang. 2023. Fast-detectgpt: Efficient zero-shot detection of machine-generated text via conditional probability curvature. arXiv preprint arXiv:2310.05130.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. CoRR, abs/2004.05150.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116.
Evan Crothers, Nathalie Japkowicz, Herna Viktor, and Paula Branco. 2022. Adversarial robustness of neural-statistical features in detection of generative transformers. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE.
Liam Dugan, Daphne Ippolito, Arun Kirubarajan, and Chris Callison-Burch. 2020. RoFT: A tool for evaluating human detection of machine-generated text. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 189–196, Online. Association for Computational Linguistics.
Liam Dugan, Daphne Ippolito, Arun Kirubarajan, Sherry Shi, and Chris Callison-Burch. 2023. Real or fake text?: Investigating human ability to detect boundaries between human-written and machinegenerated text. In Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pages 12763–12771. AAAI Press.
Chujie Gao, Dongping Chen, Qihui Zhang, Yue Huang, Yao Wan, and Lichao Sun. 2024. Llm-as-a-coauthor: The challenges of detecting llm-human mixcase. arXiv preprint arXiv:2401.05952.
Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. 2019a. GLTR: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 111–116, Florence, Italy. Association for Computational Linguistics.
Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. 2019b. Gltr: Statistical detection and visualization of generated text. arXiv preprint arXiv:1906.04043.
Katy Ilonka Gero, Vivian Liu, and Lydia Chilton. 2022. Sparks: Inspiration for science writing using language models. In Designing interactive systems conference, pages 1002–1019.
Jesse Graham, Jonathan Haidt, Sena Koleva, Matt Motyl, Ravi Iyer, Sean Wojcik, and Peter Ditto. 2012.
Moral foundations theory: The pragmatic validity of moral pluralism. Advances in Experimental Social Psychology, 47.
Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. CoRR, abs/2301.07597.
Abhimanyu Hans, Avi Schwarzschild, Valeriia Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2024. Spotting llms with binoculars: Zero-shot detection of machine-generated text. arXiv preprint arXiv:2401.12070.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: decoding-enhanced bert with disentangled attention. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Xinlei He, Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. 2023. Mgtbench: Benchmarking machine-generated text detection. CoRR, abs/2303.14822.
Benjamin D Horne, Jeppe Nørregaard, and Sibel Adali. 2019. Robust fake news detection over time and attack. ACM Transactions on Intelligent Systems and Technology (TIST), 11(1):1–23.
Xiaomeng Hu, Pin-Yu Chen, and Tsung-Yi Ho. 2023. Radar: Robust ai-text detection via adversarial learning. arXiv preprint arXiv:2307.03838.
Daphne Ippolito, Daniel Duckworth, Chris CallisonBurch, and Douglas Eck. 2019. Automatic detection of generated text is easiest when humans are fooled. arXiv preprint arXiv:1911.00650.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. A watermark for large language models. CoRR, abs/2301.10226.
Ryuto Koike, Masahiro Kaneko, and Naoaki Okazaki. 2023. Outfox: Llm-generated essay detection through in-context learning with adversarially generated examples. arXiv preprint arXiv:2307.11729.
Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. 2023. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense. arXiv preprint arXiv:2303.13408.
Tharindu Kumarage, Joshua Garland, Amrita Bhattacharjee, Kirill Trapeznikov, Scott Ruston, and Huan Liu. 2023. Stylometric detection of aigenerated text in twitter timelines. arXiv preprint arXiv:2303.03697.
Jenny S Li, John V Monaco, Li-Chiou Chen, and Charles C Tappert. 2014. Authorship authentication using short messages from social networking sites. In 2014 IEEE 11th International Conference on e-Business Engineering, pages 314–319. IEEE.
Xiaoming Liu, Zhaohan Zhang, Yichen Wang, Hang Pu, Yu Lan, and Chao Shen. 2022. Coco: Coherenceenhanced machine-generated text detection under data limitation with contrastive learning. arXiv preprint arXiv:2212.10341.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Dominik Macko, Robert Moro, Adaku Uchendu, Ivan Srba, Jason Samuel Lucas, Michiharu Yamashita, Nafis Irtiza Tripto, Dongwon Lee, Jakub Simko, and Maria Bielikova. 2024. Authorship obfuscation in multilingual machine-generated text detection. arXiv preprint arXiv:2401.07867.
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. 2023. Detectgpt: Zero-shot machine-generated text detection using probability curvature. CoRR, abs/2301.11305.
Shaoor Munir, Brishna Batool, Zubair Shafiq, Padmini Srinivasan, and Fareed Zaffar. 2021. Through the looking glass: Learning to attribute synthetic text generated by language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1811–1822.
Rafael Rivera Soto, Kailin Koch, Aleem Khan, Barry Chen, Marcus Bishop, and Nicholas Andrews. 2024. Few-shot detection of machine-generated text using style representations. arXiv e-prints, pages arXiv– 2401.
Zhouxing Shi, Yihan Wang, Fan Yin, Xiangning Chen, Kai-Wei Chang, and Cho-Jui Hsieh. 2023. Red teaming language model detectors with language models. arXiv preprint arXiv:2305.19713.
Lei Shu, Liangchen Luo, Jayakumar Hoskere, Yun Zhu, Canoee Liu, Simon Tong, Jindong Chen, and Lei Meng. 2023. Rewritelm: An instruction-tuned large language model for text rewriting. arXiv preprint arXiv:2305.15685.
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. 2019. Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203.
Jinyan Su, Terry Yue Zhuo, Di Wang, and Preslav Nakov. 2023. Detectllm: Leveraging log rank information for zero-shot detection of machine-generated text. arXiv preprint arXiv:2306.05540.
Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. 2020. Authorship attribution for neural text generation. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 8384–8395.
Adaku Uchendu, Zeyu Ma, Thai Le, Rui Zhang, and Dongwon Lee. 2021. Turingbench: A benchmark
environment for turing test in the age of neural text generation. arXiv preprint arXiv:2109.13296.
Saranya Venkatraman, Adaku Uchendu, and Dongwon Lee. 2023. Gpt-who: An information density-based machine-generated text detector. arXiv preprint arXiv:2310.06202.
Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. 2021. Adversarial glue: A multitask benchmark for robustness evaluation of language models. arXiv preprint arXiv:2111.02840.
Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Alham Fikri Aji, et al. 2023. M4: Multigenerator, multi-domain, and multi-lingual black-box machine-generated text detection. arXiv preprint arXiv:2305.14902.
Zhuohan Xie, Trevor Cohn, and Jey Han Lau. 2023. The next chapter: A study of large language models in storytelling. In Proceedings of the 16th International Natural Language Generation Conference, pages 323–351.
Feng Xiong, Thanet Markchom, Ziwei Zheng, Subin Jung, Varun Ojha, and Huizhi Liang. 2024. Finetuning large language models for multigenerator, multidomain, and multilingual machine-generated text detection. arXiv preprint arXiv:2401.12326.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 9051–9062.
Xuandong Zhao, Prabhanjan Ananth, Lei Li, and Yu-Xiang Wang. 2023a. Provable robust watermarking for ai-generated text. arXiv preprint arXiv:2306.17439.
Xuandong Zhao, Yu-Xiang Wang, and Lei Li. 2023b. Protecting language generation models via invisible watermarking. CoRR, abs/2302.03162.
Wanjun Zhong, Duyu Tang, Zenan Xu, Ruize Wang, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2020. Neural deepfake detection with factual structure of text. arXiv preprint arXiv:2010.07475.
Appendix
A Task 3 Template Prompt
Figure 2 provides template prompts used for task 3, instructing LLMs to continue to write peer reviews and student essays with human-written part as the context.
B Human Evaluation Results
Table 9 presents the overall performance and generator-wise accuracy for human evaluation on task 2.
C Results of Multi-way Classification
Table 10 provides the overall performance and class-wise (generator) F1-scores.
D Examples of Task 1 and 2
For English, we exhibit two examples from GPT-4-arXiv and GPT-4-OUTFOX. For Arabic, we present two examples generated by ChatGPT and Jais-30B respectively based on news. Afterwards, two examples generated by ChatGPT for German news and Wikipedia are demonstrated, followed by an Italian example produced by a fine-tuned LLaMA-2-70B using Italian corpus, the model named CAMOSCIO-70B.
Field Content
Language English LLM Generator GPT-4 Domain arXiv
Title The Spitzer c2d Survey of Large, Nearby, Insterstellar Clouds. IX. The Serpens YSO Population As Observed With IRAC and MIPS
Human We discuss the results from the combined IRAC and MIPS c2d Spitzer Legacy observations of the Serpens star-forming region. In particular we present a set of criteria for isolating bona fide young stellar objects, YSO’s, from the extensive background contamination by extra-galactic objects. We then discuss the properties of the resulting high confidence set of YSO’s. We find 235 such objects in the 0.85 deg2̂ field that was covered with both IRAC and MIPS. An additional set of 51 lower confidence YSO’s outside this area is identified from the MIPS data combined with 2MASS photometry. We describe two sets of results, color-color diagrams to compare our observed source properties with those of theoretical models for star/disk/envelope systems and our own modeling of the subset of our objects that appear to be star+disks. These objects exhibit a very wide range of disk properties, from many that can be fit with actively accreting disks to some with both passive disks and even possibly debris disks. We find that the luminosity function of YSO’s in Serpens extends down to at least a few x .001 Lsun or lower for an assumed distance of 260 pc. The lower limit may be set by our inability to distinguish YSO’s from extra-galactic sources more than by the lack of YSO’s at very low luminosities. A spatial clustering analysis shows that the nominally less-evolved YSO’s are more highly clustered than the later stages and that the background extra-galactic population can be fit by the same two-point correlation function as seen in other extra-galactic studies. We also present a table of matches between several previous infrared and X-ray studies of the Serpens YSO population and our Spitzer data set.
Prompt Write an abstract of an article from ArXiv given its title so that it is indistinguishable from a genuine human-written abstract. Write in the style that most abstracts from ArXiv are written. Title: The Spitzer c2d Survey of Large, Nearby, Insterstellar Clouds. IX. The Serpens YSO Population As Observed With IRAC and MIPS. Abstract:
Machine This paper presents the results of the Spitzer c2d survey of the Serpens star-forming region, focusing on the young stellar object (YSO) population as observed with the Infrared Array Camera (IRAC) and the Multiband Imaging Photometer for Spitzer (MIPS). The survey identified a significant number of YSO candidates, providing a comprehensive view of star formation in this region. The data obtained from IRAC and MIPS observations were used to construct color-color and color-magnitude diagrams, which were then utilized to classify the YSOs into different evolutionary stages. The spatial distribution of these objects was also analyzed, revealing a correlation with the dense molecular gas in the region. The results contribute to our understanding of the star formation process in large, nearby, interstellar clouds and provide a valuable resource for future studies.
Language Arabic LLM Generator CHATGPT Domain News Title ©k. @Q K Q ®k H@ñ̄ð ÊK. @Q¢ . XYj. J K ¼PAªÖÏ @ . . AJ . J Ë úÍ@ PñJ.ªÊË é JK QëAg. áÊªK ú » Q Ë @ m.Ì'@
Human QK PA ® JË @ HXA ̄ @ð , ÊK. @Q£ éJ . J ÊË @ éÖÞ AªË@ ¡J jÖß. éªÒm.Ì'@ ÐñJ Ë @ ¼PAªÖÏ @ HXYm.
' m.Ì'@ Y» @ á g ú ̄ , Aî EXĀ PQK. @ Yg @ É J ®Óð Q ®k é ®J Ê g Y«A ® JÖÏ @ Z @ñÊË @ H@ñ̄ ©k. @Q K.
éÓñºk áÓ ø Qº « PY Ó XA ̄ @ Ȳð . HAÒJ Êª JË @ éJ ®Ê K Pñ ̄ AJ . J ÊË ék. ñ JÊË é JK QëAg. ú »
Q Ë @ H@ñ̄ HXAª J @ð . ÊK. @Q£ ú G. ñ Jk. ÈA J ®Ë @ PðAm× I. Ê « AK. éjÊ ÖÏ @ HA¿ AJ. B@ XYj. JK. A ̄ñË @
Qº ªÓ Ê g úÍ@ H. Aj B@ úÎ« Q ®k H@ñ̄ HQ .g. @ð , K. A Īð ú ̄ Aî EQå k ©̄ @ñÓ A ̄ñË @ ú ̄ éJ ÓAÓ B@  ñ¢ mÌ'@ ú Î KA ®Ó à @ A ̄ñË @ H@ñ̄ áÓ PXA Ó HQ» Xð . ÊK. @Q£ ¡J jÖß. ¼ñÓQ Ë @
Y éJ ËA J̄ HAJ ÊÔ« àð Y ® JK áK YË @ , éJ ðQË@ \\Q «A ̄ "" é»Qå ú jÊ Ó áÓ @ñ KA¿ Q ®k H@ñ̄
H@ñ ®Ë éJ Ë @ñÓ ÐC«@ É KA ð HY» @ð . é JK Yg éJ «ñ K éjÊ @ Ð@Y j J AK. A ̄ñË @ éÓñºk H@ñ̄
. ú GC ÖÏ @ YÒm× ú j J ̄ ú«YK ð á»P YJ ®« éJ. KP ÉÒm ' @ PPAK. AK Qº « @Y KĀ HY ® ̄ Aî E @ Q ®k
Field Content éÖÞ AªË@ H. ñ Jk. ¼ñÓQ Ë @ Pñm× ú ̄ ú kðPA  ®K. éªÒm.Ì'@ ÐñJ Ë @ é«Qå Ó ú GC ÖÏ @ ù ®Ëð
H@ñ ®Ë ©K. A JË @ 106 Z@ñÊËAK. HA«PYÖÏ @ð HAK. AK. YË @ hC Xñ ®K ú GC ÖÏ @ àA¿ð . ÊK. @Q£
¡J jÖß. éK Qº « H@ Q»QÖ ß éªÒm.Ì'@ ÐñJ Ë @ Q ®k H@Q KA£ I ̄ Yî D @ , é K @ X AJ Ë@ ú ̄ .Q ®k H@PA « HC JË Ò mÌ'@ Ó
@ I Qª K Ȳ é JK YÖÏ @ I KA¿ð . XCJ. Ë @ ú G. Q « ÈAÖÞ éK ð@ QË @ èA ® Ó
úÎ« t ' P@ñ  ñ ® éj. J K áK Q k @ éJ K AÖ ß hQk. ð á J K @ á J KYÓ É J ®Ó á« Aë@Yg@ HQ ® @ éK ñk.
XY J K èQ Ó é JK YÖÏ @ ú ̄ Ik. Q k éªÒm.Ì'@ ÐñJ Ë @ð . éK ð@ QË @ ú ̄ ú ÎgA Ë@ K Q¢ËAK. èY« HCm×
. á J KYÖÏ @ áÓ AK Am  ñ ® úÍ@ ø X ñ Kð éJ ¢ ® JË @ H A ÖÏ @ ¬Yî D ú æË @ Q ®k H@ñ̄ H@PA ªK. @ðXY Kð , ÊK. @Q£ éÖÞ AªË@ úÎ« Q ®k H@ñ̄ é J ø
YË@ Ðñj. êÊË ÑîD ̄P á« àðQëA ¢ JÖÏ @ Q .«ð ú ̄ éK Qº « PXA Ó HY»
@ , AîD. K Ag. áÓQÓ@ð B@ PA ¢ J K @ ú ̄ . éË éÔ«@YË@ ÈðYË@ ̄ @ñÖß. ÈAg AêÓAêÖß. ÐAJ ®Ë @ð AJ . J Ë úÍ@ ék. ñ JÊË èYª J Ó éJ »Q Ë @ éjÊ ÖÏ @ H@ñ ®Ë@ à @ èQ ® K @
@YK XA K ¨A ̄ YË @ èP@ Pð Õæ AK. é KYj JÒÊË ù ®m QÖ ß ñÓ I.
®« ½Ë X ZAg. . HAÒJ Êª JË @ AîD ®Ê K úÍ@ ék. ñ JË @ á«ð .2019 ÐAªË@ ÈC g èP@ PñË@ é¢ B AÒJ J ® K éJ ̄ IÓȲ H. ñ J» @ Õ æJ.
ÐAêÖÏ @ ¨@ñ K @ é ̄ A¾K. ÐAJ ®ÊË èYª J Ó éJ »Q Ë @ éjÊ ÖÏ @ H@ñ ®Ë@ à@ PXA ÖÏ @ IËA ̄ , AJ . J Ë I. J £ I. k. P ú » Q Ë @ KQË @ ©̄ñ K , Ò mÌ'@ Ó @ð . HAÒJ Êª JË @ AîD ®Ê K ÈAg AJ »Q K h. PA gð É g@X
ú GA JË @ QK A JK áÓ áÓA JË @ ú ̄ àAÖÏQ . Ë @ áÓ ñ ® K úÎ« é JÓñºk Èñ k àA «ðXP @
èXCK. à @ Y» @ð . éJ «Qå Ë @ éÓñºmÌ'@ èñ«YË éJ J. Ê K , AJ . J Ë úÍ@ Xñ Jk. ÈA P@ Ég. @ áÓ ÉJ. ®ÖÏ @
é ®J Ê g ú G. C ® KB@ È@Q m.Ì'@ Y AêkA ®» ú ̄ ÊK. @Q£ éÓñºmÌ Ñ«YË@ ¨@ñ K @ ©J Ôg. ÐY
® J YJ ® J JË @ Q g IÊ gX Ò mÌ'@ Qm. ̄ ð . é ®Ê J m× éJ K. Q«ð éJ K. ðPð @ ÈðX áÓ Ðñ«YÖÏ @ ,Q ®k
ú ̄ AëQå YªK. AJ . J Ëð AJ »Q K á K. ø Qº ªË@ð ú æÓ B@ àðAª JËAK. é A mÌ'@ ÑëA ® JË @ èQ»
YÓ @ PQË@ YJ.« AJ »Q K ú ̄ ú æ. J ÊË @ Q ® Ë@ ÈĀ , K. A Īð ú
̄ ð . éJ »Q Ë @ éJ ÖÞ QË @ èYK Qm.Ì'@ , A mÌ'@ I. K PY JË @ð ø ñm.Ì'@ ¨A ̄ YË @ ú ̄ AK Qº « AÔ«X h. A Jm ' èXCK. à@ PXA ®Ë @ YJ.« PA J m×
Ñ«X úÎ« Èñ jÊË èQ ® K @ úÍ@ AJ ÖÞ P AJ. Ê£ A ̄ñË @ éÓñºk Õç' Y ® K á« ZAJ. K @ É £ ú ̄ ½Ë Xð éjÊ Ó ¼PAªÓ ÊK. @Q£ ¡J m× YîD , ú æ AÖÏ @ ÉK QK. @ 4 Y JÓð .ø ñk. ð ø QK. ð ø Qm '. ø Qº «
.Q ®k é ®J Ê g Y«A ® JÖÏ @ È@Q m.Ì'@ H@ñ̄ð AJ ËðX AîE. ¬Q ªÖÏ @ A ̄ñË @ éÓñºk H@ñ̄ á K.
Field Content
Prompt Given the following details for an Arabic news Article, generate the content in Arabic.
Make sure the content is around 385 words.
Title:
©k. @Q K Q ®k H@ñ̄ð ÊK. @Q¢ . XYj. J K ¼PAªÖÏ @ . . AJ . J Ë úÍ@ PñJ.ªÊË é JK QëAg. áÊªK ú » Q Ë @ m.Ì'@
Description: H@ñ̄ ©k. @Q K. QK PA ® JË @ HXA ̄ @ð , ÊK. @Q£ éJ . J ÊË @ éÖÞ AªË@ ¡J jÖß. éªÒm.Ì'@ ÐñJ Ë @ ¼PAªÖÏ @ HXYm. '
AJ . J ÊË ék. ñ JÊË é JK QëAg. ú » Q Ë @ m.Ì'@ Y» @ AÒ J K. , Aî EXĀ PQK. @ Yg @ É J ®Óð Q ®k é ®J Ê g Y«A ® JÖÏ @ Z @ñÊË @
. HAÒJ Êª JË @ éJ ®Ê K Pñ ̄
Ref: HBA¿ð + èQK Qm.Ì'@ : PY ÖÏ @
Content:
Machine iÊ ÖÏ @ ¨@Qå Ë@ QÒ J IJ k , AJ . J ÊK. ÊK. @Q£ é ®¢ JÓ ú ̄ ¼PAªÖÏ @ XYj. JK. éªÒm.Ì'@ ÐñJ Ë @ QK PA ® JË @ HXA ̄ @
©k. @Q K J Kñ K Õç ' Ȳð .Q ®k é ®J Ê g Y«A ® JÖÏ @ Z @ñÊË @ H@ñ̄ð ú æ£ñË@ A ̄ñË @ éÓñºmÌ éJ Ë @ñÖÏ @ H@ñ ®Ë@ á K.
. ¼PAªÖÏ @ ÈC g áK PPAJ. Ë @ Aî EXĀ Yg @ É J ®Ó á« QK PA ® K ©Ó ,Q ®k H@ñ̄
@ Pñ ̄ I. J j. é K @ úÍ@ @ Q Ó , AJ . J Ë úÍ@ ék. ñ JÊË é JK QëAg. ú » Q Ë @ m.Ì'@ áÊ« @ , AJ Ë@ @ Yë ú ̄
éJ JÓ B@ ¨A ð B@ Ð P A Kð é ®¢ JÖÏ @ ú ̄ Q Kñ JË @ Y«A É £ ú ̄ àC«B @ @ Yë ú G AK ð . AëA ®Ê JK HAÒJ Êª K ø B
. AJ . J Ë ú ̄
, AJ . J Ë ú ̄ ú æ£ñË@ A ̄ñË @ éÓñºk Ñ«YË éJ »Q Ë @ éÓñºmÌ'@ ú «A Ó PA£@ ú ̄ ú » Q Ë @ m.Ì'@ èñ¢ k ú G A K HBA B@ J Jº K Õç ' Ȳð . XCJ. Ë @ ú ̄ áÓ B@ð P@Q ® J B@ éK AÒmÌ ùª ð Q ®k H@ñ̄ É KA ® K ú
æË @ . ú æ AJ Ë@
J JË @ð éK Qº ªË@ HĀCªË@ QK Qª JË á J. K Am.Ì'@ á K. J JË @ð A K. Qk XCJ. Ë @ YîD IJ k , AJ . J Ë ú ̄ èQ g B@ H@Pñ¢ JË @ Z @ P @ YK Y
Ê ®K. éJ »Q Ë @ éÓñºmÌ'@ Qª ð J ®m 'ð éÓ P B@ Ég ú ̄ A J KP @ PðX I. ªÊ K à @ AJ »Q K ÉÓ A Kð .2019 ÉK QK.
@ Y JÓ èQª J Ó éK ñÓX . XCJ. Ë @ ú
̄ P@Q ® J B@ áÓ áºËð , ú » Q Ë @ àAÖÏQ . Ë @ é ® ̄ @ñÓ ú «Y J AJ . J Ë úÍ@ ú » Q Ë @ m.Ì'@ Èñ kX à @ úÍ@
èPA B @ I. m. ' à AK. àñJ »Q K àñËð ñ Ó hQå Ȳð . èñ¢ mÌ'@ è Yë YJ ® J JË éJ Kñ KĀ K@ñ« ø @ PðAm. ' Õ æK à @ ©̄ñ JÖÏ @
. XCJ. Ë @ ú ̄ áÓ B@ àAÖÞ ð AJ . J Ë ú ̄ AêmÌ'A Ó éK AÒmÌ éÓ PCË@ H@Z@Qk. B @ é ̄ A¿ Y j J K ÑëXCK.
ÐC Ë@ J ®m ' Ég. @ áÓ ú ÍðYË@ ©Ò Jj. ÖÏ @ ©Ó àðAª JÊË èYª J Ó AJ »Q K Q . Jª K , É JÓ YJ ª úÎ« P@ñmÌ'@ úÍ@ èXñªË@ð éK Qº ªË@ HAJ ÊÒªË@ ̄ñK. éJ »Q Ë @ éÓñºmÌ'@ I. ËA¢ ð . AJ . J Ë ú
̄ P@Q ® J B@ð . éK QmÌ'@ð ÐC ÊË ú æ. J ÊË @ I. ª Ë@ HAªÊ¢ éJ J. Ê Kð ù £@Q ®Öß YË@ ÈA ® J KB@ J ®j JË , ú æ AJ Ë@
Field Content
Language Arabic LLM Generator JAIS Domain News Title ©k. @Q K Q ®k H@ñ̄ð ÊK. @Q¢ . XYj. J K ¼PAªÖÏ @ . . AJ . J Ë úÍ@ PñJ.ªÊË é JK QëAg. áÊªK ú » Q Ë @ m.Ì'@
Human Z @ñÊË @ H@ñ̄ ©k. @Q K. QK PA ® JË @ HXA ̄ @ð , ÊK. @Q£ éJ . J ÊË @ éÖÞ AªË@ ¡J jÖß. éªÒm.Ì'@ ÐñJ Ë @ ¼PAªÖÏ @ HXYm. '
AJ . J ÊË ék. ñ JÊË é JK QëAg. ú » Q Ë @ m.Ì'@ Y» @ á g ú ̄ , Aî EXĀ PQK. @ Yg @ É J ®Óð Q ®k é ®J Ê g Y«A ® JÖÏ @ éjÊ ÖÏ @ HA¿ AJ. B@ XYj. JK. A ̄ñË @ éÓñºk áÓ ø Qº « PY Ó XA ̄ @ Ȳð . HAÒJ Êª JË @ éJ ®Ê K Pñ ̄ , K. A Īð ú ̄ Aî EQå k ©̄ @ñÓ A ̄ñË @ H@ñ̄ HXAª J @ð . ÊK. @Q£ ú G. ñ Jk. ÈA J ®Ë @ PðAm× I. Ê « AK. PXA Ó HQ» Xð . ÊK. @Q£ ¡J jÖß. ¼ñÓQ Ë @ Qº ªÓ Ê g úÍ@ H. Aj B@ úÎ« Q ®k H@ñ̄ HQ .g. @ð Q «A ̄ @ é»Qå ú jÊ Ó áÓ @ñ KA¿ Q ®k H@ñ̄ ú ̄ éJ ÓAÓ B@  ñ¢ mÌ'@ ú Î KA ®Ó à @ A ̄ñË @ H@ñ̄ áÓ
. é JK Yg éJ «ñ K éjÊ @ Ð@Y j J AK. A ̄ñË @ éÓñºk H@ñ̄ Y éJ ËA J̄ HAJ ÊÔ« àð Y ® JK áK YË @ , éJ ðQË@ á»P YJ ®« éJ. KP ÉÒm ' @ PPAK. AK Qº « @Y KĀ HY ® ̄ Aî E @ Q ®k H@ñ ®Ë éJ Ë @ñÓ ÐC«@ É KA ð HY» @ð Pñm× ú ̄ ú kðPA  ®K. éªÒm.Ì'@ ÐñJ Ë @ é«Qå Ó ú GC ÖÏ @ ù ®Ëð . ú GC ÖÏ @ YÒm× ú j
J ̄ ú«YK ð 106 Z@ñÊËAK. HA«PYÖÏ @ð HAK. AK. YË @ hC Xñ ®K ú
GC ÖÏ @ àA¿ð . ÊK. @Q£ éÖÞ AªË@ H. ñ Jk. ¼ñÓQ Ë @ éK Qº « H@ Q»QÖ ß éªÒm.Ì'@ ÐñJ Ë @ Q ®k H@Q KA£ I ̄ Yî D @ , é K @ X AJ Ë@ ú ̄ .Q ®k H@ñ ®Ë ©K. A JË @ H@PA « HC JË Ò mÌ'@ Ó @ I Qª K Ȳ é JK YÖÏ @ I KA¿ð . XCJ. Ë @ ú G. Q
« ÈAÖÞ éK ð@ QË @ èA ® Ó ¡J jÖß. úÎ« t ' P@ñ  ñ ® éj. J K áK Q k @ éJ K AÖ ß hQk. ð á J K @ á J KYÓ É J ®Ó á« Aë@Yg@ HQ ® @ éK ñk. H@PA ªK. XY J K èQ Ó é JK YÖÏ @ ú ̄ Ik. Q k éªÒm.Ì'@ ÐñJ Ë @ð . éK ð@ QË @ ú ̄ ú ÎgA Ë@ K Q¢ËAK. èY« HCm×
Q .«ð . á J KYÖÏ @ áÓ AK Am  ñ ® úÍ@ ø X ñ Kð éJ ¢ ® JË @ H A ÖÏ @ ¬Yî D ú æË @ Q ®k H@ñ̄ ̄ @ñÖß. @ðXY Kð , ÊK. @Q£ éÖÞ AªË@ úÎ« Q ®k H@ñ̄ é J ø
YË@ Ðñj. êÊË ÑîD ̄P á« àðQëA ¢ JÖÏ @ H@ñ ®Ë@ à @ èQ ® K @ ú
̄ éK Qº « PXA Ó HY» @ , AîD. K Ag. áÓQÓ@ð B@ PA ¢ J K @ ú ̄ . éË éÔ«@YË@ ÈðYË@ I. ®« ½Ë X ZAg. . HAÒJ Êª JË @ AîD ®Ê K ÈAg AêÓAêÖß. ÐAJ ®Ë @ð AJ . J Ë úÍ@ ék. ñ JÊË èYª J Ó éJ »Q Ë @ éjÊ ÖÏ @ èP@ PñË@ é¢ B AÒJ J ® K éJ ̄ IÓȲ H. ñ J» @ Õ æJ. @YK XA K ¨A ̄ YË @ èP@ Pð Õæ AK. é KYj JÒÊË ù
®m QÖ ß ñÓ ÐAJ ®ÊË èYª J Ó éJ »Q Ë @ éjÊ ÖÏ @ H@ñ ®Ë@ à@ PXA ÖÏ @ IËA ̄ , AJ . J Ë úÍ@ ék. ñ JË @ á«ð .2019 ÐAªË@ ÈC g ú » Q Ë @ KQË @ ©̄ñ K , Ò mÌ'@ Ó
@ð . HAÒJ Êª JË @ AîD ®Ê K ÈAg AJ »Q K h. PA gð É g@X ÐAêÖÏ @ ¨@ñ K @ é ̄ A¾K.
ÉJ. ®ÖÏ @ ú G A JË @ QK A JK áÓ áÓA JË @ ú ̄ àAÖÏQ . Ë @ áÓ ñ ® K úÎ« é JÓñºk Èñ k àA «ðXP @ I. J £ I. k. P
¨@ñ K @ ©J Ôg. ÐY ® J èXCK. à @ Y» @ð . éJ «Qå Ë @ éÓñºmÌ'@ èñ«YË éJ J. Ê K , AJ . J Ë úÍ@ Xñ Jk. ÈA P@ Ég. @ áÓ éJ K. ðPð @ ÈðX áÓ Ðñ«YÖÏ @ ,Q ®k é ®J Ê g ú G. C ® KB@ È@Q m.Ì'@ Y AêkA ®» ú ̄ ÊK. @Q£ éÓñºmÌ Ñ«YË@ ø Qº ªË@ð ú æÓ B@ àðAª JËAK. é A mÌ'@ ÑëA ® JË @ èQ» YÓ YJ ® J JË @ Q g IÊ gX Ò mÌ'@ Qm. ̄ ð . é ®Ê J m× éJ K. Q«ð AJ »Q K ú ̄ ú æ. J ÊË @ Q ® Ë@ ÈĀ , K. A Īð ú ̄ ð . éJ »Q Ë @ éJ ÖÞ QË @ èYK Qm.Ì'@ ú
̄ AëQå YªK. AJ . J Ëð AJ »Q K á K. , A mÌ'@ I. K PY JË @ð ø ñm.Ì'@ ¨A ̄ YË @ ú ̄ AK Qº « AÔ«X h. A Jm ' èXCK. à@ PXA ®Ë @ YJ.« PA J m× @ PQË@ YJ.«
ø Qº « Ñ«X úÎ« Èñ jÊË èQ ® K @ úÍ@ AJ ÖÞ P AJ. Ê£ A ̄ñË @ éÓñºk Õç' Y ® K á« ZAJ. K @ É £ ú ̄ ½Ë Xð H@ñ̄ á K. éjÊ Ó ¼PAªÓ ÊK. @Q£ ¡J m× YîD , ú æ AÖÏ @ ÉK QK. @ 4 Y JÓð .ø ñk. ð ø QK. ð ø Qm '.
.Q ®k é ®J Ê g Y«A ® JÖÏ @ È@Q m.Ì'@ H@ñ̄ð AJ ËðX AîE. ¬Q ªÖÏ @ A ̄ñË @ éÓñºk
Field Content
Prompt éJ K. QªË@ é ªÊËAK. øñ JjÖÏ @ ZA A K. Õ ̄ , éJ K. Q« PAJ. k @ éËA ®ÖÏ éJ ËA JË @ ÉJ A ® JË @ úÎ« ZA JK.
. HAÒÊ¿ 385 Èñk hð@Q K øñ JjÖÏ @ à @ áÓ Y» A K.
Q ®k H@ñ̄ð ÊK. @Q¢ . XYj. J K ¼PAªÖÏ @ . . AJ . J Ë úÍ@ PñJ.ªÊË é JK QëAg. áÊªK ú » Q Ë @ m.Ì'@ : à@ñ JªË @ ©k. @Q K ©k. @Q K. QK PA ® JË @ HXA ̄ @ð , ÊK. @Q£ éJ . J ÊË @ éÖÞ AªË@ ¡J jÖß. éªÒm.Ì'@ ÐñJ Ë @ ¼PAªÖÏ @ HXYm.
' :  ñË@ é JK QëAg. ú » Q Ë @ m.Ì'@ Y» @ AÒ J K. , Aî EXĀ PQK. @ Yg @ É J ®Óð Q ®k é ®J Ê g Y«A ® JÖÏ @ Z @ñÊË @ H@ñ̄
:øñ JjÖÏ @ HBA¿ð + èQK Qm.Ì'@ : PY ÖÏ @ : ©k. QÖÏ @ . HAÒJ Êª JË @ éJ ®Ê K Pñ ̄ AJ . J ÊË ék. ñ JÊË
Machine ©k. @Q K Q ®k H@ñ̄ð ÊK. @Q¢ . XYj. J K ¼PAªÖÏ @ . . AJ . J Ë úÍ@ PñJ.ªÊË é JK QëAg. áÊªK ú » Q Ë @ m.Ì'@
á K. éªÒm.Ì'@ ÐñJ Ë @ HA¿ AJ. B@ XYj. JK. ÊK. @Q£ éJ . J ÊË @ éÖÞ AªË@ ¡J m× áÓ èXP@ñË@ QK PA ® JË @ HXA ̄ @ HXA ̄ @ AÒ» .Q ®k é ®J Ê g Y«A ® JÖÏ @ Z @ñÊË @ H@ñ̄ð AJ ËðX AîE. ¬Q ªÖÏ @ ú æ£ñË@ A ̄ñË @ éÓñºk H@ñ̄
. Aî EXĀ PQK. @ Yg @ É J ®Óð Q ®k H@ñ̄ ©k. @Q K. QK PA ® JË @
HAÒJ Êª JË @ éJ ®Ê K Pñ ̄ é K @ñ̄ Qå ð AJ . J Ë úÍ@ PñJ.ªÊË é JK QëAg. á« ú » Q Ë @ m.Ì'@ áÊ« @ ,Q k @ Pñ¢ ú ̄ ð é A KQK. , Ò mÌ'@ ÐñK ú » Q Ë @ ú ×ñ ®Ë@ áÓ CË ¨AÒ Jk. @ YªK. àC«B @ @
Yë ZAg. ð . éJ »Q Ë @ éÓñºmÌ'@ áÓ . àA «ðXP
@ I. J £ I. k. P KQË @ éÓðA ®Ó Iêk. @ð Aî DºË , ÊK. @Q£ úÎ« èQ¢J Ë@ K. A Īð ú
̄ IËðAg Ȳ Q ®k H@ñ̄ I KA¿ð / ÉK QK. @ Y JÓ èQ K @YË@ ¼PAªÖÏ @ HQ ® @ð . AJ ËðX AîE.
¬Q ªÖÏ @ éÓñºjÊË éJ Ë @ñÖÏ @ H@ñ ®Ë@ áÓ èYK Y . ¬B B@ YK Qå ð HA JÖÏ @ É J ®Ó á« ú æ
AÖÏ @ àA K H@ñ̄ Z @ P@
é JÓA É ¢ áË èXCK. à @ àA «ðXP @ I. J £ I. k. P ú »
Q Ë @ KQË @ Y» @ ,øQ k
@ éêk. áÓ
ú ̄ Z AÓYË@ ½ ® ú ̄  Pñ JËAK. ÈðYË@ è Yë àA «ðXP @ Ñî E@ð . AJ ðPð Qå Óð H@PAÓB @ AêÔ«Y K ú æË @ Q ®k
. AJ . J Ë Z @ P @
Ê ®ËAK. Qª èXCK. à@ ú æ Kñ» ú æ. K
Pñk. éJ ËA¢ B @ éÓñºmÌ'@ KP ÈĀ , É JÓ AJ ú ̄ ð . È@ñ̄ @ Ëð Éª ̄ úÍ@ h. A Jm ' AJ . J Ë à @ @Y» ñÓ , AJ . J Ë ú
̄ H@Pñ¢ JË @ áºË . XCJ. Ë @ ú ̄ Q K @YË@ ÈA J ®Ë @ ZAî EB AJ . J Ë ú ̄ èQkA J JÖÏ @ ¬@Q£ B@ á K. ¡ ñ JË @ èYj JÖÏ @ Õ× B@ ÈðAm 'ð
. à B@ ú æk éJ AJ éK ñ úÍ@ É ñ JË @ áÓ áºÒ J K ÕË éJ ËðYË@ éÒ ¢ JÖÏ @ Ȳ AJ . J Ë ú ̄ ÈA J ®Ë @ à @ áÓ éJ ËðYË@ ñ ®ªË@ éÒ ¢ JÓð èYj JÖÏ @ Õ× B@ Aî DJ K. áÓ éJ ËðX HAÒ ¢ JÓ HP
Yg Ȳð . HAJ ® ÖÏ @ð P@YÖÏ @ð á J KYÖÏ @
¬@Yî D @ ©Ó I. «QË@ áÓ èYK Yg. HAK ñ J Ó úÍ@ É ð
Bð . PA JË @ C£@ ̄ñË éJ ËðYË@ Xñêm.Ì'@ Ñ «P P B@ úÎ« @YJ ª YîD AJ . J Ë ú ̄ ¨A ð B@ È@ Q K Bð
. ú Í@ñ JË @ úÎ« XA Ë@ QîD ÊË ÊK. @Q¢Ë AëPA k ú ̄ èQÒ J Ó Q ®k H@ñ̄ È@ Q K HBA¿ð + èQK Qm.Ì'@ : PY ÖÏ @ - : ©k. @QÖÏ @
Field Content
Language German LLM Generator CHATGPT Domain Wikipedia
Title Abwasserpumpe
Human Schmutzwasserpumpen sind in der Lage, auch grob verunreinigtes Wasser (aber auch andere fluide Medien) mit hohem Feststoffanteil wie Steinen, Geröll und Ähnlichem zu fördern. Sie arbeiten in der Regel nicht selbstansaugend, sondern sind als Tauchpumpe ausgelegt, bei denen ein Schaufelrad das Fluid fördert. Dafür werden besondere Laufradformen wie Freistromrad, Einkanalrad, Mehrkanalrad, Diagonalrad, Schraubenrad wie auch Propellerrad verwendet. Kleinere Pumpen haben meistens ein Freistromrad, da dies am wenigsten störanfällig ist und einen großen freien Durchgang für die Feststoffe bietet. Pumpen mit Freistromrädern haben jedoch schlechte Wirkungsgrade und erreichen auch keine großen Förderhöhen; deshalb wählt man bei größeren Maschinen meistens andere Laufradformen. Schmutzwasserpumpen sind auf die Fördermenge ausgelegt – typische Förderleistungen von 5.000 bis 15.000 Liter pro Stunde (etwa 3 bis 5 Liter je Sekunde) – und weniger auf den Druck, die Förderhöhe beträgt häufig nur einige Meter. Der Antrieb erfolgt in der Regel durch einen Elektromotor, je nach Anwendungsfall aber auch mittels eines Verbrennungsmotors erfolgen. Eingesetzt wird dieser Pumpentyp etwa im Tiefbau zum Auspumpen von Baugruben, im Hochwasserschutz und in der Katastrophenhilfe, zur Bewässerung in Landwirtschaft und Gartenbau oder in der Landschaftsarchitektur zur Befüllung von künstlichen Wasseranlagen. Kleinere Modelle werden auch im Heimbereich verwendet, um das Wasser für den Rasensprenger nicht dem „teuren“ Trinkwasser zu entnehmen, sondern aus nahen Fließgewässern oder Teichen (hierbei sind aber wasserrechtliche Vorschriften zu beachten).
Prompt This is a sample German Wikipedia summary section for the title "WBDJ": Der sozialistisch ausgerichtete Weltbund der Demokratischen Jugend (WBDJ) wurde am 10. November 1945 auf der Weltjugendkonferenz in London gegründet. Er ist von den Vereinten Nationen als internationale nichtstaatliche Jugendorganisation anerkannt. Dem WBDJ gehören über 150 Mitgliedsorganisationen (nach eigenen Angaben 2012 über 270) aus 109 Ländern an, die rund 30 Millionen Jugendliche vertreten. Diese Jugendorganisationen sind kommunistisch oder sozialistisch orientiert und/oder sie gehören Befreiungsbewegungen an. Der WBDJ ist der Hauptträger der in unregelmäßigen Abständen seit 1947 stattfindenden Weltfestspiele der Jugend und Studenten.Der WBDJ war eine kommunistische Frontorganisation. In Folge des Zusammenbruchs des sozialistischen Lagers geriet der WBDJ 1990/91 in eine schwere Krise. Der künftige Charakter der Vereinigung wurde in Frage gestellt. Der WBDJ überstand jedoch diese Krise und stellte seine reguläre Arbeitsweise wieder her. 1997 wurden erstmals wieder Weltfestspiele auf Kuba durchgeführt. Höchstes Organ des WBDJ ist die Generalversammlung (bis 1957 Weltkongress), die zuletzt 2019 in Nikosia stattfand. Diese war mit 160 Vertretern von 93 Organisationen die größte Generalversammlung der letzten 30 Jahre. Using a similar formatting and style, write a German Wikipedia summary section for the title "Abwasserpumpe". The summary should have at least 250 words.
Machine Abwasserpumpe ist eine elektrische Pumpe zur Förderung von Abwasser und Schmutzwasser. Zusätzlich muss sie in der Lage sein, Feststoffe wie Haare, Papier oder Faserstoffe zu zerkleinern, um Verstopfung zu vermeiden. Man zählt die Abwasserpumpe zu den "Tauchmotorpumpen", da das gesamte Gerät unter Wasser getaucht wird. Das Pumpwerk (bestehend aus Saugkorb, Hydroscheider und Laufrad) wird dabei in einer Pumpstation eingesetzt. Der Antrieb des Laufrads geschieht durch einen Elektromotor. Die Pumpe trägt dazu bei, dass Abwasser und Schmutzwasser von der Einleitungsstelle zu den Kläranlagen oder sammelnden Kanalsystemen weitergeleitet werden. Insbesondere in Privathaushalten und Großküchen ist die Abwasserpumpe nützlich, da sie den Transport von Abwasser wesentlich erleichtert. Allerdings sollte man auf Handelsnamen achten: nicht alle Tauchmotorpumpen sind auch für den Einsatz in der Abwasserverarbeitung zugelassen. Eine häufig auftretende Schwierigkeit bei der Abwasserpumpe ist der sogenannte "Longpumpen". Hierbei kann der Abwasserstrom nicht mehr vollständig durch den Pumpenkopf geleitet werden, wodurch sich der Anteil sedimenter Stoffe erhöht. Darüber hinaus kann lang anhaltender Stress auf wenige Teile der Pumpe - etwa die Achse oder das Laufrad - schnell zu Abnutzungserscheinungen führen. Regelmäßige Wartung und Einhaltung von Grenzwerten für die Höhe des Abwasserdrucks oder die Größe der durchgeleiteten Partikel können diese Probleme vermindern. Die Qualität von Abwasserpumpen lässt sich anhand des maximalen zulässigen Drucks, der maximale Durchflussmenge, dem verwendeten Material oder auch in Hinblick auf Effizienz merklich voneinander unterscheiden. Es kann auch sinnvoll sein, beim Kauf einer Abwasserpumpe auf vorhandene Siegel und Zertifikate wie CE, VDE oder TÜV zu achten.
Field Content
Language German LLM Generator CHATGPT Domain News
Title Journalisten und Publikum sägen an den Ästen, auf denen sie sitzen
Human Verlogen, unglaubwürdig, verwöhnt: So werden Journalisten heute taxiert. Sie werden alle in einen Topf geworfen, ihre Arbeit scheint wertlos, beliebig und ein Ärgernis. Das muss aufhören. Medienpublikum, Medienmanager, Medienforscher und Journalisten sägen derzeit selber an den Ästen, auf denen sie sitzen. Gewiss gibt es Journalismus, der über die Stränge schlägt oder abgehoben ist. Und es gibt zu viele Journalisten, die selbstgefällig sind, keine Fehler zugeben und sich diebisch freuen, wenn sie einen anderen in die Pfanne hauen können. Das sei nicht kleingeredet, aber ins Verhältnis gesetzt. Wir wissen, dass es Banker gibt, die in die eigene Tasche wirtschaften und zahllose Kunden ruinieren, Automobilclubs, die Ranglisten manipulierten, Unternehmer, die mit illegalen Abschalteinrichtungen Abgasnormen umgingen, Politiker, die politische Gegner ausspionierten, korrupte Fußballfunktionäre, quacksalbernde Ärzte, Winkeladvokaten. Und wir wissen das alles oft gerade, weil es eben neben schwarzen Schafen noch andere Journalisten gibt. Ende Februar wurde Spotlight mit einem Oscar ausgezeichnet: ein Film über eine Recherche, die die Missbrauchsfälle der katholischen Kirche in Boston enthüllte. Für diese Recherche gewann das Investigativteam des Boston Globe Jahre zuvor einen Pulitzerpreis. Doch nicht nur der Enthüllung von Missständen wegen müssen wir eine Lanze brechen für einen aufklärenden, verantwortungs- und werteorientierten Informationsjournalismus. Wir brauchen Medien, die uns Arenen öffnen, in denen wir um die besten Argumente ringen und diskutieren, wie wir künftig in unserer Gesellschaft leben wollen. Hören wir auf, an Ästen zu sägen, und fordern wir einen Journalismus ein, der sich bewusst seinem Publikum zuwendet, der Zweifel, Ängste und Sorgen ernst nimmt und auch konstruktiv aufzeigt, wie Probleme gelöst werden können.
Prompt Schreibe einen deutschen Nachrichtenartikel zu einem vorgegebenen Titel. Der Artikel muss mindestens 250 Wörter und höchstens 500 Wörter enthalten. Schreibe ausschließlich den Artikel und keine Anweisungen, Kommentare, Werbeanzeigen, Hashtags oder ähnliches. Titel: Journalisten und Publikum sägen an den Ästen, auf denen sie sitzen Artikel:
Machine Heute leben wir in der Welt des Internets. Dies hat dazu beigetragen, dass jeder seinen Standpunkt darlegen und seine Meinung veröffentlichen kann. Aber haben wir dadurch eine Situation geschaffen, die zur Unabhängigkeit führt, oder haben wir uns in ein Gefängnis verwandelt, das von einer Gruppe von Menschen kontrolliert wird, die unsere Freiheit zensieren und unsere Perspektive beeinflussen? In den Tageszeitungen sind Journalisten in einer ähnlichen Situation gefangen. In dem Bestreben, die Meinungsfreiheit zu fördern und Nachrichten zu verbreiten, wird auch diese Gruppe von Menschen von der Unabhängigkeit des Internets gefangen genommen. Aber es gibt eine große Lücke zwischen der Zensur im Internet und der Zensur bei Tageszeitungen. Es ist wahr, dass Tageszeitungen dazu neigen, sich auf die Meinungen ihrer Leser zu konzentrieren, um einen zufriedenstellenden Inhalt bereitzustellen, der alle belohnt. Aber es ist auch wahr, dass diese Eigenarten der Tageszeitungen, als informelles Werkzeug für die politische Agenda genutzt werden kann. Werbewirtschaftliche und politisch motivierte Entscheidungen zwingen Journalisten dazu, die Wahrheit von dem zu unterscheiden, was derzeit trendig ist. Die Leserschaft selbst, die mit Zeitungen und anderen Publikationen sozialisiert wurde, erwartet manchmal einen sehr spezifischen Nachrichteninhalt, der möglicherweise mehr mit ihrer eigenen ideologischen Perspektive als mit einer objektiven Wahrheit übereinstimmt. Eine solche Publikumsbindung kann dazu führen, dass Journalisten gezwungen werden, in eine bestimmte Richtung zu tendieren und sich zwangsläufig von der Wahrheit zu entfernen. Schließlich werden Journalisten dazu gezwungen, Artikel zu schreiben, die das Publikum möglicherweise nicht hören will. Sie schreiben über schwierige Themen wie Armut, Arbeitslosigkeit, Drogenmissbrauch, Hunger und Kriege. Aber einige Journalisten sind sich nicht sicher, wie sie über diese Probleme schreiben sollen, ohne das Publikum nicht zu verlieren, das dazu tendiert, einfache Geschichten mit einer positiveren Note zu bevorzugen. Dies kann dazu führen, dass Journalisten eine eingeschränkte Rolle in der Informationsbeschaffung übernehmen und das Publikum den verbleibenden Raum besetzt. Das Problem der Journalisten und Publikum, die an den Ästen sägen, auf denen sie sitzen, ist also ein beunruhigender Trend, der unsere Freiheit und Möglichkeiten beeinträchtigen kann. Wenn wir befreit und unabhängig bleiben wollen, benötigen wir mehr professionelle und mutige Journalisten, die bereit sind, eine Vielzahl von Lösungen für unser komplexes Leben zu präsentieren, unabhängig von den eigenen Interessen. Die Herausforderung besteht darin, das kritische Denken und die Unabhängigkeit der Medien zu fördern, um sicherzustellen, dass unsere Freiheit und unsere Weltsicht nicht von einer einflussreichen Gruppe von Menschen dominiert werden. Wir sollten das Bestreben der Journalisten in ihrem Bemühen um Unabhängigkeit unterstützen und uns dafür einsetzen, damit jeder die Freiheit hat, seine Meinung zu äußern und die Wahrheit zu suchen.
Field Content
Language Italian LLM Generator CAMOSCIO-70B Domain News
Human Francesco Schettino rilancerà il turismo delle Marche? Non c’è ancora nessuna ufficialità, ma già è scoppiata la bufera. Infatti, léx comandante della Costa Concordia, secondo alcuni quotidiano locali, sarebbe pronto a diventare testimonial di Numana per il rilancio del turismo nella Riviera del Conero. Il sindaco però si è subito affrettato a precisare che si tratta di una idea malsana e fantasiosa. Sarà. Intanto però il legale di Schettino ha annunciato che il suo cliente avrà un colloquio di lavoro nei prossimi giorni. " Auspichiamo vivamente che questa buia fantasia non abbia fondamento alcuno: una persona divenuta famosa per un evento così negativo sicuramente non deve essere associata a una località che vive di turismo, bella e solare ", ha tuonato Carlo Neumann , presidente dell’Associazione Riviera del Conero. "Apprezzo lo sforzo di fantasia, ma bisogna partorire qualcosa di più interessante ", ha commentato il sindaco di Numana, Marzio Carletti (Pd). Insomma, ancora non si sa se Schettino metterà piede a Numana. Quello che è indubbio è che non ci sarà nessun inchino ad attenderlo
Prompt ### Istruzione: Dato il testo ’Schettino testimonial per rilanciare il turismo nella Riviera del Conero?’ scrivete un articolo di almeno 1000 parole in Italiano di cui quello è il titolo. ### Risposta:
Machine "La Riviera del Conero, situata sulla costa adriatica della regione Marche dell’Italia centrale, è una delle destinazioni turistiche più popolari d’Europa. È conosciuta per le sue spiagge di sabbia dorata e acque cristalline, così come per i suoi pittoreschi villaggi mediterranei che si snodano lungo la riva. Tuttavia, negli ultimi anni il numero di visitatori nella zona è diminuito significativamente a causa della crisi economica globale. Per cercare di invertire questa tendenza negativa, l’amministrazione locale ha deciso di rivolgersi al capitano Francesco Schettino, ex comandante della nave da crociera Costa Concordia naufragata nel 2012, per aiutare a promuovere nuovamente la Riviera del Conero come meta vacanziera ideale. Secondo gli amministratori locali, poiché il capitano Schettino è diventato un volto familiare in tutto il mondo grazie alla sua associazione con la tragedia della Costa Concordia, potrebbe essere uno strumento efficace per attirare attenzione sulla bellezza naturale ed esotica della Riviera del Conero"
E Examples of Boundary Identification
We demonstrate five examples below for boundary detection, including: ChatGPT-PeerRead, LLaMA-27B-PeerRead, GPT-4-OUTFOX, LLaMA-2-13B-OUTFOX and LLaMA-2-70B-OUTFOX.
Field Content
Language English LLM Generator LLAMA-2-7B Domain PeerRead
Title Evaluation of Defensive Methods for DNNs against Multiple Adversarial Evasion Models Abstract Due to deep cascades of nonlinear units, deep neural networks (DNNs) can automatically
learn non-local generalization priors from data and have achieved high performance in various applications. However, such properties have also opened a door for adversaries to generate the socalled adversarial examples to fool DNNs. Specifically, adversaries can inject small perturbations to the input data and therefore decrease the performance of deep neural networks significantly. Even worse, these adversarial examples have the transferability to attack a black-box model based on finite queries without knowledge of the target model. Therefore, we aim to empirically compare different defensive strategies against various adversary models and analyze the crossmodel efficiency for these robust learners. We conclude that the adversarial retraining framework also has the transferability, which can defend adversarial examples without requiring prior knowledge of the adversary models. We compare the general adversarial retraining framework with the state-of-the-art robust deep neural networks, such as distillation, autoencoder stacked with classifier (AEC), and our improved version, IAEC, to evaluate their robustness as well as the vulnerability in terms of the distortion required to mislead the learner. Our experimental results show that the adversarial retraining framework can defend most of the adversarial examples notably and consistently without adding additional vulnerabilities or performance penalty to the original model.
Prompt Task: Complete a partially-written peer review of the following paper. Make sure that: 1. The completion is of at least 155 words. 2. You only complete the partial review and not write it from the beginning. Title of the paper: {paper title} Abstract of the paper: {paper abstract} Review: {paper partial review}
Gold Boundary 46
Human Segment I reviewed the manuscript as of December 6th. The authors perform a systematic investigation of various retraining methods for making a classification network robust to adversarial examples. The authors achieve lower error rates using their RAD and IAEC methods perform better then previously introduced distillation methods for
Machine Segment adversarial training. They demonstrate that their approach does not introduce any significant performance degradation compared to the baseline model. Overall, this work provides valuable insights into improving the robustness of deep learning models against adversarial attacks.
Language English LLM Generator GPT-4 Domain OUTFOX
Problem Statement Describe Luke’s experience on the cattle boat trip and how it impacted his life. Include details about his background, the draft board’s reaction to his trip, his work on the boat, and his overall perspective on the experience. Additionally, discuss how this experience influenced Luke’s future interactions with international visitors.
Prompt Act as an experienced essay writer. Given the following problem statement (essay topic), please write an essay of at least 152 words with a clear opinion. The written essay should look like human-written. Partial essay: Luke was at his part time job when his friend Don came in and said (do you want to come on a cattle boat trip with me.) Luke knew it was a opportunity of a life time and couldn’t say no. Luke was
Gold Boundary 43
Human Segment Luke was at his part time job when his friend Don came in and said (do you want to come on a cattle boat trip with me.) Luke knew it was a opportunity of a life time and couldn’t say no. Luke was 18 before he arived in Greece which meant he could be drafted for military service. When the draft board learned he was on a cattle baot trip they said that he was able to keep going on the cattle boat trip. The cattle boat trip was an unbeleivable opportunity for a small towns boy like hime. Luke helped out on his aunt Katie’s farm whch prepared him for hard work but not for the dangers at sea. Luke also found a way to have fun on board even on return trips after the animals had been unloaded. When Luke got home he was very grateful that he was able to go on the trip. To the cowboys it was an adventure but to Luke it was a opportunity to open up a whole new world. And that awareness stayed with him, leading his family to host of intternational students and exchange visitors for many years.
Field Content
Machine Segment Luke was at his part time job when his friend Don came in and said (do you want to come on a cattle boat trip with me.) Luke knew it was a opportunity of a life time and couldn’t say no. Luke was a simple man from a small town in Kansas, with dreams bigger than the vast plains that surrounded him. The cattle boat trip was his first step towards realizing those dreams. The draft board, however, was not as enthusiastic about his trip. They saw it as a reckless adventure, a deviation from the norm. But for Luke, it was a chance to break free from the monotony of his life. Working on the boat was far from easy. The long hours, the grueling physical labor, and the constant motion of the sea tested his endurance. But it also taught him resilience, patience, and the value of hard work. He learned to appreciate the simple joys of life - the beauty of a sunrise over the open sea, the camaraderie among the crew, and the satisfaction of a job well done. The trip had a profound impact on Luke’s life. It broadened his horizons and gave him a new perspective on life. He realized that the world was much bigger than his small town, and that there were countless cultures and ways of life to explore. This realization influenced his future interactions with international visitors. He welcomed them with open arms, eager to learn about their cultures and share his own experiences. He became a bridge between his small town and the rest of the world, fostering understanding and friendship among people from different backgrounds. In conclusion, Luke’s experience on the cattle boat trip was a turning point in his life. It challenged him, shaped him, and ultimately, enriched his life in ways he could never have imagined. It was a testament to the transformative power of travel and the endless possibilities it offers.
Language English LLM Generator LLAMA-2-13B Domain OUTFOX
Problem Statement Discuss the dangers of using cellular devices while driving and the impact it has on individuals and society. Include statistics and examples to support your argument.
Gold Boundary 38
Prompt Act as an experienced essay writer. Given the following Problem statement (essay topic) complete the following Partial Essay writing at least 318 words with a clear opinion. The written essay should look like human-written (please start writing the essay without any additional text). Problem statement (essay topic): Discuss the dangers of using cellular devices while driving and the impact it has on individuals and society. Include statistics and examples to support your argument. Partial Essay: Over the past 4 decades cell phones have evolved extremely. Nowadays people use their phones for just about any and everything. Things such as texting, calling, playing games, booking appointments, even checking latest posts on social media and much
Human Segment Over the past 4 decades cell phones have evolved extremely. Nowadays people use their phones for just about any and everything. Things such as texting, calling, playing games, booking appointments, even checking latest posts on social media and much more. Cell phones started to become a problem when people were just looking down at their phones while crossing the street. Something that could be fixed. Now, people are choosing to use their phones while driving which has become an issue and is very dangerous. Which puts not only the driver in danger put others as well. Each day people are dying or being injured from using their cellular devices while being behind the wheel. Each year 421,000 people in the United States are injured due to a driver being distracted by their device. It’s more common for teens and young adults to text and drive (ages 18-24). 14% of fatal crashes are involved from texting and driving in this age division. In 2017 the "Hands Free Law" had been issued. Which means drivers are allowed to use hand free devices only while being behind the wheel of a car and being completely free of a device. Although people may think the hands free law has prevented less accident it hasn’t it only makes people have the urge to reach for their devices even more. 11 teenagers die every day due to texting and driving. Teenagers are 400% more likely to get into an accident from texting and driving. AT&Tś Teen Driver Survey found that 97% of teenagers think it’s dangerous, while 43% of them engage in the activity anyway. In conclusion I strongly disagree with texting and driving because it puts more than one person in danger. I feel as if you need to use your device while behind the wheel of a moving vehicle be mindful and pull to the side. Weather itś for making a quick call or sending a quick message even typing in a location into the GPS pull over because anything could happen in the blink of an eye.
Field Content
Machine Segment Over the past 4 decades cell phones have evolved extremely. Nowadays people use their phones for just about any and everything. Things such as texting, calling, playing games, booking appointments, even checking latest posts on social media and much more. However, one of the most dangerous things people do while using their phones is texting and driving. According to the National Highway Traffic Safety Administration (NHTSA), texting and driving has become one of the leading causes of car accidents in the United States. In 2018 alone, there were over 4,500 deaths and 140,000 injuries caused by motor vehicle crashes involving distracted drivers. Using cellular devices while driving poses a significant threat not only to the driver but also to passengers, pedestrians, and other drivers on the road. The act of texting and driving takes the driver’s focus away from the road, causing them to be less aware of their surroundings. This lack of attention can lead to devastating consequences such as accidents, injuries, and even fatalities. Moreover, using cellular devices while driving can also have a negative impact on society as a whole. For instance, accidents caused by distracted driving can result in property damage, medical expenses, lost productivity, and even legal fees. These costs can be staggering and can have long-lasting effects on individuals, families, and communities. Additionally, the frequency of distracted driving incidents has led to increased insurance premiums for all drivers, which can be a significant financial burden for many people. Furthermore, the dangers of using cellular devices while driving are not limited to the driver alone. Pedestrians, cyclists, and other drivers are also at risk of being injured or killed by distracted drivers. In fact, according to the NHTSA, pedestrian fatalities have increased by 3.4% from 2018 to 2019, with distracted driving being a major contributing factor. This highlights the need for stricter laws and regulations to prevent the use of cellular devices while driving. In conclusion, using cellular devices while driving poses a significant danger to individuals and society as a whole. The frequency of distracted driving incidents has led to increased accidents, injuries, and fatalities, as well as financial burdens for individuals, families, and communities. Therefore, it is crucial that we take measures to prevent the use of cellular devices while driving, such as implementing stricter laws and regulations, increasing public awareness campaigns, and investing in technology that can help reduce the temptation to use cellular devices while driving. By taking these steps, we can create safer roads for everyone and reduce the dangers associated with using cellular devices while driving. Language English LLM Generator LLAMA-2-70B Domain OUTFOX
Gold Boundary 116
Problem Statement Explain the benefits of the Electoral College system and argue for its continuation in the United States presidential elections, citing its effectiveness over the past century and the role of electors in ensuring informed and responsible voting.
Prompt Act as an experienced essay writer. Given the following Problem statement (essay topic) complete the following Partial Essay writing at least 318 words with a clear opinion. The written essay should look like human-written (please start writing the essay without any additional text). Problem statement (essay topic): Explain the benefits of the Electoral College system and argue for its continuation in the United States presidential elections, citing its effectiveness over the past century and the role of electors in ensuring informed and responsible voting. Partial Essay: I would say that we keep the Electoral College because it has worked over the past century so why change it now. The people voting don’t always know what they want in a president they might be undecided and just choose the best looking one I mean yeah some have their facts straight and know how the president they are voting is going to help the united states become a better country overall When people have to much freedom they don’t know what to do and do wrong choices so i think choosing a elector is a good thing they have studied the president and have heard countless promises that he has done. I mean he
Field Content
Human Segment I would say that we keep the Electoral College because it has worked over the past century so why change it now. The people voting dont́ always know what they want in a president they might be undecided and just choose the best looking one I mean yeah some have their facts straight and know how the president they are voting is going to help the united states become a better country overall When people have to much freedom they dont́ know what to do and do wrong choices so i think choosing a elector is a good thing they have studied the president and have heard countless promises that he has done. I mean he or she most likley will vote for the candiate running for president you did because he or she is a elector he or she gets the people who also voted for the same person as he or she wants. I mean we people are still voting we still have the vote just that our vote goes to choosing a certain elector instead of just directly leaving it to the people to vote for a ceratin candiate running for president. By this the number is much smaller since each country is weighed and is worth certain points and decided by electors you choosed to trust to vote for the same president you did. You still are in control just that you choose somebody he/her is not likely to change his or her mind and vote for the exact person you want he or she to vote for. It stated in the article that "Each candidate running for president in your state has his or her own group of electors. The electors are generally chosen by the candiateś politcal party." I say that if the candidate political party chooses the electors then they must be good and the party must be sure the elector wont́ change his mind out of no where. This does make the candiate running for president try to convience the electors he is the one they need and that he will help our country be better. This also helps the candiate running for president know what states he should go to since some states they already know they lost It stated in the article that "A solid regional favorite, such as Romney was in the south,has no incentive to campaign heavily in those states, for he gains no electoral votes by increasing his plurality in states that he knows he will win."So this keeps the president from having to go to every state because its just no based of popular vote and want every individual person votes for. I think the Electoral college is great and well organized the process is good and the only thing is does is make the voting process more easier and ordered. I think we should keep it i mean it has had his troubles her and there but it has worked so good for such a long time. We dont́ want to change it and create choas since a lot of people are not use to change and dont́ take it quite well
Machine Segment I would say that we keep the Electoral College because it has worked over the past century so why change it now. The people voting don’t always know what they want in a president they might be undecided and just choose the best looking one I mean yeah some have their facts straight and know how the president they are voting is going to help the united states become a better country overall When people have to much freedom they don’t know what to do and do wrong choices so i think choosing a elector is a good thing they have studied the president and have heard countless promises that he has done. I mean he has to convince the elector that he is the right person for the job. The Electoral College system has been a cornerstone of American democracy for over two centuries, and it continues to serve the country well. Despite calls for its abolition, the system has proven its effectiveness time and time again, and it remains an essential component of the United States presidential elections. In this essay, I will argue for the continuation of the Electoral College system, citing its success over the past century and the critical role of electors in ensuring informed and responsible voting. First and foremost, the Electoral College system has a proven track record of effectively electing presidents who have gone on to serve the country with distinction. Over the past century, the system has produced some of the most iconic and influential leaders in American history, including Franklin D. Roosevelt, John F. Kennedy, and Barack Obama. These presidents have demonstrated exceptional leadership, vision, and commitment to the country, and their elections were made possible by the Electoral College system. Moreover, the Electoral College system ensures that the president is elected by a broad coalition of states, rather than just the most populous ones. This ensures that the president is accountable to voters across the country, rather than just those in heavily populated urban areas. For example, in the 2016 presidential election, Donald Trump won the presidency despite losing the popular vote to Hillary Clinton. However, he won a majority of the electoral votes, ensuring that he had a broad base of support across the country. Another significant advantage of the Electoral College system is the role of electors in ensuring informed and responsible voting. Electors are chosen by each state to cast votes for the president and vice president. They are typically party loyalists who have a deep understanding of the candidates and their policies. They are also responsible for ensuring that the president and vice president meet the constitutional requirements for office. This means that they must carefully consider the candidates’ qualifications, character, and policy positions before casting their votes.
Field Content
Furthermore, the Electoral College system provides a safeguard against the tyranny of the majority. In a direct popular vote system, the candidate who wins the most votes would automatically become the president, regardless of whether they have the support of a broad coalition of voters. This could lead to a situation where a candidate with extreme views or policies could win the presidency with a simple majority, even if they are opposed by a large majority of voters. The Electoral College system prevents this by requiring the president to win a majority of electoral votes, which ensures that the president has broad support across the country. Finally, the Electoral College system is an essential component of the federal system of government established by the Constitution. The system ensures that the president and vice president are elected by a coalition of states, rather than just the most populous ones. This ensures that the federal government remains accountable to the states and their citizens, rather than just the national population. In conclusion, the Electoral College system has been a cornerstone of American democracy for over two centuries, and it continues to serve the country well. It has a proven track record of effectively electing presidents who have gone on to serve the country with distinction, ensures that the president is accountable to voters across the country, provides a safeguard against the tyranny of the majority, and is an essential component of the federal system of government established by the Constitution. Therefore, I strongly argue for the continuation of the Electoral College system in the United States presidential elections.

heading: Future Work
text: Moreover, detecting texts that contain multiple changing points is an interesting research direction for future work, but it would bring new challenges with analyzing shorter spans. Extremely, if a human added an extra word in a machine-generated sentence, it would be much harder for blackbox detectors to identify this word. Metrics Mean Absolute Error (MAE) is used to evaluate the boundary detection model’s performance. It measures the average absolute difference between the predicted position index and the actual changing point. 4 Human Evaluation
We investigate the complexity of differentiating the language model used for creating machinegenerated texts for human readers. Considering that annotators may lose patience to learn and summarize the patterns of seven distinct generators, we control the number of unique models they could identify and simplify to four-class classification with five demonstration examples (five-shot) as practice to learn before the formal annotation. Data for Human Evaluation There are 35 unique domain-generator combinations — five domains × seven generators (six LLMs plus humanwritten texts) for task 2 data. With the concern of annotators’ workload, we sample 4 examples for each combination, in total of 140 examples, split into four groups as Figure 1. Without consideration of out-of-domain classification, we follow the best-performing setup: see all, classify all. If each annotator needs to check all domains, they need to learn 20 demonstration examples, which is a time consuming and laborious undertaking. Let us consider that each annotator annotates 3 domains. They would need to check 4 models × 3 domains × 4 examples = 48, which is
in a more reasonable range. Five demonstration examples are randomly se-
lected, to ensure that four classes are all included. Annotator background. Group 1, 2, 3, and 4 are annotated by native Italian, Chinese, English, and Russian speakers, respectively; they are either postdoc or PhD students in NLP. Results Intuitively, it would be challenging to distinguish between generators based on 1-shot per class. Annotators may make decisions depending on the observations that generations by Cohere tend to be repetitive, Dolly-v2 has high-quality and coherent texts, while BLOOMz texts are more likely to contain a bunch of numbers or a bunch of “YES” or “NO” characters repetitively at the end of the texts, compared to other models. For GPT series, annotators distinguish them depending on their quality level assumption that GPT-4 > ChatGPT > davinci-003. They spot human-written texts based on formatting patterns (e. g. , initial double newlines, initial space, completely missing new lines in the paragraph), typos, inconsistencies within texts, or specific references and URLs. As shown in Table 4, the best distinction accuracy for humans is 27%, and the average accuracy
21% is less than random guess. This implies that it is extremely difficult for humans to learn patterns from five demonstration examples and then distinguish generations of different LLM generators (see the F1-score for each separate class in Appendix B). 5 Experiments
We conduct an experimental evaluation with several common baselines representing a diverse set of detection techniques: standard fine-tuning of an encoder-based Transformer model, elaborated feature engineering for the similar task, ranking-based features which form patterns for different LLMs. For classification tasks 1 and 2, we use five baseline classifiers suggested by Wang et al. (2023). They include fine-tuned RoBERTa and XLM-R classifiers, logistic regression with GLTR features (LR-GLTR) (Gehrmann et al. , 2019a), SVM with Stylistic (Stylistic-SVM), and SVM with NELA features (NELA-SVM) (Horne et al. , 2019). For the boundary detection task 3, we applied two detectors based on semantic features. GLTR features These features are based on the assumption that in order to generate fluent and natural-looking text, most LLM decoding strategies sample high-probability tokens from the head of the distribution (Gehrmann et al. , 2019a). Thus, word ranking information of an LM can be used to distinguish machine-generated texts from humanwritten. Following Wang et al. (2023), we select two categories of these features: (1) the number of tokens in the top-10, top-100, top-1000, and 1000+ ranks from the LM predicted probability distributions (4 features); and (2) the Frac(p) distribution over 10 bins ranging from 0. 0 to 1. 0 (10 features). Frac(p) describes the fraction of probability for the actual word divided by the maximum probability of any word at this position. We train a logistic regression model based on these 14 features to perform the binary classification task with the parameter of the maximum iteration = 1,000. Stylistic and NELA features We use stylistic features including (1) character-based features, e. g. , the number of characters, letters, special characters, etc. , (2) syntactic features, e. g. , a number of punctuation and function words, (3) structural features, e. g. , a total number of sentences, and (4) word-based features, e. g. , a total number of words, average word length, average sentence length, etc. NEws LAndscape (NELA) features (Horne et al. , 2019) involves six aspects: (1) style: the style and the structure of the article; (2) complexity: how complex the writing is; (3) bias: overall bias and subjectivity; (4) affect: sentiment and emotional patterns; (5) moral based on the Moral Foundation Theory (Graham et al. , 2012), and (6) event: time and location. An SVM classifier is applied to perform the binary distinction. 5. 1 Monolingual Human vs. Machine
For Task 1, we simulate a practical scenario where a detector have to deal with texts generated from a new LLM while it is trained on the outputs from
a limited number of other generators. A practical detector should be able to generalize over different generators and be robust on unseen inputs. Experimental Setup We conduct experiments in two setups. (1) In the first setup, we select an LLM and combine its generations across all domains with the upsampled human-written texts to create a test set. Texts produced by other LLMs and humans are divided into training and development sets with a 4:1 ratio. (2) In the second setup, we take the data from all domains and generators and randomly split it into train and test sets (row All in Table 5). We train detectors with five different seeds for all experiments and present the mean values of the metrics. Results and Analysis Classification results of five detectors are presented in Table 5. We analyze the results to answer three research questions: (1) For a given detector, on which unseen generator does it generalize best (highlighting F1-score and accuracy)? (2) Given a setting that training without the data of the generator to test, which detector performs the best (underlined Acc). (3) On average across both settings, which detector is the most accurate and has the best generalization capabilities? Which detector has higher recall than precision?
RoBERTa, XLM-R, and LR based on stylistic features give the best F1-score and accuracy on the setting where the classifier is trained with generations of all generators and tested across all generators. GLTR features perform the best in the setting where the detector is trained on data without generations from Cohere, and tested over Human vs. Cohere data, while NELA has the highest accuracy in distinguishing human vs. GPT-4 when it has not seen any data from GPT-4. The results for RoBERTa and XLM-R suggest that such detectors encountered notable challenges when confronted with text from new generators not encompassed in their training data, exhibiting the large accuracy gap between the setting of All and others. This problem is especially conspicuous in the case of BLOOMz for all detectors (All vs. BLOOMz), followed by Dolly-v2, davinci-003, ChatGPT, GPT-4, and Cohere. XLMR and GLTR demonstrate better generalization performance (underlines numbers) over (davinci-003, Dolly-v2, and BLOOMz) and (ChatGPT, GPT-4, and Cohere) respectively. XLM-R is the best on average. It has higher
accuracy and better generalization ability. LR with NELA features is the worst. Recall of RoBERTa, XLM-R, and GLTR is higher than precision, while it is the opposite for stylistic and NELA features. In the scenario of detecting misuse, we expect higher recall to avoid less misses of potentially malice. Overall, detectors show poor generalization performance when testing on unseen generators. This is well-known due to exposure bias. The generations by BLOOMz are significantly different from other generators, leading to the result that when training on the data combination of other generators and testing on BLOOMz, the performance is extremely low for all detectors. The detector based on NELA features overall performs the worst, which demonstrates that the features used for fact verification are not suitable to serve as signals to distinguish human vs. machine text. 5. 2 Multilingual Human vs. Machine Similar to monolingual setups, we evaluate detectors generalization on new languages (the language to test) and train on the combination of other languages. In the setting of All in Table 6, we merge data of all languages, and randomly split into train, validation and test sets. Results and Analysis We categorize the performance of the detector (XLM-R) across ten configurations into three tiers based on the F1 score. (1) F1<50: Bulgarian, Russian and Indonesian; (2) 60<F1<85: English, German, Italian, Chinese; and (3) F1>90: Arabic, Urdu, All. For the first level, Bulgarian and Russian are both Slavic languages, and Indonesian is a member of the Malayo-Polynesian branch of the language
family, training on other distantly-unrelated languages (Latin, Chinese and Arabic family) would naturally result in low performance. For the second level, Chinese is a high-resource language, so the detector could benefit from the learned patterns during XLM-R pretraining. For German and Italian, English training data will keep the same family languages in a medium performance, vice versa. High recall and low precision with respect to MGT across all testing languages implies that the majority of examples are recognized as machinegenerated text. This leads to lower accuracy on human-written text than that on MGT (unweighted average macro-F1 < MGT F1). Overall, the detector often predicts texts as MGT. when dealing with low-resource languages or training on language families that are distantly related, the accuracy tends to be moderate. 5. 3 Multi-way Detection
Experimental Setup For task 2, the classifiers are trained to distinguish multiple generators (seven-label classification), including human, davinci-003, ChatGPT, GPT-4, Cohere, Dolly-v2, BLOOMz. We measure the capabilities of detectors to generalize on a new domain that was not presented during training. The detectors are trained on the whole dataset except the domain used for testing. For this task, we calculate accuracy, macro F1-score, and the class-wise F1-scores. We experiment with six baselines: fine-tuned RoBERTa and XLM-R, logistic regression (one-vsrest) and SVM (one-vs-one) with GLTR features, and SVM (one-vs-one) with stylistic and NELA. Results and Analysis We try to answer three questions: (1) given a detector, which domain is the hardest to predict; (2) which generator is the most challenging one to distinguish; and (3) which detector performs the best on task 2. As results in Table 7, Transformer-based detectors show poor performance in this task when facing with text from new domains not included in their training data. Taking RoBERTa as an example, low accuracy is presented when testing on any unseen domains, such as arXiv and Wikipedia with accuracy of 36 and 52. Only when training over all domains and testing over all domains, accuracy reaches 97. The detectors exhibit significant challenges in generalizing to unfamiliar content, reflecting a notable weakness in their ability to handle previously unseen domains. The domain of
arXiv is the hardest for most detectors to identify if they are trained without arXiv examples. For most domains, davinci-003 is the most challenging generator to distinguish, followed by ChatGPT. BLOOMz appears to be the easiest one, for all classifiers (see class-wise F1-scores in Appendix C). Even the worst detector NELA-SVM can identify BLOOMz with >90% F1-score. This may result from the fact that the generative distri-
bution of BLOOMz is significantly different from other generators, while the distribution between davinci-003 and ChatGPT are more similar to each other, making it hard to make distinctions. RoBERTa is the best in this task followed by XLM-R. Based on the GLTR features, SVM onevs-one mode is better than logistic regression onevs-rest mode. Under the same SVM, GLTR features are more helpful than stylistic and NELA. 5. 4 Boundary Identification Experimental Setup We perform evaluation for task 3 in two settings: (1) train on PeerRead and test on PeerRead (same or different generators); and (2) train on PeerRead and test on a new domain with multiple generators (test column in Table 3 with 11,123 examples). As a baseline, we use a sequence tagger based on Longformer (Beltagy et al. , 2020) considering the long context (> 1,024 tokens) and DeBERTav3 (He et al. , 2021), where the position of the first word that is predicted as machine-generated is our prediction. The label was predicted at the token level: human-written words are tagged as 0 and machine-generated words are tagged as 1. We remap the tagged tokens back to the word level and calculate the MAE. We run experiments three times with different random seeds and calculate the mean and standard deviation. Results and Analysis In Table 8, when training on generations from LLaMA-2 PeerRead and testing on ChatGPT PeerRead and vice versa, even within the same domain, predicted boundary shows large gap with the gold label. This reveals that it is much harder for boundary detection models to predict changing points for unseen generators. Testing on an unseen domain OUTFOX with mixture of generators including ChatGPT, GPT-4 and LLaMA-2 series, results in worse MAE, i. e. , column all test. Taking the setting of All as an example, MAE changes from <5 (column
PeerRead LLaMA-2-7B* and PeerRead ChatGPT) to MAE>21 and >15 (column all test) respectively for Longformer and Deterta-v3. Particularly for Longformer trained with LLaMA-2 data, MAE is greater than 53 when testing on all test, demonstrating challenges on unseen domains and generators, consistent with findings in task 1 and 2