Idiosyncrasies in Large Language Models
Authors: Mingjie Sun
Yida Yin
Zhiqiu Xu
J Zico Kolter
Zhuang Liu
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models; Idiosyncrasies; Dataset bias; Synthetic data;
Primary area: deep_learning->large_language_models
TL;DR: We study idiosyncrasies in LLMs.

Abstract: In this work, we unveil and study idiosyncrasies in Large Language Models (LLMs) -- unique patterns in their outputs that can be used to distinguish the models. To do so, we consider a simple classification task: given a particular text output, the objective is to predict the source LLM that generates the text. We evaluate this synthetic task across various groups of LLMs and find that simply fine-tuning text embedding models on LLM-generated texts yields excellent classification accuracy. Notably, we achieve 97.1\% accuracy on held-out validation data in the five-way classification problem involving ChatGPT, Claude, Grok, Gemini, and DeepSeek. Our further investigation reveals that these idiosyncrasies are rooted in word-level distributions. These patterns persist even when the texts are rewritten, translated, or summarized by an external LLM, suggesting that they are also encoded in the semantic content. Additionally, we leverage LLM as judges to generate detailed, open-ended descriptions of each model's idiosyncrasies. Finally, we discuss the broader implications of our findings, including training on synthetic data, inferring model similarity, and robust evaluation of LLMs.

Idiosyncrasies in Large Language Models
Mingjie Sun
Yida Yin 2
Zhiqiu Xu
J. Zico Kolter Zhuang Liu
Abstract
ferent LLMs, either between the outputs of entirely different models between those of different variants of the same model family. If possible, the ability to distinguish between source models in this manner would be valuable for num- ber of applications: it could shed light on the relative uptake of different LLMs, beyond what is reported by individual companies, and on the nature of data used to build differ- ent models. Additionally, it could offer insights into what features of generated text most "unique" to each LLM.
In this work, we unveil and study idiosyncrasies in Large Language Models (LLMs)- unique pat- terns in their outputs that can be used to distin- guish the models. To do so, we consider simple classification task: given particular text output. the objective is to predict the source LLM that generates the text. We evaluate this synthetic task across various groups of L.LMs and find that sim- ply fine-tuning text embedding models on LLM- generated texts yields excellent classification ac- curacy. Notably, we achieve 97.1 1% accuracy on held-out validation data in the five-way classifica- tion problem involving ChatGPT, Claude, Grok, Gemini, and DeepSeek. Our further investiga- tion reveals that these idiosyncrasies are rooted in word-level distributions. These patterns persist even when the texts are rewritten, translated, or summarized by external LLM, suggesting that they are also encoded in the semantic content. Ad- ditionally, we leverage LLM as judges to generate detailed, open-ended descriptions of each model's idiosyncrasies. Finally, we discuss the broader implications of our findings, including training synthetic data, inferring model similarity, and robust evaluation of LLMs. Code is available
In this paper, we investigate whether LLMs exhibit idiosyn- crasies that enable their outputs to be reliably differentiated. Inspired by recent studies on dataset bias in computer vision (Liu & He, 2025; Zeng et al., 2024), which showed that images from different large-scale datasets can be accurately distinguished by standard neural networks, we consider a similar synthetic classification task to assess the separability of responses generated between different LLMs. Specifi- cally, we sample a large number of text outputs from each LLM using the same set of prompts and then train a clas- sifier to recognize which model generates a specific text. Figure provides an overview of our framework. The lustrated example on ChatGPT, Claude, Grok, Gemini, and DeepSeek presents five-way classification problem.
We find that a classifier based upon simple fine-tuning text embedding models on LLM outputs is able to achieve re- markably high accuracy on this task. This indicates the clear presence of idiosyncrasies in LLMs. The observation is highly robust over a large variety of LLM combinations. For instance, trained the combined set of texts from ChatGPT, Claude, Grok, Gemini, and DeepSeek, model can achieve 97.1% % classification accuracy on the held-out validation data, compared to 20.0% chance-level guess. Within the same model family, we obtain a non-trivial 59.8% accuracy across 4 model sizes in Qwen-2.5 series (Qwen et al., 2024). Further, we observe strong out-of -iito---istribttion generalization of these classifiers when tested on responses from prompts outside the training distribution.
1. Introduction
As the adoption of generative models such as LLMs accel- erates, becomes increasingly important to understand the origin and provenance of such generated content. While great deal of past work has focused the classification of numann-ritten and Al-written content (Krishna al., 2023; Mitchell et al., 2023; Sadasivan et al., 2025), there has been little work on classifying between content generated by dif-
'Equal contribution 'Carnegie Mellon University *UC Berkeley "University of Pennsylvania "Princeton University. Correspon- dence to: Mingjic Sun <mingjies @ andrew.cmu.cdu>. Yida Yin <davidyinyida0609 berkeley.cdu:
We observe several interesting properties of this task. When controlling the length and format of outputs through prompt instructions, we still obtain high classification accuracy. Fur- thermore, for post-trained LLMs, the classifier demonstrates non non-trivial accuracy even with only the first few tokens of the generated text. However, when classifying generations
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Idiosynerasies in Large Language Models
Prompt: Now everyone in a room silent working on their own thing, and you need to break the silence three-sentence joke. What would you say?
P
Claude
Grok
Gemini
DeepSeek eek
Why don'ttiintistt trust atoms? Because they make up everything! And speaking of making things up, told my computer1 needed a break, and said, 'No problem, I1l go on byte-sized vacation!"
Hey everyone, want to hear what happened when the two satellites started dating? They had great first date, but things fell apart after that.. Turns out they had too much space between them!
Hey, did you know got new job at the calendar factory? The hours are great but the days are really hard get off. And the worst part? Every time I by to ta aeke break, get a month behind!
A snail went to buy car. The salesman told him, "We have great deal on this one, but doesn't come with a shell." The snail looked surprised and asked, "Well, how am I supposed drive it?
Why did the scarecrow win an award?
Because he was outstanding his field!
...Or maybe he just knows how to keep people silent straw-ng presence.
Neural Network Classifier
Cha Claude, Grok, Gemini, or DeepS eek?
Figure 1. Our framework for studying idiosyncrasies in Large Language Models (LLMs). We show that each LLM unique in its expression. In the example shown here on ChatGPT, Claude, Grok, Gemini, and DeepSeek, a neural network classifier is able to distinguish them with a near-perfect 97.1 % accuracy.
from the same LLM but using different sampling strategies, we achieve accuracy only slightly above the chance level. In addition, we observe certain behaviors of this task that resemble those of standard text classification, where im- provements in text embeddings and availability of larger training datasets lead to better classification performance.
manipulate voting- bbatng leaderboards, therefore highlight- ing the need for more robust evaluation methodologies.
2. Evaluating Idiosyncrasies in LLMs
Large Language Models (LLMs) share several core charac- teristics. The majority of them are based on the Transformer architecture (Vaswani al., 2017), which is shared by all models we consider in this paper. Second, they are trained using an auto-regressive objective (Radford et al., 2019), where they predict the next token in a sequence based on pre- ceding context. Lastly, their training datasets significantly overlap, often incorporating vast and diverse sources such as Common Crawl, Wikipedia, and Stack Overflow. Given these similarities, it is natural to ask: do LLMs speak in the same way? If not, how can effectively measure the degree of their differences?
We analyze the sources of these idiosyncrasies by applying text transformations that isolate different levels of infor- mation. We find that randomly shuffling words in LLM- generated responses leads to only a slight decrease in classi- fication accuracy. This suggests that a substantial portion of distinctive features is encoded the word-level distribution. We then highlight distinct sets of characteristic phrases that consistently associated with each LLM. We also observe that markdown formatting contributes to moderate degree of idiosynerasies in the LLMs following post-training.
At the same time, we obtain over 90% accuracy when the word distribution is disrupted through transformations that preserve semantics, such as rephrasing or translating. Even with the most aggressive transformation summarizing, classification accuracy remains well above chance-level guess. This finding implies that semantic information also shapes the idiosyncrasies in LLMs. Through open-ended language analysis, we provide further insights into these characteristics. For instance, ChatGPT has a preference for detailed, in-depth explanations, whereas Claude produces more concise and direct responses. prioritizing clarity.
To address these questions, we construct a synthetic task focused on classifying outputs from different LLMs. Con- sider N LLMs, denoted as f1....,fx where each fi takes an input prompt p and outputs text completion o. For a given dataset D of prompts, the outputs produced by each LLM fi are denoted as O, We approach this problem with straightforward setup. For N output sets O, we formulate a N-way classification task, where the objective is to predict which LLM produced each output. If outputs of different LLMs were drawn from the same distribution, classification accuracy would not be better than random chance. Thus, we use the classification performance of this synthetic task as measure of idiosyncrasies in LLMs.
Last, we discuss the broader implications of our findings. One should be cautious when using synthetic data to train LLMs, as we show that many of these idiosyncrasies can be inherited in such a process. Our framework also serves as a tool for assessing model similarities among frontier models, either open-source or proprietary. In addition, we discuss how the idiosynerasies in LLMs can be used maliciously to
Our task is formulated as sequence classification problem, for which fine-tuning BERT-style models a common ap- proach (Sun et al., 2019). In this work, we fine-tune more recent and competitive sequence embedding model based
2

---

Idiosynerasies in Large Language Models
ChatGPT
Claude
Grok
Gemini
DeepSeek
acc. (chat)
99.3
97.7
98.7
97.2
99.7
99.6
99.6
99.4
98.7
99.9
97.1
(a) chat APIs
Llama
Gemma
Qwen
Mistral
acc. (instruct)
acc. (base)
99.9
98.3
97.8
81.7
97.0
96.3
99.9
98.3
/
99.9
98.4
/
96.1
95.7
/
96.3
87.3
(b) instruct and base LLMs
Table Classification accuracies various LLM combinations. Top: results for chat APIs. Bottom: results for instruct and base LLMs. Check marks (V) denote the models included in each combination. We observe high classification accuracies consistently across all model combinations, indicating the presence of distinct idiosyncrasics in LLMs.
on decoder-only Transformers: LLM2vec (BehnamGhader et al., 2024). We attach a N-way classification head to the extracted embeddings and use LoRA-based fine-tuning (Hu al., 2022) the model weights. Input sequences are cated maximum length of 512 tokens. We report the classification accuracy held-out validation set. Addi- tional training details are provided in Appendix A.2.
2. Instruct LLMs ("instruct")): These models are trained to generate high-quality responses from human instruc- tions. consider four LLMs of similar sizes across different families: Llama3.1 (Dubey et al., 2024), Gemma2-9h (Riviere et al., 2024), Qwen2.5-7b (Qwen et al., 2024) and Mistral-v3-76 (Jiang et al., 2023). We will refer to them as Llama, Gemma, Qwen and Mistral.
3. Base LLMs ("base"): These base versions of instruct LLMs. They are obtained by pretraining on extensive text corpora without any post-training stage.
2.1. Main Observations
We observe surprisingly high accuracies by neural networks to classify LLM outputs. This observation is robust across different settings, e.g., across model families and sizes.
Throughout the paper, refer these three categories "chat", "instruct", and "base" respectively. For given prompt dataset, we collect 11K text sequences, splitting them into 10K and 1K as training and validation sets, re- spectively. The same split is used across all LLMs. For chat APIS and instruct LLMs, we generate outputs from UltraChat (Ding et al., 2023). a diverse dialogue and in- struction dataset. For base LLMs, synthesize new texts using prompts from Feb (Penedo et al., 2024), high- quality LLM pretraining dataset. More details on response generation are in Appendix A.I.
describe the LLMs we use to generate the output datasets O O1...,. ... N. For a comprehensive and fair comparison across model families, we categorize three groups of LLMs:
Chat APIs ("chat"): This category includes state-of- the-art LLMs that primarily accessible via APIs. We consider GPT-4o (OpenAI, 2024), Claude-3.5- Sonnet (Anthropic, 2024), Grok-2 (xAI, 2024), Gemini- 1.5-Pro (Google, 2024), and DeepSeek-VV V3 (DeepSeck- AI, 2024). For simplicity, we refer to them as ChatGPT, Claude, Grok, Gemini and DeepSeek. Their architec- tures and weights remain proprictary and undisclosed, with the exception of DeepSeek.
Across model families. In Table 1, we report the results for classifying outputs from various combinations of chat APIs (Table la) and instruct base LLMs (Table lb). In each
3

---

Idiosynerasies in Large Language Models
of the three LLM groups, we enumerate all (C%) possible pairwise combinations when choosing 2 out of N models in the top panel of each table, as well as the case including N models in the bottom row. For the binary classification task, the neural network consistently achieves over 90% accuracy, with only one exception. Notably, for chat APIs and instruct LLMs, many combinations reach as high as 99% accuracy. In the more challenging N-way classification tasks, our classifiers maintain strong performance, achieving at least 87.3% accuracy across three groups. These results highlight the idiosyncrasies across different L.MSs. We refer readers to Appendix B.3 for the confusion matrices of our classifiers.
classifier on the remaining three datasets. Table shows the results instruct L.MMs. Our classifiers generalize well across different datasets, indicating that they learn very robust and transferable patterns.
train / test
Ultrahhat Cosmopedia LmsysChat WildChat
UltraChat
96.3
98,9
89.9
92.4
Cosmopedia
95.7
99.8
88.3
94.9
LmsysChat
94.7
97.2
91.8
92.0
WildChat 95.1
99.1
90.2
95.7
Table 3. Robust generalization out-of-distribution responses. We train classifiers on LLM outputs from one prompt datasct and tested on those from another.
Within the same model family. We evaluate sequence classification performance when distinguishing responses from LLMs within the same model family. Note that models from the same family typically share common training pro- cedures, e.g. pretraining datasets and optimization schedule. First, we analyze the impact of model size by consider- ing four Qwen2.5 instruct LLMS with 7B, 14B, 32B, and 72B parameters. As shown in Table 2, the classification task is more difficult here, but our classifiers remain well above chance accuracy when distinguishing LLMs within the same family. In the binary classification setup, the high- est accuracy reaches 85.5% whereas in the full combination setup, the accuracy becomes 59.8%. In addition, we observe high accuracies when classifying responses from base and instruct versions of the same model. For example, our clas- sifiers achieve 96.8% accuracy when distinguishing outputs from Qwen2.5-7b base and instruct models.
2.2. Controlled Experiments
We analyze the behaviors of the synthetic classification task in several controlled settings. From now on, we only report accuracies of the N-way classification lask in each group.
Prompt-level interventions. We assess the degree of id- iosyncrasies in LLM outputs with explicit prompt-level in- terventions. Specifically, we modify the original prompt by incorporating additional instructions to constrain response length and format. We then perform sequence classification on the resulting outputs. Our interventions are:
Length control: Please provide a concise response in single paragraph, limited to maximum of 100 words. Format control: Please provide your response in plain text only, avoiding the use of italicized or bold text, lists, markdown, or HTLL formating..
14b
32b
72b
instruct
77.0
LLM outputs after these interventions are presented in Ap- pendix C. We find that LLMs can follow the additional instructions in generating responses.
/
81.2
/
83.4
d
63.1
/
V
85.5
original length control format control
84.8
instruct LLMs
96.3
93.0
91.4
59.8
Table 2. Classification within Qwen2.5 model family. The clas- sifier can differentiate responses between LLMs within the same model family with reasonably well accuracics.
Table 4. Controlling LLM outputs with prompts. An instruction is added to the original prompt to specify the output length and format, Length control limits responses to one paragraph. Format control ensures that responses are in plain text without any format.
Generalization to out-of-distribution responses. We find that our classifiers generalize robustly to responses beyond their training distribution. To evaluate this, we collect re- sponses from instruct LLMs across four diverse datasets: i.e.. UlraChat, Cosmopedia (Ben Allal al., 2024), Lm- sysChat (Zheng et al., 2024), and WildChat (Zhao et al., 2024). These datasets originate from different sources and are designed for various purposes - Cosmopedia is intended for synthetic data generation, LmsysChat and WildChat cap- ture real-world user interactions, while UltraChat consists primarily of synthetic responses. For each dataset, we train a classifier on a group of model responses and evaluate the
The results are shown in Table 4, where "original" means the classification accuracy without interventions. We can see that neural networks still perform excellently for classi- fying LLM outputs applied with length and format control prompts. These findings suggest that LLM characteristies are deeply embedded in the generated text, persisting despite surface-level constraints on length and formatting
Input length of text embedding models. We control the number of input tokens to the text embedding models. Specifically, we truncate each response to a fixed number of tokens in left-to-right fashion. Figure 3 presents the results.
4

---

Idiosynerasies in Large Language Models
Our products feature innovative sustainable materials, such
According to the text, Kal usser
1. Deliver Exceptional Service: The foundation of word-of-mouth marketing consisten isttsttent excellence. Providing top-notch services
1. Deliver E Exceptinna Service Consistently exceed customer expectations
believes traditional cardio ...
Certainly! If you're looking for cheese
Based on the lext provided, here are
altematiies to replace Brie your... ...
key details about Armon Binns' ...
Focus on quality and attention
yverall, while there are challenges,
While many winter sports in the
Ingredients:
Ingredients:
Tanzania making progress....
Pyrenees are similar to those found...
2 (3 oz) packages orange-
boxes orange-flavored Jlllo
Sure! Here's simple guide cooking
This appears
fragment
flavored J ello
can evaporated milk
a juicy salmon fillet...
postry that creates pastoral
1 cup tonic water (this is what
Tonic water
PT
Claude
ChatGPT PT
Claude
(a) characteristic phrases
(b) unique markdown formatting
Figure Example responses from ChatGPT and Claude, showcasing their idiosyncrasies: characteristic phrases (left) and unique markdown formatting (right). For clarity, we highlight each characteristic phrase with underline and model-specific color.
Across three groups of LLMs, the classification task benefits from seeing an increased number of tokens. Intriguingly, for chat APIs and instruct LLMs, we observe around 50% accuracy using only a single text token. This suggests that the initial token in a response contains certain distinctive signals for the classification problem. In Section 3.1, we provide further evidence supporting this observation.
Table shows the results. All sequence embedding models can achieve very high accuracies. The classification perfor- mance improves with more advanced sequence embedding models. Among all methods, LLM2vec demonstrates the best performance, achieving 97.1% % on chat APIs, 96.3% on instruct LLMs, and 87.3% on base LLMs.
3. Concrete Idiosyncrasies in LLMs
100
accuracy (%)
75
In this section, we identify concrete idiosyncrasies in LLMs across three dimensions: words and letters, markdown for- matting elements, and semantic meaning. For each dimen- sion, we apply text transformations to isolate potential id- iosyncrasies and assess their impacts on classification per- formance. We then highlight specific patterns within each dimension that distinguish LLMs.
50
chat instruct base
25
A -
A
0
1
2
128 256 512 input sequence length for the classifier
16 32
64
Figure 3. Ablations on input length of text embedding models. Classification accuracies improve as the text emhedding models capture more context. Performance begins to saturate beyond an imput sequence length of 256. Note that the three lines represent different groups of LLMs and not directly comparable.
3.1. Words and Letters
Text shuffling. To decouple the effects of words and let- ters from other factors, we remove special characters in LLM-generated responses, such as punctuations, markdown elements, and excessive white spaces. This ensures that each response consists solely of words separated by a white space. Additionally, we apply two shuffling strategies to the preprocessed text: word-level and letter-level shuffling. These transformations disrupt the natural order and force the classifier learn patterns from raw text statisties. Table 6 presents the classification results.
Text embedding models. We vary the underlying pretrained embedding models for sequence classification. The default setting we used in previous parts is fine-tuning the LLM22ec embedding models. We consider various generations em- beddings models spanning across architectures and training methods: ELMo (Peters et al., 2018), BERT (Devlin al., 2018), T5 (Raffel et al., 2020), GPT-2 (Radford et al., 2019), and LLM2vec (BehnamGhade et al., 2024). Details on the fine-tuning setting can be found in Appendix A.2.
chat
instruct
base
original
97.1
96.3
87.3
removing special characters
95.1
93.8
75.4
shuffling words
88.9
88.9
68.3
method
chat
instruct
basc
shuffling letters
39.1
38.6
38.9
ML
90.8
91.0
69.8
BERT
91.1
91.5
66.0
Table 6. Classifications with only words and letters. While re- moving special characters and shuffling words have little impact on accuracies, shuffling letters greatly reduces the performance.
90.5
89.8
67.9
GPT-2
92.1
92.3
80.2
LLM2vec
97.1
96.3
87.3
Classifiers trained on responses without special charac- ters achieve accuracies close to those using the original responses, i.e., 95.1% for chat APIs, 93.8% for instruct I.LMs, and 75.4% for base LLMss. Likewise, using word-
Table 5. Different sequence embedding models. LLM2vee achieves the best performance in classifying outputs from vari- ous LLMs among the five embedding models we study.
5

---

Idiosynerasies in Large Language Models
such such as enhance involves certainly
here according to according the text while based here's a summary appears to both when
ChatGP
Claude
the
here
title
based the
certainly
creating
according
overall
to here ssure yes
title how this from
sure
utilize
various
typically
aas
in
GPT-40
Claude
0.12
0.24
0.36
0.12 0.24
0.36 DeepSeek
which
crucial other even here's breakdown
below is in summary for example
Grok
Gemini
where not here here is
here
the
the
the
this
certainly title creating to
certainly title to
ppython relara
key improvements doesn't within essentially also etc
certainly summary may certainly! below know If example DeepS eek
remember
Iyes while
might also not only helps in
here comprehensive step
creating yes
rain
introduction step
creating my
based
Grok
Gemini
0.12 0.24 0.36
0.12 0.24 0.36
0.12 0.24 0.36
Figure 4. Characteristic phrases. We train a logistic regression model on TF-IDF features of chat APIs' outputs and extract the top 10 phrases for each LLM based on the coefficients of these features. We remove common words shared across these LLMs.
Figure 6. First word. We analyze the distribution of the first word in chat APIs' responses, with the top 10 most frequent words for each model. These differences in the first-word usage explain the non-trivial accuracy with only the first word Figure 3.
shuffled responses yields high accuracies comparable to the original ones. Further, we plot the frequencies of several commonly used words from five chat APIs in Figure 5 (left). We observe distinct patterns among models, even for fre- quent English words: Claude has much lower frequencies for words like "the", "and"", "to", and "of" than other chat APIs. These results suggest that special characters and word order are not essential for distinguishing LLMs; word choices reflect substantial idiosyncrasies across models.
phrases inside LLM-generater responses that reflect each model's word choices. Formally, we treat each LLM re- sponse as a document and then extract TF-IDF features on all uni-gram and bi-gram words. We then train a N-way lo- gistic regression model to predict the origin of responses on the extracted features. This simple linear classifier achieves 85. 5.5% 83. 33.7% accuracy on chat APIs / instruct LLMs. close to 95.1 % 993.8% achieved with fine-tuning embedding mod- els on responses without special characters (Table 6).
ChalGPT
Since the coefficients of a logistic regression model provide a natural ranking for its features, we leverage these coef- ficients to highlight important phrases the classification task. Figure 4 presents the top 10 phrases with the largest logistic regression coefficients for each of the five chat API models (excluding the 20 most frequently used words in Fig- ure 4). Notably, these phrases often serve as transitions or emphasis in sentences. For example, ChatGPT likes to gen- erate "such as", "certainly", and "overall", whereas Claude prefers "here", "according to", and "based on".
Claude
0.06
Grak
0.15
Gemini
DeepSeek
0.02
abcdeT gh 1K imopqre LYwwyy
Figure 5. Frequencies of words and letters. The top 20 most frequently used words of M. (left) exhibit distinct patterns for each model, but their letter frequencies (right) are very similar. Results are on the chat API models.
Figure illustrates these characteristic phrases with exam- ple responses from ChatGPT and Claude. While ChaiGPT begins responses with "certainly" and "below is". Claude usually references the original prompt using the phrases like "according to the text" and "based on the text". Moreover, Figure 6 reveals noticeable differences in the distribution of first word choices among chat APIs. Appendix B.5 provides characteristic phrases for other LLMs.
In contrast, shuffling at the letter level results in a substantial drop in accuracy (49%-56%) approaching chance-level per- formance. This indicates that letter-level statistics alone are not sufficient for predicting LLM identities. To qualitatively visualize distinctions in letter distributions across models, Figure 5 (right) shows the frequency distribution of letters in responses generated by chat APIs. Different LLMs share almost identical letter distributions, indicating that letters contribute minimally to idiosyncrasies in LLMs.
3.2. Markdown Formatting
Characteristic phrases. We use Term Frequency- Inverse Document Frequency (TF-IDF) to highlight characteristic
We seek to understand how each LLM formats their re- sponses, particularly in markdown. To this end, we focus on
6

---

Idiosynerasies in Large Language Models
common markdown elements used by LLMs: (I) bold text, (2) italic text, (3) header, (4) enumeration, (5) bullet point, (6) code block. We transform the LLM outputs by retaining only these formatting components while replacing other text with the marker "xxx". Appendix C provides examples of the transformed outputs. Table 7 shows the classification results after this transformation.
3.3. Semantics
Rewriting. One potential reason for the high classification accuracy is the unique writing style (e.g.. word choice, sen- tence structure) of cach LLM. To isolate this factor, we lever- age another LLM (e.g., GPT-4o mini) to rewrite LLM re- sponses. Our rewriting approaches include (see Appendix C for example responses after rewriting):
Paraphrase the above text while maintain- ing the semantic meaning of the original text.
Paraphrasing:
chat
instruct
base
original
97.1
96.3
87.3
Translating: Translate the above text into Chinese
markdown elements only
73.1
77.7
38.5
Summarizing: Summarize the above text in one para-
Table 7. Classifications with only markdown elements. Using markdown elements can achieve high accuracies for chat APIs and instruct LLMs, but marginally better results for base LLMs.
graph.
chat
instruct
basc
original
97.8
96.3
87.3
paraphrasing
91.4
92.2
71.7
translating
91.8
92.7
74.0
Surprisingly, we observe our classifiers achieve high accura- cies of 73.1% for chat APIs and 77.7% for instruct LLMs. However, the classification accuracies with base LLMs' re- sponses are near chance-level guess (25%). This is likely because base L.MMs tend to generate responses in plain text.
summarizing 58.1
57.5
44.7
Table 8. Classifications on rewritten responses. Paraphrasing or translating I.I outputs achieves an accuracy comparable to that using original counterparts. However, summarizing these texts makes the model less capable of predicting LLM identities.
We count the occurrence of a markdown formatting element in each response. We then plot the distribution of these counts over all responses in Figure 7. Each model exhibits unique way to format its responses. For instance, Claude (vellow) has a high density at zero in the bold text and header count distributions, indicating that it generates many responses without bold texts or headers. On the contrary. other LLMs exhibit lower values at zero and thus decorate text with these formatting elements more often.
We show the results in Table 8 (see Appendix B.7 for re- sults using alternative LLM for rewriting). The classifiers trained on paraphrased LLM responses maintain similar ac- curacy levels to those using original responses. Likewise, when using translated text, the classifiers are also able to differentiate between LLMs. These findings suggest that the semantic meanings of words play a more significant role in predicting LLM origins than the exact word choice.
Figure 2b visualizes how ChatGPT and Claude structure their responses in markdown. Interestingly, ChatGPT tends to emphasize each key point within enumerations in bold and highlight a title with markdown headers, but Claude for- mats text with simple enumeration and bullet points. More analysis for other models can be found in Appendix B.6.
Moreover, despite a noticeable accuracy drop (i.e., >38%) with the summarized text, the resulting performance remains well above chance-level guess. This remarkable ability to classify the summarized texts shows the high-level semantic difference in LLM- -aM-raeeedtedd responses.
Open-ended language analysis. In this part. we focus on studying the semantic difference in responses generated by LLMs. We employ another LLM (e.g., ChatGPT) as a judge to provide open-ended, descriptive characterizations for each LLM's outputs. The results with other LLM judges for our language analysis are available in Appendix B.8.
ChalGPT
Clsude
Grok
1.00
0.75
Gemini
0.75
0.50
SeepSeak
0.50
0.25
0.25
Specifically, we present an LLMM judge with two responses generated by different models based on the same prompt- and ask to analyze these responses from different angles (e.g., tone and content). This process is repeated multiple times to gather comprehensive collection of analyses. Fi- nally, we query the LLM judge to summarize these analyses into bullet points that capture the characteristics of each model. The prompts are detailed in Appendix A.3.
bold text count
20
header count
12
1.00
1.00
0.75
0.75
0.50
0.50
0.25
10
10
enumeration count
bullat point count
30
Figure 7. Markdown formatting elements. Each LLM has distinctive distribution of markdown formatting elements.
The results of open-ended language analysis on ChatGPT vs. Claude are shown in Figure 8. For detailed pairwise
7

---

Idiosynerasies in Large Language Models
1. Descriptive and Detailed Tone: Often uses narrative styles with an informative, engaging, or vivid tone.
1. Concise and Straightforward Tone: Generally adopts a more direct, factual, or succinct tone.
Specific and Technical Word: E Empooys descriptive and technical
2. Functional and Clear Word Choices: Prefers simple or action-oriented
vocabulary, enhancing depth and specificity.
language prioritizing clarity and practicality.
Structured and Contextual Opening Lines: Typically begins with
3. Immediate and Direct Opening Lines: Often starts with straightforward statement or summary without extended context.
context-setting or narrative introductions.
4. Markdown Formatting for Organization: Utilizes various markdown
4. Minimal Markdown or List Use: Relies on plain lists or simple
elements like headings, lists, and bold text for clarity,
formatting for quick reference.
Comprehensive and In-Depth Content: O ffers rich detail, focusing on explanations, background, and broader topics.
5. Focused and Summarized Content:
Concentrates on essential points and specific phenomena, avoiding extensive detail.
ChatG PT
Claude
Figure 8. Results of our open-ended language analysis on ChatGPT and Claude, ChatGPT features descriptive language, sophisticated markdown formatting, and in-depth details, while Claude highlights straightforward tone, minimal structure, and summarized content.
comparison of the responses, see Figure in Appendix C.
Prediction of ChatGPT outputs
Prediction of Claude outputs
90%
ChatGPT is characterized by descriptive and detailed re-
60%
sponses in an engaging tone. In contrast, Claude prioritizes simplicity with only key points and straightforward lan- guage. Additional results on chat API models and instruct LLMs are provided in Appendix B.8.
30%
28
31.2
T.2
82
6.6
Grok Gemini DeepSeek Phi-4
ChatGPT Grok
Gemini DeapSBek Phi4
Prediction Grok outputs
Prediction of Gemini outputs
90%
60%-
40.3
30%
1.1
1.2
3:1
22,0
Cha TORude Gemini Geminl Seek Phi-4
ChatiPTCiaud
Gmok DeeppSek Ph14
4. Implications
Prediction of Seep outputs.
Prediction of outputs
90%
76.0
In this section, we explore the broader implications of our framework, regarding synthetic data and model similarity.
60%
46A
30%
16.2
0.2
6.6
0.3
677 CHstGi TCauee Grok Gemini DeepSeek
144
02
Charg PC aude
Grok Gemini
Phi-I
Idiosynerasies via synthetic data. Using synthetic data has become a common practice when training frontier LLMs (Abdin et al., 2024a;b; Liu et al., 2024). We conduct supervised fine-tuning (SFT) on two base LLMs (Llama3. 8b and Gemma2-96 using Ultrachat, i.e., dialogues gener- ated by ChatGPT. After the SFT stage, we train classifier to distinguish between responses from two fine-tuned models. We find that SFT on the same synthetic dataset significantly reduces the classification accuracy from 96.5% to 59.8%, narrowing down the differences between these two models.
Figure 9. Inferring model similarity. We consider 6 L.MMs, in- cluding chat APT models and Phi-4. In each subfigure, we evaluate five- way classifier outputs from the excluded LLM and present the distribution of predicted model origins. There a strong tendency for LLM outputs to be predicted ChatGPT.
source is considered the closest match to the excluded LLM. This process is repeated for each of the N models. For this analysis, we include the open-weight Phi-4 (Abdin et al., 2024b) alongside 5 chat API models. Note that Phi-4 uses substantial amount of synthetic data in its training.
In addition, we generate responses from Llama3.1-8P and Gemma2-9B in instruct LLMs using UltraChat prompts. Then we fine-tune Qwen2.5-7B base LLM on each set of re- sponses respectively. Interestingly, responses from the two resulting fine-tuned models can be classified with 98.9% accuracy, suggesting that each fine-tuned model retains the unique characteristics in its SFT data. These findings sug- gest that training with synthetic data can propagate the id- iosyncrasies in the source model.
Figure 9 shows the results. Intriguingly, for Claude, Grok, and Gemini, we observe a strong tendency for their outputs to be classified as ChatGPT. For instance, when Grok is the excluded model, 82.8% of its responses are classified as ChatGPT. In addition, responses from ChatGPT and DeepSeek are frequently identified as coming from Phi-4, with 55.9% and 76.0% of their responses, respectively. In turn, most of Phi-4's outputs are classified as originating from ChatGPT or DeepSeek.
We note that this behavior is not limited synthetic datasets; in fact, training on different datasets often leads to idiosyn- cratic patterns in model outputs (Mansour & Heckel, 2024).
Robust evaluation of LLMs. Our findings reveal a poten- tial vulnerability in widely used LLM evaluation method- ologies. It has become common strategy to incorporate human judgement in evaluating L.MSs, for instance, Chatbot Arena (Chiang et al., 2024). It is a voting-based leaderboard where users submit preferences of the responses from two randomly chosen models. This benchmark has gained sig- nificant traction and is now a key reference point for frontier
Inferring model similarity. Our framework offers a quan- titative approach for assessing similarities between propri- etary and open-weight LLMs. Given set of N LLMs, we omit one model and train a classifier on responses from the remaining N- 1 models. We then evaluate which L.LM the classifier associates the responses of the excluded model with. The model that is most frequently predicted as the
8

---

Idiosynerasies in Large Language Models
model development. However, exploiting the idiosyncratic property of L.LM outputs, a malicious attacker can identify the model behind the candidate responses and consistently vote for the target model, thereby manipulating the leader- board rankings. Concurrent work by Huang et al. (2025) has demonstrated the feasibility of this attack in simulation. We hope our work brings attention to potential weaknesses in current evaluation pipelines, as they can misguide model development and optimization efforts (Singh et al., 2025).
human writing. Other works have examined stylometric features in Al-generated content (Zaitsu & Jin, 2023). Most recently, Chakrabarty et al. (2024) studied the Auman-AI alignment in the writing process, finding that professional writers can effectively identify and edit undesirable idiosyn- crasies common in LLM LLM-generated text. Our work seeks to provide a deeper and principled understanding behind the observed idiosyncrasies from our framework in Figure 1.
Understanding differences between distributions. A line of research (Dunlap et al., 2024; Zhong et al., 2024) has used foundation models to describe qualitative differences between pairs of data distributions (e.g.. image datasets). Gao et al. (2025) and Cai et al. (2025) conducted auditing studies on L.LM APIs to detect instances of model substi- tution, such as watermarking and quantization. The most relevant work to us is Dunlap et al. (2025), which proposed VibeCheck to understand user-aligned traits in LLM out- puts. They found that LLMs often vary in styles, such as being more formal or friendly. In contrast, our work aims to identify broader generalizable patterns to interpret the high classification performance.
5. Related Work
Dataset classification. Torralba& & Efros (2011) introduced the "Name That Dataset" experiment a decade ago to high- light the bias present in visual datasets at that time. Re- cently, Liu & He (2025) revisited this problem (termed dataset classification) and found that current large-scale, supposedly more diverse visual datasets are still very biased. Zeng et al. (2024) further identified structural and semantic components in images as key contributors to these biases. You et al. (2024) and Mansour & Hockel (2024) applied the dataset classification framework to study bias in syn- thetic images and LLM pretraining datasets, respectively. While the synthetic task shown in Figure 1 is conceptually similar to dataset classification, we focus not on training datasets but on the distinctive characteristics inherent to LLMss. Shortly after our initial arXiv preprint, Suzuki et al. (2025) demonstrated that subtle variations in the training process can result in distinguishable outputs across LLMs.
6. Conclusion
We demonstrate the presence of idiosyncrasies in Large Language Models (LLMs) and investigate a synthetic task designed to quantify their extent. We find that simply fine- tuning pretrained text embedding models on LLM outputs leads to exceedingly high accuracy in predicting the origins of the text. This phenomenon persists across diverse prompt datasets, LLM combinations, and many other settings. We also pinpoint concrete forms of these idiosyncrasies within LLMs. We hope our work encourages further research into understanding idiosyncrasies in LLMs.
Human vs. machine-generater texts. Many prior works have studied the problem of determining whether text is authored by human or Al system (Mitchell et al., 2023; Wu et al., 2023; Cai et al., 2025). Model-free ap- proaches typically use linguistic properties such as n-gram frequencies (Badaskar et al., 2008; OpenAI, 2019), en- tropy (Lavergne et al., 2008; Gehrmann et al., 2019) or negative probability curvature (Mitchell et al., 2023; Bao et al., 2024). Other works leverage neural network features to perform this task, such as fine-tuning BERT (Uchendu et al., 2021; Ippolito et al., 2020). Russell et al. (2025) found that experienced chatbot users are good at distinguishing between AI- and suman-written articles. Neural authorship attribution (Uchendu et al., 2020; Antoun et al., 2023; Huang et al., 2024) seeks not only to identify machine-generated text but also to attribute it to specific text generators. In this work, we focus the distinguishability between LLMs rather than between AI vs. human.
We conclude by outlining several directions for future work:
It is worth investigating whether our observations gener- alize to LLMs beyond the Transformer architecture, such as state space models (Gu & Dao, 2024) and diffusion- based language models (Nie et al., 2025).
Understanding how the training process results in these idiosyncrasies remains an important open question.
Third, our setup does not consider scenarios where the list of source L.MSs could be large and even unknown be- forehand, which require further investigation. Our initial results suggest it is promising: a 10-way classification problems involving 5 chat APIs and 5 instruct LLMs including Phi-4 achieves 92.2% accuracy.
Characteristics of machine-generatee texts. Beyond de- tection, it is also important to understand the distinctive properties of have long served basic signal for such purpose. McGov- et al. (2024) showed that L.M outputs contain unique lexical and syntactic fingerprints that distinguish them from
It is interesting to examine how these idiosynerasies relate to model distillation (Hinton et al., 2015), a tech- nique that has become increasingly prevalent in practice.
texts. N-gram frequencies
9

---

Idiosynerasies in Large Language Models
Acknowledgments
Ben Allal, L.., Lozhkov, A., Penedo, G., Wolf, T., and
Werra, L. Cosmopedia. huggingface.co cos.p/ediacoooppaa.....ooo. 2024.
We thank Zekai Wang for valuable discussions. Mingjie Sun was supported by funding from the Bosch Center for Artificial Intelligence.
/ /ttggggggaaceetg///a//aa
datasets/
Cai, W. Shi, T., Zhao, X., and Song, D. Are you getting what you pay for? auditing model substitution in llm apis.
Impact Statement
arXiv preprint arXiv:2504.0471: 2025.
Our study investigates the cistinguishability of LLMs. On the positive side, our results offer insights into LLM behav- iors that enhance transparency, accountability and prove- nance in tracking AI generated content. This has valuable applications such as combating misinformation, and ensur- ing responsible use of LLMs. However, our results highlight challenges in model training. Reliance on synthetic data or model distillation risks propagating biases and potentially infringing on the intellectual property of proprietary models.
Chakrabarty, T., Laban, P., and Wu, C.-S. Can ai writing be salvaged? mitigating idiosyncrasies and improving human-ai alignment in the writing process through edits. arXiv preprint 2409 14509, 2024.
Chiang, W.-L.,, Zheng, L.,, Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, Jordan, M. I., Gonzalez, J. E., and Stoica, I. Chatbot arena: An open platform for evaluating 1lms by human preference. In ICML, 2024.
DeepSeek-Al. Deepseek-33 v3 technical report. arXiv preprint arXiv:2412.19437, 2024.
References
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan,
Devlin, J., Chang. M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In NAACL,
A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., Benhaim, A., Bilenko, M., Bjorek, J., Bubeck, S., Cai, M., Cai, Q. Chaudhary, V., Chen, D., Chen, D., Chen, W., Chen, Y.-C., Chen, Y.-I.,, and et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.142199 2024a.
Ding, N., Chen, Y., Xu, B., Qin, Y., Zheng, Z., Hu, S., Liu, Z., Sun, M., and Zhou, B. Enhancing chat language mod- els by scaling high-quality instructional conversations. In EMNLP, 2023.
Abdin, M., Aneja, J.. Behl, H., Bubeck, S., Eldan, R., Gu- nasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, P., Lee, J. R., Lee, Y. T., Li, Y., Liu, W., Mendes, C. T., Nguyen, A., Price, E., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Wang, X., Ward, R., Wu, Y., Yu, D., Zhang, C., and Zhang, Y. Phi-4 technical 2024b.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang. A., Fan, A., and et al. The llama 3 herd of models. arXiv preprint arXiv: 2:X7v22400.22 407.21183, 2024.
report. arXiv preprini 24
Dunlap, I.,, Zhang, Y., Wang, X., Zhong, R., Darrell, T., Steinhardt, J., Gonzalez, J. E., and Yeung-Levy, S. De- scribing differences in image sets with natural language. In CVPR, 2024.
Anthropic. Claude-3.5-sonnet www.anthropic.com news/cllas claude- -3 -onneet, 2024.
Antoun, W., Sagot, B., and Seddah, D. From text to source: Results in detecting large language model-generated con- tent. arXiv preprint 2xXiv22309.13322 2023.
Dunlap, L., Mandal, K., Darrell, T., Steinhardt, J., and Gon- zalez, J. E. Vibecheck: Discover and quantify qualitative differences large language models. ICLR, 2025.
Badaskar, S., Agarwal, S., and Arora, S. Identifying real or fake articles: Towards better language modeling. In IJCNLP, 2008.
Gao, I., Liang, P., and Guestrin, C. Model equality testing: Which model is this api serving? In ICLR, 2025.
Gehrmann, S., Strobel, H., and Rush, A. M. Gltr: Statistical detection and visualization of generated text. In ACL, 2019.
Bao, G., Zhao, Y., Teng, Z.,, Yang, L.,, and Zhang, Y. Fast-detectgpt: Efficient zero-shot detection of machine- generated text via conditional probability curvature. In ICLR, 2024.
Google. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.05533 2024.
BehnamGhader, P., Adlakha, V., Mosbach, M., Bahdanau, D., Chapados, N., and Reddy, S. Llm2vec: Large lan- guage models are secretly powerful text encoders. In COLM, 2024.
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. In COLM, 2024.
10

---

Idiosynerasies in Large Language Models
Hinton, G., Vinyals. O., and Dean, J. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.02333 2015.
Mitchell, E., Lee, Y., Khazatsky, A., Manning, C. D., and Finn, C. Detectgpt: Zero-shot machine- -enerrrted text detection using probability curvature. In ICML, 2023.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L.,, and Chen, W. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J.,
Lin, Y.. Wen, J.-R., and Li, C. Large language diffusion models. arXiv preprint arXiv:2502.09992 2025.
Huang, B., Chen, C., and Shu, K. Authorship attribution in the era of llms: Problems, methodologies, and challenges. arXiv preprint arXiv:2408.08940 2024.
OpenAL. Gpt-2 output dataset detection. gi hub. com/ openai http://gott-oottttttttttee-eee-e 2019.
OpenAL. Gpt-4o. openaa.comm/nndex//elllooo 40/, 2024.
Huang, Y., Nasr, M.. Angelopoulos, A., Carlini, N., Chi- ang, W.-L.,, Choquette-Choo, C. A., Ippolito, D., Jagiel- ski, M., Lee, K., Liu, K. Z., Stoica, I., Tramèr, F., and Zhang, C. Exploring and mitigating adversarial manip- ulation voting-based leaderboards. arXiv preprint arXiv:2501.07499 2025.
Penedo, G., Kydlíček, H., Allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. The fineweb datasets: Decanting the web for the finest text data scale. arXiv preprint arXiv:2406.1733 17575, 2024.
Peters, M. E., Neumann, M., Iyyer. M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. Deep contextualized word representations. In NAACL, 2018.
Ippolito, D., Duckworth, D., Callison-Burch, C., and Eck, D. Automatic detection of generated text is easiest when humans are fooled. In ACL, 2020.
2018.
Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J.. Dang, K., Lu, K., Bao, K., Yang. K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., and Qiu, Z. Qwen2.5 technical report. arXiv preprint 2rXiv:2412.15111 2024.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C.,, Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, arXiv preprint arXiv:2310.0663 2023.
Krishna, K., Song. Y., Karpinska, M., Wieting, J., and Iyyer, M. Paraphrasing evades detectors of ai- ai-generated text, but retrieval is an effective defense. In NeurIPS, 2023.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, Language models are unsupervised multitask learners. Technical Report, 2019.
Lavergne, T., Urvoy, T., and Yvon, F. Detecting fake content with relative entropy scoring. In International Confer- ence on Uncovering Plagiarism, Authorship and Social Software Misuse, 2008
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with unified text-to-text transformer. JMLR, 2020.
Lin, C.-Y. ROUGE: A package for automatic evaluation of summaries. In ACL, 2004.
Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussonot, L., Mesnard, T., Shahriari, B., and et al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.00 0044000.01110 00///1 18, 2024.
Liu, R., Wei, J., Liu, F., Si, C., Zhang, Y., Rao, J., Zheng, S., Peng, D., Yang, D., Zhou, D., and Dai, A. M. Best practices and lessons learned on synthetic data. In COLM, 2024.
Russell, J., Karpinska, M., and Iyer, M. People who fre- quently use chatgpt for writing tasks accurate and robust detectors of ai-generated text. In ACL, 2025.
Liu, Z. and He, K. A decade's battle on dataset bias: Are there yet? In ICLR, 2025.
Sadasivan, V. S., Kumar, A., Balasubramanian, S., Wang, W., and Feizi, S. Can ai-generated text be reliably de- tected? TMLR, 2025.
Mansour, Y. and Heckel, R. Measuring bias of web-filtered text datasets and bias propagation through training. arXiv preprint arXiv:2412.02855 2024.
Singh, S., Nan, Y., Wang, A., D'Souza, D., Kapoor, S., Üstün, A., Koyejo, S., Deng, Y., Longpre, S., Smith, N. A., Ermis, B., Fadace, M., and Hooker, S. The leader- board illusion. arXiv preprint 200799504.20879, 2025.
McGovern, H., Stureborg. R., Suhara, Y., and Alikaniotis, D. Your large language models are leaving fingerprints. In ICCL. Workshops, 2024.

---

Idiosynerasies in Large Language Models
Sun, C., Qiu, X., Xu, Y., and Huang, X. How fine-tune bert for text classification? arXiv preprint 2019.
Suzuki, T. Ri, R., and Takase, Natural fingerprints of large language models, 2025.
Torralba, A. and Efros, A. A. Unbiased look at dataset bias. In CVPR, 2011.
Uchendu, A., Le, T., Shu, K., and Lee, D. Authorship attribution for neural text generation. In EMNLP, 2020.
Uchendu, A., Ma, Z., Le, T., Zhang, R., and Lee, Turing- bench: A benchmark environment for turing test in the age of neural text generation. In EMNLP, 2021.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, Attention is all you need. In NeurIPS, 2017.
Wu, J., Yang, S., Zhai, R., Yuan, Y. Wong, D. F., and Chao, L. S. A survey on Ilm-generated text detection: Necessity, methods, and future directions. arXiv preprint arXiv:2310.14722 2023.
xAL. Grok-2 beta release. x.
2024.
You, Z., Zhang, X., Guo, H., Wang, J., and Li, C. Are im- ages indistinguishable to humans also indistinguishable to classifiers? arXiv preprint 2rXiv:2405.18022 2024.
Zaitsu, W. and Jin, M. Distinguishing chatgpt(-33 -4)-
generated and hurtnewritten papers through japanese stylometric analysis. arXiv preprint 23Xi::2304.05334, 2023.
Zeng, B., Yin, Y., and Liu, Z. Understanding bias in large- scale visual datasets. In NeurlPS, 2024.
Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., and Deng, Y. Wildchat: 1m chatgpt interaction logs in the wild. ICLR, 2024.
Zheng, L., Chiang, W.-L., Sheng, Y., Li, T., Zhuang, S., Wu, Z., Zhuang, Y., Li, Z., Lin, Z., Xing, E. P., Gonzalez, J. E., Stoica, I., and Zhang, H. Lmsys-chat-Imm: A large-scale real-world Ilm conversation dataset. In ICLR, 2024.
Zhong, R., Wang, H., Klein, D., and Steinhardt, J. Explain- ing datasets in words: Statistical models with natural language parameters. NeurlPS, 2024.
12

---

Idiosynerasies in Large Language Models
Appendix
A. Implementation Details
A.I. Response Generation
We report our procedure for generating responses from chat APIs, instruct LLMs, and base LLMs. For chat APIs, we access stable version of each model, including EPTT-4--2224-08 and DeepSeek-Chat through its official API between November 28, 2024, and February 2025, generating responses with their default sampling setting. For instruct LLMs, use greedy decoding sample outputs. For base LLMs, set the temperature to T = 0.6 and apply repetition penalty of 1.1 to avoid repetitive completions.
3.5ude-3.-Sommet-2000
Grok-Bete Gemini- 1.5-Pro-002,
A.2. Training Setup
In this part, describe fine-tuning process using the text embedding models LLM responses. We use the first 512 tokens of each generated response for training and evaluation. To perform sequence classification, we add a linear layer as the classification head on top of each text embedding model. For ELMo, BERT, LLM2vec, this layer is applied to the average embeddings over all tokens sequence. For T5 and GPT-2, we follow the original setups (Radford et al., 2019; Raffel et al., 2020) and apply the head on the output the last token.
For smaller text embedding models, such ELMo, BERT, T5, and GPT-2, we fine-tume the entire model along with the classification head, searching over base learning rates (3e-3, 1e-3, 3e-4, le-4, 3e-5, 1e-5, 3e-6, le-6). For the largest LLM2vec model, we employ the parameter--eficiiet LoRA (Hu al., 2022) fine-tuning method with rank of 16, LoRA of 32, dropout rate of 0.05, and base learning rate of 5e-5. Table 9 details our basic training recipe.
config optimizer weight decay
value AdamW 0.001
optimizer momentum
B1. 82 0.9, 0.999
training epochs
3
batch size
8
learning rate schedule
cosine decay linear
warmup schedule
warmup ratio
10%
gradient clip
0.3
Table 9, Our fine-tuning recipe.
13

---

Idiosynerasies in Large Language Models
A.3. Prompts for Open-ended Language Analysis
We detail the procedures of our open-ended language analysis in Section 3.3. Given the same input, we sample pair of responses from two LLMs and present them, along with analysis prompt (see Figure 10a), to an LLM judge for comparison. To avoid the LLM judge exploiting any prior knowledge of the models, an index distribution. This process is repeated for 35 response pairs, yielding set of detailed analyses. Finally, we use the summarization prompt (see Figure 10b) to distill these analyses into 5 bullet points that characterize the idiosyncrasies of each model.
anonymize model identities using
Below 35 summaries that compare the unique features text distributions paint by point.
Here are some samples from different distriuttions of text
(Analys 1} (Analys 2}
Sample from distribution 0. 23 (ext sample 1}
Sample from distribution 0. 23 text sample 2}
(Analys 35}
Determine the unique characterin ris tics of the 2 distribttoos, and summarize them with bullet points for each, Each bullet point should analyze the senponse from the same angle and can be easily compared by human. Focus on key aspects that differentiate the distributons. Focus the overall tine, specific word choices, opening lines, markdown formatting, an content. Oulput only 5 bullet points per distribution, additonal text.
Condense all 35 summaries each summary evaluates the overall tone, specific word choices, opening lines, markdown formatting. and content Provide exactly five bullet points for each distribution, one for each feature. Each bullet point should consist of descriptive title chara cterizing the feature and short sentence explaining connceell. E Emphasize recurring and significant features, remove the redundant ones, and ensure the for easy human ca comparison. Output only bullet points per distribution, no additional text.
single summary.
how
sentences clearly differentiste the distributions
(a) analysis prompt
(b) summarization prompt
Figure 10. Prompts in our open-ended language analysis.
B. Additional Results
B.1. Controlled Studies
We present additional studies on our classification framework, to complement our results in Section 2.2.
Sampling methods. We consider outputs when sampled using different decoding strategies. Specifically, we use four widely used sampling methods: greedy decoding, temperature softmax. top-k, and top-p sampling. For each method, we generate of responses from the same LLM. We then fine-tune the LLM2vec embedding model to predict the sampling method responsible for each response.
greedy
softmax
top-k
top-p
greedy 59.6
softmax
top-k
58.2
50.0
top-p
52.9
51.0
52.1
Table 10. Classifications with different sampling methods. Distinguishing responses generated by the same model using different sampling strategies is only marginally better than chance accuracy. The results are on Llama3.1-8h instruct model's responses.
Table 10 presents the results for all pairs of sampling methods. Notably, the accuracy of distinguishing between responses generated by the same LLM remains relatively low, with the highest accuracy across all configurations being 59%. Furthermore, in a more fine-grained 5-way classification task distinguishing softmax sampling at five different temperatures (T = 0, 0.25, 0.5, 0.75, 1), obtain an accuracy of 37.9 9%, only marginally better than the random chance level of 20%. These results suggest that outputs from the same LLM are not easily separable based on decoding strategies.
Training data size. We vary the number of training samples generated by LLMs and train the classifier with the same total number of iterations. We present the results in Figure 11. The classification performance increases with more training samples. This trend holds consistently across chat APIs, instruct LLMs, and base LLMs. Furthermore, as few as 10 samples, the classifier achieves non-trivial accuracy (e.g., 40.3% on chat APIs), surpassing 20% chance-level guess.
14

---

Idiosynerasies in Large Language Models
100
accuracy (%)
75
50
chat
instruct
25
base
10
100
1K
10K
# training samples
Figure I.. Different numbers of training samples. Our sequence classifiers benefit from more training samples. The classification performance converges when using about 10K training samples.
B.2. Text Similarities
Here we use text similarity metries quantify differences between LLM outputs. We consider three standard metries ROUGE-1 (Lin, 2004), ROUGE-L (Lin, 2004), and BERTScore (Devlin et al., 2018) to measure lexical and semantic similarity. We compute the mean Fl-score for each metric across all response pairs generated by any two different chat API models given the same prompt. For comparison, we also measure the similarity between responses sampled within the same model. As shown in Table 11, responses from different LLMs exhibit lower text similarities than those from the same model.
across LLMs
within an LLM
0.499 ROUGE-L
ROUGE-1
0,660
0.256
0.414
BERTScore" 0,220
0.482
Table 11. Text similarity scores, We evaluate the text similarity of L.M outputs using ROUGE-1, ROUGE-L, and BERTScore. Responses from different LLMs exhibit low lexical similarity.
B.3. Confusion Matrix
In Figure 12, we present the confusion matrix for the N-way classifiers that trained on responses generated by chat APIs, instruct LLMss, and base LLMs, respectively. The results demonstrate that our classifiers accurately predict the origin of LLM-generated responses, with minimal confusion between different LLMs.
Grak Claude ChatoPT
96.1%
0.2%
1.0%
0.6%
2.1%
Liama
96.4% %
0.1%
1.3%
2.2%
Liama
80.3%
0.6%
16.2%
2.9%
99. 59.5%%
0.0%
0.1%
ground-ruuth Genmra
ground Inuh Genma
ground-rruty tanddtmitt
1.0%
0.1% 9
Inth
1.2%
2.4%
0.22
97.0%
0.0%
Deepseek Gemini
2.2%
0.2%
94.9% 9
2.7%
20.9%
1.8%
75.2%
2.1%
0.8%
0.0%
0.6%
98.6%
0.0%
Misrral
Mistral
3.8%
0.3%
0.7%
0.0%
95.2%
3.1% %
0.0%
2.4%
94.5%
4.3%
1.9%
1.7%
92.1 1%
ChaiGPT
Claude
Grak prediction
Gemini
Deepseck
Liama
Gemma
Owen
Mistral
Liema
Gemma
Owen
Mistral
prediction
prediction
(a) chat APIs
(b) instruct LLMs
(c) base LLMs
Figure Confusion matrices for N-way classifiers on three groups LLMs: chat APIs, instruct LLMs, base LLMs.
B.4. Words and Letters
Figure 13 presents the frequencies the 20 most commonly used words (left) and all English letters (right) across instruct and base LLMs. Consistent with our observations in Section 3.1, we find notable differences in the distribution of commonly used words between these models, such as "the", "and", "to".. In contrast, the letter distributions are nearly identical.
15

---

Idiosynerasies in Large Language Models
0.06
Liama
0.15
0.06
Gerria
Owen
Liama
0.15
0.33
Misiral
0.10
0.02
0.05
abodat
imnopq istuvwxy2
ono Dig from
fightghh imopgr tuwwyy
(a) instruct LLMs
(b) base LLMs
Figure 13, Word and letter frequencies in instruct and base LLMs.
B.5. Characteristic Phrases
We provide additional results for characteristic phrases presented in Section 3.1. We follow the same methodology in Figure 4 to extract characteristic phrases of instruct and base L.Mss. Specifically, we train four-way logistic regression classifier on the TF-IDF features of their responses and use the coefficients to select important phrases of each model.
As shown in Figure 14, each instruct LLM contains quite distinct characteristic phrases. For example, Llama frequently employs terms "including" and "such as" to introduce specific examples in the output, whereas Gemma tends to engage with users using phrases "let me" and "know if". In contrast, the extracted phrases from base LLMs are less distinctive, primarily consisting of common words such "the", "to", and "you".
Figure 15 illustrates the distribution of first word choices in instruct and base LLMs. Similar to chat APIs (Figure 6), instruct LLMs display varied distributions. However, base LLMs exhibit substantial overlap in their most frequent first words, e.g., "the", "and", "of", "to", and "in".
including the following example use such use case which explanation that
essentially know if know let me here's e breakdown
cerlainly below certainly! here certa nly! below enhance delailed
Ulle abso kuttly
and strong h2
5
the and are to
also he ip Lill comprehensive
the
blockquote
you
breakdown
there here after
specific might
specific remember
can broader
an however during Milstral
to
you your
2020 summary
for
throughout
mlama
eemma
Qwen
Llama
Gemma
Qwen
Mistral
(a) instruct LLMs
(b) base LLMs
Figure 14, Characteristic phrases for instruct' and base LLMs.
Llama
Gemma
Llama
Gemma
here the
the
the
the
here
and
according based
1o
of
as
of
a5
please you okay
is
it
s
litle
for
this
comprehensive
python according
we
for
this
we
0.12
0.24
0.36
0.12
0.24
0.36
0.02
0.04
0.06
0.b8
0.02
0.04
0.06
0.08
Qwen certainly
Mistral
Qwen
Mistral
title
the
the
the
the
and
and
in title to creating based comprehensive according sure
in
Df
to
to here absolutely yes based subject creating
to
of
nn
1s
this
for
for
with
you
0.12
0.24
0.36
0.12
0.24
0.36
0.02
0.04
0.06
0.b8
0.02
0.04
0.06
0.08
(a) instruct I.IMs
(b) base I.I.Ms
Figure 15. Distribution of first word choices instruct and base LLMs.
IIn Llama of instruct L.LMs, the phrase "explanation the" corresponds markdown header bold text for "explanation" followed by a new sentence starting with "the".
16

---

Idiosynerasies in Large Language Models
B.6. Unique Markdown Formatting
In this part, we provide additional results for the analysis of markdown formatting presented in Section 3.2. Figure 16 illustrates the distribution counts of six markdown formatting elements across different models. For both chat API models (Figure 16a) and instruct LLMs (Figure 16b), observe distinct differences in the usage of bold texts, headers, enumerations, and bullet points, while italic texts show less variation. Intriguingly, Gemini uses much more italic texts (a lower density at zero in the italic text) than other chat APIs, where similar observations can be found on Gemma2.
ChatGPT
Claude
1.00
Grok
1.00
1.00
0.75
Gemini
0.75
0.75
0.50
DeepSeek
0.50
0.50
0.25
0.25
0.25
10
10
bold text count
20
italic text count
20
header count
12
30
30
18
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0
0
0
enumeration count
bullet point count
code block count
30
18
(a) chat APIs
Llama
1.00
Gemma
1.00
1.00
0.75
Qwen
0.75
0.75
0.50
Mistral
0.50
0.50
0.25
0.25
0.25
0
10
10
bold text count
20
italic text count
20
header count
12
30
30
18
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0
10
10
enumeration count
bullet point count
20
code block count
12
30
30
18
(h) instruct L.MSs
Figure 16. Markdown formatting elements for chat APIs (top) and instruct L.LMs (bottom).
17

---

Idiosynerasies in Large Language Models
B.7. Rewriting LLM outputs
In Section 3.3, we used GPT-4o-mini rewrite LLM outputs. Here, we present results using an alternative model: Qwen2.5- 7B-Chat As shown in Table 12, our observations remain consistent, indicating that our findings are robust to the choice of LLM used for rewriting.
LLM for rewriting
original
paraphrase
translate
summarize
GPT--4-mini Qwwe2.5577-Instrrccl
97.8
93.6
93.9
63.7
97.8
92.6
94.3
71.5
Table 12. Classification accuracies on rewritten responses. The results are on Chat API responses.
B.8. Open-ended Language Analysis
Ablation on LLM judges. Here we demonstrate our findings in Figure 8 of Section 3.3 remains consistent under several LLM judges. Specifically, we change the LLM judge from ChatGPT to Claude, Grok, and Gemini. We show the results in Figure 17. Regardless of the choice of LLM judges, our language analysis reveals that ChatGPT often uses detailed explanations and complex formatting structures, whereas Claude emphasizes key contents without extensive elaboration.
Tone: Authoritatve and formal, using academic language with detailed explanations and complex sentence structures
Tone: Analytical and cautious, frequently referencing source material and acknowledging limitations
Word Choice: Rich descriptive vocabulary with technical terms, metaphors, and lative language ("renowned," "instrumental," "exquisite")
Word Choice: Simpler. more practical vocabulary focused on clarity and accessibility rather than style
Opening Style: Direct statements enthusiastic "Certainly!" responses that establish context and preview co content
Opening Style: Meta-references like "According to the text" explicit disclaimers about content limitations
Markdown Usage: Extensive formatting with bold headers, nested bullet points, and hierarchical section markers
Markdown Usage: Minimal formatting with basic bullet points or simple numbered lists
Content Structure: Comprehensive paragraphs with detailed explanations, examples, and flowing transitions between ideas
Content Structure: Concise buille points and stalements organized in categorical lists without extensive elaboration
ChatGPT
Claude
Claude as the LLM judge.
Tone: Warm and Informative Uses an inviting, educational tone to engage and imform the reader.
Direct and Practical Often concise, focusing actionable insights or stra trrighhioowwrd information.
Word Choices: Descriptive and Detailed Employs rich, technical, or emotive language enhance understande inderstanding evoke emotion.
Simple and Functional P refers straightforward language, emphasizing utlliyy over emotional detailed descriptions.
Word Choices:
Opening Lines: Contextual and Engaging Provides context
Opening Lines: Immediate and Focused J umps directly tapic
narrative hook to draw readers in.
key points w thout much preamble.
Markdown Formatting: Structured and Emphasized Extensive use of headings, lists, and bold text for clarity and organization.
Markdown Formatting: Minimalist- Uses basic markdown elements like bullet simple lists for clariy.
Content: Comprehensive and Insightful Focuses on in-depth explanation, examples, and broad implications.
Content: Succinct and Focused rovides essential Information depth, often aiming for quick readabilit or practicolity.
ChatGPT
Claude
(b) Grok as LLM judge.
Formal Tone: Often adopts formal, academic, instructional tone, prioritizing clarity and detail.
Informal Tone: Generally lly uses a more informal, conversational, concise tone, prioritizing brevity and access
Precise Diction: E Employs specific, descriptive, and sometimes technical vocabulary relevant the subject
Simplified Language: Favors simpler, more direct vocabulary, avoiding jargon and focus ing on key information.
Typically lly begins with introductory statements, context-setting, or clear purpose statements.
Contextual O peninggs:
Direct Openings: Often
directly with main point summary
key information.
Structured Formatting: Frequently utlizes headings, lists, bolding, and other markdown features for organization and emphasis.
Minimal Formatting: Uses less markdown fo formatting, relying primarily basic lists, builet points, or plain text.
Comprehensive Content: Provides detailed explanations, examples, and in-depth information, often covering broader scope.
Concise Content Focuses on summarizing key points, offering practical advice, providing concise overviews.
CT
Claude
(c) Gemini as the L.M judge.
Figure 17. Results of our open-ended language analysis ChatGPT and Claude with different L.MM judges.
18

---

Idiosynerasies in Large Language Models
Open-ended language analysis results other LLMs. In Section 3.3, we presented the results of open-ended language analysis for ChatGPT and Claude. Here, extend analysis other chat API models and instruct L.Mss. The full results are shown in Figure 18, where we use ChatGPT as the LLM judge to compare responses generated by two models within the same category (chat APIs / instruct LLMs). Our analysis highlights several interesting characteristics of each model. For example, Grok's responses tend to feature rich language and comprehensive content, whereas Gemini's outputs are more concise with direct openings.
Descriptive Tone: Emphasizes detail, narrative, or objective perspectives, often exploring themes w itth depth and nuance.
Concise Tone: Often adopts a straightfo rward, direct. or expressive approach focusing on succinctness and dramatic tension.
Rich Language: incorporates emative, analytical, or descriptive language to enhance understanding
Specific Language: Utilizes precise or vivid language to convey themes like tension, urgency, and significant events outcomes.
engagement.
Detailed Openings: Introduces the subject with context. thematic ele elements, guiding points, establishing comprehensive view. Structured Formatting: Utilizes formatting elements like headings, lists, and bold text organize information clearty.
Direct Openings: Typically begins with immediate context action, setting focused tone premise from the start
Simple Formatting: R Relles mainly on plain text or minimal formatting to maintain clarity and directness.
Comprehens Content: Delivers in-depth analysis or exploration of subjects, highlighting diverse factors broader implications.
Core Content: Centers on key insights or motivations, providing essential information narratives with limited embellishment.
Grok
Gemini
(a) chat APIs
Tone Characterization: O Otten formal, factual, and exacting, emphasizing detailed analysis and structured presentations.
Tone Charac tharacttrizaaion:: Generally more conversational, engaging, and accessible, often blending storytelling or personal insights.
Word Choices Depiction: Technical and precise, leveraging industry- specific academic te terminoloyy convey depth credibility.
Word Choices Depiction: S simple, expressive, and often emotive language, creating relatability and narrative flair.
Opening Line Initiation: Frequently starts directly with context, setting immediate formal informative tone.
Opening Line Initiation: Begins engaginglyy often with context or an emational hook, capture the reader's interest.
Markrlown Use: Prefers structured formats with headings, bullet points, or numbered lists to enhance clarity.
Markdown Use: Employs varied markdown elements like bullet points emphasis for readability and structure.
Content Focus: Delivers comprehensive, detailed content w substantial data, examples, or specific tactics.
Content Focus: Centers on overarching concepts, insights, and experiences, often prioritizing engagement over exhaustive detail.
Llama
Gemma
Detailed and A Analytical Tone: The tone is often formal, focusing on analysis, exploration, organized narratives. Descriptive Formal
Concise and Direct Tone: The tone is generally straightforward, practical, and somelimes conversational, focusing on direct explanations. Simplified and Emotive Word Choices: Prefers straightforward language, accasionally prioritizing emational or motivational impact. Immediate 0 Openngg Lines: Begins directly with subject matter actions,
Choices: technical and descriptive
voca bulary that highlights complexity and depth.
Comprehensive Opening Lines: S Starts often with broad context detailed introductions, setting formal framework.
often in conversational or narrative styles.
Structured Markdown Formatting: U Utiliess headers, sections, and lists to enhance readabllity and organization.
Minimalist Markdown Formatting: Uses simpler formatting with lists or plain text maintain clarity without embellishments.
Content Depth Exploration: Provides comprehensive narratives with detailed dis cussions, focusing systematic approaches and analysis.
Summary Focused Content: Concentrates key points with succinct explanations, prioribzing clarity and accessibility over comprehensive detail. Mistral
Qwen
(b) instruct LLMs
Figure 18. Additional results of our open-ended language analysis on chat APIs (top) and instruct LLMs (bottom).
19

---

Idiosynerasies in Large Language Models
C. Response Demonstrations
In this part of the appendix, we present examples of LLM responses. Table 13 and 14 illustrate responses before and after our prompt-level interventions (Section 2.2). Table 15 shows the transformed responses when only markdown elements are retained (Section 3.2).
Table 16 presents the LLM rewritten responses (Section 3.3), For paraphrasing, we find that formatting style remains largely unchanged, e.g., the number of enumerated lists are the same. Most of the differences lie in their word choices, paraphrased texts use different words with similar meanings but do not change the high-level semantic meaning of the original texts.
Additionally, Table 17 provides a pairwise comparison of responses, supporting our findings in Table 8. Table 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30 contain example responses from each L.M considered in Section 2, including ChatGPT, Claude, Grok, Gemini, DeepSeek, Llama3, 1-8b (instruct), Gemma2-9b (instruct), Qwen2.5-7h (instruct), Mistral-v3-7b (instruct), Llama3.1 1-8b (base). Gemma2-96 (base), Qwen2.5-7b (base), and Mistral-v3-1 (base).
Can you suggest any other cheese alternatives that can use instead of Brie for this recipe? Generate according to: Add one two slices Brie the beef and let it melt. Cut the green apple in thin slices. Top with the mixed salad and finish with the apple slices.
If you're looking for cheese alternatives to Brie for your recipe, consider the following options:
1. Camembert: This cheese is quite similar to Brie in texture and flavor, offering creamy consistency and a mild, earthy taste that will complement the beef and apple.
2. Taleggio: An Italian cheese with soft lexture and langy, fruity flavor. It melts well and add unique twist to your dish.
3. Fontina: Known for its smooth melting properties, Fontina has a nutty and slightly sweet flavor that pairs nicely with beef and apples.
4. Goat Cheese: If you prefer tangier option, goat cheese can provide a creamy texture and a distinct flavor that contrasts well with the sweetness of the apple.
5. Havarti: A semi-sofi cheese with buttery and slighty sweet taste, Havarti melts beautifully and can be great substitute for Brie.
These alternatives should work well in your recipe, providing delicious and melty addition to your beef and apple combination.
(a) original
Can you suggest any other cheese alternatives that can use instead of Brie for this recipe? Generate according to: Add one two slices Brie the beef and let it melt. Cut the green apple in thin slices. Top with the mixed salad and finish with the apple slices.
Please provide a concise response in a single paragraph, limited to maximum of 100 words.
For delicious alternative Brie, consider using Camembent or triple cream cheese, both of which offer a similar creamy texture and rich flavor. you prefer slightly tangier option, goat cheese can provide a delightful contrast the beef and apple. For milder taste, Havarti or Fontina can melt beautifully and complement the dish's 's flavors. Each of these cheeses will add unique twist while maintaining the creamy, melt-in- cereiin-you---oooe mouth experience that Brie provides.
(b) length control
Can you suggest any other cheese alternatives that can use instead of Brie for this recipe? Generate according to: Add one or two slices Brie o the beef and let it melt. Cut the green apple in thin slices. Top with the mixed salad and finish with the apple slices.
Please provide your response in plain text only, avoiding the use of italicized or bold text, lists, markdown, or HTML formatting.
You can use Camembert or triple cream cheese as alternatives to Brie in your recipe. Both have a similar creamy texture and rich flavor that will complement the beef and other ingredients.
(c) format control
Table 13, Examples of prompt-level interventions on ChatGPT.
20

---

Idiosynerasies in Large Language Models
Can you suggest any other cheese alternatives that I can use instead of Brie for this recipe? Generate according to: Add one or two slices Brie the beef and let melt. Cut the green apple in thin slices. Top with the mixed salad and finish with the apple slices.
If you'te looking for alternatives to Brie cheese your recipe, here are some options you can consider:
1. Goat Cheese:
A soft and creamy cheese with tangy flavor that pairs well with the sweetness of the apple and the savory flavor of
the beef.
2. Feta Cheese: A salty and crumbly cheese that adds a nice contrast to the softness of the beef and the crunch of the apple.
3. Camembert: A French cheese with a creamy texture and a mild, earthy flavor that complements the beef and apple nicely.
4. Blue Cheese: A strong and pungent cheese that adds bold flavor to the dish. Use a milder blue cheese if you't 're not a fan of strong flavors.
5. Mascarpone: A rich and creamy cheese that adds a luxurious touch to the dish. It's a good option if you want a milder flavor than
Brie.
6. Ricotta: A soft and creamy cheese that's a good option if you want a lighter flavor than Brie. It's also good choice if you' ree
looking for lower-calorie alternative.
7. Taleggio:
An Italian cheese with a creamy texture and a mild, fruity flavor that pairs well with the apple and beef.
When substituting Brie, keep the following in mind:
using a stronger cheese like Blue Cheese or Goat Cheese, use a smaller amount avoid overpowering the other flavors.
using a milder cheese like Mascarpone or Ricotta, you may want to add more to achieve the desired creaminess. using a crumbly cheese like Feta, you may want to crumble it instead of slicing it to distribute the flavor evenly.
Remember, the key is to balance the flavors and textures in your dish, so feel free to experiment and find the combination that works best for you!
(a) original
Can you suggest any other cheese alternatives that can use instead of Brie for this recipe? Generate according to: Add one or two slices Brie the beef and let melt. Cut the green apple in thin slices. Top with the mixed salad and finish with the apple slices.
Please provide concise response in single paragraph, limited maximum of 100 words.
If you'r re lonking for alternatives to Brie, consider using other soft-ripened cheeses like Camemhert, Goat Cheese, or Feta. Alternatively, you could opt for a semi-soft choese like Havarti or Muenster. For a stronger flavor, try using Blue Cheese or Gorgonzola. If you prefer a milder taste. Boursin or Mascarpone could work well. Each of these cheeses will melt and complement the beef and apple nicely, adding a unique twist to your dish. Simply replace the Bric with your chosen cheese and proceed with the recipe as instructed.
(h) length control
Can you suggest any other cheese alternatives that use instead of Brie for this recipe? Generate according to: Add one two slices Brie the beef and let melt. Cut the green apple in thin slices. Top with the mixed salad and finish with the apple slices.
Please provide your response in plain text only, avniding the use of italicized or hold text, lists, markdown, HTML. formatring.
You can consider the following cheese alternatives to Brie in your recipe:
Camembert:
This French cheese has similar creamy texture and mild, earthy flavor to Bric, making it a suitable substitute. A tangy and creamy cheese that pairs well with the sweemess of the apple and the savory flavor of the heef.
Goat cheese:
Feta: crumbly Greek cheese that adds salty flavor to the dish, which can halance out the sweetness of the apple.
Blue cheese: A strong and pungent cheese that can add a bold flavor to the dish, but be aware that it may overpower the other ingredients.
Ricotta: A mild and creamy cheese that can add a light and airy texture to the dish, but may not provide the same richness as Brie.
Mascarpone: An Italian cheese with rich and creamy texture that can add a luxurious feel to the dish, but it may be sweeter than Bric. Keep in mind that each these alternatives will change the flavor profile of the dish, so you may need to adjust the other ingredients accordingly.
(c) format control
Table 14, Examples prompt-level interventions on Llama3. 1-8b--nstruct.
21

---

Idiosynerasies in Large Language Models
What are some effective tactics for huilding a referral network and gaining new business through word-of-mouthi
Building strong referral network and gaining new business through word-of-mouth be highly effective and cost-efficient. Here are some lactics achieve this:
1. Deliver Exceptional Service: The foundation word-of-moutt marketing is consistent excellence. Providing top-notch services or products encourages satisfied customers to share their positive experiences with others.
2. Identify and Nurture Key Relationships: Focus on building strong relationships with clients, partners, and industry peers who can refer you to others. Personalize your interactions and maintain regular communication to keep these relationships strong.
3. Ask for Referrals: Don't hesitate ask satisfied clients for referrals. Timing crucial mucial-make the request at the peak of their satisfaction. clear about the type of referrals you seeking.
4. Referral Incentives: Offer incentives such as discounts, gift cards, or exclusive access to services to encourage customers or partners to refer new business to you. Ensure these incentives are enticing enough to motivate action.
5. Leverage Social Proof: Encourage satisfied customers to leave reviews or lestimonials online. Positive reviews on platforms
such as Google. Yelp. or LinkedIn can influence potential clients.
6. Engage on Social Media: Use social media platforms to share success stories, client testimonials, and updates about your business. Engage with your audience through comments, shares, and likes to build community around your hrand.
7. Host Events: Organize events such workshops, webinars, or networking meet-ups. These events provide value, showcase your expertise, and encourage attendees refer you others.
8. Partner with Complementary Businesses: Collaburate with businesses that offer complementary services or products. This creates mutual referral opportunities and extends your reach to their customer base.
9. Create a Referral Program: Formalize your referral process with a structured program, ensuring referrers are aware of the benefits and the procedure. Clearly communicate the value proposition and streamline the referral process.
10. Stay Top-of-Mo Mind: Regularly connect with your network through emails, newsletters. personalized messages. Share updates. industry insights, tips that position you valuable and knowledgeable resource.
Thank Referrers: Always express gratitude those who refer business you. Acknowledging their support strengthens relationships and encourages further referrals.
Measure and Optimize: Track where your referrals are coming from and effectiveness different tactics. Use this data to optimize your approach, focusing on strategies that yield the best results. By implementing these tactics and consistently nurturing your network, you can elfectively build a robust referral system that generates new business through word of mouth recommendations.
(a) original
XXX
1. XXXXXX
2. XXXXX
3. XXXXXX
4. XXXXX
5. XXXXXX
6.
7.
8. XXXXXX
9. XXXXXX
10. XXXXXX
11. XXXXXX
12.
XXXXXX
XXX
markdown elements only
Table 15. Examples of only using markdown formatting elements and replacing text content with "xxx" placcholders.
22

---

Idiosynerasies in Large Language Models
What are some ways to cope with feelings of loneliness or isolation?
Coping with feelings of loneliness or isolation can he challenging, but there are several strategies that might help:
1. Connect with Others: Reach out to friends and family through phone calls, video chats, or messaging apps. Joining clubs, groups, or online communitics related to your interests can also help you meet new people.
2. Volunteer: Helping others can provide sense of purpuse and community. Look for local organizations virtual opportumities
where you can volunteer your time or skills.
3. Engage in Hobbies: Pursue activities hobbies that you enjoy, whether it's reading, painting, cooking, or playing musical instrument. These can fulfilling and help distract from loneliness.
4. Physical Activity: Exercise is known to boost mood and alleviate stress. Try wolging.joggnng yoga, or any form physical activity that you enjoy.
5. Mindfulness and Meditation: Practices such as meditation. deep breathing, or mindfulness can help anchor you in the present moment and reduce feelings of loneliness.
6. Professional Help: Consider speaking with a therapist or counselor, who can provide support and coping strategies tailored
to your situation.
7. Pet Companionship: If feasible, consider getting a pet. Animals can offer companionship and can be source of comfort and joy.
8. Limit Social Media:
While it might seem connecting, excessive use of social media can sometimes exacerbate feelings of loneliness. Set boundaries for healthier use.
9. Self- Compossmssiinn Practice being kind to yourself. Acknowledge your feelings without judgment and remind yourselí that
it's okay to feel lonely at times.
10. Set Goals: Setting small, achievable goals can provide sense of accomplishment and forward momentum, fostering sense of purpose.
11. Seeking Community Spaces: Libraries, community centers, or cafes can be great places to be around others, even if you 're not directly interacting with them.
Remember, everyone experiences loneliness differently, and it's important to find what works best for you.
(a) original
Dealing with feelings of loneliness or isolation can be tough, but there are various strategies that may help:
1. Reach Out: Contact friends and family through phone calls, video chats. or messaging platforms. Joining clubs, groups, online communities that align with your interests can also help you connect with new people.
2. Volunteer Your Time: Helping others can give you sense of purpose and community. Look for local charities or online opportunities to contribute your time or skills.
3. Pursue Your Hobbies: Engage in activities you enjoy, such as reading, painting. cooking, or playing a musical instrument. These pursuits can be satisfying and provide a distraction from feelings of loneliness.
4. Stay Active: Regular exercise is known to improve mood and reduce stress. Try activities like walking, jogging, yoga, or any form of physical exercise that appeals to you.
5. Practice Mindfulness: Techniques such as meditation, deep breathing, mindfulness can help ground you in the present and alleviate feelings of loneliness.
6. Consider Professional Guidance: Talking to a therapist or counselor can provide helpful support and coping techniques
tailored to your needs.
7. Get a Pet: If it's an option for you, think about adopting a pet. Animals can be wonderful companions and bring joy and comfort into your life.
8. Limit Social Media Use: Though it may seem like way connect, extensive use of social media can sometimes increase feelings of loneliness. Estahlish limits for a healthier engagement.
9. Be Kind to Yourself: Practice self-compassion by acknowledging your feelings without judgment and reminding yourself that it's normal feel lonely occasionally.
10. Set Achievable Goals: Creating small, realistic goals can foster a sense of accomplishment and help you move forward. enhancing your sense of purpose.
11. Use Community Spaces: Places like libraries, community centers, or cafes can provide an opportunity to be around others, even if you're re not directly interacting with them.
Remember, everyone experiences loneliness in different ways, so it's essential to discover what strategics work hest for you.
(b) paraphrasing
Coping with loneliness can be challenging, but several strategies can help. Connecting with friends and family, volunteering, and engaging in enjoyable hobbies can foster a sense of community. Physical activity and mindfulness practices can improve mood and reduce stress. Seeking professional help and considering companionship from pets may also provide comfort. It's essential to limit social media usage, practice self-compassio, set small goals for achievement, and explore community spaces like libraries or cafes. Ultimately, finding individualized approaches that resonate with you is key to managing feelings of loneliness.
(c) summarizing
Table 16. Examples of LLM rewriting using GPT-4o-mini.
23

---

Idiosynerasies in Large Language Models
What are some ways to cope with feelings of loneliness or isolation?
There are several healthy ways to cope with loneliness and isolation:
Coping with feelings of loneliness or isolation can be chal- lenging. but there are several strategies that might help:
1. Maintain social connections:
1. Connect with Others: Reach out to friends and family through phone calls, video chats, or messaging apps. Joining clubs, groups, online communities related to your interests can also help you meet new people.
Call or video chat with friends and family
Send messages, emails, or letters
Join online communities or social media groups
Schedule regular virtual meetups
2. Volunteer: Helping others can provide a sense of pur- pose and community. Look for local organizations or virtual opportunities where you can volunteer your time or skills.
2. Pursue hobbies and interests:
Take up a new hobby
Join clubs or groups related to your interests
3. Engage in Hobbies:
Pursue activitics or hobbies that you enjoy, whether it's reading, painting, cooking, playing musical instrument. These can be fultilling and help distract from loneliness.
Take online classes
Learn new skill
3. Physical activities:
4. Physical Activity: Exercise is known to boost mood and alleviate stress. Try walking, jogging. yoga, or any form of physical activity that you enjoy.
Exercise regularly
Take walks outdoors
Practice yoga or meditation
5. Mindfulness and Meditation: Practices such as medi- tation, deep breathing. or mindfulness can help anchor you in the present moment and reduce feelings of loneli- ness.
Stay physically active
4. Connect with your community:
Volunteer
6. Professional Help:
Consider speaking with therapist or counselor, who can provide support and coping strate- gies lailored to your situation.
Join local organizations
Attend community events
Participate in religious or spiritual groups
7. Pet Companionship: If feasible, consider getting a pet. Animals can offer companionship and can be a source
5. Self-care practices:
of comfort and joy.
Maintain a routine
8. Limit Social Media: While it might seem connecting, excessive use of social media can sometimes exacerbate feelings of loneliness. Set boundaries for healthier use.
Practice mindfulness
Keep a journal
Get enough sleep
9. Self- Colf-Compassion: Practice being kind to yourself. Ac- knowledge your feelings without judgment and remind yourself that it's okay to feel lonely at times.
Eat healthy meals
6. Seek professional help:
10. Set Goals: Setting small, achievable goals can provide sense of accomplishment and forward momentum, fostering sense of purpose.
Talk to therapist counselor
Join support groups
Consider online therapy options
11. Seeking Community Spaces: Libraries, community centers, or cafes can be great places to be around others, even you're not directly interacting with them. Remember, everyone experiences loneliness differently, and it's important to find what works best for you.
7. Adopt care for pet:
Pets provide companionship
Create a sense of purpose
Encourage routine and responsibility
Remember that feeling lonely is a common human experience, and it's okay to reach out for help when needed.
(a) ChalGPT
(b) Claude
Table 17. Examples from ChatGPT and Claude that illustrate results of our open-ended language analysis in Figure 8.
24

---

Idiosynerasies in Large Language Models
Can polar bears migrate to new habitats if their Arctic environment changes too drastically?
Polar bears are highly specialized animals adapted to the Arctic environment, primarily relying on ice for hunting seals, their main prey. their Arctic environment changes drastically, such as through significant sea ice loss due climate change, their ability to migrate to new habitats is limited by several factors:
1. Species Adaptation: Polar bears are uniquely adapted to life on sea ice and cold climates. They have thick fur, layer of fat for insulation, and physical characteristics like large paws for swimming and walking on ice. These adaptations make it difficult for them to survive in non-pon non-polar environments.
2. Dietary Needs: Their diet consists mainly of seals, which they hunt on sea ice. New habitats may not support a sufficient seal population alternative prey that polar bears can easily exploit.
3. Geegraphical Barriers: Polar bears are largely confined the Arctic and areas surrounding the North Pole. There are no direct routes migrate other cold environments that would support them in similar way.
4. Competition and Predation: Moving to new habitats would likely involve entering territories with existing ecosystems that include competition for food from other predators or unfamiliar threats.
5. Habitat Fragmentation: The fragmentation of ice habitats does not create viable paths for migration but rather isolates populations, reducing genetic diversity and making survival more precarious.
In conclusion, while polar bears might be able to make small relocations within the Arctic as sea ice conditions shift, extensive migration to entirely new habitats is not feasible due to their specific adaptations and the absence of suitably similar environments elsewhere. Conservation efforts focus more mitigating climate change and preserving their current habitat.
Table 18. Example response of ChatGPT.
Can polar bears migrate to new habitats if their Aretic environment changes too drastically?
Polar bears have limited ability to migrate to new habitats if Arctic conditions become unsuitable. They are highly specialized for life on sea ice and depend on it for hunting seals, their primary food source. While they can swim long distances, moving to southern territories would be challenging because: 1) they are not well-adapted to warmer climates, 2) their hunting techniques are specific to ice-based prey, and 3) competition with other bear species and humans would increase. Their survival is closely tied maintaining Arctic sea ice habitat.
Table 19. Example response of Claude.
Can polar bears migrate new habitats if their Arctic enviromment changes too drastically?
Polar bears are highly specialized for their Arctic environment, and their ability to migrate to entirely new habitats is limited by several factors:
1. Adaptation: Polar bears adapted to cold enviromments with diet primarily consisting of seals, which they hunt from sea ice. Their thick fur, large size, and fat reserves are suited for Arctic conditions, not temperate or tropical climates.
2. Habitat Requirements: They need sea ice for humting. breeding, and resting. Without ice, their ability to catch seals, which is their primary food source, is severely compromised. Alternative habitats would need to provide similar conditions or abundant altermative prey, which is rare outside the Arctic.
3. Physiological Limitations: Polar bears might find it challenging to adapt to significantly different temperatures or ecosystems. For instance, their large bodies and fur make them prone to overheating in warmer climates.
4. Behavioral Flexibility: While bears in general can be quite adaptable in terms of diet (e.g.. grizzly hears), polar bears have evolved in an environment with very specific conditions. Their behavioral adaptations are finely tuned to the Arctic, reducing the likelihood of successful migration to drastically different environments.
5. Human Activity: As polar bears move southward search food or better conditions. they increasingly encounter human populations and infrastructure, leading to conflicts, habitat fragmentation, and often, negative outcomes for the bears.
6. Genetic Considerations: Polar bears have evolved in relative isolation in the Arctic. Mixing with other bear species or adapting to entirely new environments could lead to genetic issues or dilution of their specialized adaptations.
However, there are some considerations:
Hybridization: There's evidence of polar bears mating with grizzly bears (creating "pizzly" "grolar" bears) in areas where their ranges overlap due to climate change. This might be seen as form of adaptation or migration at a genetic level, though it's not migration in the traditional sense.
Range Expansion: Some polar bears have been observed traveling further south than usual, likely in search of food as ice recedes. However, this is more about extending their current habitat rather than migrating to a completely different one. Conservation Efforts: Efforts might focus on preserving or creating corridors where polar bears can move between suitable habitats or on habitat restoration to ensure the Arctic remains viable for them.
In summary, while polar bears could potentially expand their range or adapt behaviorally to some extent, true migration to entirely new, non-Aretic habitats unlikely due to their specific ecological needs and physiological adaptations. Conservation efforts are more focused on mitigating the impacts of climate change their current habitat facilitating their movement within the Arctic region.
Table 20, Example response of Grok.
25

---

Idiosynerasies in Large Language Models
Can polar bears migrate to new habitats if their Arctic environment changes too drastically?
While polar bears have some capacity for movement and adaptation, large-scale scale migration to entirely new habitats in response to Arctic change is highly unlikely and faces significant obstacles:
Physiological Specialization: Polar bears are highly specialized for Arctic life. Their thick blubber. fur, and hunting techniques are specifically adapted for cold, icy environments. They rely heavily on sea ice for hunting seals, their primary food source. Relocating warmer environments with different prey and terrain would be extremely challenging. Competition and Existing Ecosystems: Other bear species or large predators already occupy potential alternative habitats. Introducing polar bears would create competition for resources and potentially disrupt existing ecosystems. Polar bears are apex predators, and their presence would have cascading effects.
Limited Land-Bascd Food Sources: While polar hears can sometimes scavenge on land, terrestrial food sources are insufficient to sustain their large body size and energy demands. They are primarily dependent on the rich calorie content of seals. Land-based based food would not be abundant enough to support large populations.
Human- Human-Wildlife Wildlife Conffict: As polar hears are forced further south or into more populated areas in search of food due to shrinking sea ice. encounters with humans are likely increase, leading conflict and potential danger for buth bears and people. Relocation efforts could exacerbate this issue.
Distance and Barriers: Potential alternative habitats are often geographically distant from the Arctic. Natural barriers such as mountains, oceans, and unsuitable terrain would make migration extremely difficult. Furthermore, suitable habitat may already be accupied by other species or fragmented by human development.
Genetic Adaptation Takes Time: While some level of adaptation is possible within a species, evolving to thrive in completely different environment takes many generations. The rapid pace of Arctic change is outpacing the ability of polar bears adapt genetically to significantly different conditions.
Instead of large-scale migration. polar bears more likely exhibit smaller-scale shifts in range, dietury changes (though with limited success), and increased hybridization with grizzly bears. However, the long- survival of the species ultimately depends on mitigating climate change and preserving their Aretic habitat.
Table 21. Exampic response of Gemini.
Can polar bears migrate to new habitats if their Arctic environment changes too drastically?
Polar bears are highly specialized for life in the Aretic, relying on sea ice to hunt their primary prey, seals. If the Aretic enviromment changes too drastically, particularly due to climate change and the loss sea ice, polar bears would face significant challenges migrating to new habitats. Here are some key points to consider:
1. Specialization: Polar bears are adapted to the cold, icy conditions of the Aretic. Their physical and behavioral traits are finely tuned to this environment, making it difficult for them to thrive in vastly different habitats.
2. Prey Availability: Polar bears primarily hunt seals, which are also dependent sea ice. If sea ice diminishes, both polar bears and their prey would be affected. Moving new habitats would require finding alternative food sources, which may not be abundant accessible.
3. Geographic Barriers: The Arctic is relatively isolated, and polar bears would face significant geographic barriers if they attempted to migrate to new areas. Moving southward would bring them into competition with other bear species, such as grizzly bears, and into environments that are not suited to their specific needs.
4. Human Impact: Many potential new habitats for polar bears are already occupied by humans. Human activities, such as urhan development, agriculture, and industry, could pose additional challenges for polar hears trying to adapt to new areas.
5. Genetic A Adaptation: Over long periods, species can adapt to new environments through genetic changes. However, the rapid pace of climate change may outstrip the ability of polar hears to adapt quickly enough to survive in significantly altered or new habitals.
6. Conservation Efforts: Conservation strategies are being developed to help polar bears cope with changing environments. These include protecting critical habitats, reducing greenhouse gas emissions to mitigate climate change, and researching potential relocation or assisted migration options.
In summary, while polar bears have some capacity to adapt to changing conditions, their specialized nature and the rapid pace of cavironmental change pose significant challenges to their ability to migrate to and thrive in new habitats. Conservation efforts are crucial to ensuring their survival in the face of these challenges.
Table 22, Example response of DeepSeek
26

---

Idiosynerasies in Large Language Models
What effective tactics for building referral network and gaining business through word of mouth?
Building referral network and leveraging word-of-mouth marketing can be powerful way to grow your business, Here are some effective tactics to help you achieve this:
1. Deliver Exceptional Service:
The foundation a strong referral network is delivering exceptional service to your existing customers. Ensure that you consistently meet or exceed their expectations, and they will be more likely to refer you to others.
2. Build Relationships: Foster strong relationships with your customers, partners, and colleagues. This can be done through regular communication, shared experiences, and mutual support.
3. Encourage Referrals: Make it easy for your customers refer you providing incentives, such discounts, rewards, exclusive offers. You can also ask for referrals directly, but be
do in non-intrusive and respectful manner.
4. Leverage Social Proof: Use customer testimonials, case studies, and reviews to demonstrate your expertise build credibility. This can be done through social media, your website, other marketing channels.
5. Host Events and Webinars: Hosting events, webinars, or workshops can help you build relationships with potential customers and establish yourself as an expert in your field.
6. Partner with Other Businesses: audiences.
Collaborate with complementary businesses expand your network and reach new
7. Use Referral Marketing Platforms:
Utilize platforms like ReferralCandy, Ambassador, B2B referral platforms streamline your referral process and track referrals.
leads to ensure they have positive experience and gather feedback. 9. Be Authentic and Transparent: Be genuine and transparent your interactions with your network. This will help build trust and credibility, making it more likely that people will refer you to others.
8. Follow Up: Follow up with your referrals
10. Measure and Analyze: Track the effectiveness of your referral efforts and analyze the data to refine your strategy and optimize your results.
Additionally, consider the following:
Employee Advocacy: Encourage your employees hecome brand amhassadors and share their positive experiences with their networks.
Customer Advocacy: Identify and empower your most loyal customers to hecome advocates for your business.
Content Marketing: Create valuable content that educates and entertains your audience, making them more likely to share with others.
Influencer Marketing: Partner with influencers in your industry expand your reach and credibility.
By implementing these tactics, you can build a strong refernal network and leverage word- of mouth marketing to drive growth and success for your business.
Table 23. Example response of Llama3.1-88 (instruct).
27

---

Idiosynerasies in Large Language Models
What are some effective tactics for building a referral network and gaining new business through word-or of mouth?
Building Referral Network Gaining New Business Through Word-of cond-of-Mutthe
Deliver Exceptional Service:
Exceed expectations: Go above and beyond for every client. Surprise and delight them with unexpected value.
Build strong relationships: Get to know your clients personally, understand their needs, and build genuine connections.
Be responsive and reliable: Respond promptly inquiries, meet deadlines, and follow through promises.
2. Make it Easy to Refer:
Create a referral program: Offer incentives for successful referrals, such as discounts, free services, or exclusive perks. Provide referral materials: Make it easy for clients to share your information with others by providing referral cards, social media share buttons, and email templates.
Streamline the process: Make simple for clients to refer friends and family with clear and concise process.
3. Nurture Your Network:
Stay in touch: Regularly communicate with your existing clients through email newsletters, social media updates, or personalized messages.
Attend industry events: Network with other professionals and potential referral partners.
Join relevant online communities: Engage in online forums and groups related to your industry.
Leverage Social Media:
Share valuable content: Provide helpful tips, insights, and resources that your target audience will find valuable. Encourage user-generated content: Ask clients to share their experiences with your business social media.
Run contests and giveaways: Offer incentives for people to share your content and refer their friends.
5. Ask for Referrals:
Don't be afraid to ask: Politely request referrals from satisfied clients.
Make it personal: Explain why you value their referrals and how they can help your business grow.
Offer specific suggestions: Provide examples of people who might benefit from your services.
Track Your Results:
Monitor your referral rate: Track the number of referrals you receive and the conversion rate of those referrals.
Analyze your Identify which tactics are most effective and adjust your strategy accordingly. Celebrate your successes: Recognize and reward your top referrers.
Remember, building strong referral network takes time and effort. patient, consistent, and focus on providing exceptional value to your clients.
Table 24. Example response of Gemma2-9 2b (instruct).
28

---

Idiosynerasies in Large Language Models
What are some elfective lactics for building a referral network and gaining new business through word-of-mouth?
Building a referral network and leveraging oord-of-mouth to gain new business powerful strategy. Here are some effective factics consider:
1. Identify Your Ideal Referral Sources:
Determine who your ideal customers are and who they might trust to recommend your services or products. This could be current customers, industry influencers, or other businesses in your network.
2. Create a Referral Program:
Develop clear and attractive referral program that incentivizes both the referrer and the referred. Incentives range from discounts, free services, or even just a simple thank you note.
3. Educate Your Network:
Provide your network with the necessary information to make a referral. This could include testimonials, case studies, or simple one-pager that explains the benefits of your product or service.
4. Make It Easy to Refer:
Provide easy-to-use tools for your network to make referrals. This could be referral form, link, or simple email template they can use.
5. Leverage Social Media:
Use social media platforms to connect with potential referrers and share your referral program. Platforms like Linkedln, Twitter, and Facebook can be particularly effective.
6. Host Events and Networking Opportunities:
Organize events or participate in networking opportunities where you can meet potential referrers in person. Personal connections can often lead to more referrals.
7. Provide Excellent Customer Service:
Ensure that your current customers have positive experience with your product service. Satisfied customers
more likely refer others.
8. Follow Up and Show Appreciation:
Keep touch with your network and show appreciation referral.. simple thank you note or small token
appreciation can go a long way.
9. Use Testimonials and Case Studies:
Share positive restimonials and case studies from satisfied customers. This can help build trust and encourage others to refer to you.
10. Be Patient and Persistent:
Building referral network takes time. Be patient and persistent in your efforts. Consistency is key.
Monitor and Analyze:
Keep track of your referral program's performance and analyze the data to see what's working and what isn't. Use this
information to refine your approach.
12. Offer Referral Bonuses:
Consider offering bonuses for repeat referrals for referring certain number of new customers. This can help keep the momentum going.
By implementing these tactics, you build strong referral network and leverage wond-of-mouth to grow your business.
Table 25. Example response of Qwen2.5-7b (instruct).
29

---

Idiosynerasies in Large Language Models
What are some effective tactics for building a referral network and gaining new business through word-of word-of-mouth?
1. Provide exceptional service: The foundation of a strong referral network is built on delivering outstanding service your current clients. Happy customers more likely refer you others.
2. Ask for referrals: Don't be shy about asking satisfied clients for referrals. Make easy for them providing referral cards, email templates, or simple script they can use introduce you their contacts.
3. Offer incentives: Consider offering incentives for referrals, such as discounts, freebies, or other rewards. This can motivate
your clients to refer more people to your business.
4. Networking: Attend industry events, trade shows, and networking events to meet potential referral sources. Joining
professional organizations and participating in local business groups can also help you expand your network.
5. Leverage social media: Use social media platforms connect with potential clients and engage with your current network.
Share valuable content, participate in discussions, and don't forget ask for referrals on your social media profiles.
6. Follow up: Follow up with leads promptly and professionally. A quick response can make positive impression and increase the chances of converting referral into client.
7. Build relationships: Focus building long-term relationships with your clients referral sources. Regularly touch base with them, show appreciation for their referrals, and keep them updated on your business and industry news.
8. Provide referral training: Teach your clients how to effectively refer you to others. This can include providing them with tips on how to introduce you, what information to share, and how to follow on referrals.
9. Collahorate with non-competitive husinesses: Partner with other non-competitive businesses cross-refer clients. Thiss can help you tap into their network and vice versa.
Ask for restimonials: Positive testimonials can help build trust with potential clients and encourage referrals, Ask satisfied clients they would willing provide a lestimonial for your website or social media profiles.
Table 26. Example response of Mistral-v3- -3-7b (instruct).
30

---

Idiosynerasies in Large Language Models
Newswise Sepsis is a serious medical condition caused by an overwhelming immune response to infection. The body's s delocttion-ighhinn chemicals trigger widespread inflammation
which can lead to blood clots and leaky blood vessels. It damages the tiny capillaries in organs such as lungs, liver, kidney and brain. According to recent estimates, million people die from sepsis each year. In United States alone, it affects more than one million hospital patients annually and accounts for over $20 billion in healthcare costs, making it the most expensive condition treated in intensive care units (ICUs). A research team led by Dr. Robert W. Day at Washington University School of Medicine found that common drug used for treating type diabetes may reduce mortality rates among ICU patients with severe sepsis or septic shock. The rescarchers examined data more than 340,000 adult patients admitted to ICUs across the U.S. hetween January 2015 and December 2017. Approximately 50% were diagnosed with severe sepsis septic shock. They discovered that the of metformin the first-line treatment for type 2 diabetes was associated with a significant reduction in the risk of death within 30 days after admission to the ICU. The benefit was even greater among those who didn't have gre-exisinng diabetes. The findings suggest that metformin could be considered as part of standard practice for all ICU patients with severe sepsis or septic shock. Metformin has been shown to improve outcomes in animal models of sepsis. According to the researchers, this study provides further evidence linking metformin use with better survival rates in human subjects experiencing extreme illness.
Table 27. Example response of Llama3.1 (base).
Newswise Sepsis is serious medical condition caused by overwhelming immune response infection. The body's infection-fighting chemicals trigger widespread inflammation
which can lead to organ damage and failure.
While sepsis affects only 1% of the U.S. population each year, it kills more than half million people globally every year-man saar-making it as dangerous as cancer malaria. In fact, sepsis accounts for about in six deaths worldwide each year. It also ranks among the top five leading causes death from infectious diseases. When we think of 'sepsis," we often imagine elderly patients with multiple comorbidities, said Michael Miller, M.D., chief of the Division of Pulmonary, Critical Care Sleep Medicine at UT Southwestern Medical Center and holder of the Dallas Heart Ball Chair in Cardiovascular Disease Research. "But our research has shown that 40% of people who develop severe sepsis are young adults.
Table 28. Example response of Gemma2-9b (hase).
Newswise Sepsis is serious medical condition caused by overwhelming immune response to infection. The hody's 's infection-fightinn chemicals trigger widespread inflammation,
which can damage healthy organs and blood vessels. Sepsis is treated in the hospital with antibiotics and fluids. If sepsis isn 't detected early or if treatment duesn't It work, multiple organ failure occurs and may lead to death.
Table 29. Example response of Qwen2.5-7b (base).
31

---

Idiosynerasies in Large Language Models
Newswise Sepsis is serious medical condition caused overwhelming immune response infection. The hody's infection-fighting chemicals
which are normally good, can cause damage to the blood vessels and organs. Sepsis is one of the leading causes of death in hospitals. Although there have been recent advances in sepsis treatment, researchers are still working hard to identify new ways prevent and treat this complex discase. Researchers at Washington University School of Medicine in St. Louis recently published study that may change how sepsis diagnosed and treated. Their work suggests that two different types immune cells are involved in the progression sepsis: one cell type that's 's protective and another that's harmful. These findings could help doctors hetter identify who is most at risk for severe complications from the disease.
"The results from our study show that there are two distinct populations of monocytes," said co-author Dr. Michael Jaffe, professor of medicine at Washington University. "One population promotes inflammation and is associated with worse outcomes; the other helps resolve inflammation and promotes healing."
Monocytes type of white blood cell that are part of the innate immune system. They play a key role in fighting off bacteria and viruses. When someone has sepsis, their monocytes hecome activated. That means they increase in number and start producing cytokines, which are small proteins that regulate the body's 's immune response. Typically, when the body encounters an infection, these cytokines signal the immune system to fight back. But too many cytokines produced too quickly can result in hyperinflammatory state, which can lead to organ failure and death.
In the past, doctors have used blood tests to diagnose sepsis. However, those tests only measure the overall levels of cytokines in person's 's blood. The new study suggests that it might be possible to determine whether someone has particular type of monocyte that could make them more susceptible sepsis. If doctors could potentially use this information to personalize treatments and improve patient outcomes.
"Our findings suggest that there may be opportunities develop therapies that target specific monocyte subsets," said co-author Dr. Daniel Levy. assistant professor of medicine at Washington University. "This could potentially reduce the risk of developing sepsis or even improve outcomes once someone already has sepsis."
There's ss still much to learn about how these monocyte subsets contribute to sepsis development and progression. but the results of this study provide promising insights into why some patients respond differently to treatment than others.
Table 30, Example response of Mistral-v3-77 (base).
32


Reviews:
[{"review_id": "vTGCpmvPeA", "reviewer": "Reviewer_4tcg", "summary": "This paper studies the samples generated by various LLMs. Specifically, they show that it is possible to effectively determine from which LLM a piece of text was sampled. Furthermore, they connect this predictability to \"idiosyncrasies\" in the word-level patterns, which persist even when the text has been transformed.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive assessment of our paper and the constructive comments. We are happy to address your concerns.\n- **Broader impact may be limited**  \n\nIn our submission, we have discussed the broader impact of our results mainly in the first paragraph of the  introduction and Section 5. \nOverall, we believe our results are useful for studying frontier models given that many of their training details are missing in the first place (e.g., training data and model weights). Our results provide a framework that can be used to quantitatively study such differences and reveal the characteristic features of each model. This could potentially help us to build tools for attributing generated text to specific LLMs.  \n\nIn addition, as described in Section 5, we show that our framework could help study the differences of LLMs when training on synthetic data, and measure model similarity. While synthetic data is a promising direction for scaling training data, our results show that the student model could inherit idiosyncrasies of the teacher model that produce those synthetic data. We further use our framework to show that many AI models are easily classified as ChatGPT while ChatGPT is easily confused with Phi-4 – a model trained with large amounts of synthetic data (Figure 9). These results suggest that our work could help detect the practice of model distillation.  \n\nLast, our results can have implications on AI security and safety. One interesting example would be that the distinguishability of AI models could allow adversaries to attack voting based evaluation leaderboard, e.g., ChatbotArena [1]. Specifically, since the identity of the model could be determined from the texts, then an adversary could manipulate the leaderboard by voting consistently for a target model.  \n\nWe will discuss and highlight these broader impacts of our results in the next revision of our paper.\n\n- **In the paraphrasing experiments, how different is the paraphrased text from the original input text?**   \n\nIn our submission, we have provided a comparison of the original generated texts and paraphrased texts in Table 15 (page 20) of Appendix. The formatting style remains largely unchanged, e.g., the number of enumerated lists are the same. Most of the differences lie in their word choices, paraphrased texts use different words with similar meanings but do not change the high-level semantic meaning of the original texts. We will add these observations and more text examples in the next revision.  \n- **Also, did you try using other LLMs, aside from GPT-4o mini for paraphrasing? Were the results consistent if so?**    \n\nBesides GPT-4o-mini, we use Qwen2.5-7b-chat for rewriting LLM responses. Here are the results of classifying the rewritten texts by Qwen2.5-7b-chat on Chat APIs’ responses. \n\n| chat APIs | original | paraphrase | translate | summarize |\n|-|-|-|-|-\n| GPT-4o-mini     | 97.8     | 93.6       | 93.9      | 63.7      |\n| Qwen2.5-7b-chat  | 97.8     | 92.6       | 94.3      | 71.5      |\n\nThe results show that our results are robust to the choices of LLMs for rewriting. We will add these results in the next revision.  \n\n[1] Chatbot Arena. https://lmarena.ai/", "claims_and_evidence": "The claims about identification, word-level patterns, persistence under transformation, and the nature of the idiosyncrasies are well-supported by evidence presented by the authors.", "methods_and_evaluation_criteria": "The evaluation methods make sense. The authors determine they can train classifiers which work to identify the source LLM for a given text completion and do so with both held out completions and held out prompt datasets. Additionally, the methods for identifying the nature of idiosyncrasies which make these texts predictable seem sound.", "theoretical_claims": "There are no proofs in this paper.", "experimental_designs_or_analyses": "Experiments seem broadly sound and effective.", "relation_to_broader_scientific_literature": "This paper relates to work on identification of LLM-generated text using model-based features (e.g. embeddings) or text features (e.g. n-grams). It also relates more broadly to work which studies the composition of real and synthetic natural language datasets.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n\n* The paper is well-written\n* Experiments and analysis are thorough and conclusions are well justified \n* Experiments for identifying the sampling method as well as showing that predictability persists after transforming the text (e.g. paraphrasing) are interesting\n\nWeaknesses:\n\n* Broader impact may be limited", "other_comments_or_suggestions": "No other comments", "questions_for_authors": "In the paraphrasing experiments, how different is the paraphrased text from the original input text? Also, did you try using other LLMs, aside from GPT-4o mini) for paraphrasing? Were the results consistent if so?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BpCxckySAA", "reviewer": "Reviewer_TS1f", "summary": "This paper shows that deployed large language models have idiosyncracies in their output that makes it possible to distinguish which model generated a particular piece of text. These idiosyncracies seem to transcend the surface structure of the output, persisting after text is shuffled in various ways and even after rewriting.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful feedback and recognition of the novelty of the question our paper seeks to answer. We are happy to address your concerns.\n- **The claims are primarily focused on being able to classify the output well, and the models perform far better than chance although no inferential statistics are presented to support this. It would be good to add error bars to the figures.**  \n\nHere we report the classification performance on responses from four instruct LLMs, using prompts (10K for train set and 1K for val set) independently sampled from UltraChat three times: 96.3%, 96.1%, 96.0%. The standard deviation in this case is less than 0.2%, which indicates that our reported numbers in the paper are statistically stable.  \n\n- **The primary analysis presented on page 3 is based on a classification pipeline that is only explained in the Appendix. Given the centrality of this analysis more details in the paper would be helpful.**  \n\n\nThank you for raising this problem. We agree that moving classification setup in Appendix A to the main paper would be better for readers to understand our methodology. In the next revision of our paper, we will describe these details in Section 3 of the main text.  \n\n- **I think this question is interesting but not necessarily something that will lead to further technical innovations for large language models -- it seems more motivated by curiosity about the behavior of these models.**  \n\n\nWe agree that our work is mostly motivated by curiosity and observation – how current LLMs behave differently in everyday interactions. While our proposed classification framework might not lead to direct technical innovation, it can assist researchers to evaluate and understand differences in LLMs. This is especially important since current frontier models are often not open-sourced (e.g., training data and model weights). Our scientific study can provide several useful insights: as shown in Section 5, our framework can help analyze how synthetic data affects model idiosyncrasies and infer model similarity. These implications may offer valuable insights into current LLMs and, in turn, inform future technical developments.  \n\n- **The primary weakness of this paper is that it doesn't have a great deal of technical depth -- it primarily uses off the shelf models and classification techniques.**  \n\nOur paper does not focus on proposing new methods but rather focus on characterizing and understanding the differences between LLMs, where we conduct comprehensive analysis to evaluate and understand model idiosyncrasies. Our novelty mostly lies in the results and findings from our carefully designed experiments. Given the growing number of model releases, we hope this work provides valuable insights for both users and model developers.\n\n- **p 3. The order of Base LLMs and Instruct LLMs in the numbered list should be exchanged. p 6. \"Charateristic\" -> \"Characteristic\"**   \n\nThank you for pointing out these writing errors. We will correct them in the next revision of our paper.", "claims_and_evidence": "The primary claims in the paper are well-supported by the results: classification performance is high across the models considered, and even within model families. The authors checked that their classifiers generalize out of distribution and conducted a number of manipulations to the text to try to identify the factors that were making it possible to classify the output of the models. The claims are primarily focused on being able to classify the output well, and the models perform far better than chance although no inferential statistics are presented to support this. It would be good to add error bars to the figures.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are reasonable for the questions being asked. Some of the methods weren't particularly clear in the main text of the paper. For example, the primary analysis presented on page 3 is based on a classification pipeline that is only explained in the Appendix. Given the centrality of this analysis more details in the paper would be helpful.", "theoretical_claims": "There were no theoretical claims that required assessment -- the primary claims in the paper are empirical.", "experimental_designs_or_analyses": "The basic design of the experiments was sound, and the manipulations of the output that were presented in Section 4 were clever and well designed.", "relation_to_broader_scientific_literature": "This work situates itself well in the broader literature on large language models, as well as the literature on dataset classification in machine learning. The question that is being asked seemed novel to this paper. I think this question is interesting but not necessarily something that will lead to further technical innovations for large language models -- it seems more motivated by curiosity about the behavior of these models. The most interesting finding was the semantic depth of the idiosyncracies, which suggests that there are relatively deep differences resulting from small changes in architecture and training regimes.", "essential_references_not_discussed": "I did not identify any essential references that were missing.", "other_strengths_and_weaknesses": "The primary weakness of this paper is that it doesn't have a great deal of technical depth -- it primarily uses off the shelf models and classification techniques. However, this isn't the focus of the paper so I don't think this is a major weakness.", "other_comments_or_suggestions": "p 3. The order of Base LLMs and Instruct LLMs in the numbered list should be exchanged.\n\np 6. \"Charateristic\" -> \"Characteristic\"", "questions_for_authors": "No questions, in general this was a clearly written paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5h7PGM5ere", "reviewer": "Reviewer_RqnH", "summary": "This paper studies the question of how idiosyncratic the responses of different LLMs are. The authors frame this as a classification task where models are trained to predict which LLM (among a fixed set) generated a particular output. The experimental results show that these classifiers have high accuracy, >95% (compared to a base rate of 25%). \n\nThe paper then studies specific characteristics that contribute to these idiosyncracies. For example, they find differences in word-level distributions across models, even at the unigram level. They also show that LLM responses can be predicted when responses are rewritten, translated, or summarized. The paper concludes by discussing implications of the results for synthetic data training and using their metrics as a measure of model similarity.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. We are happy to address your concerns.\n- **Contrived setting**\n\nWe choose this setting for several reasons:\n1. We are motivated by prior works [1,2] showing the bias in computer vision datasets and thus adopt a standard classification setup to study the differences of LLMs.\n2. It helps us evaluate and understand model idiosyncrasies systematically. In section 3 and 4 of our submission, we have designed many analysis experiments around our classification framework, from which we draw many valuable insights. \n3. Our choice of 4 LLMs per classification group (4 base models together and similarly for instruct models and chat APIs) is mostly for fair comparisons.\n\nWe acknowledge that our setup may not consider scenarios where the list of source LLMs could be large and even unknown beforehand. To address this, we conduct a classification experiment with more LLMs (10 in total): ChatGPT, Claude, Grok, Gemini, DeepSeek, Llama, Gemma, Mistral, Qwen and Phi-4, achieving 92.2% accuracy. This indicates our results can be useful in a practical setup with more models. We will include these discussions and new result in our next revision.\n- **Synthetic data**\n\nThis result is indeed not specific to synthetic data. We verified fine-tuning LLMs on two non-synthetic datasets yield distinguishable models, e.g., 99.1% for GSM8k versus Math; 98.9% for MMLU versus ARC. We focus on synthetic data in the paper since synthetic data becomes a popular use case of LLM generated outputs. For example, UltraChat dataset consists of mostly GPT-4 responses. Thus we want to highlight this idiosyncratic behavior of training on synthetic data where idiosyncrasies of teacher models can be inherited. We will clarify this point in our next revision.\n- **Model similarity**\n\nTwo recent papers have proposed other model similarity metrics, [3] and [4], which we have cited in our submission. Specifically, [3] focused on “vibe”-based similarities based on tones and writing styles. [4] proposed a metric to measure statistical differences between models and their quantized, watermarked or fine-tuned versions. [3]’s method can be extensive and biased as it uses a LLM judge to obtain the vibe scores. [4] is more suitable to detect small changes occurred to one model. Our metric is more general than [3] (not limited to vibes), as we show in Section 4. As opposed to [4], we focus mainly on comparing two different models but not variations of one model.\n- **LLM monoculture**\n  \nA recent study that explores LLM monoculture [5] finds that LLM-generated book reviews are more positive than human-written ones. Following their setup, we generate book reviews using Llama-2-7b-chat and Vicuna-13b-v1.5 and find these responses can be classified with near-perfect accuracy. Note that our results are not contradictory with [5] as it is possible to express similar sentiments with different stylistic / lexical choices. Therefore, our work offers a tool to capture idiosyncrasies that characterize LLM monoculture.\n- **Training data overlap**\n\nWe added experiments to study the effect of training data overlap. We split GPT-2 pretraining dataset OpenWebText into three parts: a, b and c. We start from training two models on data splits a and b. Then we replace 33% or 66% percent from split a and b with data from split c, which makes the resulting splits overlapping. The classification results on the trained models are:  \n|overlapping between a and b|accuracy|\n|-|-\n|0% c|86.4|\n|33% c|86.4| \n|67% c|79.7| \n|100% c|72.6|  \n\nAs the overlapping increases, the accuracy to classify two trained models is lower, suggesting that pretraining data overlap affects idiosyncrasies.\n- **Experiment setup**\n\nThanks for pointing this out. We construct the held-out sets by sampling 1K prompts and use the responses generated from these fixed 1k prompts. The prompts for generating the train and held-out sets are separate. We do not include the prompt in the input to the classifier but we find this choice little affects our results. The total number of prompts in UltraChat we sampled from is around 200K. We will add these details in our next revision.\n- **Confusing tables**\n\nIn our submission, we clarified at the start of Section 3.2 that “From now on, we report the accuracy of the four-way classification task.”. The base rates would be the results under “original”. We are sorry for the confusion and will make these clearer.\n- **Strict description of LLMs**\n\nWe were meaning that we consider Transformer-based LLMs in our paper. We will rephrase this and add related references on diffusion-based LLMs.\n\n[1] A Decade’s Battle on Dataset Bias: Are We There Yet?. ICLR 2025  \n[2] Unbiased look at dataset bias. CVPR 2011  \n[3] VibeCheck: Discover and Quantify Qualitative Differences in Large Language Models. ICLR 2025  \n[4] Model Equality Testing: Which model is this API serving? ICLR 2025\n[5] Generative Monoculture in Large Language Models. ICLR 2025", "claims_and_evidence": "The paper's central claim is that which LLM has written a given text can be easily predicted. In one sense, this claim is very well supported:\n- Extensive analysis shows compelling results that LLM outputs are very predictable (when conditioning on up to 4 LLMs).\n- Creative range of experiments from different prompts and text shuffling\n- The qualitative analysis was nice, focusing on different frequencies of words and letters. The characteristic phrases analysis was also interesting.\n\nHowever, in another sense, the setting studied by the authors is very narrow with unclear implications:\n- The setting -- conditioning on knowing the text is from an LLM and that it's one of 2 or 4 known models -- is contrived. When are we going to come across this in the real-world? As the authors note, this is different from testing whether a response is LLM-generated or not.  Why not a larger classification setting that considers many models?\n- The first point in the implications section -- that fine-tuning two different models on the same synthetic data makes their answers harder to distinguish -- is unclear. Why does the data have to be synthetic? This seems like this could be a point about models being trained on different data being easier to differentiate. Wouldn't we have a similar result when models are trained on non-synthetic data? In other words -- why is the synthetic nature of the data important to this claim?\n- The second point in the implication section is that the predictability metric can be used as a metric of model similarity. While this is interesting, there isn't much discussion or results about why this would be a useful metric. Is this capturing something that other model similarity metrics aren't capturing? And is there evidence for this?", "methods_and_evaluation_criteria": "The experimental methodology is generally sound, although some details aren't clearly described. For example, the paper doesn't clearly explain how held-out sets are constructed (e.g. whether responses to the same prompt can appear in both train and test set). See the \"Other strengths and weaknesses\" section for more discussion. See also the points in \"Claims and Evidence\"", "theoretical_claims": "The paper doesn't make substantial theoretical claims requiring proofs.", "experimental_designs_or_analyses": "See responses to \"Methods and Evaluation Criteria\" and \"Other Strengths and Weaknesses\"", "relation_to_broader_scientific_literature": "The paper positions itself well within the literature on dataset bias and human vs. machine-generated text detection. However, it could connect to the LLM monoculture literature -- see \"Other Strengths and Weaknesses\".", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "As mentioned above, the biggest weakness of the paper is the unclear implications. A couple of suggestions for improvement below:\n- One possible implication worth exploring is the LLM monoculture literature. How do these results inform our understanding of LLM monoculture, showing e.g. that LLMs make similar mistakes in answering questions or behave similarly when recommending jobs?\n- More analysis about what makes pairs of models more predictable would be interesting. E.g. how important is the training data overlap (for models where training data is known)?\n\nThe other main drawback of the paper was unclear writing:\n- There's not enough detail in the second paragraph of Section 3 to make the experimental setup clear. E.g. how are the held-out sets constructed? Can responses to the same prompt appear in both the train and held-out sets? Does the classifier condition on prompt? What's the total number of prompts in each dataset? These details aren't in the appendix either.\n- The tables are very confusing. E.g. what is Table 4 reporting, and what are the units? For many of the tables (e.g. Figure 2, Table 8) it's not clear what's the base rate -- are these all 4-way classification?\n- The description of LLMs in lines 125-129 is too strict (i.e. that they \"all utilize the Transformer architecture with self-attention\" and \"trained using an auto-regressive objective\"). For example we're beginning to have diffusion-based LLMs ([1] for code specifically, [2] more recently as a general purpose LLM)\n\n[1] Singh, Mukul et al. \"CodeFusion: A Pre-trained Diffusion Model for Code Generation\".  \n[2] https://www.inceptionlabs.ai/news", "other_comments_or_suggestions": "See above", "questions_for_authors": "See above\n\n=====================\n\nPOST-REBUTTAL UPDATE\n\n=====================\n\nI thank the authors for addressing my comments. In response I raised my review.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e8MGAaDslC", "reviewer": "Reviewer_UaTq", "summary": "This paper explores how distinguishable the outputs of different LLMs are from each other. The main results demonstrate that the LLM to produce a given output can easily be predicted by a trained classifier. Further experiments show commonly-used phrases and other idiosyncrasies that differentiate LLMs from one another. Ultimately, the authors conclude that LLM outputs convey deeply-seated differences in their use of language, despite frequently being trained on common data sources.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. We are happy to address your concerns.\n- **Different prompt datasets**  \n\nIn our experiments, we primarily use the UltraChat dialogue dataset to generate responses from instruction  LLMs and Chat APIs, and the high-quality pretraining dataset FineWeb for base LLMs. Note that these datasets are already diverse by construction. In addition, in Table 3 of our submission, we have presented results with other prompt datasets (e.g., Cosmopedia, LmsysChat, and WildChat). We find the classifier trained on each of them can achieve >90% accuracy (see diagonal entry). This indicates our results are robust across prompt datasets.\n\n-  **Takeaway of “generalization to OOD responses” and “length control”**  \n\nThe goal of these two experiments is to study idiosyncratic behavior in controlled experiments. The results of “generalization to OOD responses” show that our classifiers capture robust distinguishable features, indicating  that our observation is general and not dataset specific. The “length control” experiment is to make sure that the high classification accuracy is not due to the short-cut solution which is simply based on the response length.  \n\n- **Sampling with restricted, common set of vocabulary (especially without characteristic phrases)**\n\nWe restrict the LLMs to a common set of K words during response generation by only sampling relevant tokens composing each word and special character along with special tokens for each LLM. In addition, we do not allow the LLM to output characteristic phrases in Figures 5 and 13 by setting relevant logits to -inf.  Below are the classification results of instruct LLMs using the above sampling strategy:  \n| | Llama vs Gemma | Llama vs Qwen | Llama vs Mistral | Gemma vs Qwen | Gemma vs Mistral | Qwen vs Mistral | four way |\n|-----|-----|-----|-----|----- |-----|-----|-----\n|all words | 99.9 | 97.8| 97.0 | 99.9| 99.9|96.1 |96.3|\n|K=2000|98.4| 97.1|99.5 | 99.4 | 99.5 | 98.6|96.9 |\n|K=4000| 99.1 | 97.5|98.5 | 99.4 | 99.6 | 98.0 | 97.1|  \n\nThe results show that even restricting to a common set of vocabulary, our classifiers can still predict LLM identity with high accuracy.  \n\n- **Main results with top-k and top-p sampling**  \n\nWe use the softmax sampling (T=0.8), top-k sampling (k=100), top-p sampling (p=0.9) to generate responses for instruct LLMs and report their results below:  \n| | Llama vs Gemma | Llama vs Qwen | Llama vs Mistral | Gemma vs Qwen | Gemma vs Mistral | Qwen vs Mistral | four way |\n|-----|-----|-----|-----|----- |-----|-----|-----\n|greedy | 99.9 | 97.8| 97.0 | 99.9| 99.9|96.1 |96.3|\n|softmax (T=0.8) | 99.2| 97.3 | 96.1| 99.5| 99.5 |95.7 |95.2|\n|top-k (k=100)  | 99.3| 96.8 |96.2 |99.5 |99.5 | 96.1|95.1 |\n|top-p (p=0.9)| 99.2 | 97.2 | 96.4 |99.6 |99.6 |95.9|95.5|\n\nWe observe that using different sampling methods does not alter the results very much (often <2%).  \n\n- **Implication too specific and does not expand to broader impact**  \n\nWe acknowledge that the current implication section contains too many experiments. In the next revision, we will focus more on discussing and highlighting the broader impact of our results in this section. Besides our analysis on synthetic data and model similarity, we also want to mention another implication on privacy and safety. One interesting example would be that the distinguishability of AI models could allow a malicious party to manipulate voting based evaluation leaderboard, e.g., Chatbot Arena. As the identity of the model could be determined from the texts, such an adversary would be able to consistently vote for a desired target model. \n\n- **Grammar errors and LLM type order**  \n\nThank you for pointing this out. We have corrected it in the current version of our paper. We will run a systematic grammar check to proofread the paper.  \n-  **Blurry figure**  \n\nThank you for raising this concern. The font size is indeed a little bit small there. We will increase the font size to match that of the figure captions to ensure the results are more readable.", "claims_and_evidence": "Claims appear to be supported.", "methods_and_evaluation_criteria": "Most of the results only use one dataset per group of LLMs, as stated in column 2 lines 141-148. These results could change significantly when prompted using a different dataset.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "- Many results are surprising, such as the high degree of distinguishability between LLMs (Section 3.1) and that the distinguishability is retained even when the outputs are paraphrased or translated by another LLM (Section 4.3).\n- Some experiments seem a little superfluous or could be designed to be more interesting. For example, the “Generalization to out-of-distribution responses” in Section 3.1 or the length control aspect of “Prompt-level interventions” experiment in Section 3.2 do not seem to lead to very useful takeaways. Comparatively, it might be interesting to see if the LLMs become less distinguishable when limiting all the LLMs to a restricted, common set of vocabulary during sampling (especially one without their most-commonly used phrases shown in Figure 5). Similarly, the “Sampling methods” experiment in Section 3.2 may instead be more interesting by investigating if the sampling method impacts how recognizable different LLMs are from each other. In other words, repeat the Table 1 main results experiment using top-k sampling, and again for top-p sampling, etc.", "relation_to_broader_scientific_literature": "- The perspective of the paper is different from many other LLM papers, leading to some findings that may be a useful contribution to the LLM research community. \n- The “Implications” section does not expand very much on broader implications of the paper, but rather introduces additional experimental results. Moving these additional results to the appendix would allow the authors to better clarify the importance of their work to readers in this section.", "essential_references_not_discussed": "None I am aware of", "other_strengths_and_weaknesses": "- Some minor grammatical errors and other issues: for example, the list of LLM types in 3.1 refers to base LLMs mentioned above, when they are the last item in the list\n- Figure 4 (and the others of this style) is very useful for results visualization, but looks a little blurry and difficult to read in print.", "other_comments_or_suggestions": "- Some minor grammatical errors and other issues: for example, the list of LLM types in 3.1 refers to base LLMs mentioned above, when they are the last item in the list\n- Figure 4 (and the others of this style) is very useful for results visualization, but looks a little blurry and difficult to read in print.", "questions_for_authors": "Similarly to the point I mentioned earlier, would you be able to share results of how distinguishable the LLMs are from each other (as in Table 1) when using each sampling method?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper shows that deployed large language models have idiosyncracies in their output that makes it possible to distinguish which model generated a particular piece of text. These idiosyncracies seem to transcend the surface structure of the output, persisting after text is shuffled in various ways and even after rewriting.

The reviewers generally agree that the work is well executed and the claims are sound. Several reviewers question the implications of this work and broader impact. The author response addresses these concerns to an extent, and incorporating an improved discussion of these issues will strengthen the paper.
Decision: Accept (poster)