MOSAIC: Multiple Observers Spotting AI Content
Authors: Matthieu Dubois, François Yvon, Pablo Piantanida
Accepted at: acl25

MOSAIC: Multiple Observers Spotting AI Content

Matthieu Dubois1 and François Yvon1 and Pablo Piantanida2,3,4
1Sorbonne Université, CNRS, ISIR, Paris France
2CNRS, International Laboratory on Learning Systems, Montréal, Canada
3Mila - Québec AI Institute, Montréal, Canada
4CentraleSupélec, Université Paris-Saclay, Gif-sur-Yvette, France
Correspondence: duboism at isir.upmc.fr

Abstract

The dissemination of Large Language Models (LLMs), trained at scale, and endowed with powerful text-generating abilities, has made it easier for all to produce harmful, toxic, faked or forged content. In response, various proposals have been made to automatically discriminate artificially generated from human-written texts, typically framing the problem as a binary classification problem. Early approaches evaluate an input document with a well-chosen detector LLM, assuming that low-perplexity scores reliably signal machine-made content. More recent systems instead consider two LLMs and compare their probability distributions over the document to further discriminate when perplexity alone cannot. However, using a fixed pair of models can induce brittleness in performance. We extend these approaches to the ensembling of several LLMs and derive a new, theoretically grounded approach to combine their respective strengths. Our experiments, conducted with various generator LLMs, indicate that this approach effectively leverages the strengths of each model, resulting in robust detection performance across multiple domains. Our code and data are available at https://github.com/BaggerOfWords/MOSAIC.

1 Introduction

Large Language Models (LLMs) have greatly improved the fluency and diversity of machine-generated texts. The release of ChatGPT and GPT4 by OpenAI has sparked global discussions regarding the new opportunities offered by AI-based writing assistants. These advances have also introduced considerable threats such as fake news generation (Zellers et al., 2019), and the potential for harmful outputs such as toxic or dishonest content (Crothers et al., 2023), among others. As it seems, the research on methods to detect the origin of a given text to mitigate the dissemination of forged content and to prevent technology-aided plagiarism still lags behind the rapid advancement of AI itself.1

Many studies have focused on tools that could spot such AI-generated outputs and mitigate these underlying risks. From a bird’s eye view, this typically involves using detector models to discriminate generator models’ outputs from legitimate human writings. Multiple versions of this generic text classification task have been considered, varying, e.g. the number of possible categories to distinguish and the amount of supervision (see Section 2). Owing to its large user base and applications, the largest effort has focused on one specific generator, ChatGPT, for which training and test data are easily obtained. Yet, the corresponding supervised binary problem, with a unique known generator, is not the only way to frame this task. A more challenging problem, that we study here, is generator-agnostic artificial text detection, where the models to be detected are not predefined.

Our contributions. In this paper, using fundamental information-theoretic principles from universal compression, we derive a new ensemble method (depicted in Figure 2) that combines the strength of multiple LLMs into a single score to detect forged texts. By using several models, this

![Unsupervised methods rely on fixed detector models, but which one should you use in a generator-agnostic setting?](page_1012_370_388_246.png)

Figure 1: Unsupervised methods rely on fixed detector models, but which one should you use in a generator-agnostic setting?

1As illustrated by the discontinuation of OpenAI’s detector https://openai.com/index/new-ai-classifier-for-indicating-ai-written-text/.
approach dispenses with the need to look for the optimal set of detector models, and thus does not require a validation dataset. Our experiments use both standard benchmarks comprising multiple domains and languages, as well as a new corpus of machine-generated texts. They confirm that ensembling strong LLMs yields detectors that can robustly identify a multiplicity of generators and that compare favourably with several recent proposals using a predefined set of detector models. We also report a number a complementary analyses exploring the effect of incrementally growing the ensemble, of considering adapted versions of one single model, of using a naïve ensembling method, and of processing noised versions of artificial texts. We publicly release the code and data produced for this study.

2 Related Work

Overview of the field. The improved text generation abilities of LLMs raise concerns about potential misuses such as disinformation (Zellers et al., 2019), abusive content (Crothers et al., 2023), forged academic publications (Liu et al., 2024), or cheating during exams (Vasilatos et al., 2023). Since such fake texts seem difficult for humans to spot (Gehrmann et al., 2019), the issue of automatically detecting machine-generated texts has been subject to an increasing focus. This problem can be framed as a binary human vs. non-human decision, as the problem of detecting one known artificial agent (e.g., ChatGPT (Mitrović et al., 2023; Liu et al., 2024)), or as discriminating the correct model in a predefined list (Li et al., 2023). Some works aim to differentiate between “machine-generated” or “machine-polished” (Abassy et al., 2024; Liu et al., 2024). Another distinction is between closed-domain (e.g. scientific (Liyanage et al., 2022), academic (Liu et al., 2024) or user-generated content (Fagni et al., 2021; Kumarage et al., 2023)) vs. open-domain text detection. Assuming the generator models are known, various settings can be considered, depending on whether models can be openly queried (open parameters), whether they expose their full logits, or just the top prediction (and associated probability), etc.

Supervised methods. Supervised detection with a single generator often achieves detection accuracy rates in the high 90s (Zellers et al., 2019; Guo et al., 2023; Liu et al., 2024), using classifiers based on Roberta (Conneau et al., 2020) or T5 (Raffel et al., 2020). However, these approaches are brittle and their success depends on particular generator-detector pairs (Antoun et al., 2024), prompting e.g. Verma et al. (2024) to design automatic feature extractors from multiple detectors to improve the robustness of their system.

Zero-shot methods. Unsupervised detection is more challenging. Most approaches rest on the idea that human-written texts are more “surprising” than artificial texts2, leading to a difference in token-wise log-probability3. This idea is used in GPTzero4 and thresholding perplexity usually provides strong baselines (see, inter alia, (Gehrmann et al., 2019; Ippolito et al., 2020; Mitchell et al., 2023)). Such techniques heavily rely on the detector model(s) used to compute the log-probabilities of input texts, which must be robust to variations in domains, genres, styles, and languages (Wang et al., 2024b); and to variations in the generator itself (Antoun et al., 2024).

Perturbation-based. Mitchell et al. (2023) and Bao et al. (2024) exploit a similar intuition, arguing that small random perturbations of an artificial text will on average make it less likely, unlike human-written texts. They develop a statistical criterion based on the curvature of the log-probability function and achieve near-perfect detection scores across three types of texts generated by five different models. The Binoculars score of Hans et al. (2024) also relies on a function of the per-token log-perplexity, contrasted with the cross-entropy of an auxiliary model.

These valuable works point to the over-reliance on one specific detector model as a major limitation of the state-of-the-art. Our proposed mitigation relies on ensemble techniques, that are also considered in the supervised detection setting, e.g. in (Verma et al., 2024; Wang et al., 2023; El-Sayed and Nasr, 2023; Liyanage and Buscaldi, 2023).

Other methods. Abandoning generator-detector based techniques altogether, (Mao et al., 2024; Yang et al., 2024) develop effective detection approaches based on regeneration, prompting the (known) generator to regenerate part of the input text. The intuition is that artificial inputs are likely to be regenerated exactly, unlike human

2Assuming generation does not use random sampling, in which case the reverse is likely to be observed, as long artificial texts drift away from natural writings (Zellers et al., 2019).
3(Mitchell et al., 2023) argues that the difference is better seen at the level of log-ranks.
4https://gptzero.me/
texts. Other strategies include text watermarking (Kirchenbauer et al., 2023, 2024; Liu and Bu, 2024), though its efficiency and robustness are still subject to discussions, e.g., (Zhang et al., 2024).

Robustness issues. Recent works focus on detection robustness. (Wang et al., 2024a) find that after simple modifications, only watermarking remains able to accurately identify artificial documents. Dugan et al. (2024) present artificial texts generated with multiple models and sampling strategies, additionally subject to various adversarial attacks, observing that most detectors suffer large drops in performance, and Binoculars (Hans et al., 2024) stands out, achieving decent True Positive Rates at False Positive Rates under 1%.

3 Detecting AI-Generated Text with Multiple Models

3.1 Background

We consider models for language generation tasks that define a probability distribution over strings. Formally, language models are probability distributions over an output space \( \mathcal{Y} \) which contains all possible strings over vocabulary \( \Omega \): \( \mathcal{Y} \triangleq \{ \text{BOS} \circ y \circ \text{EOS} \mid y \in \Omega^* \} \), BOS and EOS denote respectively the beginning-of-sequence and end-of-sequence tokens, and \( \Omega^* \) is the Kleene closure of \( \Omega \).

Today’s models for language generation are typically parameterized with trainable weights \( \theta \in \Theta \). These models follow a local-normalization scheme, meaning that \( \forall t > 0,\ p_\theta(\cdot|y_{<t}, ) \) defines a conditional probability distribution over \( \bar{\Omega} = \Omega \cup \text{EOS} \). The probability of a sequence \( \mathbf{y} = \langle y_0, \ldots, y_T \rangle \) is:

\[
p(\mathbf{y}) = \prod_{t=1}^T p_\theta(y_t|\mathbf{y}_{<t})
\]

\( \mathbf{y}_{<t} = \langle y_0, \ldots, y_{t-1} \rangle, y_0 = \text{BOS} \) and \( y_T = \text{EOS} \).

Measuring information. A fundamental relationship in information theory relates the probability of a message and the quantity of information it carries, using the relationship (Cover and Thomas, 2006): information \( = -\log(\text{probability}) \), assuming the use of coding techniques such as Huffman and Arithmetic codes (Shields, 1996) which allow to achieve message lengths closely approximating this ideal length in binary digits.

Explanations of data. Given a body of text represented in a finite string \( \mathbf{y}_{<t} = \langle y_0, \ldots, y_{t-1} \rangle \), an “explanation” of this next token \( y_t \) is a binary string encoding the symbol with minimum length \( \mathcal{L}_p(y_t|\mathbf{y}_{<t}) \triangleq -\log p(y_t|\mathbf{y}_{<t}) \). \( \mathcal{L}_p(y_t|\mathbf{y}_{<t}) \) is also often referred to as the model’s surprisal (Samson, 1953) on input \( y_t \). Its expected value is termed the conditional entropy:

\[
\mathcal{H}_p(Y_t|\mathbf{y}_{<t}) = \sum_{y_t \in \Omega} p(y_t|\mathbf{y}_{<t}) \mathcal{L}_p(y_t|\mathbf{y}_{<t}).
\]

Finally, another important concept is the conditional mutual information (MI) between two random variables \( \mathbb{M} \) and \( Y_t \), given a sequence value \( \mathbf{y}_{<t} \), defined as (Cover and Thomas, 2006):

\[
\mathcal{I}_p(\mathbb{M}; Y_t|\mathbf{y}_{<t}) = \mathcal{H}_p(Y_t|\mathbf{y}_{<t}) - \mathcal{H}_p(Y_t|\mathbb{M}, \mathbf{y}_{<t}), \\
\mathcal{H}_p(Y_t|\mathbb{M}, \mathbf{y}_{<t}) = \mathbb{E}_{m \sim \mu(m|\mathbf{y}_{<t})} \mathcal{H}_p(Y_t|m, \mathbf{y}_{<t}).
\]

Conditional MI captures the amount of information we get about \( \mathbb{M} \) when observing \( Y_t \), and already knowing \( \mathbf{y}_{<t} \).

3.2 Multi-model Detection Methods

When detecting machine-generated texts in a zero-shot setting, the most promising methods rely on a language model (Guo et al., 2023). These techniques are becoming less effective as LLMs’ capabilities improve over time. The results of FastDetectGPT and Binoculars suggest that detection can be significantly improved by simultaneously using two models: in their study, detection scores are obtained by comparing a model’s surprisal against the cross-entropy with respect to the other model, averaged over the input tokens.

Here, we explore a key question: how can we leverage multiple models for improved detection? A straightforward approach is to systematically search for the best model pair, achieving optimal detection scores, as reported in Table 3 p.15 in (Hans et al., 2024) and Table 7 p.19 in (Bao et al., 2024). However, this method lacks robustness, as the best-performing model pair may vary depending on the validation dataset used for selection, leading to performance fluctuations when the test domain or language changes. Additionally, this approach may struggle to scale to larger model ensembles due to the exponential increase in possible model combinations that must be explored.

Before addressing our main question, we revisit and reformulate Binoculars and FastDetectGPT\footnote{We focus on Binoculars here, while the analysis of FastDetectGPT is provided in Appendix A.} using the previously introduced concepts, we explore potential variations and extensions.
Algorithm 1 MOSAIC Scoring

1: Input: text \( \mathbf{y} = (y_0, y_1, \ldots) \), LLMs \( m \in \mathcal{M} \), with \( m^* \) the reference model
2: for \( y_t \) in \( \mathbf{y} \) do
3:   \( \mu^*(m|\mathbf{y}_{<t}) \leftarrow \) Blahut–Arimoto (\( \mathcal{P}_\mathcal{M}(\mathbf{y}); \mathbf{y}_{<t} \)) \( \triangleright \) Obtain the \( \mu^* \) weights
4:   \( q^*(y_t|\mathbf{y}_{<t}) \leftarrow \sum_{m \in \mathcal{M}} \mu^*(m|\mathbf{y}_{<t}) p_m(y_t|\mathbf{y}_{<t}) \) \( \triangleright \) Build the mixture
5:   \( s_t(\mathbf{y}) \leftarrow \mathcal{L}_{q^*}(y_t|\mathbf{y}_{<t}) - \sum_{y \in \Omega} p_{m^*}(y \mid \mathbf{y}_{<t}) \mathcal{L}_{q^*}(y \mid \mathbf{y}_{<t}) \) \( \triangleright \) Compare surprisal and cross-entropy
6: end for
7: \( S_\mathcal{M}(\mathbf{y}) \leftarrow \frac{1}{T} \sum_t s_t(\mathbf{y}) \) \( \triangleright \) MOSAIC score for the whole text

3.3 Revisiting the Binoculars Method

The Binoculars score \( B_{p,q}(\mathbf{y}) \) for an input sequence \( \mathbf{y} = (y_0, y_1, \ldots) \), using two language models \( q \) and \( p \) expressed as in (1), is defined by:

\[
B_{p,q}(\mathbf{y}) \triangleq \frac{\sum_{t=1}^T \sum_{y \in \Omega} \mathbb{I}[y = y_t] \mathcal{L}_q(y_t|\mathbf{y}_{<t})}{\sum_{t=1}^T \sum_{y \in \Omega} p(y|\mathbf{y}_{<t}) \mathcal{L}_q(y|\mathbf{y}_{<t})},
\]

with \( \mathcal{L}_q(y_t|\mathbf{y}_{<t}) = -\log q(y_t|\mathbf{y}_{<t}) \), and \( p(y|\mathbf{y}_{<t}) \) and \( q(y|\mathbf{y}_{<t}) \) represent the probabilities assigned by models \( p \) and \( q \), respectively, to token \( y \) conditioned on the current context \( \mathbf{y}_{<t} \). It is important to note that Eq. (2) is only valid when \( q \) and \( p \) share the same underlying vocabulary and tokenizer.

From an information theory perspective, we can interpret the numerator as the shorter average encoding length of the observed text according to model \( q \), while the denominator represents the encoding length if the text were generated by sampling from model \( p \) instead. For this reason, in all that follows, we call \( p \) the reference model.

Interestingly, it is easy to see that FastDetect-GPT leverages the same concept but calculates a difference rather than a ratio. While equivalent, it normalizes the score for each token instead of averaging it over the entire sentence (see Appendix A).

How to choose the most promising reference model? Both scoring methods are based on the intuition that the numerator—the log-probability of the text—tends to be smaller for machine-generated texts compared to natural ones. To enhance these distinctions, they also rely on the idea that, conversely, the denominator term should be smaller for artificial texts. This hints at the fact that when having a family of models \( \mathcal{P}_\mathcal{M}(\mathbf{y}) = \{p_m(\mathbf{y}) : m \in \mathcal{M}\} \), given a human sample of text \( \mathbf{y}_{\text{hum}} \), we can use the following criterion:

\[
m^*(\mathbf{y}_{\text{hum}}) \triangleq \argmin_{m \in \mathcal{M}} - \sum_{t=1}^T \log p_m(y_t|\mathbf{y}_{<t}). \tag{3}
\]

In other words, the reference model \( p_{m^*} \) needs to be the model in the ensemble \( \mathcal{P}_\mathcal{M}(\mathbf{y}) \) with the lowest log-perplexity for human samples of text \( \mathbf{y} \). This often turns out to be the largest LLM in the ensemble, which is consistent with the tables in the original papers (Bao et al., 2024; Hans et al., 2024) and confirmed experimentally in Section 6.3.

The methodology introduced in (3) enables us to select the most promising reference model, \( p_{m^*} \), from a given family of available models \( \mathcal{P}_\mathcal{M}(\mathbf{y}) \). However, it does not provide a practical criterion for selecting the best model \( q \) needed to evaluate (2). This will be addressed in the next section.

4 Introducing MOSAIC

Building on the principles used in previous systems, we now present MOSAIC, a scoring method designed to leverage multiple models simultaneously. The key difference compared to previous approaches is that instead of using a single fixed LLM for \( q \), MOSAIC defines it as a position-dependent mixture of all models in the ensemble. The weights of this mixture are formally defined in the next proposition and depicted in Figure 2.

![Mixture model, where {μ_i} denote the time-varying weights associated to LLMs in the mixture.](page_1012_1342_388_180.png)

Figure 2: Mixture model, where \( \{ \mu_i \} \) denote the time-varying weights associated to LLMs in the mixture.

Proposition 1 (Optimal model). The optimal model, \( q^* \)—which minimizes the encoding length for tokens—is the position-dependent mixture:

\[
q^*(y_t|\mathbf{y}_{<t}) \triangleq \sum_{m \in \mathcal{M}} \mu^*(m|\mathbf{y}_{<t}) p_m(y_t|\mathbf{y}_{<t}), \tag{4}
\]
<table>
  <tr>
    <th></th>
    <th>chatgpt</th>
    <th>cohere-c</th>
    <th>cohere</th>
    <th>gpt2</th>
    <th>gpt3</th>
    <th>gpt4</th>
    <th>llama-c</th>
    <th>mistral-c</th>
    <th>mistral</th>
    <th>mpt-c</th>
    <th>mpt</th>
  </tr>
  <tr>
    <td>Bino (best) Config.</td>
    <td>0.996</td>
    <td>0.986</td>
    <td>0.986</td>
    <td>0.935</td>
    <td>0.999</td>
    <td>0.969</td>
    <td>1.000</td>
    <td>0.999</td>
    <td>0.953</td>
    <td>0.997</td>
    <td>0.976</td>
  </tr>
  <tr>
    <td>Bino (min)</td>
    <td>0.650</td>
    <td>0.671</td>
    <td>0.606</td>
    <td>0.207</td>
    <td>0.871</td>
    <td>0.293</td>
    <td>0.826</td>
    <td>0.668</td>
    <td>0.436</td>
    <td>0.698</td>
    <td>0.473</td>
  </tr>
  <tr>
    <td>Bino (avg)</td>
    <td>0.893</td>
    <td>0.894</td>
    <td>0.874</td>
    <td>0.616</td>
    <td>0.971</td>
    <td>0.675</td>
    <td>0.967</td>
    <td>0.925</td>
    <td>0.755</td>
    <td>0.935</td>
    <td>0.795</td>
  </tr>
  <tr>
    <td>Fast (best) Config.</td>
    <td>0.996</td>
    <td>0.977</td>
    <td>0.982</td>
    <td>0.947</td>
    <td>0.996</td>
    <td>0.972</td>
    <td>1.000</td>
    <td>0.998</td>
    <td>0.954</td>
    <td>0.995</td>
    <td>0.983</td>
  </tr>
  <tr>
    <td>Fast (min)</td>
    <td>0.512</td>
    <td>0.433</td>
    <td>0.504</td>
    <td>0.366</td>
    <td>0.477</td>
    <td>0.343</td>
    <td>0.693</td>
    <td>0.438</td>
    <td>0.522</td>
    <td>0.360</td>
    <td>0.641</td>
  </tr>
  <tr>
    <td>Fast (avg)</td>
    <td>0.848</td>
    <td>0.816</td>
    <td>0.834</td>
    <td>0.680</td>
    <td>0.859</td>
    <td>0.681</td>
    <td>0.932</td>
    <td>0.850</td>
    <td>0.781</td>
    <td>0.826</td>
    <td>0.853</td>
  </tr>
</table>

Table 1: Summary of Bino(ulars) & Fast(DetectGPT) AUROC on RAID with the Falcon family. Best, avg and min cells respectively report the best, average and worst score among all configurations.“-i” and “-c” respectively denote the instruct and chat version of the model.

<table>
  <tr>
    <th></th>
    <th>chatgpt</th>
    <th>cohere-c</th>
    <th>cohere</th>
    <th>gpt2</th>
    <th>gpt3</th>
    <th>gpt4</th>
    <th>llama-c</th>
    <th>mistral-c</th>
    <th>mistral</th>
    <th>mpt-c</th>
    <th>mpt</th>
  </tr>
  <tr>
    <td>Bino (best) Config.</td>
    <td>0.996</td>
    <td>0.985</td>
    <td>0.979</td>
    <td>0.812</td>
    <td>0.999</td>
    <td>0.969</td>
    <td>1.000</td>
    <td>0.998</td>
    <td>0.915</td>
    <td>0.999</td>
    <td>0.946</td>
  </tr>
  <tr>
    <td>Bino (min)</td>
    <td>0.511</td>
    <td>0.688</td>
    <td>0.711</td>
    <td>0.459</td>
    <td>0.945</td>
    <td>0.376</td>
    <td>0.741</td>
    <td>0.560</td>
    <td>0.609</td>
    <td>0.661</td>
    <td>0.637</td>
  </tr>
  <tr>
    <td>Bino (avg)</td>
    <td>0.837</td>
    <td>0.900</td>
    <td>0.870</td>
    <td>0.652</td>
    <td>0.983</td>
    <td>0.720</td>
    <td>0.928</td>
    <td>0.876</td>
    <td>0.774</td>
    <td>0.895</td>
    <td>0.798</td>
  </tr>
  <tr>
    <td>Fast (best) Config.</td>
    <td>0.994</td>
    <td>0.981</td>
    <td>0.979</td>
    <td>0.858</td>
    <td>0.996</td>
    <td>0.974</td>
    <td>1.000</td>
    <td>0.993</td>
    <td>0.923</td>
    <td>0.995</td>
    <td>0.966</td>
  </tr>
  <tr>
    <td>Fast (min)</td>
    <td>0.505</td>
    <td>0.673</td>
    <td>0.705</td>
    <td>0.501</td>
    <td>0.914</td>
    <td>0.363</td>
    <td>0.740</td>
    <td>0.552</td>
    <td>0.606</td>
    <td>0.647</td>
    <td>0.636</td>
  </tr>
  <tr>
    <td>Fast (avg)</td>
    <td>0.802</td>
    <td>0.853</td>
    <td>0.860</td>
    <td>0.684</td>
    <td>0.961</td>
    <td>0.691</td>
    <td>0.918</td>
    <td>0.869</td>
    <td>0.796</td>
    <td>0.866</td>
    <td>0.830</td>
  </tr>
</table>

Table 2: Summary of Bino(ulars) & Fast(DetectGPT) AUROC on RAID with Llama and Tower models. Best, avg, and min cells report respectively the max, average and worst score among all configurations. “T” and “L” respectively stand for TowerBase and Llama-2-7b, while “-c” denotes the “chat” version.

where the distribution \( \mu^*(\cdot|\mathbf{y}_{<t>}) \) of the random variable \( \mathbb{M} \) over LLM indices in \( \mathcal{M} \) satisfies:
\[
\mu^*(\cdot|\mathbf{y}_{<t>}) \triangleq \underset{\mu \in \mathcal{P}(\Omega)}{\operatorname{argmax}} \mathcal{I}_p(\mathbb{M}; Y_t|\mathbf{y}_{<t>}) .
\] (5)

Furthermore, the weights \( \{\mu^*(m|\mathbf{y}_{<t>})\}_{m \in \mathcal{M}} \) depend on the prefix \( \mathbf{y}_{<t>} \); they can be efficiently computed using the well-known Blahut–Arimoto algorithm (Arimoto, 1972; Blahut, 1972).

Definition 1 (MOSAIC Score). For an input sentence \( \mathbf{y} = (y_0, y_1, \ldots) \), and models indexed by \( \mathcal{M} = \{1, \ldots, M\} \) sharing a common tokenizer, the MOSAIC score is then defined as:
\[
S_{m^*, \mathcal{M}}(\mathbf{y}) \triangleq \frac{1}{T} \sum_{t=1}^T \sum_{y \in \Omega} \left[ \underbrace{\mathbb{I}_{\{y = y_t\}} \mathcal{L}_{q^*}(y_t \mid \mathbf{y}_{<t>})}_{\text{(codelength for observed token)}} \right.
\]
\[
\left. - \underbrace{p_{m^*}(y_t \mid \mathbf{y}_{<t>}) \mathcal{L}_{q^*}(y_t \mid \mathbf{y}_{<t>})}_{\text{(codelength for generated token from model } m^*)} \right].
\] (6)

where \( \mathcal{L}_{q^*}(y_t|\mathbf{y}_{<t>}) = -\log q^*(y_t|\mathbf{y}_{<t>}) \) and \( m^* \) is the reference model (3) with lowest perplexity on human texts. This formulation highlights the similarity between MOSAIC and Binoculars scores, differing only in how they compute average surprisal: a fixed model for Binoculars vs. a position-dependent mixture for MOSAIC. This score is used to detect artificial texts as follows: given an appropriate threshold \( \delta > 0 \) and a sample text \( \mathbf{y} \), if \( S_{m^*, \mathcal{M}}(\mathbf{y}) \geq \delta \), the text is classified as human; otherwise, it is considered AI-generated.

A formal description of how this scoring system is implemented is provided in Algorithm 1.

5 Experimental Settings

5.1 Datasets & Metrics

<table>
  <tr>
    <th>Corpus Name</th>
    <th># Gen</th>
    <th># texts/avg len</th>
    <th># texts/avg len</th>
  </tr>
  <tr>
    <td>RAID (sampling)</td>
    <td>11</td>
    <td>1k</td>
    <td>452</td>
    <td>11k</td>
    <td>373</td>
  </tr>
  <tr>
    <td>RAID adversarial</td>
    <td>11</td>
    <td>1k</td>
    <td>452</td>
    <td>11k</td>
    <td>658</td>
  </tr>
  <tr>
    <td>RAID+</td>
    <td>2</td>
    <td>1k</td>
    <td>452</td>
    <td>2k</td>
    <td>410</td>
  </tr>
  <tr>
    <td>M4 (Multilingual)</td>
    <td>1</td>
    <td>15k</td>
    <td>729</td>
    <td>15k</td>
    <td>649</td>
  </tr>
</table>

Table 3: Dataset details: RAID+ was specifically generated for this study using the models considered in our ensembling experiments. # Gen indicates the numbers of generators used for the artificial texts, avg len represents the average length of texts in Llama-2 tokens.

We evaluate our method on a diverse set of texts and generative models from the literature: RAID (Dugan et al., 2024) and M4 (Wang et al., 2024b).

RAID contains about 15k natural texts in English from a variety of domains; the artificial part version contains approximately 500k, generated
Figure 3: Comparing MOSAIC AUROC on the RAID dataset with different methods using the Falcon family (top plot) and Llama and Tower models (bottom plot). Model names represent the LLM used to generate the dataset.

with a diverse set of recent models, also varying the sampling procedure. As the test set is not publicly released, we select a random subset of 1,000 human texts and their generated counterparts with 11 different models and ancestral sampling for our experiments. RAID also includes an artificially noised subcorpus, we report the results for the different adversarial attacks in Table 9 in the Appendix.

The M46 corpus is a massive dataset of natural texts collected from a diverse set of sources (Wang et al., 2024b). Comparable artificial texts are generated by 6 LLMs, with prompts such as article titles, headlines, or abstracts depending on their domain. In our experiments, we only use one multilingual generator (ChatGPT, https://chatgpt.com/), and the “balanced” sets made of 3,000 pairs of (artificial, natural) texts in Chinese, Russian, Bulgarian, Arabic, and German.

We augmented the RAID extracts with texts generated using models that are in our ensembles,7 using the same prompts as in the RAID dataset. We refer to this augmented RAID as RAID+.

These datasets, presented in Table 3, represent a large variety of genres, themes, languages, sampling strategies, and generators, allowing us to thoroughly assess our detection strategy in different settings. Using RAID+, we can also evaluate detection performance for texts produced by one of our detectors.

Metrics. As in most studies, we report the AUROC score as our main evaluation metric. Depending on the application, True Positive Rate (TPR) for a predefined False Positive rate (e.g., 5%) is also worth looking at and reported. All these scores are obtained using scikit-learn (Pedregosa et al., 2011).

5.2 Choice of Models

In the next experiments, we use two ensembles of four models each, sharing a common vocabulary (65k for Falcon, 32k for Llama-2):

• The Falcon family (Almazrouei et al., 2023) (Falcon-7b, Falcon-7b-instruct, Falcon-40b, Falcon-40b-instruct).

• Llama (Touvron et al., 2023) model variations (Llama-2-7b, Llama-2-7b-chat, TowerBase-7b and TowerBase-13b (Alves et al., 2024)).

Both ensembles contain LLMs used in the original Binoculars paper, to which we added models to deepen the analysis of our ensembling techniques. We only consider pairs or ensembles of models within one group, as computing the cross-entropy term in equations (2) and (6) requires models to share the same tokenizer.

7For Multi-Lingual, Multi-Domain, Multi-Generator Machine-Generated text.
Generation uses ancestral sampling.
<table>
  <tr>
    <th>AUROC</th>
    <th>chatgpt</th>
    <th>cohere-c</th>
    <th>cohere</th>
    <th>gpt2</th>
    <th>gpt3</th>
    <th>gpt4</th>
    <th>llama-c</th>
    <th>mistral-c</th>
    <th>mistral</th>
    <th>mpt-c</th>
    <th>mpt</th>
  </tr>
  <tr>
    <td>MOSAIC-2</td>
    <td>0.928</td>
    <td>0.978</td>
    <td>0.968</td>
    <td>0.803</td>
    <td>0.994</td>
    <td>0.902</td>
    <td>0.998</td>
    <td>0.970</td>
    <td>0.895</td>
    <td>0.990</td>
    <td>0.920</td>
  </tr>
  <tr>
    <td>MOSAIC-3 (i)</td>
    <td>0.917</td>
    <td>0.978</td>
    <td>0.971</td>
    <td>0.818</td>
    <td>0.994</td>
    <td>0.898</td>
    <td>0.996</td>
    <td>0.969</td>
    <td>0.903</td>
    <td>0.988</td>
    <td>0.925</td>
  </tr>
  <tr>
    <td>MOSAIC-3 (ii)</td>
    <td>0.972</td>
    <td>0.980</td>
    <td>0.979</td>
    <td>0.802</td>
    <td>0.997</td>
    <td>0.956</td>
    <td>0.998</td>
    <td>0.987</td>
    <td>0.909</td>
    <td>0.994</td>
    <td>0.922</td>
  </tr>
  <tr>
    <td>MOSAIC-4</td>
    <td>0.970</td>
    <td>0.979</td>
    <td>0.980</td>
    <td>0.815</td>
    <td>0.997</td>
    <td>0.955</td>
    <td>0.998</td>
    <td>0.988</td>
    <td>0.914</td>
    <td>0.994</td>
    <td>0.927</td>
  </tr>
  <tr>
    <th>TPR@5%FPR</th>
    <th>chatgpt</th>
    <th>cohere-c</th>
    <th>cohere</th>
    <th>gpt2</th>
    <th>gpt3</th>
    <th>gpt4</th>
    <th>llama-c</th>
    <th>mistral-c</th>
    <th>mistral</th>
    <th>mpt-c</th>
    <th>mpt</th>
  </tr>
  <tr>
    <td>MOSAIC-2</td>
    <td>0.674</td>
    <td>0.923</td>
    <td>0.893</td>
    <td>0.365</td>
    <td>0.984</td>
    <td>0.636</td>
    <td>0.999</td>
    <td>0.881</td>
    <td>0.577</td>
    <td>0.960</td>
    <td>0.643</td>
  </tr>
  <tr>
    <td>MOSAIC-3 (i)</td>
    <td>0.629</td>
    <td>0.922</td>
    <td>0.904</td>
    <td>0.377</td>
    <td>0.984</td>
    <td>0.610</td>
    <td>0.998</td>
    <td>0.871</td>
    <td>0.584</td>
    <td>0.948</td>
    <td>0.644</td>
  </tr>
  <tr>
    <td>MOSAIC-3 (ii)</td>
    <td>0.844</td>
    <td>0.937</td>
    <td>0.924</td>
    <td>0.314</td>
    <td>0.990</td>
    <td>0.769</td>
    <td>1.000</td>
    <td>0.938</td>
    <td>0.546</td>
    <td>0.978</td>
    <td>0.586</td>
  </tr>
  <tr>
    <td>MOSAIC-4</td>
    <td>0.831</td>
    <td>0.936</td>
    <td>0.928</td>
    <td>0.351</td>
    <td>0.991</td>
    <td>0.771</td>
    <td>1.000</td>
    <td>0.943</td>
    <td>0.568</td>
    <td>0.976</td>
    <td>0.624</td>
  </tr>
</table>

Table 4: Performance of MOSAIC on the RAID dataset with a varying number of underlying models. “-c” indicates the chat version of a model. 2 models: Llama-2-7b and its chat version, 3 models (i): both Llama+ TowerBase-7B, 3 models (ii): both Llama+ TowerBase-13B, 4 models: all 4 Llama and Tower models.

6 Experimental Results

6.1 Results on RAID

In Table 1, we perform a systematic search to figure out the “best” performing configuration for each generator model used in RAID, that we call “Oracle” in Figure 3. The optimal model selection varies by dataset, making it incompatible with a “generator-agnostic” approach. As the average Binoculars and FastDetectGPT scores across combinations indicate, most settings yield poor performance, highlighting the need for a robust criterion to identify effective model combinations.

In Figure 3, we observe that MOSAIC performs as well as the other methods’ oracle configurations, with the exception of GPT2 generations. This can be explained by the poorer quality of the outputs, increasing the surprisal of artificial texts, thus misleading the detectors. The poor average performance across all combinations on the data generated by this smaller model supports this argument. In contrast, most static two-model combinations struggle to detect GPT4 accurately. However, since GPT4’s outputs are of very high quality, MOSAIC remains effective in identifying them.

6.1.1 Augmenting the Ensemble

To demonstrate the effectiveness of ensembling, we showcase the results of MOSAIC using 2, 3 and 4 models. We first look at the results of the Llama-2-7b and Llama-2-7b-chat pair, then add TowerBase-7B-v0.1, then TowerBase-13B-v0.1. Following our criterion of “lowest surprisal over the human texts” defined in Section 3.3, the model used to compute conditional entropy for the 2 and 3 model setting is Llama-2-7b, and TowerBase-13B-v0.1 is used for the 4-model setting.

In Table 4, we see that adding the “best” model, TowerBase-13B, leads to improved performances, with the exception of “mistral” and “mpt” generations. Adding the 7B version of Tower does not change much the results of our method, probably because its capabilities in English are similar to those of the already available Llama models. Augmenting the ensemble seems to have the least effect when the generator model is small, as GPT2, Mistral and MPT are the ones showing few improvements when adding models, even worsening results at times.

6.2 Including the Generator in the Ensemble

With our 1,000 human samples of RAID and the prompts used, we generated 1,000 texts using ancestral sampling with both Falcon-40b and Llama-2-7b, then used MOSAIC with respectively the Falcon and the Llama families. MOSAIC with Falcon models obtains an AUC of 0.965 for the text generated by Falcon-40b, and MOSAIC with the Llama and Tower models obtains an AUC of 0.986 for the texts generated by Llama-2-7b. These results are good but surprisingly average for the methods, suggesting that the size of the generator and the quality of its outputs matter more than its usage as a detector. In our setting, having the generator in the ensemble does not appear particularly advantageous.

6.3 Choosing the Best “Reference” Model

As mentioned in Section 3.3, the model m used to compute the conditional entropy needs to be the one with the least surprisal when looking at human texts. When having a family of models available, the one with the most parameters often ends up being better at this task and becomes the obvious
choice. However, we purposely chose the Tower models for their multilingual capabilities, despite a number of parameters similar to the Llama models. Table 5 reports the log-perplexities of these models on the “human” parts of the M4 dataset. Table 6 then gives us the performance of MOSAIC when using different \( m \) models to compute the conditional entropy. Our heuristic of selecting the reference model based on its low perplexity on human texts (see Section 3.3) consistently gives the best results. This simple criterion only requires human development texts; if unavailable, the largest model in the ensemble can serve as a good proxy.

<table>
  <tr>
    <th></th>
    <th>Arabic</th>
    <th>Bulgarian</th>
    <th>Chinese</th>
    <th>German</th>
    <th>Russian</th>
  </tr>
  <tr>
    <td>TowerBase-13B</td>
    <td>1.2743</td>
    <td>1.8052</td>
    <td>2.3047</td>
    <td>1.4912</td>
    <td>1.5069</td>
  </tr>
  <tr>
    <td>TowerBase-7B</td>
    <td>1.3929</td>
    <td>1.9839</td>
    <td>2.3327</td>
    <td>1.6169</td>
    <td>1.6084</td>
  </tr>
  <tr>
    <td>Llama-2-7b-chat</td>
    <td>1.7379</td>
    <td>2.3175</td>
    <td>2.6800</td>
    <td>2.1189</td>
    <td>2.2917</td>
  </tr>
  <tr>
    <td>Llama-2-7b</td>
    <td>1.3506</td>
    <td>1.8291</td>
    <td>2.1286</td>
    <td>1.6117</td>
    <td>1.7778</td>
  </tr>
</table>

Table 5: Log-perplexity values of our models for the “human” texts in M4

<table>
  <tr>
    <th>Model \( m \)</th>
    <th>Arabic</th>
    <th>Bulgarian</th>
    <th>Chinese</th>
    <th>German</th>
    <th>Russian</th>
  </tr>
  <tr>
    <td>TowerBase-13B</td>
    <td>0.9563</td>
    <td>0.9888</td>
    <td>0.9752</td>
    <td>0.9311</td>
    <td>0.9148</td>
  </tr>
  <tr>
    <td>TowerBase-7B</td>
    <td>0.9111</td>
    <td>0.9578</td>
    <td>0.9558</td>
    <td>0.8679</td>
    <td>0.8569</td>
  </tr>
  <tr>
    <td>Llama-2-7b-chat</td>
    <td>0.7768</td>
    <td>0.8262</td>
    <td>0.5849</td>
    <td>0.6751</td>
    <td>0.4321</td>
  </tr>
  <tr>
    <td>Llama-2-7b</td>
    <td>0.8947</td>
    <td>0.9762</td>
    <td>0.9059</td>
    <td>0.9200</td>
    <td>0.6814</td>
  </tr>
</table>

Table 6: AUROC of MOSAIC on the M4 dataset when varying the “reference” model \( m^* \).

6.4 Using Multiple Variants of a Single Model
A lighter and faster implementation of MOSAIC can be developed based on one single model, the logits of which are modified in order to simulate several probability distributions, that can then be ensembled. In fact, many generation techniques are based on the adaptation of the distribution of an underlying model (Meister et al., 2023), such as top-k sampling (Fan et al., 2018), top-p or nucleus sampling (Holtzman et al., 2020), \( \eta \)-sampling (Hewitt et al., 2022), etc. Using such techniques in MOSAIC only requires to load in memory and perform inference with just one model; it also readily satisfies the constraint that all distributions in the ensemble should share a common vocabulary.

We explored this with Llama-2-7b and four different values of nucleus sampling.\footnote{In our implementation, we use a smoothed version of the adapted distribution, so that all tokens have a small probability to be sampled. Having the same support for all distributions is required to compute the cross-entropy term in Eq. (6).} Results are reported in the Appendix (Table 8). This choice leads to results that are weaker than the 2-models setting depicted in Table 4, suggesting that applying this top-p in four different ways does not introduce as much diversity as the instruct model does. Another downfall of this approach is that the selection of \( m \), the “reference model”, can no longer rely on perplexity scores, as these cannot be reliably computed when with truncated vocabularies. In our experiment, we thus experimented with all potential values. Further work is needed to draw more precise analyses of this use-case.

6.5 Robustness Issues
Using the same 1000 human samples of RAID considered in all our previous experiments, we randomly sampled 1000 artificially generated texts for each adversarial attack available in RAID. Performance on this “noised” test set are in Table 9 in the Appendix. MOSAIC (with Llama and Tower) is quite resilient to such modifications, with the exception of “synonym” and “zero-width space” attacks, which significantly deteriorate the performance of the method due to the large change in surprisal they induce in the generated texts.

6.6 Uniform Ensembling
Instead of using the Blahut-Arimoto algorithm for optimal model combination, we naively assigned equal weights to each model in the ensemble. These results are reported in Table 7 on row “MOSAIC (uniform Falcon)”. We observe that this uniform mixture yields good overall results; yet, it still underperforms our theory-driven ensembling method for every generator in the RAID dataset.

7 Conclusion
Through all of our experiments, we have shown that the MOSAIC method effectively combines all the models in the ensemble, achieving very strong results across all datasets and languages. This method has multiple advantages: it is fully unsupervised, dispenses with the search for the optimal detector(s) when several are available, while offering a scalable solution that can incorporate a growing number of models. Even adversarial attacks have only a minimal effect on the performance, despite our scoring system not being optimized for perturbations. However, MOSAIC is currently computationally costly, as each model must run on the input text.

While not in the main scope on this paper, a potential improvement of computational efficiency
is proposed in Section 6.4, and further solutions are discussed in Appendix E. Furthermore, more work needs to be done in order to fully understand how to evaluate the “distance” in-between models’ outputs, in order to choose the best ensemble that would cover all potential generations.

The optimal mixture in MOSAIC, defined by Blahut-Arimoto weights to minimize encoding length in worst-case scenarios, could be useful beyond detecting machine-generated texts. We leave this exploration for future work.

8 Limitations

As mentioned above, the computational cost of the method, running multiple LLMs as well as the Blahut-Arimoto algorithm for each token is currently an issue, being able to analyse the 1000 texts of each RAID sub-dataset with 4 models in about an hour on one A100 GPU if using the Llama and Tower models, and three if using the Falcon ones. Theory-wise, the main issue with our method is the need to compute the cross-entropy between models, forcing them to have the same vocabulary, thus greatly limiting the number of models we can combine at once.

9 Ethical Considerations

It should be acknowledged that artificial text detection tools are not infallible and consequently should not be used as the sole basis for punitive actions or decisions that could affect individuals or organizations. These methods must be complemented by human oversight and verification before taking any drastic measure, to ensure fairness of treatment.

Acknowledgments

This work was performed using HPC resources from GENCI–IDRIS (Grant 2023-AD011014903).

References

Mervat Abassy, Kareem Elozeiri, Alexander Aziz, Minh Ngoc Ta, Raj Vardhan Tomar, Bimarsha Adhikari, Saad El Dine Ahmed, Yuxia Wang, Osama Mohammed Afzal, Zhuhuan Xie, Jonibek Mansurov, Ekaterina Artemova, Vladislav Mikhailov, Rui Xing, Jiahui Geng, Hasan Iqbal, Zain Muhammad Mujahid, Tarek Mahmoud, Akim Tsvigun, Alham Fikri Aji, Artem Shelmanov, Nizar Habash, Iryna Gurevych, and Preslav Nakov. 2024. LLM-DetectAIve: a tool for fine-grained machine-generated text detection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 336–343, Miami, Florida, USA. Association for Computational Linguistics.

Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérourane Debba, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. The Falcon Series of Open Language Models. Preprint. arxiv:2311.16867.

Duarte Miguel Alves, José Pombal, Nuno M Guerreiro, Pedro Henrique Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and Andre Martins. 2024. Tower: An open multilingual large language model for translation-related tasks. In First Conference on Language Modeling.

Wissam Antoun, Benoît Sagot, and Djamel Seddah. 2024. From text to source: Results in detecting large language model-generated content. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 7531–7543, Torino, Italia. ELRA and ICCL.

Suguru Arimoto, 1972. An algorithm for computing the capacity of arbitrary discrete memoryless channels. IEEE Transactions on Information Theory, 18(1):14–20.

Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. 2024. Fast-detectGPT: Efficient zero-shot detection of machine-generated text via conditional probability curvature. In The Twelfth International Conference on Learning Representations.

Andrew R. Barron, Jorma Rissanen, and Bin Yu. 1998. The minimum description length principle in coding and modeling. IEEE Trans. Inf. Theory, 44(6):2743–2760.

Richard Blahut. 1972. Computation of channel capacity and rate-distortion functions. IEEE Transactions on Information Theory, 18(4):460–473.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.

Thomas M. Cover and Joy A. Thomas. 2006. Elements of Information Theory, 2nd edition. Wiley, New York, NY.
Evan N. Crothers, Nathalie Japkowicz, and Herna L. Viktor. 2023. Machine-generated text: A comprehensive survey of threat models and detection methods. IEEE Access, 11:70977–71002.

Liam Dugan, Alyssa Hwang, Filip Trhlík, Andrew Zhu, Josh Magnus Ludan, Hainiu Xu, Daphne Ippolito, and Chris Callison-Burch. 2024. RAID: A shared benchmark for robust evaluation of machine-generated text detectors. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12463–12492, Bangkok, Thailand. Association for Computational Linguistics.

Ahmed El-Sayed and Omar Nasr. 2023. An ensemble based approach to detecting LLM-generated texts. In Proceedings of the 21st Annual Workshop of the Australasian Language Technology Association, pages 164–168, Melbourne, Australia. Association for Computational Linguistics.

Tiziano Fagni, Fabrizio Falchi, Margherita Gambini, Antonio Martella, and Maurizio Tesconi. 2021. Tweet-Fake: About detecting deepfake tweets. PLOS ONE, 16(5):e0251415. Publisher: Public Library of Science.

Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898, Melbourne, Australia. Association for Computational Linguistics.

Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. 2019. GLTR: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 111–116, Florence, Italy. Association for Computational Linguistics.

Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection. Preprint, arxiv:2301.07597.

Abhimanyu Hans, Avi Schwarzschild, Valeria Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2024. Spotting LLMs with binoculars: Zero-shot detection of machine-generated text. In Forty-first International Conference on Machine Learning.

John Hewitt, Christopher Manning, and Percy Liang. 2022. Truncation sampling as language model desmoothing. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3414–3427, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In Proceedings of the International Conference on Learning Representations, ICLR.

Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. 2020. Automatic detection of generated text is easiest when humans are fooled. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1808–1822, Online. Association for Computational Linguistics.

John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. A Watermark for Large Language Models. In Proceedings International Conference on Machine Learning.

John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. 2024. On the Reliability of Watermarks for Large Language Models. In The Twelfth International Conference on Learning Representations.

Tharindu Kumarage, Joshua Garland, Amrita Bhattacherjee, Kirill Trapeznikov, Scott Ruston, and Huan Liu. 2023. Stylistic Detection of AI-Generated Text in Twitter Timelines. Preprint, arXiv:2306.05524. ArXiv:2303.03697 [cs].

Linyang Li, Pengyu Wang, Ke Ren, Tianxiang Sun, and Xipeng Qiu. 2023. Origin Tracing and Detecting of LLMs. arXiv preprint. ArXiv:2304.14072 [cs].

Yepeng Liu and Yuheng Bu. 2024. Adaptive text watermark for large language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 30718–30737. PMLR.

Zeyan Liu, Zijun Yao, Fengjun Li, and Bo Luo. 2024. On the detectability of ChatGPT content: Benchmarking, methodology, and evaluation through the lens of academic writing. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, CCS ’24, page 2236–2250, New York, NY, USA. Association for Computing Machinery.

Vijini Liyanage and Davide Buscaldi. 2023. An ensemble method based on the combination of transformers with convolutional neural networks to detect artificially generated text. In Proceedings of the 21st Annual Workshop of the Australasian Language Technology Association, pages 107–111, Melbourne, Australia. Association for Computational Linguistics.

Vijini Liyanage, Davide Buscaldi, and Adeline Nazarenko. 2022. A benchmark corpus for the detection of automatically generated text in academic publications. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4692–4700, Marseille, France. European Language Resources Association.

Chengzhi Mao, Carl Vondrick, Hao Wang, and Junfeng Yang. 2024. Raidar: geneRative AI detection via rewriting. In The twelfth international conference on learning representations.
Clara Meister, Tiago Pimentel, Luca Malagutti, Ethan Wilcox, and Ryan Cotterell. 2023. On the efficacy of sampling adapters. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1437–1455, Toronto, Canada. Association for Computational Linguistics.

Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. 2023. DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature. In Proceedings International Conference on Machine Learning, ICML.

Sandra Mitrović, Davide Andreoletti, and Omran Ayoub. 2023. ChatGPT or Human? Detect and Explain. Explaining Decisions of Machine Learning Model for Detecting Short ChatGPT-generated Text. Preprint, arxiv:2301.13852.

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine learning in python. J. Mach. Learn. Res., 12(null):2825–2830.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21(140):1–67.

Edward W. Samson. 1953. Fundamental natural concepts of information theory. ETC: A Review of General Semantics, 10(4):283–297.

P.C. Shields. 1996. The Ergodic Theory of Discrete Sample Paths. Graduate studies in mathematics. American Mathematical Society.

Jorge F. Silva and Pablo Piantanida. 2022. On universal d-semiautiful coding for memoryless sources with infinite alphabets. IEEE Transactions on Information Theory, 68(4):2782–2800.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aunlien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. Preprint, arxiv:2302.13971.

Christoforos Vasilatos, Manaar Alam, Talal Rahwan, Yasir Zaki, and Michail Maniatakos. 2023. HowkGPT: Investigating the Detection of ChatGPT-generated University Student Homework through Context-Aware Perplexity Analysis. Preprint, arxiv:2305.18226.

Vivek Verma, Eve Fleisig, Nicholas Tomlin, and Dan Klein. 2024. Ghostbuster: Detecting text ghostwritten by large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1702–1717, Mexico City, Mexico. Association for Computational Linguistics.

John von Neumann. 1928. Zur theorie der gesellschaftsspiele. Mathematische Annalen, 100:295–320.

Pengyu Wang, Linyang Li, Ke Ren, Botian Jiang, Dong Zhang, and Xipeng Qiu. 2023. SeqXGPT: Sentence-level AI-generated text detection. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1144–1156, Singapore. Association for Computational Linguistics.

Yichen Wang, Shangbin Feng, Abe Hou, Xiao Pu, Chao Shen, Xiaoming Liu, Yulia Tsvetkov, and Tianxing He. 2024a. Stumbling blocks: Stress testing the robustness of machine-generated text detectors under attacks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2894–2925, Bangkok, Thailand. Association for Computational Linguistics.

Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Toru Sasaki, Thomas Arnold, Alham Fikri Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. 2024b. M4: Multi-generator, multi-domain, and multi-lingual black-box machine-generated text detection. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1369–1407, St. Julian’s, Malta. Association for Computational Linguistics.

Xianjun Yang, Wei Cheng, Yue Wu, Linda Ruth Petzold, William Yang Wang, and Haifeng Chen. 2024. DNA-GPT: Divergent n-gram analysis for training-free detection of GPT-generated text. In The twelfth international conference on learning representations, (ICLR), Vienna, Austria.

Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending Against Neural Fake News. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.

Hanlin Zhang, Benjamin L. Edelman, Danilo Francati, Daniele Venturi, Giuseppe Ateniese, and Boaz Barak. 2024. Watermarks in the sand: impossibility of strong watermarking for language models. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org.
A Information-Theoretic Principles Behind FastDetectGPT and Binoculars

As discussed in Section 3.3, FastDetectGPT closely resembles Binoculars. However, rather than directly computing the cross-entropy between two models, it draws \( N \) independent samples from the model and computes the empirical cross-entropy: \( \{ \hat{y}_i \sim p(Y_t|y_{<t}) \}_{i=1}^N \) and then return the score:

\[
S^{Fast}_{p,q}(y) = \frac{-\log q(y_t|y_{<t}) + \frac{1}{N} \sum_{i=1}^N \log q(\hat{y}_i|y_{<t})}{\tilde{\sigma}(y_{<t})}
\]

where

\[
\tilde{\sigma}^2(y_{<t}) \triangleq \frac{1}{N-1} \sum_{i=1}^N \left( -\log q(y_i|y_{<t}) \right. \\
\left. + \frac{1}{N} \sum_{j=1}^N \log q(y_j|y_{<t}) \right)^2
\]

is a normalisation term which is particularly useful for individual token detection.

B Theoretical Grounding of MOSAIC

Identifying explanations of data. We turn to the problem of determining an adequate sequence of models \( \hat{m} = (\hat{m}_0, \ldots, \hat{m}_T) \).

Our goal will be to derive a robust scoring algorithm that best extracts regularity in the data, which is equivalent to identifying the model that achieves the best compression of the input tokens. Suppose we are given a family of LLMs \( \mathcal{P}_M(\mathcal{Y}) \), with corresponding Shannon codelengths:

\[
\mathcal{L}_{p_m}(y_t|y_{<t}) \triangleq -\log p_m(y_t|y_{<t}),
\]

for each \( y_t \). These can be viewed as a collection of data compressors, indexed by \( m \). We can measure the performance of encoding \( y_t \) at time \( t \) relative to \( \mathcal{P}_M(\mathcal{Y}) \). If we choose to encode the token \( y_t \) with model \( q(y_t|y_{<t}) \), the resulting expected excess codelength (or overhead) w.r.t. any distribution \( p_m \in \mathcal{P}_M(\mathcal{Y}) \) is:

\[
\mathcal{R}_m(q; y_{<t}) \triangleq \mathbb{E}_{y_t \sim p_m(y_t|y_{<t})} \left[ -\log q(y_t|y_{<t}) \right] - \mathcal{H}_{p_m}(Y_t|y_{<t})
\]

which is non-negative since \( \mathcal{H}_{p_m}(Y_t|y_{<t}) \) is the minimum expected codelength. \( \mathcal{R}_m \) represents the extra averaged number of bits needed to encode \( y_t \) using the code/LLM \( q(y_t|y_{<t}) \), as compared to \( \mathcal{H}_{p_m}(Y_t|y_{<t}) \), the number of bits needed if we would have used the best fitting LLM in \( \mathcal{P}_M(\mathcal{Y}) \) with hindsight. However, the encoder cannot know the underlying model artificially generating \( y_t \) so we take a worst-case approach and look for universal LLMs with small worst-case expected overhead, where the worst-case is over all models in \( \mathcal{P}_M(\mathcal{Y}) \). \( \mathcal{R}_m \) is our quality measure and hence, the ‘optimal’ LLM relative to \( \mathcal{P}_M(\mathcal{Y}) \), for a given context \( y_{<t} \), is the distribution minimizing:

\[
q^*(y_t|y_{<t}) \triangleq \argmin_{q \in \mathcal{P}(\Omega)} \max_{m \in \mathcal{M}} \mathcal{R}_m(q; y_{<t})
\]

where the minimum is over all distributions on \( \Omega \). The minimizer corresponds to the code with the smallest overhead (i.e., the fewest extra bits) relative to the optimal code, which is retrospectively the best choice in the worst-case model selection for generating synthetic text across all LLMs in the available family \( \mathcal{P}_M(\mathcal{Y}) \).

Leveraging codelengths for identifying AI-generated text. The averaged overhead of the optimal codelength \( -\log q^*(y_t|y_{<t}) \) obtained by solving Eq. (8) seems to be a very reasonable choice for building a robust score function to detect AI-generated text because of the following properties:

• The better the best-fitting LLM in \( \mathcal{P}_M(\mathcal{Y}) \) fits the artificially generated data, the shorter the codelength \( \mathcal{L}_{q^*}(y_t|y_{<t}) \triangleq -\log q^*(y_t|y_{<t}) \).

• No LLM in \( \mathcal{P}_M(\mathcal{Y}) \) is given a prior preference over any other since \( \mathcal{R}_m(q^*; y_{<t}) \leq \mathcal{R}_m(p; y_{<t}) \) for all \( p \in \mathcal{P}_M(\mathcal{Y}) \), i.e., we are treating all LLMs within our universe \( \mathcal{P}_M(\mathcal{Y}) \) on the same footing.

B.1 Proof of Proposition 1

Proof. We need to show the fundamental identity:

\[
\Gamma(y_{<t}) \triangleq \min_{q \in \mathcal{P}(\Omega)} \max_{m \in \mathcal{M}} \mathcal{R}_m(q; y_{<t}) \\
= \max_{\mu \in \mathcal{P}(\mathcal{M})} \mathcal{I}(\mathbb{M}; Y_t|y_{<t}),
\]

where the optimal \( q^*(y_t|y_{<t}) \) achieving the minimum is characterized by the mixture:

\[
q^*(y_t|y_{<t}) = \sum_{m \in \mathcal{M}} \mu^*(m|y_{<t}) p_m(y_t|y_{<t})
\]

and the distribution \( \mu^*(m|y_{<t}) \) of the random variable \( \mathbb{M} \) on \( \mathcal{M} \) follows by solving:

\[
\mu^*(m|y_{<t}) \triangleq \argmax_{\mu \in \mathcal{P}(\Omega)} \mathcal{I}(\mathbb{M}; Y_t|y_{<t}).
\]
To this end, we start from the definition \( \mathcal{R}_m \):
\[
\mathcal{R}_m(q; \mathbf{y}_{<t}) \triangleq \mathbb{E}_{y_t \sim p_m(y_t|\mathbf{y}_{<t})} [-\log q(y_t|\mathbf{y}_{<t})]
- \min_{q' \in \mathcal{P}(\Omega)} \mathbb{E}_{y_t \sim p_m(y_t|\mathbf{y}_{<t})} [-\log q'(y_t|\mathbf{y}_{<t})]
= \mathbb{E}_{y_t \sim p_m(y_t|\mathbf{y}_{<t})} [-\log q(y_t|\mathbf{y}_{<t})] - \mathcal{H}_{p_m}(Y_t|\mathbf{y}_{<t})
= \mathcal{D}_{\mathrm{KL}}\left(p_m(\cdot|\mathbf{y}_{<t})||q(\cdot|\mathbf{y}_{<t})\right),
\]
where \( \mathcal{D}_{\mathrm{KL}}(\cdot||\cdot) \) denotes the Kullback–Leibler divergence. Hence, we can formally state our problem as follows:
\[
\Gamma(\mathbf{y}_{<t}) = \min_{q \in \mathcal{P}(\Omega)} \max_{m \in \mathcal{M}} \mathcal{R}_m(q; \mathbf{y}_{<t})
= \min_{q \in \mathcal{P}(\Omega)} \max_{m \in \mathcal{M}} \mathcal{D}_{\mathrm{KL}}\left(p_m(\cdot|\mathbf{y}_{<t})||q(\cdot|\mathbf{y}_{<t})\right)
= \min_{q \in \mathcal{P}(\Omega)} \max_{\mu \in \mathcal{P}(\mathcal{M})} \mathbb{E}_{m \sim \mu} \mathcal{D}_{\mathrm{KL}}\left(p_m(\cdot|\mathbf{y}_{<t})||q(\cdot|\mathbf{y}_{<t})\right),
\]
(14)
where the minimum is taken over all the possible distributions \( q \in \mathcal{P}(\Omega) \), representing the expected value of regret of \( q \) w.r.t. the worst-case distribution over \( \mu \in \mathcal{P}(\mathcal{M}) \). Notice that this is equivalent to the average worst-case regret (Barron et al., 1998; Silva and Piantanida, 2022). The equality in (14) holds by noticing that
\[
\max_{\mu \in \mathcal{P}(\mathcal{M})} \mathbb{E}_{m \sim \mu} \mathcal{D}_{\mathrm{KL}}\left(p_m(\cdot|\mathbf{y}_{<t})||q(\cdot|\mathbf{y}_{<t})\right)
\leq \max_{m \in \mathcal{M}} \mathcal{D}_{\mathrm{KL}}\left(p_m(\cdot|\mathbf{y}_{<t})||q(\cdot|\mathbf{y}_{<t})\right)
\]
and moreover,
\[
\max_{m \in \mathcal{M}} \mathcal{D}_{\mathrm{KL}}\left(p_m(\cdot|\mathbf{y}_{<t})||q(\cdot|\mathbf{y}_{<t})\right)
= \mathbb{E}_{m \sim \tilde{\mu}} \mathcal{D}_{\mathrm{KL}}\left(p_m(\cdot|\mathbf{y}_{<t})||q(\cdot|\mathbf{y}_{<t})\right)
\]
by choosing the measure \( \tilde{\mu} \) to be an uniform probability over the set \( \mathcal{M} \subseteq \mathcal{M} \), which is defined as the set of maximizers:
\[
\widetilde{\mathcal{M}} \equiv \argmax_{m \in \mathcal{M}} \mathcal{D}_{\mathrm{KL}}\left(p_m(\cdot|\mathbf{y}_{<t})||q(\cdot|\mathbf{y}_{<t})\right)
\]
and zero otherwise.

The convexity of the KL-divergence allows us to rewrite expression (14) as follows:
\[
\min_{q \in \mathcal{P}(\Omega)} \max_{\mu \in \mathcal{P}(\mathcal{M})} \mathbb{E}_{m \sim \mu} \mathcal{D}_{\mathrm{KL}}\left(p_m(\cdot|\mathbf{y}_{<t})||q(\cdot|\mathbf{y}_{<t})\right)
= \max_{\mu \in \mathcal{P}(\mathcal{M})} \min_{q \in \mathcal{P}(\Omega)} \mathbb{E}_{m \sim \mu} \mathcal{D}_{\mathrm{KL}}\left(p_m(\cdot|\mathbf{y}_{<t})||q(\cdot|\mathbf{y}_{<t})\right)
\]
(17)

This follows by considering a zero-sum game with a concave-convex mapping defined on a product of convex sets. The sets of all probability distributions \( \mathcal{P}(\mathcal{M}) \) and \( \mathcal{P}(\Omega) \) are two nonempty convex sets, bounded and finite-dimensional. On the other hand, \( (\mu, q) \to \mathbb{E}_{m \sim \mu} \mathcal{D}_{\mathrm{KL}}\left(p_m(\cdot|\mathbf{y}_{<t})||q(\cdot|\mathbf{y}_{<t})\right) \) is a concave-convex mapping, i.e.,
\[
\mu \to \mathbb{E}_{m \sim \mu} \mathcal{D}_{\mathrm{KL}}\left(p_m(\cdot|\mathbf{y}_{<t})||q(\cdot|\mathbf{y}_{<t})\right)
\]
is concave and,
\[
q \to \mathbb{E}_{m \sim \mu} \mathcal{D}_{\mathrm{KL}}\left(p_m(\cdot|\mathbf{y}_{<t})||q(\cdot|\mathbf{y}_{<t})\right)
\]
is convex for every \( (\mu, q) \), respectively. Then, by classical min-max theorem (von Neumann, 1928), we have that (17) holds.

Finally, it remains to show that:
\[
\min_{q \in \mathcal{P}(\Omega)} \mathbb{E}_{m \sim \mu} \mathcal{D}_{\mathrm{KL}}\left(p_m(\cdot|\mathbf{y}_{<t})||q(\cdot|\mathbf{y}_{<t})\right)
= \mathcal{I}(\mathbb{M}; Y_t|\mathbf{y}_{<t})
\]
(18)
for any random variable \( \mathbb{M} \) distributed according to the probability distribution \( \mu \in \mathcal{P}(\mathcal{M}) \) and each distribution \( p_m(y_t|\mathbf{y}_{<t}) \).

We begin by showing that:
\[
\mathbb{E}_{m \sim \mu} \mathcal{D}_{\mathrm{KL}}\left(p_m(\cdot|\mathbf{y}_{<t})||q(\cdot|\mathbf{y}_{<t})\right) \geq \mathcal{I}(\mathbb{M}; Y_t|\mathbf{y}_{<t})
\]
for all distributions \( q(\cdot|\mathbf{y}_{<t}) \) and \( p_m(y_t|\mathbf{y}_{<t}) \). To this end, we consider the following identities:
\[
\begin{align*}
&\mathbb{E}_{m \sim \mu} \mathcal{D}_{\mathrm{KL}}\left(p_m(\cdot|\mathbf{y}_{<t})||q(\cdot|\mathbf{y}_{<t})\right) \\
&= \mathbb{E}_{m \sim \mu} \mathcal{D}_{\mathrm{KL}}\left(p_m(\cdot|\mathbf{y}_{<t})||p_m(\cdot|\mathbf{y}_{<t})\right) \\
&\quad + \mathcal{D}_{\mathrm{KL}}\left(p_m(\cdot|\mathbf{y}_{<t})||q(\cdot|\mathbf{y}_{<t})\right) \\
&= \mathcal{I}(\mathbb{M}; Y_t|\mathbf{y}_{<t}) + \mathcal{D}_{\mathrm{KL}}\left(p_m(\cdot|\mathbf{y}_{<t})||q(\cdot|\mathbf{y}_{<t})\right) \\
&\geq \mathcal{I}(\mathbb{M}; Y_t|\mathbf{y}_{<t}),
\end{align*}
\]
(19)
where \( p_m(\cdot|\mathbf{y}_{<t}) \) denotes the marginal distribution of \( p_m(\cdot|\mathbf{y}_{<t}) \) w.r.t. \( \mu \) and the last inequality follows since the KL divergence is non-negative. Finally, it is easy to check that by selecting:
\[
q^*(y_t|\mathbf{y}_{<t}) = \mathbb{E}_{m \sim \mu} [p_m(y_t|\mathbf{y}_{<t})]
\]
the lower bound in (19) is achieved:
\[
\min_{q \in \mathcal{P}(\Omega)} \mathbb{E}_{m \sim \mu} \mathcal{D}_{\mathrm{KL}}\left(p_m(\cdot|\mathbf{y}_{<t})||q(\cdot|\mathbf{y}_{<t})\right)
= \mathbb{E}_{m \sim \mu} \mathcal{D}_{\mathrm{KL}}\left(p_m(\cdot|\mathbf{y}_{<t})||q^*(\cdot|\mathbf{y}_{<t})\right)
\]
(21)
(22)
for every \( \mu \in \mathcal{P}(\mathcal{M}) \), which proves the identity in expression (18).

The claim in (10) follows by taking the maximum overall probability measures \( \mu \in \mathcal{P}(\mathcal{M}) \) at both sides of (18), and combining the resulting identity with expressions (17) and (14). The mixture in (12) follows from expression (20) which is a necessary condition to solve the min-max problem.

\hfill \Box

C Blahut–Arimoto Algorithm

C.1 Algorithm

Our channel can be specified using two discrete random variables \((\mathcal{M}, Y_t)\) with alphabets \((\mathcal{M}, \Omega)\) and probability distributions \( \mu \) and \( p_m(y_t|y_{<t}) \), respectively, conditioned on \( y_{<t} \). The problem to be solved is the maximization of the mutual information, which consists in:

\[
\Gamma(y_{<t}) \triangleq \max_{\mu \in \mathcal{P}(\mathcal{M})} \mathcal{I}_m(\mathcal{M}; Y_t|y_{<t}). \tag{23}
\]

Now if we denote the cardinality \( |\mathcal{M}| = M, |\Omega| = N \), then \( p_m(y_t|y_{<t}) \) is an \( M \times N \) matrix, which we denote the \( i \)-th row, \( j \)-th column entry by \( w_{ij} \). For the case of channel capacity, the algorithm was introduced in (Blahut, 1972; Arimoto, 1972) to solve (23). They both found the following expression for the capacity of a discrete channel with channel law \( w_{ij} \):

\[
\Gamma(y_{<t}) = \max_{\mu} \max_Q \sum_{i=1}^M \sum_{j=1}^N \mu_i w_{ij} \log \left( \frac{q_{ji}}{\mu_i} \right),
\]

where \( \mu \) and \( Q \) are maximized over the following requirements:

• \( \mu \triangleq (\mu_1, \ldots, \mu_M) \) is a probability distribution on \( \mathcal{M} \). That is, \( \sum_{i=1}^M \mu_i = 1 \).

• \( Q = (q_{ji}) \) is an \( N \times M \) matrix that behaves like a transition matrix from \( \Omega \) to \( \mathcal{M} \) with respect to the channel law. That is, for all \( 1 \leq i \leq M, 1 \leq j \leq N \):
\[
q_{ji} \geq 0,\quad q_{ji} = 0 \Leftrightarrow w_{ij} = 0,
\]
and every row sums up to 1: \( \sum_{i=1}^M q_{ji} = 1 \).

Then, upon initializing a probability measure \( \mu^0 = (\mu_1^0, \mu_2^0, \ldots, \mu_M^0) \) on \( \mathcal{M} \), we can generate a sequence \( (\mu^0, Q^0, \mu^1, Q^1, \ldots) \) iteratively as follows:

\[
(q_{ji}^t) = \frac{\mu_i^t w_{ij}}{\sum_{k=1}^M \mu_k^t w_{kj}} \tag{24}
\]

and

\[
\mu_k^{t+1} = \frac{\prod_{j=1}^N (q_{jk}^t)^{w_{kj}}}{\sum_{i=1}^M \prod_{j=1}^N (q_{ji}^t)^{w_{ij}}} \tag{25}
\]

for \( t = 0, 1, 2, \ldots \).

Then, using the theory of optimization, specifically coordinate descent, it has been shown that the sequence indeed converges to the required maximum. That is,

\[
\lim_{t \to \infty} \sum_{i=1}^M \sum_{j=1}^N \mu_i^t w_{ij} \log \left( \frac{q_{ji}^t}{\mu_i^t} \right) = \Gamma(y_{<t}).
\]

So given a channel law \( p_m(y_t|y_{<t}) \), the (23) can be numerically estimated up to arbitrary precision.

C.2 Computational complexity

The computational complexity of the Blahut-Arimoto algorithm can be characterized as follows:

• **Number of iterations.** The algorithm typically converges linearly, so the number of iterations required, denoted as \( T \), is proportional to the desired accuracy of the solution.

• **Operations per iteration.** Each iteration involves updating the probability measures in (24) and (25), and evaluating the mutual information, which requires matrix manipulations. Let \( M \) and \( N \) be the cardinalities of the input and output alphabets, respectively. Each iteration involves operations over all input-output pairs, requiring \( \mathcal{O}(M \times N) \) operations.

Combining these, the overall computational complexity of the Blahut-Arimoto algorithm is \( \mathcal{O}(T \times n \times m) \), reflecting its dependence on the sizes of \( M \) (number of LLMs in the considered family) and \( N \) (the vocabulary), and the number of iterations needed for convergence, which depends intrinsically on the underlying distributions.

D Complementary Results

This section is dedicated to additional results. Table 7 shows the results of MOSAIC with both ensemble of models on the RAID dataset, along with the uniform mixture discussed in Section 6.6.

Table 8 reports the results of Section 6.4, using only one model but modifying its logits with
nucleus sampling in order to create different probability distributions.

Table 9 shows the performance of MOSAIC under the different adversarial attacks available in the RAID dataset.

E  Complexity Improvements

Our algorithm currently processes each text in approximately 10 seconds on NVIDIA 32G V100 GPUs, and twice as fast on 80G A100 GPUs. Runtime optimization is an area that should be improved in future work. Below, we outline limitations of our system and propose potential improvements: In MOSAIC, the texts are processed one-by-one by the LLMs. Each model is loaded onto a separate GPU, and the logits are moved to a central device for performing operations such as Blahut-Arimoto, perplexity, and cross-entropy calculations, after which the final score is computed. This setup has several inefficiencies. For instance, transferring logits to a central device introduces a significant bottleneck. Additionally, while calculations are performed on one GPU, the remaining ones remain idle, resulting in suboptimal use of resources.

A more efficient method would involve computing the logits for all texts in parallel, storing them across different GPUs, and performing subsequent calculations concurrently. An even more streamlined solution would involve loading all models onto a single GPU using quantized or distilled versions, thus eliminating the need to transfer logits across devices.

While these optimizations are promising, they have not been implemented in this work, as we focus on the algorithmic methodology rather than runtime efficiency.

F  A Systematic Study of all Potential Combinations

Tables 10, 11, 12 and 13 display the whole study of all the potential combinations of the four models in our ensembles for every generator in the RAID dataset.
<table>
  <tr>
    <th>AUROC</th>
    <th>chatgpt</th>
    <th>cohere-c</th>
    <th>cohere</th>
    <th>gpt2</th>
    <th>gpt3</th>
    <th>gpt4</th>
    <th>llama-c</th>
    <th>mistral-c</th>
    <th>mistral</th>
    <th>mpt-c</th>
    <th>mpt</th>
  </tr>
  <tr>
    <td>MOSAIC (Falcon family)</td>
    <td>0.984</td>
    <td>0.981</td>
    <td>0.980</td>
    <td>0.794</td>
    <td>0.997</td>
    <td>0.955</td>
    <td>0.996</td>
    <td>0.993</td>
    <td>0.903</td>
    <td>0.994</td>
    <td>0.932</td>
  </tr>
  <tr>
    <td>MOSAIC (uniform Falcon)</td>
    <td>0.969</td>
    <td>0.947</td>
    <td>0.942</td>
    <td>0.675</td>
    <td>0.987</td>
    <td>0.876</td>
    <td>0.988</td>
    <td>0.970</td>
    <td>0.774</td>
    <td>0.983</td>
    <td>0.796</td>
  </tr>
  <tr>
    <td>MOSAIC (Llama and Tower)</td>
    <td>0.970</td>
    <td>0.980</td>
    <td>0.979</td>
    <td>0.815</td>
    <td>0.997</td>
    <td>0.955</td>
    <td>0.998</td>
    <td>0.988</td>
    <td>0.913</td>
    <td>0.994</td>
    <td>0.927</td>
  </tr>
</table>

Table 7: Performance of MOSAIC on the RAID dataset with both families of models.

<table>
  <tr>
    <th></th>
    <th>chatgpt</th>
    <th>cohere-chat</th>
    <th>cohere</th>
    <th>gpt2</th>
    <th>gpt3</th>
    <th>gpt4</th>
    <th>llama-chat</th>
    <th>mistral-chat</th>
    <th>mistral</th>
    <th>mpt-chat</th>
    <th>mpt</th>
  </tr>
  <tr>
    <td>p = 0.7</td>
    <td>0.792</td>
    <td>0.665</td>
    <td>0.577</td>
    <td>0.356</td>
    <td>0.733</td>
    <td>0.503</td>
    <td>0.869</td>
    <td>0.763</td>
    <td>0.465</td>
    <td>0.731</td>
    <td>0.488</td>
  </tr>
  <tr>
    <td>p = 0.8</td>
    <td>0.789</td>
    <td>0.665</td>
    <td>0.577</td>
    <td>0.354</td>
    <td>0.732</td>
    <td>0.502</td>
    <td>0.867</td>
    <td>0.760</td>
    <td>0.462</td>
    <td>0.731</td>
    <td>0.485</td>
  </tr>
  <tr>
    <td>p = 0.9</td>
    <td>0.781</td>
    <td>0.663</td>
    <td>0.578</td>
    <td>0.358</td>
    <td>0.725</td>
    <td>0.503</td>
    <td>0.860</td>
    <td>0.752</td>
    <td>0.457</td>
    <td>0.726</td>
    <td>0.480</td>
  </tr>
  <tr>
    <td>p = 0.95</td>
    <td>0.764</td>
    <td>0.656</td>
    <td>0.573</td>
    <td>0.370</td>
    <td>0.711</td>
    <td>0.497</td>
    <td>0.848</td>
    <td>0.736</td>
    <td>0.456</td>
    <td>0.713</td>
    <td>0.476</td>
  </tr>
</table>

Table 8: MOSAIC on RAID using Llama-2-7b with four different values of nucleus sampling, on the RAID dataset. Each row corresponds to the chosen value of p computed as \( m \), the “reference model”.

<table>
  <tr>
    <th></th>
    <th>homoglyph</th>
    <th>number</th>
    <th>article deletion</th>
    <th>insert paragraphs</th>
    <th>misspelling upper/lower</th>
    <th>whitespace</th>
    <th>zero-width space</th>
    <th>synonym</th>
    <th>paraphrase</th>
    <th>alternative spelling</th>
  </tr>
  <tr>
    <td>AUROC</td>
    <td>0.961</td>
    <td>0.936</td>
    <td>0.920</td>
    <td>0.952</td>
    <td>0.948</td>
    <td>0.928</td>
    <td>0.927</td>
    <td>0.754</td>
    <td>0.681</td>
    <td>0.944</td>
    <td>0.947</td>
  </tr>
  <tr>
    <td>TPR@5%FPR</td>
    <td>0.749</td>
    <td>0.736</td>
    <td>0.693</td>
    <td>0.785</td>
    <td>0.771</td>
    <td>0.699</td>
    <td>0.707</td>
    <td>0.007</td>
    <td>0.315</td>
    <td>0.752</td>
    <td>0.771</td>
  </tr>
</table>

Table 9: MOSAIC AUROC and TPR@5%FPR for the various attacks performed on RAID, the usual Llama and Tower models were used in this scenario. For reference, MOSAIC obtains an average AUC of 0.956 over all generators without adversarial attacks.

<table>
  <tr>
    <th>Method</th>
    <th>chatgpt</th>
    <th>cohere-chat</th>
    <th>cohere</th>
    <th>gpt2</th>
    <th>gpt3</th>
    <th>gpt4</th>
    <th>llama-chat</th>
    <th>mistral-chat</th>
    <th>mistral</th>
    <th>mpt-chat</th>
    <th>mpt</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>Binoculars 0 1</td>
    <td>0.99559</td>
    <td>0.97838</td>
    <td>0.97799</td>
    <td>0.93460</td>
    <td>0.99786</td>
    <td>0.95868</td>
    <td>0.99996</td>
    <td>0.99919</td>
    <td>0.91932</td>
    <td>0.99733</td>
    <td>0.95045</td>
    <td>0.97358</td>
  </tr>
  <tr>
    <td>FastDetectGPT 0 1</td>
    <td>0.99487</td>
    <td>0.96940</td>
    <td>0.97366</td>
    <td>0.94662</td>
    <td>0.99375</td>
    <td>0.96133</td>
    <td>0.99922</td>
    <td>0.99662</td>
    <td>0.92746</td>
    <td>0.99365</td>
    <td>0.96846</td>
    <td>0.97500</td>
  </tr>
  <tr>
    <td>Binoculars 0 2</td>
    <td>0.94267</td>
    <td>0.87411</td>
    <td>0.85786</td>
    <td>0.38375</td>
    <td>0.97858</td>
    <td>0.68632</td>
    <td>0.98387</td>
    <td>0.29252</td>
    <td>0.59812</td>
    <td>0.95399</td>
    <td>0.64461</td>
    <td>0.83034</td>
  </tr>
  <tr>
    <td>FastDetectGPT 0 2</td>
    <td>0.86155</td>
    <td>0.74420</td>
    <td>0.77979</td>
    <td>0.49724</td>
    <td>0.82475</td>
    <td>0.70021</td>
    <td>0.94406</td>
    <td>0.80315</td>
    <td>0.64382</td>
    <td>0.77470</td>
    <td>0.75213</td>
    <td>0.75687</td>
  </tr>
  <tr>
    <td>Binoculars 0 3</td>
    <td>0.98818</td>
    <td>0.92769</td>
    <td>0.90545</td>
    <td>0.41349</td>
    <td>0.98959</td>
    <td>0.84535</td>
    <td>0.99703</td>
    <td>0.98274</td>
    <td>0.66670</td>
    <td>0.98825</td>
    <td>0.69790</td>
    <td>0.85558</td>
  </tr>
  <tr>
    <td>FastDetectGPT 0 3</td>
    <td>0.94627</td>
    <td>0.82562</td>
    <td>0.83573</td>
    <td>0.52014</td>
    <td>0.87708</td>
    <td>0.82838</td>
    <td>0.98374</td>
    <td>0.91869</td>
    <td>0.69212</td>
    <td>0.89539</td>
    <td>0.78809</td>
    <td>0.82830</td>
  </tr>
  <tr>
    <td>Binoculars 1 0</td>
    <td>0.70925</td>
    <td>0.79631</td>
    <td>0.73388</td>
    <td>0.49105</td>
    <td>0.93632</td>
    <td>0.35652</td>
    <td>0.88678</td>
    <td>0.83358</td>
    <td>0.66604</td>
    <td>0.82795</td>
    <td>0.71482</td>
    <td>0.72296</td>
  </tr>
  <tr>
    <td>FastDetectGPT 1 0</td>
    <td>0.61166</td>
    <td>0.61442</td>
    <td>0.64173</td>
    <td>0.59703</td>
    <td>0.68329</td>
    <td>0.37331</td>
    <td>0.82227</td>
    <td>0.65597</td>
    <td>0.68932</td>
    <td>0.55185</td>
    <td>0.80423</td>
    <td>0.60466</td>
  </tr>
  <tr>
    <td>Binoculars 1 2</td>
    <td>0.65047</td>
    <td>0.67101</td>
    <td>0.60557</td>
    <td>0.20704</td>
    <td>0.87062</td>
    <td>0.29622</td>
    <td>0.82634</td>
    <td>0.66793</td>
    <td>0.43608</td>
    <td>0.69845</td>
    <td>0.47293</td>
    <td>0.58173</td>
  </tr>
  <tr>
    <td>FastDetectGPT 1 2</td>
    <td>0.51206</td>
    <td>0.43272</td>
    <td>0.50416</td>
    <td>0.36579</td>
    <td>0.47719</td>
    <td>0.34305</td>
    <td>0.69285</td>
    <td>0.43790</td>
    <td>0.52222</td>
    <td>0.35996</td>
    <td>0.64190</td>
    <td>0.48089</td>
  </tr>
  <tr>
    <td>Binoculars 1 3</td>
    <td>0.85179</td>
    <td>0.76976</td>
    <td>0.66409</td>
    <td>0.22159</td>
    <td>0.90261</td>
    <td>0.44492</td>
    <td>0.93861</td>
    <td>0.85577</td>
    <td>0.46290</td>
    <td>0.88676</td>
    <td>0.49854</td>
    <td>0.68158</td>
  </tr>
  <tr>
    <td>FastDetectGPT 1 3</td>
    <td>0.64419</td>
    <td>0.50397</td>
    <td>0.52859</td>
    <td>0.37228</td>
    <td>0.49602</td>
    <td>0.42459</td>
    <td>0.76787</td>
    <td>0.56583</td>
    <td>0.52658</td>
    <td>0.49533</td>
    <td>0.64131</td>
    <td>0.54241</td>
  </tr>
  <tr>
    <td>Binoculars 2 0</td>
    <td>0.97183</td>
    <td>0.97627</td>
    <td>0.97972</td>
    <td>0.82349</td>
    <td>0.99761</td>
    <td>0.89893</td>
    <td>0.99819</td>
    <td>0.99345</td>
    <td>0.94065</td>
    <td>0.99375</td>
    <td>0.96866</td>
    <td>0.95841</td>
  </tr>
  <tr>
    <td>FastDetectGPT 2 0</td>
    <td>0.97335</td>
    <td>0.96636</td>
    <td>0.97593</td>
    <td>0.85763</td>
    <td>0.99305</td>
    <td>0.91412</td>
    <td>0.99851</td>
    <td>0.98910</td>
    <td>0.94400</td>
    <td>0.98469</td>
    <td>0.98023</td>
    <td>0.96154</td>
  </tr>
  <tr>
    <td>Binoculars 2 1</td>
    <td>0.99459</td>
    <td>0.98569</td>
    <td>0.98585</td>
    <td>0.91758</td>
    <td>0.99925</td>
    <td>0.96204</td>
    <td>0.99993</td>
    <td>0.99942</td>
    <td>0.95250</td>
    <td>0.99732</td>
    <td>0.97588</td>
    <td>0.97910</td>
  </tr>
  <tr>
    <td>FastDetectGPT 2 1</td>
    <td>0.99470</td>
    <td>0.97868</td>
    <td>0.98179</td>
    <td>0.93533</td>
    <td>0.99949</td>
    <td>0.96306</td>
    <td>0.99993</td>
    <td>0.99799</td>
    <td>0.95402</td>
    <td>0.99441</td>
    <td>0.98457</td>
    <td>0.97987</td>
  </tr>
  <tr>
    <td>Binoculars 2 3</td>
    <td>0.99417</td>
    <td>0.98038</td>
    <td>0.97641</td>
    <td>0.68528</td>
    <td>0.99709</td>
    <td>0.96864</td>
    <td>0.99807</td>
    <td>0.99378</td>
    <td>0.86997</td>
    <td>0.99491</td>
    <td>0.90954</td>
    <td>0.94257</td>
  </tr>
  <tr>
    <td>FastDetectGPT 2 3</td>
    <td>0.99618</td>
    <td>0.97637</td>
    <td>0.97708</td>
    <td>0.72250</td>
    <td>0.99601</td>
    <td>0.97237</td>
    <td>0.99997</td>
    <td>0.99405</td>
    <td>0.88457</td>
    <td>0.99509</td>
    <td>0.93669</td>
    <td>0.95002</td>
  </tr>
  <tr>
    <td>Binoculars 3 0</td>
    <td>0.85327</td>
    <td>0.93915</td>
    <td>0.93887</td>
    <td>0.79709</td>
    <td>0.99887</td>
    <td>0.53252</td>
    <td>0.99325</td>
    <td>0.90276</td>
    <td>0.89972</td>
    <td>0.90142</td>
    <td>0.94402</td>
    <td>0.89675</td>
  </tr>
  <tr>
    <td>FastDetectGPT 3 0</td>
    <td>0.86065</td>
    <td>0.79226</td>
    <td>0.85151</td>
    <td>0.81354</td>
    <td>0.99874</td>
    <td>0.70672</td>
    <td>0.99806</td>
    <td>0.90161</td>
    <td>0.95339</td>
    <td>0.95395</td>
    <td>0.98687</td>
    <td>0.95867</td>
  </tr>
  <tr>
    <td>Binoculars 3 1</td>
    <td>0.92261</td>
    <td>0.93290</td>
    <td>0.93741</td>
    <td>0.99345</td>
    <td>0.99859</td>
    <td>0.99881</td>
    <td>0.99299</td>
    <td>0.87181</td>
    <td>0.98556</td>
    <td>0.99709</td>
    <td>0.98109</td>
    <td>0.97079</td>
  </tr>
  <tr>
    <td>FastDetectGPT 3 1</td>
    <td>0.92617</td>
    <td>0.93678</td>
    <td>0.93578</td>
    <td>0.99038</td>
    <td>0.99802</td>
    <td>0.94262</td>
    <td>0.99792</td>
    <td>0.98310</td>
    <td>0.89995</td>
    <td>0.97919</td>
    <td>0.93025</td>
    <td>0.91648</td>
  </tr>
  <tr>
    <td>Binoculars 3 2</td>
    <td>0.84538</td>
    <td>0.90754</td>
    <td>0.91513</td>
    <td>0.64079</td>
    <td>0.99382</td>
    <td>0.54105</td>
    <td>0.89113</td>
    <td>0.89614</td>
    <td>0.76557</td>
    <td>0.95707</td>
    <td>0.83912</td>
    <td>0.84207</td>
  </tr>
  <tr>
    <td>FastDetectGPT 3 2</td>
    <td>0.85196</td>
    <td>0.90944</td>
    <td>0.91961</td>
    <td>0.64439</td>
    <td>0.99140</td>
    <td>0.53714</td>
    <td>0.98236</td>
    <td>0.89891</td>
    <td>0.78080</td>
    <td>0.93265</td>
    <td>0.85672</td>
    <td>0.84594</td>
  </tr>
</table>

Table 10: AUROC on RAID for all configurations of Falcons for Binoculars and FastDetectGPT. Configurations are indicated by the index of the models used : Falcon-7b[0], Falcon-7b-instruct[1], Falcon-40b[2], Falcon-40b-instruct[3].
<table>
  <tr>
    <th>Method</th>
    <th>chatgpt</th>
    <th>cohere-chat</th>
    <th>cohere</th>
    <th>gpt2</th>
    <th>gpt3</th>
    <th>gpt4</th>
    <th>llama-chat</th>
    <th>mistral-chat</th>
    <th>mistral</th>
    <th>mpt-chat</th>
    <th>mpt</th>
    <th>Average</th>
  </tr>
  <tr><td>Binoculars 0 1</td><td>0.98900</td><td>0.91600</td><td>0.98450</td><td>0.89000</td><td>0.99200</td><td>0.80400</td><td>1.00000</td><td>0.99600</td><td>0.62100</td><td>0.99400</td><td>0.72900</td><td>0.87600</td></tr>
  <tr><td>FastDetectGPT 0 1</td><td>0.99900</td><td>0.88800</td><td>0.89700</td><td>0.78700</td><td>0.98100</td><td>0.84400</td><td>1.00000</td><td>0.99000</td><td>0.71500</td><td>0.98400</td><td>0.85000</td><td>0.90260</td></tr>
  <tr><td>Binoculars 0 2</td><td>0.70200</td><td>0.55800</td><td>0.38800</td><td>0.00900</td><td>0.94900</td><td>0.08800</td><td>0.97900</td><td>0.62700</td><td>0.00500</td><td>0.80600</td><td>0.01600</td><td>0.46420</td></tr>
  <tr><td>FastDetectGPT 0 2</td><td>0.26200</td><td>0.12900</td><td>0.18500</td><td>0.00200</td><td>0.21400</td><td>0.03200</td><td>0.62800</td><td>0.10600</td><td></td><td>0.00200</td><td>0.01900</td><td>0.15350</td></tr>
  <tr><td>Binoculars 0 3</td><td>0.97300</td><td>0.72900</td><td>0.57400</td><td>0.01000</td><td>0.98100</td><td>0.35800</td><td>0.99900</td><td>0.95800</td><td>0.02400</td><td>0.98000</td><td>0.30000</td><td>0.60127</td></tr>
  <tr><td>FastDetectGPT 0 3</td><td>0.72500</td><td>0.31300</td><td>0.29250</td><td>0.00200</td><td>0.32900</td><td>0.21000</td><td>0.96100</td><td>0.49890</td><td>0.01900</td><td>0.42200</td><td>0.04100</td><td>0.48640</td></tr>
  <tr><td>Binoculars 1 1</td><td>0.00100</td><td>0.18000</td><td>0.14000</td><td>0.01000</td><td>0.68700</td><td>0.08000</td><td>0.54300</td><td>0.21500</td><td>0.02000</td><td>0.28400</td><td>0.03200</td><td>0.21455</td></tr>
  <tr><td>FastDetectGPT 1 1</td><td>0.04700</td><td>0.07400</td><td>0.14400</td><td>0.01400</td><td>0.15300</td><td>0.06000</td><td>0.37900</td><td>0.05400</td><td>0.05400</td><td>0.04800</td><td>0.01000</td><td>0.09055</td></tr>
  <tr><td>Binoculars 1 2</td><td>0.08400</td><td>0.03800</td><td>0.03200</td><td>0.00400</td><td>0.34500</td><td>0.05000</td><td>0.35700</td><td>0.01700</td><td></td><td>0.07900</td><td>0.00700</td><td>0.08745</td></tr>
  <tr><td>FastDetectGPT 1 2</td><td>0.00800</td><td>0.04100</td><td>0.04100</td><td>0.00000</td><td>0.10500</td><td>0.00000</td><td>0.03700</td><td></td><td></td><td>0.09000</td><td>0.01000</td><td>0.04009</td></tr>
  <tr><td>Binoculars 1 3</td><td>0.33000</td><td>0.29260</td><td>0.06700</td><td>0.01500</td><td>0.50200</td><td>0.09400</td><td>0.66700</td><td>0.23400</td><td>0.00300</td><td>0.46700</td><td>0.01200</td><td>0.23345</td></tr>
  <tr><td>FastDetectGPT 1 3</td><td>0.01600</td><td>0.00500</td><td>0.01200</td><td>0.00000</td><td>0.01400</td><td>0.00400</td><td>0.04100</td><td>0.00300</td><td>0.00100</td><td>0.00200</td><td>0.000400</td><td>0.00098</td></tr>
  <tr><td>Binoculars 2 1</td><td>0.87920</td><td>0.92800</td><td>0.94200</td><td>0.65000</td><td>0.36700</td><td>0.97800</td><td>0.98200</td><td>0.97300</td><td>0.28400</td><td>0.96400</td><td>0.18200</td><td>0.83180</td></tr>
  <tr><td>FastDetectGPT 2 1</td><td>0.89250</td><td>0.88200</td><td>0.92200</td><td>0.54400</td><td>0.49100</td><td>0.65300</td><td>0.97800</td><td>0.96100</td><td>0.29400</td><td>0.91200</td><td>0.36420</td><td>0.86182</td></tr>
  <tr><td>Binoculars 2 2</td><td>0.57900</td><td>0.95100</td><td>0.58700</td><td>0.39800</td><td>0.83900</td><td>0.18700</td><td>1.00000</td><td>0.99700</td><td>0.87000</td><td>0.78400</td><td>0.95500</td><td>0.09078</td></tr>
  <tr><td>FastDetectGPT 2 2</td><td>0.58500</td><td>0.90900</td><td>0.92800</td><td>0.71400</td><td>0.98110</td><td>0.99700</td><td>1.00000</td><td>0.99700</td><td>0.79720</td><td>0.98300</td><td>0.51800</td><td>0.39500</td></tr>
  <tr><td>Binoculars 2 3</td><td>0.39300</td><td>0.91110</td><td>0.90246</td><td>0.11000</td><td>0.99100</td><td>0.83106</td><td>0.97200</td><td>0.82406</td><td>0.79500</td><td>0.64180</td><td>0.58006</td><td>0.45166</td></tr>
  <tr><td>FastDetectGPT 2 3</td><td>0.25900</td><td>0.50900</td><td>0.65730</td><td>0.35500</td><td>0.24380</td><td>0.12640</td><td>0.60300</td><td>0.68300</td><td></td><td>0.35400</td><td>0.18509</td><td>0.40047</td></tr>
  <tr><td>Binoculars 3 1</td><td>0.71700</td><td>0.87100</td><td>0.88600</td><td>0.56500</td><td>0.96900</td><td>0.15000</td><td>0.99500</td><td>0.96900</td><td>0.35000</td><td>0.90400</td><td>0.73600</td><td>0.43896</td></tr>
  <tr><td>FastDetectGPT 3 1</td><td>0.78100</td><td>0.78200</td><td>0.77800</td><td>0.61100</td><td>0.62900</td><td>0.16800</td><td>0.93900</td><td>0.62900</td><td>0.40100</td><td>0.79100</td><td>0.52087</td><td>0.56900</td></tr>
  <tr><td>Binoculars 3 2</td><td>0.48200</td><td>0.73100</td><td>0.72800</td><td>0.18700</td><td>0.98900</td><td>0.06300</td><td>0.93300</td><td>0.66900</td><td></td><td>0.83900</td><td>0.19000</td><td>0.42120</td></tr>
  <tr><td>FastDetectGPT 3 2</td><td>0.51060</td><td>0.70700</td><td>0.73200</td><td>0.21200</td><td>0.98200</td><td>0.09400</td><td>0.93500</td><td>0.65400</td><td>0.32670</td><td>0.81540</td><td>0.53312</td><td>0.41504</td></tr>
</table>

Table 11: TPR@5%FPR on RAID for all configurations of Falcons for Binoculars and FastDetectGPT. Configurations are indicated by the index of the models used : Falcon-7b[0], Falcon-7b-instruct[1], Falcon-40b[2], Falcon-40b-instruct[3].

| Method | chatgpt | cohere-chat | cohere | gpt2 | gpt3 | gpt4 | llama-chat | mistral-chat | mistral | mpt-chat | mpt | Average |
|--------|---------|-------------|--------|-----|------|-------|------------|----------|---------|----------|-------|----------|
| Binoculars 0 1 | 0.98868 | 0.98521 | 0.97708 | 0.77221 | 0.99688 | 0.95900 | 1.00000 | 0.99325 | 0.87059 | 0.99826 | 0.89373 | **0.94848** |
| FastDetectGPT 0 1 | 0.98543 | 0.98087 | 0.97859 | 0.80717 | 0.99594 | 0.96687 | 1.00000 | 0.98990 | 0.89063 | 0.99503 | 0.87960 | **0.85639** |
| Binoculars 0 2 | 0.78407 | 0.95574 | 0.96148 | 0.38053 | 0.99353 | 0.75383 | 0.86650 | 0.93600 | 0.60380 | 0.98925 | 0.57427 | **0.60126** |
| FastDetectGPT 0 2 | 0.78205 | 0.95023 | 0.96258 | 0.82922 | 0.99761 | 0.76255 | 0.98419 | 0.92987 | 0.91102 | 0.95896 | 0.93200 | **0.91106** |
| Binoculars 0 3 | 0.82575 | 0.09920 | 0.09648 | 0.00681 | 0.36241 | 0.62674 | 0.79600 | 0.88800 | 0.76507 | 0.49346 | 0.81125 | **0.51732** |
| FastDetectGPT 0 3 | 0.77686 | 0.83947 | 0.84620 | 0.07997 | 0.99333 | 0.63320 | 0.95657 | 0.87310 | 0.28910 | 0.97504 | 0.91180 | **0.47048** |
| Binoculars 1 1 | 0.58200 | 0.79002 | 0.76754 | 0.79540 | 0.70400 | 0.44905 | 0.95748 | 0.84374 | 0.79509 | 0.74802 | 0.54966 | **0.40933** |
| FastDetectGPT 1 1 | 0.58414 | 0.80577 | 0.76716 | 0.82932 | 0.73723 | 0.79101 | 0.42975 | 0.89010 | 0.70147 | 0.68474 | 0.58285 | **0.47046** |
| Binoculars 1 2 | 0.57676 | 0.87465 | 0.75222 | 0.66870 | 0.69856 | 0.45973 | 0.96637 | 0.59000 | 0.70892 | 0.72164 | 0.64911 | **0.47353** |
| FastDetectGPT 1 2 | 0.73076 | 0.79100 | 0.66100 | 0.25700 | 0.63842 | 0.30812 | 0.78844 | 0.39220 | 0.60193 | 0.63043 | 0.36568 | **0.48191** |
| Binoculars 1 3 | 0.66875 | 0.71117 | 0.51711 | 0.49217 | 0.37549 | 0.75049 | 0.55618 | 0.66918 | 0.56669 | 0.56904 | 0.36920 | **0.42441** |
| FastDetectGPT 1 3 | 0.58035 | 0.76134 | 0.78952 | 0.03067 | 0.41174 | 0.36631 | 0.52417 | 0.56048 | 0.52071 | 0.42654 | 0.27568 | **0.32741** |
| Binoculars 2 1 | 0.50303 | 0.16207 | 0.40320 | 0.08516 | 0.93142 | 0.09349 | 0.76320 | 0.75770 | 0.45610 | 0.48612 | 0.27566 | **0.20891** |
| FastDetectGPT 2 1 | 0.54540 | 0.19580 | 0.39241 | 0.17907 | 0.95901 | 0.09452 | 0.74840 | 0.97889 | 0.49107 | 0.36063 | 0.20710 | **0.26278** |
| Binoculars 2 2 | 0.65576 | 0.90980 | 0.87836 | 0.20761 | 0.64851 | 0.12398 | 0.88240 | 0.71517 | 0.63097 | 0.56577 | 0.06113 | **0.31407** |
| FastDetectGPT 2 2 | 0.71411 | 0.95687 | 0.91555 | 0.48738 | 0.88216 | 0.05861 | 0.96595 | 0.93928 | 0.33204 | 0.31394 | 0.08110 | **0.33917** |
| Binoculars 2 3 | 0.36491 | 0.98091 | 0.84517 | 0.14017 | 0.49812 | 0.07417 | 0.87519 | 0.98703 | 0.57319 | 0.60755 | 0.07832 | **0.23841** |
| FastDetectGPT 2 3 | 0.48864 | 0.74572 | 0.83154 | 0.55905 | 0.57794 | 0.71733 | 0.95886 | 0.83207 | 0.09107 | 0.51997 | 0.45725 | **0.19496** |
| Binoculars 3 1 | 0.85392 | 0.58691 | 0.50785 | 0.04583 | 0.48258 | 0.37825 | 0.93517 | 0.49883 | 0.82746 | 0.38459 | 0.11659 | **0.29576** |
| FastDetectGPT 3 1 | 0.49232 | 0.57264 | 0.94486 | 0.78517 | 0.97188 | 0.80400 | 0.99952 | 0.57127 | 0.88110 | 0.81585 | 0.56715 | **0.34887** |
| Binoculars 3 2 | 0.36120 | 0.99500 | 0.95983 | 0.41517 | 0.86075 | 0.12583 | 0.90601 | 0.92135 | 0.19277 | 0.65955 | 0.33942 | **0.60588** |
| FastDetectGPT 3 2 | 0.95460 | 0.98141 | 0.71579 | 0.93732 | 0.93914 | 0.33986 | 0.99999 | 0.99617 | 0.90106 | 0.91930 | 0.94895 | **0.95689** |
<caption>Table 12: AUROC on RAID for all configurations of our Llamamodels for Binoculars and FastDetectGPT. Configurations are indicated by the index of the models used : llama-2-7b[0], llama-2-7b-chat[1], towerBase-7B-v1[2], TowerBase-13B-v0.[3].</caption>
<table>
  <tr>
    <th>Method</th>
    <th>chatgpt</th>
    <th>cohere-chat</th>
    <th>cohere</th>
    <th>gpt2</th>
    <th>gpt3</th>
    <th>gpt4</th>
    <th>llama-chat</th>
    <th>mistral-chat</th>
    <th>mistral</th>
    <th>mpt-chat</th>
    <th>mpt</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>Binoculars 0 1</td>
    <td>0.95300</td>
    <td>0.93700</td>
    <td>0.89800</td>
    <td>0.17300</td>
    <td>0.99300</td>
    <td>0.77200</td>
    <td>1.00000</td>
    <td>0.97200</td>
    <td>0.38900</td>
    <td>0.99500</td>
    <td>0.42500</td>
    <td><b>0.77336</b></td>
  </tr>
  <tr>
    <td>FastDetectGPT 0 1</td>
    <td>0.94900</td>
    <td>0.91600</td>
    <td>0.90900</td>
    <td>0.32400</td>
    <td>0.98300</td>
    <td>0.84300</td>
    <td>1.00000</td>
    <td>0.95800</td>
    <td>0.55400</td>
    <td>0.98300</td>
    <td>0.64800</td>
    <td><b>0.82445</b></td>
  </tr>
  <tr>
    <td>Binoculars 0 2</td>
    <td>0.46000</td>
    <td>0.85900</td>
    <td>0.87500</td>
    <td>0.31200</td>
    <td>0.99000</td>
    <td>0.28000</td>
    <td>0.94300</td>
    <td>0.78100</td>
    <td>0.55700</td>
    <td>0.88500</td>
    <td>0.69000</td>
    <td>0.69382</td>
  </tr>
  <tr>
    <td>FastDetectGPT 0 2</td>
    <td>0.46000</td>
    <td>0.83100</td>
    <td>0.87800</td>
    <td>0.44800</td>
    <td>0.99300</td>
    <td>0.35300</td>
    <td>0.94400</td>
    <td>0.76400</td>
    <td>0.65600</td>
    <td>0.82000</td>
    <td>0.80900</td>
    <td>0.71027</td>
  </tr>
  <tr>
    <td>Binoculars 0 3</td>
    <td>0.71000</td>
    <td>0.62600</td>
    <td>0.62600</td>
    <td>0.19700</td>
    <td>0.97900</td>
    <td>0.28500</td>
    <td>0.51300</td>
    <td>0.51300</td>
    <td>0.51300</td>
    <td>0.51300</td>
    <td>0.51300</td>
    <td>0.51300</td>
  </tr>
  <tr>
    <td>FastDetectGPT 0 3</td>
    <td>0.71900</td>
    <td>0.49900</td>
    <td>0.55900</td>
    <td>0.04800</td>
    <td>0.68700</td>
    <td>0.11800</td>
    <td>0.57800</td>
    <td>0.54600</td>
    <td>0.32700</td>
    <td>0.57900</td>
    <td>0.40200</td>
    <td>0.51627</td>
  </tr>
  <tr>
    <td>Binoculars 1 0</td>
    <td>0.67400</td>
    <td>0.38000</td>
    <td>0.29600</td>
    <td>0.17300</td>
    <td>0.90300</td>
    <td>0.01600</td>
    <td>0.22600</td>
    <td>0.17500</td>
    <td>0.19600</td>
    <td>0.27100</td>
    <td>0.20600</td>
    <td>0.26509</td>
  </tr>
  <tr>
    <td>FastDetectGPT 1 0</td>
    <td>0.65800</td>
    <td>0.34500</td>
    <td>0.27900</td>
    <td>0.15200</td>
    <td>0.88500</td>
    <td>0.01500</td>
    <td>0.21600</td>
    <td>0.14600</td>
    <td>0.13600</td>
    <td>0.22800</td>
    <td>0.19200</td>
    <td>0.24564</td>
  </tr>
  <tr>
    <td>Binoculars 1 2</td>
    <td>0.06600</td>
    <td>0.30200</td>
    <td>0.29200</td>
    <td>0.17400</td>
    <td>0.84700</td>
    <td>0.02400</td>
    <td>0.15200</td>
    <td>0.16900</td>
    <td>0.19000</td>
    <td>0.18100</td>
    <td>0.21500</td>
    <td>0.23745</td>
  </tr>
  <tr>
    <td>FastDetectGPT 1 2</td>
    <td>0.06400</td>
    <td>0.28700</td>
    <td>0.28800</td>
    <td>0.16500</td>
    <td>0.83700</td>
    <td>0.02100</td>
    <td>0.16000</td>
    <td>0.16000</td>
    <td>0.20000</td>
    <td>0.16300</td>
    <td>0.21900</td>
    <td>0.23309</td>
  </tr>
  <tr>
    <td>Binoculars 1 3</td>
    <td>0.01300</td>
    <td>0.11900</td>
    <td>0.13500</td>
    <td>0.01300</td>
    <td>0.71800</td>
    <td>0.00700</td>
    <td>0.05500</td>
    <td>0.02000</td>
    <td>0.02600</td>
    <td>0.05400</td>
    <td>0.03300</td>
    <td>0.08181</td>
  </tr>
  <tr>
    <td>FastDetectGPT 1 3</td>
    <td>0.01200</td>
    <td>0.08000</td>
    <td>0.12500</td>
    <td>0.01800</td>
    <td>0.49900</td>
    <td>0.00600</td>
    <td>0.05900</td>
    <td>0.01300</td>
    <td>0.03500</td>
    <td>0.01900</td>
    <td>0.06800</td>
    <td>0.08473</td>
  </tr>
  <tr>
    <td>Binoculars 2 0</td>
    <td>0.76200</td>
    <td>0.71000</td>
    <td>0.54000</td>
    <td>0.03100</td>
    <td>0.99000</td>
    <td>0.23200</td>
    <td>0.99000</td>
    <td>0.89700</td>
    <td>0.08200</td>
    <td>0.96300</td>
    <td>0.06200</td>
    <td>0.56900</td>
  </tr>
  <tr>
    <td>FastDetectGPT 2 0</td>
    <td>0.74800</td>
    <td>0.66300</td>
    <td>0.66400</td>
    <td>0.11700</td>
    <td>0.89900</td>
    <td>0.36900</td>
    <td>0.98200</td>
    <td>0.84400</td>
    <td>0.28500</td>
    <td>0.83200</td>
    <td>0.36000</td>
    <td>0.61482</td>
  </tr>
  <tr>
    <td>Binoculars 2 1</td>
    <td>0.98900</td>
    <td>0.85400</td>
    <td>0.76300</td>
    <td>0.03200</td>
    <td>0.99300</td>
    <td>0.71100</td>
    <td>1.00000</td>
    <td>0.98600</td>
    <td>0.12800</td>
    <td>0.99700</td>
    <td>0.08600</td>
    <td>0.68536</td>
  </tr>
  <tr>
    <td>FastDetectGPT 2 1</td>
    <td>0.97100</td>
    <td>0.79600</td>
    <td>0.77500</td>
    <td>0.11400</td>
    <td>0.93900</td>
    <td>0.81400</td>
    <td>1.00000</td>
    <td>0.96400</td>
    <td>0.25500</td>
    <td>0.96300</td>
    <td>0.33900</td>
    <td>0.72091</td>
  </tr>
  <tr>
    <td>Binoculars 2 3</td>
    <td>0.69700</td>
    <td>0.63100</td>
    <td>0.45000</td>
    <td>0.01400</td>
    <td>0.97500</td>
    <td>0.17000</td>
    <td>0.98600</td>
    <td>0.76700</td>
    <td>0.04000</td>
    <td>0.91400</td>
    <td>0.02300</td>
    <td>0.51518</td>
  </tr>
  <tr>
    <td>FastDetectGPT 2 3</td>
    <td>0.50400</td>
    <td>0.43000</td>
    <td>0.43400</td>
    <td>0.01500</td>
    <td>0.58300</td>
    <td>0.17100</td>
    <td>0.94100</td>
    <td>0.50800</td>
    <td>0.11900</td>
    <td>0.48200</td>
    <td>0.16600</td>
    <td>0.39573</td>
  </tr>
  <tr>
    <td>Binoculars 3 0</td>
    <td>0.80100</td>
    <td>0.81800</td>
    <td>0.81400</td>
    <td>0.11700</td>
    <td>0.99700</td>
    <td>0.32500</td>
    <td>0.99300</td>
    <td>0.92500</td>
    <td>0.35600</td>
    <td>0.97700</td>
    <td>0.44000</td>
    <td>0.68755</td>
  </tr>
  <tr>
    <td>FastDetectGPT 3 0</td>
    <td>0.75500</td>
    <td>0.74300</td>
    <td>0.80400</td>
    <td>0.28300</td>
    <td>0.96600</td>
    <td>0.45100</td>
    <td>0.97900</td>
    <td>0.85300</td>
    <td>0.57000</td>
    <td>0.85400</td>
    <td>0.76600</td>
    <td>0.72945</td>
  </tr>
  <tr>
    <td>Binoculars 3 1</td>
    <td>0.98800</td>
    <td>0.92500</td>
    <td>0.90100</td>
    <td>0.11000</td>
    <td>0.99900</td>
    <td>0.83900</td>
    <td>1.00000</td>
    <td>0.99300</td>
    <td>0.32000</td>
    <td>0.99400</td>
    <td>0.37100</td>
    <td>0.76727</td>
  </tr>
  <tr>
    <td>FastDetectGPT 3 1</td>
    <td>0.97700</td>
    <td>0.85900</td>
    <td>0.85400</td>
    <td>0.27200</td>
    <td>0.98100</td>
    <td>0.88200</td>
    <td>1.00000</td>
    <td>0.96900</td>
    <td>0.54000</td>
    <td>0.97200</td>
    <td>0.70000</td>
    <td>0.82145</td>
  </tr>
  <tr>
    <td>Binoculars 3 2</td>
    <td>0.70000</td>
    <td>0.85700</td>
    <td>0.86600</td>
    <td>0.26900</td>
    <td>0.99400</td>
    <td>0.35600</td>
    <td>0.98100</td>
    <td>0.91200</td>
    <td>0.54500</td>
    <td>0.96400</td>
    <td>0.69700</td>
    <td>0.74009</td>
  </tr>
  <tr>
    <td>FastDetectGPT 3 2</td>
    <td>0.69200</td>
    <td>0.79100</td>
    <td>0.83400</td>
    <td>0.41200</td>
    <td>0.96900</td>
    <td>0.43800</td>
    <td>0.97400</td>
    <td>0.86700</td>
    <td>0.64900</td>
    <td>0.85300</td>
    <td>0.81300</td>
    <td>0.75564</td>
  </tr>
</table>

Table 13: TPR@5%FPR on RAID for all configurations of our Llamamodels for Binoculars and FastDetectGPT. Configurations are indicated by the index of the models used : Llama-2-7b[0], Llama-2-7b-chat[1], TowerBase-7B-v0.1[2], TowerBase-13B-v0.1[3].

Reviewer paper summaries: This paper focuses on the generator-agnostic artificial text detection, where the models to be detected are not predefined. This paper proposes a new ensemble method using fundamental information-theoretic principles to identify MGTs.

This paper describes a method for discriminating between human and AI generated text using an ensemble of LLMs and a novel scoring method which the authors call MOSAIC.

The setup comprises an ensemble of detector LLMs whose logits are passed to a mixture model which dynamically ranks the LLMs (best choice for discriminating each successive string in the sequence of text) and modifies the LLMs weightings accordingly. This ultimately produces a MOSAIC score indicating if the text is human generated or LLM generated based on a heuristic-defined threshold score.
The authors point out this approach is similar to Binoculars scores, but MOSAIC computes the average surprisal using a position-dependent mixture of models (versus a single fixed model for Binoculars).

Experiments are conducted using various combinations of detector ensembles, on various datasets (RAID, M4, and a modified RAID which the authors refer to as RAID+) covering several languages and domains.
Summary of strengths: - This paper proposes a valuable insight -- using multiple LLMs to detect MGTs.

The method described, MOSAIC, modifies an existing approach in a way that could be promising by being, as the authors state, generator-agnostic. However, I am not convinced this has been adequately demonstrated in the paper.
Experiments were conducted using existing datasets that cover multiple domains, and in one dataset - multiple languages, and are compared in some experiments to existing approaches (Binoculars, FastDetectGPT).
Summary of weaknesses: - The proposed ensemble strategy seems to require the vocabulary of each sub-model to be the same, which limits the application. If I misunderstood, thanks for explaining it in the response phase, and I'll consider updating my score.
- I question the fairness of the comparison in Figure 2. 
   - Why do the authors compare the proposed model with the average score? I think it is not fair, since some LLM pairs might perform poorly on most datasets. It leads that these pairs will likely be filtered out.
   - I wonder the time cost of the proposed model. In my opinion,  the time cost of the proposed model and enumeration pairs is equivalent, since the time is mainly consumed in obtaining the token distributions of each model. Maybe I misunderstood, thank you for giving the time cost in the response phase, and I'll consider updating my score.
It leads to unclear motivations and advantages of the proposed model.
- Actually, I think the proposed model does not solve the brittleness (line 017). The authors just turn a fixed pair into a fixed candidate set (4 models). Meanwhile, the performance and time cost are equivalent.

§6.1 - Fig.2 suggests the MOSAIC ensemble mostly underperforms compared to  Binoculars (Oracle) and FastDetectGPT (Oracle), on the RAID dataset. In the text (393) it seems you suggest that the Oracle combination is not suited to a generator-agnostic approach, that aside, how do you respond to the criticism that MOSAIC mostly underperforms compared to Binoculars and FastDetectGPT?
Is the argument that MOSAIC, although it may be slightly poorer performing, should perform better against as yet unseen generators? If you tested against unseen generators (whether LLMs or PLMs) can you clearly point out where you did so and what the results indicate?
This experiment is, to my mind, the raison-d’etre of MOSAIC, but the Figure gives a poor impression of the efficacy of the detector and thereby the usefulness of your approach; so if you can allay the concerns above it would be most helpful.

§6.1.1 - This experiment does not definitively show that adding more models makes the ensemble perform better - it could be just that the models you added were better than the existing models in the ensemble. An ablation experiment may be useful here, or at least adding TowerBase-13B as the third model, and TowerBase-7B as the fourth model and comparing those results with the results in Table 4.

The ensemble of LLMs is limited because all LLMs must all share the same vocabulary and tokenizer. In fairness, the authors mention this in the paper, and in the Limitations section:
(248 - detectors in the ensemble must share the same vocabulary and tokenizer)
(384 - ensembles only within the same family i.e. Falcon, or Llama&Tower)
(556 - Limitations)
Comments, suggestions and typos: See the weakness above. More:
- Some references of the tables are unclear. For example, Table 3 p.15 and Table 7 p.19 (line 227). Using \ref and clearly defining the p.x could improve the readability.
- Some punctuation marks are not used consistently, such as lines 393 and 395.

423-426 Specify whether you are referring to the AUROC or TPR@5%FPR portion of Table 4.
Whichever you are referring to, the results you mention in this paragraph (423-426) do not seem to match either set of results in Table 4. You say that adding TowerBase-13B does not improve results in “mistral-chat” and “mpt-chat”, however the Table indicates the MOSAIC-4 (which includes TowerBase-13B) improves results for “mistral-chat” from 0.970 to 0.988, and for “mpt-chat” from 0.990 to 0.994 AUROC. Please explain if I have misunderstood?

036 “As it seems, ...” - this sentence is difficult to parse.
263 “chose” -> “choose”
296 “wights” -> “weights”
429 - “in the” sentence probably reads better with these two words removed.
451 and 466: “section” -> “Section”
472: “by” - presumably this is a typo
483: “satisfy” -> “satisfies”
488: “lead” -> “leads”

Releasing RAID+ would be useful for reproducibility.
Overall assessment scores: 3

3