DMAP: A Distribution Map for Text
Authors: Tom Kempton
Julia Rozanova
Parameswaran Kamalaruban
Maeve Madigan
Karolina Wresilo
Yoann Launay
David Sutton
Stuart Burrell
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Entropy
Statistical Text Analysis
Post-training
Supervised Fine-tuning
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce DMAP, a tool for analyzing statistical properties of text and large language models.

Abstract: Large Language Models (LLMs) are a powerful tool for statistical text analysis, with derived sequences of next-token probability distributions offering a wealth of information. Extracting this signal typically relies on metrics such as perplexity, which do not adequately account for context; how one should interpret a given next-token probability is dependent on the number of reasonable choices encoded by the shape of the conditional distribution. In this work, we present DMAP, a mathematically grounded method that maps a text, via a language model, to a set of samples in the unit interval that jointly encode rank and probability information. This representation enables efficient, model-agnostic analysis and supports a range of applications. We illustrate its utility through three case studies: (i) validation of generation parameters to ensure data integrity, (ii) examining the role of probability curvature in machine-generated text detection, and (iii) a forensic analysis revealing statistical fingerprints left in downstream models that have been subject to post-training on synthetic data. Our results demonstrate that DMAP offers a unified statistical view of text that is simple to compute on consumer hardware, widely applicable, and provides a foundation for further research into text analysis with LLMs.

DMAP: A DISTRIBUTION MAP FOR TEXT
Tom Kempton
University of Manchester, UK
thomas. Cemptoot@manceeterr.co
Julia Rozanova, Parameswaran Kamalaruban, Maeve Madigan Risk and Security AI Lab, Visa Inc., UK (yrozanov, kaparar rapa, mmadi mandiigan] @vi Sovisa... commom
Karolina Wresilo, Yoann Launay" University of Cambridge, UK
kwresiloChep. phy...pppp..ppp..... cam.ac..... uk, y18448cam ac uk
David Sutton & Stuart Burrell Risk and Security AI Lab, Visa Inc., UK {dsutton, s com
ABSTRACT
Large Language Models (LLMs) powerful tool for statistical text analysis, with derived sequences of next-token probability distributions offering wealth information. Extracting this signal typically relies on metrics such as perplexity, which do not adequately account for context; how one should interpret a given next-token probability dependent on the number of reasonable choices encoded by the shape of the conditional distribution. In this work, we present DMAP, a mathematically grounded method that maps a text, via a language model, to set of samples in the unit interval that jointly encode rank and probability informa- tion. This representation enables efficient, model-agnostic analysis and supports range of applications. We illustrate its utility through three studies: (i) val- idation of generation parameters to ensure data integrity, (ii) examining the role of probability curvature in machine-generated text detection, and (iii) forensic analysis revealing statistical fingerprints left in downstream models that have been subject to post-training on synthetic data. Our results demonstrate that DMAP of- fers a unified statistical view of text that is simple to compute on consumer hard- ware, widely applicable, and provides foundation for further research into text analysis with LLMs.
INTRODUCTION
A language model provides wealth of information about text (wi ... w.y through the sequence next-token probability distributions p(-1w, ... -.-1). We ask how to use this informa- tion to learn something about the text or the language model p. To date, most efforts that use language model to report statistical properties of a text use metrics that measure how unexpected each token is under p. A standard such metric is the average log-likelihood of each observed token,
T Σ log fogp(ws/w: ... w.;
1
while a related but coarser variant, log-rank, replaces p( r(w; |W ... w;-1) of the observed token in the descending list of next token probabilities. Perplexity is the exponential of the negative average pok-token log probability.
wi ....... ... w;-1) -.,-.,1))
with the ordinal rank
Work completed while Featurespace, Visa Solution.

---

Sorted distribution
Fininb) 0.
Map observed next token to interval
LLM
Poxomie)) uB
Pipuclyy)
Extract prefix of length
The robot danced the
0.0
1.0
token
Generarate DMAP Sample 2y ~ U/0. 0.7)
While < len(text) do
i+1
Output sumple
text
Visualizations
Statistical Tests
The robot danced the snow while the birds watched from the treetops...
x²
T
The DMAP algorithm. Given text w of length T, this diagram illustrates how DMAP generates a collection samples Zi .... 2T in [0, 1]. These may be analyzed quantitatively with statistical tests qualitatively by splitting [0, 1] into equal sized bins and plotting histogram, as illustrated in Figure 2. We initialize = Our experiments demonstrate these visualizations identify decoding parameters (top-p, top-k, temperature), yield insights into black box machine- generated text detection algorithms based on probability curvature, and reveal statistical fingerprints left by performing supervised fine-tuning (SFT) synthetic data.
Figure 1:
Log-likelihood. log-rank and perplexity have been widely used in the training, evaluation, and de- tection of language models. For example, body of work seeks to use perplexity to predict the readability texts (Trott Rivière, 2024), and to use the strength of the correlation between per- plexity and human reading time measure of the quality of language model (Oh & Schuler, 2023). However, in some settings these metrics are problematic (Meister & Cotterell, 2021; Fang al., 2024), and often require contextualization be useful.
In this work we use contextualization to describe the process of interpreting a raw statistic of a text (such per-token log-likelihood) in terms of the content of the text. At the broadest level, one can give better answers to questions such as 'is per-token fog-likelihood score of 4 unusually high?' one knows whether the text under consideration is factual essay about chemistry or piece of creative writing. A finer-grained approach is to compare the log-likelihood of token present in text the expected log- ligeeiill of alternative tokens randomly sampled from language model. The crux of the contextualization problem is that the way one should interpret token being the third most likely, or having model probability 0.1, depends on the number of reasonable token choices. These differences, encoded by the shape of the conditional distributions, are prone to persist over long passages of text.
Recently, a body of work initiated by DetectGPT (Mitchell et al., 2023) has tried to language model p to address the contextualization problem and extract more nuanced information from next token probability distributions in the context of machine generated machine-generated text detection (Bao et al., 2023; Su et al., 2023; Hans et al., 2024). Inspired by these ideas, we introduce a distribution map. DMAP, which is statistically rigorous and lends itself better to visualization. In essence, DMAP is recipe for mapping a text w onto a density function f [0, 1] > R+ that encodes information about both rank and perplexity in principled way. As shall see, this addresses the contextualization problem and provides a simple yet surprisingly powerful mathematical lens through which to measure and compare texts.
Contributions
1. Introduce the DMAP Algorithm. We introduce DMAP, a simple method to represent a text we through language model p as set of samples in [0, 1] that jointly encode rank
2

---

and probability information. DMAP is open-sourced', computationally efficient, and can be applied effectively on consumer hardware with small models such as OPT- (Zhang et al., 2022).
2. Demonstrate Applications of DMAP.
(a) Validate generation parameters. Detecting incorrect or inconsistent generation set- tings in published data, such as top-k. top-p, temperature, or the language model itself. (b) Design machine-generated text detectors. Using DMAP, we identify design weak- nesses of existing zero-shot detectors based on probability curvature and propose al- ternative principles for detector design.
(c) Reveal statistical signatures of post-training data in instruction models. We investigate overconfidence instruction-tunet models using DMAP and reveal alignment between downstream generated text distributions and fine-tuning datasets, even without access to internal model probabilities for the generating model.
We expect that the strengths of DMAP in statistical rigor and convenient visualization will it find many applications beyond these.
2 RELATED WORK
The Probability Integral Transform. The probability integral transform (PIT, Fisher 1928) was introduced method for mapping continuous probability distribution on the real line onto the uniform distribution on [0, I]. and as such is similar in philosophy to DMAP. One could combine PIT with the distributional transform to deal with non-continuous random variables, this corresponds exactly to our step of sampling from the uniform distribution on 4, However, it only makes sense to use PIT with categorical variables if one has natural ordering of the categorical variables, and if the bias which one is hoping to detect presents itself in terms of this ordering. While we have framed our work as turning log-rank into a statistically useful measure, one could instead think of DMAP as applying PIT and the distributional transform to the case of texts generated by language models, but with a crucial additional step of dynamic re-ordering tokens by model probability.
Text Visualization. GLTR (Gehrmann al., 2019) introduced a tool color-code text according to token rank. This is an effective visualization, but inherits the issues of rank as a crude measure of where a token sits in the probability distribution. Our DMAP framework extends this visualization paradigm by providing a continuous, mathematically grounded representation that preserves both rank and probability information while enabling statistical inference.
Text Detection. Several key approaches have emerged to address the so- cially pertinent problem of detecting machine-generater text. DetectGPT (Mitchell et al., 2023) pioneered the of probability curvature, measuring how perturbing a text affects its likelihood under a language model. This approach was refined by DetectLLM (Su et al., 2023), who adapted DetectGPT to use rank information, rather than exact probabilities. FastDetectGPPT (Bao al., 2023) addressed efficiency issues related to perturbation and the contextualization problem by normaliz- ing probability distributions at each step. More recently, Binoculars (Hans et al., 2024) proposed a promising cross-model approach using probability ratios, though the theoretical justification for their normalization scheme remains unclear. Unlike these approaches, DMAP does not directly address the text detection problem. Instead, it provides unified statistical framework that maps probability information standardized representation, enabling broader text analysis applications such as informing the design of future detectors.
Model Calibration and Overconfidence. Recent work has identified systematic overconfidence in instruction-tunee language models. Luo al. (2025) demonstrated that alignment procedures can degrade calibration, while Shen et al. (2024) proposed temperature scaling methods for post- hoc calibration. Chhikara (2025) investigated the relationship between alignment objectives and confidence estimation, and Zhu et al. (2023) analyzed calibration across different model scales. Yang& & Holtzman (2025) explored how alignment training affects uncertainty quantification. DMAP
Ihttps:// com com/Peaturespacel ddaa.
3

---

contributes to this literature by providing tool to visualize and quantify distributional changes induced by post-training procedures, revealing statistical fingerprints that persist in downstream model behavior.
The key distinction of DMAP is its ability to map arbitrary probability distributions to a standardized unit interval representation, enabling both intuitive visualization and rigorous statistical analysis while addressing the fundamental contextualization challenges that limit existing approaches based on surprisal metrics.
3 DMAP: A DISTRIBUTION MAP FOR TEXT
3.1 DEFINING DMAP
Let p be language model, which we call the evaluator model, and w W1 ... WT be text with tokens belonging to vocabulary V. For each sequence position i {1, T). the candidate tokens E can be ordered by decreasing probability p(u w)...w;--1). We construct a sequence of points I1 ET [0, 1J² referred to as DMAP sample as follows, see Figure I for an overview.
Given a text w1 .....uu. WT and an index E {1, T}. DMAP works by first defining an inter- val 4 and then sampling point 2, from the uniform distribution the language model p to be the most likely token to follow W1 ... W;-1. the interval 4 will be [0, p( |w1 ... w;-1)]. 1]] More generally, I; is the interval of length p(w; W/ w;-1) 1) whose left end point a; is the total mass of the set of tokens judged more likely by the language model.
4. If w, judged by
Formally, let
y+== (u E V p(u|w) ... w)-1) > p(wa wwy ... ww-1))
be the set of tokens judged by to be more likely than w, to appear next in the sequence. Now, define two points a, and b, by
a1 =
Σ VE
p(o w ... ....,..,,,. w;-1);
and 6; :- a; + p[w:/w] I:
Finally, define the interval C [0, 1] by I, = la, by] and, for each i, let x; = [wrl [w] ... w) be chosen by sampling from the uniform distribution U( b.) on 4, yielding the desired sequence
T1 ... ZT. practice, to visualize the resulting set of samples 2, we split the unit interval [o, 1] into k equallyy-szze bins and plot the resulting histogram. For the plots in this article use k 40. Other quantitative and qualitative measures of the set {xi}s may be informative and we leave this for future work.
In the following proposition, pure sampling from language model p refers to the practice of autore- gressively generating sequence wi ... wr by, each step selecting token w: with probability p(wz/wn-. ... w.-1). We later also consider top-k, nucleus and temperature sampling, see Section B for definitions.
Proposition 3.1. When generating a text Wh ... .r by pure sampling from language model p. the
corresponding sequence II...IT VT obtained by applying DMAP to wi -o....00 wT with evaluator model p will be independent and identically distributed (i.id.) according to the uniform measure on [0, 1].
Proof. See Appendix A.
This proposition is particularly useful in Section 5.1, since the i.i.d. structure of the sequence J1 Er allows one to use off the shelf results about convergence rates. particular, it allows one to compute the chi-squared statistic of the distribution of the set of points (I1,-.. , uncov- cring errors in the process of text generation with a high degree of confidence.
We can also use DMAP to visualize texts using an evaluator language model modified by de- coding strategy such as top-k, top-p or temperature sampling. For example, to see how W1 ... Wr looks to language model ities (( ... -.;-1) with the probabilities q( [olw,- ----;1) resulting from sampling from p with temperature т. Proposition 3.1 continues to apply: see Appendix A.
temperature т, one simply needs replace the next token probabil-
4

---

3.2 DEFINING ENTROPY- WEIGHTED DMAP
There are two practical issues with the definition of DMAP in Section 3.1. First, there is randomness in the selection of xi from that introduces noise and makes DMAP non-deterministic. Second, if the language model p has high certainty about the choice of next token, then a choice of x; contains little useful information and we should assign less weight this choice of zς. We present an alternative version of DMAP that mitigates these shortcomings by removing this randomness and weighting the outcome of each token by the entropy of the next token probability distribution. A version of Proposition 3.1 continues to hold, see Appendix A. In Appendix F we make DMAP plots restricted to times where the next-token probability distribution has low entropy, and see that these plots contain little useful information, justifying our deweighting of these points.
In broad terms, our map D defined below makes two changes DMAP. Firstly, rather than weight- ing each sample equally, we use the entropy of the next-token probability distribution to weight each sample point, up to a chosen maximum clipped weighting of X for stability. Secondly, we mitigate the randomness in the selection of z; and, instead of considering the (weighted) proportion of sam- ples in each bin, plot the expectation of this, removing the randomness. A precise mathematical description goes as follows.
Given language model p and text W1 U1...0 wr, generate the sequence of intervals ..... IT with DMAP. Also, compute h1 ... ht where
h4 = Enleler ... log
w;-1)
VEV
is the entropy of the probability distribution p(-lw) ...--i..)))))) Let 14 == max(hi, A} and return the
function D ((w)) [0, > IR given by
1== M X m
D(w) D(w) wr.p)
h24
where XI, is the characteristic function equal to 1 T; and O elsewhere, and [7g| is the length of interval 7. This gives that XI,/(T,l [ is the function of integral 1 taking value 0 outside of 7, and constant value on I, Thus, D is step function of integral one supported on [0. 1]. Finally, we may make an weighty DMAP plot by splitting the unit interval into, for example, and averaging the value of Đ over each bin. Since the entropy potentially very large, in practice recommend clipping it at À = 2 for stability.
40 bins
The entropy h; can viewed as the expected information obtained by revealing the next token wi- Therefore, entropy entropy-weighting places more weight on the times when the choice of next token is more uncertain, and so we have more to learn. This amplifies the differences between our output distribution and the uniform distribution at [0, 1]. by reducing the weight associated with times i where the next token is known with extremely high probability, making DMAP a more sensitive entropy-weighter DMAP unless otherwise specified. See Appendix F for
tool. Hereafter, more information on entropy-weighting
3.3 INTERPRETING DMAP VISUALIZATIONS
DMAP samples may be analyzed quantitatively, as in the x2 tests of Section 5.1, or qualitatively via simple histograms. These visualizations reveal whether certain portions of the model next-token probability distribution are systematically over- or underrepresented in the text. Three particularly common behaviors that observe are: Head Bias (e.g. Figure 2 (c)) tokens viewed as likely by the evaluator model are over-represented in the text. Tail Bias (e.g. Figure 2 (f)) tokens viewed as unlikely by the evaluator model are over-represented in the text. Text generated by one base model and evaluated by another base model with similar entropy will typically display this behavior. Tail Collapse (e.g. Figure 2 (e)) small portion the bottom of the evaluator model distribution is strongly under-represented in the text. Often seen in human- written text, this is consistent with the folklore intuition that model distributions place too much weight on tokens which are realistic, an oft-cited motivation for top-p (nucleus) sampling.
5

---

Donsity
0.5
1.0
008
1.0
1.3
50
[h] 1.0
(6)
(0
(g)
DMAP Samples
Figure 2: Illustrative DMAP histograms. The first row shows plots of XSum data (Narayan et al., 2018) generated by OPT-125m, evaluated by OPT-125m. The generation strategies (left to right) are (a) pure sampling, (b) top-p 0.8 sampling, (c) temperature = 0.8 sampling, and (d) top- = 50. The second row shows various different types of text evaluated by DMAP: (c) a news dataset of human text from RAID, (f) text generated by Mistral 7B (Albert Q. Jiang and others, 2023) using pure sampling (top-p=1). (g) text generated by Mistral 7B Instruct, and (h) text generated by ChatGPT from the Ghostbusters dataset (Verma et al., 2024). OTP- VTP-122m was used the scoring model to generate DMAP samples. See Appendix N for examples with larger evaluation models.
4 ILLUSTRATIVE VISUALIZATIONS FROM DMAP
This section takes a first look at some histogram plots from D. We take text from different sources, both human and machine, and evaluate them using OPT-125m as recommended by Mireshghallah et al. (2024). This evaluator demonstrates DMAP can be run effectively on consumer hardware in few minutes. Throughout, we run DMAP over 300 texts each of around 300 tokens.
The top row of Figure 2 considers the case where the language model used to generate the texts is also used generate the DMAP plot. We consider pure sampling (a), where token wy is selected time according to its model likelihood p(w:/wr [0q ... w;-1), along with top-p (nucleus) (b), tempera- ture (c), and top-k (d) sampling. See Appendix B for an explanation of these decoding strategies. As predicted in Proposition 3.1, the DMAP plot from pure-sampled text with the same generator and evaluator language models is close to the uniform distribution. Temperature, top-p and top-k sam- pling are methods for increasing the probability of sampling from the head of the model distribution, corresponding to the left side DMAP histogram. They each produce head-biased DMAP plots with highly characteristic shape. In particular, the plot for top-p sampling is flat on [0, before rapidly dropping off, and the plot for top-k sampling is flat on roughly the first half the interval before smoothly dropping off. These shapes be explained in terms of the statistics of the space of conditional probability distributions; see Appendix B.
In the bottom row of Figure things can be seen in the distribution of human-written text (e). The distribution generally shows that human-writter tokens are somewhat surprising OPT-125m. However, there is a sharp drop off on the very right hand side of the distribution, reflecting the fact that the bottom five percent of the OPT- 125m distribution places too much weight on tokens which are not representative of human writing. This drop-off is much less pronounced when using more modern language models as detectors. Several other phenomena are easy to observe. Text generated by Mistral 7B base model) and evaluated by 12Tm125m (f) is tail-biased. This means, average, Mistral 7B generated text is surprising to 125-125m. This phenomena should be expected when evaluating base models, Section C, and is repeated when Mistral (Albert Q. Jiang and others, 2023), Falcon (Ebtesam
produce DMAP plots for generated in various ways. Two

---

(a)
(b)
(C)
1.0
1.0
0.8
0.8
Density
0.6
0.6
0.4
0.4
-8
0.2
0.2
White box
-10
Black box
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
2000
4000
6000
8000
10000
DMAP Samples
Figure 3: Quantitative validation of decoding parameters. (a) shows DMAP plot for the black-box case of Llama 3.1 8B generated text evaluated by Mistral 7B. (b) shows DMAP plot for the white- box case of Llama 3.1 8B generated text evaluated by Llama 3.1 8B. (c) plots the log10 p-values resulting from our x2 uniformity test. This demonstrates how quantitative evidence can be extracted from DMAP samples investigate hypothesis about possible generated strategies. For example, (c) tells us that after evaluating 10000 tokens of Llama generated text with Mistral-7B as the evaluation model, the probability that Mistral would produce text with such an extreme x2 distribution less than 10-10 We can conclude that it is not plausible that the text under review was generated by pure sampling from Mistral 7B.
Almazrouei and others, 2023) and Ilama 3.1 8B (Grattafiori al., 2024) look the text generated by one another, see Figures 8 and 9. In contrast to text generated by base models, text generated by instruction-tune
models Mistral-Instruct (g) and ChatGPT (h) is head-biased; on average these models much more likely to pick tokens that unsurprising OPT. One might speculate that something in the post-training regime has profoundly altered the language model distribution. We study this question in more detail in Section 5.3.
5
APPLICATIONS
This section contains three example applications of DMAP, though we anticipate further use cases.
5.1 VALIDATING GENERATION PARAMETERS
Research claims in natural language processing are often sensitive to the precise decoding param- eters used to generate text, see Section 5.2. Therefore, when generating text, or using publicly available samples, it is crucial to able to validate the reported language model and generation hyperparameters such top-p, top-k, temperature.
For example, using DMAP, we discovered a major data-error affecting most of the top papers in zero-shot machine-generatee text detection (Mitchell et al., 2023; Su al., 2023; Bao et al., 2023; Hans et al., 2024; Dugan et al., 2024). At the time of their writing, HuggingFace enabled top-k by default with & 50, making very easy for researchers innocently and accidentally leave top-k enabled while reporting that experiments were run on texts generated with pure sampling. Further works such as Dubois al. (2025) use texts generated by other papers where the was present. In this section, we present how DMAP may be used as qualitative or quantitative tool to help researchers validate their data to prevent such occurrences.
Qualitative Validation. DMAP is an easy method for visually checking the integrity of machine- generated text for open-weight models. When text is claimed be generated by pure sampling. Proposition 3.1 proves that the DMAP samples should be approximately uniformly distributed. Text generated by a given decoding strategy clearly identified from the DMAP plots, see Figure 2. Alternatively, to test specific combination of decoding parameters, could test whether uniform distribution recovered when modifying DMAP to use the probability measure q arising from sampling from p with the specified decoding strategy.

---

Quantitative Validation. To go beyond visual check, have the following method for deter- mining the consistency of the data with a reported generation method. Firstly, DMAP applied to generate points 21 2r, adapted for the parameter settings, for example, by multiplying logits by 1/r for temperature sampling at temperature r. Next, the unit interval is divided into & equal-sized bins, and frequencies f1 computed, where
# ( E (1,.... T} - æj
f:
T
We follow the Terrell-Scott rule (Terrell & Scott, 1985) for choosing the number of bins, letting
k = -27)) and compute the x2 statistic
k Tk
1
1=1 1:
We then compute the probability that text generated by the reported generation method would have such an extreme X2 statistic. This p-value determines the consistency of the text with the reported generation method, see Figure 3 and Appendix D for further details, and Goodman (2008) for guid- ance on the interpretation of p-values for this test.
5.2 DESIGNING METHODS TO DETECT MACHINE-GENERATIRR TEXT
This section describes how to use DMAP to gain insight into differences between human- and machine-generater text, and what conclusions can be drawn on the performance of current detec- tors and the design of future detectors. This is pertinent since, discussed in Section 5.1, there are data errors in substantial portion of the research on the detection of machine-generated text. Consequently, we discover that the principles underlying the design of many Al detectors are not universally true, the primary being the probability curvature thesis. Other observations for the interested reader are provided in Appendix L.
5.2.1 PROBABILITY CURVATURE FOR BASE AND INSTRUCTION-TUNED MODELS
The main idea behind most statistical approaches to detecting machine-generated text is that humans tend to choose words from further down the probability distribution than machines do. This idea was presented as probability curvature in DetectGPT (Mitchell et al., 2023), and the approach was broadly followed in DetectLLM (Su et al., 2023), Fast-DetecIGPT (Bao et al., 2023), and Binoculars (Hans et al., 2024).
Probability curvature is effective in detecting text generated by an instruction cittucttion-uueed model, or sampling strategy that weights text generation towards the head of the distribution. However, the probability curvature idea is not supported by DMAP plots when pure sampling is used; in the white box case, Figure (a) and (e), is much weaker than previously reported and in the black box case, Figure 23, it appears false. To further validate this claim, rerun experiments efficacy of DetectGPT, Fast-DetecIGPT and Binoculars in the black box detection setting. When top-k sampling is used the detectors remain effective as previously reported, since this decoding strategy impacts probability curvature as expected by the methods. However, Table shows that when pure sampling is used their performance is worse than coin toss due to inversion of the probability curvature. While an AUROC under 0.5 may suggest inverting classifications on either side of the threshold, this is not possible here due to the zero-shot nature of these techniques: the fixed directionality is dictated by the method and based on the probability curvature thesis.
the
Table shows that base models present an acute vulnerability current detectors based on prob- ability curvature. Fortunately, typical users will use instruction-tuned models which often exhibit head-bias as shown in Figure 2. However, new methods are required prevent a determined ad- versary bypassing existing detectors through the use of base models with careful prompting. In addition, despite enterprise detectors most often being exposed to from instruction-tur tunee mod- els, the vast majority of the existing literature performs experiments on base models. Our results show these two classes of models should be tested separately when researchers are designing and validating future detection algorithms. Further experimental details may be found in Appendix J.
8

---

Table I: Performance comparison across different machine-generated text black box detection meth- ods, language models, and datasets (XSum (Narayan et al.. 2018), SQuAD (Rajpurkar et al., 2016), and WritingPrompts (Writing, Fan al. 2018)). Results are shown for two sampling configurations: top-k 50 and pure sampling (top-k: None). As predicted by DMAP, sfae-of-the--a detectors based on "probability curvature' are effective when top-k sampling is used but not with pure sampled text.
XSum
SQuAD
Writing
Method
Model
k: 50
k None
50
k None
= 50
k None
Alama-3.1-81
0.702
0.200
0.739
0.208
0.915
0.289
FAST-DETECTGPT
Mistral-7B-v0.3
0.770
0.276
0.819
0.299
0.906
0.339
Qwen3-8B
0.765
0.289
0.612
0.320
0.923
0.377
Llama-3.1-8 0.606
0.408
0.527
0.299
0.723
0.422
DETECTGPT
Mistral-7B-
0,679
0,486
0.586
0.365
0.688
0,457
Qwen3-8B
0.635
0.445
0.463
0.380
0.724
0.479
Llama-3, 3.1-88. 1-88B
0.825
0.325
0.849
0.365
0,942
0,410
BINOCULARS
Mistral-7B-v0.3
0.823
0.350
0.851
0.416
0.931
0.404
Qwen3-8B 0.857
0.416
0.752
0,467
0.949
0.492
5.3 STATISTICAL SIGNATURES OF POST-TRAINING DATA IN INSTRUCTION MODELS
This section studies the effect of instruction fine-tuning on DMAP plots. Figure we saw that two instruction tuned models, ChatGPT and Mistral Instruct, systematically over-sample from the head of the OPT-125m probability distribution, whereas the non-instruction tuned base model Mistral 7B is tail-biased. This tail-biased behavior of the base model is understood and expected (see Appendix C). The real question what causing instruction tuned models to systematically over-select from tokens which OPT- 125m finds likely?
has previously been noted (Luo et al., 2025; Shen et al., 2024; Chhikara, 2025; Zhu et al., 2023; Yang & Holtzman, 2025) that instruction-tuned models are over-confident, in the sense that when answering questions they assign too much weight to answers they believe are likely correct. We hypothesize this may be explanation for what we observe in head-biased DMAP plots that place too much weight on likely tokens at the head of the distribution.
DMAP plots provide indirect method for studying model overeonfidence by checking for this head-bias in DMAP samples. We analyze whether this bias is present in the training data, the base model, and the corresponding fine-tuned model, thereby shedding light how biases in training data may propagate to downstream model generations. This analysis is particularly relevant given the common practice of using papperatrre-sapllee responses as fine-tuning data (Dubois et al., 2023).
We fine-tuned two sizes of Pythia models (Biderman al., 2023) on the OASST2 dataset (Köpf et al., 2023) with responses provided by humans, Llama 3.1 8B at temperature 1, and Llama 3.1 8B at temperature 0.7. Figure contains DMAP plots for texts generated by Pythia IB with four fine- tuning configurations. For experimental details, DMAP plots for the fine-tuning data, and repeated
results on the other model, see Appendix
Our main finding is that the only head-biased model was the one fine-tuned on temperature-sampled data, which was turn the most head-biased fine-tuning data. We also see that human-written instruction fine-tuned data has a dramatic tail-collapse, much larger than seen in other human-written text. In addition, we see increased density in the final bin for fine-tuned models, which might demonstrate how DMAP could be used to detect mild overfitting during SFT and inform early- stopping strategies, a hypothesis we leave to explore further in future work. Finally, we note that all of our instruction-tuned models had less tail-biased distribution than the original Pythia model. These initial experiments point to the utility of DMAP in studying how instruction-tumi tuninn affects the distributional properties of machine text produced by downstream models.

---

10-
1.0
1.3
1.-
Decsity
0.8-
0.3-
0.8
0.2-
0.2
0.2-
0.0
0.0
0.0
0.4
0..6
0.0
0.2 0.4
DMAP Samples
Figure 4: Using DMAP to investigate the effect of SFT post-training on synthetic data. DMAP plots generated with the evaluator model (OPT-125) on text generated by Pythia 1B models with (a) fine-tuning, (b) fine-tuned on OASST2 human data (Köpf et al., 2023), (c) fine-tuned on OASST2 with responses regenerated by Llama 3.1 8B at temperature 0.7, and (d) fine-tuned on OASST2 with responses regenerated by Llama 3.1 8B at temperature 1.0.
6 CONCLUSION
We introduced DMAP, a mathemattcally principled method for mapping text through language mod- els to a standardized statistical representation. DMAP addresses the fundamental contextualization problem that has limited previous approaches to statistical text analysis with language models. Three initial case studies demonstrate the broad utility of this tool. Through parameter validation, we demonstrate how DMAP can validate data integrity, helping to prevent natural and inevitable hu- man errors propagating through the research ecosystem. Our re-examination of detection methods showed that the widely-accepted probability curvature principle fails for pure sampling from base models, challenging foundational assumptions in the field and pointing towards principles for de- signing more robust detection strategies. Importantly, our experiments highlight that existing detec- tors are critically vulnerable to adversarial attacks using base models with pure sampling in the black box setting due to inversion of the expected curvature. Finally, our analysis of instruction models reveals how statistical fingerprints of training data persist in downstream model outputs, providing new insights into the sources of overconfidence in aligned models.
Beyond these specific applications, DMAP offers several key advantages: it is compatationally effi- cient, requiring only forward passes through models such as OPT- 125m that may be run on consumer hardware; it provides intuitive visualizations that make complex distributional patterns immediately apparent; and it enables rigorous statistical testing. The method's model-agnostic nature means can be applied across different architectures and scales, making it a versatile tool for research and industry.
Our initial investigations point toward promising future research directions. For instance, DMAP might used for data curation and efficient fine-tuning (Ankner et al., 2025), or to advance calibra- tion approaches for instruction-tuned models, rather than using temperature scaling to align per- plexity with pre-trained models. Alternatively, future work may calibrate DMAP distributions to match human text patterns, potentially offering more nuanced control over model confidence dur- ing frontier model training. The distinct DMAP signatures we observed across different text do- mains (poetry, news, technical writing) suggest the method could formalize how language model performance varies across contexts, separating inherent prediction difficulty from model-specific limitations. Perhaps most intriguingly, the dramatic differences in DMAP plots when generator and evaluator models differ point toward entirely new approaches to language model identification and forensic analysis. We are most excited for the applications we have yet anticipate and believe that we have only scratched the surface of what we can learn through close examination of next- token probability distributions. DMAP provides clear, principled window into that rich statistical landscape.
10

---

7 REPRODUCIBILITY STATEMENT
All datasets used are public or generated as detailed our experimental setup in Section 5 and Appendices J and Code implementing DMAP is available on GitHub."
ACKNOWLEDGMENTS
K.W. financially supported by the Science and Technology Facilities Council (STFC) through the STFC DiS-CDT at the University of Cambridge. Y.L. was supported by the STFC DiS-CDT scheme and the Kavli Institute for Cosmology, Cambridge. We would also like thank our review- ers for helpful feedback and comments.
REFERENCES
Albert Q. Jiang and others. Mistral 7B, 2023. URL
2655.
Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew L Leavitt, and Man- sheej Paul. Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: /ttp::://ppeeeee wttp//peeeeee.ee//////// net. http://wwww.eeeeoooeeeeeee-----.
Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. Fast-DetectGPT: Effi- cient Zero-Shot Detection of Machine- Generattd Text via Conditional Probability Curvature. In The Twelfth International Conference on Learning Representations, 2023. 20233
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Au- toregressive Language Modeling with Mesh-Tensorflow, March 2021. URL https::/ //oi.. org/10.528 zenodo. 5297715.
Prateek Chhikara. Mind the confidence gap: Overconfidence, calibration, and distractor effects in large language models. arXiv preprint arXiv:2502.11008, 2025.
N. David and N. L. Johnson. The probability integral transformation when parameters are esti- mated from the sample. Biometrika, 35(1/2):182-190, 1948.
Matthieu Dubois, François Yvon, and Pablo Piantanida. MOSAIC: Multiple Observers Spotting AT Content. In Findings of the Association for Computational Linguistics: ACI. 2025, pp. 24230- 24247, 2025.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani. Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback. Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: http://pppeeeeeeeeeeeepe.eo wtattt/ooommmmdddttteotta LcKX..
Liam Dugan, Alyssa Hwang, Filip Trhlik, Andrew Zhu. Josh Magnus Ludan, Hainiu Xu. Daphne Ip- polito, and Chris Callison-Burch. RAID: A Shared Benchmark for Robust Evaluation of Machine- Generated Text Detectors. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pp. 22463-12492 2024.
Ebtesam Almazrouei and others. The Falcon Series of Open Language Models, 2023. https:::
//arxiv. wwtp://.rrgaab////////o/ooosse// /231. 11111.68887 16867.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical Neural Story Generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2018.

---

Lizhe Fang, Yifei Wang, Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, and Yisen Wang. What is Wrong with Perplexity for Long-context Language Modeling? arXiv preprint arX1v:211022571, 2024.
Ronald Aylmer Fisher. Statistical methods for research workers. Number 5 in Biological Mono- graphs and Manuals. Oliver and Boyd, Edinburgh, 1928.
Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. GLTR: Statistical Detection and Visualization of Generated Text. In Proceedings of the 57th Annual Meeting of the Association for Compulational Linguisties: System Demonstrations, pp. 111-116, 2019.
Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E. Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69(2): 243-268, 2007.
Steven Goodman. A dirty dozen: twelve p-value misconceptions. In Seminars in hematology, volume 45, pp. 135-140. Elsevier, 2008.
Aaron Grattafiori et al. The Llama 3 Herd of Models, 2024. URL URR.ttpss//exxxv.rrrg 2407.22 21733.
Abhimanyu Hans, Avi Schwarzschild, Valeriaa Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Spotting LLMs With Binoculars: Zero- Shot Detection of Machine-Generatee Text. In Proceedings of the 41 International Conference on Machine Learning, pp. 17519-17537, 2024.
Daphne Ippolito, Daniel Duckworth, and Douglas Eck. Automatic Detection of Generated Text is Easiest when Humans are Fooled. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1808-1822, 2020.
Tom Kempton and Stuart Burrell. Local Normalization Distortion and the Thermodynamic Formal- ism of Decoding Strategies for Large Language Models. In Christos Christoddullpoollss, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Findings of Association for Computational
Linguistics: EMNLP 2025, pp. 22216-22231, Suzhou, China, November 2025. Association for doi: 10. 10.18653/v1/0223 12n2://///......nngeeeeemmm findings--mmlpp 1210. URL https: http:///aaaaaattaaaaaeee org. 2225. findings emnlp 1210/.
Computational Linguistics. ISBN 979-8-89176-335-7.
Tom Kempton, Stuart Burrell, and Connor J Cheverall. TempTest: Local Normalization Distortion and the Detection of machine-generated Text. In International Conference Artificial Intelli- gence and Statistics, pp. 1972-1980. PMLR, 2025.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnay Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. OpenAssistant conversations democratizing large language model align- ment. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023, Curran Associates Inc.
Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. Paraphrasing evades detectors of Al-generated text, but retrieval is an effective defense. Advances in Neural Information Processing Systems, 36, 2024.
Beier Luo, Shuoyuan Wang, Yixuan Li, and Hongxin Wei. Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator. arXiv preprint 1xXiv:2505.16600 2025.
Clara Meister and Ryan Cotterell. Language Model Evaluation Beyond Perplexity. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3328, Online, August 2021. Association for Computational Linguistics. doi: 10. 1411883311/222 Mol--oong..1. 414. URL. https: //a //aclaa woclannooooogoo org..yyy.....11111 11121..al-ooon..... 111/lonnn/.../....
12

---

Niloofar Mireshghallah, Justus Mattemn Sieun Gao, Reza Shokri, and Taylor Berg-Kirppatrick. Smaller Language Models are Better sero -eenrrttee Text Detectors. Proceed- ings of the 1 18th Conference of the European Chapter of the Association for Computational Lin- guistics ( (Volume 2: Short Papers), pp. 278-293, 2024.
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. De- Text Detection using Probability Curvature. In Interna-
tectGPT: Zero-Shot tional Conference on Machine Learning, pp. 14950-249622 PMLR, 2023.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2018.
Byung-Dob Oh and William Schuler. Transformer Based Language Model Surprisal Predicts Hu- man Reading Times Best with About Two Billion Training Tokens. In Findings the for Computational Linguistics: EMNLP 2023, pp. 1915-1921, 2023.
the Association
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ Ques- tions for Machine Comprehension of Text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp.
2016.
Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can AI-Generated Text be Reliably Detected?, 2025. URL petps::/arxxoe org/ / 11156.
Maohao Shen, Subhro Das, Kristjan Greenewald, Prasanna Sattigeri, Gregory Wornell, and Soumya Ghosh. Thermometer: Towards Universal Calibration for Large Language Models. In Proceed- ings of the st International Conference on Machine Learning, pp. 44687-44711, 2024.
Jinyan Su, Terry Zhuo, Di Wang, and Preslav Nakov. DetectLLM: Leveraging Log Rank Infor- mation for Zero-Shot Detection of Machine-Generaal Text. In Findings of the Association for Computational Linguisties: EMNLP 2023, pp. 124121 2023.
Yutao Sun, Zhenyu Li, Yike Zhang, Tengyu Pan, Bowen Dong, Yuyi Guo, and Jianyong Wang- Efficient Attention Mechanisms for Large Language Models: A Survey, 2025. URL https:
// //arxiv. http://..ggg.aaaa/.....1
George R Terrell and David W Scott. Oversmoothed nonparametric density estimates. Journal of the American Statistical Association, 80(389):209-211 1985.
Sean Trott and Pamela Rivière. Measuring and Modifying the Readability of English Texts with GPT-4. In Proceedings of the Third Workshop on Text Simplification, Accessibility and Readability (TSAR 2024), pp. 134, 2024.
Vivek Verma, Eve Fleisig, Nicholas Tomlin, and Dan Klein. Ghostbuster: Detecting Text Ghostwrit- ten by Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vol- ume 1: Long Pupers). pp. 1702-1717, 2024.
An Yang, Anfeng Li, Baosong Yang, al. Qwen3 Technical Report, 2025. URL https:// arxiv. org/ abs / 250.. 09388.
Chenghao Yang and Ari Holtzman. How Alignment Shrinks the Generative Horizon. arXiv preprint arXiv:2506.11 17871, 2025.
Susan Zhang, Stephen Roller. Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open Pre-trained Transformer Language Models. arXiv preprint Xtx:::2225.00066 2022.
Chiwei Zhu, Benfeng Xu, Quan Wang. Yongdong Zhang, and Zhendong Mao. On the Calibration of Large Language Models and Alignment. In The 2023 Conference Empirical Methods in Natural Language Processing. 2023.
13

---

A PROOF OF PROPOSITION 3.1
Suppose that text Wi ... u. has been generated by pure sampling from language model At time token is selected with probability p(u] we w;-1). . Token defines interval [a, of length p(u|to w--); (see the definition of DMAP, Section 3.1). The points a, are functions of both and the context w1 ... w;-1, to keep notation clean we suppress this dependence here.
Then, following the DMAP algorithm, point 2; is selected according to the uniform distribution on [a, b). Let P denote the partition of [0, 1) into intervals (a,b) corresponding different tokens given fixed context
-.
In order to prove that the process of picking v according to p(-lwr ... w;-1) and then picking a; according to U[a, bl) produces points distributed according to [ 1)), it is enough to show that for any interval (c. d), P(x, (c,d)) = c. enough to prove this for intervals (c. d) which
are fully contained in one of the intervals in the partition P, the result for larger (c, follows by standard rules of probability.
Now, given context - let (c. d) be subset of some interval (a, b) in P, where (a, b) is the interval corresponding to selection of some token v. Then
P(x; (c,d)) P(x; E (a, b))
P(a, E (c, P(æ. E (a, b))
(d - c)
p(ulw1 wi-1)
- a)
(b a) (d c)
d
(b a)
required. Here used that 2; E (a, exactly when the token v defining interval (a, was selected, which happens with probability p(ulw)- ... w;-1 = b a.
We have shown here that, for any choice of context wT W;-1 1, 24 will be distributed according UO0, 1). Thus we have shown that o; is independent of w) ... 15-1, and so the resulting sequence T1 ST is ii.d.
Finally we note that no properties of the language model were assumed in the above proof. In particular, one could define p(-lw1 ... w;-1) be the next token probability distribution resulting from sampling from Llama 3.1 at temperature 0.7. In that case, our result about being i.i.d. according to the uniform distribution on [0, 1] continues hold, provided the same language model decoding strategy pair is used in the generation of text and generation of DMAP plots.
Proposition 3.1 does not hold directly for entropy weighted DMAP, since entropy weighted DMAP does not produce points 24. One can make corresponding statements about the expected integral of [T over any interval and see that entropy weighted DMAP plots for texts generated by the evaluator model do converge to the uniform distribution, but rates of convergence do not follow so easily.
B ON THE SHAPE OF DMAP PLOTS FOR TEMPERATURE, TOP-P AND TOP-K SAMPLING
Recall that, for a language model and context wi ... W;-1, and for given values of r, æ and k temperature sampling picks token p; with probability
p(w,/w] ... wi-1
Cr (w wn ... w.: Evey p(ulwz 1
Top-k sampling defines the top-k set V3 to be the & tokens with highest model (conditional) proba- bility p(- p(-|w] w;-1 1) and assigns probability
p(wx/w) w;-1)
qb (w: (w/ww ... wi-1)
Σn Ever, p(u/w0 -.W
14

---

to tokens w; in the top-k set, and zero mass outside of the top-k set.
Finally top-p (nucleus) sampling orders the tokens in V by decreasing conditional probability p(-[wi -.-11)) and defines the top-p set Vx to be the first tokens, where is the smallest integer for which Σi=1 p(u; w;-1) >= π. Top-p sampling then chooses token w, with probability
p(w; W1 wi-1)
qm (w w;-1) Ever, p(ul11 ... wx-1)
We then ask, given a long sampled by top-k sampling, what are the limiting statisties of the size of the top-k set. The expected shape of the DMAP plot for top-k sampling is the function
The total mass of the top-k set is greater than .²)
renormalized to have integral one. This is a decreasing function, typically nearly flat on [0, 0.5), before smoothly decreasing towards 0.
Similarly, for top-p sampling, the expected shape the function
P( total mass of the top-sr set is greater than 2))
again renormalized have mass one. This flat on [0, T.. before decaying very rapidly. There is sharp cut off reflecting the majority of cases where the size of the top-I set is only just larger than
π, and a non-vanishing tail reflecting the fact that the proportion of times for which the top-1r set has mass close to is non-zero.
For the shape of the DMAP plots in the case of temperature sampling, we mention only that it is smooth deformation. There is an interval close to 0 upon which the DMAP is nearly flat, but this is much smaller than in the top-k case it is function of the mass of the most likely token, rather than of the top-k most likely tokens.
C WHY ARE BLACK-BOX BASE MODELS TAIL BIASED?
The fact that our plots of texts generated by one base language model and evaluated by another base language model (Figures and 23) are tail-biased partially supported by theory. In Kempton & Burrell (2025). it was shown that for a language model P and generation length T, pure sampling from P is the unique way of maximizing the sum entropy and per-token log-likelihood (as judged by P). That is, whenever we sample from a language model Q for which the entropy is at least as large as the entropy of P, the resulting plot text must have lower expected per-token lig-likli (as measured by P). This is statement about liglliielihood, not statement about position in the probability distribution, and so doesn't 't directly correspond to saying the DMAP plots should be tail-biased, but it gives strong indication in that direction.
D RATES OF CONVERGENCE IN PROPOSITION 3.1
Proposition 3.1 implies that DMAP plots of texts generated by pure sampling from language model, evaluated by the same language model, should look roughly flat. One might wonder whether make this statement detailed the following proposition.
precise with rate of convergence. The answer is positive, as
Proposition D.1. Given a language model text WT .T.WT generated by and number of bins k. let Z1, Zr be the set of points generated by DMAP with P as the evaluator model. Further define frequencies
ft = #li ... <
T
and the x2 statistic
2
X2 Tk Σ
ft
k
as Section 5.I. Then x2 is asymptotically distributed according to the Xb-1 (the x2 distribution with k- degrees of freedom).
15

---

An immediate application of this is that evaluate the plausibility of the statement "Text w1 wr was generated by pure sampling from language model P'. To do this, we compute the x² statistic c arising from the text w1 Wr and use standard statistics packages to compute the probability that points drawn according to the x distribution would be larger than c. For large T, this is arbitrarily close to the probability that text generated by language model P would generate as extreme a x² statistic.
We mention one note of caution about the word "asymptotically"" in Proposition D.1. For finite T, the x2 statistic is not perfectly distributed according the x² distribution (indeed supported on finite set), and so our computations of p-values are not exact. A conservative rule of thumb is that T should be at least 10k for reliable p-values, so one should evaluate least 400 tokens when plotting histograms with 40 bins.
Proposition D.I standard result in statistics, proved by a slightly delicate application of the central limit theorem.
One should be cautious in interpreting p- p-values, as they are famously prone to misinterpretation. Many articles can be found explaining these misconceptions, see for example Goodman (2008).
E EFFICIENCY AND EMPIRICAL CONVERGENCE RATES
This section provides a discussion algorithmic efficiency and empirical results the convergence of DMAP as the number tokens, and therefore DMAP samples, grows. This latter evidence is to supplement the theoretical analysis of convergence in Proposition D.I. Figure 5 shows DMAP plots for 200, 2000, and 20, 000 tokens. We see that for extremely small token sizes there is significant identifiable patterns emerging, and by
noise, while this quickly stabilizes. By 2000 tokens, 20, 000 the noise is fairly negligible with strong characteristic patterns present. If you are working with extremely small amounts of data with DMAP, you may reduce the impact of the noise by reducing the number of bins in your histogram. Even at 200 tokens this approach allows us to recover information about decoding strategies, see Figure
The runtime efficiency and speed of DMAP is dominated by the underlying language model. For vanilla transformers, we therefore have quadratic time and memory complexity, though this could be improved if considering language models based on more efficient alternative attention mechanisms (Sun al., 2025).
F ENTROPY WEIGHTING
In Section 3.2 we introduced entropy weighting as a way of suppressing the effect of tokens selected at times where uncertainty around the next token is very low and a single option has extremely high likelihood (say probability 0.995). In such situations, the points plotted by DMAP tend to have distribution close to the uniform distribution irrespective of the method of text generation.
To illustrate this point, we consider DMAP plots of texts generated by Mistral-7B-v0.3 and evaluated by Llama-3.1 1-BB. The full weiropy- DMAP plot is present as the first plot on the second row of Figure 8. In Figure 7 we separate out points by the entropy of the next token probability distribution. plotting the low entropy entropy case in (c). As predicted, in the low entropy case the DMAP plot is extremely flat, since very little information is uncovered from these points. This illustrates weight to points of entropy below 2. this example, tokens where the entropy is below 0.5 make up 7474 of the 77321 tokens evaluated. In contrast, (b) and (c) contain noticeable peaks.
in (a), the medium entropy case in (b), and the high
reasons for giving lower
G PROMPT SENSITIVITY ANALYSIS
This section includes experiments investigating the sensitivity of DMAP to the inclusion or exclusion of the prompt. In addition, we test the sensitivity of DMAP to the first few tokens for which there is no or small amounts of context shape the conditional distributions.
16

---

Sconi 200, tangle &. achae
Saee 208 Seapet polli hele
2.50
Takres 24, Tragedh Tap Nome
Telres Thagg Tapikito
1.70
1.75
1.50
3.5
3.50
1.35
1.25
1.00
1.00
1.00
0.75
0.75
0.75
0.5
0.30
0.50
0.20
0.0
0.00
3.6
0.0
0.2
0.4
1.15
1.2
1.2
Density
1.0
1.0
1.25
0.6
0.75
0.15
6.4
004
0.2
0.35
0.0
0.2
0.4
3.5
1.0
3.0
0.2
0.4
6-5
0.5
1.0
0.0
0.2
0.4
0-5
0.5
000
02
2.5
3.0
1.50
1.25
1.0
1:00
0.0
0.5
0.72
0.4
0.50
0.2
0.2
60
52
0.4
3.6
0/3
1.0
0.0
0.2
0.4
0.5
0.8
2.0
3.0
DMAP Samples
Figure 5: Empirical convergence analysis of DMAP with 40-bin histograms. This figure illustrates the convergence of 40-bin DMAP histograms the number of tokens, and thus DMAP samples, increases from 200 to 20, 000. At extremely low sample sizes we see high variability and noise, while the strong characteristic shapes we expect emerge the number of tokens increases. We recommend choosing the number of bins as a function of the number of tokens being evaluated, for example using the Terrell-Scout rule, so as mitigate this noise.
First, we repeated a large scale white box and black box experiment, to compare the efficacy of DMAP with and without access to the prompt. The results shown in Figure 8 (without access to the prompt) and Figure 9 (with access to the prompt). These plots show that in realistic settings where one may wish to apply DMAP, the utility is equal with without access to the prompt. However, note that without access to the prompt, despite the characteristic shapes being strongly present, there is slight increase in the noisiness or irregularity in the plot. This is expected, since the reduced context will cause increased variance in the conditional distributions, particularly for tokens near the start of the text.
To explore sensitivity to the initial tokens further, for both settings including and excluding the prompt, we conducted an additional 32 experiments. Here, we also consider an initial cutoff- For an initial cutoff of N, we use the first N tokens to compute conditional probabilities for tokens N 1,... but do not plot the corresponding N DMAP samples. This is intended to remove noise associated with the early tokens with insufficient context. We consider four core configurations i) include prompt and initial cutoff = 30, ii) include prompt and initial cutoff 0, iii) exclude prompt and initial cutoff 30, and iv) exclude prompt and initial cutoff = 0. Figure 10 analyses these four configurations for texts of length 300 tokens, while Figure 11 considers texts of length 50 tokens. These results show that at realistic token counts both the prompt and initial cutoff have only minor effect. However, at very small token counts, DMAP is affected by the prompt. This is to be expected as before, since the prompt a significant proportion of the total tokens in this case and therefore the statistical effect is larger. This gives a further reason to caution against using DMAP on extremely small samples of data.
17

---

Scaw 20, tanguuA Sugpat shee
Takres 24, Tegach Tap Tapk
Telres thega LA
1.75
3.2
12
1.25
03
64
048
0.4
0.2
2.6
0.4
08
Density
1.2
1.0
0.0
0.0
0.6
6-2
0.2
0.0
022
0.4
3.5
1.0
3.0
022
0.4
88
0.5
1.0
0.0
0,2
84
64
0.8
00
04
088
2.5
3.0
Thry
1.0
1.5
1.0
1.4
04
0.5
0.4
0.2
02
60
52 3.6
000
0.2
0.4
0.5
DMAP Samples
Figure 6: Empirical convergence analysis of DMAP with 5-bin histograms. This figure shows that noise due to extremely small sample sizes may be mitigated by using fewer bins. In this regime, we still see approximate versions the characteristic patterns present the top row of Figure 2, in contrast the noisy estimates found in the first of Figure 5.
(a)
(c)
0.8
Density
0.6
0.2
6.9
8m
0.2
0.4
06
0. 1.0 0.
07
0.8
0.5
0.8
10
0.
07 044 06
0.0 1.0
DMAP Sumples
Figure 7: DMAP plots of text generated by Mistral and evaluated by Llama. Plot (a) shows only those tokens for which the entropy of the next token probability distribution is below 0.5, (b) shows entropy in the range (0.5,1), and (c) shows points where the entropy is greater than 2. We see that the plot (a) of low entropy tokens looks flat even though are in the tail-biased case of pure sampling from a base model with different models used for generation detection.
H RELATIONSHIP TO PROBABILITY INTEGRAL TRANSFORM
One way to view DMAP is that it builds on the classical probability integral transform (PIT), see Fisher (1928); David & Johnson (1948); Gneiting et al. (2007), as discussed in the related work section. However, a key difference between DMAP and PIT is that DMAP dynamically orders tokens by sorting the logits prior to generating sample the unit interval. As such, it also encodes rank information as studied in works such as Su et al. (2023). For completeness, in Figure 12 we
18

---

LJama-3. 3.1-BB, Llame-3.1-88
Lleme-3.1-88, Mistral-7B-v0.3
Llama-3.1-8B, Falcon3-7B-Bou
1.4
1.0
1.2
0.3
3.0
1.0
0.0
0.5
0.5
0.6
0.4
0.4
0.2
0.2
0.0
0.2
0.4
0.55
0.80
1.0
0.0
0.2
0.4
00.5
1.0
0.0
0.2
0.6
0.6
0.8
1.0
Mistral-7 B-.0.3, Llama-3311
Mistral-7B-r0..3 Mistral-7B-v0..3
Mistral-7B-r0..3 Palconn--78
1.0
1.0
1.0
0.8
Density
0.5
0.03
0.8
0.5
01.5
0.4
0.4
0.2
0.2
0.0
Folcon3-7B- BBBBane--B-BBsse, Llama-331 3.1-80
Falcoo3- BDD-aa,,, Mistral-7B-vO.3
Felcon3-77 Base, Falcon3-7B-B8 7---Be
1.6
1.4
1.4
1.2
3.0
1.0
0.6
0.8
0.0
0.4
0.4
0.2
0.0
0.2 0.4
0.6
0.4
0.6
3.0
0.0
0.2
0.6
0.6
0.8
1.0
DMAP Samples
Figure 8: White and black box experiments excluding the prompt with texts of length 300. For these plots, the prompt was excluded. White and black box evaluation. Plots are labeled: (Generator Model, Evaluator Model). Plots on the main diagonal (green) correspond to the white box setting, while off-diagonal plots (pink) correspond to the black box setting. Observe that in the black box setting we see plots with heavy tails.
perform this extension of PIT to categorical variables where dynamic token ordering is replaced with a random token ordering. We have been unable to extract useful information from these plots.
I ANALYSIS OF ADVERSARIAL PARAPHRASING
DMAP is a general tool for the statistical analysis of text, and not itself a specialized detector of machine-generatte text. That said, we have seen throughout this paper that it can highlight important differences between machine and human generated text, for example, see Figure 2.
Paraphrasing is well-known and important technique that has been shown to be effective ad- versarial attack against common machine-generatee text detectors (Sadasivan et al., 2025). this appendix, we ask what the impact is of paraphrasing DMAP visualizations. In partieular, do paraphrasing attacks make DMAP visualizations of synthetic data look more like that of human data, analogous to their impact on common aechine-generated text detectors? To answer this, performed experiments using DIPPER (Krishna et al., 2024), popular model used to evaluate ro-
19

---

LJama-3. 3.1-BB, Llame-3.1-88
Lleme-3.1-88, Mistral-7B-v0.3
Llama-3.1-8B, Falcon3-7B-Bon
1.0
3.4
122
0.3
1.0
0.2
0.0
0.4
0.2
0.0
0.2
0.4
0.55
0.80
1.0
0.0
0.2
0.4
00.5
1.0
0.0
0.0
0.2
0.6
0.6
0.8
1.0
Mistral-7 B--003,, Llama-3.1
Mistral-7B-r0..3 Mistral-7B--00.3
Mistral-7B-vv0.3 Palconn- BBB-BBe
1.2
1.0
Density
1.0
0.0
0.8
0.8
0.6
0.4
0.4
0.2
0.2
0.9
Pelcon3-TB-Brss Llama-331 3.1-80
Falcoo3- --- TD-Base, Mistral-7B-vO.33
Felcoo3-77 Base, Falcon3-7B-BB 7---Be
1.75
1.0
0.8
1.00
3.0
0.0
0.8
0.4
0.25
0.0
0.2
0.4
0.6
0.4
0.6
3.0 0.0 0.2
0.6
0.6
0.8
1.0
DMAP Samples
Figure 9: White and black box experiments including the prompt with texts of length 300. For these plots, the prompt used in computation of next token probabilities of later tokens, but DMAP samples were not generated for tokens the prompt. White and black box evaluation. Plots are labeled: (Generator Model, Evaluator Model). Plots the main diagonal (green) correspond the white box setting, while off-diagonal plots (pink) correspond to the black box setting. Observe that in the black box setting we see plots with heavy tails.
bustness to adversarial paraphrasing attacks, for example, see Sadasivan et al. (2025); Kempton et al. (2025). Our experiments show that DMAP visualizations robust paraphrasing attacks with DIPPER (Krishna et al., 2024). In particular, we see that paraphrased machine-generane text and human text are clearly distinct, Figures 13, 14 and 15.
In addition, these figures also show that DMAP identifies subtle shifts between synthetic data and paraphrased synthetic data, shedding some new light on how paraphrasing attacks alter the distri- bution of text. Collectively, these results are strong evidence that an effective text detector may be built on top of DMAP that far more robust paraphrasing attacks than existing approaches. In addition, these results prompt the use of DMAP to explore the whole array of paraphrasing attacks the literature obtain deeper understanding of their underlying statistical mechanisms.
20

---

1.2
300
64
632
0.0
0.2
0.1
3.6
0.0
1.0
3.0
8.6
0.8
1.0
800
0.3
9.4
6.5
0.8
1.0
0.2
0.6
0.6
3.0
10
1.0
Density
6.2
3.6
0.8
0.2
0.0
(12)
(2)
0o) (p)
6.8
6.9
69
DMAP Samples
Figure 10: Prompt sensitivity analysis with texts of length 300 tokens. Each column corresponds to a generation strategy. From left to right columns, we have pure, top-p = 0.8, temperature T = 0.8, and top-k 50 sampling. (a)-(d) include the prompt and set initial cutoff 30. (e)-(h) include the prompt and set initial cutoff O. (i)-(I) exclude the prompt and set initial cutoff 30. (m)-(p) include the prompt and set initial cutoff = 0. The second and fourth
show setting initial cutoff increased variability. Comparing the first and third rows shows excluding the prompt causes slight increase in noise. In contrast Figure 11, overall we stable characteristic curves and low sensitivity to both the prompt inclusion and initial cutoff in this setting where the sample size realistic. This be expected, since the prompt a small proportion the total tokens this case.
EXPERIMENTAL SETUP FOR SECTION 5.2
We follow setup similar for example, Mitchell al. (2023) Kempton al. (2025). In par- ticular, given a large language model and sample of human text from XSum (Narayan et al., 2018), SQuAD (Rajpurkar al., 2016), or WritingPrompts (Writing) (Fan et al., 2018), we construct 150 tokens of synthetic text using the first 30 tokens as context. We then perform zero-shot classification on the balanced dataset of real and synthetic data. For Fast-DetectGPT (Bao al., 2023) and De- tectGPT (Mitchell et al., 2023) we use GPT-Neo 2.7b (Black et al., 2021), which Fast-Detectto report as being empirically superior for black box detection with their method. For Binoculars we use the recommended combination of Falcon 7b for the observer model and Falcon 7b Instruct for
21

---

12
3.0
048
0.0
0.2
0.1
3.6
0.0
1.0
3.0
0.2
8.6
0.8
1.0
80
0.3
9.4
0.5
0.0
0.2
0.6
0.6
3.0
1.0
3.2
1.0
0.8
0.0
0.4
Density
6:2
2.6
0.4
0.8
6.9
6.4
0.2
0.0
(12)
(2)
001
(p)
666
0.2
DMAP Samples
Figure 11: Prompt sensitivity analysis with texts of length 50 tokens. Fach column corresponds to a generation strategy. From left to right columns, we have pure, top-p = 0.8, temperature T = 0.8, and top-k = 50 sampling. (a)-(d) include the prompt and set initial cutoff = 30. (c)-(h) include the prompt and set initial cutoff = O. (i)-(l) exclude the prompt and set initial cutoff = 30. (m)-(p) include the prompt and set initial cutoff O. All four rows show setting significantly increased variability and noise compared to the setting with 300 tokens per text (Figure 10). Compared to Figure 10, we increased sensitivity sizes are very small. This is to be expected, since the prompt is a significant proportion of the total tokens in this case.
both the prompt inclusion and initial cutoff in this setting where the sample
the performer model (Ebtesam Almazrouei and others, 2023). The generation models we use Llama-3.1 8B (Grattafiori et al., 2024), Mistral-7B-v0.3 (Albert Q. Jiang and others, 2023), and Qwen3-8B (Yang et al., 2025). For each method, report AUROC the evalulation score in (Mitchell et al., 2023; Bao et al., 2023; Kempton et al., 2025).
K EXPERIMENTAL SETUP FOR SECTION 5.3
We conduct our experiments on two sizes of Pythia models (410m, IB) fine-tuned with one of three instruction fine-tuning datasets: including only human-written written text and two with responses that have been partially regenerated with an external language model.
22

---

0.8
1..6
3.0
1.0
0.40
1.8
3.8
3.0
(g)
50
0.2
04
3.6
0.8
3.0 0.0
02
3. 0.6
0.
1.0
08
82
0.4
0.5
3.0
1.5
60
02 04
3.6
0.8
10
DMAP Samples
Figure 12: Comparison of DMAP with Probability Integral Transform. Each column corresponds to a generation strategy. From left to right columns, we have pure (top-y=1), top-p = 0.8, temperature 0.8, and top-k = 50 sampling. The top row (a)-(d) shows PIT with uniformly random token order. The bottom row (c)-(h) shows standard DMAP for comparison. In this scenario, we see plots for PIT with uniformly random token order do not effectively differentiate decoding strategies.
Real
Synthetic
Synthetic (Paraphrased)
1.2
0.8
Densly
0.4
0.2
0.0
DMAP Samples
Figure 13: DMAP Analysis of Paraphrasing Attacks. These plots show DMAP visualizations for text generated by real, synthetic, and paraphrased synthetic data. The dataset used was SQuAD (Rajpurkar et al., 2016). Synthetic data was generated using Llama-3.1-88 completions. Para- phrased data was created using the model DIPPER developed for adversarial paraphrasing attacks on machine-generated text detectors (Krishna al., 2024). These plots show that paraphrased machine- generated text and human text are clearly distinct DMAP visualizations. addition, DMAP sheds light on subtle changes in the distribution between standard synthetic and paraphrased text, where we see a slight flattening of the distribution.
Fine-Tuning The instruction fine-tuning datasets are the following: (1) OASST2 (Köpf et al. (2023)), human-written instruction tuning dataset, (2) OASST2 with responses regenerated with Llama 3.1 8B model with generation temperature 0.7, and (3) The same as (2) but with generation temperature 1.0. OASST2 is a tree- structurdd multi-turn dataset with various conversation paths, but for fine-tuning purposes response to single query. Although this means we are using a portion of the text in the dataset, we continue refer to it as OASST2 throughout this work. The data is split into train and validation set following the original splitting OASST2 on the Huggingface- hosted dataset 2 Because the
extract only the first prompt-response pair,
only evaluate the
Phttps::/mmitt hittts://huggingfface co co/datasets https//eeeeennnneeeeeecsssss hatp:///......ttttatt///////anna
23

---

Real
Synthetic
Synthetic (Paraphrased)
1.2
1.0
0.8
Density
0.4
0.0
0.4
0.8 1.0
0.0
9.3
0.4
0.8
0.8
1.0
1.0
DMAP Samples
Figure 14: DMAP Analysis of Paraphrasing Attacks. These plots show DMAP visualizations for text generated by real, synthetic, and paraphrased synthetic data. The dataset used was Reddit Writing (Fan et al., 2018). Synthetic data generated using Alama-3.1-88 completions. Paraphrased data was created using the model DIPPER developed for adversarial paraphrasing attacks on machine- generated text detectors (Krishna et al., 2024). These plots show that paraphrased machine-generated text and human text are clearly distinct in DMAP visualizations. In addition, DMAP sheds light on subtle changes in the distribution between standard synthetic and paraphrased text, where see slight flattening of the distribution.
Real
Synthetic
Synthetic (Paraphrased)
1.2
0.8
Density
0.4
0.2
0.0
0.0
0.2
0.86
DMAP Samples
Figure 15: DMAP Analysis of Paraphrasing Attacks. These plots show DMAP visualizations for text generated by real, synthetic, and paraphrased synthetic data. The dataset used was XSum (Narayan et al., 2018). Synthetic data was generated using Llama-3.1-8B completions. Paraphrased data was created using the model DIPPER developed for adversarial paraphrasing attacks machine- generated text detectors (Krishna et al., 2024). These plots show that paraphrased machine-generatee are clearly distinct in DMAP visualizations. In addition, DMAP sheds light on
text and human subtle changes in the distribution between standard synthetic and paraphrased text, where we see slight flattening of the distribution.
responses in fine-tuning datasets (2) and (3) are generated using a non-instruction-tune base model, keep the first 20 tokens of the human response text part of the response generation prompt. For the creation of the DMAP plots, we only consider the tokens beyond this point.
For the fine-tuning step, the prompt-response pairs formatted to be separated with line break without any additional role tags or special tokens, so as to ensure naturalistic output and to allow for exact comparison between fine-tuned and con-iinettuned models.
Evaluation. We use OPT-125 as the evaluator model throughout, creating DMAP plots for the val- idation split of the respective datasets. In each case, the evaluated models are prompted in the same way as with the creation of the the training data, with only the newly-generated tokens contributing to the creation of the DMAP plot.
24

---

10-
1.0
1.0
Decaity
0.8-
0.6
0.8-
0.2-
0.-
022
0.0
0.2
0.0
DMAP Samples
Figure 16: DMAP plots created using OPT-125 for the three fine-tuning datasets: a) OASST2 human-written prompt-response pairs, b) OASST2 with responses regenerated by Llama 3.1 8B at temperature 0.7. and c) OASST2 with responses regenerated by Llama 1 8B at temperature 1.0.
(a)
(b)
(c)
(d)
1.4
1.4
1.5
1.-
1.2
1.2
Deasity
0.8-
0.3-
0.6-
0.6-
0:5-
0.-
0.4-
0.4
0.4
0.4
0.0
0.2
0.6
0.2
1.0
0.0
0.8
0.9
0.0
1.0
0.0
0.2
0.8
0.00
3.0 0.2
0.2
0.4
0.6
1.0
DMAP Samples
Figure 17: DMAP plots generated by pure sampling from Pythia 410m, models with (a) fine- tuning, (b) fine-tuned on OASST2 human data, (c) fine-tuned on OASST2 with responses regener- ated by Llama 3.1 8B temperature 0.7, (d) fine-tuned OASST2 with responses regenerated by Llama 3.1 8B at temperature 1.0.
K.1 RESULTS
We first visualize the DMAP distribution for each of our three fine-tuning datasets in Figure 16, each of which exhibit relatively typical shape for the respective data sources: human-generated. synthetic pure-sampled and synthetic lemperature-sampled at temperature 0.7.
We observe next the extent to which fine-tuning on these datasets affects the DMAP plots. We present the results for a single model (Pythia IB) in Figure 4, with the remaining experiment for Pythia 410m found here Figure showing the same visual pattern.
OBSERVATIONS ON DETECTOR DESIGN
L. USING DIFFERENT DETECTOR AND GENERATOR MODELS CAN BE USEFUL
Methods based on probability curvature seem to work best when the generating and detecting models are the same (although Hans et al. (2024) and Dubois et (2025) use more than one model in detection). Comparing the distributions of human-written text (Figure 18), pure-sampled text where generator and detector models are the same (Figure 2) and pure-sampled text with different generator and detector models (Figure 23), one that human- written text looks most different from machine text when a different detector model is used. Perhaps an alternative conclusion to be drawn from Table I is that variant Past-DetectGiP
remain effective detector of pure-sampled text, but DMAP should be first used to calibrate whether machine generated text is head-biased or tail-biased.
25

---

L.2 HUMAN- WRITTEN TEXT HAS MORE SUBTLE DISTRIBUTION THAN CURRENTLY EXPLOITED
Detectors based on probability curvature exploit the fact that human-written text is tail-biased. In Figure 18 we see that, while this true, it is also true that they contain tokens from the very tail of the language model probability distribution less often (tail-collapse). This is particularly important as the tail of the probability distribution is where log-likelihoor values are at their most extreme. Detectors using log-likelihood or log-rank to distinguish between human and machine generated text would presumably be more effective if they discounted this bottom five percent of the probability distribution where human-written text is under-represented.
L.3 INSTRUCTION TUNED MODELS ARE EASIER TO DETECT THAN BASE MODELS
It noted that in Ippolito et al. (2020) that it is easiest for machine detect machine-gennrater text when decoding strategies such as top-p, top-k and temperature have been used. Figures 2 and 18 would support this conclusion, the distributional differences between human and machine text are clearly much greater when top-p, top-k or temperature sampling have been used.
M FURTHER QUESTIONS
1. In Shen al. (2024) the authors attempt to curb the overconfidence of instruction fine-tuned language models by using temperature scaling with temperature larger than I in order to bring the perplexity in line with pre-trained language models. Does DMAP give any insight to this process? What would happen if, instead of trying to bring perplexity in line with the pre-trained language model, one tried to bring the DMAP plots line with human text? DMAP plots are much better tool for checking calibration with pre-trained language model than the currently used likelihood of the top token.
2. In Figure 18 we show DMAP plots for different examples of numan-writted text. These plots not perfectly flat, and are slightly different in different settings (e.g. poetry vs news). Do these plots effectively describe the underperformance
the evaluator model in different settings in way that separates the underperformance of the language model next token prediction from the inherent difficulty of next-token prediction (which varies by setting)? Can one formalise this and extract useful metrics?
3. Figure 9 shows texts generated by Llama, Mistral and Falcon, evaluated by Llama, Mistral and Falcon. In each case, the shape of the DMAP plot is very different depending on whether the evaluator and generator models are the same. Can these ideas form the basis for new approach to language model identification?
N FURTHER PLOTS
N.1 SYNTHETIC TEXT
In this section we include illustrative plots with larger evaluation models. We see similar phenomena as in Figure? 2.
N.2 HUMAN-WRITTER TEXT
We plot humtn-written text in Figure 18, evaluated by OPT-125m. We use the RAID dataset across four different categories: scientific abstracts, books, news and poetry.
We see differences in these plots, corresponding to the differing competence of OPT-125m in these areas.
N.3 BLACK BOX BASE LANGUAGE MODELS
In Figure 8 we plotted pure generated text from base language models Llama 3.1 8B, Mistral 7B and Qwen3 8B, evaluated by each other. In Figure 23 we see the same texts evaluated by OPT-125m.
26

---

1.
Decalty
0..5
0.10
DMAP Samples
Figure 18: Illustrative DMAP histograms. Plots of XSum data (Narayan et al., 2018) generated by OPT-6.7b, evaluated by OPT-6.7b. The generation strategies (left to right) (a) pure sampling, (h) top-p = 0.8 sampling, (c) temperature 7 0.8 sampling, and (d) top-k = 50.
(a]
69
(0)
(d)
1.2
0.8
Deesity
0.3
0.0
0.2
0.6
0.6
0.2
1.0
0.0
0.8
9.8
0.8
1.0
0.0
0.2
0.8
0.6
00.5
3.0
03
0.2
0.6
0.6
0.5
1.0
DMAP Samples
Figure 19: Illustrative DMAP histograms. Plots of XSum data (Narayan al., 2018) generated by OPT-2.7b, evaluated by OPT-2.7b. The generation strategies (left to right) are (a) pure sampling, (b) top-p 0.8 sampling, (c) temperature 0.8 sampling, and (d) top-k 50.
1.2
Density
0.8
0.4
0.0
DMAP Samples
Figure 20: Illustrative DMAP histograms. Plots of XSum data (Narayan et al., 2018) generated by OPT-1.3b, evaluated by OPT-1.3b. The generation strategies (left to right) (a) pure sampling, (b) top-p= = 0.8 sampling, (c) temperature т 0.8 sampling, and (d) top-k = 50.
The images here further support the claim in the main text that pure-sampled text looks flat when the generator and evaluator models are the same, and looks heavy tailed (tail bias) when another base model is used for evaluation.
LLM USAGE
LLMs were used to polish written text in the preparation of this manuscript.
27

---

[e]
[5)
Decatty
0.03
0..
0.3
0.0
Samples
Figure 21: Illustrative DMAP histograms. Plots of XSum data (Narayan et al., 2018) generated by OPT-350m, evaluated by OPT-350m. The generation strategies (left to right) are (a) pure sampling, (b) top-p 0.8 sampling, (c) temperature 7 = 0.8 sampling, and (d) top-k 50.
(8)
1.0
1.0
0.6
0.5
0.5
Density
0.55
0.2
0.2
0.0
0.6
DMAP Samples
Figure 22: Human-written text four categories: abstracts (a), books (b), news (c), and poetry (d).
LJama-3.1-80 upt-125m
Mistral-7B-x0.3 opt-125m
Palcon3- 7B-Base, opt-125m
Qwea3-6B-Bens opt-125m
1.2
1.2
1.0
1.2
1.0
1.0
0.8
Denslty
0.8
0.6
0.8
0.4
0.4
0.4
0.4
0.2
0.2
0.0
0.0
0.0
DMAP Samples
Figure 23: Llama 3.1 8B, Mistral 7B, Falcon and Qwen3 8B generated XSum data, evaluated by 025m..5m.
28


Reviews:
[{"review_id": "3IkJysO0SZ", "reviewer": "Reviewer_BUmF", "summary": "The paper introduces DMAP, a mathematically grounded method to map next-token distributions from an LLM into samples on the unit interval, producing a unified representation that captures both rank and probability information. The authors show DMAP can (i) validate generation parameters, (ii) highlight weaknesses in machine-generated text detectors based on probability curvature, and (iii) reveal statistical fingerprints from instruction-tuning, especially when synthetic data is used.", "questions": "Please refer to the discussion in weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is easy to read. \n\n- The paper has discussed mathematical connections and also provided empirical results to support the claims made. \n\n- This work provides some fresh ideas of going beyond perplexity while using LLMs to evaluate a given text. \n\n- The paper presents nice visualization to explain the ideas.", "weaknesses": "- As a reviewer, I am confused about the problem setting in introuction, specifically the discussion of \"contextualization\nproblem \", and why it is important. \n\n- Following to that, the connection to AI generated text detection seems vague and not clearly established, and then authors introduce the DMAP algorithm without motivating for the need for it. This raises concerns about the novelty and not clear what is the key contribution of this work.\n\n-  The authors talk about rannk being higher of v, but in the mathematical notation, it is lower? Am I missing something? \n\n- The authors have mentioned about the limitaitons of DMAP and then proposed entropy weighted DMAP, this section discussion is confusing, is it possible to empirically highlight or show the limitations to make it clear what is being addressed with entropy weighted DMAP? \n\n- As discussed in Table 1, it isnot clear why one would not use top-k sampling in practice? Ad what exact DMAP is predicting for these experiments? \n\n- Can author comment more about the connection with uncertainty quantification literature?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VqPh2XbCyi", "reviewer": "Reviewer_bpqg", "summary": "This work presents DMAP that maps texts onto a distribution in the unit interval under a reference evaluating LLM, which provides both a direct visualization and quantifiable metrics for texts against the LLM, enabling identification of LLM-generated texts, validation of generation parameters, and the analysis of downstream LLMs.", "questions": "Q1. What, if anything, is stopping DMAP from developing into a tool for machine-generated text detection? \n\nQ2. Why is the entropy directly used to weight the step functions at different tokens for Section 3.2's definition of DMAP? Did you consider other approaches rather than clipping, such as using the entropy of top-K tokens? \n\nQ3. Figure 4 sees a heavy \"tail\" around the far end of the interval at 1.0. If I understand correctly, this corresponds to highly unusual tokens according to the validating LLM. Can you explain this further? Is this a common artifact of fine-tuning?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The main proposal, DMAP, is well-grounded, intuitive, innovative, and immediately useful. \n\n2. The effectiveness of DMAP is demonstrated through both comprehensive visualizations and analyses, and a good suite of downstream problems backed by solid experiments. \n\n3. The authors points out a crucial problem with current state-of-the-art machine-generated text detection methods, and demonstrated DMAP's power in its identification.", "weaknesses": "To clarify, despite these weaknesses, I still think the paper has significant contributions to recommend acceptance. I'd like my review & subsequent discussions with the authors to work towards improving the paper quality. \n\n1. As an antithesis to DMAP's ability to validate model sampling decoding strategies, DMAP is sensitive to decoding strategies and may not work as well if the temperature / top-K / top-p parameters change. This also may hinder DMAP's ability of becoming an effective tool for actual machine-generated text detection. I suspect there are ways to improve this, but it would require extensive additional experiments, especially more studies of DMAP on human-generated texts. \n\n2. While the idea of DMAP becomes simple once understood, the paper could do a better job presenting the mathematical formulation. Here are some specific pointers: \n    1. The introduction of $V_i^a$ and $V_i^b$ make the presentation bloated and unintuitive, and there should be some way around it. It may also be helpful if the intuitive explanation on Lines 155-157 are moved to the front, so the technical passage become easier to read (this is my recommended principle for writing in general). \n    2. I fully understand the motivation of Section 3.2, and why it's a natural extension of the discrete, unweighted version of DMAP in Section 3.1. However, it will be helpful if, before the technical definitions are given, a quick explanation of the two things being done is given on Line 187, for example describing the extension of a single, uniformaly sampled point in an interval to a more \"quantum\" approach of keeping the interval intact as a step function, and the entropy weighting technique. \n    3. Proposition 3.1 mentions pure sampling without first defining it. \n    4. The example of temperature $\\tau$ in Lines 172-176 can also end with \"sampling from $p$ with temperature $\\tau$\" instead of \"sampling from $p$ with the appropriate decoding strategy\". \n\n3. There's currently no analysis on the efficiency and sample complexity of DMAP, theoretical or empirical. \n\n4. The visualization in Figure 1 is an important direct illustration that almost immediately explains the mechanism of DMAP. However it's too small to read and could use some more work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3t1h8y9ZRx", "reviewer": "Reviewer_soHE", "summary": "The paper proposes DMAP, a simple yet effective algorithm that maps natural language text into a probability distribution on the [0,1] interval using a language model. DMAP provides an intuitive statistical visualization of how closely a text aligns with the model's generation configuration. The authors further demonstrate three possible applications of DMAP through concrete examples, supported by theoretical analysis or experimental evidence, to present its versatility and potential.", "questions": "Please check my questions in the Weaknesses section.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper introduces a clear and effective mapping algorithm that projects text into a probability distribution over [0,1] via a language model, enabling intuitive visualization and analysis.\n2. The authors explore three potential applications of DMAP, each supported by either experimental results or theoretical analysis.", "weaknesses": "1. The core objective of DMAP is similar to the well-known Probability Integral Transform (PIT) used for model calibration [1–3]. Both map predicted probabilities to the [0, 1] interval and yield a uniform distribution when predictions are perfectly aligned with the model. DMAP can be interpreted as an LLM-specific extension of PIT that replaces i.i.d. sampling with conditional sampling and adds an entropy-weighting. The authors should explicitly discuss this connection and compare DMAP against a simple PIT-based baseline to clarify what additional insight DMAP provides in the context of LLM.\n\n2. The paper claims that existing metrics such as log-likelihood and entropy fail to reflect \"context\" of the next-word prediction, yet these metrics already capture contextual information through the softmax normalization over the full next-token prediction's distribution. DMAP does not introduce new contextual information beyond this. Instead, it re-expresses and visualizes the same distributional information through a conditional sampling perspective. The authors should clarify how their notion of context differs from that already implicit in standard probability-based metrics.\n\n3. DMAP effectively visualizes the inherent limitation of curvature-based AI-text detection methods: AI-generated text tends to be more \"self-consistent\" with the scoring model than human text, but this assumption breaks when the source model or decoding parameters differ substantially. While the paper illustrates this limitation, it does not demonstrate how DMAP may mitigate it as the authors claim in the paper. The authors should show how DMAP could improve detection robustness under cross-model and cross-config scenarios by applying DMAP to existing curvature-based AI-text detection methods empirically.\n\n4. The paper does not specify how the reliability of DMAP visualizations changes with the input length. Since DMAP samples one value per token, short sequences may yield unstable or misleading results. The authors should provide theoretical or empirical guidance on the minimal input length required by DMAP, given the model and decoding parameters used.\n\n\n[1] Fisher, Ronald Aylmer. Statistical methods for research workers. No. 5. Oliver and Boyd, 1928.\n\n[2] David, F. N., and N. L. Johnson. \"The Probability Integral Transformation When Parameters Are Estimated from the Sample.\" Biometrika, vol. 35, no. 1/2, 1948, pp. 182–90.\n\n[3] Gneiting, Tilmann, Fadoua Balabdaoui, and Adrian E. Raftery. \"Probabilistic forecasts, calibration and sharpness.\" Journal of the Royal Statistical Society Series B: Statistical Methodology 69.2 (2007): 243-268.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QA2SfnFmIm", "reviewer": "Reviewer_5pHM", "summary": "This paper presents a straightforward method for visualizing a probability distribution on the unit interval which measures how the tokens in a piece of text relate to the token distribution of an evaluator language model. The visualization method has the property that if the same model M is used both to sample the piece of text, and to generate the visualization, then a uniform distribution should be produced. Furthermore, standard statistical tests of uniformity can be used to confidently reject the hypothesis that a given text dataset was produced by a particular model.\n\nThe authors then use this visualization method to point out an error in a significant line of work on detecting LLM generated text. In particular, this line of work inadvertently used top-k sampling when generating text, which clearly produces a bias towards more likely tokens. The property of bias towards more likely tokens was then proposed as a method for LLM generated text detection. The visualization method of the submission can be used to immediately see that this property is true for top-k sampling, but not for standard sampling. This implies that standard sampling can easily bypass these prior detection techniques. The method can also be used to as a straightforward statistical validation step, to verify the sampling model and settings (temperature, top-k, nucleus sampling etc) for any given text dataset.", "questions": "Does the method still at least give interesting empirical results without access to the prompt, but only the model response? Evidence in this direction would cause me to increase my score.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed method is straightforward and the visualizations are easy to understand. Most importantly, the method was used to identify a serious error in prior work on LLM generated text detection. The identification of the error alone seems to me a valuable contribution. I am not an expert in the LLM-generated text-detection research area, and so would defer to other reviewers if they have additional insight into the significance of this result.", "weaknesses": "Beyond the particular case of finding the error in prior work on LLM text detection, it is unclear how useful this method will be. In particular, the method is rigorous and clearly useful for saying that a particular text was *not* generated by an LLM, but it is unclear how robust this method is. For example, if the prompt used to generate the text is not available (as it generally is not when attempting to detect LLM generated text) it is unclear how this method will perform. The rigorous theoretical results do not hold without access to the prompt, and it is very unclear whether the empirical results will either.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)