SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zuhao Yang $^{1*}$ , Yingfang Yuan $^{2*}$ , Yang Xu $^{3*†}$ , Shuo Zhan $^{1}$ , Huajun Bai $^{4}$ , Kefan Chen $^{2}$
$^{1}$  School of Computer Science and Engineering, Nanyang Technological University
$^{2}$  School of Mathematical and Computer Sciences, Heriot-Watt University
<sup>3</sup> Department of Computer Science, Southern University of Science and Technology
4 Genify
Measuring the distance between machine-produced and human language is a critical open problem. Inspired by empirical findings from psycholinguistics on the periodicity of entropy in language, we propose FACE, a set of metrics based on Fourier Analysis of the estimated Cross-Entropy of language, for measuring the similarity between model-generated and human-written languages. Based on an open-ended generation task and the experimental data from previous studies, we find that FACE can effectively identify the human-model gap, scales with model size, reflects the outcomes of different sampling methods for decoding, correlates well with other evaluation metrics and with human judgment scores.
The concept of entropy from Information Theory is broadly applied in Natural Language Processing (NLP) technology and computational linguistic studies. The most notable example is the use of cross-entropy in training and evaluating language models, where the exponentiation of cross-entropy, perplexity, is adopted to measure models' performance in next-word (or masked-word) prediction task. However, low perplexity alone does not guarantee good performance in language generation tasks, which not only depend on model sizes but are also closely related to the sampling techniques used in decoding stage. The complexity of the generation task makes it especially important to have different metrics that can reflect the generation quality from multiple angles. One particular perspective is that the language generated from a good model should have a similar distribution of words/tokens as in the "natural" human language.
Recent advances in psycholinguistics put forward new directions for developing more sophisticated metrics other than Zipf's coefficient. In particular, studies on temporal and spectral patterns in dialogue [7, 47] reveal that cross-entropy (or referred to as surprisal, information density in the psycholinguistics literature) changes periodically in natural language, which points out the potentials of using fine-grained transformation of cross-entropy to quantify the differences in language data (see Section 3 for a detailed review). It motivates the basic idea of this study: Can we effectively quantify the periodical pattern of the cross-entropy, and use it as an indicator to distinguish human and model-generated languages?
We summarize our contributions as follows: 1. We propose a set of metrics based on the frequency spectra obtained from the Fast Fourier Transform (FFT) of the cross-entropy sequences of language data, named FACE (Fourier Analysis of Cross-Entropy). 2. We empirically show FACE's performance on identifying human-model gap and how it scales with model sizes in Section 4.1. 3. We explore
*Equal contribution, in random order
†Corresponding author, email: xuyang@sustech.edu.cn
Figure 1: Overall workflow of this study.
FACE's correlations with sampling methods and human evaluation in Section 4.2 and Section 4.3, respectively. 4. We validate the statistical soundness of FACE in Section 4.4. 5. We discuss an intuitive interpretation of the metrics and how it reflects the characteristics of language use in Section 4.5. 6. Implementation and experiments code are available in this public repository: https://github.com/CLCS-SUSTech/FACE.
2 FACE
The basic idea of FACE is to obtain the spectra of cross-entropy from different data sources (human or models) and compute their similarities. The overall workflow is shown in Figure 1, which we describe in five steps:
1. Collect the datasets for human-written and model-generated texts,  $\mathcal{D}_h$  and  $\mathcal{D}_m$ .
2. Use a third pre-trained language model  $m_{\mathrm{est}}$  to estimate the cross-entropy of text in  $\mathcal{D}_h$  and  $\mathcal{D}_m$ , resulting in two sequences of cross-entropy output,  $\mathcal{E}_h$  and  $\mathcal{E}_m$ .
3. Obtain the frequency spectra for each cross-entropy sequences,  $\mathcal{E}_h\Rightarrow \mathcal{F}_h$  and  $\mathcal{E}_m\Rightarrow \mathcal{F}_m$
4. Develop FACE metrics that quantify the spectral similarity between  $\mathcal{F}_h$  and  $\mathcal{F}_m$ .
5. Evaluate FACE on different model types/sizes, sampling methods, and the correlations with other metrics for Natural Language Generation (NLG).
We describe the steps in detail from Section 2.1 to Section 2.3.
2.1 Estimate cross-entropy
We use a pre-trained language model  $m_{\mathrm{est}}$  as the estimator for cross-entropy, which runs in the evaluation model (no gradients produced). It takes as input a sequence of  $T$  tokens,  $[t_1, t_2, \dots, t_T]$ ; for each position  $i = 1, \dots, T$ , it predicts the probability of the next token  $P(t_{i+1} | t_1, \dots, t_i)$ ; the cross-entropy between this probability and the ground truth token  $t_{i+1}$  is then computed, resulting in the cross-entropy sequence that consists of  $T - 1$  real values  $\mathcal{E} = [c_1, c_2, \dots, c_{T-1}]$ , as the first token is not predicted:
$$
\mathcal {E} = \left[ c _ {1}, c _ {2}, \dots , c _ {T - 1} \right] \triangleq \left[ - \log P \left(t _ {2} \mid t _ {1}\right), - \log P \left(t _ {3} \mid t _ {1}, t _ {2}\right), \dots , - \log P \left(t _ {T} \mid t _ {1}, t _ {2}, \dots , t _ {T - 1}\right) \right] \tag {1}
$$
Note that  $\sum c_{i} = -\sum_{i=2}^{T} \log P(t_{i}|t_{1}\dots t_{i-1})$  is exactly the definition of negative log-likelihood loss, i.e., cross-entropy loss, for training a language model, where  $c_{i}$  is the negative logarithm of the predicted probability for each token  $t_{i+1}$ . In psycholinguistic studies, this  $c_{i}$  quantity is usually referred to several different terms, including surprisal [16, 17], information density [25, 20, 48], and entropy [13, 14, 46, 47], each of which has a specific theoretical flavor. There have been debates over the justifiability of using "entropy" to denote the negative log-likelihood, because it is not a weighted summation as originally defined in [37]. Albeit, we decide to use cross-entropy as it is the most broadly communicated term and we believe it will not cause confusion as its mathematical form is clearly defined. Apparently, the choice for  $m_{est}$  will influence the next steps, because better
Figure 2: Definitions of four FACE metrics.
language models produce lower perplexity scores, that is, lower cross entropy. Therefore, we discuss how different choices for  $m_{est}$  affect our metrics in Section 4.4.
2.2 Fast Fourier transform
We treat the estimated cross-entropy sequence  $[c_1, \ldots, c_{T - 1}]$  as a finite discrete signal in the time domain, where the sampling interval is approximated with the average duration of one token. With this simplified assumption, we find that the discrete Fourier transform (DFT) is the most suitable spectral analysis tool [39]  $^3$ . The formula for DFT is as follows:
$$
X \left(\omega_ {k}\right) \triangleq \sum_ {n = 0} ^ {N - 1} x \left(t _ {n}\right) e ^ {- j \omega_ {k} t _ {n}}, k = 0, 1, \dots , N - 1 \tag {2}
$$
in which  $x(t_{n})$  is the signal at time  $t_n$ , corresponding to the  $n$ -th cross-entropy value  $c_{n}$  ( $n = 1, \ldots, T - 1$  and  $N \triangleq T - 1$ ).  $X(\omega_{k})$  is a complex number that reflects the magnitude (strength) of the  $k$ -th frequency component  $\omega_{k} = 2\pi k / N$ . In practice, DFT is implemented with an efficient algorithm known as Fast Fourier Transform [5] that runs in  $O(n \log n)$  time.
We compared two methods, periodogram and vanilla FFT. The periodogram approach computes the Fourier transform after applying auto-correlation and time-averaging windows to the signal for de-noising purposes [43]. However, we think de-noising is inappropriate because our "signal" is a time series of cross-entropy, whose value reflects the sampling result at each time step from a large. Auto-correlation or time averaging will remove the distinctiveness of rare tokens. Therefore, we use vanilla FFT and take the real part of  $X(\omega_k)$  to represent the magnitude spectrum for the frequency component  $\omega_{k}$ , which is written as  $X(\omega_{k})$  for brevity.
For an input cross-entropy sequence  $\mathcal{E} = [c_1, \ldots, c_{T - 1}]$  obtained from Section 2.1, the resulting frequency spectrum can be represented as a list of tuples of the same length,  $\mathcal{F} = [\langle \omega_1, X(\omega_1) \rangle, \ldots, \langle \omega_{T - 1}, X(\omega_{T - 1}) \rangle]$ , where  $[\omega_1, \ldots, \omega_{T - 1}]$  are the  $T - 1$  sample frequencies, and  $[X(\omega_1), \ldots, X(\omega_{T - 1})]$  are the corresponding magnitudes.
2.3 Spectral similarity metrics
We develop four metrics to measure the similarity between spectra  $\mathcal{F}_h$  and  $\mathcal{F}_m$ : Spectral Overlap (SO), Spectrum Angle Mapper (SAM) [6], Pearson's correlation (CORR), and Spearman's correlation (SPEAR), as summarized in Figure 2. Before computing the metrics, two spectra  $\mathcal{F}_h$  and  $\mathcal{F}_m$  which are of different lengths  $N_1$  and  $N_2$ , are first interpolated to the same length:  $\mathcal{F}_h \in \mathbb{R}^{N_1} \Rightarrow \mathcal{F}_h' \in \mathbb{R}^{N_C}$ ,  $\mathcal{F}_m \in \mathbb{R}^{N_2} \Rightarrow \mathcal{F}_m' \in \mathbb{R}^{N_C}$ . Here,  $N_C$  is the maximum length of the spectrum in our data. Thereafter, the computation of the subsequent metrics can commence.
Spectral Overlap (SO) is inspired by the power spectrum overlap proposed in [28], which is used in [47] for measuring the spectral similarity between dialogue participants. The frequency magnitudes in  $\mathcal{F}_h^\prime$  and  $\mathcal{F}_m^\prime$  are converted to absolute values, i.e.,  $X(\omega_k)\Rightarrow |X(\omega_k)|$ , and then compute the Area-Under-Curve (AUC) for the interaction  $\mathcal{F}_h^\prime \cap \mathcal{F}_m^\prime$  and the union  $\mathcal{F}_h^\prime \cup \mathcal{F}_m^\prime$ , respectively. SO is defined as the ratio of the two:  $SO = \mathrm{AUC}(\mathcal{F}_h^\prime \cap \mathcal{F}_m^\prime) / \mathrm{AUC}(\mathcal{F}_h^\prime \cup \mathcal{F}_m^\prime)$ . The procedure of converting
<sup>3</sup>https://ccrma.stanford.edu/~jos/sasp/Fourier_Transforms_Continuous_Discrete_Time_Frequency.html
to absolute values is indispensable, since negative values in  $X(\omega_k)$  will result in negative AUCs.  $SO$  has the range [0, 1], and a higher value indicates a stronger resemblance between the two spectra.
Spectrum Angle Mapper (SAM) calculates the angles between  $\mathcal{F}_h^\prime$  and  $\mathcal{F}_m^\prime$ , treating them as two vectors in a space [22]. The angle is measured in radians, which is calculated by the inverse function  $\arccos (\mathcal{F}_h^\prime \cdot \mathcal{F}_m^\prime /\| \mathcal{F}_h^\prime \| \cdot \| \mathcal{F}_m^\prime \|)$ , producing a value within  $[0,\pi ]$ . We understand SAM is equivalent to the cosine similarity score, which is more commonly-used in NLP, but here we just follow the conventions in [22, 2]. A smaller SAM value indicates a greater similarity between  $\mathcal{F}_h^\prime$  and  $\mathcal{F}_m^\prime$ .
Pearson's correlation (CORR) can also be leveraged to measure spectral similarities as discussed in [22].  $CORR = cov(\mathcal{F}_h',\mathcal{F}_m') / \sigma (\mathcal{F}_h')\sigma (\mathcal{F}_m')$ , with a  $[-1,1]$  range. A positive CORR value indicates high similarity (negative for dissimilarity), and 0 indicates weak correlation between  $\mathcal{F}_h^\prime$  and  $\mathcal{F}_m^\prime$ .
Spearman's correlation (SPEAR) [41] is commonly used to assess the monotonic relationship between the comparison and reference groups and to capture the presence of non-linear associations between the two. It has not been used for spectral similarity to the best of our knowledge, but we test it in our experiments. SPEAR also has the range  $[-1, 1]$  with meanings similar to CORR.
Entropy as a metric in psycholinguistics. The entropy of human language has long been a research interest in computational linguistics and psycholinguistics. The entropy of written text is estimated with the average per-word negative log-probability in sentences, and then used to validate the principle of entropy rate constancy (ERC) in human language [13, 14]. Similar studies were conducted in dialogue [46, 32]. Entropy is also defined in probabilistic grammars to describe the capacity of a language [40], and is used to develop complexity metrics to measure the cognitive load of processing syntactic expressions [16, 24, 17]. In the line of work on language production, a different term information density with the same mathematical formulation is used instead of entropy. It is found that speakers reduce syntactic complexity when the information density (or entropy) is high [25, 20]. In parallel with the concept of ERC, this line of work summarizes the tendency of distributing information evenly in human language with the term uniform information density (UID), which is commonly used as a equivalent term as ERC, for example, in [48, 15]. In conclusion, entropy is commonly used as a metric for essential properties of human language.
Periodical change of cross-entropy in language. We draw inspiration from the following studies about the distribution of information in dialogue. Humans are sensitive to the peaks and troughs of entropy in speech, with evidence from human-system dialogues and crowd-sourced ratings from human judges [7]. The entropy of utterances from two speakers converge towards each other within the scope of topical segments in spontaneous dialogues [48]. They measure the entropy of utterances from two participants of a task-oriented dialogue, and have found that the frequency domain features – power spectrum overlap and phase delay – are useful predictors of task outcomes. Both works reviewed above suggest that the periodical up-and-downs of entropy are commonly observable in the human language. It naturally leads to the question of whether and to what extent model-generated language aligns with this empirical finding.
Automatic measures for text generation. Entropy and its related variants have already used as a metric for evaluating generated text, for instance, entropy provides good visualization for the difference between GPT2-generated text and human written ones [12]. Other than entropy, there is a rich body of existing metrics targeted on discriminating human-written text and model-generated text, which we summarize in three branches: (1) statistics-based; (2) language modeling; (3) reference-based. Table 1 gives a brief summary of these three categories, as well as our proposed frequency-based FACE.
Statistics-based measures compare the model-generated distribution  $M$  with respect to the human-written distribution  $H$  in terms of some statistic. The Zipf coefficient [30] is used in [19] to describe the distribution of word frequencies in text. Self-BLEU [52] is derived by calculating the BLEU [29] score for each generated text utilizing all other generations as references. Repetition measures the sequence-level degree of repetition on the basis of the percentage of duplicated n-grams in the generated continuations  $x_{\mathrm{cont}} \sim M$  [44]. Meanwhile, we aggregate the 2-gram, 3-gram, and 4-gram repetition rates to evaluate the lexical diversity in an inverse manner.
Type	Metric	Measure	Definition/Approximation
Statistics	Zipf Coefficient	Unigram rank-frequency statistics	-
Self-BLEU	N-gram diversity	-
Repetition	Sequence-level percentage of repetition	1 - |unique n-grams (xcont)| / total n-grams (xcont) |
Diversity	Inverse of n-gram repetition rates (n = 2, 3, 4)	Πn=24(1.0 - Repetition)
Language Modeling	Perplexity	Evaluation-set perplexity	EH[log M(x)] / EMB(xpre) · EMB(xcont) / ||EMB(xpre)|| |
Coherence	LM quality (cosine similarity between sentence embeddings)
Divergence Curve	MAUVE	Quality & diversity via the divergence frontiers	C(H, M) at all λ ∈ (0, 1) [31]
Frequency Domain	FACE (this work)	Quality & diversity via the spectral similarities (four metrics)	FACE(Fh, Fm)
Human Judgment	Bradley-Terry Score	Human preference via the pairwise evaluation	P(i beats j) = 1/(1+e-(βi-βj)/100)
Table 1: Summary of metrics (automatic & non-automatic) we employed for evaluating open-ended text generation. FACE provides a way to approximate the human-model gap in the frequency domain.
Language modeling metrics measure how un(certain) human text  $\pmb{x} \sim H$  follows the model distribution  $M$ , using the probability distribution  $M(\pmb{x})$ . In our work, the perplexity is calculated upon the set of human texts to quantify how well the distribution  $M$  predicts a text continuation. Coherence is approximated by cosine similarity between the sentence embeddings of prompt  $x_{\mathrm{pre}} \sim H$  and continuation  $x_{\mathrm{cont}} \sim M$  as proposed in [42], where the embedding  $\mathrm{EMB}(\cdot)$  is produced by the pre-trained SimCSE sentence embedding [11]. Metrics under this category never observe model-generated text samples, and hence, they cannot justify how likely  $x_{\mathrm{cont}}$  is under the human distribution  $H$ .
Reference-based measures assess the generated text with respect to a small set of reference text, rather than calculating over the full sequence distributions. Some recent reference-based approaches encompass: (1) [4, 36, 38, 50] aim to capture distributional semantic information in high-dimensional space; (2) [51] concerns Euclidean distance between vector representations of  $n$ -grams and their document frequencies; (3) [31] straightforwardly computes the similarity of one learned distribution from a text generation and the other distribution of human-written text using information divergence frontiers [9, 23, 34]. Reference-based metrics are well-suited for targeted generation tasks (e.g., machine translation). Nevertheless, they become unfavorable in the open-ended generation scenario where multiple reasonable and diverse continuations are preferred.
Non-automatic metrics. Recent works [12, 31, 19, 26] on evaluation metrics and decoding strategies for natural language generation rely on human judgments, assuming that human annotations are the gold standard. Considering the expense of Human Unified with Statistical Evaluation (HUSE) [18], we adopt a pairwise evaluation protocol based on human preferences, to serve as a non-automatic complement of FACE metrics. We leverage the Bradley-Terry model [3] to predict the outcome of a head-to-head comparison given  $n$  players with scores  $\beta_{1},\dots ,\beta_{n}$ .
Task formulation. Given an input text passage as prefix, the open-ended generation aims to produce texts that form a fluent and coherent continuation. More formally, given a sequence of  $m$  tokens denoted  $[x_{1} \ldots x_{m}]$ , as the prompt, the goal is to generate the next  $n$  continuation tokens to form a complete sequence  $[x_{1} \ldots x_{m + n}]$ . The continuation probability at the decoding time by conditioning on the preceding context is defined as:  $P(x_{m + 1} \ldots x_{m + n} \mid x_{1} \ldots x_{m}) = \prod_{i = m + 1}^{m + n} P(x_{i} \mid x_{1} \ldots x_{i - 1})$ , where  $P(x_{i} \mid x_{1} \ldots x_{i - 1})$  is the next-token distribution.
4.1 Model sizes
We consider such a text completion task in three domains: Wiki text, News, and Stories. Intuitively, the generated texts involving different domain knowledge may have different language usages and writing style, which may reflect on metrics. We generate completions from large-scale language models (LMs). In particular, we adopt three representatives of state-of-the-art pre-trained autoregressive LMs: Generative Pre-trained Transformer 2 (GPT2) [33], Open Pre-trained Transformer LMs (OPT) [49], and BigScience Large Open-science Open-access Multilingual LM (BLOOM) [35]. We explore two sizes for each model to illustrate that our FACE metrics generalize across multiple LM families and sizes. Details regarding our task and input data are summarized in Table 2. Different
Domain	Model	Dataset	Prompt Length	Maximum Generation Length	Number of Generations
Wiki text	GPT2/OPT/BLOOM	WikiText-103	35 tokens	1024 tokens	5000
News	GPT2/OPT/BLOOM	RealNews	35 tokens	1024 tokens	5000
Stories	GPT2/OPT/BLOOM	WritingPrompts	varying	1024 tokens	5000
Table 2: Dataset and task summary. In our research, we set the maximum generation length to 1024 for all models on three datasets. Note that the WritingPrompts dataset [10] contains ready-to-use prompts, so the length of prompts varies. For WikiText-103 [27] and RealNews [1] datasets, we cleaned them before extracting the texts corresponding to the first 35 tokens (tokenized by GPT2Tokenizer) to form our prompt sets.
Domain	Metric	GPT2-sm	GPT2-xl	vs.	Voting	OPT-125m	OPT-6.7b	vs.	Voting	BLOOM-560m	BLOOM-7.1b	vs.	Voting
Wiki text	Diversity (↑)	0.733	0.753	L	0.645	0.789	L	0.533	0.732	L
Coherence (↑)	0.595	0.624	L	0.614	0.634	L	0.926	0.819	S
Zipf Coefficient (↓)	0.990	0.975	L	0.989	1.016	S	1.092	0.980	L
Self-BLEU (↓)	0.459	0.424	L	0.423	0.379	L	0.280	0.422	S
MAUVE (↑)	0.677	0.186	S	0.169	0.265	L	0.517	0.184	S
SO (↑)	0.414	0.406	S	0.424	0.436	L	0.426	0.432	L
CORR (↑)	0.806	0.781	S	0.771	0.769	S	0.675	0.789	L
SAM (↓)	0.199	0.213	S	0.216	0.217	S	0.258	0.208	L
SPEAR (↑)	0.022	0.023	L	0.026	0.029	L	0.059	0.023	S
News	Diversity (↑)	0.890	0.897	L	0.853	0.876	L	0.740	0.870	L
Coherence (↑)	0.613	0.640	L	0.663	0.663	S	0.897	0.785	S
Zipf Coefficient (↓)	0.961	0.958	L	0.965	0.968	L	0.964	0.966	S
Self-BLEU (↓)	0.619	0.573	L	0.611	0.543	L	0.384	0.501	S
MAUVE (↑)	0.393	0.281	S	0.162	0.130	S	0.014	0.095	L
SO (↑)	0.424	0.412	S	0.438	0.440	L	0.436	0.437	L
CORR (↑)	0.757	0.723	S	0.746	0.732	S	0.615	0.733	S
SAM (↓)	0.224	0.240	S	0.229	0.236	S	0.281	0.234	L
SPEAR (↑)	0.021	0.019	S	0.017	0.021	L	0.048	0.019	S
Stories	Diversity (↑)	0.743	0.785	L	0.769	0.875	L	0.527	0.830	L
Coherence (↑)	0.421	0.420	S	0.440	0.388	S	0.880	0.660	S
Zipf Coefficient (↓)	1.097	1.085	L	1.021	1.003	L	0.999	1.058	S
Self-BLEU (↓)	0.617	0.565	L	0.587	0.511	L	0.180	0.455	S
MAUVE (↑)	0.504	0.121	S	0.025	0.013	S	0.006	0.008	L
SO (↑)	0.411	0.402	S	0.406	0.405	S	0.350	0.418	L
CORR (↑)	0.813	0.787	S	0.737	0.705	S	0.573	0.772	L
SAM (↓)	0.195	0.209	S	0.231	0.245	S	0.300	0.214	L
SPEAR (↑)	0.023	0.022	S	0.036	0.041	L	0.050	0.027	S
Table 3: Domain-specific generation quality with respect to different models (GPT2/OPT/BLOOM) and model sizes (large model on the left and small model on the right) using top- $k$  ( $k=50$ ) sampling under various existing metrics, as well as proposed FACE metrics.  $\uparrow$  indicates the larger the metric value, the better, whereas  $\downarrow$  indicates the opposite. The vs. column indicates the better-performing model in each comparison, where L/S denotes the large/small model wins and E represents a tie. We applied the majority voting to determine the winner. OPT and BLOOM are postfixed with their number of parameters.
models may generate vastly different numbers of continuations in each length interval (see Appendix). To ensure the fairness of investigating the correlation between FACE and other widely-used metrics with respect to different models (with different sizes), we compute the weighted arithmetic mean for every metric across five length intervals.
The evaluation metrics we are interested in are based on various motivations and principles. Specifically, MAUVE and FACE emphasize the parallels between human and machine-produced texts, as stated in Section 3. Therefore, we group MAUVE together with four FACE metrics. To further obtain intuitive results, we utilize the voting approach to explore the correlations between these metrics on large/small models across three task domains. The results are shown in Table 3.