SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In our investigations, the GPT2-x1 model consistently outperforms its small counterpart among statistics-based and language modeling metrics as all relevant "vs." columns indicate, apart from the Coherence in the Stories domain. In the GPT2 experimental group, it is astonishing that the small model always performs better when referring to the voting results from MAUVE and FACE rows. Across three task domains, the performances of OPT and BLOOM models in two sizes differ. Large models have better overall performance, and small models only win four out of twelve comparisons by voting. Nonetheless, it is noteworthy that four FACE metrics we proposed maintain a relatively high level of consistency with MAUVE across all models. At least two FACE metrics yield the same results (in eight out of nine sets of human-model language comparisons) with MAUVE. Concretely
Figure 3: FACE-SO scores on OPT, BLOOM and GPT2 original output data. Model sizes compared: small vs. large for OPT and BLOOM; -sm vs. -xl for GPT2. Error bars represent  $95\%$  confidence intervals from bootstrap. The significant levels are based on  $t$ -test between the two model-size groups.
Figure 4: FACE scores (conditional generation) on original experimental data of [19] and [26]. Nine sampling methods are compared: greedy, beam search, stochastic beam search, pure sampling, temperature, top-  $k$ , top-  $k$  with temperature, nucleus, and contrastive. Note that logarithmic normalization on parameter values as well as enlarged markers for greedy decoding, pure sampling, and contrastive decoding are adopted for better visualization effect. Best viewed when zoomed in.
speaking,  $SO$  and  $SAM$  show a higher positive correlation to MAUVE than CORR and SPEAR, given that seven out of nine voting results (marked with yellow in Table 3) are identical.
To further evaluate model sizes, we apply FACE to the original GPT2 output data (webtext)  $^{4}$  generated from GPT2-sm and GPT2-x1. GPT2-x1 has a higher SO score than GPT2-sm, which is confirmed with the  $t$ -test, but non-significant effects are found on the other three metrics. Combining our generation task with the original GPT2 data, we illustrate the results for SO in Figure 3.
To conclude, we discover three keypoints: (1) FACE is consistent with MAUVE in evaluating three different model types (two sizes for each); (2) the metrics estimating similarity between human-written and model-generated text generations (e.g., FACE, MAUVE) may produce opposite results to the text-centered metrics (e.g., Diversity, Coherence); (3) the four metrics of FACE show relatively homogeneous results, and using these metrics together helps to identify model-generated texts with a more comprehensive evaluation.
4.2 Sampling methods
Recent work [19, 26] has indicated three clear trends in open-ended text generation using autoregressive LMs: (1) maximization-based decoding algorithms (e.g., beam search, greedy decoding, etc.) lead to copious repetition, while sampling with temperature may result in incoherence; (2) truncation-based sampling methods like nucleus sampling produce text with higher quality; (3) contrastive decoding outperform nucleus sampling in terms of both fluency and coherence. Accordingly, to demonstrate the effectiveness of our approach, FACE should follow the inequality: maximization-based/temperature-based  $\prec$  nucleus  $\prec$  contrastive in terms of the quality relationship.
Figure 4 visualizes the correlation between FACE scores and various decoding algorithms. The contrastive decoding approach yields the best performance among the four FACE metrics. It can be clearly observed that the maximization-based sampling methods behave worse than other algorithms. Moreover, adding the temperature parameter to top- $k$  sampling results in incoherent text generations,
<sup>4</sup>https://github.com/openai/gpt-2-output-dataset
Sampling Method	Perplexity	Self-BLEU	Zipf Coefficient	Repetition	SO (↑)	CORR (↑)	SAM (↓)	SPEAR (↑)
Human	12.38	0.31	0.93	0.28	-	-	-	-
Greedy	1.50	0.50	1.00	73.66	0.20	0.56	0.31	0.04
Beam (b=16)	1.48	0.44	0.94	28.94	0.21	0.31	0.40	0.04
Stochastic Beam (b=16)	19.20	0.28	0.91	0.32	0.37	0.49	0.33	0.04
Pure Sampling	22.73	0.28	0.93	0.22	0.41	0.63	0.28	0.03
Sampling (t=0.9)	10.25	0.35	0.96	0.66	0.42	0.61	0.29	0.03
Top-k (k=40)	6.88	0.39	0.96	0.78	0.40	0.64	0.28	0.03
Top-k (k=640)	13.82	0.32	0.96	0.28	0.42	0.63	0.28	0.03
Top-k (k=40, t=0.7)	3.48	0.44	1.00	8.86	0.34	0.61	0.29	0.03
Nucleus (p=0.95)	13.13	0.32	0.95	0.36	0.42	0.63	0.28	0.03
Contrastive Decoding	14.39	0.54	1.04	0.24	0.44	0.75	0.23	0.17
Table 4: Results for comparing all sampling methods with selected parameters regarding the conditional generation. The values closest to human scores are **bolded**, except for our proposed FACE scores, where the highest (for SO, CORR, and SPEAR) or the lowest (for SAM) values are in **bold**.
which explains the gap between the red curve (top-  $k$  w/o temperature) and the gray curve (top-  $k$  w/ temperature). We also plot the correlation graphs of unconditional generation (in the Appendix) with fewer sampling methods involved. The trends and patterns in the visualization of unconditional generation are basically consistent with its conditional counterpart.
In Table 4, FACE scores on different decoding algorithms are summarized. FACE metrics correctly match the expected quality relationship of the sampling methods examined by assigning the best  $SO$  (.44),  $CORR$  (.75),  $SAM$  (.23), and  $SPEAR$  (.17) scores to contrastive decoding. Other evaluation metrics fail to capture the correct relationship, for example, the perplexity rates nucleus-sampled text as better than contrastive-decoded text, which is irrational suggested by Li et al. [26].
4.3 Human judgments
We also explore the correlation between FACE and human judgement scores, using the crowd-source dataset collected in [31] when human evaluation is available. The dataset contains model-generated continuations (by GPT2-sm, -md, -lg, and -xl with ancestral and nucleus sampling), human-written continuations using the same prefix, and the crowd-source workers' answers on which completion is more human-like, interesting, and sensible. We follow the same experimental settings and protocol to verify whether the FACE scores of the text completions correlate well with the human quality judgements by computing the Spearman's rank correlation coefficient. The results are presented in Table 5.
We observe a high and positive correlation between FACE-SO and human judgments scores, which outperforms five out of the six evaluation metrics reported in [31] and achieves a comparative performance against MAUVE. The remaining three FACE metrics have insignificant correlations. However, we consider human judgments to be subjective and sometimes biased. Including more fine-grained questions to perform human judgments may lead to more accurate correlation statistics. Additionally, we recomputed the correlations with human judgement scores to keep those pairs in which there are exactly one item from human and the other item from model (i.e., a subset of data used for the analysis in Table 5). As shown in SO-S and MAUVE-S columns, FACE-SO has a stronger correlation than MAUVE among two of these three dimensions.
4.4 Sanity tests
Sanity test on validity. We evaluate the validity of FACE by examining whether its scores on human-human split is lower than human-model groups – an expected result based on our assumption that the spectral difference between human's "natural" language and models' "artificial" ones should be amplified by FACE. Therefore, the sanity tests are conducted as follows: first, we evenly and randomly split the human data into two folds (across three domains) to serve as control groups. The FACE scores between these control folds are then computed. As for the human-to-model experimental group, we create other two folds using human data and the best model-generated data (from contrastive decoding [26]) in terms of text quality. If FACE can effectively capture the fundamental difference
Metric	Generation Perplexity	Zipf Coefficient	Repetition	Distinct-4	Self-BLEU	SO	MAUVE	SO-S	MAUVE-S
Human-like/BT	0.810	0.833	-0.167	0.738	0.595	0.881	0.952	0.357	0.214
Interesting/BT	0.643	0.524	-0.143	0.524	0.405	0.762	0.810	0.524	0.667
Sensible/BT	0.738	0.690	-0.071	0.595	0.524	0.786	0.857	0.995	0.706
between human and model languages, then we expect to observe higher scores in control groups than in the experimental group. The results are shown in Table 6.
Table 5: Spearman's rank correlation coefficients of  $SO$  and five other metrics with human judgments. Higher scores mean better correlation. All the numbers except the  $SO$ ,  $SO-S$ , and MAUVE-S columns are sourced from [31]. "BT" denotes the Bradley-Terry score of the pairwise human evaluation, which is employed to compute the Spearman's rank correlation with the scores of other metrics. Additionally, it is important to note that the original human judgments encompass certain pairs in which both texts are generated by models, albeit different models. Therefore, we refine the original human judgment dataset to only include judgments involving both human and model-generated languages, and the results are shown in  $SO-S$  and MAUVE-S.
SO (↑)	CORR (↑)	SAM (↓)	SPEAR (↑)
h-h (wiki)	0.45	0.76	0.22	0.05
h-h (news)	0.45	0.76	0.22	0.07
h-h (stories)	0.47	0.79	0.21	0.05
h-m	0.44	0.75	0.23	0.17
Table 6: Results of sanity test on FACE's validity. The top three rows are the control groups, and "h-h" stands for human-to-human folds. The last row is the experimental group, where "h-m" is for human-to-model fold. Better FACE scores are in bold. The scores in the bottom row are retrieved from Table 4.
It can be seen from Table 6 that the control groups show significantly better FACE scores than the experimental group: FACE-SO and FACE-CORR are higher in human-to-human folds, while FACE-SAM scores are lower. The only exception is FACE-SPEAR, while we will show it is a good metric in later sections. Nonetheless, these tabulated results have proved the validity of FACE in effectively capturing human-to-model spectral differences.
Choice of estimator model. We examine how different choices of the estimator model  $m_{\mathrm{est}}$  affect the resulting spectra, using GPT2-sm, -md, -lg and -xl as  $m_{\mathrm{est}}$ , respectively. The spectra of webtext and the original GPT2 output data are computed. It is found that the spectra obtained from  $m_{\mathrm{est}}$  have different magnitudes, but their aggregated curves have the same shape (see Appendix). Therefore, the choice of  $m_{\mathrm{est}}$  will not affect FACE scores as long as the same  $m_{\mathrm{est}}$  is used for all data.
Stationarity tests. One of the assumptions of the Fourier transform is that the signal is stationary [21], i.e., the mean and variance do not change over time. We applied the Augmented Dickey-Fuller (ADF) test [8] to examine the stationarity of the cross-entropy sequences for all the human and model-generated data used in this study. The null hypothesis  $H_0$  of the ADF test is non-stationarity, and thus a  $p < .05$  testing result rejects  $H_0$  and accepts the alternative hypothesis of stationarity in the series. We calculate the proportions of cross-entropy sequences that pass the ADF test with  $p < .05$  for all model-generated and human data:  $97.4\%$  for GPT2,  $92.1\%$  for OPT,  $74.5\%$  for BLOOM, and  $97.9\%$  for human. Thus, the majority of data meets the stationarity requirement.
4.5 Interpretation of spectra
As the frequency spectrum reflects the key characteristics of a signal, we attempt to interpret the spectra to see if they tell how the "signals" – entropy of human and machine languages – differ. Without aggregation, the raw spectra of single cross-entropy sequence look indistinguishable between GPT2-sm, GPT2-x1, and human (see the left plot in Figure 5). By aggregating 5,000 spectra from each group and smoothing the curves, it can be seen that GPT2-x1's curve is closer to human than the GPT2-sm curve (readers can find this by zooming in the middle plot in Figure 5). Here, the smoothing is done with generalized additive models (GAMs) [45]. Results from other models are included in the Appendix.
Figure 5: Intuitive observations on the spectra from GPT2 and human data (webtext). Left: Spectra of three randomly sampled entropy sequences from GPT2-sm, GPT2-x1, and webtext. Middle: Smoothed plot of 5,000 aggregated spectra with absolute values,  $|X_{\omega_k}| \sim \omega_k$ . Right: Typical smoothed plot of raw spectra  $X_{\omega_k} \sim \omega_k$ , with peaks and troughs annotated.
When plotted separately, the aggregated spectra from human and different models have similar shapes: First, the majority of components exist in the low-frequency range  $(\omega < 0.05)$ . In addition, the locations of peaks and troughs are almost the same between groups. For instance,  $\omega_{1} = 0.06$  is the first trough, and  $\omega_{2} = 0.12$  is the first peak (see the right plots in Figure 5). Thus, roughly speaking, the main difference between human and model spectra is not in the locations of peak and trough frequencies but in the relative magnitudes of those frequencies.
We propose a simple way to interpret the peaks in spectra: the reciprocal of a frequency component  $T_{k} = 1 / \omega_{k}$  denotes the corresponding cycle in the time domain. Because the time interval (i.e., sampling interval) of an entropy sequence is not measured in seconds but fixed as one token, the measurement unit of  $T_{k}$  is also in number of tokens. For example, the first frequency peak in Figure 5 (right plot) implies  $\omega_{2} = 0.12 \Rightarrow T_{2} = 1 / 0.12 \approx 8.3$  (tokens), which approximately means that tokens of the same cross-entropy levels tend to recur every 8.3 tokens. This pattern is consistent in both human and model data. However, the degree of this recurrence can mark the difference between the human and model languages. We leave more detailed interpretations of spectra to future work.
We propose FACE, a set of metrics based on the Fourier analysis of cross-entropy, which is able to distinguish human and model-generated language with satisfactory performance in the open-ended generation task. The metrics scale with model sizes; reflect the effect of various sampling methods; correlate well with other existing metrics and outperform most of them in alignment with human judgement scores. Among the four implementation methods of FACE experimented, Spectral Overlap (SO) has the best overall performance.
FACE is computationally efficient with easy-to-interpret output. As a method inspired by psycholinguistic studies on the predictability (entropy/surprisal/information density) of human language, we believe FACE is a good example of incorporating knowledge from different fields for better human-centered AIs. We can generally conclude that better language models can produce spectral representations of information that are more similar to human.
Our current work has several limitations: Firstly, for open-ended generation experiments (Section 4.1), a broader set of sampling methods other than top- $k$  can be used. Secondly, larger models (with more than 100 billion parameters) need to be included for more comprehensive comparisons. We will improve from these aspects in future work.
This material is based upon work supported by the National Science Foundation under Grant No. (2105192).
[50] T. Zhang*, V. Kishore*, F. Wu*, K. Q. Weinberger, and Y. Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SkeHuCVFDr.
[51] W. Zhao, M. Peyrard, F. Liu, Y. Gao, C. M. Meyer, and S. Eger. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 563-578, Hong Kong, China, Nov. 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1053. URL https://aclanthology.org/D19-1053.
[52] Y. Zhu, S. Lu, L. Zheng, J. Guo, W. Zhang, J. Wang, and Y. Yu. Texygen: A benchmarking platform for text generation models. In The 41st international ACM SIGIR conference on research & development in information retrieval, pages 1097-1100, 06 2018. doi: 10.1145/3209978.3210080.
1 Broader Impacts
FACE measures the distance between human and model-generated languages, therefore it is technically possible to be used for designing or augmenting systems that mimic humans. We acknowledge the risks of FACE (and other metrics) being utilized in applications that deliberately confuse human-authored and model-produced text. We call for the collective efforts from the community to come up with a systematic framework that unifies different metrics, for developing more reliable and natural language generation systems.
2 Implementation Details
Preprocessing. We utilize three raw datasets: WritingPrompts, WikiText-103, and RealNews. For WritingPrompts, the prompt set has already been well-curated, so we just extracted the first 5,000 prompts (the length may vary) for our generation task. WikiText-103 and RealNews contain many complete texts. For each complete text, we further truncate it corresponding to the first 35 tokens as a prompt. To fairly evaluate the performance of metrics, we also divide text generations according to five predefined length (from 0 up to 1024) intervals for each dataset. Thereby, the human-written texts and model-produced texts used to evaluate the performance of metrics may be generated by different prompts (i.e., unpaired comparison).
Hyper-parameters. We have several hyper-parameters during the text generation and evaluation phases. For both conditional and unconditional generation, we preset a random seed integer (32 by default). Furthermore, the maximum length of each text (1024 by default) as well as the batch size (which varies according to GPUs capacity) for perplexity computation have to be determined before automatic evaluation.
3 Miscellaneous Details
Software. Our experiments were performed on Ubuntu 20.04.1 system with Python 3.9.16. The versions of key Python libraries include: Transformers 4.27.4, PyTorch-CUDA 11.6, PyTorch 1.13.1, Scipy 1.5.4.
Hardware. For the text generation task, we use the remote workstation that has two NVIDIA RTX A6000 graphics cards. It should be noted that all models were run in parallel when available.
Computation time for text generation. We spent 10 and 25 hours or so obtaining 5,000 text continuations by GPT2-sm, -xl, respectively. OPT-125m, -6.7b cost our GPU resources roughly 11 and 44 hours to output the same number of text continuations, respectively. When it comes to BLOOM-560m, -7b, they took approximately 18 and 48 hours, respectively, to generate 5,000 continuations per task domain.
Evaluation time for FACE. Computation time of four FACE metrics for a single pair of references are:  $5.96 \times 10^{-8}$  seconds for SO,  $5.01 \times 10^{-8}$  seconds for CORR,  $4.53 \times 10^{-8}$  seconds for SAM, and  $4.29 \times 10^{-8}$  seconds for SPEAR, respectively. The cross-entropy, which should be calculated beforehand, takes  $5.65 \times 10^{-2}$  seconds. All of the above measurements take place on an AMD Ryzen Threadripper PRO 3995WX 64-Cores CPU (frequency range  $\in [2200.00\mathrm{MHz}, 4308.40\mathrm{MHz}]$ ). Users can leverage more advanced GPU resources to perform the whole computation process with a faster speed.
4 Additional Experimental Results
4.1 Model sizes (generation length)
It should be emphasized that LMs have diverse designs and were pre-trained using different strategies on different datasets, giving them distinct preferences on the generation length. The numbers of text generations in each length interval are summarized in Table 7.
Domain	Length Interval	GPT2-sm	GPT2-xl	OPT-125m	OPT-6.7b	BLOOM-560m	BLOOM-7b
Wiki text	0-200	403	485	964	1522	4928	803
201-400	571	672	888	929	61	599
401-600	251	316	441	417	8	388
601-800	260	310	268	285	1	316
801-1024	3515	3217	2439	1847	2	2894
News	0-200	750	836	844	1119	4978	1371
201-400	1222	1336	1220	1325	20	917
401-600	824	759	1194	939	1	628
601-800	584	678	764	593	0	427
801-1024	1620	1391	978	1024	1	1657
Stories	0-200	549	745	2731	3588	4924	1608
201-400	625	757	715	501	63	688
401-600	296	404	241	176	9	410
601-800	241	324	160	95	4	271
801-1024	3289	2770	1153	640	0	2023
Table 7: Domain-specific generation length with respect to different models (GPT2/OPT/BLOOM) and model sizes (one large model and one small model) using top-  $k$  ( $k = 50$ ) sampling corresponding to five continuous length intervals.
Figure 6: FACE scores of GPT2 (our generated data), OPT, and BLOOM with different model sizes.
To ensure the consistency of our experiments, we run six LMs separately (using their own tokenizers) with the same prompt sets and settings as described in Table 2 to generate 5,000 pieces of continuations in each domain. Besides, we utilize the GPT2Tokenizer to calculate the numbers of continuations for each interval, which allows us to compare FACE scores with other metrics more objectively, as we believe it is unfair to explicitly compare texts of varying lengths. Then, we compute weighted arithmetic mean to evaluate a model in each domain, by  $s' = \sum_{i=1}^{n} \frac{m_i}{M} s_i$ , where  $s'$  denotes the weighted mean;  $n$  denotes the number of length intervals;  $m_i$  is the number of generated continuations in the length interval  $i$ ;  $M = \sum_{i=1}^{n} m_i$ , and  $s_i$  means a certain metric value in the interval  $i$ .
Figure 6 conveys a more intuitive representation (via bar plots) of Table 3.
4.2 Sampling methods (unconditional generation)
We also carried out experiments on unconditional text generation. Here, the prompt is not required as we generate continuations from a random seed (set to 32 empirically). Four sampling methods, which are greedy decoding, beam search, stochastic beam search, and contrastive decoding, are not involved in this set of experiments.
The results are displayed in Figure 7. The overall trends are same as its conditional counterpart, where the previous quality relationship (maximization-based/temperature-based  $\prec$  nucleus  $\prec$  contrastive) is satisfied. Yet, it is crucial to note that the advantages of top- $k$  sampling w/o temperature become more obvious compared to the conditional case.
<sup>5</sup>https://github.com/ari-holtzman/degen
Figure 7: FACE scores (unconditional generation) on original experimental data<sup>5</sup> of nucleus sampling. Five sampling (decoding) methods are compared: pure sampling, temperature, top- $k$ , top- $k$  with temperature, and nucleus. Note that logarithmic normalization on parameter values as well as an enlarged marker for pure sampling are adopted for better visualization.
Model	Sampling Method (parameter)	SO	CORR	SAM	SPEAR
GPT2-x1	Nucleus Sampling (p=0.95)	0.481	0.821	0.191	0.359
Ancestral Sampling	0.472	0.807	0.199	0.331
GPT2-lg	Nucleus Sampling (p=0.95)	0.480	0.819	0.193	0.356
Ancestral Sampling	0.472	0.814	0.196	0.338
GPT2-md	Nucleus Sampling (p=0.9)	0.478	0.815	0.194	0.358
Ancestral Sampling	0.462	0.813	0.197	0.310
GPT2-sm	Nucleus Sampling (p=0.9)	0.476	0.817	0.194	0.359
Ancestral Sampling	0.468	0.816	0.195	0.319
Table 8: FACE results based on MAUVE's original experimental data  ${}^{6}$  .
Table 8 shows the FACE scores based on the output texts from MAUVE. Each column of FACE scores is used to compute the Spearman's rank correlation coefficient between a specific FACE metric and Bradley-Terry scores (4 model sizes  $\times$  2 sampling methods  $= 8$  scores in total) from one criterion (three criteria correspond to three questions in total).
4.4 Choice of estimator model