FACE: Evaluating Natural Language Generation with Fourier Analysis of Cross-Entropy
Authors: Zuhao Yang
Yingfang Yuan
Yang Xu
SHUO ZHAN
Huajun Bai
Kefan Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: natural language generation; evaluation metrics; cross-entropy; language model
TL;DR: Fourier transform of cross-entropy sequences can be used to distinguish human from model-generated language.

Abstract: Measuring the distance between machine-produced and human language is a critical open problem. Inspired by empirical findings from psycholinguistics on the periodicity of entropy in language, we propose FACE, a set of metrics based on Fourier Analysis of the estimated Cross-Entropy of language, for measuring the similarity between model-generated and human-written languages. Based on an open-ended generation task and the experimental data from previous studies, we find that FACE can effectively identify the human-model gap, scales with model size, reflects the outcomes of different sampling methods for decoding, correlates well with other evaluation metrics and with human judgment scores.

FACE: Evaluating Natural Language Generation with Fourier Analysis of Cross-Entropy
Zuhao Yang'; Yingfang Yuan? Yang Xu Shuo Zhan', Huajun Bai", Kefan Chen?
School of Computer Science and Engineering, Nanyang Technological University
2 School of Mathematical and Computer Sciences, Heriot-Wais University
Department of Computer Science, Southern University of Science and Technology Genify
Abstract
Measuring the distance between machine-produce and human language is a critical open problem. Inspired by empirical findings from psycholinguistics on the periodicity of entropy in language, we propose FACE, a set of metrics based on Fourier Analysis of the estimated Cross- Entoroy of language, for measuring the similarity between open-endes generation task and the experimental data from previous studies, we find that FACE effectively identify the modanl gap, scales with model size, reflects the outcomes of different sampling methods for decoding, correlates well with other evaluation metrics and with human judgment scores.
and human- -wrtten languages. Based on
Introduction
The concept of entropy from Information Theory is broadly applied in Natural Language Processing (NLP) technology and computational linguistic studies. The most notable example is the use of cross-entropy training and evaluating language models, where the exponentiation of cross-entropy. perplexity, is adopted to measure models' performance in next-word (or masked-wond word) prediction task. However, low perplexity alone does not guarantee good performance in language generation tasks, which not only depend model sizes but are also closely related to the sampling techniques used in decoding stage. The complexity of the generation task makes it especially important to have different metrics that can reflect the generation quality from multiple angles. One particular perspective is that the language generated from good model should have similar distribution of words/tokens in the "natural" human language.
Recent advances in psycholinguistics put forward new directions for developing more sophisticated metrics other than Zipf's s coefficient. In particular, studies on temporal and spectral patterns in dialogue reveal that cross-entee cross-entropy (or referred as surprisal, information density in the psycholinguistics literature) changes periodically in natural language, which points out the potentials of using ene-grained transformation of cross-entropy to quantify the differences in language data (see Section 3 for detailed review). It motivates the basic idea of this study: Can we effectively quantify the periodical pattern of the cross-entropy, and use as an indicator to distinguish human and model-generated languages?
We summarize our contributions as follows: 1. We propose set of metrics based on the frequency spectra obtained from the Fast Fourier Transform (FFT) of the cross-entropy sequences of language data, named FACE (Fourier Analysis of Cross- -ntropy 2. We empirically show FACE's performance on identifying human-model gap and how it scales with model sizes in Section
3. We explore
"quual contribution, in random order t Corresponding author, email: xuyang xuyang@sustechi wustecch.des..ccc
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Dr
entropy
Spectrum
Human languago
bit
E
A
Fk
Similarity metric FACE (F Fm)
Estimated by a 3ro-party pretraned
Fourler Transform (FFT)
Cross-entropy
Spectrum
M
mm
F
Generated language
Other metrics
Compare with FACE~father I
fother
Figure 1: Overall workflow of this study.
FACE's correlations with sampling methods and human evaluation in Section 4.2 and Section 4.3
respectively. 4. We validate the statistical soundness of FACE in Section 4.4 5. We discuss
an intuitive interpretation of the metrics and how it reflects the characteristics of language use in Section 4.5 6. Implementation and experiments code are available in this public repository:
https:// gi thub. wtw.auu...omm.iieeeemeeooee SULLCS--USTTechFACE
2 FACE
The basic idea of FACE is to obtain the spectra of cross-entropy from different data sources (human models) and compute their similarities. The overall workflow is shown in Figure T which describe in five steps:
Collect the datasets for human-written and model--geneattel texts, Dr, and Dm-
2. Use third pre-trained language model Mest to estimate the cross-entropy of text in Dn, and Dm- resulting in two sequences of cross-entropy output, & and &m-
3. Obtain the frequency spectra for each cross-entropy sequences, &, + Fb, and &m + Fm-
4. Develop FACE metrics that quantify the spectral similarity between F, and F...
5. Evaluate FACE different model types/sizes, sampling methods, and the correlations with other metrics for Natural Language Generation (NLG).
We describe the steps in detail from Section 2.1 to Section 2.3
2.1 Estimate cross-entropy
We use a pre-trained language model mes the estimator for cross-entropy, which runs in the evaluation model (no gradients produced). It takes as input a sequence of 7 tokens, [41, 12, 12,..., br]: for each position 1, ...... T. it predicts the probability cross-entropy between this probability and the ground truth token 44+1 is then computed, resulting in the cross-entropy sequence that consists T real values E [ct, (2,,c,,..... ,T -T- as the first token is not predicted:
the next token P(t+ [f,..... ........ f); the
log P(22(4), log P(ts|1 (2),.... log P(tr|4 22.rl,,,2........
bT-1) (I)
Note that - 2 log P(t:(t1 ... 1 is exactly the definition of negative log- ligllikeldoood loss, i.c., cross-entropy loss, for training language model, where c; is the negative logarithm of the predicted probability for each token f In psycholinguistic studies, this quantity is usually referred several different terms, including surprisal [7], information densiry 25 20 48 and
entropy [13 14 46 47), each of which has a specific theoretical flavor. There have been debates over the justifiability of using "entropy" to denote the negative log-likelihood, because it is not weighted summation originally defined in (37. Albeit, we decide to use cross-entropy as it is the most broadly communicated term and we helieve it will not cause confusion as its mathematical form clearly defined. Apparently, the choice for mest will influence the next steps, because better

---

Spectra: 'h 1R* Fr
Interpolated & absolute: 1g F' hr
Four FACE metrics
2
AUC aN [F"
F'FFF' m (fF (F" m))
SOurCO
SAM
arccos
model
AUC(IF'A al AI
Xire
Cov(F'n..'l (FF'))((F',. m)
6 - (m (w))
CORR
SPEAR
Ne 1)
Figure 2: Definitions of four FACE metrics.
language models produce lower perplexity scores, that is, lower cross entropy. Therefore, we discuss how different choices for mest affect our metrics Section 4.4
2.2 Fast Fourier transform
We treat the estimated cross-entropy sequence [cI...., CT-1 finite discrete signal in the time domain, where the sampling interval is approximated with the average duration of one token. With this simplified assumption, we find that the discrete Fourier transform (DFT) is the most suitable spectral analysis tool [39 3 The formula for DFT is as follows:
N
X(wk)=
x(tn) le juatn h: = N
m=0
in which xz(n)) is the signal at time ln corresponding the n-th cross-entropy value On (n
1 I..., - and N é T 1). X (wx) a complex number that reflects the magnitude (strength) of the k-th frequency component W/ = 2x /N. In practice, DFT is implemented with an efficient algorithm known as Fast Fourier Transform that runs in On log n) time.
We compared two methods, periodogram and vanilla FFT. The periodogram approach computes the Fourier transform after applying auto-correlation and time-averaging windows the signal for de-noising purposes [3]. However, we think de-noising is inappropriate because our "signal" is time series of cross-entropy, whose value reflects the sampling result at each time step from large. Auto-correlation or time averaging will remove the distinctiveness of rare tokens. Therefore, we use vanilla FFT and take the real part of X (wk) to represent the magnitude spectrum for the frequency component Wk, which is written as ((w)) for brevity.
For an input cross-entropy sequence & [c,,..., cr-1) obtained from Section 2.1 the sulting frequency spectrum can be represented as a list of tuples of the same length, F = [wr,, X(w))) ..., (WT-1,X((a)) (wT-1))), where "ub....,yy-1. are the T - sample frequencies, and
are the corresponding magnitudes.
2.3 Spectral similarity metrics
We develop four metrics to measure the similarity between spectra F,, and Fm Spectral Overlap (SO), Spectrum Angle Mapper (SAM) (6), Pearson's 's correlation (CORR), and Spearman's 's correlation (SPEAR), as summarized in Figure? 2 Before computing the metrics, two spectra Fn and Fm which of different lengths N1 and N2, are first interpolated to the same length: F, E R = F Ne, Fm E N2 F E Ne, Here, No is the maximum length of the spectrum in our data. Thereafter, the computation of the subsequent metrics can commence.
Spectral Overlap (SO) is inspired by the power spectrum overlap proposed in 28 which is used in [47] for measuring the spectral similarity between dialogue participants. The frequency magnitudes in F2 and Fy are converted to absolute values, i.e., X X(wx) > XX [W(w)) and then compute the Area-Under-Curve (AUC) for the interaction Fy, F' and the union F U U. respectively. SO defined as the ratio of the two: SO AUC(F) nF' // UF ). The procedure of converting
attps: //eer/a//eaaa.../ stanford. edu/" jtwp:///wooo/eeee/ooooeessa www.rroooriee...rrrrrrraaooa Transforme
Traasforre_Cottiiuuuus Cow.iinonnuuuu.o..m Sttbiiiuuu_DDsccete... www.cetteeeeeeeeee Saaaztte_TieeFFeeueaayyy

---

to absolute values is indispensable, since negative values in (wx) will result in negative AUCs, SO has the range [0. 1), and higher value indicates a stronger resemblance between the two spectra.
Spectrum Angle Mapper (SAM) calculates the angles between Fi and Fy treating them as two The angle is measured in radians, which is calculated by the inverse function F [|FFml)), producing a value within TT. We understand SAM is equivalent to the cosine similarity score, which is more commonly-used in NLP, but here we just follow the
vectors in space 22
ar F
conventions in 22 2] A smaller SAM value indicates greater similarity between F% and Fm
Pearson's correlation (CORR) can also be leveraged to measure spectral similarities as discussed in [22]. CORR = (m)/ high similarity (negative for dissimilarity), and 0 indicates weak correlation between FV and Fr
with a [-1, 1] range. A positive CORR value indicates
Spearman's 's correlation (SPEAR) 4 is commonly used to assess the monotonic relationship between the comparison and reference groups and to capture the presence of non-linear associations between the two. It has not been used for spectral similarity to the best of our knowledge, but we test it in our experiments. SPEAR also has the range [-1, 1] with meanings similar to CORR.
3 Related Work
Entropy as a metric in psycholinguisties. The entropy of human language has long been research interest in computational linguistics and psycholinguistics. The entropy of written text is estimated with the average per-word negative log-probability in sentences, and then used to validate the principle of entropy rate constancy (ERC) in human language [3 14). Similar studies were conducted in
dialogue 46 32). Entropy is also defined in probabilistic grammars to describe the capacity of
language 40 and is used to develop complexity metries to measure the cognitive load of processing syntactic expressions [16, 24 17. In the line of work on language production, different term information density with the same mathematical formulation is used instead of entropy. It is found that speakers reduce syntactic complexity when the information density (or entropy) is high [25] 20). In parallel with the concept of ERC, this line of work summarizes the tendency of distributing information evenly in human language with the term uniform information density (UID), which is
commonly used as a equivalent term as ERC, for example, in [48 [5]. In conclusion, entropy is commonly used as a metric for essential properties of human language.
Periodical change of cross-entropy in language. We draw inspiration from the following studies about the distribution of information in dialogue. Humans are sensitive to the peaks and troughs of entropy in speech, with evidence from human-system dialogues and crowd-sourced ratings from human judges (7). 'he entropy of utterances from two speakers converge towards each other within the scope topical segments in spontaneous dialogues [48]. They measure the entropy of utterances from two participants of task-oriented dialogue, and have found that the frequency domain features power spectrum overlap and phase delay are useful predictors of task outcomes. Both works reviewed above suggest that the periodical up-and-downs of entropy are commonly observable in the human language. It naturally leads to the question of whether and to what extent model-generated language aligns with this empirical finding.
Automatic measures for text generation. Entropy and its related variants have already used as a metric for evaluating generated text, for instance, entropy provides good visualization for the difference between GPT2-generated text and human written ones [2]. Other than entropy, there is a rich body of existing metrics targeted on discriminating human-written text and model- generated text, which we summarize in three branches: (I) statistics- (2) language modeling;
(3) reference-basee Table I gives a brief summary of these three categories, as well our proposed frequency-based FACE.
Statistics-basdd measures compare the model-generated distribution M with respect to the human- written distribution H in terms of some statistic. The Zipf coefficient [30] used in [19 to describe the distribution of word frequencies in text. Belf-BLEU [2 is derived by calculating the BLEU 29 score for each generated text utilizing all other generations as references. Repetition measures the sequence-level degree of repetition on the basis of the percentage of duplicated n-grams in the generated continuations Fromnt ~ M 44. Meanwhile, we aggregate the 2-gram, 3-gram, and 4-gram repetition rates to evaluate the lexical diversity in an inverse manner.

---

Type
Metric
Meusure
Definition/ Appriiiioonppprrimiiiioi
Zipf Coefficient
Unigram rink-frequency statisties
Statisties
Self-BLEU
N: gram diversity
Repetition
Soquence-level peccentage of repetition
Diversity
of v-gram repetition rates (r 2,3,
[T.0 Repetition)
Language Modeling
Perplexity
Evaluation-set perplexity
En log M[()]]
Coberence
LM quality (cosine similarity between sentence embeddings)
[EMB II1 EMBio
Divergence Curve
MAUVE
Quality & diversity via the divergence frontiers
C,T,, M) at C (0,1) 31
Prequency Domain
(this work)
Quality & diversity via spectral similarities (Four metries)
Human Judgment
Bradley- Terry Score
Human preference via the puirwise evaluation
P(j beuts f)
Table Summary of metrics (automatic & non-automatic) we employed for evaluating open-ended text generation. FACE provides way to approximate the human-model gap in the frequency domain.
Language modeling metrics measure how un(certain) human text H follows the model distribu- tion M, using the probability distribution (x). ). In our work, the perplexity is calculated upon the set of human texts quantify how well the distribution M predicts text continuation. Coherence is approximated by cosine similarity between the sentence embeddings of prompt pre ~ and contin- uation conm M as proposed in [42], where the embedding EMB(-) is produced by the pre-trained SimCSE sentence embedding [II.. Metrics under this category never observe model-generated text samples, and hence, they cannot justify how likely Ccont is under the human distribution H.
Reference- based measures assess the generated text with respect to a small set of reference text, rather than calculating over the full sequence distributions. Some recent reference-baseee based approaches encompass: (1) 36 38 50 aim to capture distributional semantic information in high-dimensional space; (2) [11 concerns Euclidean distance between vector representations of n- n--rams and their document frequencies; (3) B1 straightforwardly computes the similarity of one learned distribution from a text generation and the other distribution of -wiiten text using information divergence frontiers 9 23 34
Reference-based metrics are well-suited for targeted generation tasks (e.g.. machine translation). Nevertheless, they become unfavorable in the open-ended generation scenario where multiple reasonable and diverse continuations are preferred.
evaluation metrics and decoding strategies for natural language generation rely on human judgments, assuming that human annotations are the gold standard. Considering the expense of Human Unified with Statistical Evaluation (HUSE) (18), we adopt a pairwise evaluation protocol based on human preferences, to serve as a non-automatic complement of FACE metries. We leverage the Bradley-Terry model [3] to predict the outcome of head-to-hin comparison given n players with scores $1,".. An.
Non-automatic metrics. Recent works [2]
31 19 26
4 Experiments
Task formulation. Given an input text passage as prefix, the open-ended generation aims to produce texts that form fluent and coherent continuation. More formally, given sequence of m tokens denoted [ ..... Æm,. as the prompt, the goal is to generate the next n continuation tokens to form complete sequence [z] ... Im+n]. The continuation probability at the decoding
time by conditioning on the preceding context is defined as: P (Im+1 Imta I x1....mm) I 11
P (x: Ii-)), where P (x; Ø1 .I-.) is the next-token distribution.
4.1 Model sizes
We consider such text completion task in three domains: Wiki text, News, and Stories. Intuitively, the generated texts involving different domain knowledge may have different language usages and writing style, which may reflect on metrics. We generate completions from large-scale language models (LMs). In particular, we adopt three representatives of tfth-oofthheart pre-trained auto- regressive LMs: Generative Pre-trained Transformer 2 (GPT2) (3), Open Pre-trained Transformer LMs (OPT) 49 and BigScience Large Open-science Open-access Multilingual LM (BLOOM) (35) We explore two sizes for each model to illustrate that our FACE metrics generalize across multiple LM familics and sizes. Details regarding our task and input data are summarized in Table 2 Different
5

---

Domain
Model
Dataset
Prompt Length
Maximum Generation Length
Number of Generations
Wiki tex
GPT2/OFT/BLOOM
WikiText-103
35 tokens
1024 tokens
5000
News
GPT2/OPT/BLOOM
RealNews
35 tokens
1024 tokens
5000
Stories
GPT2/OPT/BLOOM
WritingPrompes
varying
1024 tokens
5000
Table 2: Dataset and task summary. In our research, we set the maximum generation length to 1024 for all models on three datasets. Note that the WritingPrompts dataset [T]] contains ready-to-use prompts, so the length of prompts varies. For WikiText- 103 [27] and RealNews [ datasets, we cleaned them before extracting the texts corresponding to the first 35 tokens (tokenized by GPT2TTokenizrr) to form our prompt sets.
Domain
Metric
GPT2-sm
GPT2-l
Voting
OPT-1225
OPT46.7b
Voting
BLOOM-S60m
BLOOM-7.I6
Voting
Divenity (?)
0.733
0.753
0.645
0.789
1
0.33
0.732
L
Coherence (?)
0.595
0.624
L
0.614
0.534
1.
L
0.935
0819
Zipt Cocticiant (1)
0.990
0.975
L
0.589
1,016
5
1,092
0.980
L
Self BLBU (.)
0.459
0.424
0.425
0.379
L
0.280
0.422
S
Wikd text
MADVEC)
0.077
0.150
0.704
0.253
L
0317
0.184
0.414
0.466
0.424
0.436
0.426
0.432
CORR (1)
0,806
0.781
S
0.771
0.789
L
0.675
0.789
L
SAM (L)
0.199
0:213
0.316
0.217
5
0.558
0.88
I.
SPEAR (1)
0.022
0.033
1.
0.026
0.000
1.
0.059
0.023
Diverity (1)
0,890
0.897
1.
0.853
0.876
1.
0.740
0870
Cohrsence (?)
0.613
0.60
1.
L
0.663
0.663
5
L
0.897
0
S
Zipl Conffictent (1)
0.961
0:958
L
0.965
0,968
L
0.961
0.966
Solf-Br BL--BBBUL4 (L)
0.619
0.573
L
0.611
0.543
1.
0.364
0.501
News
MAUUE.
0.371
0:281
0.707
0.130
8
SO113
0.424
0,412
0.438
0.440
L
0.435
0.457
CORE (1)
0.757
0.725
S
0.346
0,732
5
S
0.615
0.333
L
SAM (L)
0.234
0.340
0.229
0.236
0.281
0.234
SPEAR (1)
0.021
0.019
0,017
01021
L
0.348
0.019
Divenity (1)
0,743
0.785
0.769
0.875
L
0.527
0.830
Cohesence
0.421
0.421
L
1.440
03388
5
L
0,800
0.660
S
7ipl Coefficient
1.097
1.085
1.01
1.003
1.
0.999
1058
Sal--R (L)
0.617
0.565
0.587
0.511
0.180
0.455
Skoris
MAUVEC
0.50
E.121
UT005
013305
80/1)
0.411
0:402
0,406
0:405
0,350
0.418
CORR
0.813
0.787
S
0.737
0,705
S
0.573
0.772
L
SAMFLL
0,195
0.209
0.245
0.214
SPEART1:
0,023
0.022
0,041
0.050
0.027
Table 3: Domain-specific generation quality with respect to different models (GPT2/OPT/BLOOM)
and model sizes (large model on the left and small model the right) using top-k (k=50) sampling under various existing metrics, as well as proposed FACE metrics. indicates the larger the metric value, the better, whereas 4 indicates the opposite. The column indicates the better setter-prrfommig model in each comparison, where L/S denotes the large/small model wins and E represents a tie. We applied the majority voting determine the winner. OPT and BLOOM postfixed with their number of parameters.
models may generate vastly different numbers of continuations in each length interval (see Appendix). To ensure the fairness of investigating the correlation between FACE and other widely-used metrics with respect to different models (with different sizes), we compute the weighted arithmetic mean for every metric across five length intervals.
The evaluation metrics we are interested in are based on various motivations and principles. Specifi- cally, MAUVE and FACE emphasize the parallels between human and machine-produced texts, as stated in Section 3 Therefore, we group MAUVE together with four FACE metrics. To further obtain intuitive results, we utilize the voting approach to explore the correlations between these metrics on large/small models across three task domains. The results are shown in Table 3
In our investigations, the GPT2-xl model consistently outperforms its small counterpart among statistics-bact based and language modeling metries all relevant "vs." " columns indicate, apart from the Coherence in the Stories domain. In the GPT2 experimental group, it is astonishing that the small model always performs better when referring to the voting results from MAUVE and FACE rows. Across three task domains, the performances of OPT and BLOOM models in two sizes differ. Large models have better overall performance, and small models only win four out of twelve comparisons by voting. Nonetheless, it is noteworthy that four FACE metrics we proposed maintain a relatively high level of consistency with MAUVE across all models. At least two FACE metrics yield the same results (in eight out of nine sets of numan-mode language comparisons) with MAUVE. Concretely

---

1442
50
0.0
50
0422
50
P.
1423
000000
3352
0.0
3333-
OPT
BLOON
GPT2
Figure 3: FACE-SO scores on OPT, BLOOM and GPT2 original output data. Model sizes compared: small vs. large for OPT and BLOOM; -sm vs. -xl for GPT2. Error bars represent 95% confidence intervals from bootstrap. The significant levels based on (-test between the model-size groups.
affirne Brirtry
Figure 4: FACE scores (conditional generation) on original experimental data of 19] and 26 Nine sampling methods are compared: greedy, beam search, stochastic beam search, pure sampling, tem- perature, top-k, top-k with temperature, nucleus, and contrastive. Note that logarithmic normalization on parameter values as well as enlarged markers for greedy decoding, pure sampling, and contrastive decoding are adopted for better visualization effect. Best viewed when zoomed in.
speaking, SO and SAM show a higher positive correlation to MAUVE than CORR and SPEAR, given that seven out of nine voting results (marked with yellow in Table B) are identical.
To further evaluate model sizes, we apply FACE to the original GPT2 output data (webtext) generated from GPT2-sm and GPT2-xl. GPT2-xl has higher SO score than GPT2-sm, which is confirmed with the (-test, but non-significant effects are found on the other three metrics. Combining our generation task with the original GPT2 data, we illustrate the results for SO in Figure
To conclude, we discover three keypoints: (1) FACE is consistent with MAUVE in evaluating three different model types (two sizes for each); (2) the metrics estimating similarity between
and model-generated text generations (e.g., FACE, MAUVE) may produce opposite results the text-centered metrics (e.g., Diversity. Coherence); (3) the four metrics of FACE show relatively homogeneous results, and using these metrics together helps to identify more comprehensive evaluation.
texts with
4.2 Sampling methods
Recent work 19 26 has indicated three clear trends in open-ended text generation using auto- regressive LMs: (1) maximization-based decoding algorithms (e.g., beam search, greedy decod- ing, etc.) lead to copious repetition, while sampling with temperature may result in incoherence; (2) truncation-baat
sampling methods like nucleus sampling produce text with higher quality; (3) contrastive decoding outperform nucleus sampling in terms of both fluency and coherence. Ac- cordingly, to demonstrate the effectiveness of our approach, FACE should follow the inequality: iaaiiiizationbbssd//eeppetntt based nucleus contrastive in terms of the quality relationship.
Figure visualizes the correlation between FACE scores and various decoding algorithms. The contrastive decoding approach yields the best performance among the four FACE metrics. It can be clearly observed that the maximization-base sampling methods behave worse than other algorithms. Moreover, adding the temperature parameter to top-k sampling results in incoherent text generations,

---

Sampling Method
Perplexity
Self-BLEU
Zipf Coefficient
Repetition
SO (t)
CORR (1)
SAM ())
SPEAR (1)
Human
12.38
0.31
0.93
0.28
Groody
1.50
0.50
1.00
73.66
0.20
0.56
0,31
0.04
Beam (b=16)
1.48
0.44
0.94
28.94
0.21
0.31
0.40
0.04
Stochastic Beam (1=16)
19.20
0.28
0.91
0.32
0.37
0.49
0.33
0.04
Sampling
22.73
0.28
0.93
0.22
0.41
0.28
Sampling (1=0.9)
10.25
0.35
0.96
0.66
0.42
0.61
0.29
0.03
Top k (k=40)
6.88
0.39
Top-A (k=640)
13.82
0.32
0.96
0.28
0.42
0.63
0.28
0.03
Tog. A (K=40, (=0.7)
3.48
0.44
0.34
Nucleus (na0.95)
13.13
0.32
0.95
0.36
0.42
0.63
0.28
0.03
Contrastive Decoding
14.39
0.54
Table 4: Results for comparing all sampling methods with selected parameters regarding the condi- tional generation. The values closest to human scores are bolded, except for our proposed FACE scores, where the highest (for SO, CORR, and SPEAR) or the lowest (for SAM) values are in bold.
which explains the gap between the red curve (top-k w/o temperature) and the gray curve (top-k w/ temperature). We also plot the correlation graphs of unconditional generation (in the Appendix) with fewer sampling methods involved. The trends and patterns in the visualization of unconditional generation are basically consistent with its conditional counterpart.
In Table FACE scores on different decoding algorithms are summarized. FACE metrics correctly match the expected quality relationship of the sampling methods examined by assigning the best SO (.44), CORR (.75), SAM (.23), and SPEAR (.17) scores to contrastive decoding. Other evaluation metrics fail to capture the correct relationship, for example, the perplexity rates nucleus-sampled text as better than contrastive-decoded text, which is irrational suggested by Li et al. [26).
4.3 Human judgments
We also explore the correlation between FACE and human judgement scores, using the crowd-source dataset collected in (31 when human evaluation is available. The dataset contains model-generated continuations (by GPT2-sm, -md, -Ig, and -xl with ancestral and nucleus sampling), continuations using the same prefix, and the crowd-source workers' answers on which completion is more human-like interesting, and sensible. We follow the same experimental settings and protocol to verify whether the FACE scores of the text completions correlate well with the human quality judgements by computing the Spearman's 's rank correlation coefficient. The results are presented in Table
We observe a high and positive correlation between FACE-SO and human judgments scores, which outperforms five out of the six evaluation metrics reported in [31 and achieves comparative performance against MAUVE. The remaining three FACE metrics have insignificant correlations. However, we consider human judgments to be subjective and sometimes biased. Including more fine-grained questions to perform human judgments may lead to more accurate correlation statistics. Additionally, we recomputed the correlations with human judgement scores to keep those pairs in which there are exactly one item from human and the other item from model (i.c., a subset of data used for the analysis in Tj. As shown in SO-S and MAUVE-S columns, FACE-SO has stronger correlation than MAUVE among two of these three dimensions.
4.4
Sanity tests
Sanity test on validity. We evaluate the validity FACE by examining whether its scores on human- human split is lower than human-moder groups an expected result based on our assumption that the spectral difference between human's 's "natural" language and models' "artificial" ones should be amplified by FACE. Therefore, the sanity tests are conducted as follows: first, we evenly and randomly split the human data into two folds (across three domains) to serve as control groups. The FACE scores between these control folds are then computed. As for the numan-to-modes experimental group, we create other two folds using human data and the best model genee-generrted data (from contrastive decoding (26) in terms of text quality. If FACE effectively capture the fundamental difference
8

---

Metric
Generation Perplexity
Zipl Coefficient
Repetition
Distinet-4
Self-BLEU
S0
MAUVE
S0-S
MAUVE-S
like/BT
0.738
0.337
0.214
Interestin/BBT
0.643
0.524
-0.143
0.524
0.405
0.762
0.810
0.521
0.667
Sensible/BT
0,738
0.690
0.071
0.595
0.524
0,786
0.857
0.995
0.706
Table 5: Spearman's 's rank correlation coefficients of SO and five other metrics with human judgments. Higher scores mean better correlation. All the numbers except the SO, SO-S, and MAUVE-S columns are sourced from [3I]. "BT" denotes the Pradley-Terry score of the pairwise human evaluation, which is employed to compute the Spearman's 's rank correlation with the scores of other metrics. Additionally, it is important to note that the original human judgments encompass certain pairs in which both texts generated by models, albeit different models. Therefore, we refine the original human judgment dataset to only include judgments involving both human and model-generated languages, and the results are shown in SO-S and MAUVE-S.
between human and model languages, then expect observe higher scores in control groups than in the experimental group. The results are shown in Table 6
so (t)
CORR (1)
SAM (1)
SPEAR (1)
h-h (wiki)
0.45
0.76
0.22
0,05
h-h (news)
0.45
0.76
0.22
0..07
h-h (stories)
0.47
0.79
0.21
0.05
h-m
0.44
0.75
0.23
0.17
Table 6: Results of sanity test on FACE's validity. The top three rows are the control groups, and "h-h" stands for human-to-human folds. The last row is the experimental group, where "h-m" is for human-to-mode fold. Better FACE scores are in bold. The scores in the bottom row are retrieved from Table
can be seen from Table 6 that the control groups show significantly better FACE scores than the experimental group: FACE-SO and FACE-CORE FACE-SAM scores are lower. The only exception is FACE-SPEAR, while we will show it is a good metric in later sections. Nonetheless, these tabulated results have proved the validity of FACE in effectively capturing human-to-model spectral differences.
higher in human-to- to-mmmmn folds, while
Choice of estimator model. We examine how different choices of the estimator model mes affect the resulting spectra, using GPT2-sm, -md, -Ig and -xl as Mest, respectively. The spectra of webtext and the original GPT2 output data are computed. It is found that the spectra obtained from Mest have different magnitudes, but their aggregated curves have the same shape (see Appendix). Therefore, the choice of Mest will not affect FACE scores as long as the same Mest is used for all data.
Stationarity tests. One of the assumptions of the Fourier transform is that the signal is stationary (21] i.c., the mean and variance do not change over time. We applied the Augmented Dickey-Fuller (ADF) test X to examine the stationarity of the cross-entropy sequences for all the human and model-generated data used in this study. The null hypothesis Hu of the ADF test is non-stationarity. and thus p < .05 testing result rejects Ho and accepts the alternative hypothesis of stationarity the series. We calculate the proportions of cross cross-entropy sequences that pass the ADF test with < .05 for model-generated and human data: 97.4% for GPT2, 92.1 % for OPT, 74.5% for BLOOM, and 97.9% for human. Thus, the majority of data meets the stationarity requirement.
4.5 Interpretation of spectra
As the frequency spectrum reflects the key characteristics of a signal, we attempt to interpret the spectra to see if they tell how the "signals" entropy of human and machine languages differ. Without aggregation, the raw spectra of single cross-entropy sequence look indistinguishable between GPT2-sm, xl,,,xl, and human (see the left plot in Figure 5). By aggregating 5,000 spectra from each group and smoothing the curves, it can be seen that GPT2-xI's curve is closer to human than the GPT2-sm curve (readers can find this by zooming in the middle plot in Figure 5). Here, the smoothing is done with generalized additive models (GAMs) 45 Results from other models are included in the Appendix.

---

Sample spectra
Smoothed spectra
oPb
GPT2-al
120
than GPT2-cm
X5000
Human
orrban
Smoothed spectra (log scale)
1e-22
0.12
0.18
0.1
0.
Figure 5: Intuitive observations on the spectra from GPT2 and human data (webtex(). Left: Spectra of three randomly sampled entropy sequences from GPT2-sm, GPT2-xI, and webtext. Middle: Smoothed plot of 5,000 aggregated spectra with absolute values, |.Kwy] ~ wk. Right: Typical smoothed plot of raw spectra Xwm wk. with peaks and troughs annotated.
When plotted separately, the aggregated spectra from human and different models have similar shapes: First, the majority of components exist in the low-frequency range (w < 0.05). In addition, the locations of peaks and troughs are almost the same between groups. For instance, w1 = 0.06 the first trough, and w2 0.12 is the first peak (see the right plots in Figure (5. Thus, roughly speaking, the main difference between human and model spectra is not in the locations of peak and trough frequencies but in the relative magnitudes of those frequencies.
We propose a simple way to interpret the peaks in spectra: the reciprocal of a frequeney component Th 1/w/ denotes the corresponding cycle in the time domain. Because the time interval (i.e., sampling interval) of an entropy sequence is not measured in seconds but fixed as one token, the measurement unit of T3 is also in number of tokens. For example, the first frequency peak in Figure (right plot) implies W2 = 0.12 = T2 1/0.12 8.3 (tokens), which approximately means that tokens of the same cross-entropy levels tend to recur every 8.3 tokens. This pattern is consistent in both human and model data. However, the degree this recurrence can mark the difference between the human and model languages. We leave more detailed interpretations of spectra to future work.
Conclusion and Limitations
We propose FACE, a set of metries based on the Fourier analysis of cross-entropy, which is able to distinguish human and model model-generated language with satisfactory performance in the open- endee generation task. The metrics scale with model sizes; reflect the effect of various sampling methods; correlate well with other existing metrics and outperform most of them in alignment with human judgement scores. Among the four implementation methods of FACE experimented, Spectral Overlap (SO) has the best overall performance.
FACE is computationally efficient with easy-to-interpret output. As a method inspired by psy- cholinguistic studies on the predictability (entroy/surprislliffommaato density) of human language, we believe FACE is good example of incorporating knowledge from different fields for better human-centered AIs. We can generally conclude that better language models can produce spectral representations of information that are more similar to human.
Our current work has several limitations: Firstly, for open-ended generation experiments (Section 4.1 broader set of sampling methods other than top-k can be used. Secondly, larger models (with more than 100 billion parameters) need to be included for more comprehensive comparisons. We will improve from these aspects in future work.
Acknowledgement
This material is based upon work supported by the National Science Foundation under Grant No. (2105192).
10

---

References
[1] H. Ahmed, I. Traore, and S. Saad. Detection of online fake news using n-gram analysis and machine learning techniques. In Intelligent, Secure, and Dependable Systems in Distributed and Cloud Environments: First International Conference, ISDDC 2017. Vancouver, BC, Canada, October 26-28 28, 2017, Proceedings I, pages 127-138. Springer, 2017.
[2] J. Boardman. Sips user's s guide spectral image processing system, version 1.2. Center for the Study of Earth from Space: Boulder, CO, USA, 1992.
[3] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-445) 345, 1952. ISSN 00063444, URL http:
www. www.jstor.... wwt.oo/gtg//e/////////
[4] E. Clark, A. Celikyilmaz, and N. A. Smith. Sentence mover's s similarity: Automatic evaluation for multi-sentence texts. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2748-2760, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10.18653//1/P19-1222 1264. URU/attss///cclaattooo. org/
P19- 1264
[5] J. W. Cooley and J. W. Tukey. An algorithm for the machine calculation of complex fourier series. Mathematics of computation, 19(90))297-301, 1965.
[6] O. A. De Carvalho and P. R. Meneses. Spectral correlation mapper (scm): an improvement on the spectral angle mapper (sam). In Summaries of the 9th Airborne Earth Science Workshop, JPL Publication 00 18, volume 9, page 2. JPL publication Pasadena, CA, USA, 2000.
[7] N. Dethlefs, H. Hastie, H. Cuayáhuitl, Y. Yu, V. Rieser, and O. Lemon. Information density and overlap in spoken dialogue. Computer speech & language, 37:82-97, 2016.
[8] D. A. Dickey and W. A. Fuller. Distribution of the estimators for autoregressive time series with a unit root. Journal of the American statistical association, 74(366a):427-431, 1979.
[9] Djolonga, M. Lucic, M. Cuturi, O. Bachem, O. Bousquet, and S. Gelly. Precision-recall curves using information divergence frontiers. In International Conference on Artificial Intelligence and Statistics, 2019.
[10] A. Fan, M. Lewis, and Y. Dauphin. Hicrarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 889 9988,, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 1/P18 1082.
http:/loyyy.rg/...... 1082
[11] T. Gao, X. Yao, and D. Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 202 Conference on Empirical Methods in Natural Language Processing, pages 6894 6910, Online and Punta Cana, Dominican Republic, Nov. 2021. Association
for Computational Linguistics. doi: 10. 10.18653/11/2021.10 1665///12221..cmmmmmmn...5333
URL https://
ac aclanthology otg/2021...22...oo 11021.eemn1. emnlp waanllp-main... 552
[12] S. Gehrmann, H. Strobelt, and Rush. GLTR: Statistical detection and visualization of generated text. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 111-116, Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10. 10.18663/1//11-3300 URL httpttt//ccaaatthllaae aclattology org/P19-3019
[13] D. Genzel and E. Charniak. Entropy rate constancy in text. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 199-206, 2002.
[14] D. Genzel and E. Charniak. Variation of entropy and parse trees of sentences function of the sentence number. In Proceedings of the 2003 conference on empirical methods in natural language processing, pages 65-72, 2003.
[15] M. Giulianelli, A. Sinclair, and R. Fernández. Is information density uniform in task-oriented dialogues? Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 8271-8283, 2021.
11

---

[16] J. Hale. A probabilistic earley parser as psycholinguistic model. In Second meeting of the north american chapter of the association for computational linguistics. 2001.
[17] Hale. information-thheorttiall complexity metrics. Language and Linguistics Compass, 10(9): 397-412, 2016.
[18) T. B. Hashimoto, H. Zhang, and P. Liang. Unifying human and statistical evaluation for natural language generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 1689-1701, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. 10.18653/1/N11-11666 URL org/N19-1169
[19] A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. The curious case of neural text degencration. In International Conference Learning Representations, 2020. URL https //openreview.o
[20] T. F. Jaeger. Redundancy and reduction: Speakers manage syntactic information density. Cognitive psychology, 61(():23-62, 2010.
[21] I. Kaplan. Dft of non-saationnryy time series, Sep 2001. URL [RL/tt:://eearrave.cc///1l// http///////iipaaa/oossstt//iaaae html
[22] A. Kruse, A. Lefkoff, J. Boardman, K. Heidebrecht, A. Shapiro, Barloon, and A. Goetz. The spectral image processing system (sips) interaiitte visualization and analysis of imaging spectrometer data. Remote sensing of environment, 44(2-3): (::(23))145--- 145-163, 1993.
[23] T. Kynkäänniemi, T. Karras, S. Laine, J. Lehtinen, and T. Aila. Improved precision and re- call metric for assessing generative models. In H. Wallach, H. Larochelle, A. Beygelzimer, d''Alcéé-Buc Biché-Bucc.5 E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Sys- tems, volume 32. Curran Associates, Inc., 2019. URL btpp newriiis...i.n..............i..s nwwrips...... cc/
paper_ files/
pdf
[24] R. Levy. Expectation-based syntactic comprehension. Cognition, 106(3): 126-111777 2008.
[25] R. Levy and T. Jaeger. Speakers optimize information density through syntactic reduction. In
B. Schölkopf, J. Platt, and T. Hofmann, editors, Advances in Neural Information Processing Systems, volume 19, pages 849-856, 2007.
[26] X. L. Li, A. Holtzman, D. Fried, P. Liang, J. Eisner, T. Hashimoto, L. Zettlemoyer, and M. Lewis. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint 10. 15097, 2022.
[27] S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models. In Interna- tional Conference on Learning Representations, 04 2017.
[28] O Oullier, G. C. De Guzman, K. J. Jantzen, J. Lagarde, and J. Scott Kelso. Social coordination dynamics: Measuring human bonding. Social neuroscience, 378):188-192, 2008.
[29] K. Papineni, S. Roukos, T. Ward, and W.-J Zhu. Bleu: a method for automatic evalua- tion of machine translation. In Proceedings of the 40th Annual Meeting of the Associa- tion for Computational Linguistics, pages 31 1-318. Philadelphia. Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.1115/0033033.1171133 URL https://aclanttolegg
Satrg/P02-1044 1040
[30] S. Piantadosi. Zipf's word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review, 21, 03 2014. doi: 10.37788/s33423-04--8833
[31] K. Pillutla, S. Swayamdipta, R. Zellers, J. Thickstun, S. Welleck, Y. Choi, and Z. Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 4816-4828 Curran Associates, Inc., 2021. URL htw.stttpss:... httpaaa:///ooccoeeeeeeeeee neurips...........oe...........e cc/ httpscccpppppae sicc/appr__ffe files/ieeeeeeee waapperffles/ppperr///2000 ppple//ppperr/2///iieee 20000003000002234300000000081020
In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and
pdf
12

---

[32] T. Qian and T. F. Jaeger. Topic shift in efficient discourse production. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 33, 2011.
[33] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners, 2019.
[34] M. S. M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and S. Gelly. Assessing generative models via precision and recall. In Advances in Neural Information Processing Systems, page 5234-5243, Red Hook, NY, USA, 2018. Curran Associates Inc.
[35] T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilié, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, al. Bloom: A 1760-parameter open-access multilingual language model. arXiv preprint arXiv:22/1.088.
2022.
[36] T. Sellam, D. Das, and A. Parikh. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages 7881-7892, Online, July 2020. Association for Computational Linguistics. doi: 10. 18653/1vv1/ 2020.acl-main. 704. URL https: hat::/acllacc http/aclaanttooogog org/ 202020220.222 a00.0mclmmaa... 704
[37] C. E. Shannon. Communication theory of secrecy systems. The Bell System Technical Journal, 28(4):656 -75, 1949.
[38] H. Shimanaka, T. Kajiwara, and M. Komachi. RUSE: Regressor using sentence embeddings for automatic machine translation evaluation. In Proceedings of the Third Conference Machine Translation: Shared Task Papers, pages 751-758, Belgium, Brussels, Oct. 2018, Association for Computational Linguistics. doi: 10. 10.18653/11/11--1 6456. URL Oxg/W18-6456
[39] J. O. Smith. Spectral Audio Signal Processing. http://ccrma. stanforr..taaaodd.ooo wwt.oorr.ddddd////eeoeeee edte//js/////////////c/ae/e/e "jssssapp accessed 3/14/2023. Online book, 2011 edition.
[40] S. Soule. Entropies of probabilistic grammars. Information and Control, 25(1):57-74, 1974.
[41] C. Spearman. The proof and measurement of association between two things. The American Journal of Psychology, 1961.
[42] Y. Su, T. Lan, Y. Wang, D. Yogatama, L. Kong, and N. Collier. contrastive framework for neural text generation. In A. H. Oh. A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. URL http://///ppeeeeee.eeeee/etroree J88BafmH9P]]
[43] P. Welch. The use of fast fourier transform for the estimation of power spectra: a method based on time averaging over short, modified periodograms. IEEE Transactions on audio and electroacousticc, 15(2):70-73, 1967.
[44] S. Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston. Neural text generation with unlikelihood training. In International Conference on Learning Representationss 2020. URL
[45] S. N. Wood. Generalized additive models: an introduction with R. CRC press, 2017.
[46] Y. Xu and D. Reitter. Entropy converges between dialogue participants: Explanations from an information -thorretcct perspective. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 537-546, 2016.
[47] Y. Xu and D. Reitter. Spectral analysis of information density in dialogue predicts collabo- rative task performance. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 223-633,22 2017.
[48] Y. Xu and D. Reitter. Information density converges in dialogue: Towards an information- theoretic model. Cognition, 170:147-163, 2018.
[49] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Out, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
13

---

[50] T. Zhang*, V. Kishore*, F. Wu², K. Q. Weinberger, and Y. Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL
https://openrevier.
[51] W. Zhao, M. Peyrard, F. Liu, Y. Gao, C. M. Meyer, and S. Eger. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference
Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pages 563-578, Hong Kong, China, Nov. 2019. Association for Computational Linguistics. doi: 10.18653/v//D19 1053. URL hel.ihttps//aallatthllgg.
[52] Y. Zhu, S. Lu, L. Zheng. J. Guo, W. Zhang, J. Wang, and Y. Yu. Texygen: benchmarking platform for text generation models. In The 41st international ACM SIGIR conference research & development in information retrieval, pages 1097-1100, 06 2018. 3209978.3210080.
doi: 10.1145/
14

---

Appendix
Broader Impacts
FACE measures the distance between human and model-generated languages, therefore it is techni- cally possible to used for designing or augmenting systems that mimic humans. We acknow ledge the risks of FACE (and other metrics) being utilized in applications that deliberately confuse human- authored and model-produced text. We call for the collective efforts from the community to come up with systematic framework that unifies different metrics, for developing more reliable and natural language generation systems.
2 Implementation Details
Preprocessing. We utilize three datasets: WritingPrompts. WikiText-103, and RealNews. For WritingPrompls, the prompt set has already been well-curated, so we just extracted the first 5,000 prompts (the length may vary) for generation task. WikiText-103 and RealNews contain many complete texts. For each complete text, we further truncate corresponding to the first 35 tokens as a prompt. To fairly evaluate the performance of metrics, also divide text generations according to five predefined length (from 0 up to 1024) intervals for each dataset. Thereby, the human- written texts and model-produced texts used to evaluate the performance of metrics may be generated by different prompts (i.e., unpaired comparison).
Sapperpprrametes. We have several hyper-parameterss during the text generation and evaluation phases. For both conditional and unconditional generation, we preset a random seed integer (32 by default). Furthermore, the maximum length of each text (1024 by default) well the batch size (which varies according to GPUs capacity) for perplexity computation have to be determined before automatic evaluation.
Miscellaneous Details
Software. Our experiments were performed on Ubuntu 20.04.1 system with Python 3.9.16. The versions of key Python libraries include: Transformers 4.27.4, -TTorch-CUDD 11.6, PyTorch 1.13.1, Seipy 1.5.4.
Hardware. For the text generation task, we use the remote workstation that has two NVIDIA RTX A6000 graphics cards. It should be noted that all models were in parallel when available.
Computation time for text generation. We spent 10 and 25 hours or so obtaining 5,000 text continuations by GPT2-sm, -xl, respectively. OPT-125m, -6.7b cost our GPU resources roughly 11 and 44 hours to output the same number of text continuations, respectively. When it comes to BLOOM- -bb, they took approximately 18 and 48 hours, respectively, to generate 5,000 continuations per task domain.
Evaluation time for FACE, Computation time of four FACE metrics for single pair of references are: 5.96 x 10-8 seconds for S0,5.01 5.01 x 10-8 seconds for CORR, 4.53 x 10-8 seconds for SAM, and 4.29 x 10-8 seconds for SPEAR, respectively. The cross-entropy, which should be calculated beforehand, takes 5.65 x 10 10 seconds. All of the above measurements take place on an AMD Ryzen Threadripper PRO 3995WX 64-Cores CPU (frequency range E can leverage more advanced GPU resources to perform the whole computation process with a faster speed.
4308 (00.4)) Users
Additional Experimental Results
4.1 Model sizes (generation length)
It should be emphasized that LMs have diverse designs and were pre-trained using different strategies on different datasets, giving them distinct preferences on the generation length. The numbers of text generations in each length interval are summarized in Table

---

Domain
Length Interval
GPT2-sm
GPT2-xl
OPT-125m
OPT-6.7b
BLOOM--600
BLOOM-7b
0-200
403
485
964
1522
4928
803
201-400
571
672
888
929
61
599
Wiki text
401-600
251
316
441
417
8
388
601-800
260
310
268
285
316
801-1024
3515
3217
2439
1847
2894
0-200
750
836
844
1119
4978
1371
201-400
1222
1336
1220
1325
20
917
News
401-600
824
759
1194
939
628
601-800
584
593
427
801-1024
1620
1024
1657
0-200
549
745
2731
3588
4924
1608
201-400
625
757
715
501
688
Stories
401-600
296
404
410
601-800
95
271
801-1024
3289
2770
1153
640
2023
Table Domain-specific generation length with respect to different models (GPT2/OPT/BILOOM
and model sizes (one large model and one small model) using jop-k (k 50) sampling corresponding to five continuous length intervals.
339
304
Sbe
681
Figure 6: FACE scores of GPT2 (our generated data), OPT, and BLOOM with different model sizes.
To ensure the consistency of our experiments, we run six LMs separately (using their own tokenizers) with the same prompt sets and settings as described in Table generate 5,000 pieces of continuations in each domain. Besides, we utilize the GPT2Tokenizer calculate the numbers of continuations for each interval, which allows us to compare FACE scores with other metrics more objectively, as we believe it is unfair to explicitly compare texts of varying lengths. Then, we compute weighted arithmetic mean to evaluate a model in each domain, by s Σ m. weighted mean; n denotes the number of length intervals; m, the number generated continuations in the length interval i; M Σ m,, and s; means a certain metric value in the interval i.
MM 8,. where s' denotes the
Figure 6 conveys more intuitive representation (via bar plots) of Table
4.2 Sampling methods (unconditional generation)
We also carried out experiments unconditional text generation. Here, the prompt is not required as we generate continuations from a random seed (set to 32 empirically). Four sampling methods, which are greedy decoding, beam search, stochastic beam search, and contrastive decoding, not involved this set of experiments.
The results are displayed in Figure 7 The overall trends are same as its conditional counterpart, where base//temperature based nucleus contrastive) is satisfied. Yet. it is crucial to note that the advantages of top-k sampling w/o temperature become more obvious compared the conditional case.
the previous quality relationship
httpp::/ggithu. com. htwprihhootttmma/eeeeeeee

---

Figure 7: FACE scores (unconditional generation) on original experimental date of nucleus sampling. Five sampling (decoding) methods compared: pure sampling, temperature, top-k, top-k with temperature, and nucleus. Note that logarithmic normalization on parameter values as well as an enlarged marker for pure sampling adopted for better visualization.
Model
Sampling Method (parameter)
SO
CORR
SAM
SPEAR
Nucleus Sampling (f=0.95)
0.481
0.821
0.191
0.359
GPT2-xl
Ancestral Sampling
0.472
0.807
0.199
0.331
Nucleus Sampling (f=0.95)
0.480
0.819
0.193
0.356
GPT2-Ig
Ancestral Sampling
0.472
0.814
0.196
0.338
Nucleus Sampling (7=0.9)
0.478
0.815
0.194
0.358
GPT2-md
Ancestral Sampling
0.462
0.813
0.197
0.310
Nuclcus Sampling (p=0.9) Ancestral Sampling
0.476
0.817
0.194
0.359
GPT2-sm
0.468
0.816
0.195
0.319
Table 8: FACE results based on MAUVE's original experimental data
4.3 Human judgments
Table shows the FACE scores based on the output texts from MAUVE. Each column of FACE scores used compute the Spearman's 's rank correlation coefficient between specifie FACE metric and Bradley-Tere scores (4 model sizes x sampling methods = 8 scores in total) from one criterion (three criteria correspond to three questions in total).
4.4 Choice of estimator model
We examine how different choices of estimator model mest affect the resulting spectra of cross-entropy. Five input data sources are examined (webtext plus four GPT2 original output datasets), on which four different estimator models applied: mes {GPT2-sm, GPT2-md, GPT2-Ig, GPT2-xI), resulting 4 = 20 aggregated spectra curves in Figure S can be found that on the same input data, the spectra from four estimators largely overlap. It indirectly suggests that FACE should be stable across different MegS. We leave the full inspection for future work.
4.5 Intuitive interpretation of spectra
As pointed out Section 4.5 the aggregated spectral shapes from human and different models are nearly identical. A set of higher resolution plots from GPT-xl, OPT, BLOOM and human (webtext) are shown Figure can be seen that although the X (ww) has different ranges on y-axis, the coordinates of the peaks and troughs are the same.
4.6 Corner Cases
Two examples highlight the difference between our proposed FACE-SO and MAUVE in their ability to recognize -enar-geeeerted and model-generate texts. In the filtered datasets for human judgments, the average values for FACE-SO and MAUVE are 0.4738 and 0.9549, respectively. In Case 1, human evaluators noted high level of similarity between the model-generated text and human text, resulting

---

Spectis of vetreal tree eeltrators
Spectta 777--am actimators
Specia of Gtt.2 forr entirattis
Fregerrs
Fincantty
Specra SFTT--aag
Specre GPT2-d
Figure 8: Aggregated spectra (using GAM smoothing) from four estimator models Mest [GPT2-sm, GPT2-md, GPT2-Ig, GPT2-xI3. Inputs are from GPT2 original output and webtext.
GPT2-XL
OPT
108
30
0.0
0.2
0.3
0.
02
0.
35
205
BLOOM
Human
158
106
X(e)
1-
4
Figure 9: Aggregated spectra for GPT-xI, OPT, BLOOM, and human (webtext).
in ties for human-like, interesting, and sensible aspects. However, the MAUVE score in Case is lower than the average value, while the FACE-SO score surpasses its mean. This discrepancy suggests that SO aligns more consistently with human opinions. Conversely, in Case 2. human judgement indicates significant dissimilarity between the model-generated text and human text, making them easily distinguishable. However, the MAUVE score exceeds its mean, suggesting the two texts are similar each other. Our FACE-SO score lower than mean, indicating better alignment with human opinion.

---

D
Peee
opeaninng go
<panding wan thomsed
anetting
The sock ruult excacied time
le
world'
safl
of givnt room into
intjanaanni--een que't bede the seeil the line Voal
abown it out scoed tou
dasidaa vuut
dany
eouda mupy ta plua Allnniingg pernalraria
right ow us devin camp stadiss
tead toul the is and (ree
in receed surpooee by 2018, buallan low sartana
vinenting and Sarob
sneleri--che cebooeee build' mnaaria
semposturns oen
serggdd moue shank
welive maticall derti slocc wnlinpert cove
soki insu ditv vertaree loferr
base growed caige
netseiing romee rocr
but the
emerenee he anaity meirorr mnsee dios dkile
geverneee the dlarddee Searts was to wirch og
renuate acciiee
lopwaan, wilkod lata busr
voroigh rpprrpctioo Nu modeea mas vari's
ciatan Maald gaiaa
sub-dto//B--be Bostte/lB-----ggetttr par from Marza Asantio and tartee
Beromna
Doas ley Mntenteeleeuueee atppts.
ti be oe
sowest irssyy since Zofine solk co opp-gg Rigu, vho uaas
undeng betts
chanttrreed recondctnn oftere,
City loals
Vikkay the fignt pnseerutu Ailees chall.
player
bneccss gam
the protecpp-ggoda qu onan the de seant bave fat karos) the
reparter
3/45
mongynne Sneu le
coloan trows Pigus, inpitten the sorpost
sipeeprittiiigg the
du Sieeuae toui clab. the rleak Uie class sach
postible mome forword olpeage
upr d/arrpolyy i veep some und lur, copongin di anyhie anact the actirt cal cosut that
Mididian are than momant) rasrarti
shacks therlp
Figure 10: Example of two corner cases. For each case, the prompt text, model- text, human text, MAUVE and FACE-SO scores, as well as the results from human judgments are tabulated.
5


Reviews:
[{"review_id": "UNfWqDL0gq", "reviewer": "Reviewer_v6cq", "summary": "This paper proposes a set of metrics to measure the distance between model-generated and human-written languages. Specifically, this paper uses FFT to analyze the cross-entropy sequences of the language data.", "questions": "1. Could the authors be more specific about their motivations for using spectral similarity as a metric?", "limitations": "This paper is a good step towards addressing some of the problems brought by generative AI.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This new metric is efficient. Given the fact that our models are getting exponentially bigger, it is essential that we do not waste energy during evaluation.\n2. This new metric correlates well with human judgment, and is statistically sound.\n\nI personally really like the authors' attempt to interpret the metric. Understanding the why is sometimes much more important than understanding the how.", "weaknesses": "1. The related work on psycholinguistic motivation is limited. Entropy is also a popular metric in computational linguistics, which is probably worth citing.\n2. The model size categorization seems to be very coarse.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Regarding Weakness 1: The related work on psycholinguistic motivation is limited. Entropy is also a popular metric in computational linguistics, which is probably worth citing.\n\nThanks for pointing this out. We will read through our paper in detail to include more comprehensive citations once our work gets accepted.\n\nRegarding Weakness 2: The model size categorization seems to be very coarse.\n\nIndeed, in our experimental setting, we only consider \"polarized\" groups where there is an extremely small language model having fewer parameters paired with a fairly large language model owning more than ten times the parameters for each model family, with the aim of showing FACE's scalability with model sizes. Adopting a finegrained model size categorization (e.g., five language models with linearly increased model sizes in each family) is definitely helpful to demonstrate the robustness of our proposed metrics. Nevertheless, we believe our current setting is sufficient for validating how FACE scales with model sizes since we have already taken two models with a \"stark\" contrast in the number of parameters across three LLM families into account. We have stated that larger models (with more than 100 billion parameters) need to be included for future work in line 299, but a more finegrained categorization strategy is also worth considering.\n\nRegarding the question \"Could the authors be more specific about their motivations for using spectral similarity as a metric?\"\n\nPlease read our general response about the research motivation."}, {"review_id": "3557UgLtHF", "reviewer": "Reviewer_AtQ2", "summary": "This paper proposes a new language generation evaluation metric.\nPrior work in psycholinguistics has shown that surprisal changes periodically in natural language, with natural utterances displaying moments of high and low surprisal.\nThis paper thus proposes to evaluate natural language generation models by quantifying how similar its surprisal frequency patterns are to natural language's.\nMore specifically, this paper proposes to: (1) estimate the surprisal of natural and model-generated text using a separate pretrained language model; (2) get frequency spectra for this surprisals using discrete fourier transforms; (3) compute 4 different metrics of similarity between the frequency spectra of model- and human-generated surprisals.\nThey then experiment with this metric, showing how it evaluates models of different sizes, and with different decoding strategies.\nIn a final experiment, they present the correlation between their metric and human judgment scores for 8 gpt2-based language generation systems (small, medium, large, xl with either ancestral or nucleus sampling); this experiment shows that while the proposed method does better then some prior metrics (e.g. self-bleu) it produces worse correlations than mauve.\n", "questions": "Questions:\n* Which model was used to compute $m_{\\text{est}}$ in the experiments?\n* Line 21 cites Piantadosi (2014) for the claim that \"For example, Zipf’s law can be used to distinguish between human and model distributions.\" I don't think this is a conclusion of Piantadosi (2014), however. Could you clarify where in the paper he reaches that conclusion? If this is about their comparison with \"random typing models\", those are qualitatively different from language models. When examining proper language models, Meister et al. (2021) reach the opposite conclusion (that language models follow a similar rank-frequency relationship to natural language).\n\nLarger Suggestions: \n* From the paper's text in section 2.2, I interpret that the discrete Fourier transform operates in a single sentence at a time, and thus the proposed metric was developed to compare the spectra of two sentences; not of two corpora. Figure 1, however, implies the Fourier transform takes as input all sentences in a dataset at once. Explaining section 2.2 in more detail could be helpful.\n* Although the authors do discuss this in their paper, I believe the word cross-entropy is not accurate to describe what is being measured here. The word surprisal is the correct one. The authors themselves note this in line 62, but decide to use the term \"cross-entropy\" anyway because it was used this way before, as in, e.g. Genzel and Charniak (2002). Personally, I do not believe this to be a good reason—it's not because prior work used the wrong terminology that you should propagate it.\n\nSmaller Suggestions:\n* Line 156 states that Mauve \"straightforwardly computes\" the similarity of the model- and human-text distributions. However, computing this divergence is actually intractable (starting from the fact that human-text distributions are unknown), and so this is not actually straightforward. They just approximate this using clusters of word embeddings. I’d rewrite this as “attempt to compute” or “estimate”.\n* Line 158 states that reference-based metrics are suited for close-ended generation settings, putting Mauve in that group. Mauve, however, was actually developed to analyse open-ended generation settings. \n* Figure 4 is too small. At the current scale this figure is unreadable.\n\nMeister et al. (2021). Language Model Evaluation Beyond Perplexity.\n", "limitations": "I believe the paper doesn't mention at any point which language their data is in (i.e., English) and that most of the cited psycholinguistics research is English-centric (or at least Indo-European-centric). Addressing that as a potential source of limitation is important.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "I found this paper quite interesting.\nThe motivation was relatively clear and the proposed metric is well-motivated.\nIn particular, operationalising a psycholinguistic hypothesis of what consists human-like text, and then using it to evaluate language generation systems seems like a promising approach.\nFurther, I appreciate the idea of using Fourier transforms to analyse the frequency spectra of information/surprisal in text.", "weaknesses": "I believe that the evaluation part of the paper could be improved:\n* First, section 4.1 discusses how the proposed metric evaluates models of different sizes. (They generate text while prompting models with a few initial tokens from sentences of three datasets.) These results, however, seemed confusing to me; sometimes smaller or larger models are better with no clear explanation, and the main comparison point used is whether or not the proposed metric agrees with mauve. If mauve was to be considered a gold standard, however, we would not need a new metric.\n* Second, section 4.2 evaluates the impact of decoding strategy on the evaluated scores. In these experiments, the authors (coherently) find that their proposed metric always evaluates contrastive decoding as the best strategy. How their evaluation metric fares when comparing other decoding strategies (e.g. nucleus vs ancestral sampling), however, is less clear. Further the table does not show any scores for human text, which I believe could work as an interesting sanity test. These could be computed by evaluating the proposed metric using half of this dataset against its other half.\n* Third, the correlations with human judgement scores in section 4.3 are worse than mauve's. While I do not believe a paper needs to have state-of-the-art scores to be published, the paper does not put forward other reasons why it should be accepted besides these correlations, and treats these negative results as positive in its discussion.\n\nIn summary, this paper focuses on how its proposed evaluation metric produces good scores of what is human-like text, but does not demonstrate to be better than mauve at this. Further, it does not offer any other justifications (besides being a good metric of human-like text) for why one should use it. Together, this makes me think the impact of this paper might be quite limited.\n\nAdding a longer and more detailed comparison between this proposed metric and previously proposed ones could help improve this paper's impact.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Regarding Weakness 1:\n\nRegarding the results from various model sizes, there are indeed some inconsistent cases where small models out-perform big ones. The data from our current experiments are insufficient to explain this inconsistency, but we are planning a more complete future study to address this. The reason for comparing with MAUVE, however, is not that we treat it as a gold standard, but rather as a means to prove the validity of FACE -- in the worst case, the new metric will be totally uncorrelated with existing ones (such as MAUVE), which did not happen in our study. \n\nRegarding Weakness 2:\n\nThe main purpose of having Table 4 in the paper is to show the advantage of FACE in reliably identifying the superior decoding strategies, which, as shown in the previous work (Li et al., 2022), should be contrastive decoding. We admit it is possible that conrtastive decoding may not be the optimal strategy under some conditions, and so we agree with the reviewer that a thorough comparison on other strategies is a plus. \n        Just in case we understand it correctly, if the reviewer suggests that we should look into how FACE scores change when we change the $k$ in top-k sampling from 10 to 640 continuously, or to change the $p$ from 0.1 to 0.995 (as was done in Holtzman et al. 2020), then we agree that these more thorough experiments can better reveal the performance of our metrics. \n        Holtzman et al (2020), i.e., the nucleus sampling paper provided a complete output dataset with a various range of sampling parameters, so we would love to include them in our future investigation. \n\nWe appreciate the suggestion of using half the human data to conduct a sanity test. We have conducted the sanity test and found that human text data have better FACE-SO, -CORR, and -SAM scores, which prove the validity of FACE. Please find the results in the attached PDF document in our general response.\n\nRegarding Weakness 3:\n\nWe recomputed the correlations with human judgement scores to keep those pairs in which there are exactly one item from human and the other item from model (i.e., a subset of data used for the analysis in Table 5 in section 4.3), and found that FACE-SO has stronger correlation than MAUVE in 2 out of the three dimensions (see table below; full results are in the attached PDF in the general response)\n\n                | MAUVE | SO\n    Human-Like | 0.214 | 0.357\n    Interesting | 0.667 | 0.524\n    Sensible    | 0.706 | 0.995\n\nIt indicates that our metric is better than MAUVE in distinguishing human from model languages, rather than between different models. It also leads to the conclusion that Fourier analysis of cross-entropy can identify some hidden difference between human and model generated text, which is highly likely due to the cognitive capability of human beings in language production.\n\nRegarding Question 1: Which model was used to compute $m_{est}$ in the experiments?\n\nGPT2-small (345M parameters) is used.\n\nRegarding Question 2 about Piantadosi (2014)'s work\n\nThanks for pointing out this inaccurate citation. It should be corrected to Holtzman et al. (2020), who used the Zipfian coefficient $s$ to compare the distribution in a given text to a theoretically perfect exponential curve, where $s = 1$. Here Holtzman et al. cited Piantadosi (2014). After checking the original Piantadosi paper, we believe the $s$ coefficient referred by Holtzman should be the $\\alpha$ in Piantadosi (2014). So, technically, the method of using Zip'f coeffient to distinguish human and machine was first introduced by Holtzman (2020). Please correct us if there is our understanding here is wrong. \n\nIt is interesting to read Meister et al. (2021)'s results, which can be potentially useful to our future work.\n\nReference: A. Holtzman, J. Buys, M. Forbes, and Y. Choi. The Curious Case of Neural Text Degeneration. In Proc. of ICLR, 2020\n\nRegarding Larger Suggestion 1:\n\nIt is true that FFT operates on a single sentence at time, and then FACE metrics are computed between two sentences. What Figure 1 shows is the process: we apply FFT to each and every sentence in the corpus, and FACE metrics are computed pair-wise between two shuffled corpora. \n\nRegarding Larger Suggestion 2:\n\nThanks for pointing this out. Yes, `surprisal` is accepted more broadly in psycholinguistics, but we believe `cross-entropy` (CE) is also compatible here, because `cross` indicates CE measures the difference across two distributions, ground-truth and predictions. The way estimate surprisal also needs the use of a well trained model, and to run predictions with the model. The prediction at each word position is a set of probabilities whose size is the entire vocabulary, among which only one probability is picked based on the ground-truth. Thus, we believe the surprisal itself in nature is the cross-entropy between input sentence and the model. Anyway, it is useful and interesting discussion here. "}, {"review_id": "oJSfXQAoYe", "reviewer": "Reviewer_mMGf", "summary": "This paper proposes a set of metrics based on Fourier Analysis of the estimated Cross-Entropy (FACE) of language. The main idea is to compute the similarity between the spectra of cross-entropy in model-generated texts and human-written texts. Experimental results show that FACE as a computationally efficient metric can scale with model size and reflect the outcomes of different sampling methods for decoding.", "questions": "I have included my questions in the weaknesses part.", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea to introduce the spectra of cross-entropy into the evaluation task of open-ended text generation is interesting since it may include some patterns (e.g. periodical patterns) to identify the difference between model-generated texts and human-written texts.\n\n2. This paper is overall well-written and easy to follow.", "weaknesses": "1. The proposed method lacks deeper analysis on the spectrum of cross entropy in the evaluation task. The authors only use the spectrum of cross entropy as a feature vector of texts to compute similarities without clearly describing the characteristics of texts it can reflect. This seems like an empirical try without definite intuitions or theoretical supports. In comparison, the features which are commonly used in the existing metrics such as n-gram statistics (in BLEU) and contextual hidden vectors (in BERTScore) intuitively indicate the surface-level and semantic-level representation of texts, respectively.\n\n2. From Table 5, the performance of SO is still worse than that of MAUVE proposed in 2021. I understand that pursuing SOTA is not necessary for each paper. But the authors should provide more insights into the advantages of SO over MAUVE in other aspects.\n\n3. In Section 4.4, the authors mention that they use GPT-2 of different scales to compute the spectra of GPT-2 output data. I wonder whether this setting can introduce potential bias because the cross entropy may be exceptionally low when using GPT-2 to evaluate its own output data from my experience.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Regarding Weakness 1:\n\nIndeed, in our preliminary research, we conducted a comprehensive analysis of the spectrum of cross entropy and observed its ability to effectively reflect the periodic patterns of high/low entropy words. \nIt is important to note that our work is not just an empirical trial; rather, it is theoretically inspired and supported by relevant literature. References 7 and 46 in our research demonstrate that cross-entropy changes periodically in natural language, which presents a potential means to quantify differences in texts.\n\nIn daily-life language use cases, we may not explicitly pay attention to word entropy, as our primary task is to convey semantic message. \n        To raise an analogy, the process of transmitting messages between speakers is like delivering cargo between two merchants, who care most about `what` has been delivered, that is, the semantic meanings of words, sentences, ... (whether they make sense or not, etc). However, another important factor that determines the transportation efficiency is `how much`, or, the `weight` of the cargo, which corresponds to the entropy of words. To utter and to understand a high-entropy word/sentence is expensive in terms of cognitive effort needed, and thus the merchants (speaker) need to arrange the cargo's weight in a reasonable way, so that it is neither too tiring for the delivery driver (language production devices) nor too busy for the receiver side (language comprehension devices). \n\nThe most relevant previous work in psycholinguistics is from Xu et al. (2017). They pointed out that in natural dialogue, rational speakers should avoid overlapping `peaks` of entropy (cargo weight) to achieve better communication, which can be reflected on the similarity of entropy spectra. \n        It leads to the basic assumption of our study: probably a human speaker (biological being) tends to save effort  by following some periodical patterns in delivering `heavy` words, while a machine speaker does not show such a tendency (because it does not need it).\n\nRegarding Weakness 2:\n\nThanks! We understand and agree with your perspective on evaluating the quality of generated texts. Indeed, it is a subjective task, and different individuals may have varying opinions, making the state-of-the-art (SOTA) evaluation flexible.\n\nOur approach, which focuses on detecting differences in periodicity and cross-entropy, seems to implicitly and profoundly explore human language usage. By delving into these aspects, our research aims to uncover and understand the nuances of how humans use language, which is an essential and valuable contribution to the field. It provides insights into the intricacies of human language expression and recognition, potentially leading to significant advancements in natural language processing and understanding. If needed, we could add examples once our paper is accepted. For more insights, please check our PDF file, which provides corner cases and experimental results demonstrating SO is better on recognising the model-generated and human texts.\n\nRegarding Weakness 3:\n\nThank you for pointing this out. In our preliminary research, we tried GPT-2 with three different sizes, and we found that while the absolute values changed, they did not affect the periodicity of the results.\n\nIt is true that GPT2-xl will result in cross-entropy of smaller absolute values than GPT2-sm, but this will not affect the periodical patterns of cross-entropy. An analogy is that cross-entropy of the same sequence estimated by different models are the same signal amplified to different amplitudes, but they still have the same period. \n"}, {"review_id": "r6znKDQNVq", "reviewer": "Reviewer_s437", "summary": "This paper proposes a new measure of natural language generation (NLG) quality based on similarity between the spectrum of cross-entropy in natural vs. generated text. Fourier Analysis of the Cross-Entropy of language (FACE) is inspired by NLP and psycholinguistic studies suggesting that surprisal is not uniformly distributed in natural text (e.g., content words tend to be more surprising than function words), occurring periodically. For a given generated text, FACE computes a discrete Fourier transform of the token-level cross-entropy sequence (under a separate FACE evaluation LM). Similarity between the vector of frequency magnitudes and that from a randomly selected, natural text corpus are then computed. The paper considers several definitions of FACE metrics, including spectral overlap, cosine similarity, and Pearson/Spearman’s rank correlation coefficients.\n\nLMs from 125 million to over 7 billion parameters are evaluated on NLG of Wikipedia articles, news articles, and stories (with a short prompt of 35 subword tokens provided). Ultimately, FACE is found to be correlated with human judgments of how “human-like”, “sensible”, and “interesting” the generations are. The relationship is not as strong as an existing intrinsic measure, MAUVE. The relative ranking of decoding methods according to FACE agrees with prior works (e.g., greedy decoding < nucleus), as do model size (smaller models produce lower quality generations than larger models).", "questions": "Have you also considered the spectrum of hidden LM embeddings rather than cross-entropy, and considered how such a metric might differ from FACE?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The metric is well-motivated, evaluating whether generated text matches the surprisal statistics of natural text. The algorithm is simple and described sufficiently clearly. FACE is an automatic measure of NLG quality that is, on the face of it, complementary to existing measures. This paper would be of interest to many who work on (large) language models.", "weaknesses": "While FACE is motivated by the desire to match surprisal statistics of natural text, it was not clear how different FACE is from existing metrics. Computing correlation between FACE and existing metrics would help alleviate this, as would providing anecdotes of cases with high/low FACE score vs. high/low MAUVE score, for instance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Regarding Weakness:\n\nThank you for your review. If needed, we can include examples in our paper once it is accepted. It is important to note that our approach considers the distinctions between human and model-generated languages in terms of cross entropy and periodicity, which sets it apart from most current mainstream metrics. Our research delves into the implicit aspects of human language usage, for example, exploring the periodicity of high/low-frequency words. For more insights, please check our PDF file, which provides corner cases and experimental results demonstrating SO is better on recognising the model-generated and human texts.\n\nRegarding Question:\n\nOur approach is designed based on the theory of FFT (Fast Fourier Transform), making the investigation of periodicity most intuitive and sensible using cross-entropy as inputs. Using hidden embeddings may induce other problems, such as defining the strength of a signal or making analogies. Additionally, our aim is to investigate the periodicity of high/low-frequency words, making the use of cross-entropy more appropriate."}, {"review_id": "EfNxKCHutS", "reviewer": "Reviewer_wEMM", "summary": "In order to distinguish between human-generated text and machine-generated text, the authors propose the use of the periodicity of cross entropy for discrimination. More specifically, they suggest analyzing cross entropy through the Fourier transform.", "questions": "see weakness", "limitations": " the authors adequately addressed the limitations", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-written and easy to follow.\n2. The experimental section of this paper is fairly comprehensive.  The authors' experimental objects have broadly encompassed the latest open-source large models. Although it lacks large language models like GPT-3.5 (the cross entropy can still be obtained through APIs).\n\n", "weaknesses": "1. Motivation. The motivation of the paper is not clear, as the authors do not clearly explain why the CE of human language would exhibit periodicity. In the related work section, they briefly mention previous works, but in my view, dialogue tasks are just a specific case of text generation. Overall, skipping the motivation part significantly reduces the soundness of this paper.\n\n2. Method. The authors' method simply involves applying a FFT to the CE sequences, which I believe lacks substantial novelty. Why haven't the authors considered using the information in the frequency domain as input to a deep neural network to incorporate a powerful NN? Why only analyze information in the frequency domain using spectral similarity metrics? Additionally, most of these metrics have already been presented in [1]. Which method would better utilize this information for discrimination? In conclusion, the proposed method by the authors lacks both sufficient contribution and profound insight.\n\n3. Experiments.  In the experimental section, the authors did not compare against sufficient baselines. For instance, could we achieve good results by only training a contrastive model using human-generated text and LLM-generated text? How helpful is the frequency domain information in discriminating texts?\n\n[1] Y. Xu and D. Reitter. Spectral analysis of information density in dialogue predicts collaborative task performance. ACL", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Regarding Weakness 1:\n\nPlease read our general response where we addressed the motivation issue. \n\nRegarding Weakness 2:\n\nActually applying FFT to CE sequences is our biggest innovation. First of all, as we discussed in detail in the general response, applying Fourier analysis on cross-entropy sequence is motivated by two pieces of previous work, Xu et al. (2017) and Dethlefs et al. (2016), in which the periodical patterns of cross-entropy (or surprisal, information density etc.) is discovered (Xu et al., 2017), and speakers are sensitive to the peaks and troughs of cross-entropy in human-machine dialogue (Dethlefs et al., 2016).\n        These evidence all pointed out the potentials of using frequency-domain information to distinguish human and machine languages. To the best of our knowledge, this perspective is never explored before in the community of natural language processing or AI in general.\n\nRegarding the reason of why not using neural networks to take the spectra as input and cast it as a classification problem, we would like to argue that this paper is to answer \"whether-or-not\", rather than \"how\" or \"how good\". Indeed we have positive results on using the spectral features to classify human vs. model-generated languages, and this could potentially lead to a next step of making the most out of the spectral features by adding stronger NN-based models. But so far at this point, we believe that the current content of the paper has done its job in presenting all the promising proof-of-concept results.\n\nRegarding Weakness 3:\n\nThanks for your review. In our experimental setting, we regard all the other existing open-ended text generation metrics as our baselines, which corresponds to the `metric` column in Table 3, as well as the heading rows in Table 4 and Table 5, respectively. Indeed, leveraging a \"trained and tuned\" contrastive model can be a reasonable baseline. However, considering all these widely-adopted metrics in open-ended text generation research, it is sufficient to claim that our proposed FACE metrics have explicable consistency compared to previous work and comparable performance compared to the state-of-the-art MAUVE. If needed, we could add the results of our hand-crafted baseline (e.g., evaluation scores from a contrastive model) to existing tables once our paper is accepted.\n\nSpeaking of the advantages of using frequency domain knowledge for distinguishing generated texts, one can refer to the vivid analogy which we present to Reviewer mMGf (quoted as follows): \n\n```\n the process of transmitting messages between speakers is like delivering cargo between two merchants, who care most about `what` has been delivered, that is, the semantic meanings of words, sentences, ... (whether they make sense or not, etc). However, another important factor that determines the transportation efficiency is `how much`, or, the `weight` of the cargo, which corresponds to the entropy of words. To utter and to understand a high-entropy word/sentence is expensive in terms of cognitive effort needed, and thus the merchants (speaker) need to arrange the cargo's weight in a reasonable way, so that it is neither too tiring for the delivery driver (language production devices) nor too busy for the receiver side (language comprehension devices). \n```\n\nIn short, utilizing the cross-entropy sequences from the frequency spectra of texts can effectively perceive periodic patterns of high/low-entropy words and thus quantify the difference between two families of generated texts without concerning their surface-level or semantic-level representation. Moreover, our approach is computationally efficient as it only requires cross-entropy inferences with a pre-trained language model $m_{est}$, getting rid of the tedious and time-consuming feature-space operations (e.g., encoding texts into feature space before computing the cosine similarity).\n"}]

Author rebuttal:
We would like to thank all the reviewers for providing the useful feedback for further improving our paper. We notice that some reviewers suggest us to strengthen the motivation part, especially the reasons of using Fourier analysis on the cross-entropy sequence of language. We believe that the motivation is sufficiently done in the Introduction and Related Work sections, but probably due to that some of the reviewers are not from psycholinguistics background and thus lack the bigger context, so here we would like to compose a general response to better address this paper's motivation with as plain language as possible. 

The most relevant two pieces of work from previous computational psycholinguistics studies are Xu et al. (2017) and Dethlefs et al. (2016). Xu's main finding is that the cross entropy of dialogue utterances has periodogical patterns, which can be used to predict the task success. Dethlefs' results tell about that human speakers' experiences are sensitive to the peaks and troughs in utterances from a human-machine dialogue system. Combining their findings, we distilled our main assumption/idea: capturing the periodical patterns of cross-entropy with FFT, and then test whether the spectral features is an indicator of "good" and "natural" language. It is following this assumption that we complete the entire study step by step. 

As for why cross-entropy works, if we are to give a brief explanations of our assumption/, we would say it reflects human beings cognitive capability of producing and comprehending languages. This capability naturally fluctuates in time as humans adapt to information dynamically based on its cognitive load, which produces the temporal periodical pattern of cross-entropy in language. Machines/models, on the other side, do not present such a pattern because it does not have the cognitive load issue. We believe the current way models are trained i.e., via maximum-likelihood estimation-based learning of which words produced at when, does not capture the dynamics of cognitive load behind the curtain. 
    Perhaps reviewers outside the field will find our response to Reviewer mMGf an interesting explanation on why it works to use cross-entropy for evaluating language quality. 

To sum, this paper is to answer "whether-or-not", rather than "how" or "how good". Fortunately we have positive results on using the spectral features to tell apart human vs. model-generated languages, and this could potentially lead to a next step of making the most out of the spectral features by adding stronger predicting models (as one of the reviewers suggest). But so far at this point, we believe that the current content of the paper has done its job in presenting all the promising proof-of-concept results.  

In the attached PDF document, we included three experiments as suggested by the reviewers: 1) Sanity test of FACE by splitting human text data in half. 2) Corner cases of text whose MAUVE scores are higher than FACE, but actually did not read well; some qualitative analysis is included. 3) FACE-SO show higher correlation with human judgement scores than MAUVE, when we use a more reasonable subset of evaluation data.

Meta-review: The paper proposes an evaluation metric for natural language generation. The proposed methodology is novel. The evaluation section is rather weak and several reviewers have highlighted it. The paper can easily be improved and strengthened based on the detailed comments and discussion done with Reviewer AtQ2 and mMGf. I intend to accept the paper since the use of Fourier analysis is valuable for the community. I would encourage authors to incorporate the comments of the reviewers in the final version of the paper.
Decision: Accept (poster)