SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We examine how different choices of estimator model  $m_{\mathrm{est}}$  affect the resulting spectra of cross-entropy. Five input data sources are examined (webtext plus four GPT2 original output datasets), on which four different estimator models are applied:  $m_{\mathrm{est}} \in \{\mathrm{GPT2 - sm}, \mathrm{GPT2 - md}, \mathrm{GPT2 - lg}, \mathrm{GPT2 - x1}\}$ , resulting in  $5 \times 4 = 20$  aggregated spectra curves in Figure 8. It can be found that on the same input data, the spectra from four estimators largely overlap. It indirectly suggests that FACE should be stable across different  $m_{\mathrm{est}}$ s. We leave the full inspection for future work.
4.5 Intuitive interpretation of spectra
As pointed out in Section 4.5, the aggregated spectral shapes from human and different models are nearly identical. A set of higher resolution plots from GPT-x1, OPT, BLOOM and human (webtext) are shown in Figure 9. It can be seen that although the  $X(\omega_{k})$  has different ranges on  $y$ -axis, the  $x$  coordinates of the peaks and troughs are the same.
4.6 Corner Cases
Two examples highlight the difference between our proposed FACE-SO and MAUVE in their ability to recognize human-generated and model-generated texts. In the filtered datasets for human judgments, the average values for FACE-SO and MAUVE are 0.4738 and 0.9549, respectively. In Case 1, human evaluators noted a high level of similarity between the model-generated text and human text, resulting
$^{6}$ https://github.com/krishnap25/mauve-experiments
Figure 8: Aggregated spectra (using GAM smoothing) from four estimator models  $m_{\mathrm{est}} \in \{\mathrm{GPT2 - sm}, \mathrm{GPT2 - md}, \mathrm{GPT2 - lg}, \mathrm{GPT2 - x1}\}$ . Inputs are from GPT2 original output and webtext.
Figure 9: Aggregated spectra for GPT-x1, OPT, BLOOM, and human (webtext).
in ties for human-like, interesting, and sensible aspects. However, the MAUVE score in Case 1 is lower than the average value, while the FACE-SO score surpasses its mean. This discrepancy suggests that SO aligns more consistently with human opinions. Conversely, in Case 2, human judgement indicates a significant dissimilarity between the model-generated text and human text, making them easily distinguishable. However, the MAUVE score exceeds its mean, suggesting the two texts are similar to each other. Our FACE-SO score is lower than its mean, indicating better alignment with human opinion.
Figure 10: Example of two corner cases. For each case, the prompt text, model-generated text, human text, MAUVE and FACE-SO scores, as well as the results from human judgments are tabulated.