SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table A3: Performance evaluation (AUROC) with RADAR-Unseen Paraphraser.
Method	Dataset	Target Language Model
Pythia-2.8B	Dolly-V2-3B	Palmyra-base	Camel-5B	GPT-J-6B	Dolly-V1-6B	LLaMA-7B	Vicuna-7B
log p	Xsum	0.173	0.285	0.212	0.530	0.141	0.237	0.144	0.408
SQuAD	0.247	0.413	0.328	0.550	0.241	0.305	0.238	0.425
WP	0.572	0.665	0.603	0.772	0.562	0.641	0.546	0.767
TOFEL	0.256	0.436	0.305	0.701	0.295	0.442	0.413	0.653
rank	Xsum	0.429	0.429	0.478	0.530	0.382	0.437	0.414	0.368
SQuAD	0.42	0.463	0.480	0.486	0.400	0.431	0.410	0.395
WP	0.685	0.676	0.691	0.662	0.580	0.693	0.497	0.569
TOFEL	0.323	0.377	0.310	0.397	0.335	0.366	0.300	0.326
log rank	Xsum	0.201	0.293	0.248	0.560	0.164	0.264	0.150	0.376
SQuAD	0.289	0.426	0.372	0.595	0.272	0.340	0.245	0.425
WP	0.579	0.644	0.611	0.771	0.569	0.629	0.525	0.731
TOFEL	0.263	0.405	0.305	0.673	0.296	0.445	0.376	0.602
entropy	Xsum	0.846	0.743	0.804	0.525	0.881	0.788	0.898	0.747
SQuAD	0.801	0.650	0.726	0.537	0.778	0.725	0.808	0.651
WP	0.556	0.513	0.545	0.306	0.573	0.512	0.587	0.396
TOFEL	0.735	0.597	0.737	0.530	0.739	0.577	0.590	0.436
DetectGPT	Xsum	0.260	0.389	0.291	0.521	0.217	0.254	0.213	0.374
SQuAD	0.288	0.447	0.343	0.554	0.252	0.288	0.204	0.311
WP	0.404	0.518	0.439	0.652	0.393	0.393	0.305	0.550
TOFEL	0.459	0.662	0.573	0.827	0.572	0.618	0.582	0.746
OpenAI (RoBERTa)	Xsum	0.885	0.809	0.899	0.687	0.824	0.764	0.774	0.668
SQuAD	0.727	0.597	0.757	0.486	0.665	0.638	0.612	0.552
WP	0.791	0.708	0.806	0.596	0.780	0.748	0.718	0.663
TOFEL	0.453	0.265	0.511	0.308	0.388	0.309	0.348	0.332
RADAR	Xsum	0.966	0.957	0.976	0.996	0.904	0.985	0.910	0.947
SQuAD	0.872	0.873	0.873	0.988	0.763	0.934	0.698	0.887
WP	0.843	0.898	0.824	0.989	0.726	0.965	0.655	0.907
TOFEL	0.784	0.775	0.772	0.918	0.634	0.809	0.584	0.827
Dolly-V1-6B, Camel-5B, Dolly-V2-3B, LLaMA-7B, GPT-J-6B, Palmyra-base, Pythia-2.8B) to generate completions for each text, then use GPT-3.5-Turbo API service to perform paraphrase to all completions. Thus, we get 64 AI-texts in total (32 completions, 32 paraphrases).
Case Selection. Each AI-text has a source model, for the detection of one given text, we use the RADAR-detector trained for its source model. Then, we collect the one with the largest confidence to be machine-generated (most likely to be correctly detected) and the one with the smallest confidence to be machine-generated (mostly likely to evade detection) for the following case study.
Analysis. We show two detection cases in Table A5. One with a higher probability (0.9999) is an easy-to-detect case, and another with a lower probability (0.0031) is a difficult-to-detect case. Specifically, the latter can only be detected when our detection threshold drops below 0.0031. From the table, we can see that the misclassification has a good explanation, because the AI-text is nearly identical to the original Human-text, except for the inclusion of several additional words. In fact, the AI-text can be seen as the Human-text with a suffix composed of several words appended to it.
Table A4: Performance evaluation (AUROC) with RADAR-Seen Paraphraser. Red text means the evaluating method is not capable to perform detection for some samples in the evaluating dataset and the reported value is the detection AUROC after removing these samples.
Method	Dataset	Target Language Model
Pythia-2.8B	Dolly-V2-3B	Palmyra-base	Camel-5B	GPT-J-6B	Dolly-V1-6B	LLaMA-7B	Vicuna-7B
log p	Xsum	0.269	0.191	0.287	0.328	0.217	0.174	0.142	0.228
SQuAD	0.279	0.057	0.248	0.233	0.158	0.065	0.081	0.130
WP	0.555	0.165	0.347	0.296	0.271	0.182	0.179	0.203
TOEFL	0.251	0.025	0.204	0.153	0.172	0.052	0.087	0.099
rank	Xsum	0.429	0.261	0.410	0.387	0.340	0.251	0.324	0.267
SQuAD	0.502	0.149	0.377	0.301	0.293	0.132	0.259	0.243
WP	0.631	0.258	0.439	0.329	0.333	0.267	0.291	0.305
TOEFL	0.326	0.042	0.215	0.140	0.239	0.062	0.177	0.106
log rank	Xsum	0.306	0.197	0.320	0.339	0.241	0.189	0.154	0.214
SQuAD	0.328	0.064	0.280	0.244	0.184	0.075	0.091	0.134
WP	0.582	0.167	0.366	0.292	0.282	0.182	0.187	0.187
TOEFL	0.267	0.025	0.227	0.144	0.181	0.057	0.085	0.085
entropy	Xsum	0.778	0.783	0.772	0.647	0.814	0.851	0.871	0.848
SQuAD	0.771	0.903	0.769	0.728	0.850	0.930	0.913	0.895
WP	0.612	0.835	0.708	0.693	0.760	0.840	0.804	0.852
TOEFL	0.813	0.952	0.825	0.793	0.879	0.935	0.891	0.922
DetectGPT	Xsum	0.270	0.214	0.280	0.284	0.224	0.129	0.078	0.143
SQuAD	0.231	0.093	0.201	0.181	0.111	0.030	0.027	0.043
WP	0.258	0.135	0.192	0.223	0.073	0.061	0.025	0.066
TOEFL	0.327	0.107	0.322	0.237	0.187	0.081	0.040	0.111
OpenAI (RoBERTa)	Xsum	0.844	0.839	0.829	0.716	0.786	0.830	0.883	0.842
SQuAD	0.851	0.905	0.871	0.758	0.830	0.835	0.847	0.838
WP	0.887	0.903	0.918	0.856	0.901	0.898	0.895	0.881
TOEFL	0.711	0.621	0.692	0.610	0.716	0.681	0.681	0.649
RADAR	Xsum	0.879	0.920	0.913	0.913	0.890	0.945	0.972	0.924
SQuAD	0.917	0.926	0.917	0.856	0.904	0.953	0.988	0.958
WP	0.873	0.868	0.953	0.724	0.974	0.959	0.990	0.925
TOEFL	0.969	0.894	0.926	0.809	0.952	0.947	0.994	0.964
G Effectiveness of Ensembling Detectors from RADAR
We study the ensembling detection performance by combining two RADAR-detectors' prediction probability and report the detection AUROC calculated using the combined prediction probability. The combined prediction probability  $E(A, B, \beta, x) = (1 - \beta) \mathcal{D}_{\phi}^{A}(x) + \beta \mathcal{D}_{\phi}^{B}(x)$  is a weighted sum of the prediction probability of the base model  $A$  and the augmented model  $B$ . The detection performances are shown in Table A6. We explore various ensembling ratios and ensembling models. Ensembling with the base model itself and setting the ensembling ratio to 0 both mean no ensembling. Setting the ensembling ratio to 1 is another extreme case, which refers to the transfer detection scheme mentioned in Section 4.3. From the results, we can see that the ensembling detection's effectiveness can be influenced by both the ensemble model and the ensemble ratio.
Table A6: Ensembling Detection results averaged on 4 datasets. Ensembling detection means combining base model  $A$ 's output prediction and augmented model  $B$ 's output prediction to detect  $A$ 's generation. The ensemble ratio  $\beta$  varies from 0 to 1. Red text represents a better detection AUROC than that of no-ensembling
Base Model	Ensemble Ratio	Augmented Model
Vicuna-7B	Dolly-V1-6B	Camel-5B	Dolly-V2-3B
Vicuna-7B	0	0.906
0.5	0.906	0.861	0.866	0.814
1	0.906	0.822	0.749	0.684
Dolly-V1-6B	0	0.859
0.5	0.884	0.859	0.846	0.829
1	0.877	0.859	0.722	0.764
Camel-5B	0	0.920
0.5	0.929	0.926	0.92	0.897
1	0.882	0.883	0.92	0.764
Dolly-V2-3B	0	0.769
0.5	0.803	0.763	0.753	0.769
1	0.794	0.727	0.642	0.769
(a) w/o paraphrase
(b) RADAR-Unseen paraphrase
Figure A2: Detection AUROC of RADAR toward AI-generated texts with different lengths. The reported value is the AUROC of 8 LLMs averaged over 4 human-text datasets. The value on the X-axis means the texts' relative length rank position in the whole evaluation dataset. For example,  $20\%$  means the shortest  $20\%$  sentences.
H Text Detection Towards Different Lengths
We study RADAR's effects on AI-generated texts with different lengths. We grouped the evaluation dataset into 5 subsets according to the length of the AI-text. The results are shown in Figure A2. We summarize our observations below:
- For the group {log probability, rank, log rank, DetectGPT}, without paraphrasing, these methods are not really sensitive to the length of the text. When facing paraphrasing, however, their performance increases with a longer text length.
- For the group {entropy, OpenAI (RoBERTa), RADAR}, without paraphrasing, these methods have a better detection performance for longer texts. On the contrary, their performance degrades when facing longer paraphrased AI-text (even though RADAR seems much better for short-text detection, it still outperforms other methods by a large margin, see Figure A2b).
I Detection on Dipper Paraphrasing
We also explore the use of RADAR to detect other advanced paraphrasers. We use Dipper proposed in [18] (L60-O60 version) to paraphrase the evaluation dataset following the same setup of RADAR-Unseen paraphrasing and RADAR-Seen paraphrasing. The results are shown in Figure A3. We can see from the results that the green bars are much higher than the red bars except for DetectGPT, which means in general, Dipper is less destructive than GPT-3.5-Turbo to these detectors. RADAR's detection AUROC on Dipper reaches 0.9. One thing to be noted is that though the OpenAI (RoBERTa) can perform well under Dipper paraphrasing, it still cannot be regarded as robust because it can be bypassed by other paraphrasers (RADAR-Seen and RADAR-Unseen).
Figure A3: Augmented version of Figure 2. The added green bar represents RADAR's detection performance under Dipper paraphrasing.
J Sensitivity Analysis of RADAR on the Hyperparameter  $\lambda$
Please refer to our discussion in Section 4.6.
Figure A4: Sensitivity analysis for  $\lambda$  in Eq. (3). The RADAR models are trained with Vicuna-7B with different  $\lambda$  values ranging from 0.1 to 1.0. We then compare these RADAR detectors with other baselines on Vicuna-7B's 4 evaluation datasets. The horizontal/vertical axis shows the AUROC of unperturbed/paraphrased AI-text.
K Detection on GPT-4 Generated Texts
Figure A5 shows RADAR's transfer detection for GPT-4. We use RADAR detectors trained on weaker LLMs (Vicuna-7B, Camel-5B, etc.) to detect the generation of GPT-4. We prompted GPT-4 with the instruction: You are a helpful assistant to complete given text:, to generate texts based on the same evaluation datasets (Xsum, SQuAD, WP and TOFEL) used when evaluating other LLMs. The detection performance have been discussed in Section 4.3.
Figure A5: RADAR detection performance on GPT-4. The detector is trained with a source model, and the evaluation is measured with the texts generated from GPT-4.
L Use GPT-3.5 to Assess RADAR-paraphrasers
After RADAR training, we not only get a detector but also a paraphraseer. We use GPT-3.5-Turbo to assign a score to these paraphraseers' generation to assess the language generation capability of these paraphraseers and compare with their initial version (T5-large) to see how adversarial training benefits them. We compare these paraphraseers on a WebText subset with 100 samples. For each sentence in this subset, we first use GPT-2-XL to generate one sentence, and then use the 5 paraphraseers (T5-large and 4 paraphraseers trained on 4 instruction-tuned models' generation) to paraphrase this sentence respectively. Then we input these 5 paraphrased sentences combining an instruction to GPT-3.5-Turbo. Our instruction is You are given an array of five sentences. Please rate these sentences and reply with an array of scores assigned to these sentences. Each score is on a scale from 1 to 10, the higher the score, the sentence is written more like a human. Your reply example: [2,2,2,2,2]. Then we sniff the score for each sentence from the returned answer and make the comparison. The results have been discussed in Section 4.5.