SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 28: Accuracy for different tasks with counterfactual demonstrations.
Dataset	Counterfactuals	Model	Zero-shot	CF	Demo	Demo+CF
SNLI-CAD	SNLI-RP	GPT-3.5	0.74	0.90	0.83 ± 0.01	0.85 ± 0.02
GPT-4	0.90	0.89	0.91 ± 0.02	0.91 ± 0.01
SNLI-RH	GPT-3.5	0.75	0.88	0.84 ± 0.01	0.88 ± 0.02
GPT-4	0.90	0.90	0.92 ± 0.01	0.92 ± 0.01
MSGS	main Verb	GPT-3.5	0.49	0.57	0.51 ± 0.01	0.61 ± 0.04
GPT-4	0.62	0.84	0.76 ± 0.11	0.86 ± 0.05
syntactic_category	GPT-3.5	0.55	1.00	0.81 ± 0.05	0.92 ± 0.06
GPT-4	0.81	0.99	0.97 ± 0.01	1.00 ± 0.00
control_raising	GPT-3.5	0.50	0.53	0.52 ± 0.01	0.84 ± 0.06
GPT-4	0.53	0.91	0.54 ± 0.04	0.87 ± 0.04
irregular_form	GPT-3.5	0.63	0.91	0.56 ± 0.02	0.86 ± 0.06
GPT-4	0.82	0.96	0.89 ± 0.01	0.94 ± 0.02
Figure 22 shows an example of adding a counterfactual example at the end of the demonstration. By comparing the performance between  $Zero - \text{shot}$  and  $CF(x)$ , and the performance between  $Demo$  and  $Demo + CF(x)$ , we can find out how the counterfactual examples would affect model predictions. We repeat three times for randomly sampling the demonstration examples in  $Demo$  and  $Demo + CF(x)$ , and report the accuracy scores.
Results. The results on different tasks with counterfactual demonstrations are shown in Table 28. On SNLI-CAD datasets, including the counterfactual example of the test input in the demonstration improves the performance of GPT-3.5, and the performance of GPT-4 is basically unchanged. It suggests both GPT-3.5 and GPT-4 are not misled by counterfactual demonstrations. On four linguistic task from MSGS dataset, we find that including the counterfactual example significantly improves the model performance for both GPT-3.5 and GPT-4, which indicates that they can understand the difference between the input text and its counterfactual text according to the task descriptions.
- Both GPT-3.5 and GPT-4 are not misled by the counterfactual example in the demonstrations.
- GPT-3.5 and GPT-4 will benefit from counterfactual demonstrations in general.
Table 29: Six heuristic types from the HANS dataset that we used to construct spurious correlations in our experiments. For each heuristic type, we provide an entailment example and a non-entailment example.
Heuristic Type	Label	Example
Passive (passive voice)	Entailment	Premise: The authors were supported by the tourist . Hypothesis: The tourist supported the authors.
Non-entailment	Premise: The managers were advised by the athlete . Hypothesis: The managers advised the athlete.
L_RC (lexical overlap: relictive clause)	Entailment	Premise: The judges recommended the tourist that believed the authors. Hypothesis: The tourist believed the authors.
Non-entailment	Premise: The actors who advised the manager saw the tourists. Hypothesis: The manager saw the actors.
S_RC (subsequence: relative clause)	Entailment	Premise: The managers admired the authors who called the actor. Hypothesis: The managers admired the authors
Non-entailment	Premise: The artists that supported the senators shouted . Hypothesis: The senators shouted.
PP (prepositional phrase)	Entailment	Premise: The secretaries advised the senators by the athletes. Hypothesis: The secretaries advised the senators.
Non-entailment	Premise: The managers next to the professors performed . Hypothesis: The professors performed.
Verb (embedded under verb)	Entailment	Premise: The professors knew that the students ran . Hypothesis: The students ran.
Non-entailment	Premise: The lawyers believed that the tourists shouted . Hypothesis: The tourists shouted.
Adverb (adverb differences)	Entailment	Premise: Clearly the author encouraged the actors . Hypothesis: The author encouraged the actors.
Non-entailment	Premise: Hopefully the presidents introduced the doctors . Hypothesis: The presidents introduced the doctors.
G.2 Robustness against spurious correlations in demonstrations
Here we aim to explore if LLMs would be misled by demonstration examples with designed spurious correlations. Spurious correlations represent features that are statistically associated with the target labels but not causally related.
Data. We construct spurious correlations based on the fallible heuristics provided by the HANS dataset [113]. The HANS dataset is a commonly used challenging dataset for examining spurious correlations on the Natural Language Inference (NLI) task. It annotates a heuristic subcase (e.g. "ce_adverb") for each example. Based on the annotated heuristic subcases, we first construct six paired heuristic subsets where the examples display the same heuristic type. Each heuristic type describes a superficial property of the relationship between the premise and the hypothesis. For example, heuristic type "Adverb" indicates that the difference between the premise and the hypothesis is an adverb. As shown in Table 29, the six heuristic types we use in the experiments are "Passive", "L_RC (lexical_overlap: relative_clause)", "S_RC (subsequence: relative_clause)", "PP (prepositional phrase)", "Verb (embedded_under Verb)" and "Adverb".
Based on each heuristic type, we form two types of demonstrations with spurious correlations: entailment-correlated and non-entailment-correlated demonstrations. For a target heuristic type, we construct an entailment-correlated demonstration by randomly sampling 8 entailment examples, which display this heuristic type, and randomly sampling 8 non-entailment examples from the SNLI dataset [23]. As a result, an entailment-correlated demonstration with 16 examples exhibits a spurious correlation that the target heuristic type leads to entailment. Similarly, we can construct a non-entailment-correlated demonstration, which exhibits a spurious correlation that the target heuristic type leads to non-entailment, following the above strategy.
Evaluation setup. For each heuristic type, we evaluate the entailment-correlated demonstration and the non-entailment-correlated demonstration on its paired heuristic evaluation subset, respectively.
The paired heuristic evaluation subset of each heuristic type consists of 1000 entailment cases and 1000 non-entailment cases which display that heuristic type, and this ensures that each heuristic type is not causally related to the label in the test set. We report the overall accuracy and also report the prediction gap between the accuracy of entailment cases and the accuracy of non-entailment cases  $|\Delta| = |Acc_e - Acc_n|$ . For each type of demonstration, we randomly sample demonstration examples three times.
When we use a demonstration with a spurious correlation based on a heuristic type, there are two types of possible outputs of models: 1) The model is misled by the spurious correlations in the demonstrations. Since both entailment examples and non-entailment examples in the evaluation subset display the same heuristic type, the model will predict the inputs as the class which correlates to the spurious heuristic type in the demonstration. As a result, the overall accuracy on the heuristic evaluate subset would drop and the prediction gap between the two balanced classes would be large compared to the zero-shot setting. 2) The model is able to identify the true causal features and will not be affected or even benefit from the demonstrations with the spurious correlation. As a result, the overall accuracy on the heuristic evaluate subset would not drop, and the prediction gap between the two balanced classes would be small compared to the zero-shot setting.
Results. Table 30 shows the model performance given demonstrations with spurious correlations based on different heuristic
types. For each heuristic type, Figure 23 further shows the ratio at which the overall model accuracy with demonstration containing a spurious correlation is lower than that in zero-shot setting, indicating that the predictions are misled by the spurious correlations. First, we find that different types of spurious correlations have different impacts on model predictions. In terms of NLI, the spurious correlations based on the heuristics "Verb" and "Passive" in the demonstration can mislead the predictions of GPT-3.5 and GPT-4. For example, GPT-4 is misled by the "Verb" spurious correlation via non-entailment-correlated demonstrations and makes totally biased predictions. This highlights the risks of GPT models potentially overfitting to the spurious correlations in the demonstrations. On the other hand, the spurious correlations based on the heuristic "L_RC" has a small impact on both GPT-3.5 and GPT-4.
We find that GPT-3.5 is more easily misled by the spurious correlations in the demonstrations than GPT-4 on the NLI task. For instance, the performance of GPT-3.5 on the heuristic subset "S_RC" drops when we use the entailment-correlated demonstration, while GPT-4 is able to identify the true causal features in the demonstrations with the spurious correlations and improves the overall performance on that heuristic evaluation subset.
Figure 23: The prediction ratio at which the overall model prediction accuracy with demonstrations containing spurious correlations is lower than that in the zero-shot setting, indicating that the model is misled by spurious correlations in demonstrations.
- Different types of spurious correlations have different impacts on model predictions.
- Certain kinds of spurious correlations exhibited in the demonstration (e.g., heuristic "Verb" in the NLI task) would mislead GPT models to make worse predictions. For some other spurious correlations (e.g., heuristic "L_RC"), however, GPT models can still recognize the underlying causal features from the demonstrations and leverage them to improve the model performance.
- GPT-3.5 is more likely to be misled by the spurious correlations in the demonstrations than GPT-4 on the NLI task.
G.3 Robustness against backdoors in demonstrations
Detailed evaluation setup We further design more experiments to understand how backdoored demonstrations would affect the model predictions.
Experiment I: different backdoor approaches under diverse backdoor setups. We use four backdoor generation approaches to add different backdoors into the demonstrations following Open-Backdoor [41]: BadWord [34], AddSent [43], SynBkd [138] and StyleBkd [137]. BadWord randomly inserts two irregular tokens ("cf") to the original texts. AddSent inserts a neutral sentence ("I watch this 3D movie") to the original texts. SynBkd paraphrases normal texts into sentences with a pre
Table 30: Model performance given demonstrations with spurious correlations from different heuristic types.  $|\Delta | = |Acc_{e} - Acc_{n}|$  characterizes the accuracy gap between entailment and non-entailment examples.
Heuristic	Model	Zero-shot	Entailment-correlated	Non-entailment-correlated
Acc	|Δ|	Acc	|Δ|	Acc	|Δ|
Passive	GPT-3.5	1.00	0.01	0.97±0.01	0.06±0.02	0.95±0.03	0.08±0.06
GPT-4	1.00	0.00	1.00±0.00	0.00±0.00	1.00±0.00	0.00±0.00
L_RC	GPT-3.5	0.90	0.16	0.96±0.02	0.07±0.04	0.90±0.03	0.09±0.05
GPT-4	0.98	0.02	1.00±0.00	0.01±0.00	0.99±0.00	0.01±0.00
S_RC	GPT-3.5	0.91	0.10	0.83±0.09	0.23±0.20	0.90±0.02	0.06±0.05
GPT-4	0.95	0.09	1.00±0.00	0.01±0.01	1.00±0.00	0.00±0.00
PP	GPT-3.5	0.89	0.16	0.92±0.06	0.11±0.11	0.85±0.05	0.22±0.16
GPT-4	0.96	0.08	1.00±0.00	0.00±0.00	1.00±0.00	0.00±0.00
Verb	GPT-3.5	0.59	0.81	0.56±0.03	0.86±0.07	0.78±0.02	0.30±0.11
GPT-4	0.58	0.84	0.67±0.10	0.66±0.20	0.51±0.02	0.98±0.03
Adverb	GPT-3.5	0.57	0.85	0.54±0.04	0.92±0.07	0.80±0.08	0.39±0.16
GPT-4	0.85	0.29	0.80±0.16	0.39±0.32	0.97±0.02	0.05±0.04
specified syntactic structure ("S(SBAR)(,)(NP)(VP)(.)"). StyleBkd manipulates texts by transforming the text style to Bible style.
We use "positive" as the target class and adopt the following three backdoor setups to form the backdoorsed demonstration examples.
- Setup 1: We randomly select 16 demonstration examples. Among them, we randomly choose 8 of them to inject the trigger and change their labels to the target class (i.e., positive).
- Setup 2: We randomly select 16 negative demonstration examples. Among them, we randomly choose 8 of them to inject the trigger and change their labels to the target class (i.e., positive).
- Setup 3: We randomly select 16 demonstration examples. We inject the trigger to all demonstration examples and make all the labels the target class (i.e., positive).
For each backdoor approach and backdoor setup, we evaluate the attack success rate (ASR) and clean accuracy (CACC). Attack success rate refers to the accuracy of a backdoored testing set. Clean accuracy stands for the accuracy of a clean testing set. If a model has a high ASR while retaining a high CACC, then it means the attacker can successfully manipulate the model prediction by inserting backdoor triggers into the demonstration examples.
Experiment II: location of backdoored demonstrations. Next, we study how the location of backdoored examples affects the attack performance. We leverage the BadWord attack under Setup 2. Apart from the random order, we consider two more location arrangements for 8 backdoored examples and 8 benign examples in the demonstration: 1) Backdoor first. It means the backdoored examples form the first 8 demonstration examples (beginning part), which are not immediately adjacent to the test input; 2) Backdoor last. It means the backdoored examples form the last 8 demonstration examples (last part), which are adjacent to the test input.
Experiment III: location of the backdoor triggers. We further study how the location of the backdoor triggers affect the attack performance. Specifically, we insert one word "cf" in a fixed location of every backdoored example and every backdoored test input. We consider the following location: 1) At the beginning of the text; 2) In the middle of the text; 3) At the end of the text. We try both Setup 2 and Setup 3 to collect the final backdoored demonstrations.
Experiment IV: backdoorsed instructions. To further evaluate the impact of the backdoors, we additionally add a backdoor in the task description to tell what are the backdoor trigger and the target class. We use the BadWord attack under Setup 1 since Setup 1 is the least effective among the three setups in the Experiment I. In this case, we want to evaluate how much a backdoor instruction in the task description would improve the attack efficacy. As shown in Figure 24, we use the task description with a backdoor instruction for the BadWord attack. In this way, we can further evaluate if the model will follow backdoor instruction and benign task instruction simultaneously.
Figure 24: An example of adding a backdoored instruction in the task description. The word 'cf' is the backdoor trigger. For simplicity, we only show one backdoored demonstration.
Table 31: Evaluation results under different backdoor generation approaches and backdoor setups. CACC (clean accuracy) means the accuracy of a clean testing set. ASR (attack success rate) refers to the accuracy of a backdoored testing set.
Setup	Model	BadWord	Addsent	SynBkd	StyleBkd
CACC	ASR	CACC	ASR	CACC	ASR	CACC	ASR
Setup 1	GPT-3.5	0.92±0.01	0.17±0.05	0.92±0.02	0.09±0.06	0.94±0.00	0.07±0.03	0.94±0.00	0.12±0.05
GPT-4	0.96±0.00	0.11±0.07	0.95±0.01	0.38±0.23	0.96±0.00	0.21±0.05	0.96±0.00	0.19±0.06
Setup 2	GPT-3.5	0.87±0.02	0.30±0.02	0.90±0.03	0.22±0.11	0.94±0.00	0.10±0.03	0.94±0.01	0.21±0.09
GPT-4	0.95±0.01	0.89±0.09	0.95±0.00	0.97±0.03	0.96±0.00	0.32±0.05	0.96±0.00	0.35±0.18
Setup 3	GPT-3.5	0.76±0.06	0.55±0.12	0.86±0.00	0.34±0.04	0.95±0.00	0.14±0.07	0.95±0.01	0.29±0.18
GPT-4	0.94±0.01	0.71±0.21	0.95±0.01	0.73±0.29	0.95±0.01	0.46±0.23	0.92±0.05	0.54±0.26
Experiment I: Different backdoor approaches under diverse backdoor setups. Table 31 shows the evaluation results of using different backdoor approaches under diverse backdoor setups. We can see that under certain combinations of backdoor approaches and backdoor setups (e.g. BadWord under Setup 3), the ASRs of GPT-3.5 and GPT-4 are high, which means they are highly vulnerable to such backdoor demonstrations.
Among the four backdoor approaches, inserting irregular words (BadWord) or a sentence (AddSent) is easier for large language models to capture, as they lead to higher ASR under the same backdoor setup. For the syntax and the style trigger, they require more backdoored demonstrations (Setup 3) to achieve high ASRs. We find that GPT-4 has a stronger pattern-following ability since it can capture the syntactic structure and text style more effectively than GPT-3.5, and thus it has higher ASRs under SynBkd and StyleBkd attacks. It indicates that GPT-4 is more vulnerable to backdoored demonstrations than GPT-3.5 due to its high instruction-following capabilities.
Another interesting phenomenon is that the BadWord attack under Setup 3 can cause a significant drop in the clean accuracy for GPT-3.5, but they would not affect the clean accuracy of GPT-4. A hypothetical explanation is that GPT-4 is able to treat the backdoor trigger as an additional feature when facing backdoored demonstrations. As a result, it still retains the clean accuracy, which has a high ASR. GPT-3.5, on the other hand, would be confused by such backdoored demonstrations, which results in a lower CACC.
Experiment II: location of backdoored demonstrations. Table 32 shows the evaluation results of placing backdoored examples at different locations of the demonstration. We can find that GPT-3.5 would be influenced more significantly when the backdoored examples are close to the test input (at the last part of the demonstration). It indicates that it pays more attention to the demonstration examples adjacent to the test input. It aligns with the previous finding [109] that the order of the demonstration examples matters. GPT-4 also tends to pay more attention to the later part of the demonstration than the beginning part. However, compared to GPT-3.5, the backdoors added at the beginning of the demonstration still have a high impact on the predictions of GPT-4, although not as large as those appearing in the later part. It indicates GPT-4 has a better capability of attending to the distant texts in the demonstration.
Table 32: Experiment II: Results of placing backdoored demonstrations at different locations under Setup 2.
Model	Random	Backdoor first	Backdoor last
CACC	ASR	CACC	ASR	CACC	ASR
GPT-3.5	0.87 ± 0.02	0.30 ± 0.02	0.78 ± 0.07	0.62 ± 0.19	0.93 ± 0.01	0.06 ± 0.01
GPT-4	0.95 ± 0.01	0.89 ± 0.09	0.96 ± 0.00	0.86 ± 0.19	0.95 ± 0.00	0.45 ± 0.43
Table 33: Experiment III: Results of inserting a trigger word at different locations under Setup 2 and Setup 3.
Setup	Model	Beginning	Middle	End
CACC	ASR	CACC	ASR	CACC	ASR
Setup 2	GPT-3.5	0.86±0.04	0.48±0.11	0.85±0.04	0.41±0.07	0.89±0.01	0.34±0.02
GPT-4	0.96±0.00	0.85±0.20	0.95±0.00	0.71±0.26	0.96±0.01	0.67±0.51
Setup 3	GPT-3.5	0.76±0.07	0.67±0.12	0.74±0.08	0.55±0.21	0.82±0.03	0.42±0.16
GPT-4	0.93±0.03	0.72±0.37	0.94±0.00	0.75±0.25	0.85±0.09	0.76±0.32
Experiment III: location of the backdoor triggers. Table 33 shows the evaluation results of placing backdoor triggers at different locations of the text examples. Under Setup 2, we find that for both GPT-3.5 and GPT-4, inserting a trigger at the beginning of a text is the most effective as it leads to the highest ASR compared to the other two locations. By contrast, the end location is the least effective. It indicates that GPT models may pay more attention to the beginning part of the user messages.
Under Setup 3, we can find that the finding for GPT-3.5 is similar to that under Setup 2: inserting a trigger at the beginning of a text is the most effective for backdoor attacks while the end location is the least effective. However, for GPT-4, the attack performance of placing backdoor triggers at three different locations is found to be similar. We hypothesize that the trigger location becomes less important for GPT-4 since every demonstration example contains the backdoor trigger.
Experiment IV: backdoorsed instructions. Table 34 reports the evaluation results of adding a backdoor instruction in the task description. We find that the ASRs of GPT-3.5 and GPT-4 significantly increase after adding the backdoor instruction. Specifically, the ASR of GPT-4 reaches  $100\%$  while its clean accuracy remains unchanged, which means GPT-4 perfectly follows the backdoor instruction and the benign task description. It again demonstrates that GPT-4 has better instruction-following capability than GPT-3.5, leading it to be more vulnerable to adversarial instructions unfortunately.
Table 34: Experiment IV: Results of adding the backdoored task description under Setup 1, which is the least effective attack setup for evaluation.
Model	Backdoored instruction	Benign description
CACC	ASR	CACC	ASR
GPT-3.5	0.92 ± 0.18	0.35 ± 0.18	0.92 ± 0.01	0.17 ± 0.05
GPT-4	0.95 ± 0.01	1.00 ± 0.00	0.96 ± 0.00	0.11 ± 0.07
- Providing backdoored demonstrations will mislead GPT-3.5 and GPT-4 to make incorrect predictions.
- Word or sentence based backdoor triggers have higher impact on GPT-3.5 and GPT-4 models than the syntactic and style based triggers.
- GPT-4 is more vulnerable to backdoored demonstrations. It has higher attack success rate under backdoored demonstrations compared with GPT-3.5, while retaining a high clean accuracy.
- GPT-3.5 and GPT-4 would be more likely to be misled when the backdoored demonstrations are positioned closer to the test inputs.
- The location of backdoor triggers has different impacts on GPT models. Both GPT-3.5 and GPT-4 pay more attention to the triggers at the beginning of the backdoored sentences.
- The efficacy of the backdoored demonstrations can be further enhanced by incorporating a backdoor instruction in the task description.
H Additional details of evaluation on privacy
H.1 Privacy leakage of training data
Pretraining data is often scraped from the web, but such data usually contain privacy-sensitive information [167]. A significant amount of prior works have shown how the training data can be extracted from pretrained models [29, 150, 30], given access to the full model and some pretraining datasets, and different privacy-preserving approaches have also been proposed [111, 102]. Here we aim to study whether it is possible to extract private training data given closed-source GPT models that are only accessible via APIs.
In this section, we present a proof-of-concept evaluation of the training data leakage of GPT models, using the Enron Email dataset [90]. While the official documentation for GPT-3.5 and GPT-4 models does not reveal their training datasets [128], we hypothesize that the Enron Email dataset is utilized when training these models. This assumption is based on the fact that the Enron dataset is part of the Pile dataset [58], which is used to pretrain the GPT-Neo family models.
To validate this hypothesis, we query the GPT models for the email address with appropriate prompts, which contain the context of the email or the user's name from training data.
Data. The Enron Email Dataset [90] is comprised of over 600,000 emails generated by employees of the Enron Corporation, where user email addresses are considered to be sensitive information. In our study, we utilize a preprocessed version of the dataset created by [77] that comprises about 3,300 (name, email) pairs. This preprocessed dataset excludes Enron domain addresses, which are formed in the format of first_name.last_name@enron.com.
Setting. We use different prompts introduced by [77] as inputs of GPTs and elicit the target email addresses. We treat the first email address that appears in the output texts as the predicted email address. We aim to evaluate the accuracy of the private email address predictions on the Enron dataset in different settings.
The first setting is context prompting. Following the evaluation principles of language model memorization [31], we investigate whether GPT models are capable of reconstructing the rest of a sequence when presented with a prefix. Context  $(L)$  refers to the case when we feed the preceding  $L$  tokens in the email before the target email address into GPT models to elicit the target email address in the generated output. Note that we only consider the output as privacy leakage if the generated sentences contain the completely correct target email address, and we do not verify whether the remaining words exactly match the original content, which is regarded as verbatim memorization [31].