Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations
Authors: Wei-Lin Chen
Cheng-Kuang Wu
Yun-Nung Chen
Hsin-Hsi Chen
Venue: EMNLP 2023 Main (2023)
Keywords: in-context learning
zero-shot
bootstrapping
TL;DR: This work presents Self-ICL, a framework bootstrapping LLMs' intrinsic capabilities for zero-shot in-context learning via self-generated demonstrations.

Abstract: Large language models (LLMs) have exhibited striking in-context learning (ICL) ability to adapt to target tasks with a few input-output demonstrations.
For better ICL, different methods are proposed to select representative demonstrations from existing training corpora.
However, such settings are not aligned with real-world practices, as end-users usually query LMs without access to demonstration pools.
In this work, we introduce Self-ICL---a simple framework which bootstraps LMs' intrinsic capabilities to perform zero-shot ICL.
Given a test input, Self-ICL first prompts the model to generate pseudo-inputs.
Next, the model predicts pseudo-labels for the pseudo-inputs via zero-shot prompting.
Finally, we perform ICL for the test input with the pseudo-input-label pairs as demonstrations.
Evaluation on 23 BIG-Bench Hard tasks shows Self-ICL outperforms zero-shot baselines on both average accuracy and head-to-head comparison.
Moreover, with zero-shot chain-of-thought, Self-ICL achieves results comparable to using real demonstrations.
Additionally, we conduct a range of analyses to validate Self-ICL's effectiveness and provide insights for its behaviors under different settings.

SELF-ICL:
Zero-Shot In-Context Learning with
Self-Generated Demonstrations
Wei-Lin Chen"
Cheng-Kuang Wu' Yun-Nung Chen Hsin-Hsi Chen National Taiwan University, Taiwan (wlchen, ckwu)@nlg, csie. ntu. edu. tw
y v. www.cceeeeee..om org hhchen@ntu edu. tw
Abstract
Step 1
Following example instance for Evaluate the result of random Boolean expression.
Large language models (LLMs) have exhib- ited striking in-context learning (ICL. abil- ity to adapt to target tasks with a few imput- output demonstrations. For better ICL, dif- ferent methods proposed to select repre- sentative demonstrations from existing train- ing corpora. However, such settings are not aligned with real-world practices, end-users usually query LMs without access to demon- stration pools. this work, we introduce SELF-ICI--a simple framework which boot- straps LMs' intrinsic capabilities to perform zero-shot ICL, Given first prompts the model to generate pseudo- inputs. Next, model predicts pseudo-labed for the cnrudo-inputs via cero-shot prompting. Finally, we perform ICL. for the test input with the pseudo-input-labe pairs as demonstrations. Evaluation 23 BIG-Bench Hard tasks shows SELF-ICL outperforms zero-shot baselines on both average accuracy and head-to-head com- parison. Moreover, with zero-shot chain-of- thought, SELF-ICL achieves results comparable using real demonstrations. Additionally, conduct range of analyses to validate SELF- ICL's effectiveness and provide insights for its behaviors under different settings."
Please come up with new, diverse, creative instances for the task.
Example Instance: Q: not True [True]
New instance 1:
Large LM
False (Tue
New instance
True ) and (alse) or Fallee
input, SELF-ICL
instance
C: not ( (False) and True or ( (False)
Large LM
Step
Task description: Evaluate the result expression.
Q: (alse) (Fasse and (True)
A: False
Q: True and (False) or False)
A: False
C: not (Fais) and True or False
A:
Q: not True and [Tuee)lé
LM
False
1 Introduction
Figure I: Our proposed SELF-ICI framework for zero- shot in-context learning. SELF-ICL involves three steps: (I) Given query and corresponding task description, the LLM is prompted to generate k (e.g.. k=3) pseudo-inputs. (2) Collect the pscudo-inputs and pre- diet their pseudo-labels via zero-shot prompting. (3) Perform ICI. with pscudo-demoossrratios. constructed from the generated LLM is used in all steps.
Large language models (LMs) have shown strik- ing ability to adapt to tasks at test time by prompting with few onput-output exemplars, i.e.,
demonstrations (Brown et al., 2020: Wei et al.. 2022; Chowdhery et al., 2022; Wei et al., 2023). This ability is refereed to as in-context learning (ICL; Brown et al., 2020). Towards better ICL performance, approaches for selecting representa- tive demonstrations have been investigated exten- sively (Sorensen et al., 2022; Levy et al., 2022;
pairs. The same
Zhang et al., 2022a; Gonen al., 2022). Most tech- niques assume the access to large-scale external sources (c.g., training dataset or relevant text cor- pus) is available, from which demonstrations can
"Equal contribution.
'https: http:///ttt...ccmtuuu///////a///

---

be selected with methods such nearest neighbor search other pre-defined, sophisticated similarity metries (Liu et al., 2022; Rubin et al., 2022; Wu et al., 2022). However, in most real-world scenario, users query LLMs (e.g., through APIs or web in- terface) without the access to existing corpus for their target tasks. Also, spending additional effort to handcraft demonstrations may negatively affect their workflows.
Method
Inputs
Labels
AUTO-CoT
from training set from external corpus
need
Z-ICL
need
SG-ICL
need
given
SELF-ICI. (ours)
need
need
Table A comparison to prior attempts on zero-shot ICL. SELF-ICL does not not require any real inputs or labels to construct demonstrations. See Section for detailed discussions.
Recently, series of studies has been proposed to shed lights on the inner working of ICL (Xie et al., 2021; Reynolds and McDonell, 2021; Min et al., 2022b). The evidence suggests that instead of contributing explicit signals for learning new tasks, demonstrations mainly expose LLMs' in- trinsic functionalitics and guide models towards target domains (Razeghi et al., 2022; Lyu al., 2022). Similar clues are also partly observed in chain-of-thought (CoT) and ICL (Madaan and Yazdanbakhsh, 2022: Webson and Pavlick, 2022). These findings indicate, to some degree, LLMs carry underestimated zero-shot abilities and are already equipped to fulfill various target tasks.
all-task-average accuracy and in head-to-head com- parisons. For instance, the results are 18-0-5 (win- tie-lose) for SELF-ICL versus standard zero-shot on the 23 tasks. Furthermore, with zero-shot Chain-of- Thought (Kojima et al., 2022), SELF-ICI. reaches performance on par with using few-shot demonstra- tions sampled from real data instances.
In addition, we perform an array of analyses to validate SELF-ICL's effectiveness under different settings. We investigate various approaches for gen- erated pseudo-inputs, the effect of number of shots, and the impact of random pscudo-labels, providing better insights for SELF-ICL's 's behaviours. To the best of our knowledge, we present the first attempt for true zero-shot ICL. that does not require any external data from real distribution or pre-defined label sets (See Table 1).
Inspired by the mentioned literature, propose SELF-ICL-a simple prompting frame- work for zero-shot in-context learning. SELF-ICL bootstraps LLM's intrinsic capabilities by self- generated demonstrations which inform the input and label space for performing ICL. Given a query. i.e., test input, SELF-ICL'S 's involves three steps:
SELF-ICL
This section details the design of SELFF-CI, for constructing pseudo-inputs and pseudo-labels form ideal pscudo-demonssrattion
1. The model is prompted to generate pseudo- inputs conditioned on the given query and the corresponding task description.
2.1 Pseudo-Input Construction (Step 1)
Generating pseudo-inputs can be easily achieved by zero-shot prompting LLMs with the simple prompt as shown in Figure 1 (Step 1). The given query (from real distribution) provides outline of ground-truth inputs, and the corresponding task description T guides the model generate rele- vant information associated with the task domain. From q and T, model infers the underlying for- mat and creates a new query (i.e., pseudo-input, By specifying a number (number of shots) the instruction, this process can generate multiple pseudo-inputs with one inference pass.
2. The model predicts pseudo-labels for pseudo- inputs via zero-shot prompting.
3. The sneudo-input-latee pairs form pseudo- demonstrations, which are then prepended to the query and proceed with standard ICL.
All steps adopt the same frozen LLM. Without the requirement of candidate pool for demonstration selection, SELF-ICL bridges the gap for end-user's practical needs.
To evaluate SELF-ICL's effectiveness chal- lenging, unexpected tasks for which existing demonstrations are hard to come by, we perform evaluation on set 23 tasks from BIG-Bench Hard (BBH; Suzgun et al., 2022). Results show that SELF-ICL exhibits significant improvements on the
2.2 Pseudo-Label Construction (Step 2)
After obtaining the pseudo-inputs, we then pre- dict their labels (the pscudo-labels for constructing pseuudoddmmonstraioon))) via zero-shot prompting

---

Step?
Task description: Recommend similar the of movies.
Format:
Starting with Therefore, the correct answer "eefore giving final answer. anewer.
options available, you ss very important that you stick to the format.
Find movie simler The Hangover, Bridesmaids, Superbed, Knookad Up: Options:
[A] The 4e-Vear--aaar Old Virgin
Pinsappie Exprass
Step Brothers
(D) This is The End
A: Let's st's think step
Step
description: Recommend
given movies,
Format:
Starting "Therefore, correct
giving your finel
options are avaiiable, you must pick one as the final answer.
it's very important that you stick to the format.
Find a movie similar to The Hangover, Bridesmaids, Superbad, Knocked Up:
Options:
The 40 Year-Dld Virgin
Pineapple Express
(C) Step Brothers
This The End
step step. Hangover, Bridesmaids, Superbad, Knocked Up are comedies that feature group of friends. Therefore, the correct answer is Pineapple Express.
Q: [...]
A: Let's think step by step.
O: [..]]
A: Let's think step by step. [..]]
Batman, Fugitive, Pretty Woman:
Options:
The Front
(C) The Lion King
Lamerica
step
Figure 2: Example prompts of SEL.F-CI. Steps and 3 for the CoT prompting setting (movie recommendation).
the same LLM. Specifically, we employ two zero- shot methods: Direct prompting and CoT prompt- ing, described as follows.
guiding the model to generate its intermediate rea- soning steps which lead to a more accurate final answer. We then take the trigger phrase and the gen- erated reasoning chain containing the answer for pseudo-inputs as the pseudo-labels for construct- ing pseudo-demonstrations (See Figure for an example prompt).
Direct prompting In the direct prompting setup. we construct pseudo-labels via standard zero-shot prompting schema. Namely, we prompt the L.MM with only the task description and the generated pseudo-input for direct answer prediction (See Figure for an example prompt). We predict pseudo-labels one by one, i.e., for k-shot demon- stration, & inference passes are required for the pseudo-inputs.
2.3 Prediction (Step 3)
Here in Step 3. we construct pseudo- demonstrations, i.e., pseudo-shots, by the pseudo-inputs paired with their corresponding pseudo-labels from previous steps, and predict the final answer for the test input by the typical few-shot ICL workflow. Namely, the pseudo-shots (with instructions) are prepended the test input as the context for prompting the LLM. For CoT prompting, only the final answers are evaluated. For the example prompts on Step 3, see Figure 8 and 2. Note that both direct prompting and CoT
CoT prompting For the CoT prompting setup, SELF-ICL generates pseudo-labels by zero-shot COT (Kojima al., 2022). Specifically, prompt the LLM with the task description, the current test input, and a trigger phrase, "Let's think step by step." for performing CoT reasoning. The trigger phrase is appended at the very end of the prompt,

---

Direct Prompting
CoT Prompting
BBH Task
3-shot
ZS-Direct
Self-ICL
ZS-CoT
Self-ICI
Bnolcan Expressions
84.00
87.20
3.20
85.60
85.60
0.00
89.60
Causal Judgement
55.61
61.50
5,88
58.29
58.82
0.53
62.03
Date Understanding
54,00
56.80
2.80
66.80
69.20
2.40
58.80
Disambiguation QA
64.00
68.00
4,00
64.80
67.60
2.80
68.80
Formal Fallacies
56,00
55.20
-0,80
55.20
56,80
1.60
58.80
Geometric Shapes
34.40
36,00
1.60
29.60
33,60
4.00
36.80
Hyperbaton
56.80
59.20
2,40
53.60
53,60
0.00
57.60
Logical Deduction (five objects)
41,20
40.40
-0.80
37.60
40,80
3.20
45.60
Logical Deduction (seven objects)
43.60
40.00
-3.60
39,60
36.00
-3.60
40.00
Logical Deduction (three objects)
56.00
65.60
9.60
58.80
64,80
6.00
64.40
Movie Recommendation
63.60
75.20
11.60
64,80
70,40
5.60
77.20
Navigate
49.20
68,00
18.80
53.60
57.60
4.00
52.80
Penguims Table
58.90
63.70
4.79
60.96
64.38
3.42
62.33
Reasoning about Colored Objects
59.60
61.20
1.60
68.00
67.60
-0.40
65.20
Ruin Names
54.40
66.40
12.00
48.80
56.40
7.60
84.00
Salient Translation Error Detection
51,20
57.20
6.00
50.80
54.00
3.20
65.20
Snarks
54.49
62.92
8.43
37.08
55.06
17.98
65.73
Sports Understanding
67.60
65.60
-2.00
71,20
69.20
-2.00
71.20
Temporal Sequences
57.60
35.60
-22.00
64.80
52.00
-2288
39.20
Tracking Shufiled Objects (five objs)
18.00
19.60
1.60
25.20
29.60
4.40
16.40
Tracking Shufiled Objects (seven objs)
14.40
18.40
4.00
31.60
27.20
-4.00
15.20
Tracking Shuffled Objects (three objs)
26.40
28.00
1.60
36.00
46.00
10.00
Web Lies
53.20
57.20
4.00
61.20
65.60
4.40
56.40
Tasks (avg)
50.81
53.93
3.12
53.22
55.54'
55.49
Table 2: The main results of our proposed SELF-ICL evaluated on BBH. The 3-shot results are standard few-shot prompting with real data as demonstrations. SELF-ICL exhibits consistent trends outprrforming both direct and CoT prompting baselines. We adopt one-sided McNemar's test (McNemar, 1947) to test the statistical significance of Self-ICL's performance gain over baselines, where f denotes value < 0.05.
prompting methods shared the same Step prompt (Figure 7).
Implementation details For all LMs' hyperpu- rameters, we the temperature to O and the the maximum number of tokens as 1024. Other argu- ments are kept as their default values. Regarding the number of choose 3=3 for our main experiments.
3 Experiments
shots k, we
To evaluate the effectiveness of our proposed method, we conduct a set extensive experiments for better comparison and analysis. We describe the experimental settings and discuss the results in detail.
Dataset We adopt the BIG-Bench Hard (BBH) benchmark for our evaluation. BBH consists of suite of tasks from the BIG-Bench benchmark (Sri- vastava et al., 2022), which existing LMs have dif- ficulty reaching the average numan-rater perfor- mance and are considered beyond current models' capabilities. BBH contains a total of 27 tasks, from which we select 23 tasks that are multiple-choice tasks
3.1 Configurations
Language models We use InstructGPT (text-davinci-003: Ouyang et al., 2022) for all the experiments presented in Sec- tion 4.1 and 5. We also conduct additional experiments validate the generalizability of SELF-ICL, using text-bison-0011 from the PaLM-2 model family (Anil al., 2023) and gpt-3. 1---..--uurro-nstrus from the GPT-3.5 model family. 3 The results are presented in Section 4.2.
evaluation testbed for SELF-ICL. Each BBH tasks has around 150 250 examples, and the total number of instances is 5,511.
3.2 Baselines
ZS-Direct The baseline of direct prompting is the typical zero-shot prompting setup, denoted 7.--Direct. Concretely, the L.LM prompted with the task description and the current test input for a direct answer prediction.
hnttps://developee httpe///////eeeeeeeeeeeeeeeeeeee models/language
generativeai..oogle.
hhttps://plafoom aoppalaatoooooooaiii....mm www.a/..co/oooc...ooooeooo com/docss/mdeess/ gpt-3-5

---

Prompting
CaT prompting
Self-CL (w/ CaT) v.s. Z5-CaT
10
Accuracy
-20
Figure 3: The head-to-head comparison on the 23 tasks from BBH. The accuracy delta indicates the accuracy difference between SELF-ICL and the baseline method (blue/orange indicates method wins/loses). The results are 18-0-5 (win-tie-lose) for the direct prompting setting; 16-2-5 for the CoT prompting setting; and 14-1-8 for SELF-ICL without CoT (i.e., direct) versus ZS-CoT.
ZS-CoT For CoT prompting, the baseline is the zero-shot CoT prompting proposed by Kojima et al. (2022), which is of the state-of-the- art method for solving reasoning-heas heavy task in zero-shot. We denoted it as ZS-CoT. Specifically, the LLM is prompted by the task description, the current test input, and a reasoning trigger phrase "Let's 's think step by step." same as SELF-ICI.
4.2 Generalizability
To assess whether our proposed SELF-ICL frame- work is able to generalize to other models, we perform experiments on two popular LLMs, GPT- 3.5 and PaLM-2, beside InstructGPT. We com- pare their results under direct prompting set- ting. The results are present Table 3. As oh- served, SELF-ICL demonstrates stronger overall performance over the direct prompting baseline on both PaLM-2 and GPT-3.5. Moreover, although PaLM-2 exhibits relatively poor scores comparing to InstructGPT and GPT-3.5, it can still improve upon itself with our proposed SELF-ICL, Interest- ingly, we find GPT-3.5 has a slightly inferior perfor- mance comparing to InstructGPT. We hypothesize this is because GPT-3.5 has a lower controllability. thus, it is more prone to generate unintended con- tent. For instance, the model might not follow the formatting instructions presented in the prompts (see Figure 8). In addition, the generated pseudo- inputs are more likely be invalid and could not accurately represent the underlying tasks. sum, the results still suggests SELF-ICL generalizable for different models.
4 Results
4.1
Main Results
We present our main experimental results in Ta- ble 2. On the all tasks average performance, SEL.F- ICL surpasses baselines in both the direct and CoT prompting settings. We also observe SELF-ICL with direct prompting is comparable (slightly bet- ter) with ZS-CoT prompting. Furthermore, SELF- ICL with CoT prompting reaches performance on par with few-shot prompting which uses real demonstrations (the 3-shot column).
We illustrate head-to- tead-to-head head comparisons the 23 tasks in Figure 3. The results of direct prompt- ing are 18-0-5 (win-tie-lose) for SELF-ICL versus the ZS-Direct baseline: for the CoT prompting set- ting, the results are 16-2-5 for SEL.FICI. versus the ZS-CoT baseline. Interestingly, the results are 14-1- 8 for SELF-ICL without CoT (SELF-ICL with direct prompting) versus 7S-CoT, and comparable or bet- ter performance is exhibited on all tasks average as well. This highly competitive result demonstrated by SELF-ICL with direct prompting sheds light on an alternative to elicit LMs' reasoning ability in zero-shot, without generating potentially biased or misleading reasoning chains (Turpin et al., 2023).
Analysis
In this section, we first illustrate the concept of copying effect, and discuss its implication for SELF- ICL. Next, investigate SELF-ICL'S behaviors under different settings, including different ap- proaches for generated pseudo-inputs, performance with varied number of shots, and the effect of ran- domly assigning pseudo-labels. Following analy- ses all focus on the setting of direct prompting.

---

text-bison-000
300-3.5-turbo-instrree
BBH Task
ZS-Direct
Self-ICI
ZS-Direct
Self-ICL
8
Boolcan Expressions
60.16
58.94
-1.22
84.80
88,40
3.60
Causal Judgement
47.37
48.54
1.17
42.25
12.30
29,995
Date Understanding
42.40
41.20
-1,20
59.20
57.60
-1.60
Disambiguation QA
33.33
33.82
0.49
60.00
63.20
3,20
Formal Fallacies
57.20
56,00
-1,20
52.00
50.40
-1.60
Geometric Shapes
15.20
19.20
4,00
34.00
36.40
2.40
Hyperbaton
57.43
68.27
10.84
82.40
82.80
0,40
Logical Deduction (five objects)
18.34
23,58
5.24
42.00
38.40
-3,60)
Logical Deduction (seven objects)
10.88
13.99
3.11
41.60
34,80
-6.00
Logical Deduction (three objects)
37.89
37,00
-0.88
56.00
59.20
3.20
Movie Recommendation
26.23
26.23
0.00
74.80
76.00
1.20
Navigate
58.71
57.21
-149
42.80
64,80
22.00
Penguins in Table
42.76
42.76
0.00
51.37
55.48
4.11
Reasoning about Colored Objects
62,40
70.80
8.40
$4.80
56.40
1.60
Ruin Names
30.81
26.16
-4.65
70.80
64.80
-6.00
Salient Translation Eroo Detection
22.13
22.54
0.41
41.60
51.20
9.61)
Snarks
54.86
50.86
-4.00
63.48
60.67
-2.81
Sports Understanding
46.45
45.90
-0.55
62.00
50.00
-12.00
Temporal Sequences
28.51
30.58
2.07
32.80
Tracking Shuffled Objects (five objects)
14.52
17.74
3.23
18.00
16.40
-1.60
Tracking Shufiled Objects (seven objects)
20.00
19.60
-0.40
12.40
-5.20
Tracking Shuffled Objects (three objects)
29.32
32.93
3.61
32.40
36.80
4.40
Web of Lies
$7.20
50.40
15.20
38.40
23.20
All Tasks (avg)
37.78
38.831
1.05
49.72
Table 3:
The results of SELF-ICL using text-bi -on--81 and 3pt-3..-turbb--te 5-turbo-instruct evaluated on BBH. Over-
all, SELF-ICL exhibits consistent trends outperforming direct prompting. This suggests SELF-ICL is generalizable for different models. We adopt one-sided McNemar's 's test (McNemar, 1947) to test the statistical significance of performaee
Self-ICL's performance gain over baselines, where f denotes value < 0.05.
5.1 Preliminary
5.2 The Entanglement of Input Space and Input-Label Mapping
Given a set of k-shot demonstrations denoted as [(11,00 U1),.., (II. and ys is the label. As suggested by Min et al. (2022b), four aspects are considered for the con- struction of demonstrations, namely: (1) The input- label mapping: whether æ; is paired with correct Yi (2) The input space: the underlying distribution behind f1,-.., (3) The label space: the possible label set inferred from V1,..., yk- (4) The pairing format: the format representing the ri-Y, pair. Min et al. (2022b) inspect the role of demonstrations along these four aspects, and present surprising finding-the input-label mapping is neces- sary criteria for successful ICL. Empirically, they find randomly swapping the ground-truth label of demonstrations barely degrades end-task perfor- mance. On the contrary, the other three aspects all demonstrate great impacts. With these four as- pects in mind, we now analyze the construction of pseudo-demonstrations for SELF-ICI.
yk)), where is the input text
Among the four aspects, the label space is usu- ally specified the input (e.g., options presented for multiple-choice) or described in the task de- scription. For example, the label space ("True", "False") } of the boolean expressions task can be eas- ily inferred from its description "Evaluate the result of random Boolean expression." "; the pairing for- mat is the least concern as pseudo demoossttmossstiionnns are well formatted as "Q: input text, label text".
The potentially problematic aspects are the in- put spuce and input-label mapping. Naturally, one may think input-label mapping is not an issue as described Section 5. 5h1 input does not need to be paired with the correct label. However, this intriguing discovery by Min et al. (2022b) is estab- lished under the setting of standard ICL, where the inputs randomly sampled from the training set.
As the pseudo-inputs created by SELF-ICL is based on only one reference, i.e., the given input, the generated pseudo-inputs are likely be of great semantic similarity with that test input, and fail to capture the correct input space distri-
'Input with instruction-iin like descriptions could also inform model of the label space.

---

bution. In such case, Min al. (2022b)'s finding does not hold since has been shown that models tend to copy the labels paired with inputs that are very similar to the test input, known as the copy- ing effect (Lyu et al., 2022). With no guarantee for the correctness of SELF-ICL's pseudo-labels, the copying effect would potentially hurt the ICL performance.
0.68
Similarily
0.66
0.62
5.3 Different Approaches for Generating Pseudo-Inputs
Real
Batch
Diverse
Diverse
Figure 4: Semantic similarity between the pscudo- inputs generated by different Step approaches and the test input. The similarity value is averaged across three shots and all tasks.
To mitigate the possible impact of copying effect, increasing the pscudo-inputs' diversity is essential. Typically, this can be resolved by sampling demon- stration inputs from different clusters of training set inputs (Zhang al., 2022b). However, real data is available in our SELF-ICL framework. To gain better understanding of SELF-ICL'' 's pseudo-input generation and the potential copying effect, we study three different approaches for constructing pseudo-inputs: (1) Batch inference, (2) Prompt- ing with diversity hints, and (3) Prompt without diversity hints.
55
54
53
All Task Aug (56)
51
49
Batch inference In batch inference, we assume an access to multiple test inputs in Step 1. Specif- ically, the number of example instances in the prompt equals the number of given test input, i.e., the batch size. The LM then generates the same number of pseudo-inputs in the original stream- ing inference where prompt one test input a time. The prompting template is provided in Figure In batch inference setup, all test in- puts share the same sacudo-inputs, thus the same pseudo-demoonstattion in Step 3.
Randam
Batch
Panuto-L4be 4ba
Figure 5: The all-task-average performance of using pseudo-inputs generated by different Step approaches. different number of shots, random pseudo-labels Step 2.
of using real inputs from BBH dataset for estab- lishing a similarity baseline. We encode all inputs by sentence transformers (Reimers and Gurevych, 2019) following Liu et al. (2022); Zhang et al. (2022b).
Prompting with diversity hints Prompting with diversity hints is the method we adopt our main experiments. As shown in Figure (Step I), the model is explicitly instructed to provide "new", "diverse", and "creative" pseudo-input instances.
As observed, batch inference produces pseudo- inputs that is most similar to using real inputs. This somewhat intuitive as batch inference has access to more example instances. Interestingly, looking at results in Figure 5 we find using pseudo-inputs generated by prompting with diversity hints (the 3-shot bar) and batch inference achieve essentially the same final accuracy, although it exhibits the lower similarity. This may suggest over diversify- ing demo-inputs have little impact on empirical per- formance. For prompting without diversity hints, demonstrates the highest similarity to test input and lower final accuracy, which could be explained by copying effect.
Prompting without diversity hints For prompt- ing without diversity hints, we simply remove the key words "new", "diverse", and "creative" the instruction, and keep all other settings unchanged.
Our analysis results are shown in Figure 4. We compute the cosine similarity between the pseudo- inputs generated by the aforementioned approaches and the test input. For each method, the reported value averaged across three pscudo-inpute (3- shots) and all BBH tasks. We also report the result
'We adopt the all-Minil.M-L-r2 model.

---

5.4 Effect of Different Number of Shots
0.2
Here we investigate SELF-ICL'S 's performance under different number of croudo-demonstration shots. The results are presented in Figure 5. The 3-shot setting is our adopted method in SELI-ICL'S 's main experiment, and I-shot setting used randomly sampled shot from the shots (here a shot refers to a (seuud-demmonstration))) The (-shot setting is the ZS-Direct baseline. As observed, the 3-shot is the top performing setup. Note that although inferior to 3-shot, 1-shot still exhibits notable gain over O-shot, indicating the empirical effectiveness of SELF-ICL.
Similarity Gap
0.0
-0.1
logical deduction objects
tracking shufied objects
movi recommmnndatio web of lies hyporbaton geometric shapes disambiquation co sallent franclation error detection comporo sequ sequences
sports und understanddnng
navigate
farnal lallaaalloooo
panquin in table
bonlean expresstons
snarko
date understanding
causal judgemennt
rousonnnng about colored objects
5.5 Effect of Random Pseudo-Labels
To verify the quality our pseudo-labels, we replace the pseudo-labels obtained in step 2 by randomly assigned labels, and construct pseudo- demonstration with such random pseudo-labels to predict the test inputs' answers. As shown in Fig- ure 5, the performance with random pseudo-labels is inferior 3-shot, 1-shot, and the no diverse setup, but still benefit the performance comparing no demonstration all (O-shot).
Figure 6: The similarity gap the query-inputs distance between pseudo- and real-inputs. Most tasks fall into samll =5% range (the dotted lines), indicating the pseudo-inputs are close the real-inputs and are likely robust against the copying effect.
Although the performance drop using random labels may indicate the possibility that some in- stances encounter the copying effect, we hypothe- size the LLMs' abilities to overwrite the semantic prior when demonstrations have contradicting la- bels is another big factor (Wei et al., 2023). That is, LLMs would recognize the demonstration labels as the correct answers, and make predictions ac- cordingly. Moreover, this phenomenon is further extrapolated when using LMs with instruction tun- ing. Exploring the underlying relationship between the copying effect and Wei et al. (2023)'s findings are left future works.
between pseudo- and real-inputs. Given query 9. i.e., test input, randomly selected inputs ((11,31). (æk, yk)) from the training set, and a set of k pseudo- inputs {(11, 91), ..., (æg, 9x)) generated by SELF- ICI. conditioned on We first define the query- input distance d(-) between using pseudo-input and real-input as
of k
1 Σ sim g, 22)) sim x4) (1) K
d(4)
where the query and input are encoded by the same sentence transformers model used in Section 5.3.
Next, we compute the similarity gap ((-) as
5.6 A Deeper Look of SELF-ICI''S Pseudo-Inputs
G(Q) - d(g:)
(2)
To increase the diversity of the generated pseudo- inputs and mitigate the risk facing the copy- ing effect, we apply simple and straightforward method: prompting LLMs to be diverse with key words "new", "diverse". and "creative". To pro- vide more fine-grained analysis for individual tasks, following we attempt to quantitatively ver- ify whether our generated pseudo-inputs are di- verse enough comparison with the real inputs randomly sampled from the training data, by mea- suring the similarity gap of the query-input distance
where Q is a set of n queries (q1, 4n) for a task. The similarity gaps for 23 tasks from BBH are presented in Figure 6. The results are averaged across five different random seeds (for training data sampling) and we provide standard deviation error bars. The larger the gap indicates the more closer the queries are with the pseudo-inputs than with real- real-inputs sampled from training set, and the more likely to suffer from the copying effect. As

---

observed, most of the tasks fall inside the 15% similarity range (dotted lines), suggesting our de- signed prompt is able to encourage the generation diverse pseudo-inputs, and sufficiently resemble inputs sampled from real distributions to mitigating the potential risk of the copying effect. We also observe the three tasks with substantially higher lower similarity gap require heavy, multi-step reasoning solve. Thus the initial difficulties of understanding those task could explain model's failure to capture suitable input spaces.
shared among all inputs. On the contrary, SELF- ICL generates different input-dependent options for the multiple-choiee tasks, and can easily extend to other generation tasks.
The most similar work to ours is by Kim et al. (2022), where they explore the possibility of gen- erating pseudo-imputs by the LLM itself, without any external data source. However, their frame- work requires assess to the label set. They generate the pseudo-input by conditioning the LM on a la- bel given in the prompt. Such a design dose not align with practical usage as it greatly restricts the scenario to fixed classification tasks. As a result, their evaluation is limited to only text classifica- tions (sentiment classification and natural language inference), which are relatively simple and well- studied comparing BBH in our evaluation.
6 Related Work
Understanding ICL With the popularization of various LLMs, ICL has emerged as new paradigm for the field of natural language processing (NLP). However, the mechanisms behind ICL's superior ability are still an open question in the research communities. To develop a deeper understanding of ICL, Chan et al. (2022) investigate the train- ing data distribution of LLMs, and find specific distributional properties and the transformer-bol -bsse architecture (Vaswani et al., 2017) could drive the ICL behaviors. Recent studies also provide planations viewing LMs as meta-optimizers with meta-gradiemt applied in the forward passes, and show evidence of resemblances between ICL and the explicit fine-tuning process (Dai et al., 2022; von Oswald et al., 2022; Akyürek et al., 2022).
7 Conclusions
In this work, we introduce SELF-ICL-a simple yet effective framework for zero-shot in-context learning, where only test input and its task de- scription are required. SELF-ICL consists of three steps: (1) Construction of pseudo-inputs, (2) Con- struction of pscudo-labels, (3) ICL with pseudo- demonstrations, i.e., pseudo-input-label pairs. Eval- uations on BBH show SELF-ICL outperforms zero- shot (CoT) baselines on head-to-head and all-task average accuracy. Additionally, we conduct exten- sive analyses provide better insight of SELF- ICL. To the best of our knowledge, we present the first true zero-shot approach for ICL, and demon- strate the potential of bootstrapping LMs" inner capabilities to improve zero-shot performance.
Towards Zero-Shot ICL To achieve better em- pirical performance with ICL, approaches for de- signing ideal prompts and demonstrations have been vastly investigated (Min et al., 2022a; Su et al., 2022; Zhou et al., 2022; Lu et al., 2022a; Fu et al., 2022: Lu et al., 2022b).
Limitations
Recent work from Zhang et al. (2022b) ad- dressed the need of human-annotated few-shot CoT by utilizing zero-shot CoT to construct demon- strations. Their method differs from ours as they require an existing training set from which shots are sampled as inputs to zero-shot COT. Lyu et al. (2022) attempt to exclude the need of pre-given demonstration candidate set by selecting semanti- cally relevant sentences from an raw text corpus (which is not from the task datasets) as pseudo- inputs. And pair the selected pseudo-inputs with randomly assigned labels as demonstrations for ICL. Though more similar to our setting, they still need an access to external sources for construct- ing pscudo-inpute Moreover, they are limited to classification tasks where a fixed set of labels is
Reliance on
models To follow instructions, understand unseen target tasks and generate pseudo-inputs and pseudo-labels via zero-shot prompting, a key driver of our SEL.F-ICI framework is the powerful inslroction-folloving LM. If the model is not equipped with such zero- shot generalization capability, the results of SELF- ICL would be inferior.
Better diversify approaches To mitigate poten- tial risks of suffering from the copying effect, we simply construct heuristic prompts tell the LM to generate diverse pseudo-inputs. Due to the limited budget, we do not perform comprehen- sive prompt searching or experiment with temper-

---

ature adjustments. In the future, others should ex- plore methods along the line of one- or few-shot data augmentation for constructing optimal pseudo- demonstrations.
in language models via perplexity estimation, arXiv preprint arXiv:2212.04037.
Hyuhng Joon Kim, Hyunsoo Cho, Junyeob Kim, Taeuk
Kim, Kang Min Yoo, and Sang-goo Lee. 2022. Self-gencrated in-context learning: Leveraging auto- regressive language models as a demonstration gen- erator, arXiv preprint 220.:2206..8082.. 08082.
Acknowledgements
We thank the reviewers for their insightful com- ments. This work was financially supported by the National Science and Technology Council (NSTC) in Taiwan, under Grants 1 111-222--E-00-- 013-MY3, 0-122--E-000-00--MY5 110-2221- 1,002-12--MY- and 11 111-2634-F-002-022. and Ministry of Education (MOE) in Taiwan, under grants NTU-1121 121900901.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. arXiv preprint arXiv:2205.1 1196..
Itay Levy, Ben Bogin, and Jonathan Berant. 2022. Diverse demonstrations improve in-context compositional generalization. arXiv preprint arXiv:2212.06800.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLJO 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 100-114, Dublin, Ireland and Online. Associa- tion for Computational Linguistics.
References
Ekin Akyürek, Dale Schuurmans. Jacob Andreas, Tengyu Ma, and Denny Zhou. 2022. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint 1xxx::221111660
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm technical report. arXiv preprint arXi::2305..0000 10403.
Jinghui Lu, Rui Zhao, Brian Mac Namee, Dongsheng Zhu, Weidong Han, and Fei Tan, 2022a. What makes pre-trained language models better zero/few-shot learners? arXiv preprint arXiv:2209.1520
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022b. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 8086 8098, Dublin, Ireland. Association for Compu- tational Linguistics.
Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. 2022. Data distributional properties drive emergent in-context learning trans- formers. Advances in Neural Information Processing Systems, 11118888-88911
Xinxi Lyu, Sewon Min, Iz Beltagy. Luke Zettlemoyer. and Hannanch Hajishirzi. 2022. Z-icl: Zero-shot in- context learning with peeudo-demonstrations.. arXiv preprint 22X1v:2212.09665.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.03311.
Aman Madaan and Amir Yazdanbakhsh. 2022. Text and patterns: For effective chain of thought, it takes two to tango. arXiv preprint arXiv:2209 07686.
Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153-15)
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. 2022. Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers. arXiv preprint arXiv:2212. 12.555999
Sewon Min, Mike Lewis. Hannanch Hajishirzi, and Luke Zettlemoyer. 2022a. Noisy channel language model prompting for few-shot text classification. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 531 6-5330, Dublin, Ireland. As- sociation for Computational Linguistics.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2022. Complexity-based prompt- ing for multi-step reasoning. arXiv preprint arXiv:2210.00720.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022b. Rethinking the role of demonstrations:
Hila Gonen. Srini Iyer, Terra Blevins, Noah Smith, and Luke Zettlemoyer. 2022. Demystifying prompts

---

What makes in-context learning work? In Proceed- ings of the 2022 Conference on Empirical Methods in
Zhou, et al. 2022. Challenging big- big-bench tasks and whether chain-of-thouug of-thoughtt can solve them. arXiv preprint arXiv: 2tt.::2200009..1
Natural Language Processing, pages I 1048-1 1064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Miles Turpin, Julian Michael, Ethan Perez, and Samuel R Bowman. 2023. Language models don't always say what they think: Unfaithful explana- tions in chain-of- prompting. arXiv preprint arXiv:2305.0433
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot reasoning. arXiv preprint arXiv:220.007206.
Johannes von Oswald, Eyvind Niklasson, Ettore Ran- dazzo, João Sacramento, Alexander Mordvintsev, An- drey Zhmoginov, and Max Vladymyrov. 2022. Trans- formers learn in-context by gradient descent. arXiv preprint arXiv:2212.07677.
Nils Reimers and Tryna Gurevych. 2019. Sentence-bers Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics.
A lbert Webson and Ellic Pavlick. 2022. Do prompt- based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pages 2300-2344, Seattle, United States. Association for Computational Linguistics.
Laria Reynolds and Kyle McDonell. 2021. Prompt pro-
gramming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Comput- ing Systems, pages 1-7.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. 2022. Emergent abilities of large language models. arXiv preprint arX6::2206.07777 0766222...
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning, In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistiex: Human Language Tech- nologies, pages 2655-2671, Seattle, United States. Association for Computational Linguistics.
Jerry Wei, Jason Wei, Yi Tay, Dustin Tram, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, al. 2023. Larger language models do in-context learning differently. arXiv preprint ar 2033:.330.
Taylor Sorensen, Joshua Robinson, Christopher Ryt- ting, Alexander Shaw, Kyle Rogers, Alexia Delorey, Mahmoud Khalil, Nancy Fulda, and David Wingate. 2022. An information- -thooretthcec approach to prompt engineering without ground truth labels. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Pa- pers). pages 819 819-862, Dublin, Ireland. Association for Computational Linguistics.
Zhiyong Wu, Yaoxiang Wang. Jiacheng Ye, and Ling- peng Kong. 2022. Self-adaptive in-context learning. arXiv preprimt arXiv:2212.103::
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2021. An explanation of in-context leam- ing as implicit bayesian inference. arXiv preprint arXiv: 21t::2111.2200 1.02000.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso,
Yiming Zhang, Shi Feng, and Chenhao Tan. 2022a. Ac- tive example selection for in-context learning. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 9134- 9148, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint 0rXiv:2206.06655.
Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf. Luke Zettlemoyer. Noah A Smith, et al. 2022. Selec- tive anmotation makes language models better few- shot learners arXiv preprint arXiv:2209.01975
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022b. Automatic chain thought prompt- ing in large language models. arXiv preprint 210.::2210.03493.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.0191)
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Se- bastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny

---

A Appendix
Stop
A.1 Details of Experimental Cost
Task description: Recommend movies similar given list of movies.
Direct Prompting
Q: Find movie similar to The Hangover, Bridesmaids, Superbad, Knocked Up: Options:
3-shot
118.35
I-shot
27.29
(A) The 40- Year-Ole Virgin
SEIF-ICI,
prompting without diversity hints
135.58
(B) Pineapple Express
random mseudo-labels batch inference
51.27
(C) Step Brothers
63.15
(D) This Is End
3-shot
49.98
Standard
0-shot
15.27
Step
CoT Prompting
Task description: Recommend similar the given list of movies.
SELF-ICI
3-shot
203.10
Standard
0-shot
28.71
Q: Find movie similar to The Hangover, Bridesmaids, Superbad, Knocked Up: Options:
Sum
692.70
(A) The 40- (0-Year-Oid Virgin
Table 4: The cost (in US dollar) of experiments using InstructGPT (Section 4.1 and 5), estimated based number of tokens.
(B) Pineapple Express
C Step Brothers
(D) This Is the End
Q: [...]
A: [...]
A.2 Example Prompts
[...]
[...]
Step1
Q: Find movie similar to Batman, The Mask, The Fugitive, Pretty Woman:
Following is an example instance for the task:
Recommend movies similar to the given list of movies.
Options:
Please come up with new, diverse, creative instances for the task.
(A) The Front Page
(B) Maelstrom
(C) The Lion King
Example instance:
(D) Lamerica
Q: Find movie similar Balman, Mask, The
Fugitive, Pretty Woman;
Options:
(A) The Front Page
Figure 8: Example prompts of SELF-ICL Steps 2 and 3
(B) Maelstrom
The Lion King
the the direct prompting setting (The movie recom- mendation task).
(D) Lamerica
New instance
Batch Inference Step
Figure 7: An example prompt of SELF-ICL Step (The movie recommendation task). Note that both the direct prompting and CoT prompting settings shared the exact same Step 1 prompt.
Following are [BATCH SIZE] example instances for the task: [task description). Please come up with [NUM SHOT] new, diverse, and creative instances for
Example instances 1:
Q: (test input
Example instances 2:
Q: (test input 2)
Example instances BATCH_SIZE Q: [test input BATCH. BATCH_SIZE]
New instance 1:
Q:
Figure 9: The prompt template batch inference (Step
1).


Reviews:
[{"review_id": "T6XgjLVDxX", "reviewer": "Reviewer_B1V1", "summary": "", "questions": "(1) Is the observation generalizable? Can the proposed idea work on some other models such as LLaMA, Falcon, Flan-T5 or other models in\n\nhttps://github.com/FranxYao/chain-of-thought-hub\n\n(2) How broadly can the proposed approach work (in which tasks)? When the LLM generate incorrect few shot examples, it can be risky to use the proposed approach, and further use them in the in-context learning. It will be great to include some related discussions of the limitations.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback! Regarding the points mentioned in “Reasons To Reject”:\n\n> It is unclear how generalizable the observation is. The paper only uses InstructGPT. The paper may also evaluate the proposed idea on some other models such as LLaMA, Falcon, Flan-T5 or other models in ```github.com/FranxYao/chain-of-thought-hub```.\n\n- Thank you for the suggestion! We agree that generalization across different models is important and other instruction-tuned LLMs have been examined in our preliminary experiments, in which promising qualitative results are exhibited. These results are relatively incomplete (in terms of the sample size and number of tasks), and we plan to provide a more comprehensive and systematic evaluation in the future. We adopt InstructGPT as it is one of the most popular and capable LLMs with relatively stable controllability available (during the time we conduct our experiments).\n\n> The paper may further evaluate the performance in some other common NLP datasets used in tasks such as summarization, text generation, and natural language understanding.\n\n- Please refer to the response corresponding to “How broadly can the proposed approach work (in which tasks)?” in \"Questions For The Authors\".\n\n> Improvements by the proposed solution are not very surprising in Table 2, in the sense that adding some examples in the in-context learning can outperform the zero-shot prompting. The interesting part might be the part that, the LLM can augment itself by generating some examples.\n\n- Indeed, one of the core ideas we want to demonstrate in this work is that via self-generated demonstrations, we can leverage the intrinsic capabilities of LLMs to bootstrap itself for better zero-shot performance. We will emphasize more on this point in our final version.\n\n> The paper may also include the results by previous approaches (mentioned in Table 1) in Table 2 for benchmarking purposes (so we understand whether the performances are comparable, or there are big gaps). Sometimes the assumptions made by the previous works in Table 2 are not that restrictive or impractical.\n\n- Thanks for the suggestion, we will re-examine the restrictiveness and practicality of previous approaches (mentioned in Table 1) for a more comprehensive and contributive evaluation.\n\nRegarding the “Questions For The Authors”:\n\n> Is the observation generalizable? Can the proposed idea work on some other models such as LLaMA, Falcon, Flan-T5 or other models in ```github.com/FranxYao/chain-of-thought-hub```\n\n- Please refer to our responses corresponding to the first point in “Reasons To Reject”.\n\n> How broadly can the proposed approach work (in which tasks)? \n\n- The effectiveness and generalizability of our proposed Self-ICL could be supported based on the coverage of the BIG-Bench Hard (BBH) benchmark used in our evaluation. BBH encompasses a variety of tasks considered hard and beyond common NLU tasks for LLMs to solve, such as data understanding, object tracking, judging geometric shapes, etc. We agree further evaluation and an in-depth analysis comparing performance categorized by the types of tasks is beneficial, and is left as our future work.\n\n> When the LLM generate incorrect few shot examples, it can be risky to use the proposed approach, and further use them in the in-context learning. It will be great to include some related discussions of the limitations.\n\n- For the purpose of boosting end-task performances, as shown in Figure 5, the LLM could still benefit from potentially incorrect examples and outperform the original 0-shot setting (i.e., ZS-Direct). Also, the entire Self-ICL framework can be executed “behind the scenes” so that end-users will not be exposed to incorrect examples. Nevertheless, we agree it is necessary to include more discussions in the limitations section regarding potential risks. We will incorporate this in our final version. Thank you for pointing this out!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The paper studies an interesting and emerging topic.\n\n(2) The developed solution is simple and works very well. \n\n(3) The developed solution achieves the state-of-the-art performance in some tasks, in the zero-shot setting.", "reasons_to_reject": "(1) It is unclear how generalizable the observation is. The paper only uses InstructGPT. The paper may also evaluate the proposed idea on some other models such as LLaMA, Falcon, Flan-T5 or other models in\n\nhttps://github.com/FranxYao/chain-of-thought-hub\n\nThe paper may further evaluate the performance in some other common NLP datasets used in tasks such as summarization, text generation, and natural language understanding.\n\n(2) Improvements by the proposed solution are not very surprising in Table 2, in the sense that adding some examples in the in-context learning can outperform the zero-shot prompting. The interesting part might be the part that, the LLM can augment itself by generating some examples.\n\n(3) The paper may also include the results by previous approaches (mentioned in Table 1) in Table 2 for benchmarking purposes (so we understand whether the performances are comparable, or there are big gaps). Sometimes the assumptions made by the previous works in Table 2 are not that restrictive or impractical.", "questions_to_authors": "(1) Is the observation generalizable? Can the proposed idea work on some other models such as LLaMA, Falcon, Flan-T5 or other models in\n\nhttps://github.com/FranxYao/chain-of-thought-hub\n\n(2) How broadly can the proposed approach work (in which tasks)? When the LLM generate incorrect few shot examples, it can be risky to use the proposed approach, and further use them in the in-context learning. It will be great to include some related discussions of the limitations.", "paper_topic_and_main_contributions": "Large language models (LLMs) have exhibited striking in-context learning (ICL) ability to adapt to target tasks with a few input-output demonstrations. For better ICL, different methods are proposed to select representative demonstrations from existing training corpora. However, such settings are not aligned with real-world practices, as end-users usually query LMs without access to demonstration pools. \n\nThis paper introduces Self-ICL---a simple framework which bootstraps LMs' intrinsic capabilities to perform zero-shot ICL. The paper presents the first attempt for true zero-shot ICL that does not require any external data from real distribution or pre-defined label sets.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "o7n3l90qVk", "reviewer": "Reviewer_EFGJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments! To alleviate your concerns raises in “Reasons To Reject”:\n\n> The idea that using pseudo-inputs as in-context examples is not novel.\n\n- Although there exist previous works leveraging pseudo-inputs as in-context learning examples, their methods differ from ours in several ways as compared in Table 1 and elaborated in Section 6 (Lines 474 - 505). For example, [1][2] requires access to external text corpora/training set for constructing pseudo-inputs, and labels need to be given beforehand in [3]’s approach, thus restricting their applicabilities. On the contrary, Self-ICL does not have any above-mentioned restrictions. Also, a key idea we want to demonstrate is that via self-generated demonstrations, the LLM can leverage its intrinsic capabilities to bootstrap itself for better performance.\n\n- Reference\n   - [1] Lyu, X., Min, S., Beltagy, I., Zettlemoyer, L., & Hajishirzi, H. (2023, February). Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models.\n   - [2] Zhang, Z., Zhang, A., Li, M., & Smola, A. (2022, September). Automatic Chain of Thought Prompting in Large Language Models. In The Eleventh International Conference on Learning Representations.\n   - [3] Kim, H. J., Cho, H., Kim, J., Kim, T., Yoo, K. M., & Lee, S. G. (2022). Self-generated in-context learning: Leveraging auto-regressive language models as a demonstration generator. arXiv preprint arXiv:2206.08082.\n\n> The paper lacks an in-depth analysis of the impact of various types of pseudo-inputs. For instance, it would be insightful to understand which kind of pseudo-inputs are most effective as in-context examples and how to evaluate their quality.\n\n- Although the current scope of our work focus on demonstrating the LLM’s potential to bootstrap itself by its inner capability, and we have provided analysis along several axes of Self-ICL’s behaviors in Section 5, we agree further investigations focusing on the pseudo-inputs would provide more actionable insights. We will base our analysis in future works on existing literature[1][2][3] discussing how to select representative in-context examples.\n\n- Reference\n   - [1] Sorensen, T., Robinson, J., Rytting, C., Shaw, A., Rogers, K., Delorey, A., ... & Wingate, D. (2022, May). An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 819-862).\n   - [2] Zhang, Y., Feng, S., & Tan, C. (2022, December). Active Example Selection for In-Context Learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 9134-9148).\n   - [3] Gonen, H., Iyer, S., Blevins, T., Smith, N. A., & Zettlemoyer, L. (2022). Demystifying prompts in language models via perplexity estimation. arXiv preprint arXiv:2212.04037.\n\n> Despite leveraging diversity hints, the language model may generate a mixture of high and low-quality pseudo data (e.g., incorrect pseudo-labels). Therefore, it is crucial to develop methods for automatically filtering out noise data and mitigating the impact of low-quality examples, rather than relying solely on the use of \"new,\" \"diverse,\" and \"creative\" prompts to enhance diversity.\n\n- Thanks for the instructive feedback, we acknowledge that developing automatic mechanisms capable of selecting ideal pseudo-demonstrations is of great importance to improve Self-ICL’s robustness on prompting formats. We will incorporate this in our future works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Using the generated pseudo-inputs to improve zero-shot ICL performance is rational. \n2. The paper is well-written and easy to follow.", "reasons_to_reject": "1. The idea that using pseudo-inputs as in-context examples is not novel.\n2. The paper lacks an in-depth analysis of the impact of various types of pseudo-inputs. For instance, it would be insightful to understand which kind of pseudo-inputs are most effective as in-context examples and how to evaluate their quality.\n3. Despite leveraging diversity hints, the language model may generate a mixture of high and low-quality pseudo data (e.g., incorrect pseudo-labels). Therefore, it is crucial to develop methods for automatically filtering out noise data and mitigating the impact of low-quality examples, rather than relying solely on the use of \"new,\" \"diverse,\" and \"creative\" prompts to enhance diversity.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents a novel approach, called Self-ICL, for zero-shot in-context learning (ICL) using pseudo-inputs as contextual examples. The proposed method involves generating pseudo-inputs and predicting pseudo-labels for these inputs via zero-shot prompting. Using the pseudo-input-label pairs as demonstrations, Self-ICL performs ICL for the test input. The paper demonstrates the superiority of Self-ICL over zero-shot baselines in challenging BIG-Bench Hard tasks. In addition, Self-ICL achieves performance comparable to using real demonstrations by leveraging zero-shot chain-of-thought.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "77NvJu48jU", "reviewer": "Reviewer_aCDP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback! To alleviate your concerns raises in “Reasons To Reject”:\n\n> The proposed method requires the user to prompt the LLM to generate pseudo-inputs and answers to each of the test questions, and the user is then required to concatenate the pseudo-demos to the test queries. In contrast, competing methods like AutoCoT or [1] generate a set of pseudo-demos per task but requires the user to provide an unlabelled dataset (which is cheap). On balance, it seems there is still a significant burden on the users, and since the users now need to prompt for model-generated pseudo-inputs and the final predictions, the users need to query the model many more times, so I'm not sure to what extent the claim of \"bridging user's practical need\" is satisfied.\n\n- The claim “bridging user's practical need” is in the sense that prompting LLMs requires much lower technical barrier for end-users in contrast to execute methods like AutoCoT or [1], which requires collecting unlabelled datasets which may or may not be cheap, usually require a large number of instances, and likely need to be stored locally for their own tasks.\n\n- In fact, Self-ICL can be implemented completely behind the scenes by developers such that users have zero burden and can simply prompt the LLMs as usual by giving only instructional test queries like in most LLM web interfaces. Specifically, suppose a test query is entered by users, developers can easily generate pseudo-inputs, pseudo-labels, and concatenate the pseudo-shots (without exposing to end-users) to the test query via a chain of LLM API calling. Therefore, the users do not need to change their behaviors at all since the nature of Self-ICL is a prompting pipeline, that is, everything can be off-loaded to API chaining by developers.\n\n> Moreover, AutoCoT paper also has a section on the streaming setup where the unlabeled queries arrive in an online fashion; this is a practical enough setup, as we can always collect unlabeled user queries when the model is serving requests. I am requesting the authors to clarify better in what scenarios are their purely zero-shot setup particularly advantageous compared to these previous works?\n\n- Though collecting unlabelled user queries online is a practical enough setup, one of the biggest challenges is that these methods require user’s agreement in advance to obtain and store their queries data. Such preconditions for consent might greatly affect users’ willingness to use the system. Moreover, in scenarios such as medical applications the sensitivity of personal data makes collecting user data infeasible.\nBeing purely zero-shot gives Self-ICL advantages over previous works when facing the above-mentioned limitations.\n\n> I wonder whether the authors could 1) state how much more expensive their method is compared to zs-cot and zs-direct, and 2) if the baselines are allowed to use the same amount of compute via self-consistency, is their method still better than the baselines? I think some evidence of this could alleviate the concern that the improvement is simply due to more computation used and can be easily bridged with a simpler alternative.\n\n- Regarding point 1 - For our BBH evaluation, Self-ICL (three shots) is about 7.75 times more expensive than ZS-Direct (zero shot); and 4.12 times more expensive than ZS-CoT (zero shot). The full experimental cost is listed in Appendix A.1.\n\n- Regarding point 2 - Thanks for bringing this up! We have also considered including self-consistency as one of the baselines in our early discussion phase. However, to the best of our knowledge, there were no publicly available results on applying self-consistency to BBH at the time, and due to budget constraints we did not perform such experiments. We agree it would be beneficial to observe the effect of more computation via self-consistency, which we will revisit and try to incorporate in our future version.\n\n> Not a reason to reject, but simply a request for authors to comment: [1] is considered to be a concurrent work, but it considers a very related problem of using ICL in (transductive) zero-shot (similar to AutoCoT). They focus on a different angle in retrieving the best zero-shot demonstrations using a certainty estimate from self-consistency. For example, will the proposed method by the authors benefit from a similar selection procedure? I'd much appreciate if the authors could include discussions of these more recent related works in a revised version of their manuscript.\n\n- Thanks for the suggestion! After reviewing [1], our initial belief is that Self-ICL could further benefit from applying a similar selection procedure like the consistency-based self-adaptive prompting (COSP) proposed in [1]. It is reasonable to assume such a procedure (which is more likely to select “correct” pseudo-labels) can help construct better pseudo-demonstrations and boost Self-ICL’s performance. We are glad to include [1] and other concurrent works as related works and provide a more thorough discussion in our camera-ready paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The methodology is sensible, and I also see the value of this problem setting of extending in-context learning to a strictly zero-shot setup; previous works, like AutoCoT, are *transductive* zero-shot as unlabeled data are typically required -- I do have some concerns and questions regarding this, please see *Reasons to Reject* below.\n\n- Experiments are largely thorough, and the choice of the datasets (the entire BIG-Bench Hard suite of 25 tasks) and models is convincing. The outperformance over the baselines seems significant, which is also supported by statistical tests. \n", "reasons_to_reject": "- Related to the first point I mentioned in *Reasons to Accept*, my concern is as follows: while I see the academic value of the extension to ICL to a strictly zero-shot setup, I wonder whether the authors could clarify the practical significance? \n  - The proposed method requires the user to prompt the LLM to generate pseudo-inputs and answers to *each* of the test questions, and the user is then required to concatenate the pseudo-demos to the test queries. In contrast, competing methods like AutoCoT or [1] generate a set of pseudo-demos *per task* but requires the user to provide an unlabelled dataset (which is cheap). On balance, it seems there is still a significant burden on the users, and since the users now need to prompt for model-generated pseudo-inputs *and* the final predictions, the users need to query the model many more times, so I'm not sure to what extent the claim of \"bridging user's practical need\" is satisfied. \n   \n  - Moreover, AutoCoT paper also has a section on the streaming setup where the unlabeled queries arrive in an online fashion; this is a practical enough setup, as we can always collect unlabeled user queries when the model is serving requests. I am requesting the authors to clarify better in what scenarios are their purely zero-shot setup particularly advantageous compared to these previous works?\n\n- Related to what I mentioned above, given that we need to prompt the LLM more times per test query, the method is also more expensive. On the other hand, the baselines, such as zero-shot-CoT and zero-shot-direct, do not take advantage of the higher budget at all. One simple but effective way to improve the performance of the baselines is through self-consistency [2], especially for the CoT tasks. I wonder whether the authors could 1) state how much more expensive their method is compared to zs-cot and zs-direct, and 2) if the baselines are allowed to use the same amount of compute via self-consistency, is their method still better than the baselines? I think some evidence of this could alleviate the concern that the improvement is simply due to more computation used and can be easily bridged with a simpler alternative.\n\n- Not a reason to reject, but simply a request for authors to comment: [1] is considered to be a concurrent work, but it considers a very related problem of using ICL in (transductive) zero-shot (similar to AutoCoT). They focus on a different angle in retrieving the best zero-shot demonstrations using a certainty estimate from self-consistency. For example, will the proposed method by the authors benefit from a similar selection procedure? I'd much appreciate if the authors could include discussions of these more recent related works in a revised version of their manuscript.\n\n### References\n\n[1]  Xingchen Wan, Ruoxi Sun, Hanjun Dai, Sercan Arik, and Tomas Pfister. 2023. Better Zero-Shot Reasoning with Self-Adaptive Prompting. In Findings of the Association for Computational Linguistics: ACL 2023, pages 3493–3514, Toronto, Canada. Association for Computational Linguistics.\n\n[2] Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., ... & Zhou, D. (2022, September). Self-Consistency Improves Chain of Thought Reasoning in Language Models. In The Eleventh International Conference on Learning Representations.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes Self-ICL, a method for in-context learning in a zero-shot setup. Specifically, the model is first prompted to generate some pseudo-inputs and the model is then asked to respond to these pseudo-inputs to obtain a sequence of pseudo-demonstrations. The pseudo-demonstrations are then concatenated to the test queries in a manner similar to few-shot in-context learning to perform inference. The authors show effectiveness of their method on the BIG-Bench Hard problems with popular GPT models.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Large language models (LLMs) have exhibited striking in-context learning (ICL) ability to adapt to target tasks with a few input-output demonstrations. For better ICL, different methods are proposed to select representative demonstrations from existing training corpora. However, such settings are not aligned with real-world practices, as end-users usually query LMs without access to demonstration pools.
This paper introduces Self-ICL---a simple framework which bootstraps LMs' intrinsic capabilities to perform zero-shot ICL. The paper presents the first attempt for true zero-shot ICL that does not require any external data from real distribution or pre-defined label sets.

Pros:

Well motivated problem, relates to an interesting and relevant topic

Simple intuitive solution of Using the generated pseudo-inputs to improve zero-shot ICL performance

The paper presents the first attempt for true zero-shot ICL that does not require any external data from real distribution or pre-defined label sets.

Cons:

Generalizability of their solution. The paper only uses InstructGPT. The paper may also evaluate the proposed idea on some other models such as LLaMA, Falcon, Flan-T5

Needs to also compare with the methods in Table 1 (AutoCOT, Z_ICL, SG_ICL)  when benchmarking in Table 2

More analysis/ablations on the robustness of the method - e.g. if LLM generates incorrect few shot examples etc
Decision: Accept-Main