Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations
Year: 2023

Abstract:
Although large language models can be prompted for both zeroand few-shot learning, performance drops significantly when no demonstrations are available. In this paper, we introduce Z-ICL, a new zero-shot method that closes the gap by constructing pseudo-demonstrations for a given test input using a raw text corpus. Concretely, pseudodemonstrations are constructed by (1) finding the nearest neighbors to the test input from the corpus and pairing them with random task labels, and (2) applying a set of techniques to reduce the amount of direct copying the model does from the resulting demonstrations. Evaluation on nine classification datasets shows that Z-ICL outperforms previous zero-shot methods by a significant margin, and is on par with incontext learning with few-shot labeled training data. Overall, Z-ICL provides a significantly higher estimate of the zero-shot performance levels of a model, and supports future efforts to develop better pseudo-demonstrations that further improve zero-shot results.1

heading: 1 Introduction
text: Large language models (LMs) can perform new tasks simply by conditioning on input-label pairs from the training data, known as demonstrations (Brown et al., 2020). This in-context learning (ICL) is significantly better than zero-shot methods that do not use demonstrations. Recent work suggests that in-context-learning demonstrations are primarily specifying the domain and the format that the target task, instead of providing explicit training signal (Reynolds and McDonell, 2021; Xie et al., 2022; Razeghi et al., 2022; Min et al., 2022). This implies that current zero-shot performance (with no demonstrations) levels must be significantly underestimated, since all the required information must already be in the model.
1Code available at github.com/alrope123/z-icl.
In this paper, we introduce Z-ICL: Zeroshot In-Context Learning through creating pseudodemonstrations, which achieves results on par with in-context learning from gold demonstrations (Figure 1). The key idea is to construct the pseudodemonstrations following two criteria: (a) they should inform the correct input distribution and the label space, as the k-shot demonstrations do (Xie et al., 2020; Min et al., 2022);2 and (b) they should be constructed to avoid the copying effect—our new observation that the LM predictions are heavily influenced by demonstration inputs that are very close to the test input.
To satisfy (a), Z-ICL retrieves a set of nearest neighbors from a raw text corpus and assigns a random label to each. To satisfy (b), we propose two techniques. We take physical neighbor (adjacent sentences in the corpus) of the nearest sentences instead of the nearest sentences themselves,
2We use pseudo-demonstrations to refer to demonstrations that do not use any training data (either labeled or unlabeled). We use k-shot demonstrations to refer to the more typical demonstrations from the k-shot training data.
2304
so that the sentences in the pseudo-demonstrations are from a similar distribution as the text input but are more distant. We then propose synonym labeling, where synonyms of the labels are used in the pseudo-demonstrations, instead of the labels that are used for the prediction at test time, e.g., {great, terrible}↔{good, bad}. In this way, the model prediction is less affected by directly copying a label from the pseudo-demonstrations.
We evaluate Z-ICL on nine text classification datasets. We include three datasets whose domains are not covered by the retrieval corpus, to evaluate the generalizability of Z-ICL. We experiment with GPT-J (Wang and Komatsuzaki, 2021), GPT-NeoX (Black et al., 2022) and GPT-3 (Brown et al., 2020), whose sizes range from 6B, 20B to 175B. Z-ICL significantly outperforms the previous zero-shot baseline (no-demonstrations) consistently across different datasets and LMs, despite the fact that it does not require any prompt engineering. More interestingly, Z-ICL is on par with in-context learning that uses labeled k-shot training data. Ablations show that (1) constructing a paired format of the pseudo-demonstrations is key to performance, (2) our two techniques—physical neighbor and synonym labeling—are critical, since both of them are required for our pseudo-demonstrations to be on par with k-shot demonstrations, and (3) performance improves as the size and the coverage of the corpus increase.
Together, Z-ICL provides a significantly higher estimate of the ability of current LMs to perform a new task zero-shot, encourages new ways to improve zero-shot performance by designing even better pseudo-demonstrations, and poses a set of new questions about the capabilities of LMs.

heading: 2 Related Work
text: Demonstrations in ICL. A series of prior work suggests that ICL primarily exposes model functionality that was learned during pre-training. Reynolds and McDonell (2021) suggests that ICL mainly functions by activating the LM’s ability obtained during pretraining, and that the LM can achieve significantly better zero-shot performance by using a better template. Xie et al. (2022) shows that ICL can be explained as Bayesian inference for which demonstrations provide noisy evidence. In closed-set tasks, Min et al. (2022) shows that ICL benefits mainly from the correct distribution of the inputs and the labels rather than the input-label
correspondence. Our work draws intuitions from these studies and introduces a better zero-shot method by forming pseudo-demonstrations that are proxies of the input distribution and the label space and better expose the intrinsic ability of the LM.
Better Demonstrations through Retrieval. Prior work has found that, in the setting where large training data is available, choosing demonstration examples that are close to the test input significantly helps ICL. Liu et al. (2021) retrieves the nearest training examples to the test input using a sentence encoder, either unsupervised or supervised. Rubin et al. (2021) trains a retrieval system to choose examples that improve ICL. Liu et al. (2022) retrieves the nearest neighbors from unlabeled training data, assigns estimated labels, and uses them for ICL. We similarly use nearest neighbor search to retrieve sentences close to the test input, but are the first to (1) retrieve from a raw text corpus, in contrast to prior work that uses labeled or unlabeled training data collected for the task, and (2) more closely study the connection between nearest neighbor inputs and random labels, through our copying effect hypothesis.
Copying in ICL. Prior work has explored how seen token patterns affect the ICL’s prediction. Olsson et al. (2022) identifies specific attention heads that, when predicting the next token, look for the previous similar tokens of the current last token in the demonstrations, and copy the tokens following those similar tokens. Our work similarly finds that ICL is prone to copy previously seen text from the demonstrations, but specifically with the particular input-label format in the demonstrations.

heading: 3 Copying Effect Hypothesis
text: In a typical ICL evaluation, the demonstrations are sampled uniformly at random from the true distribution, e.g., the training data in case of existing NLP datasets. We observe that, when demonstrations contain input text that is very similar to the test input, the model exhibits a behavior which we call the copying effect. To study this, we evaluate ICL-gold (standard ICL) and ICL-random; both are ICL methods that use k randomly sampled examples from the training data with gold and random labels, respectively. We then evaluate nearest ICLgold and nearest ICL-random, which follow Liu et al. (2021) in retrieving the k nearest neighbors
for each test input from the training data and assign gold labels and random labels, respectively. We use GPT-J (Wang and Komatsuzaki, 2021) as the LM and SimCSE (Gao et al., 2021) for choosing the nearest inputs.
Results are reported in Figure 2. First, ICLgold and ICL-random achieve relatively comparable performance, which is consistent with Min et al. (2022) that the correctness of labels in the demonstrations matters much less than we thought. However, this does not hold with nearest ICL: using random labels is significantly worse than using gold labels. This indicates that the correctness of labels matters significantly more when the inputs in the demonstrations are closer to the test input.
Based on our observation, we define a copying effect hypothesis: the model prediction is heavily biased toward the labels paired with inputs in the demonstrations that are very similar to the test in-
put, which resembles copying. Table 1 provides an example. The second input in the demonstrations is very close to the test input both lexically and semantically, and the model prediction tends to follow the label paired with the second input, regardless of what that label is.
To better quantify the copying effect, we design an experiment where the demonstrations include an example that is identical to the test input, either with a correct label or with an incorrect label. We then see how many times the LM makes a prediction that is the same as the label paired with the identical demonstration example. Results are reported in Table 2. LM predicts the same label as the one paired with the identical input for over 90% of the times when the label is correct, and over 70% of the times when the label is incorrect, consistently over different LMs.
In the next section, we design a zero-shot method where the copying effect can specifically be problematic, and propose new techniques that reduce the copying effect.

heading: 4 Our Method: Z-ICL
text: Overview. We introduce Z-ICL, a new Zeroshot In-Context Learning method, which predicts the correct label for a given test input x and its
candidate classes Y from a task. Unlike prior methods (Liu et al., 2021; Rubin et al., 2021; Liu et al., 2022) where the target domain and labeled training data of the task are available, Z-ICL constructs pseudo-demonstrations—pairs of inputs and labels—in a zero-shot fashion by leveraging a raw text corpus C, and perform in-context learning.
Z-ICL consists of three steps (Figure 1): retrieving the sentences to approximate the input distribution of the test input (Section 4.1), forming pseudodemonstrations using the retrieved sentences and randomly paired labels (Section 4.2), and making an inference using in-context learning (Section 4.3). Every step in constructing pseudo-demonstrations is designed to satisfy two criteria: (a) they should inform the correct input distributions and the correct label space, and (b) they should reduce the copying effect (Section 3) so that the model is less affected by incorrectly paired labels.

heading: 4.1 Step 1: Retrieve Relevant Sentences
text: In the first step, Z-ICL retrieves k from C that are similar to x. We formally denote s : S × S → R, with S being all sentences from C, as a similarity function between two sentences, and let Nk(x) be a set of sentences c1, · · · , ck retrieved from C with the highest s(ci, x).
It is possible to construct pseudo-demonstrations directly using Nk(x). While this matches the input x well, it is highly likely to suffer from the copying effect (Section 3), since retrieved sentences are too similar to the test input.
To address this, we propose a method called
physical neighbor. Instead of directly using Nk(x), it selects the sentence that is physically adjacent in C to each sentence in Nk(x) as x1, x2...xk. This method allows x1, x2...xk to share similar distribution as x, while being sufficiently distant from x since they are not the k nearest neighbors of x.

heading: 4.2 Step 2: Construct pseudo-demonstrations
text: Once x1...xk are obtained, Z-ICL pairs each xi with a random label following the intuition from Min et al. (2022). While the most straightforward method is to assign the random label from the candidate set Y , this would not achieve the best performance because the LM may find similar sentences from x1...xk and follow their labels according to the copying effect (Section 3).
We therefore propose a technique called synonym labeling: we use synonyms of the labels and pair x1...xk with them, instead of the original labels that will be used for the prediction. Formally, for each xi, Z-ICL chooses a label yi ∈ Y uniformly at random, and creates a pair (xi, ỹj), where ỹj is a manually chosen synonym of yj . We only use synonyms for the pseudo-demonstrations; we use the original candidate set Y during the test prediction. This technique (1) sufficiently informs the correct semantic space of the labels, and (2) prevents the copying effect by not having the exact same words as the test labels.

heading: 4.3 Step 3: Inference
text: Finally, Z-ICL uses in-context learning by concatenating k input-label pairs (x1, ỹ1), (x2, ỹ2), · · · , (xk, ỹk) as well as the test input x, feeds
it to the LM, and obtains the prediction via argmaxy∈YP (y | x1, ỹ1, · · · , xk, ỹk, x). The prediction is made over the original set of labels Y = {y1...y|Y|}, not the synonyms of labels ỹ1...ỹ|Y|.

heading: 5 Experimental Setup

heading: 5.1 Data
text: Text corpus. We use the Demix corpus from Gururangan et al. (2021), a raw text corpus that is not designated for any downstream task. It consists of 16 diverse domains, including Wikipedia, news, Amazon reviews, Yelp reviews, Twitter, and more, all in English. A full list is provided in Table 6 in Appendix A. We subsample up to 10M paragraphs from each domain, and split each paragraph into sentences in order to perform a sentence-level retrieval. More details are provided in Appendix A.
Evaluation datasets. We evaluate our methods on nine single-sentence classification datasets: CR (Ding et al., 2008), Amz (Zhang et al., 2015), Amz5 (Zhang et al., 2015), Yelp (Zhang et al., 2015), Yelp5 (Zhang et al., 2015), Tweet-Eval (Barbieri et al., 2020), MR (Pang and Lee, 2004), SST2 (Socher et al., 2013) and SST5 (Socher et al., 2013). Six out of the nine datasets are from domains that are represented in our corpus, while the other three (MR, SST2, and SST5) are not. This split allows us to measure domain coverage effects. Statistics are reported in Appendix A.

heading: 5.2 Baselines
text: We compare Z-ICL with the following zero-shot methods. See Table 3 for their comparison.
No-demonstrations (No-demos) predicts argmaxy∈YP (y | x) without using any demonstrations. This is a previously-used zero-shot
method (Radford et al., 2019; Brown et al., 2020).
Random inputs selects x1...xk from C uniformly at random, without considering the similarity score with x. It then pairs each xi with a random label from Y and uses in-context learning as in Section 4.3. This baseline uses pseudo-demonstrations, but does not consider the similarity between the test input and the pseudo-demonstrations.
Naive Z-ICL is a version of Z-ICL that uses the most naive retrieval method without the physical neighbor adjustment (Section 4.1) or synonym labeling (Section 4.2). This method encourages the relevance of the pseudo-demonstrations the most, but does not reduce the copying effect.
We also compare with methods that use the training data, and call them Oracle baselines.
ICL-gold (Oracle) uses k input-label pairs from the training data and in-context learning. This is equivalent to the standard in-context learning, first proposed by Brown et al. (2020).
ICL-random (Oracle) uses k inputs from the training data and pairs each input with a random label sampled from Y uniformly at random, and uses in-context learning (Min et al., 2022).

heading: 5.3 Experimental Details
text: Language models. We experiment with three casual language models: GPT-J (Wang and Komatsuzaki, 2021), GPT-NeoX (Black et al., 2022) and GPT-3 (Brown et al., 2020) of sizes 6B, 20B, and 175B, respectively. We use two inference methods: direct (a regular inference used in Brown et al. (2020)) and channel (Min et al., 2021).
Similarity function. We define a similarity function s to be a cosine similarity between two sentence embeddings obtained through SimCSE (Gao et al., 2021).3
Implementation details. For GPT-J and GPTNeoX, we use 5 random seeds and report an average and standard deviation. For GPT-3, we use 2 random seeds and only evaluate on five datasets (CR, Amz, Yelp, Tweet, and SST2) due to limited access. If the dataset includes more than 2,000 test examples, we subsample 2,000 examples uniformly at random without replacement due to limited computing resources, following prior work (Zhao et al., 2021). We use k = 16 for all experiments. We use
3In our initial experiments, we explored multiple embedding methods and found SimCSE works the best.
minimal templates from Zhao et al. (2021) without template engineering, e.g., prepending Review: and Sentiment: to the input and the label, respectively, on a review sentiment classification dataset. More details are provided in Appendix B.

heading: 6 Experimental Results

heading: 6.1 Main results
text: Results using GPT-J and GPT-NeoX are reported in Table 4. No-demos outperforms the majority baseline but lags behind ICL-gold or ICL-random that access the training data, confirming the previous work. Constructing the pseudo-demonstrations using the text corpus significantly helps, e.g., even the “Random inputs” baseline is consistently better than No-demos, likely because it informs the label space and the format to the LM. Naive Z-ICL is better than No-demos in many cases but is still worse than ICL-gold. Finally, Z-ICL, our proposed method, significantly outperforms all baselines. Z-ICL improves zero-shot performance by 5–30% absolute over the existing zero-shot method (No-demos),
consistently over all datasets and all LMs.
Comparison to few-shot ICL. Compared to oracle baselines that access the training data (ICLgold and ICL-random), Z-ICL performs on par on datasets covered by C, despite being zero-shot. This is fairly consistent over all datasets and LMs.
On datasets that are not covered by C, Z-ICL still lags behind ICL-gold and ICL-random. This indicates the importance of the coverage of C in building high-quality pseudo-demonstrations. In Section 6.2, we show improving the coverage of C improves performance on these datasets.
Results with GPT-3. Results on a subset of datasets are reported in Table 5. We find that the findings with GPT-J and GPT-NeoX mostly hold with GPT-3: Z-ICL outperforms the previous zeroshot method (No-demos), and works on par with ICL-gold or ICL-random on datasets covered by C.

heading: 6.2 Ablations
text: We perform detailed ablation studies that break down the importance of each component of Z-ICL.
We evaluate on a subset of 6 datasets (CR, Amz5, Yelp5, Tweet, MR, and SST2) with channel GPT-J unless specified otherwise.
Effect of the retrieval methods. We experiment and compare three different retrieval methods. (1) nearest, a naive retrieval method that directly selects nearest neighbors Nk(x) as x1, x2...xk. (2) diverse nearest, which first retrieves K nearest neighbors with x, NK(x), where K ≫ k, then uniformly samples a random set of k sentences from NK(x) as x1, x2...xk.4 (3) physical neighbor, our main retrieval method introduced in Section 4.1. We do not claim these three methods as the exhaustive set of potential retrieval methods.
Figure 4 indicates that both ‘physical neighbor’ and ‘diverse nearest’ perform well and ‘nearest’ performs the worst consistently over all LMs. This indicates that while informing the input space of the test input, encouraging more diversity in the pseudo-demonstrations is important, presumably
4We use K = 4, 096.
because they are more effective in reducing the copying effect.
Effect of synonym labeling. We aim to answer two questions: (a) How is the effect of synonym labeling when different retrieval methods are used? (b) How important is it to keep the semantics of the label words, e.g., what if we use random words instead of synonyms? To answer these questions, we compare three different methods of assigning labels: (1) using the original test labels, (2) using random words,5 and (3) using the synonyms of the test labels, over the three different retrieval methods.
Results are shown in Figure 5. Using random words is consistently better than using the original labels, indicating that not using words from original test labels is important. Nonetheless, using
5We construct a 1-1 mapping between the original test labels and random English unigrams, and assign the labels. Thus, the number of unique words used in the pseudodemonstrations is the same as the number of unique labels.
synonyms is consistently better than using random words, indicating that informing the semantic space of the labels is still important. While these trends are consistent across different retrieval methods, the gap between using the original labels and using the synonyms is smaller when the retrieval method encourages diversity, e.g., the smallest with the physical neighbor method and the largest with the nearest method. This is likely because the physical neighbor method is already partially reducing the copying effect.
Quantifying the Copying Effect. To better quantify how much the gains are from avoiding the copying effect, we follow Anonymous (2023) in (1) identifying some attention heads in the Transformer layers that are most responsible for copying, and (2) zero-ing their weights out. If this leads to performance improvements, it is a strong indicator that the method has been suffering from the copying effect. We apply this method to three different retrieval methods: nearest, diverse nearest and physical neighbor introduced in Section 4.1.
Figure 7 reports results. First, all methods have performance improvements by zero-ing out the attention heads, indicating that all of them suffer from the copying effect to a certain degree. We then find that (1) nearest is affected the most and physical neighbor is affected the least, and (2) methods with synonym labeling are affected much less than their counterpart without synonym labeling. These are aligned with our earlier intuition that using physical neighbor instead of nearest, and using synonym labeling help reducing the copying effect.
Effect of the size of the corpus. We quantify the impact of the size of the corpus. This is important to judge whether Z-ICL can potentially achieve better results by scaling the corpus. We evaluate Z-ICL with a corpus with varying sizes, from 100% to 0.03% of the corpus.
Figure 6 demonstrates that performance goes down as the size of the corpus gets smaller. This is likely because there are less sentences that are sufficiently close to the test input when the corpus is smaller, thus the relevance of the nearest neighbors and the test input drops. This trend is clearer on the datasets covered by C than on the datasets not covered by C.
Effect of the format of demonstrations. How many input-label pairs does Z-ICL need to benefit from pseudo-demonstrations? Are gains from pseudo-demonstrations mainly from the fact that the LM conditions on relevant text, or does the LM benefit from a specific format of the pseudodemonstrations: a concatenation of input-label pairs? To answer these questions, we experiment with (1) Z-ICL with varying range of k from 1 to 64, and (2) a variant of Z-ICL where the LM conditions on a concatenation of retrieved inputs, without randomly paired labels (called “Inputs-only”).
Results are shown in Figure 8. First, Z-ICL is sig-
nificantly better than zero-shot baselines and stays on par with the oracle baselines consistently across different values of k. Moreover, using no labels (“Inputs-only”) performs significant worse than its counterparts. This suggests that Z-ICL takes advantages of the form of input-label pairs, and is beyond simply conditioning on relevant context.
Effect of the coverage of the corpus. We quantify the impact of the coverage of the corpus, and whether adding more domains in the corpus improves performance. We do so by adding the unlabeled portion of IMDB review (Maas et al., 2011) to the corpus C. The size of C increases only by 2%, but covers the domain of three datasets that were previously not covered (SST2, SST5 and MR).
Figure 9 shows the performance on three datasets before and after adding the IMDB corpus. Performance improves consistently over all LMs, even though it only adds up the size by 2%. This suggests that the coverage of the text corpus is important, and it is feasible to further improve the overall performance simply by expanding the corpus.

heading: 7 Conclusion
text: We introduced Z-ICL, a zero-shot in-context learning method that constructs pseudo-demonstrations from a raw text corpus. Our method (1) retrieves relevant text from the corpus using the nearest neighbor search, effectively informing the correct space of the inputs to the LM, and (2) adjust the pseudo-demonstrations with physical neighbor and synonym labeling to avoid the copying effect. Evaluation on nine classification datasets shows Z-ICL significantly outperforms the previous zero-shot baseline, and performs on par with the k-shot
demonstrations. Overall, Z-ICL demonstrates that significantly higher LM zero-shot performance is possible, and opens up a new research direction on the construction of better pseudo-demonstrations that expose the full capacity of a LM.
Limitation
Extension to multi-sentence tasks. Our experiments are limited to single-sentence tasks, as we only retrieve single-sentence nearest neighbors to a test input. Multi-sentence tasks such as natural language inference would require constructing pseudo-demonstrations that consists of multiple sentences, which we leave for future work.
Beyond classification. Our experiments are limited to classification. Extensions to multi-choice tasks or generation tasks requires going beyond a fixed set of options shared between inputs in the demonstrations and the test input. We leave extensions to non-classification tasks for future work.
Better construction of pseudo-demonstrations. We think future work can explore better constructing the pseudo-demonstrations. For instance, this paper uses manually chosen synonym labels (see Appendix B for more detail). We hypothesize that better pseudo-demonstrations can improve performance, which we leave for future work.

heading: Acknowledgements
text: We thank UW NLP members and anonymous reviewers for their comments in the paper. This research was supported by NSF IIS-2044660, an Allen Distinguished Award and gifts from AI2. SM is supported by a J.P. Morgan fellowship.

heading: A Data Statistics
text: Corpus. We take the same English corpus from (Gururangan et al., 2021) covering 16 diverse domains: 1B, CS, LEGAL, MED, WEBTEXT, REALNEWS, REDDIT, REVIEWS, ACL PAPERS, BREAKING NEWS, CONTRACTS, CORD-19, GITHUB, GUTENBERG, TWEETS, and YELP REVIEWS. See the descriptions and statics in Table 6. For each domain, we 1) subsample 10M paragraphs if the data is larger, 2) split each paragraph into sentences, and 3) remove duplicate sentences while keeping the ordering of the sentences as in the original paragraphs.
Evaluation datasets. Statistics and descriptions of our evaluation datasets are reported in Table 7. For each dataset, we subsample 2000 test examples uniformly at random if the test data is larger, due to limited computational resources.
B Implementation Details
All implementations are done in PyTorch (Paszke et al., 2019). We use int8 quantization (Zeng et al.,
2022) to run GPT-NeoX on 40GB A100 machines.
Format of the demonstrations. We use k = 16 demonstration examples for all the baselines and methods, unless specified otherwise. We truncate each demonstration example to have up to 256 tokens and the concatenation of them to have up to 1,024 tokens.
Nearest neighbor search. We use SimCSE (Gao et al., 2021) to embed the corpus and the test inputs. We use FAISS (Johnson et al., 2019) to build an index for the corpus offline and perform nearest neighbor search at inference.
Synonym labeling. We manually choose a synonym of each label to perform synonym labeling. A full list of synonyms is reported in Table 7.
Computational Budget. Our main experiment on the 4 public LMs in Table 4 takes around 4,000 computing hours with a 40GB A100 machine. Our experiment using GPT-3’s API costs around 4,500 US Dollars.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
In Limitation
7 A2. Did you discuss any potential risks of your work? Our paper proposes a method for constructing demonstrations for in-context learning using a raw text corpus from Demix. The raw text corpus may contain unintended bias or harmful content, despite the authors of the original paper’s best efforts to remove them.
3 A3. Do the abstract and introduction summarize the paper’s main claims? In Abstract + Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? In Section 4
3 B1. Did you cite the creators of artifacts you used? In Section 4 and Section 5
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The open-source code will point to the license and terms for use for evaluation datasets and the text corpus. They are not included in the submission in order to keep the anonymity.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? In Appendix D
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? In Appendix D
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? In Section 5
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. In Section 5.3 and Appendix A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? n Section 6
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? In Appendix D
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? There is no important hyperparameter in our experiment setting.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? In Section 6
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? In Section 5.3 and Appendix B
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Multi-sentence tasks such as natural language inference would require constructing pseudo-demonstrations that consists of multiple sentences, which we leave for future work. Beyond classification. Our experiments are limited to classification. Extensions to multi-choice tasks or generation tasks requires going beyond a fixed set of options shared between inputs in the demonstrations and the test input. We leave extensions to non-classification tasks for future work. Better construction of pseudo-demonstrations. We think future work can explore better constructing the pseudo-demonstrations. For instance, this paper uses manually chosen synonym labels (see Appendix B for more detail). We hypothesize that better pseudo-demonstrations can improve performance, which we leave for future work