Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrations
Year: 2023

Abstract:
In-context learning (ICL) is an important paradigm for adapting large language models (LLMs) to new tasks, but the generalization behavior of ICL remains poorly understood. We investigate the inductive biases of ICL from the perspective of feature bias: which feature ICL is more likely to use given a set of underspecified demonstrations in which two features are equally predictive of the labels. First, we characterize the feature biases of GPT-3 models by constructing underspecified demonstrations from a range of NLP datasets and feature combinations. We find that LLMs exhibit clear feature biases—for example, demonstrating a strong bias to predict labels according to sentiment rather than shallow lexical features, like punctuation. Second, we evaluate the effect of different interventions that are designed to impose an inductive bias in favor of a particular feature, such as adding a natural language instruction or using semantically relevant label words. We find that, while many interventions can influence the learner to prefer a particular feature, it can be difficult to overcome strong prior biases. Overall, our results provide a broader picture of the types of features that ICL may be more likely to exploit and how to impose inductive biases that are better aligned with the intended task.1

heading: 1 Introduction
text: In-context learning (ICL) is an increasingly popular paradigm for adapting large language models (LLMs) to downstream tasks (Brown et al., 2020). It works by prompting LLMs with a small set of examples that demonstrate the input and output of a task, without requiring any parameter updates. However, a key limitation of ICL is that it can only incorporate a small number of demonstration examples (e.g., 16) due to the context length limit of
*Equal contribution. 1Our code and data are publicly available at
https://github.com/NoviScl/AmbigPrompt.
Transformer models, meaning that most tasks will be highly underspecified. For example, as shown in Figure 1, we present the model with an underspecified text classification problem. The sentences with label ‘1’ are positive reviews of movies, and the sentences with label ‘0’ are negative reviews of restaurants. From the demonstrations, it is unclear whether the labels are determined by the sentiment feature (positive vs. negative) or the topic feature (movie vs. food). Moreover, due to the limited context window, it is difficult to specify the task
11289
by supplying a large number of additional training examples. Instead, ICL can succeed only if (a) the LLM has an inductive bias that happens to align well with the given task or (b) there is a mechanism for imposing an inductive bias on the system, which can specify the task (e.g., whether it is sentiment classification or topic classification) without incorporating more training examples.
In this paper, we study the inductive biases of ICL with LLMs and measure the effectiveness of different interventions to steer ICL towards a particular hypothesis. We focus on feature biases: a tendency to use one feature rather than another given a prompt in which the two features predict the label equally well. As illustrated in Figure 1, by evaluating the model on sentences where the two features disagree—such as positive restaurant reviews—we can measure the feature bias of the learning algorithm, and we can attempt to modify the feature biases through various interventions applied to the prompt.
In the first part of the paper, we characterize the feature biases of ICL by constructing underspecified demonstrations from a variety of NLP datasets and feature pairs. We find that ICL models exhibit some clear feature biases. For example, in a sentiment analysis setting, the LLMs we study exhibit a strong tendency to generalize on the basis of sentiment rather than other equally predictive features, including topics of sentences. On sentence-pair tasks, such as question answering and natural language inference, the GPT-3 model (Brown et al., 2020) generally prefers shallow lexical features, while the instruction-tuned models (Ouyang et al., 2022) generalize more consistently with the labels associated with those datasets. Such feature biases could potentially be problematic—users could have intended either of the two predictive features as the actual task. When the model’s feature bias does not align with the intended task, we want the model to be steerable with appropriate interventions.
In the second part of the paper, we measure whether simple modifications of the prompt can supply an inductive bias to the ICL learning algorithm, steering the model to generalize according to one feature rather than another. These interventions include using natural-language instructions or explanations and using label words that are semantically related to the intended feature. As a baseline, we compare these methods with unambiguous prompts, in which some demonstration
examples are consistent with one hypothesis but not the other. We find that these interventions are most effective when the model does not have a strong feature bias, or already has a bias in favor of the intended task feature. They are less effective at steering the model to use one feature when it has a strong bias to use another feature. For example, the instruction-tuned model generalizes on the basis of sentiment despite adding instructions and even disambiguating evidence in favor of lexical features like punctuation. Interestingly, we find that data-independent methods, like using semantically relevant label words, sometimes have a stronger effect than providing unambiguous data.
These results underscore some of the challenges involved with using ICL as a general-purpose machine learning method, complementing a growing body of work that has attempted to explain how ICL works from an empirical (e.g., Min et al., 2022; Webson and Pavlick, 2022; Chan et al., 2022) and theoretical (Xie et al., 2022; Akyürek et al., 2023; von Oswald et al., 2022) point of view. On one hand, the strong inductive biases of LLMs are helpful when they happen to be well aligned with the given task, enabling ICL to generalize successfully from very few training examples. Moreover, simple modifications to the prompt are often successful at steering the model towards a particular feature in underspecified settings. On the other hand, simple prompting methods cannot systematically align the model with user intention: they have limited effectiveness when the model’s feature biases conflict with the intended task.

heading: 2 Setup

heading: 2.1 Measuring Feature Biases
text: We consider text classification problems, where x ∈ X is a text input and h1, h2 : X → {0, 1} are two binary feature functions. For example, h1 may be a sentiment classifier (returning 0 if x is negative and 1 if it is positive), and h2 is a domain classifier indicating whether x is a review of a movie or a restaurant. We consider a learning algorithm `, defined as a mapping from a dataset D ⊆ X × {0, 1} to a classifier f : X → {0, 1}. Given a learning algorithm ` and a pair of feature functions h1, h2, our aim is to understand whether ` tends to return classifiers more similar to h1 or h2, given that h1 and h2 have the same accuracy on D.
In the context of ICL, we measure this property behaviorally by prompting language models with a
set of underspecified demonstrations Ddemo and evaluating the resulting function f = `(Ddemo) on a disambiguating dataset Dtest. The underspecified demonstrations are examples Ddemo ∈ X × {0, 1} such that, for all (x, y) ∈ Ddemo, y = h1(x) = h2(x); and we ensure that the labels are balanced on D. The disambiguating dataset Dtest is constructed so that, for all x, h1(x) 6= h2(x), and the dataset is balanced with respect to h1(x) and h2(x). A simple example is illustrated in Figure 1.
We measure whether f is more consistent with h1 or h2 by comparing the predictions of f , h1, and h2 on Dtest. For a given feature function h, we define the accuracy on h as the proportion of examples for which f(x) = h(x):
h-accuracy = 1 |Dtest| ∑
x∈Dtest 1[f(x) = h(x)].
The difference between h1-accuracy and h2accuracy on Dtest can be interpreted as a feature bias: for example, a high value of h1-accuracy indicates that the model is more likely to predict the same label as h1 in situations where h1 and h2 disagree. h1-accuracy and h2-accuracy always add up to 1 and, because Dtest is balanced, an h1-accuracy of 0.5 indicates that the model does not exhibit a strong bias towards either feature.

heading: 2.2 In-Context Learning
text: The learning algorithms we consider in this paper are based on in-context learning (ICL) of LLMs (Brown et al., 2020). A language model pθ(w) assigns scores to strings w ∈ V∗, where V is a discrete vocabulary. The input to ICL is a language model pθ(w) and a function that converts a dataset D and a single test example xtest into a prompt c(D, xtest) ∈ V∗. We consider the basic form of ICL, which consists of2: (1) an instance template t : X → V∗ that encodes each data instance x as a string; (2) a label verbalizer v : {0, 1} → V∗ that encodes each label as a string. For the first part of our analysis on measuring feature biases (Section 4), we adopt the simplest format and define the instance template as t(x) = “Input: $x Label: ”, and the label verbalizer as v(0) = “0” and v(1) = “1”. The prompt c is then the concatenation of t(xi) and v(yi) for all demonstration examples (xi, yi) ∈ D; and lastly
2Additionally, we can incorporate natural-language instructions and explanations, as we will discuss in Section 5.
the test instance t(xtest). The resulting classifier is:
f(xtest;D) = argmax y pθ(v(y) | c(D, xtest)).
ICL is known to be sensitive to the order of the demonstrations (Lu et al., 2022) and to demonstrate other biases that are orthogonal to this study, including majority label bias and recency bias (Zhao et al., 2021). We control for these by ordering the demonstration examples randomly and performing label calibration, following (Zhao et al., 2021).

heading: 3 Data Construction
text: We choose datasets to cover four different NLP tasks, including both single-sentence and sentencepair classification. For sentiment analysis, we use IMDb (Maas et al., 2011) and Yelp (Asghar, 2016) datasets; for toxicity classification, we use the CivilComments dataset (Borkan et al., 2019); for natural language inference, we use the MNLI dataset (Williams et al., 2018); and for question answering, we use BoolQ (Clark et al., 2019).
For each dataset, we select the original classification label as the default feature and denote it as h1. We select alternative comparison features (h2) using existing metadata or simple, deterministic functions, chosen to reflect realistic sources of ambiguity or spurious correlation that have been studied in prior work (McCoy et al., 2019; Gururangan et al., 2018; Poliak et al., 2018; Joshi et al., 2022), as well as common shallow features such as length, capitalization, and the presence of particular words or punctuation marks. All datasets and features we use are listed in Table 1, which we elaborate below:
(1) For sentiment analysis, the default feature is the sentiment, and the alternative features include: domain or source of the review (based on whether it is from IMDb or Yelp), length of the review (longer or shorter than a threshold), the final punctuation mark of the review (exclamation mark or period), whether it contains certain keywords (“food” and “nice”), and whether it contains uppercase words (e.g., “THIS”).
(2) For toxicity classification, the default feature is whether the comment is toxic. The alternative features are: gender, sexuality, religion, and race mentioned in the comment (all based on humanannotated meta-data), and its length and capitalization (whether containing uppercase words).
(3) For NLI, the default feature is the entailment relationship between the sentence pair, and we con-
dense the neutral and contradiction classes as nonentailment to cast the task as binary classification. For alternative features, we consider: domain of the text (from the genre meta-data); lexical overlap, following the definition in HANS (McCoy et al., 2019); whether the hypothesis is shorter or longer than the premise; and whether the hypothesis contains negation words (“no”, “not”, “n’t”).
(4) For question answering, the default feature is whether the answer is yes or no, and the alternative features are: the question word, whether all words from the question also appear in the passage (lexical overlap), question structure (whether it is a comparison question), and passage length. These features are potential spurious features in QA that have been documented in prior work (Pezeshkpour et al., 2022; Shinoda et al., 2022).

heading: 4 Measuring Feature Biases

heading: 4.1 Experiment Details
text: Evaluation protocol. For all experiments, we randomly sample a balanced set of 16 demonstration examples (randomly shuffled) to form the prompt. For eight of the examples, y = h1(x) = h2(x) = 0; for the other eight, y =
h1(x) = h2(x) = 1. For each experiment, we randomly sample three sets of prompts and report the average performance on a set of 1,200 test examples, balanced between examples with h1(x) = 0/h2(x) = 1 and h1(x) = 1/h2(x) = 0. In this baseline setting, the instance template is t(x) = “Input: $x Label: ”, and the label verbalizer provides no additional information about the task: v(0) = “0” and v(1) = “1”.
Models. We focus on the TEXT-DAVINCI-002 and DAVINCI checkpoints of GPT-3 mainly because smaller-scale models often perform no better than random on tasks like NLI. The main differences between the two checkpoints are the pretraining data and the additional instruction tuning for TEXT-DAVINCI-002 (Ouyang et al., 2022). For all experiments, we use a temperature value of 0 in GPT-3 decoding, and all experiments involving the OpenAI API were conducted in January 2023.
Metric. We report the h-accuracy for each feature. Higher h-accuracy indicates a higher preference for that feature. We denote the default feature h1 (sentiment, toxicity, entailment, answer) and the alternative features h2.

heading: 4.2 Results
text: We present ICL results on all the datasets and pairs of features in Figure 2, and note several interesting trends as follows:
LLMs often have clear feature biases. For example, in the sentiment analysis setting (Figure 2a), both models generally show a strong preference to predict labels according to the sentiment of the sentence rather than other features, such as sentence length or the presence of individual lexical items like the word “food”. Such biases can be helpful when they are aligned with the intended task.
On the other hand, we do not observe clear feature preferences in the CivilComments dataset (Figure 2c), suggesting that these models may not have a strong feature biases in this setting.
The instruction-tuned model is generally more aligned with standard dataset labels. While both DAVINCI and TEXT-DAVINCI-002 show similar feature biases in the sentiment analysis setting, they show very different biases on the the sentence pair datasets MultiNLI (Figure 2b) and BoolQ (Figure 2d): the DAVINCI model tends to prefer the shallow distractor features, such as lexical overlap,
while TEXT-DAVINCI-002 tends to prefer the semantic feature associated with the dataset—either the entailment relationship or the answer to the question. This may be due to some aspect of instruction tuning, which might have exposed the model to similar tasks.

heading: 5 Comparing Interventions
text: Our findings that LLMs can have strong feature biases have important implications: when the LLMs’ biases do not align with users’ intended task, such biases would hurt performance. To resolve such misalignment, we explore a set of intervention methods designed to encourage the model to prefer one feature over another, examining whether the haccuracy for the intended feature indeed increases.

heading: 5.1 Experiment Details
text: We now consider more variants of ICL that can be decomposed into four components that are commonly used in various prompting methods. In addition to the instance template t and label verbalizer v described in Section 2.2, we also introduce: (1) An instruction s ∈ V∗, which is prepended to the prompt; and (2) a free-form explanation
e : X → V∗ after each input text t(x) and before the label v(x). The prompt c is then the concatenation of s, followed by t(xi); e(xi); v(yi) for all demonstration examples (xi, yi) ∈ D; and lastly the test instance t(xtest).
Each intervention operates on a combination of the above components. We apply these interventions separately and compare with the baseline rather than applying all interventions on top of each other in order to analyze the impact of each of the methods. We compare interventions designed to steer the model towards h1 and h2 as the intended feature respectively, to understand the effectiveness of interventions towards different features.
• Recall that in the baseline setting, there is no instruction or explanation (s and e are empty strings). We simply concatenate demonstration examples as the prompt, and use “1” and “0” as the verbalizer.
• In the semantic verbalizer setting, the verbalizer selects label words that are semantically related to a chosen feature in order to hint at the intended task. For example, if the intended feature is sentiment, then v(0) = “negative” and v(1) = “positive”; and if the intended feature is length, then v(0) = “short” and v(1) = “long”. Our choice of verbalizers is inspired by prior work (Gao et al., 2021; Shi et al., 2022) and we list all of them in Table 6.
• In the instruction setting, we add a prefix string describing the intended feature and instructing the model to use this feature. We format our instructions mostly following prior work such as Natural Instructions (Mishra et al., 2021; Wang et al., 2022), and we list all our instructions in Tables 7 and 8.
• In the explanation setting, we append a template explanation after each demo example to explain why the prediction is made based on the intended feature, formatted in a similar manner as Chain-of-Thought prompting (Wei et al., 2022) and “explain-then-predict” (Ye and Durrett, 2022). Since hand-written explanations are hard to obtain, we create fixed human-written templates for each feature value. For example, for the punctuation feature, all positive examples have the explanation “The review ends with an exclamation
mark. Therefore, the answer is 1”. We list all our template explanations in Tables 9 and 10.
• Finally, we include a disambiguation setting, in which we change half of the demonstration examples to those that disambiguate the task in favor of the intended feature. For example, to steer the model towards h1, the demonstration includes examples such that h1(x) 6= h2(x) and y = h1(x). Intuitively, this provides additional evidence for the model to differentiate the intended feature.
We measure the effectiveness of the intervention in terms of the increase in h-accuracy relative to the baseline model, where h is the intended feature.

heading: 5.2 Results
text: Which interventions are effective? Table 2 summarizes the results of these experiments, comparing the effect of the different interventions on DAVINCI and TEXT-DAVINCI-002, averaged over features and datasets, and comparing between interventions designed to steer towards the default feature (h1) and the alternative features. Nearly all interventions increase the average h-accuracy, in many cases by a substantial amount. However, the effectiveness of the intervention varies depending on the model. For the DAVINCI model, the bestperforming interventions include semantic verbalizers and template-based explanations, while the worst-performing intervention is natural-language instructions. For the TEXT-DAVINCI-002 model, instructions are much more effective.
In some cases, prompt-based interventions are more effective at steering the model than providing unambiguous demonstration examples. On one hand, this suggests that ICL can be effective even given highly underspecified data, but it also indicates that ICL models may fail to exploit the information provided in the demonstrations. This finding illustrates that ICL works very differently from standard supervised learning, and calls to mind existing empirical (Min et al., 2022) and theoretical results (Xie et al., 2022) suggesting that ICL might work in part by recognizing existing tasks rather than directly learning the input-output relation from the demonstration.
When are interventions effective? Figure 3 compares the results of different interventions on the TEXT-DAVINCI-002 model, aggregated over features. (Detailed results for each feature and DAVINCI results are in Appendix A.1.) The effectiveness of the intervention varies depending on whether the prior feature bias and the intended feature are aligned. The interventions are most effective in two scenarios. First, interventions are effective when the model already has a feature bias for the intended features. This is evident in the interventions that steer the model towards h1 in
NLI and QA, settings in which the model already has a feature bias in favor of h1. Second, interventions are effective when the model has a low feature bias. This is the case in the Toxicity Classification dataset, where the model does not exhibit a strong feature bias. In this setting, all interventions are moderately successful at steering the model towards h1, and more successful at steering the model towards h2.
On the other hand, interventions are less effective at overriding feature biases. This trend is illustrated in the second row of Figure 3, in which the intervention is designed to steer the model towards h2 rather than the standard dataset label. While some interventions increase h2-accuracy, no intervention consistently gets the model to generalize according to the intended feature.
Which features are most susceptible to interventions? In Table 3, we compare the effect of interventions on different features in MultiNLI. Using meaningful label words works better on the genre features, where the label words are semantically similar to the input example, but it is more difficult to steer the model toward the use of features like length and lexical overlap, which are not related to the semantics of the sentences. More work may be
needed to develop interventions that work well for higher-order language features.
Lastly, we compile a summary of practical takeaways for steering LLMs:
• When using non-instruction-tuned LLMs (e.g., DAVINCI), specifying feature preferences as instructions is not effective, instead adding explanations or disambiguating examples can be more effective.
• When using instruction-tuned LLMs (e.g., TEXT-DAVINCI-002), all interventions can be effective.
• Features not related to semantics, such as sentence lengths or overlap, are difficult to intervene across all conditions.

heading: 6 Related Work
text: Measuring inductive biases. Our work builds on existing research on measuring the inductive biases of learning algorithms in machine learning and NLP. Dasgupta et al. (2022) introduce a methodology for measuring feature bias as well as rulevs. exemplar-based generalization, and Chan et al. (2022) apply this approach to compare rule- vs. exemplar-based learning in ICL. We use a similar framing as Dasgupta et al. (2022), but focus on feature bias. In NLP, another line of work has studied the inductive biases of neural networks in the context of the poverty of the stimulus argument (Chomsky, 1980). These studies evaluate whether neural networks generalize in a manner consistent with structural or superficial linguistic rules (McCoy et al., 2018, 2020). Several studies
have found that models such as BERT acquire a preference for structural generalizations from largescale masked language model pretraining (Warstadt and Bowman, 2020; Warstadt et al., 2020; Zhang and Hashimoto, 2021; Mueller et al., 2022). We follow a similar poverty-of-the-stimulus experimental setup but focus on in-context learning and on features arising in common NLP tasks. Lovering et al. (2021) explore whether it is possible to predict the inductive biases of pre-trained models and show that models tend to generalize on the basis of features that are more “extractable”, measured using probing techniques (Voita and Titov, 2020), but it is not straightforward to extend the notion of extrability to in-context learning. Tamkin et al. (2023) also study how LLMs generalize to ambiguous classification tasks, but focus on ambiguous instructions and use template-generated datasets.
Spurious correlations. A related line of work has explored the inductive biases of pretrained LMs in relation to spurious correlations, or shortcuts (e.g. Gururangan et al., 2018; Poliak et al., 2018; McCoy et al., 2019; Geirhos et al., 2020; Sagawa et al., 2020)—shallow features that are correlated with the classification targets. Models can generalize successfully if they have an inductive bias that tends to favor the intended feature over the shortcut. Hendrycks et al. (2019, 2020); Tu et al. (2020) found that pre-trained models can generalize more successfully to such distribution shifts.
Explaining in-context learning. A number of empirical studies have attempted to characterize the behavior of ICL and explain why it works. These studies have found that ICL can be overly sensitive
to certain aspects of the prompt, such as the order of the demonstration examples (Lu et al., 2022) and choice of label words (Zhao et al., 2021), but also surprisingly insensitive to others. In particular, Min et al. (2022) show that LLMs can largely ignore the relationship between inputs and labels in the demonstration example and Webson and Pavlick (2022) show that the performance of ICL can perform well on NLI even if the prompt is irrelevant or misleading. Relatedly, Wei et al. (2023); Pan et al. (2023) show that LLMs can perform ICL well with flipped labels or semantically-unrelated labels, but such ability of overriding semantic priors emerges with scale. Some theoretical work has also attempted to explain why prompt-based methods work, by drawing connections between the prompting setting and properties of the pretraining distribution (Saunshi et al., 2021; Wei et al., 2021; Xie et al., 2022) or by arguing that Transformers can act as meta-learners, implicitly performing gradient descent on the in-context examples (von Oswald et al., 2022; Akyürek et al., 2023). Our results provide empirical evidence that there is a strong task bias from pretraining when the LLMs must infer the task by input-output relations.
Improving in-context learning. Recent work studied the effect of including explanations in the prompt to produce better quality answers (Wei et al., 2022; Lampinen et al., 2022). While they show the benefit of high-quality human-annotated explanations for improving task performance, we demonstrated the effectiveness of simple template explanations in steering feature biases. Another line of work collects large pools of instructions from diverse tasks and uses them to tune and control language models (Wang et al., 2022; Chung et al., 2022). We also adopt instructions as an intervention method and show that it works particularly well on instruction-tuned models. In a similar manner, Si et al. (2023) studied prompting methods to make GPT-3 more reliable, such as instructing it to not rely on demographic biases.

heading: 7 Conclusion
text: In this work, we constructed underspecified prompts from real-world datasets to study feature biases of large language models. We found that the instruction-tuned InstructGPT model prefers the “default” task features over distractor features more often than the base GPT-3 model, and we demonstrated the effectiveness of various intervention
methods in steering models to use the specified feature. These results not only shed new insights into the working mechanisms of ICL, but also have practical takeaways for discouraging models from exploiting unintended features such as demographic biases or shallow statistical cues.
Limitations
Model coverage. Our study is targeted specifically at GPT-3 and it would be interesting to study feature bias patterns on other large language models such as OPT (Zhang et al., 2022) and BLOOM (Scao et al., 2022); and it is possible that our intervention methods may have different effects on these language models trained with different data sources and scales.
Task coverage. Apart from model coverage, our analysis is focused on only four common binary classification tasks. Our main metric, h-accuracy, compares the predictions between a learned function f and a feature function h. For simplicity, we only study binary functions (consistent with prior work) to illustrate the main ideas, but the framework applies equally well if f and h are multi-class classifiers. For example, in the case of the threeway NLI task, we might set h1 to predict on the basis of entailment / contradiction / neutral, and h2 to predict on the basis of the genres – e.g. fiction / government / telephone. Future work could extend our framework to more tasks, and consider how to apply it to more complex tasks such as generation.
Feature coverage. Our current experiments are limited to a set of hand-crafted features. One potential way to systematically scale our approach is to develop novel techniques for automatic feature discovery, for example, to cluster the data and treat each cluster as having a distinct feature.
Explaining feature biases. While our empirical findings shed light on the feature bias patterns of GPT-3, we do not yet have a conclusion on how these biases arise from pretraining. Future work could attempt to draw connections to the pretraining data or to theoretical accounts of in-context learning.
Risks and ethics. While we do not foresee any ethical risks resulting from our work, we caution against making extrapolations about the extent to which LLMs exhibit feature biases towards protected social attributes. Although we do not find
evidence of strong feature biases in a particular toxicity classification setting, care should be taken to evaluate the fairness and reliability of these models directly before they are deployed in downstream applications.

heading: Acknowledgement
text: We thank Alex Tamkin, Xi Ye, Sewon Min, and Jordan Boyd-Graber for their helpful feedback. This research is partially funded by the National Science Foundation (IIS-2211779), a Sloan research fellowship, Samsung Advanced Institute of Tech- nology (under the project Next Generation Deep Learning: From Pattern Recognition to AI) and AWS AI. NJ is supported by an NSF Graduate Research Fellowship under grant number 1839302.

heading: A Appendix
text: A.1 Intervention Results Across Features We present the complete set of intervention results broken down by features in Table 4 for DAVINCI and Table 5 for TEXT-DAVINCI-002.
It is worth noting that the intervention methods’ effectiveness often varies across features even on the same dataset. For example, all intervention methods can effectively steer models towards using the genre feature over the entailment feature on MNLI, but the success is limited for the lexical overlap feature on MNLI. We hypothesize this is because features like lexical overlap are harder for models to recognize.
A.2 List of Semantic Verbalizers We present the full list of semantically meaningful verbalizers for the intervention experiments in Table 6.
A.3 List of Task Instructions We present the full list of task instructions for the intervention experiments in Table 7 and Table 8.
A.4 List of Template Explanations We present the full list of template explanations for the intervention experiments in Table 9 and Table 10.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Last section
3 A2. Did you discuss any potential risks of your work? Last section
3 A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Left blank.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Future work could extend our framework to more tasks, and consider how to apply it to more complex tasks such as generation. Feature coverage. Our current experiments are limited to a set of hand-crafted features. One potential way to systematically scale our approach is to develop novel techniques for automatic feature discovery, for example, to cluster the data and treat each cluster as having a distinct feature. Explaining feature biases