SlowGuess/NAACL_2022_OCR huggingface dataset:
Daniel Khashabi‡ Xinxi Lyu† Sewon Min†  
Lianhui Qin† Kyle Richardson‡ Sean Welleck†‡  
Hannaneh Hajishirzi†‡ Tushar Khot‡ Ashish Sabharwal‡ Sameer Singh‡♥ Yejin Choi†‡
†University of Washington ‡Allen Institute for AI ℃University of California-Irvine
Fine-tuning continuous prompts for target tasks has recently emerged as a compact alternative to full model fine-tuning. Motivated by these promising results, we investigate the feasibility of extracting a discrete (textual) interpretation of continuous prompts that is faithful to the problem they solve. In practice, we observe a "wayward" behavior between the task solved by continuous prompts and the nearest neighbor discrete projections of these prompts: One can find continuous prompts that solve a task while being projected to an arbitrary text (e.g., definition of a different or even a contradictory task) and simultaneously being within a very small  $(2\%)$  margin of the best continuous prompt of the same size for the task. We provide intuitions behind this odd and surprising behavior, as well as extensive empirical analyses quantifying the effect of design choices. For instance, larger models exhibit higher waywardness, i.e., we can find prompts that more closely map to any arbitrary text with a smaller drop of accuracy. These findings have important implications relating to the difficulty of faithfully interpreting continuous prompts and their generalization across models and tasks, providing guidance for future progress in prompting language models.
Recent work has shown the surprising power of continuous prompts to language models (LMs) for controlled generation and for solving a wide range of tasks (Li and Liang, 2021; Lester et al., 2021; Min et al., 2022). Despite these successes, the resulting continuous prompts are not easy to interpret (Shin et al., 2020). Is it possible to come up with meaningful discrete (textual) interpretations of continuous prompts, especially ones that provide a faithful explanation of the prompt's behavior?
Towards addressing this question, we propose and investigate the Prompt Waywardness hypothesis (§3.2), a surprising disconnect between the
Figure 1: We show that one can find accurate continuous prompts (that do well on a given task, e.g., sentiment classification here) such that they can be projected to any arbitrary text, such as the definition of a different task (e.g., generating a question) or even an irrelevant statement (e.g., a piece of code) — suggesting a disconnect between the outcome of continuous prompts and their discrete interpretations.
intended behavior of continuous prompts and their nearest-neighbor discrete (language) representations. In particular, we show that one can find continuous prompts that perform a desired task while, at the same time, project to any given target text. This indicates that there is little correspondence between continuous prompts and their discrete interpretation. For instance, a continuous prompt that effectively solves the sentiment classification task in Fig.1, when projected onto discrete space, might appear as the definition of a different task ("flip the sentiment"). Intuitively, continuous prompt optimization is a highly non-convex problem with numerous local minima. More surprisingly, many of these prompts (e.g., those near embedded text) are very effective at solving a desired task.
We conduct extensive analyses showing Waywardness on five classification datasets (§4). Empirically, we find the existence of wayward prompts — prompts that solve each of these tasks while pro
<sup>1</sup>Nearest-neighbor projection via dot product has been previously used to study properties of continuous word embeddings (Mikolov et al., 2013; Hashimoto et al., 2016) and is commonly performed in the final layer of modern generative LMs (Radford et al., 2019; Raffel et al., 2020).
Human Language Technologies, pages 3631 - 3643
jecting to arbitrary natural language text. To study a variety of projected text, we experiment with  $60+$  sentences, either discrete prompts from other tasks (from Mishra et al. 2022b) or random sentences from a large text corpus. We observe that it is possible to find prompts that project to a given discrete prompt (token overlap  $94\%$  F1) while scoring within  $2\%$  accuracy of the best continuous prompts-based solution for the task. Further analysis shows that the effect of Waywardness gets worse for larger models and longer prompts. We explain this surprising behavior by relating it to several structural properties of large language models ( $\S 5$ ).
We discuss several social and research implications of prompt waywardness, to help guide future research on prompt based models (§6). First and foremost, despite many promising attributes of continuous prompts, interpreting them is non-trivial and will require further research. In fact, careless interpretation of continuous prompts can result in vulnerabilities against malicious attacks concealed under the guise of benign discrete representation. Further, the loose correspondence between continuous and discrete prompts poses a challenge for future research in differentiable interpretable-prompt optimization - optimization in search of human readable discrete prompts through the continuous space. Our work shows that continuous and discrete prompts, despite their seeming similarity, are quite different and the results from one may not always transfer to the other. We hope these findings will motivate further innovations in the prompting literature for NLP models.
Continuous prompts. There is a line of work focused on tuning continuous prompts (Li and Liang, 2021; Lester et al., 2021; Zhong et al., 2021; Qin and Eisner, 2021; Zhou et al., 2021; Zhong et al., 2021). These works present different approaches to discovering a continuous prompt (which is an array of real numbers) for addressing an end task, though the interpretability of the resulting prompts remains an open question. This paper investigates the feasibility of interpreting a learned continuous prompt and its connection to discrete prompts.
Discrete prompts. The release of GPT-3 (Brown et al., 2020) initiated a body of work on the emergent ability of LMs to follow discrete natural language prompts. Consequently, several follow-up studies have used manually-designed discrete
prompts for probing LMs (Petroni et al., 2019; Jiang et al., 2020), improving LMs' few-shot ability (Schick and Schütze, 2021; Gao et al., 2021; Le Scao and Rush, 2021), and their zero-shot ability as well as transferability (Mishra et al., 2022a; Reynolds and McDonell, 2021). Most importantly, discrete prompts have the advantages of being human-readable and thus easily interpretable though we do not have efficient and algorithmic ways of reconstructing them. For example, Shin et al. (2020)'s algorithm discovers discrete prompts, yet the results are not human readable. Prior work also finds that model performance is highly sensitive to small changes in wordings (Mishra et al., 2022a) and that optimization over the discrete prompt space is non-trivial and often highly unstable. Our findings here about the disconnect between continuous prompts and their discrete interpretation provides another perspective on the difficulty of discovering discrete prompts via continuous optimization algorithms that (directly or indirectly) leverage the continuous space (more discussion in §6).
3 Prompt Waywardness
3.1 Preliminaries: Setup and Terminology
We begin with some notation and the setup of our study, starting with the space of discrete and continuous prompts (Fig.2). Let  $p_d \in \{0,1\}^{L \times V}$  denote a discrete prompt represented as an  $L$ -length sequence of one-hot vectors over a lexicon of size  $V$  (corners of a hyper-cube). Similarly, let  $p_c \in \mathbb{R}^{L \times d}$  denote a continuous prompt, represented as a  $L$ -length sequence of  $d$ -dimensional real vectors.
Projection operators. We define operators that project these two spaces to one another. Define the  $c$ -projection as one that maps discrete inputs to a continuous space by multiplying with a fixed (often pre-trained) embedding matrix  $E \in \mathbb{R}^{V \times d}$ :
$$
c - \operatorname {p r o j} \left(p _ {d}\right) = p _ {d} E \in \mathbb {R} ^ {L \times d}. \tag {1}
$$
The  $d$ -projection maps the continuous inputs to nearest neighbor discrete elements, where for each position  $l$  ( $1 \leq l \leq L$ ), one of the possible (and perhaps most straightforward) methods for interpreting a continuous prompt is defined as a projection
In our experiments we use the embedding matrix of the GPT2 family (Radford et al., 2019) which is used for both mapping input text to their embeddings as well as generating text in the output layer.
3632 2
Figure 2: The general problem setup: Similar to Lester et al. (2021)'s setup, each prompt (usually a continuous one) is appended to the given input and fed to a frozen language model.
onto nearest neighbor representations (Mikolov et al., 2013; Hashimoto et al., 2016):
$$
d \text {p r o j} \left(p _ {c}\right) = \left[ \delta_ {1} \dots \delta_ {l} \dots \delta_ {L} \right] \in \{0, 1 \} ^ {L \times V}, \tag {2}
$$
where  $\delta_l$  is a one-hot vector corresponding to the word with the closest (highest dot product) embedding to the  $l$ -th position of continuous prompt  $p_c$ .
These projections are used in the first and last layer of virtually all modern LMs, such as GPT2.
Solving tasks with continuous prompts. Consider any machine learning model  $M$  (typically a pre-trained model) that takes textual input  $x$  and produces output  $y$ . Normally, the parameters of  $M$  are learned so as to optimize behavior on a task with a dataset  $\mathcal{D} = \{(x,y)\}$  of input/output pairs. In prompt tuning (Lester et al., 2021), one freezes the parameters of  $M$  and instead optimizes for a prompt  $p$  that, when fed in conjunction with  $x$ , makes  $M$  produce the desired output  $y$ . Thus,  $p$  represents the only learnable parameters in this method. When  $p$  is a discrete prompt with  $k$  tokens, it can be simply concatenated with  $x$ , denoted  $p + x$ . In our study,  $p$  will be a continuous prompt (of length equal to the embedding of  $k$  tokens). We will concatenate it with the embedding of the input  $x$ . For simplicity and with some abuse of notation, we use  $p + x$  to denote concatenation in this continuous case as well.
One can quantify the amount of loss incurred when using a continuous prompt  $p$  as follows:
$$
\ell (p; \mathcal {D}) = \mathbb {E} _ {x, y \sim \mathcal {D}} \left[ \operatorname {l o s s} (M (p + x), y) \right], \tag {3}
$$
Minimizing this loss function (empirical risk minimization) over  $p$  recovers a minimum risk continu
ous prompt for this dataset:
$$
p _ {c} ^ {*} = \underset {p _ {c} \in \mathbb {R} ^ {L \times d}} {\arg \min } \ell \left(p _ {c}; \mathcal {D} ^ {\text {t r a i n}}\right). \tag {4}
$$
Given this prompt, its generalization to the test data can be measured in terms of the loss incurred on the test set:  $\ell(p_c^*; \mathcal{D}^{\mathrm{test}})$ .
3.2 The Waywardness Hypothesis
How should one interpret the resultant continuous prompt  $\tilde{p}_c$ ? Empirically, one can easily verify that such continuous prompts are not unique (e.g., random initializations lead to different outcomes). Additionally, the resultant prompts get projected to seemingly irrelevant discrete elements. Taking this to an extreme, we hypothesize that next to the continuous projection  $c\text{-proj}(p_d)$  of any discrete prompt  $p_d$ , there exists a variety of continuous prompts  $p_c$  that trigger responses from model  $M$  that are orthogonal to the intentions described by the discrete prompt  $p_d$ . We formalize this idea as the following hypothesis, where  $L \in \mathbb{N}$  is the length of the discrete target prompt,  $M$  is a prompt-based model, and  $\mathcal{D}$  is a dataset for a desired task:
Hypothesis 1 (Prompt Waywardness) For all  $L,M,\mathcal{D}$  there is a small  $\Delta$  such that for any discrete target prompt  $p_d$  with length  $L$  ,there exists a continuous prompt  $\tilde{p}_c\in \mathbb{R}^{L\times d}$  such that: 1.  $\left|\ell (\tilde{p}_c;\mathcal{D}^{test}) - \ell (p_c^*,\mathcal{D}^{test})\right| <   \Delta$  yet 2.  $d$  -proj  $(\tilde{p}_c) = p_d$
In other words,  $\tilde{p}_c$  is nearly as effective at making  $M$  solve the task as the optimal continuous prompt (Eq.4), and yet it projects to  $p_d$ . In this statement,  $\Delta$  (prompt performance gap relative to the optimal prompt  $p_c^*$ ) is a function of the prompt length  $L$ , the model  $M$  (e.g., its embedding size and depth when  $M$  is transformer based), and inherent properties of the target dataset  $\mathcal{D}$ . The analysis in §4.3 will provide an empirical estimate of this gap  $\hat{\Delta}$  as a function of various parameters like model size and prompt length.
It is worth emphasizing that the hypothesis is stated for any task and any set of discrete prompts, even if they are irrelevant or contradictory.3
3While our focus is on the use of continuous prompts for solving datasets (one prompt shared among many instances), one can imagine applications of the same conjecture to special use cases such as controlled generation (Dathathri et al., 2019) with one prompt per instance.
3633 3
3.3 Finding Wayward Prompts
While the above hypothesis promises the existence of  $\tilde{p}_c$ , it does not say how to discover them. We now discuss a practical method for their discovery.
We learn a continuous prompt  $p_c$  using a modification of the prompt tuning objective of Lester et al. (2021). Our modification jointly minimizes the standard downstream task cross-entropy loss  $\ell(.)$  for the task (Eq.3) and a distance measure  $\mathrm{dist}(.)$  between  $p_c$  and the discrete target prompt  $p_d \in \{0, 1\}^{L \times V}$ :
$$
\begin{array}{l} \ell^ {\prime} \left(p _ {c}; \mathcal {D}, \gamma\right) = \ell \left(p _ {c}; \mathcal {D}\right) + \gamma \operatorname {d i s t} \left(p _ {c}, p _ {d}\right) (5) \\ \tilde {p} _ {c} = \underset {p _ {c} \in \mathbb {R} ^ {L \times d}} {\arg \min } \ell^ {\prime} (p _ {c}; \mathcal {D}, \gamma), (6) \\ \end{array}
$$
where  $p_c$  is the only learnable parameter, and  $\gamma$  is a hyperparameter.
When  $\gamma = 0$ , the modified objective is reduced to the standard objective (Eq.4),  $\ell^{\prime}(.) = \ell(.)$ . We refer to this case and its resulting prompt  $p_c^*$  as the 'unconstrained' setting. A large value of  $\gamma$  will make  $p_c$  even closer (possibly identical) to  $c\mathrm{proj}(p_d)$  but lead to poor accuracy on a target dataset. Most of the experiments below are conducted via a range of  $\gamma$  values to better understand the trade off between the two terms in the objective function. In practice, we find  $\gamma = 0.01$  to give a reasonable trade-off regardless of the target dataset and the choice of  $p_d$ .
There are at least two natural ways to define the distance measure  $\mathrm{dist}(p_c, p_d)$  between a continuous prompt  $p_c$  and a discrete target prompt  $p_d$ , by converting one so that both are in the same space:
$$
c - \operatorname {d i s t} \left(p _ {c}, p _ {d}\right) = \frac {\left\| p _ {c} - c - \operatorname {p r o j} \left(p _ {d}\right) \right\| _ {2} ^ {2}}{L} \tag {7}
$$
$$
d - \operatorname {d i s t} \left(p _ {c}, p _ {d}\right) = \mathrm {F} 1 \left(d - \operatorname {p r o j} \left(p _ {c}\right), p _ {d}\right) \tag {8}
$$
The first of these places both  $p_c$  and  $p_d$  in the continuous space and computes the squared- $L^2$  norm, normalized by the prompt length. This is used in our training loss (Eq.5) implementation. The second places both in discrete space (text) and computes the standard word-level token overlap F1 score. $^4$  This is used during our evaluation.
4 Empirical Support of Waywardness
We empirically investigate the Prompt Waywardness hypothesis (§3.2) using our modification
4 Ignoring punctuation marks and articles, and applying lemmatization.
$(\S 3.3)$  of the prompt tuning method from Lester et al. (2021). We show that given an arbitrary and irrelevant discrete prompt  $p_d$ , it is possible to learn a continuous prompt that is mapped to  $p_d$  while retaining its accuracy on a given dataset.
Target tasks. Following the setup of Min et al. (2022), we select a diverse set of 5 classification datasets: SST-2 (Socher et al., 2013), SST-5 (Socher et al., 2013), AGNews (Zhang et al., 2015), Subj (Pang and Lee, 2004) and TREC (Voorhees and Tice, 2000). Statistics and the unconstrained accuracy of each dataset are provided in Table 1.
Dataset	Task	|C|	Acc
SST-2	Sentiment analysis (movie)	2	92.4
SST-5	Sentiment analysis (movie)	5	50.3
AGNews	News classification (topic)	4	88.1
Subj	Subjectivity classification	2	90.5
TREC	Answer type classification	6	88.0
Table 1: The collection of downstream tasks used in the experiments (§4.1).  $|C|$  indicates the output size (number of classes); Acc indicates the unconstrained accuracy of a prompt tuning method (Lester et al., 2021) using GPT2 Large, as a reference point.
Discrete Target Projections. We compile two sets of discrete target prompts: (1) 32 target prompts for solving tasks from Natural-Instructions $^6$  dataset (Mishra et al., 2022b) that are distinct from and intentionally orthogonal to the end tasks considered here. These were chosen by excluding discrete target prompts that have high lexical overlap with other discrete prompts; this is because we found lexically similar prompts are often semantically similar even when written for different subtasks. (2) 30 random sentences from PILE, $^7$  a large-scale, diverse text corpus used to pretrain GPT-J, the largest public causal language model (Wang and Komatsuzaki, 2021). The sampled sentences were drawn from a Poisson distribution with  $\lambda = 14$ , which makes the average length of the sentence to be consistent to those in Natural-Instructions. These sentences are selected to have little or no token overlap with the true definition of the target tasks. See Table 3 for a few examples.
<sup>5</sup>Scripts needed to reproduce our results: https://github.com/Alropel23/prompt-waywardness
<sup>6</sup>https://instructions.apps.allenai.org
<sup>7</sup>https://pile.eleuther.ai
Data	pdSource	Task Accuracy (%)	Prompt F1 (%)
\(\hat{\Delta} (Acc(p_c^*) \rightarrow Acc(\tilde{p}_c))\)
SST-2	NI	0.7	(92.4 → 91.8)	99.0
PILE	0.5	(92.5 → 92.0)	97.1
Avg	0.6	(92.4 → 91.9)	98.1
SST-5	NI	3.3	(50.2 → 48.5)	95.9
PILE	0.7	(50.5 → 50.2)	92.4
Avg	2.0	(50.3 → 49.3)	94.2
AGNews	NI	1.6	(88.0 → 86.6)	97.4
PILE	-0.1	(88.1 → 88.2)	97.5
Avg	0.8	(88.1 → 87.3)	97.4
Subj	NI	2.0	(91.3 → 89.5)	97.3
PILE	0.9	(89.6 → 88.8)	94.4
Avg	1.5	(90.5 → 89.2)	95.9
TREC	NI	3.3	(87.5 → 84.7)	86.5
PILE	1.2	(88.5 → 87.5)	85.6
Avg	2.3	(88.0 → 86.0)	86.1
Table 2: Main Results: Accuracy of solving five classification datasets, in an unconstrained setting  $(p_c^*)$  vs. constrained by the projection to various irrelevant pieces of text  $(\tilde{p}_c)$ . Optimization is done using  $\gamma = 0.01$  in the objective function (Eq.5).  $\hat{\Delta}$  indicates the relative accuracy drop (in %) from unconstrained accuracy. Each reported score (Accuracy and Prompt F1) are the average over 62 discrete target prompts and 3 random seeds. Overall, it is possible to achieve  $\geq 94\%$  prompt F1 with under  $2\%$  drop in accuracy.
Evaluation metrics. For all experiments, we report two metrics: (1) the task accuracy<sup>8</sup> as well as (2) prompt F1, the word-level token overlap F1 score computed as in Eq.8, since it easy to interpret and is commonly used for evaluating the textual output of models (Rajpurkar et al., 2016).
Models. For evaluation, we use GPT2 (Radford et al., 2019) an auto-regressive LM which has extensively been used in many NLP applications. Unless otherwise specified, we use a 'large' variant consisting of 774M parameters.
Implementation details. We use a batch size of 8, learning rate 0.01, and 2000 training steps. When experimenting with a discrete target prompt  $p_d$ , we initialize the search for continuous prompts (both  $\tilde{p}_c$  and  $p_c^*$ ) using  $c\text{-proj}(p_d)$ . For all experiments, report accuracy averaged over three random seeds.
For each of the 5 tasks  $T$  and for each of the 62 discrete target prompts  $p_d$ , we use the objective in Eq.5 to find a prompt  $\tilde{p}_c$  such that it solves  $T$  with a high accuracy while, at the same time, having a discrete projection that is close to  $p_d$ . For comparison, we also train unconstrained prompts  $p_c^* (\gamma = 0.0)$  which solve task  $T$  without any projection constraint. To ensure a fair comparison between  $\tilde{p}_c$  and  $p_c^*$ , we ensure that they have the same size  $L$ . In other words, for each  $\tilde{p}_c$  (that has the same length as  $p_d$ ), we train another  $p_c^*$  with the same length. We denote the relative accuracy drop from  $p_c^*$  to  $\tilde{p}_c$  as  $\hat{\Delta}$ .
Table 2 summarizes the results. Across all datasets, we find that it is possible to learn a continuous prompt  $p_c$  whose discrete projection is very close to  $p_d$  and mostly retains the task accuracy. There is a trade-off between the task accuracy and prompt F1, which can be controlled by the choice of  $\gamma$  (more extensive ablations in the forthcoming paragraphs (§4.3)). Overall, with  $\gamma = 0.01$ , it is possible to achieve  $\geq 94\%$  prompt F1 with under  $2\%$  relative drop in task accuracy. The only outlier is the TREC dataset where we achieved a prompt F1 score of  $86\%$  for a  $\hat{\Delta} = 2.3\%$  relative drop in accuracy. This might be due to the difficulty of learning effective prompts on TREC (also discussed by Min et al. (2022)).
Example prompts with varying values of prompt F1 scores are shown in Table 3. A prompt F1  $\geq$ $94\%$  generally indicates one word mismatch with almost no semantically meaningful difference.
Effect of Gamma. Fig. 3 shows the trade-off between task accuracy and the prompt F1 when varying  $\gamma$  from 0 to 0.03. As  $\gamma$  increases, the task accuracy goes down while the prompt F1 increases. The drop in task accuracy is relatively minor—it is possible to learn a continuous prompt for which prompt F1 is near 1.00 and the accuracy drop relative to the unconstrained accuracy is less than  $1\%$ .
Effect of Prompt Length  $(L)$ . We randomly sample sentences from The PILE with a constraint that its length must be  $L$  (chosen from  $\{4,7,14,28,56\}$ ). The left and the middle parts of Fig. 4 illustrate the results. We find that when  $L$  is very small (e.g., 4) it is relatively difficult to learn a continuous prompt  $p_c$  that is close to  $p_d$  (F1<60%) while retaining the task accuracy. This is likely be
<sup>8</sup>We did not consider alternatives like Macro-F1 because all datasets are roughly balanced across different classes. <sup>9</sup>While this is different from prior work (Lester et al., 2021; Min et al., 2022) that uses a random subset of the top-5000 vocabs, we find no meaningful differences in an unconstrained accuracy between two initialization methods.
3635 5