SlowGuess/NAACL_2022_OCR huggingface dataset:
d-proj(pc)	Prompt F1	Acc(¯pc)
Task: AGNews	pd: Write down the conclusion you can reach by combining the given Fact 1 and Fact 2.
Write down the conclusion you can reach by combining the given Fact 1 and Fact 2.	100.0	89.2
Write down the conclusion you can reach by combining the given Fact 1. Fact 2.	96.3	88.1
Write down the conclusion you can reach by combining the given Fact 1 Category Fact 2.	92.9	89.0
Write Messi in conclusion you can reach by combining the given Fact 1 and Fact 2.	89.7	88.8
Task: SST-5	pd: “If they have other interests and aims in life it does not necessarily follow that they are passive sheep.”
“If they have other interests and aims in life it does not necessarily follow that they are passive sheep.”	100.0	51.2
“If they have other interests and aims in life it does not necessarily follow that they are terrible sheep.”	94.7	53.6
“If they have other interests and aims in life it does not necessarily follow that they are terrible GoPro.”	89.5	52.3
Task: SST-5	pd: int clamp(int val, int min_val, int max_val) { return std::max(min_val, std::min(max_val, val));}
int clamp(int val, int min_val, int max_val) { return std::max(min_val, std::min(max_val, val));}	100.0	50.5
int clamp(int val, int min_val, int max_val) { return std::max(min_val, std::min(max_val terrible val));}	95.7	52.0
int clamp(int val, int min_val, int max_val) { return std::max(min_val, std::min(max_val terrible val)); This}	91.7	53.3
Figure 3: The effect of  $\gamma$  on SST-2 and AGNews. Accuracy is the average over 32 discrete target prompts from Natural Instructions and 3 random seeds. A dotted line indicates unconstrained accuracy  $p_c^*$  (same as when  $\gamma = 0$ ). Numbers inside parentheses in the y-axis indicate relative drop in accuracy against unconstrained accuracy. There is a clear trade-off between the task accuracy and the prompt F1.
cause the prompt being too short significantly hurts the expressivity of the prompt. Nonetheless, when  $L$  is reasonably larger, e.g., 14 (the average length of in Natural Instructions) or longer, all cases lead to a continuous prompt with near 1.0 prompt F1 and little accuracy drop.
Effect of Model Size. We vary the size of the GPT2 models—small, medium, large, and XL—with 124M, 355M, 774M, and 1.5B parameters, respectively. Figure 5 (right) reports the result on SST-2. We find that (1) across different sizes of the LM, our findings in learning continuous prompts with the prompt F1 of near 1.0 and little drop in the accuracy generally hold, and (2) in particular, the drop in accuracy is more negligible with larger LMs (0.2% with XL, 0.5–0.7% with medium and large, 1.2% with small).
Projection onto true task definitions. In all our results so far in §4, the target projected text was
Table 3: Examples of the target prompts  $p_d$  and their reconstructions via  $d\text{-proj}(p_c)$  for different ranges of prompt F1 scores. The first  $p_d$  is from Natural-Instructions; the rest two are sampled from The PILE. The mismatches with the original prompt are color-coded.
Data	Task Accuracy (%)	Prompt F1 (%)
ΔT (Acc(p*) → Acc(˜p))
SST-2	1.0	(91.9→90.9)	98.5
SST-5	0.9	(51.4→50.5)	96.1
AGNews	1.4	(91.8→90.4)	95.7
Subj	4.1	(89.8→85.6)	100.0
TREC	0.5	(88.6→88.1)	99.3
Table 4: Accuracy of solving five classification datasets, unconstrained setting  $(p_c^*)$  vs. constrained by the projection to the true definition of tasks  $(\tilde{p}_c)$  using  $\gamma = 0.01$  in the objective function (Eq.5). Subscript  $T$  in  $\Delta_T$  denotes this being the case for true task definitions. Projecting to the true definition of a task does not help continuous prompts solve a task.
orthogonal to the tasks being solved. One might naturally wonder whether there is any benefit in projecting continuous prompts to the texts that truly describe the task being solved, i.e., a "true" prompt for the task. To this end, we manually authored 5
Figure 4: The effect of the length of the prompt  $(L)$  on AGNews. Each point computed via the average over 32 discrete target prompts from Natural Instructions and 3 random seeds ( $\gamma = 0.01$  used). The corresponding prompt F1 is reported as a orange line. The accuracy of  $p_c^*$  and  $\tilde{p}_c$  increase as a function of prompt length, however, the gap between them tends to decrease. The relative accuracy drop is marginal when  $L$  is not too small (e.g., 7 or larger).
"true" prompts for each of the tasks. We then follow the exact same setup used earlier for Table 2 to fine-tune continuous prompts  $\tilde{p}_c$  for the task while projecting onto these true task definitions. As before, we fine-tune unconstrained prompts  $p_c^*$  of the same length, without any projection requirement.
By design,  $\tilde{p}_c$  can be no more effective at solving the task than the unconstrained prompt  $p_c^*$  (barring suboptimal search issues), which is what we find in practice. For completeness, we report detailed results for "true" target prompts (analogous to Table 2) in Table 4.
More interestingly, as shown in Table 5, continuous prompts that project to "true" target prompts are no more effective at solving the task than continuous prompts that project to the 62 irrelevant target prompts considered earlier (Table 2). Specifically, the average performance gap  $\Delta$  (relative to unconstrained prompts of the same length) is about the same ( $\approx 1.5\%$ ) for continuous prompts that map to true task definitions compared to prompts that map to irrelevant text. This further bolsters the waywardness hypothesis—continuous prompts don't relate to the task being solved.
Figure 5: The effect of the size of the model—small, medium, large, and XL—on SST-2. Each point in the experiment is computed by averaging over 30 experiments each with a different discrete target prompt from PILE and 3 random seeds. We vary  $\gamma = \{0.01, 0.005, 0.003\}$  and choose the one for which prompt F1 is larger than 0.98. The relative accuracy drop (gap between the two trends) decreases as models become larger.
SST-2	SST-5	AGNews	Subj	TREC	Avg
ΔT	1.0	0.9	1.4	4.1	0.5	1.6
Δ	0.6	2.0	0.8	1.5	2.3	1.4
Table 5: Task accuracy gap comparison between unconstrained prompts and those fine-tuned to project to a true task definition  $(\hat{\Delta}_T)$  as reported in Table 4. For comparison, we also show the corresponding performance gaps with irrelevant  $(\hat{\Delta})$  from Table 2. The average performance gaps are about the same (around 1.5) for true and irrelevant target prompts—further evidence that continuous prompts don't relate to the task being solved.
5 Explaining Waywardness
Here we provide intuitions behind the factors that enable Prompt Waywardness.
The mapping between continuous and discrete spaces is not one-to-one. While a discrete target prompt is mapped to exactly one continuous prompt (via its embedding, Eq.1; cf. Fig.2), the reverse is not true. In fact, except for a very small fraction of unnatural or degenerate edge cases, $^{10}$  for every target discrete prompt, there are infinitely many continuous prompts that project back to it (via Eq.2). While simple counting-based arguments are insufficient in continuous spaces, we formally prove (Appendix B) that this property holds for all nearest-neighbor projections under any metric distance, and broadly for all but a negligible (measure zero) portion of possible projection operators.
10 Such as using a non-metric distance in nearest-neighbor mapping, or mapping all of  $\mathbb{R}^d$  to a single discrete prompt.
3637 7
subspace of continuous prompts that have the same projections but solve different tasks.
Figure 6: The projection discrete space (Eq.2) induces a clustering (a Voronoi diagram) of the continuous space. Each cluster has infinitely many points that get mapped to the same discrete token.
This intuitively suggests that there is a whole region of continuous prompts that corresponds to a fixed discrete representation (Fig.6). The remaining question is, how is this region able to have a diverse set of prompts that can solve a variety of tasks? This is addressed next.
Deep models give immense expressive power to earlier layers. The deeper a network is, the more expressivity it has with respect to its inputs (Telgarsky, 2016; Raghu et al., 2017). Since continuous prompts reside just before the first layer, they enjoy a lot of expressivity. Therefore, no matter how narrow the regions corresponding to individual tokens are (Fig.6), they are extremely powerful in solving a variety of tasks. Previously in §4.2 we provide an empirical analysis showing evidence that the effect of Waywardness is stronger in deeper models.
6 Implications of Prompt Waywardness
We discuss the implications of these findings on several inter-related lines of research. Note that all the following statements are valid within the boundaries of the existing architectures. Moving beyond these barriers likely requires major innovations in terms of LM architectures or how continuous prompts are optimized.
Faithful interpretation of continuous prompts is difficult. Given the intuitions behind and empirical support for the Waywardness hypothesis (§5), faithful discrete interpretations of continuous prompts via common discrete projections (like nearest-neighbor projection) are unlikely to be robust based on current approaches. It is an open question whether there is a better way of interpreting continuous prompts with human language,
or whether explaining and interpreting continuous prompts via human language is inherently impossible because they lie in completely different spaces. Future work may investigate more on this topic in order to improve the interpretability of prompt-based language models.
Risk of interpreting continuous prompts: concealed adversarial attacks. It is not difficult to imagine a future where proprietary model development is driven by fine-tuned continuous prompts. In such a world, not addressing the challenges involved in discrete interpretation of continuous prompts can lead to harmful (and potentially, adversarial) consequences (Slack et al., 2020; Wallace et al., 2021), as discussed below.
Figure 7: Waywardness implies that continuous prompts can be mapped to seemingly innocuous descriptions while acting maliciously.
We consider the following scenario: a model designer comes up with a set of continuous prompts that solve a target task (e.g., ranking resumes according to each applicant's qualifications and merits). Whether intentionally or not, such prompts may maliciously target, for example, a minority group. To assure their customers, the model designer uses the projection of the prompt that expresses a benign definition for the task, which does not reveal the true nature of the egregious behavior. The customers might even evaluate the prompt on a few instances but not notice this harmful behavior, e.g., when it effects a minority group not in the evaluation set. In a way, the benign discrete projections may provide a false sense of security.
Optimizing discrete prompts through continuous prompts can be degenerate. Manually-written discrete prompts have many nice properties (Schick and Schütze, 2021; Mishra et al., 2022a), yet we do not have an efficient algorithmic way of finding them. One way to operationalize this is to formulate differentiable objective functions via LMs like GPT (Radford et al., 2019). Consider the following problem which is defined in the space
of continuous embeddings  $p_c \in \mathbb{R}^d$ :
$$
\max  _ {p _ {c} \in \mathbb {R} ^ {d}} \overbrace {\mathbb {P} (D | p _ {c})} ^ {\text {u t i l i t y}} \times \overbrace {\mathbb {P} (d - \operatorname {p r o j} (p _ {c}))} ^ {\text {r e a d a b i l i t y}}, \tag {9}
$$
This a joint optimization towards a utility objective (the extent to which it can solve dataset  $\mathcal{D}$ ) and a human readability objective. According to the Waywardness hypothesis, there are  $p_c$ 's that assign high mass to the utility term while also mapping to human interpretable text that is irrelevant (or even contradictory) to the task solved by the prompt - hence, degenerate solutions.
The same challenge holds if this optimization objective, instead of continuous prompts, is reformulated in terms of word probabilities (e.g., similar to Kumar et al. (2021, Sec 2.2)). This is the case, since searching in the space of word probabilities is analogous to a search in embedding spaces.[11]
In summary, Waywardness presents a challenge for searching effective discrete prompts via continuous optimization. The recent works have used additional signals such as domain-specific constraints (Qin et al., 2020; Khot et al., 2021; Qin et al., 2022) to alleviate these challenges. We hope to see more design innovations in this direction.
Gradients alone are insufficient to reverse engineer a model. Suppose we are given a fixed (fine-tuned or otherwise) model  $M$  (e.g., an open question-answering model) and an expected output  $y$  from this model (e.g.,  $y =$  "Joe Biden"). Can we use gradients with respect to an LM's output to find a semantically meaningful input that makes a frozen model  $M$  generate a particular output?
Our findings and the earlier argument about continuous differentiable optimization suggests this may not be feasible with current methods. To see the correspondence to Prompt Waywardness, we can replace  $\mathcal{D}$  in Eq.9 with the desired outcome  $y$  and run the optimization over word distributions (cf. Footnote 11). While gradients can guide towards some input that makes  $M$  produce  $y$ , their interpretation is likely unfaithful to the task being solved by  $M$ . In the context of the above example ( $M$  being a QA system), gradients might lead to inputs maximize the probability assigned to "Joe Biden", although this input will likely be neither fluent nor semantically descriptive of "Joe Biden".
Nevertheless, as noted earlier, gradients are still useful when they are applied using domain-specific constraints. For example, one can find local (word-level) perturbations that lead to a certain adversarial outcome, if the perturbations are restricted to well-defined semantic categories (e.g., "blue" can be perturbed to any other color name) (Sha, 2020; Guo et al., 2021; Yuan et al., 2021).
Continuous prompt tuning does not necessitate task-specific initialization. Recent works on continuous prompt-tuning have shown the effectiveness of initialization from embeddings of random common words (Lester et al., 2021; Min et al., 2022), despite these words being irrelevant to the task solved by these prompts. This, however, makes sense given the observations made in this work regarding the existence of effective prompts around word embeddings.
The prompting literature has seen many parallel developments around continuous and discrete prompts, as efficient alternatives to fine-tuning models with tens of millions of parameters. Our work introduced the Prompt Waywardness hypothesis, which expresses a surprising disconnect between continuous and discrete prompts: given a downstream task, for any discrete target prompt  $p_d$ , there exists a continuous prompt that projects to  $p_d$  while achieving strong performance on the task. We provided empirical evidence for this hypothesis, studied various parameters around it, and ended with several implications of this hypothesis.
While our experiments are done on the GPT family, we expect our findings to apply to a broader set of architectures that, in one way or another, use similar mechanisms for mapping discrete elements to continuous representations and vice versa. Similarly, while our projection to the discrete space (Eq.2) is a popular operator in the field (cf. Footnote 1), the intuition explained in Propositions 1 and 2 of the Appendix suggests similar behavior for a broad class of projection operators.
Prompt Waywardness identifies challenges for future progress on algorithmic methods for the discovery of human readable prompts that are faithful to the task they solve. We hope the observations made in this work motivate architectural innovations that overcome such challenges and guide future steps in the prompting literature.
11 A distribution over words  $p \in [0, 1]^V$  corresponds to a continuous prompt  $p_c = c \cdot \operatorname{proj}(p)$  which is a weighted combination of  $V$ -many basis vectors (word embeddings) that form a linear span of  $\mathbb{R}^d$ .
3639 9
The authors are thankful to Lisa Li, Nicholas Lourie, and Vered Shwartz for helpful discussions, and the Beaker team at AI2 for their support with the experiments. This work was supported in part by DARPA MCS program through NIWC Pacific (N66001-19-2-4031), DARPA SemaFor program, and Google Cloud Compute.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019. Plug and play language models: A simple approach to controlled text generation. In Proceedings of ICLR.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of ACL.
Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. 2021. Gradient-based adversarial attacks against text transformers. In Proceedings of EMNLP.
Tatsunori B Hashimoto, David Alvarez-Melis, and Tommi S Jaakkola. 2016. Word embeddings as metric recovery in semantic spaces. TACL, 4:273-286.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? TACL, 8:423-438.
Tushar Khot, Daniel Khashabi, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2021. Text Modular Networks: Learning to decompose tasks in the language of existing models. Proceedings of NAACL, page 1264-1279.
Sachin Kumar, Eric Malmi, Aliaksei Severyn, and Yulia Tsvetkov. 2021. Controlled text generation as continuous optimization with multiple constraints. Proceedings of NeurIPS, 34.
Tven Le Scao and Alexander M Rush. 2021. How many data points is a prompt worth? In Proceedings of NAACL, pages 2627-2636.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of EMNLP.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of ACL.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of NeurIPS, pages 3111-3119.
Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Noisy channel language model prompting for few-shot text classification. In Proceedings of ACL.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. 2022a. Reframing instructional prompts to GPTk's language. In Proceedings of ACL - Findings.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022b. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of ACL.
Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of ACL.
Fabio Petroni, Tim Roktaschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of EMNLP.
Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying lms with mixtures of soft prompts. In Proceedings of NAACL.
Lianhui Qin, Vered Shwartz, Peter West, Chandra Bhagavatula, Jena D Hwang, Ronan Le Bras, Antoine Bosselut, and Yejin Choi. 2020. Backpropagation-based decoding for unsupervised counterfactual and abductive reasoning. In Proceedings of EMNLP.
Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. 2022. COLD decoding: Energy-based constrained text generation with Langevin dynamics. arXiv preprint arXiv:2202.11705.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21:1-67.
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. 2017. On the expressive power of deep neural networks. In Proceedings of ICML, pages 2847-2854.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP, pages 2383-2392.
3640 10
Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. In Proceedings of CHI.
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of EACL, pages 255-269.
Lei Sha. 2020. Gradient-guided unsupervised lexically constrained text generation. In Proceedings of EMNLP, pages 8692-8703.
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Eliciting knowledge from language models using automatically generated prompts. In Proceedings of EMNLP.
Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. 2020. Fooling lime and shap: Adversarial attacks on post hoc explanation methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 180-186.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP.
Matus Telgarsky. 2016. Benefits of depth in neural networks. In Proceedings of COLT, pages 1517-1539.
Ellen M Voorhees and Dawn M Tice. 2000. Building a question answering test collection. In Proceedings of SIGIR.
Eric Wallace, Tony Zhao, Shi Feng, and Sameer Singh. 2021. Concealed data poisoning attacks on NLP models. In Proceedings of NAACL.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax.
Lifan Yuan, Yichi Zhang, Yangyi Chen, and Wei Wei. 2021. Bridge the gap between CV and NLP! a gradient-based textual adversarial attack framework. arXiv preprint arXiv:2110.15317.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proceedings of NeurIPS.
Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing is [mask]: Learning vs. learning to recall. In Proceedings of NAACL, pages 5017-5033.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2021. Learning to prompt for vision-language models. arXiv preprint arXiv:2109.01134.
Here we include several experimental details (§4) that did not fit in the main text. For the experiments we used A100 GPUs with 40G memory. In terms of the time GPU time of the experiments, each round of training and inference for each seed took about around 6 min. Therefore, the total GPU hours for our main experiment (Table 2) adds up to 93 hours (6 mins × 3 seeds × 5 datasets × 62 prompts = 5580 mins).
B The mapping between continuous and discrete space is not one-to-one
As argued in §5, the mapping between the space of discrete input and that of word embeddings (Fig.2) is not a bijection. While a discrete target prompt is mapped to exactly one continuous prompt (via its embedding, Eq.1), the reverse is not true: except for some unnatural or rare cases (as formalized in the following propositions) there are infinitely many continuous prompts that project back to a fixed discrete target prompt (via Eq.2).