Unnatural language processing: How do language models handle machine-generated prompts?
Authors: Corentin Kervadec
Francesca Franzon
Marco Baroni
Venue: EMNLP 2023 Findings (2023)
Keywords: prompting
interpretability
large language modelling
unnatural language processing
TL;DR: We use machine-generated prompts to probe how language models respond to input that is not composed of natural language expressions.

Abstract: Language model prompt optimization research has shown that semantically and grammatically well-formed manually crafted prompts are routinely outperformed by automatically generated token sequences with no apparent meaning or syntactic structure, including sequences of vectors from a model's embedding space. We use machine-generated prompts to probe how models respond to input that is not composed of natural language expressions. We study the behavior of models of different sizes in multiple semantic tasks in response to both continuous and discrete machine-generated prompts, and compare it to the behavior in response to human-generated natural-language prompts. Even when producing a similar output, machine-generated and human prompts trigger different response patterns through the network processing pathways, including different perplexities, different attention and output entropy distributions, and different unit activation profiles. 
We provide preliminary insight into the nature of the units activated by different prompt types, suggesting that only natural language prompts recruit a genuinely linguistic circuit.

Unnatural language processing:
How do language models handle machine-generated prompts?
Corentin Kervadec and Francesca Franzon Universitat Pompeu Fabra (UPF) Barcelona {name. BaattammeJeupff edu
Marco Baroni
UPF and ICREA/ Barcelona
marco. baroni edu
Abstract
been widely observed that such prompts, while em- pirically effective, consist of nonsensical sequences of jumbled tokens. For example, using the pop- ular AutoPrompt algorithm Shin et al. (2020) and the OPT-1.3b language model (Zhang et al.. 2022), we found that the prompt "[X] Antarctica seequelsStationnoug" [Y]" outperforms reason- able human-crafted prompts such "/X] belongs to the continent of [Y]" the task of retrieving the continent geographic body belongs to. Even more extremely, recent prompt generation methods find sequences of embedding vectors that do not correspond to items in the model vocabulary, but still outperform both human-crafin crafted and machine- derived discrete prompts (Lester et al., 2021; Liu et al., 2023; Zhong et al., 2021). This state of af- fairs is paradoxical: why does a LM that has been trained reproduce the statistics of natural lan- guage respond better input sequences that are completely outside this distribution?
Language model prompt optimization research has shown that semantically and grammatically well-formed mamually crafted prompts are rou- tinely outperformed by automatically generated token sequences with no apparent meaning or syntactic structure, including sequences of vec- tors from a model's embedding space. We use -eeeerrte prompts to probe how mod- els respond input that is not composed of nat- ural language expressions. We study the behav- ior of models of different sizes in multiple se- mantic tasks response to both continuous and prompts, and com- pare to the behavior in response to human- generated natural- language prompts. Even when producing similar output, machine- generated and human prompts trigger different response patterns through the network process- ing pathways, including different perplexities, different attention and output entropy distribu- tions, and different unit activation profiles. We provide preliminary insight into the nature of the units activated by different prompt types, suggesting that only natural language prompts recruit genuinely linguistic circuit.
discrete machine-generae
We present a detailed comparative study of how LMs internally process manually-crafted prompts and both discrete and continuous machine- generated prompts. While we do not solve the puzzle of why linguistically ill-formed machine- generated prompts are better than human prompts, we discover that there are fairly deep differences characterizing the various prompt types through all the processing stages of a LM, suggesting that the latter has fortuitously developed a distinct pathway to process unnatural language.
1 Introduction
Neural language models (LMs) are parameterized probabilistic models that can assign probability to any sequence of language tokens. Given that they are trained on huge amounts of natural language. we expect their statistics to mimic those of the latter. In this paper, we study what happens when a LM trained on English must process "unnatural language", that is, sequences that are extremely unlikely in English, as they are syntactically and semantically ill-formed.
Related work
Understanding prompts. The advent of zero- shot prompting stimulated interest in the linguis- tic and semantic properties of prompts." For example. Webson and Pavlick (2022) showed
We tackle this topic through the lens of machine- generated prompts, that is, automatically discov- ered input token sequences that optimize the model's performance in a target zero-shot task (Shin et al., 2020; Deng et al., 2022). has indeed
'There is also related work on the effect of ablations such as word order permutations in the context of models fine-tuned for a specitic task, such as natural language inference (e.g., Gupta et al., 2021; Pham et al., 2021: 1; Sinha et al., 2021a.h).

---

that, with minimal fine-tuning, highly semanti- cally irrelevant prompts can be effective as prompts with pertinent semantic content. Start- ing with Wallace et al. (2019) and Shin et al. (2020), the fact that inscrutable machine-generate discrete prompts outperform natural language se- quences has also attracted attention. For exam- ple, Deng et al. (2022) showed that constraining machine-generated prompts to be more "language- like" harms performance. Ishibashi et al. (2023) and Rakotonirina et al. (2023) studied how vari- ous ablations affect the performance of machine- generated prompts. The second study also demon- strated that it is possible to find discrete machine- generated prompts that are effective across range of LMs. Khashabi et al. (2022) found that continu- prompts be optimized to be near any arbi- trary text in embedding space, while being equally effective. These studies focus on properties of the prompts themselves. We complement them with an analysis of how LMs respond when exposed to these prompts.
3 Setup
3.1 Language model and tasks
OPT family LMs We conduct our analyses on OPT-350m and OPT-1.3b (Zhang et al., 2022), two pre-trained auto-regressive transformer-based models trained on The Pile corpus (Gao al., 2020), whose pre-trained weights are publicly available from HuggingFace. We choose auto- regressive models since LM development has in- creasingly shifted to this class, and OPT models since, in informal experiments, we found them to perform better on our tasks than comparable regreseeressiv models available from Hugging- Face (e.g., the GTP2 family). OPT models use vocabulary set composed of 50,265 items.
Knowledge-retrieaal tasks We base our experi- ment on the LAMA dataset (Petroni et al., 2019). Initially designed to probe factual knowledge and commonsense in LM, this dataset is collection of (r, 8, o) triplets describing relation between subject s and object o, e.g.: (continent of. Lavoisier Island, Antarctica). In particular, we use the TREx (EISahar et al., 2018) subset, whose test set contains 41 relations, each with up to 1,000 tuples. All the machine-generated prompts are trained using the data collected by Shin et al. (2020), also containing 1,000 tuples per relation. Each relation defines different knowledge trieval task. We focused on these tasks because they require semantically contentful prompts (c.g.. for the relation above, the prompt must carry some geographic information), as opposed to other se- tups where a prompt might simply have to describe the task at linguisig level ("translate the following sentence into Chinese"; "does paragraph X entail paragraph Y?", etc.).
Understanding LMs More generally, under- standing how LMs process unnatural linguistic in- put contributes to our understanding of their inner workings. Therefore, our study is also related to work on interpretability (Lipton, 2018), defined as the analysis of trained model's decision policy. In particular, one can approach neural network inter- pretability by adopting a mechanistic paradigm, consisting in directly studying the weights and their activation in order to reverse-engineer the neu- ral network. Successful mechanistic insights have been obtained in computer vision (Voss et al., 2021; Olah et al., 2020). Cammarata et al. (2020) is an example of mechanistic interpretability applied to Tranformer LMs. In this context, the Transformer feed-forward layers have been shown behave like key-value memories (Geva al., 2021). No- tably, as shown Dai et al. (2022), these memory slots, also called knowledge newrons, encode spe- cific concepts acquired during pre-training. Even more interesting, manually editing these memo- ries allows to causally control the prediction output (Meng et al., 2022), suggesting that they play central role in language processing (see also Geva al., 2022). In the present paper, we show that unnatural language processing is achieved by re- cruiting different knowledge neurons than the ones used for natural language processing.
3.2 Prompts
Terminology We refer different methods derive prompts prompt types. We refer the actual token sequences generated by a method for a certain task as templates.
Human prompts Human prompts (human) come from augmented version of PARAREL (Elazar et al., 2021). PARAREL provides set of near- paraphrase templates capturing each LAMA rela- tion, e.g. "IX] belongs the continent of [Y]", ParaRel enlarged the initial templates provided by LAMA using paraphrases from LPAQA (Jiang et al., 2020) and additional patterns mined from

---

Wikipedia. Each prompt was then evaluated by set of human experts. We further manually aug- mented the set with more paraphrases, and we cleaned the prompt set, e.g., by removing templates not adapted to auto-regressive LMs.
entropy of the output probability vector computed across all samples of the evaluation set.
Attention distribution We quantify how atten- tion is distributed over input tokens following Ram- sauer al. (2021). For each attention head of each layer, compute the average minimal number of attention values required to get a cumulative soft- max probability mass of 0.90. This value ranges from 0% to 100%. Intuitively, given a row of an attention map of a transformer layer, corresponds to the number (in %) of attention values you have to sum to reach 90% of the total attention. Because attention values are normalized, if the attention is flat then the score will be 90%. In contrast, if all the attention is focused one token, then the score will be close 1. This score decreases as the attention distribution becomes more peaky.
Machine-generated prompts We compare hu- man prompts with both discrete (M-disc) and con- tinuous (M-cont) machine- generrttd prompts. The discrete ones are obtained using the popular Auto- prompt (Shin et al., 2020) algorithm. For a given task, this algorithm generates a sequence of N to- kens relying on a condient-guided search in the discrete LM's vocabulary space. We set template length to N 5, as it is the average human prompt length. The continuous machine generated prompts are obtained using Optiprompt (Zhong et al., 2021), For each task, Optiprompt generates sequence of continuous vectors through optimization in the LM's embedding space. Similarly to Autoprompt, we set N = 5. Machine-generated prompts are extracted using the LAMA-TREX training set (see above). 10 templates obtained for each task by initializing training with different random seeds.
Knowledge neuron activation overlap Moti- vated by Geva et al. (2021) and Dai et al. (2022), who empirically demonstrated that Transformer feed-forward (FF) layers act key-value memo- ries, or knowledge neurons, we measure the acti- vation overlap of the intermediate FF units (cor- responding to memory keys) between different prompt types. More formally, for given Trans- former layer, let æ E Id be the token-wise hidden representation contextualised by the self-attention operation. The FF layer can be expressed as:
Template filtering We only use tasks for which we have, for each prompt type, at least one template reaching > 10% accuracy. We end up with 5.9 hu- man, 8.3 M-disc, and 9.0 M-coní templates on av- erage per task (across 21 tusks) for OPT-350m, and 6.3 human, 8.9 M-disc, and 10 M-cont templates on average per task (across 24 tasks) for OPT-1.3b.2
== f(x KT + bk) FF by
(1)
3.3 Diagnostic metries
(2)
Accuracy We measure the effectiveness of a prompt type (human, M-disc or M-conf) by com- puting its micro- accuracy (following Zhong et al. (2021)), defined as the proportion of cases where the prompted LM succesfully assigned maximum completion probability to the ground-truie object. We average across templates and LAMA tasks. is worth noting that, contrary to other works, we did not perform any filtering on the LM's output.
where V,K Rdxdm are the FF parameters, bK by their respective biases, and ʃ(-) the non- linearity. can be seen as a set of dm keys, giving access to dm "memory slots" stored in V. In order to quantify which knowledge neurons are being accessed during prompt processing, we look at the units Rdm. corresponding to the weights associ- ated to each key-value pair (Eq. 1). A high overlap means that the prompts activate the same knowl- edge neurons, indicating similar processing by the LM. On the opposite, a low overlap suggests that the prompts trigger different activation pathways in the LM. The measure is described in more detail in Appendix A.
Input perplexity and output entropy We mea- sure the average perplexity for each prompt type, defined as the exponentiated average negative log- likelihood of a " "[subject] [template]" se- quence, averaged across subjects, relations and tem- plates. To characterize the LM probability distribu- tion output, we also measure the average Shannon
Input similarity For each pair of same-task tem- plates, we measure the cosine similarity of their embedded representations. We then compute the average to get similarities at the prompt-type level.
Wo attach the filtered template list as supplementary archive.

---

OPT-350m
OPT-1.3b
OPT-350m human M-disc M-cont
OPT-1.3b
human
M-disc M-cont
human M-disc M-cont
human
M-disc M-cont
100
100
human
65
26
80
human
80
60
M-disc
33
15
39
20
M-disc
20
76
32
M-cont
46
43
M-cont
43
76
20
0
Unit activation overlap (0 100) between human, M-disc and M-cont prompt types for OPT-350m (left) and OPT-1.3b (right). Higher values (more intense color, larger squares) represent larger overlap. Con- fidence intervals (CIs) are shown as square outlines: thicker lines indicate wider Cls (CIs are generally small), Within-prompp overlap is higher than betweem-types overlap, suggesting a difference processing.
Figure 1:
Figure 2: Percentage input similarity between human, M-disc and M-cont prompt types for OPT-350m (left) and OPT-1.33 (right). Higher values (more intense color. larger squares) cue high similarity. Within-prompt-lppe similarities (the scores on the diagonal) are generally higher than similarity between types. Note that the absolute values of the input similarity obtained with both model sizes are not directly comparable due to a difference in the input dimension (2048 vs. 1024).
Output agreement For each pair of same-task templates, we measure the proportion of test cases where the templates lead to the same prediction. We then compute prompt-type-level averages.
counterintuitive patterns in LM prompt processing.
4.1 Human and machine-generated prompts are processed differently.
Uncertainty quantification We provide the un- certainty estimation of our measurements by com- puting the 95% Confidence Interval (CT) of each measure. In Table 1, the CI associated to each metric (accuracy, perplexity, attention distribution score and output entropy) is obtained by comput- ing the 0.025 and 0.975 quantiles given the list of scores obtained with each templates of given prompt type (note that each template's score is averaged the level of the relation). 95%CI in Figure/ 1,2,3, and Table are obtained using boot- strapping by randomly sampling with replacement from the list of templates (the number of resamples is found by incrementally increasing it until the uncertainty estimation converge).
High accuracy and high perplexity As con- firmed in Table 1, the main motivation use machine-generaiee prompts is their good perfor- mance, M-cont prompts outperforming human ones by 25pts.. This higher accuracy comes along with lower output entropy, suggesting better L.M calibra- tion, where a larger mass of the output probability distribution is concentrated on the correct token. However, prompt perplexity quantifying the de- gree of predictability of token sequence given LM is two order of magnitude higher for M-disc than for human templates." We discuss this further in Section 4.2 below.
Low activation overlap of knowledge neurons Activation overlap statistics are provided in Fig- ure For both OPT-350m and OPT-1.3b, we observe that, while tuthin-prompt-type overlap is mild or high, ranging from 33 to 66 (on a O-to- 100 scale), between-prompt--ya overlap is always
4 Processing machine-generatee prompts
We experimentally demonstrate that differences be- tween human and machine-generated prompts exist at three different levels: (1) at the input level, when comparing prompt types in the embedding space, (2) at the output level, when analyzing predictions and output probabilities, and (3) at the level of in- termediate activation, indicating a difference in pro- cessing at work in the LMM We conclude this quan- titative analysis by showing that, although these metrics are correlated when compared within the same prompt type, the correlation is weak between prompts of different types, leading to a number of
Calibbaation in LM analysis (e.g., Liang 2023) refers the confidence that model has in predictions when the latter correct. Our output entropy measure does not directly correlate confidence and accuracy. However, as ma- chine prompts are general more likely to trigger the correct output
the same time, they have lower output entropy, the global trends suggest that they tend to produce correct answers with more confidence. We informally use term "calibration" to refer to this property.
'Due to their continuous nature, there is not trivial way to estimate perplexity for M-cout prompts.

---

OPT-350m
OPT-1.3b
human
M-disc
M-cont
human
M-disc
M-cont
Accuracy
29.5
43.4
54.9
28.8
46.1
58.0
195% CII
[11.5, 65.0]
[17.0, 79.5]
[20.7, 86.0]
[10.4, 78.21
[15.1, 83.4]
[23.8, 89.6]
Perplexity (10*)
0.60
40.9
0.40
30.3
195% CII
[0.1, 1.9
[16.0, 95.01
10.04, 1.481
12.0, 911.31
Attention distribution (%)
34.4
30.0
23.2
30.8
28.7
29.4
[95% CI]
129.2, 39.7]
32.55
25.51
[17.0, 85.8]
[16.6, 84.4]
[14.3, 74.7]
Output entropy
5.00
4.30
2.10
4.70
3.90
2.10
[95% CII
[3.2, 6.01
[1.9, 5.7]
[0.5, 4.31
[1.7, 6.0]
[1.3, 6.41
10.4,
Table 1: Human and machine-generated prompts (both M-disc and M-cont) significantly differ in at least four aspects: (I) machine-generated prompts outperform the human ones in terms of accuracy; (2) they also better calibrated on average, given their lower output entropy; while at the same time (3) machine-generatee prompts are less predictable by the LMs, reaching significantly higher perplexity; (4) in machine- machine-generated prompts, attention is concentrated on a smaller amount of tokens. For technical convenience, perplexity is not computed for M-cont.
low, ranging from 13 to 26. This pattern is more pronounced when comparing human and M-cont. Between-prompl overlap tends to be higher with OPT-1.3b, suggesting that larger LMs could show a convergence of human and muchine-gennrrtee prompts (this remains to be further explored). The low-overlap result is confirmed by the diagnostic classifier analysis presented in App. B, that shows that a simple linear classifier can distinguish be- tween any prompt type pair based on activation patterns on any layer of either LM.
4.2 Surprising aspects of LM processing
The significant differences that emerged between human and machine prompt processing suggest that these prompt types trigger different decision pathways. Furthermore, they provide interesting insights concerning the nature of LM processing, and, in particular, how it can occasionally be quite counter-intuitive. We explore this by considering some unexpected correlation patterns.
Perplexity does not predict accuracy Gonen et al. (2022) reported negative correlation be- tween perplexity and effectiveness of handcrafted prompts. However, we observe that, when using machine-genenn generated prompt, it is possible reach higher prediction accuracy while having higher perplexity. Thus, counter-intuitively. perplexity does not necessarily predict effectiveness.
Attention is focused on fewer tokens As trans- former behaviour is by-product of both FF and attention layers, we also look at the difference in at- tention distributions, shown in Table 1. Here again, we observe clear distinction between human and machine--gnerran prompts, the latter leading to attention being focused on a smaller amount of to- kens. Recall that prompt length is a hyperparameter of automated prompt induction algorithms, fixed at 5 tokens without tuning. This result might sug- gest that the algorithms only associated meaningful information to subset of these tokens.
Input similarity does not predict output agree- ment The Inpul predicts Ouiput? column of Ta- ble 2 measures the correlation between embedding- space similarities of same-task templates (c.g., a human and a M-dise template for the continent of relation) and the rate of output agreement (de- fined as the portion of times different templates lead to the same prediction) for the corresponding templates. There is a significantly lower correla- tion when templates belonging to different prompt types are compared, especially when comparing neeneae- templates (e.g.. hu- man vs. M-disc templates), than for within-type comparisons (e.g., different M-dise templates for the same task). When comparing different prompt types, counter- -nnuttrrittuitively. the degree of similarity
Machine prompts are drifting away from Hu- man prompts in the input space. Figure 2 shows that, for both OPT-350m and OPT-1.3b. in- put similarity within prompt types is higher than similarity between different prompt types. In par- ticular, the input similarity between human and Ma- chine prompts dramatically decreases when mov- ing from discrete to continuous prompts.
human

---

Input predicts Output?
Input predicts Activation?
Activation predicts Output?
OPT-350m
human vs. M-disc
0.11 [-0.06, 0.27]
0.21 [0.16, 0.26]
0.01 [-0.05, 0.07]
human vs. M-cont
0.14 [0.01. 0.26]
0.06 [0.02, 0.09]
-0.04 [-0.08, -0.00]
M-disc vs. M-cont
0.30 10.18, 0.42]
0.06 [0.03, 0.08)
0.02 [-0.01, 0.05]
human vs. human
0.53 [0.43, 0.62]
0.73 [0.69, 0.77]
0.66 [0.59, 0.72]
M-disc vs. M-disc
0.54 [0.45, 0.63]
0.85 [0.83, 0.88)
0.55 [0.49, 0.63]
M-cont vs. M-cont
0.63 [0.55. 0.72]
0.74 [0.69, 0.78)
0.54 [0.47, 0.61]
OPT-1.3b
human vs. M-disc
0.18 [0.07, 0.28]
0.24 [0.18, 0.30]
0.13 [0.05, 0.22]
human vs. M-cont
-0.06 [-0.21, 0.08]
0.08 [0.03, 0.14]
-0.03 [-0.08, 0.03]
M-disc VS. M-cont
0.12 [0.01. 0.22]
0.03 [-0.01, 0.07]
-0.00 [-0.05, 0.04]
human VS. human
0.58 [0.52, 0.66]
0.65 [0.61, 0.68]
0.60 [0.55, 0.65]
M-disc vs. M-disc
0.64 [0.58, 0.70]
0.89 [0.87, 0.90]
0.61 [0.55, 0.67]
M-cont VS. M-cont
0.78 [0.73, 0.83]
0.74 [0.71, 0.77]
0.53 [0.49, 0.57]
Notation: average 195% confidence interval]
Table 2: Pearson correlations between input similarity, output agreement and activation overlap. First, we compute a single comparative statistic (imput similarity, output agreement or activation overlap) for each pair of prompts in some comparison set (e.g., human vs. M-disc human vs. human); then, for each comparison set, look the correlation across prompt pairs between two statisties (c.g.. input similarity vs, activation overlap). Within-type correlations range from mild to high. Between-typs correlations are significantly lower. These low correlations highlight counteractive aspects of LM language processing. Results in bold significant (p < 0.01). We provide the average and [95%CI interval] correlations obtained using bootstrapped uncertainty estimation.
is not good predictor of whether the templates will trigger the same output or not.
generated prompts trigger units which are distinct from the ones triggered by human prompts. The units that are most often activated by the various prompt types also appear, to some degree, to be distributed differently across layers (ef. Figure 3). In particular, machine prompts display tendency to activate more units on the last layer and, espe- cially, on the first one (it is worth recalling that this is the first proper Transformer layer, and not the embedding layer). The M-disc profile lies some- where between human and M-cont, confirming the trends already observed in Section 4.
Activation overlap is only weakly correlated with output agreement and input similarity Ta- ble 2 also provides correlations between activation overlap and input similarity (Input predicts Activa- tion?) or output agreement (Activation predicts Output?) across different pairwise prompt type combinations. Within a prompt type, these correla- tions are mild or high, with the higher correlations pertaining to input similarity. On the contrary, ac- tivation overlap ceases to be correlated to either input similarity or output agreement as soon as we compare different prompt types. This drop in cor- relation highlights the complexity of LM internal processing. Without any prior input type, it is difficult to predict the decision pathway that will be used by the model, even in the presence of high input or output similarities.
5.2 Profiling
units through associated vocabulary items
Having shown that the three prompt types activate different pathways through the network, we seek now some insights into the nature of the units char- acterizing these different pathways.
Methodology We identify those units that are both typical of single prompt type across tasks, and significantly impacting the network output dis- tribution, in the sense that their gradients w.r.t the max output probability are in the top quartile of all network units (recall that, as usual, we fo- cus on those units we identified as knowledge neu-
5
A closer look at the typical units of each prompt type
5.1 Unit distribution across layers
The low activation overlap between prompt types reported in Section 4 taught us that machine-

---

8192
Type prompt
human M-disc M-cont
Units in the top 20% activation
6000
4000
2000
12
13
14
15
16
17
18
19
20
21
22
23
Layer
Figure 3: For each prompt type, we plot the number of units belonging to the top 20% most activated units (overall across prompt types). M-cont and M-dise have significantly more highly activated units than human on the first layer, with the effect particularly strong for M-cont. There also weaker tendency for the machine prompts to activate more units on last layer compared to human ones. Data from OPT-1.3b.
rons). We define the typical units for prompt A as those that among the top 20% most frequently activated by this prompt type, while at the same time being among the bottom 20% least frequently activated by prompt types and C.5 This filter- ing procedure leaves 14 human (resp. 6), M-dise (resp. 4) and 58 M-cont (resp. 238) units for OPT- 350m (resp. OPT-1.3b). As sanity check, we also repeated the analysis with laxer thresholds involv- ing more units, and the results were similar to the ones report here.
ble 4). Improving unit-item association extraction is left to future work.
Having obtained the list of vocabulary items sociated to each unit, we count how many times each vocabulary item occurrs in association with any typical unit each prompt type (as defined at the beginning of this Methodology paragraph), obtaining 3 frequency lists, one for each prompt type. We compare the relative frequencies of each vocabulary item in each list determine which vo- cabulary items most distinctively associated (the of typical units of) each type. In particular, using a standard method from corpus linguistics, we compute the local Mutual Information score (Evert, 2005) between each vocabulary item and each prompt type t:
Next, we associate each selected unit to set of items from the LM vocabulary that strongly trig- ger its activation. Using the Wikipedia corpus, for each item in the vocabulary we save the aver- age unit activation in a forward pass. We sort the resulting matrix to get, for each unit, the top 500 vocabulary items leading the strongest activation. We extract both input items, recording unit acti- vation when an item the input sequence, and output items, recording activation when an item is predicted by the L.M. We apply lower-casing and initial-space stripping on the resulting vocabulary set. More details are provided in Appendix C. This method has been chosen for its simplicity. How- ever, it is also noisy and sensitive citing" tokens (e.g., magikarp. see Appendix C Ta-
P(u,t) LMI = |u, log P(v)
where lv I| counts the occurrences of in the list, the joint probability P(v, t) is estimated based on lv, fl: P(v) is estimated using the cumulative occurrence count of in all lists, and P(t) is the total number of occurrences of any item in the list. Table reports the top-30 30 input vocabulary items ranked by LMI for each prompt type and both LMs.
but "ex-
Machine prompts recruit "non-linguistic" units Looking at the OPT-350m results first, nearly all characteristic human items are well-formed words,
'A unit activated when its value greater than "Subset "20220301.en enn"" from HuggingFace

---

human
M-disc
M-cont
OPT-350m
whats
gazed
ful
handler
(&
361 ancel
Óol2
HII
Concerning OPT-1.3b. we observe pretty much the same patterns for human and M-cont. For M- disc, the other hand, together with number of non-English diacritics and special symbols, there is a strong increase in regular words and word frag- ments, although the latter still clearly differ from those associated to human prompts, in that syntax- related items, such as function words or inflected verb forms, are largely missing. In line with what we observed in Figure 1, we thus observe a cline which, at least for M-disc, the difference in pro- prompts name nifty
darn
expr
avascript
Onfip2
page
pwt
why
devs
Treaking
iterator
cpp
yout
DE
(N
Sts
fuck
much these terness
addons
risome
O):
0000
1}
noticed
like
have
hillary
frameworks
+
table
really
daddy
wanna
filename
lication
ithub crossref
interstit.
thats
likes what
easy
702
Öof printin
stats
/ou
does
honestly
crappy relent been
disabled
502
warn
));
thing
workaround
601
poons
})
crip
goddamn
bothers
json
sacrific
inline
debug
OPT-1.3b
undermines
severe dictates
conducive frowned
ahO
attle
äkg Algäigat endif
Icilan
looph
curls
appalling
extreme
cram
ataнāи everal
-
makes tempt
optimized mal eff will unfold remain early monitor
dop egregious
ākj
canaver citiz %%%% äk %
archdemon
xff
does bounces
reap complex schizophren stroll hou
rep pass insancly kinda añ
marketable
manifosts
persist
rece
0000
prevail outweigh
shines gres fail hangs
añé:
acper
haunt
haun
crazy
dilig
nanto
meshes
smokes
governs
delay
shitty
prototype
cryst
wipes
poised
fills
capital
I//
devices
â-â-
leban
Table 3: Top vocabulary items associated to each prompt type ranked by LMI. Machine Machine-generated prompts respond to less language- like items than those triggered by human prompts. Nearly all human items well-formed words. Many M-disc items, on the other hand, are non- non-English diacritics, special symbols or code-related terms. M-cont items are entirely non--linuuistic.. Some strings have been abbreviated to fit column width.
and include high number of forms cuing syntac- tie processing, such as function elements (whats, why, does...). inflected verbs (noticed, gazed, liked...) .) and modifiers (really, much, honestly...) A remarkable amount of M-disc items are coding- related terms (handler, expr, iterator...) Numbers and punctuation sequences that could be coding- related or web-page boilerplate ((&,)) \-) also appear, well few regular words or word fragments (Hiillary, easy, sacrific... Finally, for M-coni, the items are entirely "inn-iinguistic", being composed sequences of non-Latin characters or punctua- tion marks, as well as code fragments.
ways, but that the units involved in these pathways tend to respond less language-like items than those triggered by human prompts. Note that these units are spread across the layers of the network, so that we are not only recording low-level differ- ences in processing the input strings or vectors.? Moreover, the results are largely mirrored by those obtained when associating units with output instead of input vocabulary items (Table 5 in App. D).
Recall that our analysis is based on units that are not only highly typical of prompt type across rela- tions, but also in the top gradient quartile, suggest- ing that they significantly contribute the model's output distribution. is puzzling that units that mostly respond to coding fragments or unusual characters could lead the network to produce the correct next token in the semantic tasks we are studying. We conjecture that distributed activation from such units might nudge the network towards the right output semantic fields through connectiv-
"The selected typical human units occur in layers 4th to last of OPT-350m and layers 2 to 14 of OPT-1.3b (counting from 0). The M disi units range from layers 3 to 10 of OPT- 350m and 2 to 13 of OPT-1.3b. The M-coot units are the only ones where, as expected given the distribution illustrated in Figure 3, significant proportion occurs on the first (0-th) layer (about one third for OPT-350m and one fourth for OPT- 1.3b), but the remaining two thirds/three fourths range from layers to 21 and 1 to 10, respoctively.
cessing human and machine-generann decreases with model size.
We tentatively conclude that machine prompts are not only triggering different activation path-

---

ity pathways that fortuitously arose during network training. This is an important topic for future work.
In particular, we plan to zoom further in into the processing of specific templates, tracking their pro- cessing throughout the network with methods such as the vocabulary-baan based unit analysis of Section 5.
6 Discussion
We have studied the phenomenon of linguisti- cally and semantically opaque machine- generrttd prompts from the perspective of how LMs process them, compared to human-crafted ones. Our study has important Limitations, that are discussed in the relevant section below. However, at least for the prompt generation methods, LMs and tasks we explored, we can draw some general conclusions.
On investigating unnatural language We be- lieve that investigating "unnatural language" as we did here (see also Khashabi al., 2022; Ishibashi et al., 2023; Rakotonirina al., 2023) should be a central concern to NLP for at least three reasons. First, understanding why LMs work as well as they do, and what are their failure modes, is one of the questions with the broadest scientific and societal implications we can ask today. It would however be dangerously limiting to narrow our in- vestigation to how LMs process natural language only, ignoring their behaviour when presented in- puts outside their training distribution.
More than a "happy accident" Our evidence suggests that the differences between human and muchine-generater prompts are not just superficial, but affect all levels of network processing, and result in the activations of qualitatively different units. Some of these units are stable across seman- tie tasks, suggesting that they are more generally recruited to process any "unnatural" input. More- over, contrary to what one could reasonably predict, there is some evidence that machine prompts are more robust than human ones, in the sense that they achieve better output calibration.
Second, unnatural language can be exploited for negative purposes, as shown by Wallace et al. (2019) and Zou et al. (2023), who derived appar- ently nonsensical prompts that could steer multiple LMs' responses towards harmful behaviour, such as generating racist language.
Finally, there is recent interest in letting LMs di- rectly communicate with each other to jointly solve tasks or build community (Park et al., 2023; Zeng et al., 2023). Based on our evidence, might be pointless to insist that LM-to-LM communica- tion takes place in natural language, given that LMs might share information more efficiently through unnatural prompts. Conversely, if being able to decode the communication flow is important (e.g., for safety reasons), care must be taken stop LMs from drifting into unnatural language.
It's unlikely that the LM has been exposed to anything like M-disc prompts during its initial train- ing, and definitely it could not have seen out-of- vocabulary M-cont prompts, so we can only as- sume that the special pathways triggered by these prompts arose through unforeseen side effects of pre-training. 8 However, they seem to be more than just lucky connectivity accidents exploited by spe- cific prompts to solve specific tasks, or clse it would be difficult explain the overall low entropy of machine prompt predictions and the commonali- ties in the units they activate. Moreover, there is evidence that M-disc prompts can transfer across Transformer-busee LMs (Rakotonirina et al., 2023; Zou et al., 2023), suggesting that unnatural lan- guage pathways might arise from the interaction of general characteristics of the Transformer ar- chitecture with Web-derived training data that are partially shared across many current pre-trained LMs. We must defer a better understanding of the nature of these unnatural pathways future work.
For all these reasons, we hope that our prelimi- nary contribution will encourage our community to pay more attention to the phenomenon of unnatural language processing.
"We experimentally verified that model training is neces- sary for effective unnatural prompts to arise. We ran both Autoprompt and Optiprompt on 3 distinct random initializa- tions of OPT-1.3b with the same hyperparameters as in our main experiments, and found that the resulting M-dise prompts achieved 0% accuracy in nearly all cases, whereas M-cont where at best able to retrieve the majority output of a task.

---

Limitations
the Barcelona Apple Machine Learning Research group and the participants in the EviL seminar for helpful feedback and suggestions. Our work was funded by the European Research Council (ERC) under the European Union's 's Horizon 2020 research and innovation programme (grant agreement No. 101019291). This paper reflects the authors' view only, and the ERC is not responsible for any use that may be made of the information it contains.
Main limitation: We presented an extended study of how two pre-trained language models process human and machine-generated inputs, but we did not provide an account of why are observing the processing differences seeing. We noticed, for example, that M-cont prompts activate units associated to punctuation marks and special characters. We do not know, however, in which way these units contribute to retrieving the correct an- swer in the target semantic tasks, nor how the optimization procedure chances upon them. This is our priority for future work.
References
Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, Ludwig Schubert, Chelsea Voss, Ben Egan, and Swee Kiat Lim. 2020. Thread: Circuits. Distill. https://disiill.pu//20ooiiiiiii
Our work is limited to the OPT family of mod- els trained on the English language, to the LAMA semantic tasks and the AutoPrompt and OptiPrompt prompt extraction methods. A straightforward direction for future work is to extend our analysis to more models (includ- ing instruction-tuned models, as instruction tuning might have a significant impact on how models respond to unnatural input), languages, data-sets and prompt extraction algorithms.
Damai Dai, Li Dong, Yaru Hao. Zhifang Sui. Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493- 8502.
Mingkai Deng, Jianyu Wang, Cheng-Ping Hsich, Yihan Wang, Han Guo, Tiammin Shu, Meng Song, Eric Xing. and Zhiting Hu. 2022. RLPrompt: Optimizing discrete text prompts with reinforcement learning. In Proceedings of EMNLP, pages 3369-3391, Abu Dhabi, United Arab Emirates.
Ethics Statement
The advent of publicly accessible LM interfaces such as ChatGPT has heated up the debate around the broader impact of LMs. While there is a variety of possible societal issues to consider (Weidinger et al., 2022), we believe that better understanding of how LMS process information is crucial part of bias and harm containment. If we do not under- stand the models, we cannot control their behaviour, and we are exposed to intentional adversarial at- tacks and other forms of unintentional model mis- use. The very existence of completely opaque but empirically effective machine-genen generated prompts is proof of how counterintuitive the behaviour of LMs can be, and of how little we understand them. We thus believe that our investigations of "unnatu- ral language processing" fit well into the broader program of improving our scientific understanding of LMs, in order to make them more predictable, controllable and, ultimately, safer.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhi- lasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. Transac- rions of the Association for Computational Linguis- tics, 9:1012-1031,
Hady BlSahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon S. Hare, Frédérique Laforest, and Elena Paslaru Bontas Simperl. 2018. T- rex: A large scale alignment of natural language with knowledge base triples. In International Conference on Language Resources and Evaluation.
Stephanie Evert, 2005. The Statistics of Word Covccur- rences. Ph.D dissertation, Stuttgart University.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang. Horace Hc, Anish Thite, Noa Nabeshima. Shawn Presser, and Connor Leahy. 2020. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2 100000277.
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Gold- berg. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 30-45.
Acknowledgements
We thank Emmanuel Chemla, Emily Cheng, Nathanaël Carraz Rakotonirina, Xavier Suau, the members of the UPF COLT lab, the members of

---

Mor Geva, Roci Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer worward layers are key- value memories. In Proceedings of EMNLP, pages 5484-5495, Online and Punta Cana, Dominican Re- public.
Zachary C Lipton. 2018. The mythos of model inter-
pretability, Communications of the ACM, 61(00)36 43.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig, 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35.
Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018. Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information. In Proceedings of the EMNLP Black- boxNLP Workshop, pages 248, Brussels, Bel- gium.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual asso- ciations in GPT. Advances in Neural Information Processing Systems, 35: 17559-7-777
Hila Gonen, Srini Iyer, Terra Blevins, Noah Smith, and Luke Zettlemoyer. 2022. Demystifying prompts in language models via perplexity estimation. https: / serrxiv.org/abs/21111 /ttp///bb22222.4323 84037.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020, Zoom in: An introduction to circuits. Distill.
Ashim Gupta, Giorgi Kvernadze, and Vivek Srikumar. 2021. BERT and family eat word salad: experiments with text understanding. In Proceedings of AAAI, pages 12954, Online.
Joon- loon-Sung Park, Joseph O' O'Brien, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. https://arxix //ass////xivvooo wrt.kii..oog//b///4/444444/ 03442.
Yoichi Ishibashi, Danushka Bollegala, Katsuhito Sudoh, and Satoshi Nakamura. 2023. Evaluating the robust- ness of discrete prompts. In Proceedings of EACI., pages 2384, Dubrovnik, Croatia.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gram- fort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vin- cent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthicu Brucher, Matthieu Per- rot, and Édouard Duchesnay, 2011. Scikit-learn Ma- chine learning in Python. Journal of Machine Learn- ing Research, 12:2825-2830.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis. Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings EMNLP, pages 2463- 2473, Hong Kong, China.
Daniel Khashabi, Xinxi Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sean Welleck, Hannaneh Ha- jishirzi, Tushar Khot, Ashish Sabharwal, Sameer Singh, and Yejin Choi. 2022. Prompt waywardness: The curious case of discretized interpretation of con- tinuous prompts. In Proceedings of NAACL, pages 3631-3643, Seattle, WA.
Thang Pham, Trung Bui, Long Mai, and Anh Nguyen. 2021. Out of Order: How important is the sequen- tial order of words in a sentence in natural language understanding tasks? In Findings of ACL, pages 1145-1 160, Online.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of EMNLP, pages 3045-3059, Punta Cana, Dominican Republic.
Nathanaël Rakotonirina, Roberto Dessi, Fabio Petroni, Sebastian Riedel, and Marco Baroni. 2023. Can discrete information extraction prompts general- ize across language models? ings of ICLR, Kigali, Rwanda. Published on- line: https: /ttpss///ppeeeee.../omm http:///eeeeeeeeeeee/rrorrrrddde www..et/grroupidd-cu.. cc/28233/Coferermn
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang. Deepak Narayanan, Yuhuai Wa, Ananya Ku- mar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher Man- ning, Christopher Ré. Diana Acosta-Navas Drew Hudson, Eric Zelikman, Esin Durmus, Faisal Lad- hak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Hender- son, Qian Huang, Ryan Chi, Sang Xie, Shibani San- turkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Ieard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2023. Holistic evaluation of language models. attps://arxils www.rg///oooo orgg/bss//// 11121111111111111111111111111111 09.10.
In Proceed-
Hubert Ramsauer, Bernhard Schäfi, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, Michael K Kopp. et al. 2021. Hopfield networks is all you need. In International Conference on Learning Representations 2021.
Taylor Shin, Yasaman Razeghi, Robert Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Elic- iting knowledge from language models with automat- ically generated prompts. In Proceedings of EMNLP. pages 4222-4233 Online.

---

Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021a. Masked language modeling and the distributional hy- pothesis: Order word matters pre-training for little. In Proceedings of EMNLP, pages 2888-2913, Punta Cana, Dominican Republic.
Andy Zou, Zifan Wang, Zico Kolter, and Matt Fredrik- son. 2023. Universal and transferable adversarial attacks aligned language models. http://arxiv. org/ org/abs/2307, 15043.
Koustav Sinha, Prasanna Parthasarathi, Joelle Pineau, and Adina Williams. 2021b. UnNatural Language Inference. In Proceedings of ACL, pages 7329-7346, Online.
Chelsea Voss, Nick Cammarata, Gabriel Goh, Michael Petrov, Ludwig Schubert. Ben Egan, Swee Kiat Lim, and Chris Olah. 2021. Visualizing weights. Distill. Hups://distill http://iiillppm22222222ccei/i/// weights.
Eric Wallace, Shi Feng. Nikhil Kandpal, Matt Gard- ner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In Pro- ceedings of EMNLP, pages 2153-2162, Hong Kong, China.
Albert Webson and Ellie Pavlick. 2022. Do prompt- based models really understand the meaning of their prompts? In Proceedings of NAACL, pages 2300- 2344, Seattle, WA.
Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang. John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, 7ac Kenton, Will Hawkins, Tom Stepleton, Abeba Birhane, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Sean Legassick, Geoffrey Irving, and Iason Gabriel. 2022. Taxonomy of risks posed by language mod- els. In Proceedings of FAcoI, pages 214 229, Scoul, Korea.
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Fed- erico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. 2023. Socratic models: Compos- ing zero-shot multimodal reasoning with language. In Proceedings of ICLR, Kigali, Rwanda. Pub- lished online: https: wwwpppewwwwweee//rrreeeeeeeeeeee id=ICLR. ccc///23//offereecce
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: Open pre-trained trans- former language models. https://arxiv.. abs/2285.00 01068.
Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing is [MASK]: Learning vs. learning to recall. In Proceedings of NAACL, pages 5017-5033,
Online.

---

A Measuring the overlap of activated knowledge neurons
Layer O, unit 248
"(
In Section 4, we measure and compare which knowledge neurons, that is, units found in the in- termediate layers of the wred-fowward Transformer blocks (Dai al., 2022; Geva al., 2022), are ac- tivated by different prompt types. In particular, we measure the knowledge neuron activation overlap (abbreviated as activation overlap). The following algorithms in pseudo-code detail how this measure is obtained.
Layer 10, unit 674
antidepress
debian minecraft awakens
frieza
magikarp
xperia
oneplus
bitcoin
Layer 16, unit 2126
filename
windows drm
sshd
misunder
m
folder
pkg
vm
First, construct a Boolean matrix recording which units are activated by template. As shown in the pseudo-code below, unit said to be ac- tivated if its value greater than on more than k = 20% of cases when instantiated with each of the subjects associated to the template relation. In the pseudo-code, Relations is the set of re- lations available in our task set; given a relation, Subjects provides the list of relevant subjects and template instantiates the template with subject s. Model is the LM being used.
Layer 22, unit 3617
everal
huge
every
risome
some
any
these
crappy
nifty
Table 4: forward layers extract the set of items leading to the strongest activation on the Wikipedia corpus. As an illustration, we selected four different units with distinet profiles and displayed them in this table, along with their top 9 most associated input items, for OPT-350m. We observe varying degrees of consistency and naturalness across units.
For each intermediate unit in the OPT's feed-
def get_act ael.actttemlatee relation): cpt
ing the activation overlap for pairs containing one human and one M-cont).
= 0.2
iterate across subject for s in Subjects(relation): inpt template(s)
B Diagnostic classifiers
for u in Model (inpt) units: cpt[u] += (u>0) u is activated if for more than kx of inputs act cpt > kelen((ubjecttsrellattt))) return act
To complement the activation--overlp--bee analy- sis presented in Section 4.1 the main paper, we run a set of shallow linear "diagnostic" classifiers (Giulianelli et al., 2018) of the activations gener- ated by the models each layer in response to inputs from each prompt type. As usual, we focus the activation of knowledge neurons.
Then, for each pair of templates (in Templates), we compute the intersection over union of their respective activation matrices:
Data As in the main paper, we use all templates with LAMA accuracy dom subset of the LAMA P176 relation human tem- plates, as this relation is greatly over-represented. We are left with and 24 tasks for OPT-350m and OPT-1.3b, respectively. As we are interested in units inherently distinguishing prompt types inde- pendently of lexico-semantic aspects associated to specific templates or tasks, partition the data that the training and test data contain disjoint tasks (and, a forotiori, disjoint templates). We consider such partitions, each time using data from 16 (OPT-350m//18 (OPT-1.3b) tasks for training and (OPT-350m) /6 (OPT-1.3b) for testing, such that
10%, filtering out ran-
overlap { for in Relations:
for t_A in Templates(r):
for t_B in (r)p
act_A get_act(t_A,r)
act_B get_act(t_B,r)
act_A act_B
u act_A act_B overlap[(t_A, t_B)] = i/u
Finally, we average these pairwise overlaps while filtering by prompt type (e.g., only averag-

---

there is test task overlap across the partitions. We instantiate each template with 10 randomly se- lected subjects from the corresponding LAMA lists. For each pairwise classification task, we balance the test instances by downsampling the larger class, so that chance/majoriyy/mnnrriyy accuracies ("base- lines" in Figure 4) are at 50%.
meaningful context. In addition, we the window stride to 15 save computation time. Furthermore, due to our limited computation resources, and given the size of the Wikipedia corpus (6B), we only used 66% the data for OPT-1.3b.
Samples As illustrated in Table 4, came up with large diversity of unit profiles, some of them being associated to more or less linguistically valid items, and with varying degree semantic consis- tency.
Classifier We use a logistic regression classifier with L1 regularization (to encourage sparseness), with the LI term coefficient fixed at a = 0.01. We fit the classifier with stochastic gradient descent, us- ing the Scikit-learn toolkit (Pedregosa et al., 201 1). For each of the 4 training partitions, we repeat the experiment with 5 different seeds, resulting in total of 20 runs for each layer.
D Profiling typical units by output vocabulary analysis
Table in the main paper reports the 30 imput vo- cabulary items with the largest LMI with respect each prompt type. Table 5 here reports the top- 30 output items. We largely confirm the same trends, although we do notice an overall tendency for the units triggered by machine prompts to be associated to more "language- -iike""ee"ke" output material, which makes sense ultimately these prompts do produce well-formed task-relevant outputs.
Results. Figure reports average per-layer clas- sification accuracies for each pairwise prompt type comparison, with standard deviations across the 20 runs. In all cases, accuracy values are well above baseline level, and typically very high. We con- clude that each single layer contains units whose activation is sufficiently discriminative for each prompt type to successfully train the classifiers, de- spite the challenging setup in which training and test tasks (and consequently templates) are com- pletely disjoint. Interestingly, few units on each layer suffice to discriminate prompt types (the aver- age classifier weight sparsity across all experiments is at 99.5% with 0.2% standard deviation for OPT- 350m and 99.6% with 0.4% s.d. for OPT-1.3b). The easiest distinctions involve M-cont as one of the classes, confirming that out-of-vocabulary embed- dings make continuous prompts particularly differ- ent from natural language. Indeed, it's remarkable that distinguishing M-dise from M-cont is gener- ally easier than distinguishing between M-disc and human prompts.
To conclude, the classification experiments bring strong convergent evidence that genuinely differ- ent pathways characterize different prompt types across all layers of the network.
Unit/vocabulary item association
Implementation details When extracting unid/vocabulary-iten association, we empirically set the window size to 15. This value is reason- able close to prompt size (5 in average), while containing a sufficient amount of tokens to get a
"As we have 21 total tasks meeting our conditions for OPT- 350m, one of the 4 partitions used for this model consists of 15 training and 6 test tasks

---

OPT-350m
OPT-1.3b
10
1.
0.8
0.8
Accuracy
Accuracy
0.7
0.7
0.5
0.5
human M-cont
human vs M-disc
0.4
M-disc vs M-cont
baselines
10
20
15
Layers
Layers
Figure 4: Average accuracies and standard deviations for 20 runs of the pairwise prompt-type classification experiments across the 24 layers of the networks.

---

compare
comple
attempt
currently
älgäigä
pref
also
L' therefore 'V:
0000
prev
ÖF inst ) anj
air
1
ahi
moreover
M M-cont
sample aq
.aa
)):
text
1."
å a-a
äń atg say
+= any
url
meanwhile
kinnikuman
services provider atataratataa
checking
forge physic supported tyr dst
tracker
insert ceremon header
%% 96 96
On
avg
äkg atai
aí i
localhost filib though
budgetary
billboards
acquies
murky
recip
disag
game
blat
772
%
foundational
outlandish
independ
M disc
xcom
invoke
mysql
oddly
vaugh
pause
scram
OPT-350m
fanc
ridic
782
":1"
698
(1
OPT-1.33 1.3b
prett
mpg
bicy
understanding
monstrous trembling check
incompet alluded
juxtap
revert insin faintly irresist
recone
adolesc
collaps disbel
unpop
aT/4
)
"
authenticated snowball earlier
scrambled
rcasonably
deprecated inherently
peeled distinctly o0z
pinpoint
tempted
tonight
messed
waging incred
impecc
tasted
right
surg
interchangeable
human
attributable solely
unmatched
overloaded
obligatory
constrained sandwic
achievable
undone
Lnybody
plugged
captcha
glanced
tweaked
figured
infused
before
glean
snug
accumulatiig
fundamentally
overpower
substantially
unparalleled
nobody
yesterday
really
awoken
bother
corrobor
parsed
bothering
nailed
reinvest
anonym
overseen
ideally
okay
elic
Table 5: Top 30 output items associated each prompt type ranked by LMI. Some strings have been abbreviated
fit column width.


Reviews:
[{"review_id": "LishdevymR", "reviewer": "Reviewer_jWaw", "summary": "", "questions": "- Question A: It is expected that input similarity would be higher with larger models, as cosine similarity is a poor distance function at high dimensional space. Did the authors try any other distance functions?\n- Question B: More simply, can the authors first show the token overlap % between human and m-disc? Would this be highly correlated to the unit activation overlap?\n- Question C: Did the authors specifically study the examples where machine generated prompts are outperforming human prompts? Is there anything special about the examples themselves in such cases?\n- Question D: When the authors study attention distribution (\"attention is focused on a smaller amount of tokens\"), is this with respect to the prompt or with respect to the input example? How does the attention distribution over the examples change?\n- Question E: The authors note that their evidence suggests the results are more than a happy accident. Did the authors study the logits of the generation, in order to quantify the \"accident\" in terms of model confidence? As in, does the model becomes more/less confident with machine generated prompts?\n\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review, with valuable comments and appreciations. Please find the answers of your questions below:\n\n## Novelty and interest\n\n> [...] the paper essentially does not present anything new or surprising with respect to the model behavior. \n> Since the surface forms of these prompts are wildly different, it is expected that the internal processing (attention distribution, activation overlap) would also be noticeably different.\n\nConcerning the interestingness of our results, please note that, for each type, we are selecting prompts that are performing the same task and have accuracy above a certain threshold (>10%). Thus, **while the inputs are very different, they produce the same or similar outputs** (note the low correlations between input similarity and output agreement in the top right cell of Table 2). We could thus reasonably expect that there are processing differences in the lower layers, but these differences vanish higher up, where the network starts converging on the output to be generated. **The fact that this is not the case is, in our opinion, highly surprising**. **It suggests that different prompt types produce similar outputs through thoroughly different generation strategies.** The diagnostic classifier experiment in appendix B goes even further, showing that, on each layer, **it is possible to predict prompt type from activation profile even when generalizing across knowledge-retrieval tasks**: that is, the activation profiles of, e.g, the place-of-birth and works-for machine prompts have something in common that allows a shallow classifier that has only seen place-of-birth examples at training time to easily distinguish machine-generated works-for examples from the corresponding human prompts on each layer. Again, we think this is very remarkable. \n\nConcerning the fact that we do not present anything “new”, we respectfully disagree: to the best of our knowledge, despite widespread interest in machine-generated prompts, **ours is the very first mechanistic analysis of how such prompts are processed by a language model**.\n\nWe will further clarify all these points when revising the paper.\n\n## Investigating the *why* instead of the *how*\n\n> Investigating unnatural language is indeed a very interesting and useful line of work. However, the main question should be why the model is differentiating its processing on these unnatural input rather than human input [...].\n\nWe fully agree that ultimately the “why” problem is the fundamental one. We think, however, that **we can’t get at the “why” without first characterizing the “how”**. We hope that the set of  “how” results we present can help our community to formulate causal explanations that can already exclude hypotheses that are rejected by our findings (e.g., that differences between prompts only pertain to the earliest processing steps). We are grateful to reviewer oqtz for perfectly capturing our spirit in their review: *“Some readers may find the lack of a causal hypothesis disappointing, but I think for many this will mostly be a hindsight effect thanks to the very clear problem statement in the paper itself: the authors make a strong case that it's worthwhile to study the mechanisms by which prompt engineering works and that too few researchers are currently tackling this question. The detailed exploratory characterization of the empirical phenomenon that this paper provides will likely be an influential early step on the path toward deeper understanding of this question.”*\n\n\n## Cosine similarity\n> Question A: it is expected that input similarity would be higher with larger models, as cosine similarity is a poor distance function at high dimensional space. Did the authors try any other distance functions?\n\nWe are using cosine similarity as we think it is **the most standard measure to compare neural-network-extracted vector representations**. We would be grateful for references about the issues you are mentioning, and we’d be happy to implement any more robust method you recommend. Still, we would be surprised if the large asymmetry in similarity increase between M-cont and human vs. the other comparisons in Fig. 2 was only due to the possible confound you mention (note also that the OPT-350m layer activation space, with its 1024 elements, is already very high dimensional). We have in any case also replicated the analysis using token overlap, as discussed below.\n\n\n## Token overlap\n> Question B: More simply, can the authors first show the token overlap % between human and m-disc? Would this be highly correlated to the unit activation overlap?\n\nThe token overlap between human and m-disc (measured as the intersection over union between the sets of tokens employed by each prompt type) is **2.7%**. As expected, this overlap is low, but this datum might be misleading for at least three reasons: \n1. Tokens found in M-disc prompts are more **diverse** than those in human prompts: we counted 742 different tokens in M-disc prompts, compared to only 96 in human prompts. \n2. The very low token overlap **does not explain the significantly positive output agreement** (counting how often two prompt types lead to the same predictions) between Human and M-disc which is 35%. \n3. The token overlap **does not fully explain activation overlap**. \n\nWe assessed (3) by conducting a small experiment with opt350m, which involved measuring the intra-prompt-type token overlap. For each type, we randomly sampled two sets of 50 templates and measured the token overlap between the sets. We repeated this operation 1000 times to obtain an average token overlap within a prompt type. While Human templates exhibited a relatively high overlap (54% token overlap), we observed a low overlap within M-disc templates (7.6%). However, as stated in the article, we demonstrated that M-disc templates tend to activate the same neurons in the LM (as evidenced by an activation overlap of 31%). Consequently, **a low overlap in input tokens does not necessarily imply a low overlap in terms of neuron activation**. While we acknowledge your point that token overlap contributes to the explanation, we firmly believe and provide evidence for the fact that it is imperative to move beyond direct correlations between token and neuron activation overlaps.\n\n\n## Isolate similar prompts\n\n> A better study would have been to isolate the prompts from each category which are mostly similar in their surface forms, and then investigate whether the model prefers one or the other.\n\nThanks for the suggestion. It is closely related to what we tried to achieve with the analysis in Table 2, which shows that input similarity (Input in the table) is only weakly correlated to activation overlap and output agreement, suggesting that superficial differences between inputs are not the main driver of processing and output mismatches. Reviewer oqtz also found Table 2 confusing, and we will try to get our message across more clearly when revising the paper. Concerning the analysis you suggest, we are specifically interested in cases where different prompt types lead to similar outputs, as these are the cases where differences in processing might cue genuinely different processing methods. When one prompt type produces the right output and another doesn’t, the processing paths might differ simply because different answers are being produced.\n\n## Focusing on machine generated prompts outperforming human prompts\n> Question C: Did the authors specifically study the examples where machine generated prompts are outperforming human prompts? Is there anything special about the examples themselves in such cases?\n\nFor the reasons we discussed above, **we are focusing on cases where all prompt types lead to relatively good performance**, precisely because we are interested in studying processing when prompts tend to produce the same or similar outputs. Our main question is: *why can these alternative, linguistically “crazy” machine-generated prompts produce coherent outputs comparable to those produced by human prompts for the same semantic tasks?* This is a different question from the one of why machine-generated prompts can even outperform human ones. We however agree that the analysis you recommend is interesting, and we will consider it in future research.\n\n\n## Attention distribution\n> Question D: When the authors study attention distribution (\"attention is focused on a smaller amount of tokens\"), is this with respect to the prompt or with respect to the input example? How does the attention distribution over the examples change?\n\nWe quantified the distribution of attention using a method introduced by Ramsauer et al. (2021) (see Figure A.3). This method operates at a global level, which means it is computed at the corpus level rather than the local sample level. Consequently, **our measurement is derived with respect to the prompt type and cannot be replicated at the input example level**. While it's possible to modify the method to achieve a local measure of attention distribution, we hold the belief that **the outcome would likely be noisy and challenging to interpret**, particularly given the diversity of templates. As a result, we opted for the more robust approach of using a global measure.\n\n\n## Quantifying the \"accident\" in terms of model confidence\n> Question E: The authors note that their evidence suggests the results are more than a happy accident. Did the authors study the logits of the generation, in order to quantify the \"accident\" in terms of model confidence? As in, does the model becomes more/less confident with machine generated prompts?\n\nWe observe that the **output entropy decreases for machine prompts** (Output entropy row of Table 1), suggesting higher model confidence. We will clarify the meaning of this statistic in the revision.\n\n\n## Typical tokens\n>  In Section 5.2, the paper uncovers the top \"typical\" tokens for each prompt type, and it is noticeably clear how linguistically different the human generated prompt and the machine generated prompts are\n\nPlease kindly note that the tokens of Section 5.2 are not input tokens, but tokens associated to the units activated by the different kinds of prompts.\n\n\n\n## Additional references\n> The paper misses a significant line of work in uncovering unnatural language processing of Transformer based Language models, including (but not limited to) the following papers:\n\nThanks for the references, we will incorporate them when revising the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is clearly written with good analysis of the differences in processing human crafted prompts and machine generated prompts. The paper defines the studies well, and come to several insights of how the processing methods employed by the LLMs internally are different, while the model clearly favors the machine generated prompts.\n\n", "reasons_to_reject": "- My main criticism is that the paper essentially does not present anything new or surprising with respect to the model behavior. In Section 5.2, the paper uncovers the top \"typical\" tokens for each prompt type, and it is noticeably clear how linguistically different the human generated prompt and the machine generated prompts are. Since the surface forms of these prompts are wildly different, it is expected that the internal processing (attention distribution, activation overlap) would also be noticeably different. A better study would have been to isolate the prompts from each category which are mostly similar in their surface forms, and then investigate whether the model prefers one or the other.\n- Investigating unnatural language is indeed a very interesting and useful line of work. However, the main question should be _why_ the model is differentiating its processing on these unnatural input rather than human input, which is clearly more abundant in the training data. In this paper, we only get a glimpse of _how_ the model is behaving differently, without any explanation to the performance improvement for unnatural input. The authors do highlight this in the Limitations and I thank them for it. However, since the finding does not improve our understanding towards the \"why\" problem, the report is not so interesting for the community.\n\n", "questions_to_authors": "- Question A: It is expected that input similarity would be higher with larger models, as cosine similarity is a poor distance function at high dimensional space. Did the authors try any other distance functions?\n- Question B: More simply, can the authors first show the token overlap % between human and m-disc? Would this be highly correlated to the unit activation overlap?\n- Question C: Did the authors specifically study the examples where machine generated prompts are outperforming human prompts? Is there anything special about the examples themselves in such cases?\n- Question D: When the authors study attention distribution (\"attention is focused on a smaller amount of tokens\"), is this with respect to the prompt or with respect to the input example? How does the attention distribution over the examples change?\n- Question E: The authors note that their evidence suggests the results are more than a happy accident. Did the authors study the logits of the generation, in order to quantify the \"accident\" in terms of model confidence? As in, does the model becomes more/less confident with machine generated prompts?\n\n", "paper_topic_and_main_contributions": "The paper studies the internal behavior of OPT 350M and OPT 1.3B models with respect to human crafted and machine generated prompts. The paper highlights that while the machine generated prompts (using popular methods such as AutoPrompt and Optiprompt) achieve better scores, they elicit processing behavior which is very different from those of the human prompts. The paper doesn't attempt to answer why such a thing exists, instead proposes analysis methods to highlight the differences in processing with \"unnatural inputs\", which is the machine generated prompts.\n\n", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- L201: \"We end up with 5.9 human\" -> \"5.9 M human\"\n"}, "extra_text": {"Missing_References": "The paper misses a significant line of work in uncovering unnatural language processing of Transformer based Language models, including (but not limited to) the following papers:\n\n- Sinha, Koustuv, et al. \"Unnatural language inference.\" arXiv preprint arXiv:2101.00010 (2020).\n- Marzoev, Alana, et al. \"Unnatural language processing: Bridging the gap between synthetic and natural language data.\" arXiv preprint arXiv:2004.13645 (2020).\n- Pham, Thang M., et al. \"Out of order: How important is the sequential order of words in a sentence in natural language understanding tasks?.\" arXiv preprint arXiv:2012.15180 (2020).\n- Gupta, Ashim, Giorgi Kvernadze, and Vivek Srikumar. \"Bert & family eat word salad: Experiments with text understanding.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 35. No. 14. 2021.\n\n"}}, {"review_id": "0GI9ib3vT2", "reviewer": "Reviewer_ZpBf", "summary": "", "questions": "(a) (209) Is “micro-accuracy” a term used in the literature? Is there a reference for this? Otherwise I would just use accuracy and qualify what you mean.\n\n(b) (216) Can you provide an example of “[subject][template]”, as I’m not exactly sure what this looks like. Would it be something like “Lavoisier Island continent of Antarctica” ?\n\n(c) (222) How is the entropy measured?\n\n(d) (230) What does this value range over? What counts as “low” and “high” values for this measurement? (More broadly, this section felt a little telegraphic. I would appreciate a little more detail in the final version here.)\n\n(e) (335) It is an interesting result that the prompts converge to similar input representations for larger LMs. Any speculations about why this is going on?\n\n(f) (354) Again, this section felt a little telegraphic. I would appreciate more detail, examples, and motivation for this analysis.\n\n(g) As noted above, what does the error of the measurement look like here? Are the differences between the prompt types big with respect to the error? \n\n(h) (421) What does “resp.” mean?\n\n(j) (520) As noted above, the idea that these connections arose by chance during training seems at odds with the finding from Rakotonirina et al., (2023) that automatic prompts can be made to generalize across LMs. How would you square this hypothesis with their results?\n\n(k) (limitations) I would also add that this work was carried out only in English and that future work could explore this in other languages as well.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review, with valuable comments and positive feedbacks. Please find the answers of your questions as follows:\n\n## Statistical analyses\n> [...] some of the analyses could benefit from more careful statistical testing\n\nWe have conducted further statistical analyses and intend to include these findings in the revised paper. We appreciate the reviewer's input, as it has significantly enhanced the clarity of our study. Here is a summary of our statistical investigation (limited here to opt350m):\n\n### Table 1\nWe added a 95% confidence interval (95% CI) with respect to the relations/tasks scores (we first averaged the sample-level scores across templates to get relation-level scores. Then we computed the final average scores and their confidence interval based on these relation-level scores).\n| Type   | Micro         \t| PPL (10^3)    \t| Entropy    \t| Attention dist.   |\n|--------|-------------------|-------------------|----------------|-------------------|\n| Human  | 29.5 [11.5, 65.0] | 0.6 [0.1, 1.9]\t| 5.0 [3.2, 6.0] | 34.4 [29.2, 39.7] |\n| M-disc | 43.3 [17.0, 79.5] | 40.9 [16.0, 95.0] | 4.3 [1.9, 5.7] | 30.0 [27.3, 32.5] |\n| M-cont | 54.9 [20.7, 86.0] | NaN           \t| 2.1 [0.5, 4.3] | 23.2 [21.1, 25.5] |\n\n### Figure 1\nWe provide a bootstrap estimation of the 95% CI (we randomly sample 350 templates 1000 times):\n| Type         \t\t | Activation overlap     |\n|-------------------    |--------------------    |\n| Human        \t\t | 65 [63, 67]   \t\t |\n| M-disc       \t\t | 31 [30, 33]   \t\t |\n| M-cont       \t\t | 45 [44, 46]   \t\t |\n| Human vs. M-disc \t | 16 [16, 17]   \t\t |\n| Human vs. M-cont \t | 13 [13, 14]   \t\t |\n| M-disc vs. M-cont     | 15 [15, 15]   \t\t |\n\n### Figure 2\nSame as in Fig1:\n\n| Type         \t\t | Input similarity     |\n|-------------------    |------------------    |\n| Human        \t\t | 96 [96, 96] \t\t |\n| M-disc       \t\t | 90 [90, 90] \t\t |\n| M-cont       \t\t | 85 [84, 86] \t\t |\n| Human vs. M-disc \t | 86 [85, 86] \t\t |\n| Human vs. M-cont \t | 48 [48, 49] \t\t |\n| M-disc vs. M-cont     | 51 [50, 51] \t\t |\n\n### Table 2\nSimilarly, we provide the bootstrap 95% CI of the correlations (again we randomly sample 350 templates 1000 times). We also rearrange the table to make it more clear:\n\n| Type                 \t\t | Input predicts Activation?     | Activation predicts Output?     | Input predicts Output?     |\n|---------------------------    |----------------------------    |-----------------------------    |------------------------    |\n| Between Human and M-disc \t | 0.21 [0.17, 0.25]     \t\t | 0.01 [-0.04, 0.06]     \t\t | 0.11 [-0.03, 0,24]\t\t |\n| Between Human and M-cont \t | 0.06 [0.03, 0,08]     \t\t | -0.04 [-0.07, -0.00]   \t\t | 0.14 [0.04, 0.24] \t\t |\n| Between M-disc and M-cont     | 0.06 [0.04, 0.08]     \t\t | 0.02 [0.00, 0.05]      \t\t | 0.30 [0.19, 0.40] \t\t |\n| Within Human         \t\t | 0.72 [0.68, 0.75]     \t\t | 0.65 [0.59, 0.70]      \t\t | 0.51 [0.43, 0.59] \t\t |\n| Within M-disc        \t\t | 0.84 [0.82, 0.86]     \t\t | 0.51 [0.48, 0.59]      \t\t | 0.52 [0.45, 0.59] \t\t |\n| Within M-cont        \t\t | 0.72 [0.68, 0.75]     \t\t | 0.51 [0.46, 0.58]      \t\t | 0.61 [0.54, 0.68] \t\t |\n\n> (g) As noted above, what does the error of the measurement look like here [Table 2]? Are the differences between the prompt types big with respect to the error?\n\nWe observe that the differences between the prompt types are big with respect to the error, in particular when comparing the correlations measured “within” vs. “between” types.\n\n### Figure 3\n\nWe have incorporated the 95% confidence intervals using the bootstrapping technique. \n\n> It would be good to get a sense of the measurement error for the results reported in Figure 3. In addition, while I do see a slight difference between the human, M-cont and M-disc patterns, these seem relatively weak, compared to the overall pattern which is shared between prompt types, namely that all prompt types clearly have more top neurons in the final layers than, for example, in layers 4, 8, and 12.\n\nAs correctly pointed out by Reviewer ZpBf, while the overall trend of having more highly activated neurons in the final layers for all prompt types is evident, the distinctions between prompt types are somewhat less pronounced. Nonetheless, we wish to emphasize that the **differences remain statistically significant, particularly in the first layers (l00 and l04), and to a lesser extent in the final layer.** In these specified layers, we have statistically demonstrated that machine-generated prompts lead to the activation of more neurons compared to human prompts. We will make appropriate revisions to the paper in light of this insight.\n\n\n## Chance vs. universality?\n> [...] how to square the authors' conclusion—that the special pathways triggered by these prompts arose by chance–with the findings from Rakotonirina et al., (2023), indicating that at least some automatically generated prompts can generalize across language models?\n\nThanks, this is a great point. When we say that the pathways arose “by chance”, **we do not mean that they arose by random happenstance** (e.g., through lucky initialization seeds), but rather that **they came about as an unintentional side effect of pre-training**. We find it fascinating that, as shown by Rakotonirina and others, very opaque token sequences trigger the same response across different language models. We conjecture that this is due to similarities in architecture (all popular language models have rather similar Transformer backbones) and training data (e.g., parts of the Pile corpus and other Web-derived data are used to train many popular models). Suppose for example that a special pathway arises in response to some quirks in the distribution of the input tokens. Then, two Transformer-based language models exposed to similar data might develop similar pathways. We realize that this is just a wild conjecture. We will discuss it, as such, in the paper.\n\n\n## Micro accuracy\n> (a) (209) Is “micro-accuracy” a term used in the literature? Is there a reference for this?\n\nThe term “micro-accuracy” is from the literature (e.g. in  Zhong et al., 2021). We will make it clearer when revising the paper.\n\n\n## Template\n> (b) (216) Can you provide an example of “[subject][template]”, as I’m not exactly sure what this looks like. Would it be something like “Lavoisier Island continent of Antarctica”?\n\nAn example of “[subject][template]” is:\nHuman prompt: *“Lavoisier Island belongs to the continent of”*\nM-disc prompt: *“Lavoisier Island Antarctica submissions Telegraph Travels in”*\n\n\n## Entropy\n> (c) (222) How is the entropy measured?\n\nThe entropy score consists in the average Shannon entropy of the output probability vector computed across all samples of the evaluation set. We will clarify.\n\n\n## Attention distribution\n(d) (230) What does this value range over? What counts as “low” and “high” values for this measurement? (More broadly, this section felt a little telegraphic. I would appreciate a little more detail in the final version here.)\n\nWe quantified the distribution of attention using a method introduced by Ramsauer et al. (2021) (see Figure A.3). This value ranges from 0 to 100. Intuitively, given a row of an attention map of a transformer layer, **it corresponds to the number (in %) of attention values you have to sum to reach 90% of the total attention**. Because attention values are normalized, if the attention is flat then the score will be 90. In contrast, if all the attention is focused on one token, then the score will be close to 1. When we spoke of low and high values, we simply wanted to state that our score decreases as the attention distribution becomes more peaky. We will clarify this.\n\n## Large LMs' machine prompts convergence\n> (e) (335) It is an interesting result that the prompts converge to similar input representations for larger LMs. Any speculations about why this is going on?\n\nThe paper by Rakotonirina you mentioned above found that better-performing machine prompts are, to some degree, more “semantically transparent”, and thus presumably more similar to human prompts. As machine prompt accuracy increases for the larger model (first row of Table 1), it’s possible we are recording a similar phenomenon: **“better” machine prompts that are more transparent, and thus more similar to human prompts**. If this is the case, **still the approximation to human prompts must hit a ceiling at a certain point, as shown by the persistence of effective unnatural language even for ChatGPT-sized models** (e.g.: https://arxiv.org/abs/2307.12507, https://llm-attacks.org/). We will explicitly discuss this in the paper.\n\n\n## Writting\n\n> (f) (354) Again, this section felt a little telegraphic. I would appreciate more detail, examples, and motivation for this analysis.\n\nWe will clarify and reformulate when revising.\n\n\n## Resp.\n> (h) (421) What does “resp.” mean?\n\nWe thought “resp.” is an abbreviation for respectively, but we will avoid it.\n\n\n## Explore other languages\n> (k) (limitations) I would also add that this work was carried out only in English and that future work could explore this in other languages as well.\n\nThat’s a very interesting research direction, and we will add it to current limitations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(a) Very well written and extremely clear to follow\n\n(b) The authors provide a number of different analyses that point towards the same results, e.g. the two analyses presented in 5.1 and 5.2.\n\n(c) The paper is on a timely topic and does a good job of situating itself within previous work on the subject (at least to my knowledge)", "reasons_to_reject": "(a) I think some of the analyses could benefit from more careful statistical testing, in particular the analysis in 5.1. It would be good to get a sense of the measurement error for the results reported in Figure 3. In addition, while I do see a slight difference between the human, M-cont and M-disc patterns, these seem relatively weak, compared to the overall pattern which is shared between prompt types, namely that all prompt types clearly have more top neurons in the final layers than, for example, in layers 4, 8, and 12.\n\n(b) One big question I have is how to square the authors' conclusion—that the special pathways triggered by these prompts arose by chance–with the findings from Rakotonirina et al., (2023), indicating that at least some automatically generated prompts can generalize across language models? Given this result, I am not sanguine about this conclusion, and I think the paper could benefit from more discussion on this point.", "questions_to_authors": "(a) (209) Is “micro-accuracy” a term used in the literature? Is there a reference for this? Otherwise I would just use accuracy and qualify what you mean.\n\n(b) (216) Can you provide an example of “[subject][template]”, as I’m not exactly sure what this looks like. Would it be something like “Lavoisier Island continent of Antarctica” ?\n\n(c) (222) How is the entropy measured?\n\n(d) (230) What does this value range over? What counts as “low” and “high” values for this measurement? (More broadly, this section felt a little telegraphic. I would appreciate a little more detail in the final version here.)\n\n(e) (335) It is an interesting result that the prompts converge to similar input representations for larger LMs. Any speculations about why this is going on?\n\n(f) (354) Again, this section felt a little telegraphic. I would appreciate more detail, examples, and motivation for this analysis.\n\n(g) As noted above, what does the error of the measurement look like here? Are the differences between the prompt types big with respect to the error? \n\n(h) (421) What does “resp.” mean?\n\n(j) (520) As noted above, the idea that these connections arose by chance during training seems at odds with the finding from Rakotonirina et al., (2023) that automatic prompts can be made to generalize across LMs. How would you square this hypothesis with their results?\n\n(k) (limitations) I would also add that this work was carried out only in English and that future work could explore this in other languages as well.\n", "paper_topic_and_main_contributions": "Summary: This paper compares how LMs process human generated vs. automatically generated prompts. They build on prior work that has demonstrated the efficacy of automatically generated prompts to investigate how the two types are processed differentially in two LMs. They show that automatically generated prompts tend to be better calibrated than human prompts and activate a different set of (more language-oriented) neurons than human prompts. Overall, I thought this paper was extremely clearly written. While its results feel a bit like “first steps” on a research topic, I believe that they will be of interest to the community and am therefore recommending this paper for acceptance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "(a) (294) “two order of magnitude” → “two orders of magnitude”\n\n(b) (392) “not only we” → “not only do we”"}, "extra_text": {}}, {"review_id": "2kX6RYVgGp", "reviewer": "Reviewer_oqtz", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review, with valuable comments and positive feedbacks. Please find the answers of your questions as follows:\n\n## Statistical analyses\n\n> I think this work would be more convincing if readers had a clearer idea about the distributional properties of the statistics being reported, in order to form intuitions about claimed differences.\n\nWe appreciate the thoughtful insights shared by oqtz, highlighting the importance of incorporating uncertainty quantification techniques. To address this concern, **we will be appending a 95% confidence interval (95% CI) to all the scores in the paper, achieved by considering variability across relations/tasks or bootstrapping**. It's important to emphasize that even when looking at confidence intervals obtained in this way, **the reported results maintain their statistical significance**. As an illustration, below are the updated outcomes for opt350m:\n\n### Table 1 \nWe added a 95% CI with regards to the relations scores (we first averaged the sample-level scores across templates to get relation-level scores. Then we computed the final average scores and their confidence interval based on these relation-level scores).\n| Type   | Micro         \t| PPL (10^3)    \t| Entropy    \t| Attention dist.   |\n|--------|-------------------|-------------------|----------------|-------------------|\n| Human  | 29.5 [11.5, 65.0] | 0.6 [0.1, 1.9]\t| 5.0 [3.2, 6.0] | 34.4 [29.2, 39.7] |\n| M-disc | 43.3 [17.0, 79.5] | 40.9 [16.0, 95.0] | 4.3 [1.9, 5.7] | 30.0 [27.3, 32.5] |\n| M-cont | 54.9 [20.7, 86.0] | NaN           \t| 2.1 [0.5, 4.3] | 23.2 [21.1, 25.5] |\n\n### Figure 1\nWe provide a bootstrap estimation of the 95% CI (we randomly sample 350 templates 1000 times):\n| Type         \t\t | Activation overlap     |\n|-------------------    |--------------------    |\n| Human        \t\t | 65 [63, 67]   \t\t |\n| M-disc       \t\t | 31 [30, 33]   \t\t |\n| M-cont       \t\t | 45 [44, 46]   \t\t |\n| Human vs. M-disc \t | 16 [16, 17]   \t\t |\n| Human vs. M-cont \t | 13 [13, 14]   \t\t |\n| M-disc vs. M-cont     | 15 [15, 15]   \t\t |\n\n### Figure 2\nSame as in Fig1:\n\n| Type         \t\t | Input similarity     |\n|-------------------    |------------------    |\n| Human        \t\t | 96 [96, 96] \t\t |\n| M-disc       \t\t | 90 [90, 90] \t\t |\n| M-cont       \t\t | 85 [84, 86] \t\t |\n| Human vs. M-disc \t | 86 [85, 86] \t\t |\n| Human vs. M-cont \t | 48 [48, 49] \t\t |\n| M-disc vs. M-cont     | 51 [50, 51] \t\t |\n\n### Table 2\nSimilarly, we provide the bootstrap 95% CI of the correlations (again we randomly sample 350 templates 1000 times). We also rearrange the table to make it more clear:\n\n| Type                 \t\t | Input predicts Activation?     | Activation predicts Output?     | Input predicts Output?     |\n|---------------------------    |----------------------------    |-----------------------------    |------------------------    |\n| Between Human and M-disc \t | 0.21 [0.17, 0.25]     \t\t | 0.01 [-0.04, 0.06]     \t\t | 0.11 [-0.03, 0,24]\t\t |\n| Between Human and M-cont \t | 0.06 [0.03, 0,08]     \t\t | -0.04 [-0.07, -0.00]   \t\t | 0.14 [0.04, 0.24] \t\t |\n| Between M-disc and M-cont     | 0.06 [0.04, 0.08]     \t\t | 0.02 [0.00, 0.05]      \t\t | 0.30 [0.19, 0.40] \t\t |\n| Within Human         \t\t | 0.72 [0.68, 0.75]     \t\t | 0.65 [0.59, 0.70]      \t\t | 0.51 [0.43, 0.59] \t\t |\n| Within M-disc        \t\t | 0.84 [0.82, 0.86]     \t\t | 0.51 [0.48, 0.59]      \t\t | 0.52 [0.45, 0.59] \t\t |\n| Within M-cont        \t\t | 0.72 [0.68, 0.75]     \t\t | 0.51 [0.46, 0.58]      \t\t | 0.61 [0.54, 0.68] \t\t |\n\n\n### Figure 3\nWe have incorporated the 95% confidence intervals using the same bootstrapping technique.\n\n\n## Discretized overlap\n> Why discretize a continuous space a way that seems designed to exaggerate subtle continuous differences? It seems a lot more straightforward to use simple correlation in place of the current factorization into (discrete) activation overlap and (continuous) similarity, which seem like two different views on the same data.\n\nThanks for the suggestion. We have devised the discretized measure because 1) we were concerned that possible patterns would have been hidden by **low-frequency noise when correlating the whole activation vectors**; 2) the discrete measure is **more straightforward to compute across layers** (so that we have a single overlap score, instead of 24); and 3) the method returns **a discrete set of “active” units that we can then further analyze** as we do in Section 5. We agree however that the measure has in turn the issues that you mention, and we will explore the direct correlation approach you suggest.\n\n\n## Calibration\n> The authors make an assumption that better-calibrated models will have lower output entropy. This assumption is false in general [...].\n\nThanks for pointing this out. We will remove the direct reference to calibration. Our reasoning is as follows: in general, machine prompts are more likely to trigger the correct output and, at the same time, they have lower output entropy. So, while this might not be true for all specific cases, the global trends suggest that machine prompts tend to produce correct answers with more confidence.\n\n## Table 3 is unclear\n\n> It's hard to figure out what is being correlated with what in Table 3. \n\n*Unless we are mistaken, it seems that there is a typos, and reviewer oqtz is referring to Table 2, and not Table 3.*\n\nWe agree on Table 2’s lack of clarity. The first column describes the data used to compute the correlations: human and M-disc prompts, human and M-cont prompts, [...], only human prompts, only M-disc prompts, etc. Then, the next columns provide the correlation between two measures (among input similarity, activation overlap and output agreement). For instance, when considering the set of human and M-disc prompts, we measure a correlation of 0.21 between the input similarity and the activation overlap. The same correlation is 0.72 when considering the set of human prompts only.\n\nHere is an improved version of the Table:\n| Type                 \t\t | Input predicts Activation?     | Activation predicts Output?     | Input predicts Output?     |\n|---------------------------    |----------------------------    |-----------------------------    |------------------------    |\n| Between Human and M-disc \t | 0.21 [0.17, 0.25]     \t\t | 0.01 [-0.04, 0.06]     \t\t | 0.11 [-0.03, 0,24]\t\t |\n| Between Human and M-cont \t | 0.06 [0.03, 0,08]     \t\t | -0.04 [-0.07, -0.00]   \t\t | 0.14 [0.04, 0.24] \t\t |\n| Between M-disc and M-cont     | 0.06 [0.04, 0.08]     \t\t | 0.02 [0.00, 0.05]      \t\t | 0.30 [0.19, 0.40] \t\t |\n| Within Human         \t\t | 0.72 [0.68, 0.75]     \t\t | 0.65 [0.59, 0.70]      \t\t | 0.51 [0.43, 0.59] \t\t |\n| Within M-disc        \t\t | 0.84 [0.82, 0.86]     \t\t | 0.51 [0.48, 0.59]      \t\t | 0.52 [0.45, 0.59] \t\t |\n| Within M-cont        \t\t | 0.72 [0.68, 0.75]     \t\t | 0.51 [0.46, 0.58]      \t\t | 0.61 [0.54, 0.68] \t\t |\n\n We would be grateful to get the reviewer’s opinion on this new version of the table, and if needed, some suggestions for improvement.\n\n\n## Terminology\n> I think the so-called \"human\" prompts are mis-labeled [...]. I think \"rule-based\" or \"human-readable\" would be a clearer label.\n\nThe prompt templates were taken from a number of sources in the literature, as originally collated in the ParaRel resource. The original templates were partially hand-written, partially harvested from the Web, partially manually or automatically paraphrased. One of the human authors of the current paper went through the whole list, filtering out templates that were not well-formed or semantically appropriate, optimizing them for causal model prediction, correcting or paraphrasing other templates, and coming up with further templates by hand. So, **while some of the templates were originally generated by semi-automated procedures, there are also plenty of templates that were hand-generated, and all prompts have been validated by multiple human judges**. In this respect, we think it’s fair to label them as “human” prompts. Note that in principle the templates generated by AutoPrompt or other algorithmic procedures could also be “human-readable” (although that’s not, for the time being, the case).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Clearly written\n2. Detailed, well-motivated analyses\n3. Potentially fundamental insights (substantial separation in the network-internal activation pathways between human-readable vs machine-optimized prompts)\n4. Detailed exposition of potential weaknesses, strong grounding of this work in the ethical implications of LM interpretability", "reasons_to_reject": "1. This work participates in a growing movement to treat LMs not only as engineering tools but as objects of scientific inquiry in their own right. As a result, questions shift from \"which LM does better at X?\" to \"why/how do LMs do X?\" This kind of work is much-needed and I strongly support it. However, I think quantitative approaches that are standard in NLP and largely used in this study (esp., leaderboard-style reporting of numerical differences between models) don't meet quantitative standards for scientific research, which place a high priority on uncertainty quantification in the form of e.g., confidence intervals, posterior distributions, hypothesis tests, etc. There are a variety of reasonable tools that could be used, but the overall goal is to distinguish the statistics of the sample from the inferences about the population, and to make explicit the reasoning by which population inferences are made from these statistics. For example, the authors say based on Table 1 that the attention distribution statistic (number of timepoints needed to cover 90% of the softmax distribution) is smaller in M-disc than human prompts. In a sense, this is uncontroversially true because the number is smaller (30 vs 34 using OPT-350m). But is this difference an inherent property of the M-disc vs human prompting strategies (that would appear again with high probability if this experiment were repeated), or is it an artifact of the particular sample of prompts used here? Without uncertainty quantification, it's hard to know whether a difference of 4 for the attention statistic big or small, meaningful or not, etc. I think this work would be more convincing if readers had a clearer idea about the distributional properties of the statistics being reported, in order to form intuitions about claimed differences.\n\n2. The activation overlap and similarity analyses are less independent than they seem: lower cosine similarity implies lower (standardized) Pearson correlation, which in turn implies lower activation overlap (due to the lower likelihood that a unit will co-activate across conditions). In addition, the activation overlap measure introduces an experimenter degree of freedom: it requires an arbitrary frequency threshold for positive activation and selects neurons for analysis based on it in a binary manner. Why discretize a continuous space a way that seems designed to exaggerate subtle continuous differences? It seems a lot more straightforward to use simple correlation in place of the current factorization into (discrete) activation overlap and (continuous) similarity, which seem like two different views on the same data.\n\n3. The authors make an assumption that better-calibrated models will have lower output entropy. This assumption is false in general, since without direct evaluation (i.e., of the frequency with which a prediction made with model-assigned probability *p* is correct) one cannot know whether a given model should have higher or lower output entropy. Maybe the authors have reason to believe this is true *for this specific evaluation*? If so, the reasons should be given. If not, I think they should not invoke the notion of calibration, which seems like a red herring here.\n\n4. It's hard to figure out what is being correlated with what in Table 3. After a lot of puzzling I think I finally got it: the correlation is between two *statistics* (activation overlap, input similarity, and output agreement) computed either within or between prompt types. So the Activation-Input statistic for the human vs. M-disc condition represents the degree to which the activation overlap statistic using human promps is correlated with the input-similarity statistic using M-disc prompts (?). I'm still not sure this is right, which I think speaks to deeper presentational problems. How the numbers in Table 3 were computed and what they mean is unclear, and more explanation is needed in the text and caption.\n\n5. I think the so-called \"human\" prompts are mis-labeled---to me, this label implies that human annotators wrote prompts for each training/testing instance, when in reality (if I understand the methods) these prompts were generated from the training/test instances using a hand-made codebook. I think \"rule-based\" or \"human-readable\" would be a clearer label.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This is an exploratory paper that attempts to characterize in detail the empirical properties of language models' (LMs) processing of human-readable vs. machine-optimized prompts. Diverse analyses suggest that machine-optimized prompts propagate through the network in qualitatively different ways than human-readable prompts that more closely resemble the models' training inputs: machine-optimized prompts have much higher perplexity than human-readable prompts, show little activation overlap with human-readable prompts, have peakier attention distributions than human-readable prompts, and have lower output entropy than human-readable prompts. Although this study is not designed to illuminate the causes of these differences, results are at least consistent with the possibility that prompt optimization works by finding and exploiting novel activation pathways through the network that work well for the target task, a finding that has also been suggested by recent closely related work (Khashabi et al 22).\n\nI think this is a carefully executed study and a well-written paper and I support publication. Some readers may find the lack of a causal hypothesis disappointing, but I think for many this will mostly be a hindsight effect thanks to the very clear problem statement in the paper itself: the authors make a strong case that it's worthwhile to study the mechanisms by which prompt engineering works and that too few researchers are currently tackling this question. The detailed exploratory characterization of the empirical phenomenon that this paper provides will likely be an influential early step on the path toward deeper understanding of this question.\n\nI do have some technical and presentational concerns that I hope the authors will consider addressing in revisions.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the behavior of models on human-constructed vs machine generated prompts, finding that machine generated prompts are processed in a qualitatively different way compared to human-constructed prompts. The findings in this study are very intuitive, and all reviewers appreciated the analysis in the paper.
Decision: Accept-Findings