Prompting with Pseudo-Code Instructions
Authors: Mayank Mishra
Prince Kumar
Riyaz Ahmad Bhat
Rudra Murthy
Danish Contractor
Srikanth G. Tamilselvam
Venue: EMNLP 2023 Main (2023)
Keywords: Instruction Finetuning
Pseudo-Code Instructions
Large Language Model
TL;DR: Pseudo-Code Instructions prompting and dataset

Abstract: Prompting with natural language instructions has recently emerged as a popular method of harnessing the capabilities of large language models (LLM). Given the inherent ambiguity present in natural language, it is intuitive to consider the possible advantages of prompting with less ambiguous prompt styles, like pseudo-code.

In this paper, we explore if prompting via pseudo-code instructions helps improve the performance of pre-trained language models. We manually create a dataset of pseudo-code prompts for 132 different tasks spanning classification, QA, and generative language tasks, sourced from the Super-NaturalInstructions dataset. Using these prompts along with their counterparts in natural language, we study their performance on two LLM families - BLOOM, CodeGen. Our experiments show that using pseudo-code instructions leads to better results, with an average increase (absolute) of 7-16 points in F1 scores for classification tasks and an improvement (relative) of 12-38% in aggregate ROUGE-L scores across all tasks. We include detailed ablation studies which indicate that code comments, docstrings, and the structural clues encoded in pseudo-code all contribute towards the improvement in performance.

To the best of our knowledge, our work is the first to demonstrate how pseudo-code prompts can be helpful in improving the performance of pre-trained LMs.

Prompting with Pseudo-Code Instructions
Mayank Mishra; Prince Kumar; Riyaz Bhat,
Rudra Murthy V, Danish Contractor, Srikanth Tamilselvam
IBM Research AI
Imayank.mishee I kumbrr 12, diyassb www.aaaa.aiisi..ccttttooooo contractor @ibm..om, (rmrrthyy,srikke wtwptir..ssiiiah..aaaa.... (amintteaaliilllllami @ein.ibm.com
Abstract
Listing 1 An example pseudo-code instruction for the task from Wang et al. (2022b). A successful model expected to use the provided pseudo-code instructions and output responses to a pool of eval- uation instances.
Prompting with natural language instructions has recently emerged as popular method of harnessing the capabilities of large language models (LLM). Given the inherent ambiguity intuitive to
def interrat_senttmene (sentencc:: str) -> str: """For the given sentence, the task is to predict the sentiment. For positive sentiment return "positive" else return "negative".
present natural language, consider the possible advantages of prompting with less ambiguous prompt styles, like pseudo- code.
this paper, we explore if prompting via pseudo-code instructions helps improve the performance of pre-trained language models. We manually create dataset! of pseudo-code prompts for 132 different tasks spanning classi- fication, QA, and generative language tasks, sourced from the huperrNaturallnsstuutioon dataset (Wang et al., 2022b). Using these prompts along with their counterparts in natural language, we study their performance two LLM families BLOOM (Scao et al., 2023), CodeGen (Nijkamp al., 2023). Our experi- ments show that using pseudo-code instructions leads to better results, with an average increase (absolute) of 7-16 points in scores for classi- fication tasks and an improvement (relative) of 12-38% in aggregate ROUGE-L scores across all tasks. We include detailed ablation studies which indicate that code comments, docstrings, and the structural clues encoded pseudo-code all contribute towards the improvement in per- formance. To the best of our knowledge, our work first to demonstrate how pseudo- code prompts can be helpful in improving the performance of pre-trained LMs.
Parameters: sentence (str): input sentence Returns:
str: sentiment of the input
predict the sentiment
if sentiment_is lerpeen__spposttiieeeeee positive(sennence)) "positive"
16
else:
"negative"
18
19
20
"that has charmingly bourbon air."
21
)
to help improve the ability of LMs to follow in- structions and performance on unseen tasks (Wei et al., 2022a; Wang et al., 2022b).
However, natural language instructions can be ambiguous and under-specified, and therefore have multiple interpretations including detailed in- structions may not always be beneficial, as can add to the complexity of reasoning for models. This has led to the growing body of work around 'prompt-engineering' where specialized prompt- ing strategies are developed for different domains and task types (7hao et al., 2021; Reynolds and McDonell, 2021; Arora et al., 2023; Liu et al., 2023; Zamfirescu-Pereira al., 2023). In ad- dition. inference-time prompting strategies that specifically aid multi-step reasoning have also been found to be helpful e.g. the inclusion of chain- of-thought reasoning in few-shot settings results
1 Introduction
Prompting with natural language instructions has recently emerged as popular method of harness- ing the capabilities of large language models. In addition to fine-tuning, models are often fine-tuned using instructions on large collection of datasets
"Equal contribution
Coode and dataset available at https://gitthul com/ mayank31 1ayankk1398//ae 39////99////////////
http://pcooodee--eeeeepepeeeee-e code- instructions

---

in improved performance over standard prompts (Wei al., 2022b), the infamous "Let's think step- by-step"-promp -prompt for boosting O-shot performance (Kojima et al., 2022).
We compare the performance of both styles of prompts on classification tasks, QA tasks, as well as a mix of other language generation tasks. Our experiments indicate that prompting with pseudo- code instructions indeed helps, and they result in an absolute gain of 7-16 points in FI scores on clas- sification tasks, and 12-38% relative improvement in aggregate ROUGE-L scores across all tasks.
Algorithm Attention Block
1:
function TRANSFORMERS. ATTENTION, ATTENTION_BLOCKQ K, V)
2: Input: Q, and input matricss
Output: The curput the attention block.
4:
scores +- Q K
attention weights 4- softmast (scores)
Contributions: In summary, our paper makes the following contributions: (i) We release a dataset of 132 pseudo-code prompts spanning 28 differ- ent task types; (ii) Through a series of detailed experiments on two publicly available open-access LLM families, we demonstrate how prompting with pseudo-code instructions results in marked improvement in performance over prompting with natural language instructions; (iii) We include de- tailed ablation studies indicating that code com- ments, doestrings, and the structural clues encoded in pscudo-code all contribute towards the improve- ment in performance.
6:
weightted yintues +- oftention woighhtt
output t- De weighted
return outyutt
end function
Given the inherent ambiguity present in natural language, it is intuitive to consider the advantages of prompting with less ambiguous prompt styles, such the use of pseudo-code. Pseudo-code is an informal set of code-like constructs, which tend to be easy to interpret for humans but are not neces- sarily compilable/executaale. They are often used to express complex ideas, processes, and flows for example, Algorithm I expresses a summarized version of what happens within a Multi-Hearl At- tention block (Vaswani et al., 2017) in pseudo-code. Arguably, expressing the same ideas in natural lan- guage could result in ambiguity and would perhaps require detailed text for clarity, which adds to the complexity.
To the best of our knowledge, our work is the first to demonstrate how pseudo-code prompts can be helpful in improving the performance of pre- trained LMs. Our findings only emphasize the significance of leveraging pseudo-code for prompt- ing but also shed light on the specific elements within pseudo-code that contribute to the observed improvements.
In light of recent successes in NLP tasks achieved by code models (Madaan et al., 2022; Zhang et al., 2023a,b), this study aims to exam- ine the efficacy of using pseudo-code instructions for prompting
Related Work
means of enhancing model per- formance. This study is driven by the hypothesis that using poeudo-code as prompts could offer a natural advantage to models in NLP tasks, owing to the concise and clearer expression of ideas in pseudo-code. To test the hypothesis that prompt- ing large language models with pseudo-code in- stead of natural language data could be helpful, we created pseudo-code prompts² for 132 different tasks spanning 28 distinct task types, sourced from the SuperrNaturalInsttutions dataset (Wang et al., 2022b) (see Listing 1 for an example). Using these prompts along with their counterparts from natural language, we study their performance on two LLM families: BLOOM (Scao et al.. 2023) and Code- Gen (Nijkamp et al., 2023). Both LLM families have been trained on natural language well as code data.
Finetuning large language models on instruction datasets enhance their performance and even their ability to generalize unseen tasks (Wei al., 2021; Chung al., 2022). Many aspects of instruc- tion finetuning such as the number of tasks, model size, and finetuning on -oain-of-thought data have been found to be useful (Chung et al., 2022). Con- sequently, significant efforts have been invested in manually creating instruction datasets, as well as using existing generative models to train and eval- uate language models (Mishra et al., 2021; Bach et al., 2022: Wang et al., 2022b.a). The instructions available in instruction tuning datasets are mostly in natural language, but have been applied for both natural language tasks and programming tasks. But alternatives to natural language instructions such as programming language code, pseudo-code, sym- bols (MacCartney and Manning, 2007) etc. have
"The pscudo-cnde instructions for cach of these tasks were created by the authors of this paper.
'In the rest of the paper, we use the words 'pscudo-code' and 'code' interchangeably when referring to prompts.

---

not been thoroughly explored even for program- ming tasks. Compared natural language, code or pseudo-code has less ambiguity due to its inherent nature of using functions or steps that contribute towards accomplishing a task. This makes them a natural choice for specifying instructions. Recently, few works (MarvinAl: Madaan et al., 2022; Zhang et al., 2023a,b) have explored code and pseudo- code inputs. Unlike contemporaneous work by Zhang et al. (2023a) we find that pseudo-code in- structions indeed provide better performance over NL instructions on a wide varicty of tasks.
function. An example docstring for the sentiment classification task presented in line numbers 2 13 Listing
This includes the bulk of the pseudo-code instruction describing how solve the particular task. To the extent possible, the function definitions do not leave out any infor- mation contained in the docstring. Pseudo-code in the function definition are written as sub-task functions. These sub-task functions are usually not defined and often use descriptive names, arguments and variables. We include in-line comments indicat- ing what is accomplished by the sub-task function and the role of the arguments if required. We some- times also define secondary sub-task functions if it requires additional details or if the descriptive func- tion name may not be adequate to specify the goal of the sub-task function. We assume the availabil- ity of basic helper functions such as concat_sta search etc., and do not include any import state- ments.
Function Definition:
3 Dataset
The Super-Naturalllnstructton dataset (Wang et al.. 2022b) comprises 1, 616 diverse NLP tasks, and each task contains the task instruction, positive/neg- ative examples, and instances. We sampled a mix- ture of 132 tasks that did not require multilingual capabilitics and re-wrote instructions for subset of this dataset using Python constructs. Note that we borrow Python constructs only to express our prompts in pseudo-code and our prompts do not result in executable Python code. Further, we do not include any additional steps/instructions that were not present in the original natural language instructions.
Line numbers 14 to 17 present function defini- tion for sentiment classification task. The function calls sent i _osis tive sub-task function which checks if the sentiment of the given sentence is positive or not. This function is not explicitly defined in the instruction.
All task instructions follow the schema as de- scribed in Listing 1. The schema consists of the following elements.
Pre- proceescesoo:: Since the pseudo-code instruc- tions expect inputs as arguments, we need to parse the inputs provided in the Super- Naturallnstructions dataset (Wang et al., 2022b) (which provides pre-formatted inputs). For each pseudo-code instruction, we also include an ex- ecutable python pre-processor which used for parsing the input.
This defines the prototype of the main pscudo-code function. The function names are deseriptive and summarize the task to be performed. They also include all variables passed as input along with their data types and return type. We follow the PEP 8+ style guidelines for writing the pseudo-code and use strongly typed prototypes. We avoid declaring global variables whenever pos- method. To
Function Prototype:
3.1 Dataset Statistics
sible and pass them as arguments the extent possible, we also avoid the use of classes and enumerations. Line number 1 in Listing pro- vides an example function prototype for a senti- ment classification task.
We created instructions for 132 tasks that have in structions and input/output pairs in English lan- guage. We group the tasks into three classes: Clas- sification Tasks (Table 1). QA tasks (Table 2) and other language generation tasks (Table 3). These tasks cover total of 28 different categories and span 72 unique datasets. For each task sample 1000 instances for evaluation.
DocString: The doestring provides detailed in- structions on the task be performed in natural language. Often, this is a paraphrased version of the original natural language instruction. The doc- string ends with a list of parameters (with their types) being passed and the return type from the
Evaluation
In order to study instruction specification via pseudo-code results improved performance over
htttps://ppps.pppp wwwpto.ogg.pppppoeeppppppppppppp /000////

---

baseline NL English instructions, we choose to ex- periment with BLOOM (Scao al., 2023), Code- Gen (Nijkamp et al., 2023) models. Our choice of models is motivated by the fact that these models have not been instruction-finetumme Instructions dataset. In addition, they have both been trained on code and natural language data.
Extractive QA
on the Natural
Generative QA
16
Answer Classifiation
Question Undervanding
Senciment Analysi
Tisk Categary
Dataseis
Tooic Langsage Detection
Extractive QA
RUPES (Lin et 20192, Odd Out (Sunovsky and Hop- kins. 2018). SQUADL.1 (Ragpurkrr 2016). Symbhric 2022b), MCSengt Sedoviiss 2018) MWSC (MaConn
Testual Entailiment
et al. 2018), PICO OPUS (Tademann, 2013), CoQA (Reddy 2019) Queee (Daiigi d 2019), McTaon Tha and Roth, 2019%, DROP (Dia al. 2019), MuliRe (Khashai sh., 201.9 83, PIQA (Bisk et al, 2000 Symbhmn (Wing i at 202263 BREAK (Woltonn etal. 2000, Nataral Questoon Ukwarkmwsti ot 2019) AmbigQA (Min al. 2000 CrQA (Roddy all 20191 TriviQA Uusii 2017) 2018), WinoGmade (Sak- apucti 2021), MuluNLI (Willamss al., 2018), ReCoRD (Zhung al., 2015), MMMLC (Handryels at sl., 2021)
12
Test Categorizziion
Generative QA
Test Matching
Otherss
MCO
Essemial. QuaRel ('ufjond
Tink Category
Datasets
Anower Insaiication
Malne (Krrndabl ot al. X0185 McTho (Ben Zhoz and Roth. 2019), TWEETQA (Xinng st, 2019)
Table Collection of QA tasks used our work
Answer Veniication
MultiKU (Khashali al. 2018)
Clessifeation
ATOMIC
2019)
Coecfertnce Selection
Numeric Fusod Head (Blazar Goldberg, 2019)
Disloguc Selection
SPOLIN (Cho May, 20200, DSTC3 (Hondess
at al. 2014)
Grammr Ence Detection
CoL.A (Wantall et 2019)
List Operation
Inieni Idensiiiation
Daily Dialog (Li at. 201
Irony Desection
SemBval2008 8 Dak3 (Ván Hos at 2015)
Opticn Gencration
Linguistic Chasification
Semtiaal (Conneau 20226)
Kiela, 2018)
Primc Nunber Classifcation
Systhetic (Wang
Parapianacigg
Program Esscution
Synthtii (Wang
20226)
Question Understanding
MeTo (Bon Zhou
DROP (uua
Question Generation
at al., 2019), TRFC and Roth, 2007) DREAM (Sun et al. 2019). FreehaseOA (Jinng et al. 2019) CODA-19 (Funng et al. 2000)
Rowniting
Section Chesifizeion
11
Sontiment Analysis
The Muttlingua Amazon Reviews Corpus (Koung at al. 2020) Sencmmnt 440 (Go al, 2009, SST-2 (Socher 2013), PerSent (Baston Resiow Polanty 2020), Semtimet (Steng and Uthus, 2020)
Misc.
2020,
PEC (Zhong
Text Citegorizatinn
MuleNLI (Willaam et al, 2018) DDO (Darmus and Curde, 2019), Seminaal 2020 Toak? (Hossai at al., 2020), Semples Lurri et al., 2021)
Tesk
Datasets
Tex: Matching
AFS (Misn et al., 2015), PAWS (Zhang et 2019) McTaco (Ben Zhou and Roth 2019)
List Operation
CaNaaa al., 2018), Systhetic (Tiedemann, 2012),
Text Quality Clavsification
Youtube C-piiun Cumections (Dak71mmily) aNLi (Nie et al., 2020, ASSBT (Alva Manchego th. 2020, RDC Sumte (Mintalazsshh et al. 2017)
Texmal Entailment
MuliNLI (Williams al. 2018) SNLI (Bow- 2015),
Uption Generation
(Cambum al., 2018). Defeasibee (Kudinger (Sap at al., 2019)
2029) ATOMIC
Paraphrasing
ZEST (Weller et al, 2020), PARANMI 5OM (Wieting and Gimpel, 2018)
Todic Language Detection
CAD (Nidgen et al., 2010). ligsaw tejodams et al., 2019), Hote Speech Ocfeasive (Davidson et di,, 2017)
Question Generation
CommorQA (Hoang al. 2019), WinGGrnnd (Sabaguchi 2021), ROPES (Lán al., 2019b) SQUADE, jputkar ot 2016), SmatggyOA (Gesa 2021. BoeQ (Clark al. 2019) ColA (Reddy et al., 2019), 2RE (Levy al. 2017)
Winng Candidate Generation
McTaco (Ben Zhou and Roth, 2019)
SQUAD20 (Rajpukkrr es
Table I: Collection of classification tasks used in our work
Rewnitig
WinoGrande (Sakaguchi 2021) ANL (Ne ital, 2020) ASSET (Aiva Manchigo et 2020), ZEST (Weller et dl. 2020. SNLI (Bowman al., 2013)
DROP
WinGGrune (Sikuguchi dl. 2021), QASC (Khot 2020), tasomuul (Khishabi al.,
201 StoryClote (Mostfunadel et 2016. Country Banade Profix dunset Country Re- gion in World dscarct, Gigawedd (Gmff et al., 2003, GAP (Weheeer et 2018), SPOLNN IN (Cho and May. X000 XI- WiL: (Raganato al. 2020))
2017), ROPES
The BLOOM models are trained on the ROOTS corpus (Laurençon et al., 2022) consisting of 46 natural and 13 programming languages. On the other hand, the CodeGen models are trained on the Pile corpus (Gao et al., 2020), Google's 's publicly available BigQuery and BigPython datascts (Ni- jkamp et al., 2023). The BLOOM models have been trained on a mixture of natural language and code simultaneously. As for the CodeGen mod- cls we utilize, they were initially trained on natu- ral language and subsequently received additional
Table 3: Collection of language generation tasks used in our work
training focused specifically on Python code.
Our choice of models allows us to setup a con- trolled environment where we can study the impact of prompting in natural language and pseudo-code.

---

Most recent instruction- models have either seen the Super-NauralInstuction dataset (Wang et al., 2022b) in some form (Longpre et al., 2023) or they do not have tokenizers that will meaning- fully process code syntax (Raffel et al., 2020), and therefore can not be used in our study. By empiri- cally studying the performance of models on these prompts, we hope to inform future work on train- ing instruction-fune model using pseudo-code instructions.
all outputs by truncating by the newline charac- ter 'An'. Furthermore, the output subjected additional post-processing, including punetuation removal and lower casing.
4.4 Results
Through our experiments we aim to answer the following questions: (i) What is the difference in performance between prompting pre-trained lan- guage and code models with pseudo-code prompts versus natural language prompts? (ii) How does increasing model size affect the efficacy of pseudo- code prompts? (iii To what extent does structured prompting, such as the use of function names, doc- strings, inline comments, and arguments, impact performance on tasks?
4.1 Model Configurations
For all of the experiments conducted in this pa- per, we use BLOOM-3B, BLOOM 7B (Scao et al., 2023), CodeGen-mono 2B, and CodeGen-mono 6B (Nijkamp et al., 2023) models. The inference was performed using A A100 80 GB GPUs. To accelerate the inference of all models, we utilized DeepSpeed- Inference (Aminabadi al., 2022) in fp16, which resulted in average inference throughput im- provement of around 1.7x, compared to the stan- dard HuggingFace (Wolf et al., 2020) inference. We used greedy decoding for all our experiments for reproducibility and restricted generated outputs to 100 tokens. Even for classification tasks, we generate the class labels using auto-regressive de- coding instead of picking the class label with low-
4.4.1 Prompting with Pseudo-code
Table 4 compares the performance of prompting with pseudo-code (referred to as code instructions) and natural language instructions in O-shot settings. Results have been grouped by model family and size.
As can be seen, for all model families and sizes, prompting with pseudo-code results in a significant improvement in performance. The performance on classification tasks is especially notable, for exam- ple, the gains on weighted Fl vary between 7-16 Fl points (absolute). Furthermore, the relative perfor- mance improvement on all other tasks, measured by ROUGE-L, varies between 12-38%. The over- all performance measured by ROUGE-L, ANLS and Exact Match also report similar trends.
est perplexity. This is done because not all class labels can be mapped to a single token for all tasks. This technique evaluating performance classi- fication tasks is often employed when using closed LLMs, such as those behind APIs (eg: OpenAl's GPT4 (OpenAl, 2023), Google's PaLM (Chowdh- ery et al., 2022) etc).
Comparison of CodeGen vs BLOOM Despite most tasks being non-code tasks, CodeGen, a model designed for code applications, outperforms BLOOM models, even when using natural lan- guage instructions (see metrics for 'All Tasks'), )) Similar behavior has been anecdotally reported (Fu and Khot, 2022; Madaan al., 2022), but has pos- sibly not been investigated using as many tasks as presented this paper. Note, however, that using pseudo-code prompts in the code models results better performance than any other prompt-model configuration.
4.2 Metrics
We adopt different metrics for each task-category: measure the performance of classification tasks using micro, macro and weighted Fl scores, and for QA and language generation tasks we use the ROUGE-L metric. We report the ROUGE-L, Exact Match (EM), and ANLS Average Normalized Levenshtein Similarity (Biten et al., 2019) for all tasks.
4.3 Output post-processing
Since the models experiment with have not been fine-tuned for instruction following, they tend generate excess text after the output for the given task. We therefore post-process the outputs to ensure models are not penalized our evalua- tion due to excess generations. We post-process
Performance on QA tasks Interestingly, we find that on QA tasks, the performance of pseudo-code instructions is better than natural-language instruc- tions, when using the CodeGen model. However, this is not the case when using BLOOM.

---

Instruction Formut
Model
Clusdiication Tasks
Tusks
Generation tasks
All Tusks
Macro Fl
Micro Fl
Weighned F1
ROUGE-L
ROUGE-L.
ROUGELL
ANLS
EM
Majority Closs
0.296
0.509
0,362
CodeGien 2B
Code Instructions
0.272
0.417
0.354
0.175
0.317
0.330
0.261
0,202
Instructions
0.306
0.254
CodeGie
Code Instructions
0.311
0.443
0.375
0,201
0.327
0.354
0.283
0.218
NL Instructions
0,052
0.278
0.215
0.132
0.271
0.257
0.187
0.134
BLOOM 3D
Code Instuctions
0.116
0.351
0,288
0.147
0.271
0,279
0.215
0.165
NL Instructions
0.082
0.275
0.214
0.159
0.234
0.250
0.180
0.132
BLOOM
Code Instructions
0.174
0.369
0.285
0.150
0.298
0.297
0.232
0.176
Instructions
0,247
0,203
0.276
0.247
0.122
Table 4: Performance of models when prompted using pseudo-code instructions and natural language instructions in O-shot settings. (i) In each model, prompting with pseudo-code instructions results in much higher performance in almost all the tasks (ii) For each model family, increasing scale helps improve performance (iii) Prompting CodeGen (a model designed for code) results in better performance than BLOOM. (iv) Prompting BLOOM models with Natural Language instructions instead of code-instructions results in higher performance on QA tasks.
CodeGen 6B
BLOOM 7B
Code Instructions
NL Instructions
Cude Instructions
NL Instructions
QA Task
EM
ROUGE-L
ANLS
EM
ROUGE-L
ANLS
EM
ROUGE-L
ANLS
| BM
ROUGE-L
ANLS
Extractive QA
0.140
0.303
0.189
|
0.045
0,188
0.077
0.047
0.184
0.077
| 0.047
0,227
0.086
Generative QA
0,045
0.129
0.068
0,029
0,095
0.045
0.028
0.101
0,042
0.032
0.115
0.047
MCQ
0,213
0,210
0.082
0.106
0.083
0.201
0.197
0.107
0.143
0.108
Table 5: 0-shot performance of CodeGen 6B and BLOOM 7B models on QA tasks from dataset. As can be seen, pseudo-code instructions applied on the CodeGen model results in the best overall performance on all categories QA tasks. However, comparing the performance Natural Language Instructions, we find that it performs marginally better than pseudo-code instructions on non-MCQ QA tasks when using the BLOOM 7B model.
We investigated this further and observed that for most QA tasks, the instructions in pscudo-code are not significantly more detailed or easier to under- stand than natural-language instructions. As an ex- ample, the pseudo-code instruction for answer gen- cration from the SQuAD dataset merely contains the following statement in its function definition: return get_answer ffowwer_rro__rm lrl_ffoooosaasee_ (passage, question) and reflects the details included in the natural instructions.
instructions as compared to other types of QA. The discrepancy in performance between Code- Gen and BL.OOM on QA tasks (see Table 5), could be attributed to the fact that the structure from code prompts could be better leveraged by code models as programming languages and aspects of code syn- tax (structure) are likely to be better represented in a code model such as CodeGen. This brings us to our next question - What is the contribution of structure that may be present in prompts?
We further analysed the results across QA task categories and found that pseudo-code instructions always help with multiple-choice questions (MCQ) tasks (see Table for comparison between Code- Gen 6B and BLOOM 7B). We believe that this is because, understanding the instructions in such tasks may be more involved. For illustration, in- structions in MCQ tasks often include details about how answers are expected eg: "choose the correct option A, B, C", "Select Option 1 Value Option 2 Value 2 ". Depending on the instructions, the models may be required to return options, values, or both which adds a degree complexity to the
4,4.2 Contribution of Structure in prompts
The reasons behind the performance improvement when using pseudo-code prompts are likely be a combination of factors, including the use of descrip- tive function names that convey the function's 's pur- pose (such as get_answer wonswerqquestiinen (question)), a model that can effectively utilize structured information, and structured prompt for task that could further benefit from few-shot examples.
We therefore experiment with different struc- tured prompting styles and report their results Table We study the performance of CodeGen and

---

Model
Instruction Format
Claseiiiation Tasks
QA Ticks
Generation Tasks
All Tasks
Macro F1
Mizu Fl
Wiighted F1
ROUCE-L
ROUCE-L.
ROUGE-1
ANLS
FM
Cade Instractions (i)
0.272
I.417
0.354
0.175
0.317
IL330
0.262
0.202
Function Declaration (0)
0.150
0.079
0.085
0.124
0.253
0.153
0.083
0.043
Codeeiinn 2B
Function Declaration (2)
0.105
012667
0.257
0,185
0.234
0.256
0.188
0.137
Function Invocation
0.253
0,183
0,296
0.251
Geuric Function Insocation
B/044
0.282
0.344
0.167
0.257
01345
Examples (2)
0.081
0.126
0.009
0.017
Carl Instractions
0
0.201
0.327
0.354
0.218
Declaration
0.162
0.273
0.17y
CinleGen
Finction Declaration (2)
0.134
0.196
0.299
0.281
0.212
0.154
Punation Invocation (2)
0.133
1129%
0.269
0,192
0,302
11.775
0.201
0.149
Generic Function Invocation (2)
0.062
0.244
0.215
0.167
0.262
0.239
0.175
0.121
NI. Ecumples (2)
0.000
0,000
0.001
0,102
0.168
01088
0.023
01006
Code Instractions (0)
0.116
0.351
0.288
0,147
0.271
0.279
0.214
0,165
Function Declaration (8)
0.000
0.014
0.016
0.108
0.229
0.116
0.054
0,015
BLOUM
Function Declaration (2)
0.164
0.249
0.225
Fundike
0.227
0.164
0.234
0:215
Function Envocation
0.161
11:003
0.066
Exiples (?)
0.000
0.025
0.031
0,150
0.208
0.122
0.056
0.024
Cade Instractions
0.298
1..997
0L376
Function Declaration (0)
0.004
0.021
0.027
0.111
0.242
0.124
0.058
0.017
BLOOM 7B
Functioon Declaration (2)
BA072
11236
0.227
0,191
0.289
11.257
0.182
0.128
Punction Envocation (2)
D.086
0.244
0.221
0,189
0.256
012200
0.176
0.125
Generia Functionn Insoccation
0.009
0.199
0.178
0,187
0.216
0.232
0.155
0.097
NL. Exumples
11000
01009
0.132
0,182
0.306
0.058
Table 6: Study of structured prompts: Performance of models when prompted using 0-shot pscudo-code instructions, function declaration O-shot and 2-shot settings as well as 2-shot prompting with a 'generic' function name and the of only examples. The number N in the brackets indicates N-shot prompt. (i) Except for the performance on QA tasks, in each model, prompting with pseudo-code instructions results in much higher performance which indicates that detailed instructions are helpful (ii) For each model family, and prompting style, increasing model scale improves performance (iii) As before, prompting model designed for code, CodeGen, results in better performance than BLOOM.
BLOOM with five types of prompts: (i) Pseudo- code instructions, (ii) Prompts that make use of function declaration (declare function name only), (iii structured prompt consisting only task ex- amples in 2-shot settings using the desc-desc function name (iv) structured prompt consisting only of task examples in 2-shot settings using generic function name "func' (v) using the Nat- ural Language examples (without instructions) in 2-shot settings. Details about each prompt have been included in the Appendix.
periment, which showed that code models more capable of exploiting structured prompts. In the case of QA tasks in our dataset, it is worth noting that since the pscudo-code instructions are not as detailed, even utilizing a simpler structured prompt with examples can significantly enhance perfor- mance compared to natural language prompts.
4.4.3 Impact of pseudo-code documentation In this section, we study the contribution of com- ments and doestrings present in pseudo-code instructions towards the improvement in perfor- mance. We first study the performance of pseudo- code prompts with and without the use of doc- strings and code comments.
We make three important observations from Ta- ble First, code-instructions in 0-shot settings con- sistently yield the best overall performance com- pared to other structured prompts. Second, on aver- age, the CodeGen model consistently outperforms BLOOM on all tasks. Lastly, the QA tasks in our dataset, which are relatively easy to express in nat- ural language instructions, also benefit from struc- tured prompts, particularly when prompted with examples.
As can be seen in Table 7, the inclusion of com- ments as well as the docstring in the pseudo-code instruction prompt helps improve performance. This indicates that not only is the structure of the prompts being exploited by the model, the models are also relying on additional helper text present in the documentation. We, therefore, also investigate if the use of these elements from pseudo-code could also benefit natural language instruction prompts.
It can be inferred from these observations that the performance gains resulting from the use of pseudo-code prompts are likely due clearer task instructions, and not just the exploitation of super- fluous patterns from in-context learning. These findings reinforce the results from the previous ex-
The lower half of table studies the performance of natural language prompts with and without the use of pseudo-code comments and docstring. We

---

Model
Instruction Format
Classification Tasks
QA Tasks
Generation Tusks
All Tasks
Macro F1
Miero F1
Weigheed F1
ROUGE-
ROUGER-
ROUGEL
ANLS
FM
Codecien 68
Code Instrutioos
0.311
0.444
0.375
0.201
0.327
0.354
0.283
0.218
docstings and comments
without
0.409
0.348
0.195
0.327
0.335
0.301
Instrutioos
Code Instructions without
docatrings comments
0.316
0.144
0.269
0.151
CodeCian
NL Instructions
0,52
0.278
0.215
0.257
0.134
NI. Instrnctoon with docatrings and comments
0.062
0.312
0.254
0.139
0,293
0.275
0.208
0.148
BLOOM 7B
NL Instructions
0.046
01247
0.203
0.156
0.276
0.247
0.172
0.122
NI. Instruction with docatrings and comments
0.044
0,303
0.233
0.165
0.263
0.266
0.199
0.147
Table 7: Ablation: Zero-Shot Setting. (i) In each model, prompting with pseudo-code instructions results in much higher performance on QA and classification tasks (ii) For each model family, increasing scale helps improve performance (iii) As before, prompting a model designed for code, CodeGen results in better performance than BLOOM. On average, the CodeGen model, the use of code comments and docstrings helps improve the performance of natural language prompts. However, it appears for BLOOM,, only the larger-sized model is able consistently use the additional details in the prompt to improve performance.
find that the performance of natural language in- structions also improves by the inclusion of com- ments and docstring for each model family and configuration. We hypothesize that the gains may be attributable to a form of step-by-step reasoning derived from pscudo-code comments especially in complex tasks.
in O-shot settings, with the increase in scale, the performance of pscudo-code instructions improves for both model families. However, when using natural language instructions, this not the case. We hypothesize, that since none of these models are instruction-tumed larger scales exacerbate the propensity of the models being primed for language completion.
4.5 Summary of findings
Code vs. Natural Language models: We find that code models are better suited for exploiting pseudo-code prompts compared to language mod- els. As can be seen from Table 4 (see metrics for 'All Tasks'), the use of natural language instruc- tions on CodeGen results in better performance than their use on BLOOM.
We now summarize our findings for easy reference. Effeet of Prompting Style: From Table 4 we ob- serve that 0-shot prompting of pre-trained models with pseudo-code prompts results in better perfor- mance than natural language prompts. This is true for both code models and language models. The gains are more pronounced for the code models. Effect of Structure in prompts: Pseudo-code prompts include many elements such the func- tion declaration, docstring, comments etc. From Table 6 find that while information from the function declaration, and a lask-indicative function name help, using the complete pseudo-code prompt is most useful.
Conclusion and Future Work
In this paper we presented our work prompting with pseudo-code instructions. We created col- lection of pseudo-code instructions comprising of 132 NLP tasks from the Super-Naturallnnstueeee dataset (Wang et al., 2022b). We evaluated the performance of the following families of models CodeGen and BLOOM at different model sizes and found that prompting all models with pseudo-code instructions results in significant gains as compared to prompting with NL instructions. Our work opens up multiple directions of future work. It is inter- esting observe that not only do pseudo-code in- structions help when used with code models, they also work better on models designed for natural language tasks. In addition, the fact that code mod-
Further, from Table 7 we find that the pseudo- code instruction still works better than any prompt created with natural language instructions, even when docstring and comments from pseudo-code are included in the natural language instruction. This suggests the gains from prompting in pseudo- code are not just due to comments and docstrings (which could help reinforce the task instructions),
but also due to clearer instructions in pseudo-code Effect of Model Size: From Table we find that

---

els used in our experiments perform better than NL models, even when prompted with natural language instructions, suggests that could be useful to ex- plore instruction tuning of code models instead of pure NL models for NI. applications. Based on the findings of this paper it may also be useful to consider the effects of instruction fine-tuning with pseudo-code instructions as opposed to NL instruc- tions.
Fernando Alva-Manchego, Louis Martin, Antoine Bor- des, Carolina Scarton, Benoît Sagot, and Lucia Spe- cia. 2020. ASSET: A dataset for tuning and evalua- tion of sentence simplification models with multiple rewriting transformations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4668-4679, Online, Association for Computational Linguistics.
Reza Yazdani Aminabadi, Samyam Rajbhandari. Minjia Zhang. Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng. Jeff Rasley, Shaden Smith, Olatunji Ruwase, and Yuxiong Hc. 2022. Deepspeed inference: En- abling efficient inference of transformer models at unprecedented scale.
Another aspect worth studying is how traditional thoinhof- may compare with pseudo-code prompts - how would reasoning enabled by pseudo- code instructions compare with chain-of-thoo -hotuught reasoning with and without fine-tuning? Further, pseudo-code instructions may not only be used as direct inputs to a model, but they could also be used to create intermediate responses that model needs generate prior to returning a response.
Simran Arora, Avanika Narayan, Mayee F Chen, Lau- rel Orr. Neel Guha. Kush Bhatia, Ines Chami, and Christopher Re. 2023. Ask me anything: sim- ple strategy for prompting language models. In The Eleventh International Conference Learning Rep- resentations.
Limitations
Stephen H Bach, Victor Sanh, Zheng-Xin Yong, Al- bert Webson, Colin Raffel, Nihal V Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, et al. 2022. Promptsource: An integrated development environment and repository for natural language prompts, arXiv preprint arXiv:2202.01279.
Our results have been reported on two model fam- ilies - CodeGen and BLOOM at scales of 2-7B parameters. remains to be seen if our findings would hold at larger model sizes. It is possible that better reasoning enabled by larger model sizes could reduce the benefit of prompting with pseudo- code instructions but we have not investigated this in our work. In addition, our work does not include any multi-lingual NLP tasks BLOOM was specif- ically trained to be able to support multiple lan- guages and it possible this model design choice could play a role in our findings when we com- pare code (CodeGen) and NL (BLOOM) models against each other. Moreover, both models have been trained on different datasets and this also af- feets the intrinsic reasoning capabilities of these models. Lastly, and importantly, the use of pseudo- code for prompting LLMs is limited by the expec- tation that it requires technical expertise to write them, thus reducing their widespread usage.
Mohaddeseh Bastan, Mahnaz Koupace, Youngsco Son, Richard Sicoli, and Niranjan Balasubramanian. 2020. Author's sentiment prediction.
Qiang Ning Ben Zhou, Daniel Khashabi and Dan Roth. 2019. "going on a vacation" takes longer than "go- ing for walk": A study of temporal commonsense understanding. In EMNLP.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Thirty- Fourth AAAI Conference on Artificial Intelligence.
Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawa- har, and Dimosthenis Karatzas. 2019. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vi- sion. pages 4291-4301.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632 632-642, Lishon, Portugal. Association for Compu- tational Linguistics.
References
2dot7 mily. Youtube captions corrections.
htpp:://gihhul htthuu..com//// 2 captions_correctto
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz AI- shamsi, Alessandro Cappelli, Ruxandra Cojocaru. Merouane Debbah, Etienne Goffinet, Daniel Hes- low, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B: an open large language model with nate-of-the-ane performance.
Oana-Maria Camburu. Tim Rockt"aschel. Thomas Lukasiewicz, and Phil Blunsom. 2018. e-snli: Natu- ral language inference with natural language expla- nations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, Cesa- DesaaBBianchi, and R. Garnett, ed- itors, Advances in Neural Information Processing

---

Systems 31, pages 9539-9549. Curran Associates, Inc.
Joint Conference on Natural Language Processing JEMNLP-IJCNLL pages 5925-5937 Hong Kong, China. Association for Computational Linguistics.
Hyundong Cho and Jonathan May. 2020. Grounding conversations with improvised dialogues. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics.
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detec- tion and the problem of offensive language. Proceed- ings of the International AAAI Conference on Web and Social Media, 11(1):512-55 1155
Aakanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay. Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Doham, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways.
Dheeru Dua, Yizhong Wang. Pradeep Dasigi. Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprchension benchmark requir- ing discrete reasoning over paragraphs. In North American Chapter of the Association for Computa- tional Linguistics.
Esin Durmus and Claire Cardie. 2019. A corpus for modeling user and language effects in argumentation on online debating. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguisties, pages 602-607, Florence, Italy. Associa- tion for Computational Linguistics.
Yanai Elazar and Yoav Goldberg. 2019. Where's my head? Definition, data and models for numeric fused-head identification and resolution. Transac- tions of the Association for Computational Linguis- rics, 7:519-535.
Hugging Face. Amazon polarity dataset. https:// wwwgingfacee.o/ccttaeettttt htt.re//ccccttttttsassoooopppppo
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling mntruction-finetume language models. arXiv preprint arXiv:2210.11416.
Hao Fu, Yao; Peng and Tushar Khot. 2022. How does gpt obtain its ability? tracing emergent abilities of language models to their sources. Yao Fu' ''s Notion.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling.
cjadams, Daniel Borkan, inversion, Jeffrey Sorensen, Lucas Dixon, Lucy Vasserman, and nithum. 2019. Jigsaw unintended bias in toxicity classification.
Christopher Clark, Kenton Lee, Ming-Way Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Techmologies, Volume (Long and Short Papers), pages 2924-2936, Minncapolis. Min- nesota. Association for Computational Linguistics.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346 361.
Allec Go, Richa Bhayani, and Lei Huang. 2009, Twit- ter sentiment classification using distant supervision. CS224N project report, Stanford, ((12):2009.
Alexis Conneau and Douwe Kicla. 2018. SentEval: An evaluation toolkit for universal sentence representa- tions. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2003. English gigaword. Linguistic Data Consor- fium, Philadelphia, 4(1):34.
Matthew Henderson, Blaise Thomson, and Jason D Williams. 2014. The third dialog state tracking chal- lenge. In 2014 IEEE Spoken Language Technology Workshop (SLT), pages 324-329. IEFE.
Pradeep Dasigi. Nelson F. Liu, Ana Marasovié, Noah A. Smith, and Matt Gardner. 2019, Quoref: A read- ing comprehension dataset with questions requir- ing coreferential reasoning. In Proceedings of the 2019 Conference on Empirical Methods Natu- ral Language Processing and the 9th International
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. 2021. Measuring massive multitask language

---

understanding, Proceedings of the International Con- ference on Learning Representations (ICLR).
Human Language Technologies, Volume / (Long Pa- pers), pages 252-262. New sociation for Computational Linguisties.
New Orleans, Louisiana. As-
Nabil Hossain, John Krumm, Michael Gamon, and Henry Kautz. 2020. SemEval-2020 task 7: Assess- ing humor in edited news headlines. In Proceed- ings of the Fourteenth Workshop on Semantic Eval- nation, pages 746-758, Barcelona (online). Interna- tional Committee for Computational Linguisties.
Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2017. Learning what essential in ques- tions. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017). pages 80-89 89, Vancouver. Canada. Association for Computational Linguistics.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos QA: Machine reading comprehension with contextual commonsense rea- soning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- val Language Processing (EMNLP-JJCNLP) pages 2391-2401, Hong Kong, China. Association for Com- putational Linguistics.
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. Qase: A dataset for question answering via sentence com- position. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8082 8090.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. ArXiv, abs/2205.11916.
Ting-Hao Kenneth Huang, Chich- Yang Huang, Chien- Kuang Comelia Ding, Yen-Chia Hsu, and C. Lee Giles. 2020. CODA-19: Using non-expert crowd to annotate research aspects on 10,000+ abstracts in the COVID-13 19 open research dataset. In Proceedings of the 1st Workshop NLP for COVID-19 at ACL 2020, Online, Association for Computational Linguistics.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wer wei Chang, Andrew Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natu- ral questions: benchmark for question answering research. Transactions of the Association of Compu- tational Linguistics.
Kelvin Jiang, Dekun Wu, and Hui Jiang. 2019. Free- baseQA: A new factoid QA data set matching trivia- style question-anse answer pairs with Freebase. In Pro- ceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 318-323, Min- neapolis, Minnesota, Association for Computational Linguistics.
Hugo Laurençon, Lucile Saulnier, Thomas Wang. Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, An- gelina McMillan-Major, Gerard Dupont, Stella Bi- derman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas. Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel Van Strien, 7aid Alyafeai, Khalid Almubarak. Minh Chien Vu, Itziar Gonzalez- Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Or- tiz Suarez, Aaron Gokaslan, Shamik Bose, David Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq. Suzana Ilic, Margaret Mitchell, Sasha Alexandra Luccioni, and Yacine Jer- nite. 2022. The bigscience roots corpus: A 1,6tb com- posite multilingual dataset. In Advances in Neural Information Processing Systems, volume 35, pages 31 809-318266 Curran Associates, Inc.
Di Jin and Peter Szolovits. 2018 PICO element de- tection in medical text via long short-term memory neural networks. In Proceedings of the BioNLP 2018 workshop, pages 67-75, Melbourne, Australia, Asso- ciation for Computational Linguistics.
Mandar Joshi, Eunsol Choi. Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol- une 1: Long Papers), pages 1601-1611, Vancouver, Canada. Association for Computational Linguistics.
Phillip Keung, Yichao Lu, György Szarvas, and Noah A. Smith. 2020. The multilingual Amazon reviews cor- pus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4563-4568, Online, Association for Computational Linguistics.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 333-342, Vancouver. Canada. Association for Computational Linguisties.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018 Looking beyond the surface: A challenge set for reading com- prehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics:
Xin Li and Dan Roth. 2002. Learning question clas- sifiers. In COLING 2002: The 19th International Conference on Computational Linguistics.

---

Yanran Li, Hui Su. Xiaoyu Shen, Wenjie Li. Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Nat- ural Language Processing |Volume I: Long Papers), pages 986-995, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Amita Misra, Brian Ecker, and Marilyn Walker. 2016. Measuring the similarity of sentential arguments in dialogue. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dia- logue, pages 276-287, Los Angeles. Association for Computational Linguistics.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 189, San Diego, California. Association for Computational Linguis- tics.
Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gard- ner. 2019a. Reasoning over paragraph effects in situ- ations. In MRQA @ EMNLP.
Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gard- ner. 2019b. Reasoning over paragraph effects in situ- ations. In Proceedings of the 2nd Workshop on Ma- chine Reading for Question Answering, pages 58-62, Hong Kong. China. Association for Computational Linguistics.
Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. 2017. Ls- dsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pages 46-51.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre- trin, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput, Surv. 55(9).
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung. Yi Tay, Denny Zhou, Quoe V Le, Barret Zoph, Jason Wei, et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 13688.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial nli: A new benchmark for natural language under- standing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. As- sociation for Computational Linguistics.
Nicholas Lourie, Ronan Le Bras. and Yejin Choi. 2021. Scruples: A corpus of community ethical judg- ments on 32,000 real-life anecdotes. Proceedings of the AAAI Conference on Artificial Intelligence. 35(15):13470-1344
Erik Nijkamp, Bo Pang. Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations.
Bill MacCartney and Christopher D Manning. 2007. Natural logic for textual inference. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 193-200.
OpenAL. 2023. Gpt-4 technical report.
Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang. and Graham Neubig. 2022. Language models of code are few-shot commonsense learners. arXiv preprint 1rXiv22210.01128.
Simon Ostermann, Ashutosh Modi, Michael Roth, Ste- fan Thater, and Manfred Pinkal. 2018. MCScript: A novel dataset for assessing machine comprehen- sion using script knowledge. In Proceedings of the Eleventh International Conference on Language Re- sources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
MarvinAI. 30 March 2023. Marvinai. https:/ / /www. askmarvin.ai//
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2019. The natural language decathlon: Multitask learning as question answering.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-555
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering biguous open-domann questions. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 5783- 5797, Online. Association for Computational Lin- guistics.
Alessandro Raganato, Tommaso Pasini, Jose Camacho- Collados, and Mohammad Taher Pilehvar. 2020. XL- WiC: multilingual benchmark for evaluating se- mantic contextualvaatin. In Proceedings of the 2020 Conference Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 7193-7206, On- line. Association for Computational Linguistics.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannanch Hajishirzi. 2021. Cross-task generaliza- tion via natural language crowdsourcing instructions. arXiv preprini 2rXiv:2104.08773.

---

Pranav Rajpurkar, Robin Jia, and Percy Liang, 2018. Know what you don't t know: Unanswerable ques- tions for SQuAD, In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia, Association for Computational Linguisties.
Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady Elsahar, Hamza Benyamina, Hieu Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dior Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jörg Frohberg, Joseph Tobing, Joy- deep Bhattacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro Von Werra, Leon Weber, Long Phan, Loubna Ben allal. Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Masoud, María Grandury. Mario Saško, Max Huang, Max- imin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad A. Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nuru- laqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Villegas, Peter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto Luis López, Rui Ribeiro, Salomey Osei Sampo Pyysalo, Se- bastian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, Shayne Longpre, So- maich Nikpoor, Stanislav Silberberg, Suhas Pai, Syd- ney Zink, Tiago Timponi Torrent, Timo Schick, Tris- tan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala. Violette Lepercq, Vrinda Prabhu, Zaid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling. Chenglei Si, Davut Emre Taşar, Eliz- abeth Salesky, Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, Andrea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajyoti Datta, Eliza Szezechla, Gunjan Chhablani, Han Wang. Harshit Pandey, Hen- drik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Saiful Bari, Maged S. Al-shaibani, Matteo Manica, Nihal Nayak, Ryan Teehan, Samuel Albanie. Sheng Shen, Srulik Ben- David, Stephen H. Bach, Taewoon Kim, Tali Bers, Thibault Fevry Trishala Neeraj. Urmish Thakker, Vikas Raunak, Xiangru Tang, Zheng-Xin Yong. Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarieh, Adam Roberts, Hyung Won Chung, Jae- sung Tae, Jason Phang, Ofir Press, Conglong Li. Deepak Narayanan, Hatim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi. Myriam Peyrounette. Nicolas Patry, Nouamane Tazi, Omar Sanseviero. Patrick von Platen, Pierre Cornette, Pierre François Lavallée, Rémi Lacroix. Samyam Rajbhandari. San- chit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Dettmers, Ahmed Baruwa, Amanpreet Singh. Anastasia Cheveleva, Anne- Ligozat, Arjun Subramonian, Aurélie Névéol, Charles Lover- ing, Dan Garrette, Deepak Tunuguntla, Ehud Reiter, Ekaterina Taktasheva. Ekaterina Voloshina. Eli Bog- danov, Genta Indra Winata, Hailey Schoelkopf. Jan- Christoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Miruna Clinciu, Na- joung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann. Shachar Mirkin, Shani Pais, Tatiana Shavrina, Thomas Scialom, Tian Yun. Tomasz Limisiewicz, Verena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada Pruksachatkun, Yonatan
Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguisties.
Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Com- putational Linguistics, 7:249-266 266.
Laria Reynolds and Kyle McDonell. 2021. Prompt pro-
gramming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHi Conference on Human Factors in Com- puting Systems, CHI EA '21, New York, NY, USA. Association for Computing Machinery.
Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chan- dra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith, and Yejin Choi. 2020. Thinking like a skeptic: Defeasible inference in natural language. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 4661-4675, Online. Association for Computational Linguistics.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavat- ula, and Yejin Choi. 2021. Winogrande: An adver- sarial winograd schema challenge at scale. Commuu nications of the ACM, 64(9):99-100) 106.
Maarten Sap, Ronan Le Bras, Emily Allaway, Chan- dra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi, 2019. Atomic: An atlas of machine commonsense for if- then reasoning. Proceedings of the AAAI Conference on Artificial Intelligence, 3301133027-3035.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné. Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Am- manamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Vic- tor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji. Amit Alfassy. Anna Rogers, Ariel Kreisberg Nitzay, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, Dragomir Radev, Ed- uardo González Ponferrada, Efrat Levkovizh, Ethan

---

Belinkov, Zachary Bamberger, Zdenčk Kasner, Al- ice Rueda, Amanda Pestana, Amir Feizpour, Am- mar Khan, Amy Faranak, Ana Santos, Anthony Hevia, Antigona Unldreaj. Arash Aghagol, Are- zoo Abdollahi, Aycha Tammour, Azadeh HajiHos- seini, Bahareh Behroozi, Benjamin Ajibade, Bharat Saxena, Carlos Muñoz Ferrandis, Damish Contrac- tor, David Lansky, Davis David, Douwe Kiela, Duong A. Nguyen, Edward Tan, Emi Baylor, Ez- inwanne Ozoami, Fatima Mirza, Frankline Onon- iwu, Habib Rezanejad, Hessie Jones, Indrani Bhat- tacharya, Irene Solaiman, Irina Sedenko, Isar Ne- jadgholi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Livia Dutra, Mairon Samagaio, Maraim EI- badri, Margot Mieskes, Marissa Gerchick, Martha Akinlolu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok. Nafis Abrar, Nazneen Ra- jani, Nour Elkott, Nour Fahmy, Olanrewaju Samuel, Ran An. Rasmus Kromann, Ryan Hao, Samira Al- izadeh, Sarmad Shubber, Silas Wang. Sourav Roy. Sylvain Viguier. Thanh Le. Tobi Oyebade, Trieu Le. Yoyo Yang. 7ach Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Anima Shukla, Antonio Miraada-Escalada, Ayush Singh, Benjamin Beilharz, Bo Wang. Caio Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel León Periñán, Daniel Molano, Dian Yu, Enrique Manjava- cas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyaseddin Bayrak, Gully Burns, Helena U. Vrabec, Imane Bello, Ishani Dash, Jihyun Kang, John Giorgi, Jonas Golde, Jose David Posada, Karthik Ranga- sai Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Mare Pamies, Maria A Castillo, Mari- anna Nezhurina, Mario Sänger, Matthias Samwald, Michael Cullan, Michael Weinberg, Michiel De Wolf, Mina Mihaljeie, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung, Patrick Haller, Ramya Chandrasekhar, Renata Fisenberg, Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok S Deshmukb, Shubhanshu Mishra, Sid Ki- blawi, Simon Ou, Since Sang-aroonsiri, Srishti Ku- mar, Stefan Schweter, Sushil Bharati, Tanmay Laud, Théo Gigant, Tomoya Kainuma, Wojeiech Kusa, Ya- nis Labrak, Yash Shailesh Bajaj. Yash Venkatraman, Yifan Xu, Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. 2023. Bloom: 176b-parameter open- access multilingual language model.
cal Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Gabriel Stanovsky and Mark Hopkins. 2018. Spot the odd man out: Exploring the associative power of lexi- cal resources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), Brussels, Belgium. Association for Computational Linguistics.
Kai Sun, Dian Yu, Jianshu Chen, Dong Yu. Yejin Choi, and Claire Cardie. 2019. DREAM: A challenge data set and models for dialogue- reading compre- hension. Transactions of the Association for Compu- tational Linguistics, 7:217-231.
Oyvind Tafjord, Peter Clark, Matt Gardner, Wen tau Yih, and Ashish Sabharwal. 2018. Quarel: A dataset and models for answering questions about qualitative relationships. ArXiv, abs/18 0tbs/88110080 11.08844
Jörg Tiedemann. 2012. Parallel data, tools and inter- faces in opus. In Proceedings of the Eight Inter- national Conference on Language Resources and Evaluation (LREC'12) 12), Istanbul, Turkey. European Language Resources Association (ELRA).
Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. SemEval-2018 task 3: Irony detection in En- glish tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 39-50, New Orleans, Louisiana. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.
Bertie Vidgen, Dong Nguyen, Helen Margetts, Patricia Rossini, and Rebekah Tromble. 2021. Introducing CAD: the contextual abuse dataset. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 2289 -2033, Online. Association for Computational Linguistics
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Al- isa Liu, Noah A Smith, Daniel Khashabi, and Han- nanch Hajishirzi. 2022a. Self-instruct: Aligning lan- guage model with self generated instructions. arXiv preprint arXiv:2212 10560.
Emily Sheng and David Uthus. 2020. Investigating societal biases in a poetry composition system. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 93-106, Barcelona, Spain (Online). Association for Computa- tional Linguistics.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran. Anjana Arunkumar, David Stap. Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Puro- hit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma. Ravsehaj Singh Puri, Rushang Karia, Savan Doshi,
Richard Socher, Alex Perelygin, Jean Wu. Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empiri-

---

Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong General- ization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 5085-5109, Abu Dhabi, United Arah Emirates. Association for Computational Linguistics.
Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Seao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-an natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, Association for Computational Linguistics.
Shen. 2022b. Super-Naturallnstreettn
Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- man. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625 641.
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gard- ner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. 2020. Break it down: A question understand- ing benchmark. Transactions the Association for Computational Linguistics.
Kellie Webster, Marta Recasens, Vera Axelrod, and Ja- son Baldridge. 2018. Mind the GAP: A balanced corpus of gendered ambiguous pronouns. Transac- tions of the Association for Computational Linguis- tics, 6:605-617.
Wenhan Xiong, Jiawei Wu, Hong Wang. Vivek Kulka- mi, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2019. TWEETQA: A social media focused question answering dataset. In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 5020- 5031, Florence, Italy. Association for Computational Linguistics.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. 2022a. Finetuned language models are zero-shot learners. In International Con- ference on Learning Representations.
Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018. Learning to mine aligned code and natural language pairs from stack overflow. In International Conference on Min- ing Software Repositoriex, MSR, pages 476-486. ACM.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoe V Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint arXiv:2109.01652.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. 2022b. Chain of thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
J.D. Zamfirescu-Peree Richmond Y. Wong, Bjoern Hartmann, and Qian Yang. 2023. Why johnny can't prompt: How non-ai experts try (and fail) to design Ilm prompts. In Proceedings of the 2023 CHI Confer- ence on Human Factors in Computing Systems, CHI '23, New York, NY, USA. Association for Computing Machinery.
Orion Weller, Nicholas Lourie, Matt Gardner, and Matthew E. Peters. 2020. Learning from task de- scriptions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 1361-1375, Online. Association for Computational Linguistics.
Li Zhang, Liam Dugan, Hainiu Xu, and Chris Callison- Burch. 2023a. Exploring the curious case of code prompts, arXiv preprint 2tti::2304.122550
Li Zhang, Hainiu Xu, Yue Yang, Shuyan Zhou, Weiqiu You, Manni Arora, and Chris Callison- Callison-Burch. 2023b. Causal reasoning of entities and events in procedural texts. arXiv preprint arXiv:2301.10896
John Wieting and Kevin Gimpel. 2018. ParaNMT-50M,
Pushing the limits of paraphrastic sentence embed- dings with millions of machine translations. In Pro- ceedings of the 56th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Pupers), pages 451 451-462, Melhourne, Australia. As- sociation for Computational Linguistics.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and ma- chine commonsense reading comprehension. ArXiv, fbs/1810.12888,
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume I (Long Papers). pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase Adversaries from Word Scram- bling. In Proc. of NAACL.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In Proceedings of ICML, pages 12697-12706.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistae, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen,
Peixiang Zhong, Chen Zhang, Hao Wang. Yong Liu, and Chunyan Miao. 2020. Towards persona-based empa- thetic conversational models. In Proceedingy of the

---

2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6556-6566 6566, Online. Association for Computational Linguistics.
Listing 2 Code instructions (0-shot prompt) for
Sentiment classification task def
str) -> str: """For the given sentence, the task predict the sentiment. For positive sentiment return "positive" else return "negative".
A
Appendix
A.1 Results additional LLMs
Parameters:
sentence (str): input sentence
We also perform experiments using the recently re- leased, Falcon-7B (Almazrouci et al., 2023) model. The results are presented in Table and also demon- strates improved performance with pseudo-code prompts.
Returns:
str: sentiment of the input
# predict the sentiment
if ittpimettssspssiiieeeeeeeee return "positive"
A.2 Pseudo-Code Validation
else: return "negative"
To ensure that the pseudo-code instructions fol- low the guidelines provided, we run automatic test. The test code calls the preprocess func- tion defined for each example from the Super- Naturallnstruccions dataset (Wang al., 2022b) for that task. The returned values from the preprocess function are compared against the ar- guments in the function prototype. Any mismatch in the data type or the number of arguments results in error. The instruction creator is given feedback to correct the errors.
>>> generate_sentiments
"that has charmingly bourbon air."
Listing 3 Code instructions (2-shot prompt) for
sentiment classification task def
str) -> str: """For the given sentence, the task predict the sentiment. For positive sentiment return "positive" else return "negative".
Parameters:
A.2.1 Prompt Styles
sentence (str): input sentence
Returns:
In this section, we describe the various prompting styles used to study the effect of pseudo-code vs NL prompting. Here, we show simple task to generate the sentiment of given sentence. This is task 833 in Super-NaturalInsturn
str: sentiment of the input
# predict the sentiment if iet.imenttss.ositieeeeeeeee return "positive"
datasct.
else:
return "negative"
A.2.2 Prompting with Pseudo-code instructions
generate_sentimee sentteee__ntmmmnt(
"tormented by the quickened blood of the "roots"
For the pseudo-code prompting, we use the in- structions that are created by the authors of this paper. The pseudo-code instructions have much richer structure than natural language instructions and are more elaborate and simple to understand. They contain docstrings, return types and might also contain comments, function invocations etc. For preparing the few shot examples and the input query, we treat the example as a python interpreter running in the linux terminal and use the special markers ">>>' for the input. We don't use any spe- cial markers for the outputs. An example for 0-shot and s-ohot shot prompting is shown in Listings and respectively.
"negative"
Cenerate_sentiment
"radiant from the mount, he stood"
"positive"
generate [
"that has charmingly bourbon air."
A.2.3 Prompting with function prototype
We try prompting the models with function proto- types with all docstrings, comments and code logic removed from the base pseudo-code instruction. The function prototype instructions are composed of the function names, arguments and their types
We also measure the impact of removing the docstrings and comments from the code instruction. An example for 0-shot and 2-shot prompting is shown in Listings and respectively.

---

Model
Instruction Formut
Clusdiication Tasks
Tusks
Generation tasks
All Tusks
Mazro Fl
Micro F!
Weighted F1
ROUGE-L
ROUGE-L
ROUGELL
ANLS
EM
Majority Closs
0.296
0.509
0,362
Code Instructions
0.068
0.339
0,259
0.152
0.265
0.275
0.207
0.161
Falcon 78
Instructions
0.017
0,206
0.197
0.172
0.273
0.242
0.149
0.102
Table Performance of models when prompted using pseudo-code instructions and natural language instructions 0-shot settings. (i) In each model, prompting with pseudo-code instructions results in much higher performance in almost all the tasks
Listing Code instructions without docstrings and comments (O-shot prompt) for sentiment classifica- tion task
Listing Function prototype (0-shot prompt) for sentiment classification task def Severate_sentmmet((CC str) str:
def generate_ ietesseeeeeeeeeteeeeeeeee str)
if sen tontimnt_ii_ppe wessssssitteeeseetteeeeeeeeeeeee return "positive"
generate_ (
"that has charmingly bourbon air."
else:
return "negative"
>>> generate_sentiment
"that has a charmingly bourbon air."
Listing Function prototype (2-shot prompt) for sentiment classification task def Seneerat_sentiment[Ce str) -> str:
Listing Code instructions without docstrings and comments (2-shot prompt) for sentiment classifica- tion task
"tormented by the quickened blood of the "roots"
def generate_sentiment lit_issssssieeee_________e_seeee (eenteeee): return "positive"
if sentiment teentteetteeeee str) -> str:
"negative"
else:
>>>
return "negative"
"radiant as moses from the mount, he stood"
>>> seterrate_sentmmentee
"pasitive"
"tormented by the quickened blood of the "roots"
>>> generate_sentiment
"that has charmingly bourbon air."
"negative"
>>> generate_sente (
"radiant as moses from the mount, he stood"
"positive"
the prompts provided part of the Super- NaturalInstructions dataset without any modifica- tion. We add special 'input:" and 'output:' markers in the few shot examples and the input query to the model as shown in Listings and 9.
>>> generate_sentiment
"that has charmingly bourbon air."
and the return types. This method of prompting is devoid of any pseudo-code. An example for O-shot and 2-shot prompting shown in Listings and respectively.
Listing 8 Natural instructions (0-shot prompt) for sentiment classification task
In this task, you need identify sentiment of the given sentence as one "positive" or "negative".
A.2.4 Prompting with NI instructions
For natural language prompts, we use the orig- inal instructions provided as part of the Super- NaturalInstructions dataset (Wang et al., 2022b). For natural language instruction prompting, we
input: that has a charmingly bourbon air. output:

---

Listing 9 Natural instructions (2-shot prompt) for sentiment classificatiin task
Listing 11 Natural instructions with docstrings (2- shot prompt) for sentiment classification task In this task, you need identify the smntiment of the given sentence "negative".
this task, you need to identify the sentiment the given sentence as one of "positive" or "negative".
of "positive"
input: tormented by the quickened blood of the roots
is predict the sentiment. For positive sentiment "positive" else return "negative".
"""For the given sentence,
output:
negative
radiant as moses from the mount, he stood output: pasitive
input:
Parameters:
sentence (str): input sentence
Returns:
that has charmingly bourbon air. output:
input:
str: sentiment the input *"*
predict the sentiment
A.2.5 Prompting with NL instructions and NL comments from the pseudo-code
input: tormented by the quickened blood of the roots
output: negative
We also try experimenting by adding the doc- strings and comments to the NL instructions from the Super=Nauurallnstructooon dataset (Wang et al., 2022b) as shown in the example in Listings and 11.
input: radiant as moses from the mount, he stood output: positive
input: that has a charmingly bourbon air. output:
Listing 10 Natural instructions with docstrings (0- shot prompt) for sentiment classification task
Listing 12 Function invocation (O-shot prompt) for
In this task, you need to identify the sentiment of the given sentence one of "positive" or "negative".
sentiment classification task
>>> ( "that has charmingly bourbon air."
"""For the given sentence, the task is to predict the sentiment. For positive sentiment return "positive" else return "negative".
Listing 13 Function invocation (2-shot prompt) for sentiment classification task
Parameters:
sentence (str): input sentence
Returns:
eeterate_sentimntte
str: sentiment of the input
toormetted by the quickened blood of the "roots"
predict the sentiment
"negative"
that charmingly bourbon air. output:
input:
generate_sentimen
"radiant as moses from mount, stood"
"positive"
generate_sentimee se__nteent((((((e
A.2.6 Prompting without instructions
"that has charmingly bourbon air."
We also study the effect of prompting without in- structions. We try this method of prompting in three settings:
Listing 14 Generic function invocation (0-shot
1. Function Invocation (refer Listings 12 and 13)
prompt) for sentiment classification task func(
2. Generic Invocation (refer Listings 14 and 15)
"that has a charmingly bourbon air."
3. Natural Language examples (refer Listings 16 and 17)

---

Model
Instruction Format
Clusdiication Tusks
QA Tasks
Generation Tosks
All Tasks
Macio F1
Miero FI
Weighted F1
ROUGEL
ROUGEL.
ROUGES.
ANLS
FM
CodeGen
Code Instructions
0.137
0.295
0.272
0.187
0.299
0.269
0.202
0.148
NL Instructions
0,000
0.004
0.006
0.022
0.130
0.011
0.017
0.006
CodeGen
Code instructions
0.145
0.317
0,292
0.194
0.304
0,285
0.219
0.139
NL Instructions
0,000
0.001
0.002
0.101
0.172
0.089
0.024
0.006
BLOOM Code Instructions
0,086
0.254
0.227
0,151
0.248
0.226
0.164
0.121
NI. Instructions
0.005
0.060
0.060
0.151
0.207
0.140
0.070
0.038
BLOOM
NI. Instructions
Instructions
0.000
BJ072
0.120
0.227
0.014
0.137
0.186
0.109
0.250
0.176
0.941
0.018
Table Performance with 2-shot prompts. (i) each model, prompting with pseudo-code instructions results much higher performance (ii) For each model family, increasing scale helps improve performance (iii) As before, prompting model designed for code, CodeGen results in better performance than BLOOM. (iv) Surprisingly, model configurations and
compared to 0-shot prompting (Table 4), there is marked drop in performance tasks, except in QA tasks, where there is improvement in performance.
Listing 15 Generic function invocation (2-shot
with 2-shot prompts. Table reports the perfor- mance of both families of models CodeGen and BLOOM when using pseudo-code prompts and natural language instruction prompts in 2-shot set- tings.
prompt) for sentiment classification task
func(
"tormented by the quickened blood of the "roots"
"negative"
Interestingly we find that, as compared to the results reported in Table the performance of each corresponding model-prompt configuration is lower than its O-shot counterpart. While this may appear surprising, similar findings have been reported in prior work (Reynolds and McDonell, 2021; Zhang et al., 2023a). Perhaps the perfor- mance in few-shot settings could improve with ad- ditional examples, but we do not experiment with more than 2-shot settings due to limitations im- posed by the size of input context length available to models.
>>> func(
"radiant as moses from the mount, he stood"
"positive"
>>> func(
"that has a charmingly bourbon air."
Listing 16 Natural examples (O-shot prompt) for sentiment classification task input: that has charmingly bourbon air.
output:
After study of outputs generated by the mod- els in 2-shot settings, we observe that in many cases, in the absence of extensive task-specific prompt-cnginecring and output processing, models are likely to generate additional continuation exam- ples instead of solving the task. The fact that the pseudo-code prompts perform better indicate that models seem "interpret" the instructions better in this form.
Listing 17 Natural examples (2-shot prompt) for sentiment classification task input: tormented the quickened blood af the roots
output:
negative
radiant as moses from the mount, he stood output: positive
input:
that has charmingly bourbon air. output:
input:
A.4 Ablation Experiments
As can be seen in Table 10 and 11, the inclusion of comments as well as the docstring in the pscudo- code instruction prompt and natural language in- structions helps improve performance for smaller models too.
A.3 2-shot Prompting with Pseudo-code instructions
Given that structured prompts, such as those based on function declarations, benefit from 2-shot prompts, we investigate whether the performance pseudo-code prompts can be further improved

---

Model
Instruction Format
Classfication Tusks
Tasks
Generation Tacks
All Tasks
Macro Fl
Miero F1
Weighted F1
ROUGEE
ROUGEFI
ROUGE-L
ANLS
FM
Nt Instructions
0.068
0,306
0.239
0.154
0.254
0.265
0.195
0.147
CodeCien 20
dntstrings and commeents
NL Instructions with
0.098
0.349
0.270
0.136
0.258
0.275
0.208
0.161
CodeGen
Instructions
0.052
0.215
0.132
0.271
0.257
0.187
0.134
Instructions
docstrings commeents
Instructions
0,082
0.275
0.214
0.159
0.250
0.132
NL Instruciios with doestrings
01946
0.200
0.121
0.213
0.111
BLOOM
NL Instructions
0.046
0.247
0.203
0.156
0.276
0.247
0.172
0.122
NL Instructions with decotrings and commments
(L0444
0,313
0.233
0.165
012033
0.266
0.199
0.147
Table 10: Ablation: On average, in the CodeGen model the use of code comments and docstrings in O-shot setting helps improve performance of natural language prompts. However, appears on BLOOM, only the larger sized model is able to consistently use the additional details in the prompt improve performance.
Model
Instruction Format
Classifiation Tasks
QA Tasks
Generation Tasks
All Tosks
Micro Fl
Micro F1
Weightnl F1
ROUGEL
ROUGE-L
ROUGE-
ANLS
FM
Codetien
Code Instructions
0.272
0.417
0.354
0.175
0.317
0.330
0.622
0.202
decstrings comtmmnts
Instructions
0,389
0.309
Code Instructions
0.354
0.283
Code Instructions without doestrings
1.363
0.348
0.195
0.335
0.266
0.201
BLOOM
Code Instructions
0.116
0.351
0.288
0.147
0.271
0.279
0.215
0.165
Code Instructions without dncstrings and comments
0,044
0.302
0.249
1.132
0.259
0.248
0.117
0.183
BLOOM 7B
Code Instructions
0.174
0.369
0.285
0.150
0.298
0.297
0,232
0.176
Code Instructions without docstrings and comments
0.145
0.516
0.247
11.144
0.291
0.269
0.204
0.151
Using 0-shot code instructions without docstrings and comments (i) In each model, prompting with pseudo-code instructions results in much higher performance QA and classification tasks (ii) For each model family, increasing scale helps improve performance (iii) As before, prompting model designed for code, CodeGen results in better performance than BLOOM.
Table 11: Ablation:


Reviews:
[{"review_id": "2Bu0dQRqP1", "reviewer": "Reviewer_45A7", "summary": "", "questions": "1. How did you calculate the classification F1, and what's the random-guess baseline for classification tasks in Table 4? The macro F1 scores for NL instructions seem very low. Did you look at the token-level probability distribution, or did you simply measure the lexical overlap between the greedy output and the groundtruth? \n2. I suspect that one advantage of pseudo-code instruction is to clearly define the output space for classification. For example, the model is expected to output \"Yes/No\" instead of \"True/False\" for a binary problem, but the NL instruction doesn't make it clear. I wonder how often this happens. \n3. It would be helpful to include more complicated examples in appendix than Figure 1. For example, a pseudo-code instruction with step-by-step comments as mentioned in L454-457 will be very illustrative.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We are pleased to note that the reviewer found our work novel and interesting! \nWe include a response to the questions below:\n\n---\n\n### **Question 1**\n\nHow did you calculate the classification F1, and what's the random-guess baseline for classification tasks in Table 4? The macro F1 scores for NL instructions seem very low. Did you look at the token-level probability distribution, or did you simply measure the lexical overlap between the greedy output and the ground truth?\n\n### **Answer 1**\n\nWe extract the predicted answer from the generated output. We use new-line character as the delimiter and extract the text till the first occurrence of new line character as the generated text. This generated text was compared with the ground truth to calculate the F1 scores. We use exact match for computing F1-scores. The Rouge-L scores in column ALL includes the results on classification task too where we measure the Rouge score between the generated output and the ground truth. \n\nWe didn't report classification performance based on token-level probability (or perplexity) of the class label in the main paper.\nHowever, to address the reviewer's question, we compute classification performance based on perplexity. The micro-F1 numbers for BLOOM-7B and CodeGen-6B are reported in the table below. We also report a random baseline (picking a class label at random) averaged over 5 different seeds. It can be clearly seen that pseudo-code instructions outperform the NL instructions by 8 points (approx.) on BLOOM-7B and 6 points (approx.) on Codegen-6B.\n\n|  | Pseudo-code instructions | Natural Language instructions | Random baseline (independent of model) |\n|---|---|---|---|\n| BLOOM-7B | **0.227** | 0.145 | 0.075 |\n| CodeGen-6B mono | **0.209** | 0.148 | 0.075 |\n\n---\n\n### **Question 2**\n\nI suspect that one advantage of pseudo-code instruction is to clearly define the output space for classification. For example, the model is expected to output \"Yes/No\" instead of \"True/False\" for a binary problem, but the NL instruction doesn't make it clear. I wonder how often this happens.\n\n### **Answer 2**\n\nThe output space is defined in both cases (pseudo-code and NL instructions). For example, in the following task we have 2 classes (Inversion & Original) for which we demonstrate both pseudo-code and NL instruction:\n\n**Pseudo-code instruction**:\n```python\ndef is_words_flipped(sentence: str) -> str:\n    \"\"\"\n    Given a sentence, check if the sentence contains two consecutive words which needs to be flipped to make the sentence grammatical and meaningful. If yes, then return 'Inversion'. If no changes are required return 'Original'\n\n    Parameters:\n        sentence (str): a given sentence\n\n    Returns:\n        str: \\\"Inversion\"\\ or \\\"Original\\\"\n    \"\"\"\n\n    # the functions checks if the sentence contains two words which needs to be flipped to make the sentence grammatical and meaningful\n    if words_need_to_be_flipped(sentence):\n        return \"Inversion\"\n    else:\n        return \"Original\"\n```\n\n**NL instruction**:\n```text\nIn this task you are given a sentence. You must judge whether there exist two consecutive words within the sentence with flipped orders, that is, whether the sentence will make sense and be correct if the order of two consecutive words changes. Label the instances as \\\"Inversion\\\" or \\\"Original\\\" based on your judgment.\n```\nAs can be seen in both cases, the class labels are included as part of the instructions.\n\n---\n\n### **Question 3**\n\nIt would be helpful to include more complicated examples in appendix than Figure 1. For example, a pseudo-code instruction with step-by-step comments as mentioned in L454-457 will be very illustrative.\n\n### **Answer 3**\n\nWe will include the examples in the accepted version of the paper. Here are a few included examples for pseudo-code instructions:\n\n**Example 1**:\n```python\ndef create_fill_in_the_blanks_question(context_word: str, mention_person1: str, mention_person2: str, target_blank: str, child_friendly_mode: bool = True) -> str:\n    \"\"\"creates a fill in the blank question based on the mentioned persons, context word and the target person to be blanked\"\"\"\n\n    # initialize gender options\n    gender = [\"male\", \"female\", \"trans\"]\n    # choose gender to be used for PersonX and PersonY\n    gender_index = random(0, size(gender))\n    # select gender\n    selected_gender = gender[gender_index]\n    # get sentence with context word and mentioned persons based on count of mentions.\n    generated_sentence = get_sentence(\n        context_word,\n        child_friendly_mode,\n        mention_person1,\n        mention_person2,\n        mention_person_1_count=2,\n        mention_person_2_count=1,\n        min_word_length=15,\n        max_word_length=30,\n        person_gender=selected_index,\n    )\n    # blank second mention of mention_person1\n    blanked_sentence = blank_sentence(generated_sentence, blank_token=\"_\", blank_target=mention_person1, blank_mention_index=2)\n    # return blanked sentence\n    return blanked_sentence\n```\n\n**Example 2**:\n```python\ndef question_category_classification(sentence: str, question: str, category: str) -> str:\n    \"\"\"\n    Given a sentence and a question around the sentence. A reasoning category is also provided. The task is to indicate if the question belongs to the provided reasoning vategory or not.\n\n    a) Event Duration: understanding of how events last\n    b) transient v. stationary: understanding of whether an event will change over time or not\n    c) event ordering: understanding of how events are usually ordered in nature\n    d) absolute timepoint: understanding of when events usually happen\n    e) frequency: how often an event is likely to be repeated\n\n    Parameters:\n        sentence (str): sentence\n        question (str): question\n        category (str): category\n    Returns:\n        str: `Yes.` or `No.`\n    \"\"\"\n\n    if \"Event Duration\" in category:\n        # this function checks if the question has an understanding of how long the events last\n        if does_question_require_event_duration_reasoning(question, category, sentence):\n            return \"Yes.\"\n        else:\n            return \"No.\"\n    elif \"transient v. stationary\" in category:\n        # this function checks if the question has an understanding of whether an event will change over time or not\n        if does_question_require_event_temporal_reasoning(question, category, sentence):\n            return \"Yes.\"\n        else:\n            return \"No.\"\n    elif \"event ordering\" in category:\n        # this function checks if the question has an understanding of how events are ordered in nature\n        if does_question_require_event_ordering_reasoning(question, category, sentence):\n            return \"Yes.\"\n        else:\n            return \"No.\"\n    elif \"absolute timepoint\" in category:\n        # this function checks if the question has an understanding of when events usually happen\n        if does_question_require_event_happen_reasoning(question, category, sentence):\n            return \"Yes.\"\n        else:\n            return \"No.\"\n    elif \"frequency\" in category:\n        # this function checks if the question has an understanding of how often an event is likely to be repeated\n        if does_question_require_event_frequency_reasoning(question, category, sentence):\n            return \"Yes.\"\n        else:\n            return \"No.\"\n```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of prompting with pseudo-code is novel and interesting. It inspires many future directions for exploration, such as pseudo-code prompt engineering/generation and instruction finetuning with pseudo-code. \n2. Pseudo-code prompting exhibits better performance than natural language instructions for LLMs pretrained on code.\n3. Extensive analysis is conducted to show the contribution of each component in pseudo-code prompting. ", "reasons_to_reject": "I don't have a clear reason to reject this paper.", "questions_to_authors": "1. How did you calculate the classification F1, and what's the random-guess baseline for classification tasks in Table 4? The macro F1 scores for NL instructions seem very low. Did you look at the token-level probability distribution, or did you simply measure the lexical overlap between the greedy output and the groundtruth? \n2. I suspect that one advantage of pseudo-code instruction is to clearly define the output space for classification. For example, the model is expected to output \"Yes/No\" instead of \"True/False\" for a binary problem, but the NL instruction doesn't make it clear. I wonder how often this happens. \n3. It would be helpful to include more complicated examples in appendix than Figure 1. For example, a pseudo-code instruction with step-by-step comments as mentioned in L454-457 will be very illustrative.", "paper_topic_and_main_contributions": "The paper introduces a new idea of prompting large language models with pseudo-code instructions. Compared with natural language, pseudo-code is often more specific in describing the task. Following this idea, the authors construct a new evaluation dataset by rewriting the natural language instructions into pseudo-code for 132 tasks from Super-NaturalInstructions. Experiments show that pseudo-code prompts lead to better performance when evaluated under zero-shot setting with models that have not been instruction-finetuned. Further analysis demonstrates that pseudo-code prompts are more powerful with models specifically trained on code. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "99eR9CaFyC", "reviewer": "Reviewer_ZMgi", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and for rating our paper strong and exciting. We include a response to a point raised by the reviewer.\n\n---\n\n### **Question 1**\n\nThe results have been reported on two model families – CodeGen and BLOOM at scales of 2-7B parameters. It remains uncertain if the findings would hold at larger model sizes.\n\n### **Answer 1**\n\nWe acknowledge that the trends could be different at larger parameter scales and we leave that to future work. Due to the large number number of tasks for evaluation running a larger model was computationally expensive, so we limited our evaluations to model size from 2-7B parameters. However, to help assuage concerns about generalization we include experiment on another model family - Falcon-7B. As can be seen in the table below, we see a similar trend where the model prompted with pseudo-code instructions outperforms model prompted with natural language instruction.\n\n|    Model   | Instruction  Format | Classification Tasks |          | QA Tasks | Generation tasks | All Tasks | \n|:----------:|:-------------------:|:--------------------:|:--------:|:--------:|:----------------:|:---------:| \n|            |                     | Macro F1             | Micro F1 | ROUGE-L  | ROUGE-L          | ROUGE-L   |\n| Falcon-7b  | Code Instructions   | 0.068                | 0.339    | 0.152    | 0.265            | 0.275     | \n|            | NL Instructions     | 0.017                | 0.206    | 0.172    | 0.273            | 0.242     |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors released a dataset containing 132 pseudo-code prompts spanning 28 different task types.\n\nPseudo-code instructions significantly improve performance compared to natural language instructions.\n\nDetailed ablation studies highlight the elements within pseudo-code that contribute to the observed improvements.\n", "reasons_to_reject": "The results have been reported on two model families – CodeGen and BLOOM at scales of 2-7B parameters. It remains uncertain if the findings would hold at larger model sizes.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper discusses the recent trend of using natural language instructions to harness the capabilities of large language models. Given the ambiguity in natural language, the authors explore the potential advantages of using less ambiguous prompt styles, specifically pseudo-code. They manually create a dataset of pseudo-code prompts for 132 tasks, which include classification, QA, and generative language tasks, sourced from the Super-NaturalInstructions dataset. The performance of these prompts is compared against their natural language counterparts using two LLM families: BLOOM and CodeGen. Results indicate that pseudo-code instructions lead to better outcomes, with an average increase of 7-16 points in F1 scores for classification tasks and a 12-38% improvement in aggregate ROUGE-L scores across all tasks. The study also delves into the factors contributing to this improvement, such as code comments, docstrings, and structural cues in pseudo-code.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gSgwiMGP5M", "reviewer": "Reviewer_wLJ1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and for rating our paper string and exciting! We are pleased that the reviewer found our experiments insightful. \n\nWe include a response to some points raised by the reviewer. \n\n---\n\n### **Question 1**\n\nThe experimented LLMs are not very recent models. I understand that the area is moving forward rapidly, but it would be much more interesting if the paper can also experiment with recent models with larger size like LLAMA.\n\n### **Answer 1**\n\nIndeed, the field is moving rapidly -  to address the concern about recency, we conducted an experiment with a relatively recent model such as Falcon-7B.  As can be seen in the table below, we see a similar trend where the model prompted with pseudo-code instructions outperforms model prompted with natural language instruction.\n\n|    Model   | Instruction  Format | Classification Tasks |          | QA Tasks | Generation tasks | All Tasks | \n|:----------:|:-------------------:|:--------------------:|:--------:|:--------:|:----------------:|:---------:| \n|            |                     | Macro F1             | Micro F1 | ROUGE-L  | ROUGE-L          | ROUGE-L   |\n| Falcon-7b  | Code Instructions   | 0.068                | 0.339    | 0.152    | 0.265            | 0.275     | \n|            | NL Instructions     | 0.017                | 0.206    | 0.172    | 0.273            | 0.242     | \n\n---\n\n### **Question 2**\n\nExperiments tasks are all from Super-NaturalInstructions, which is still very different from current popular LLM tasks like planning and open generation.\n\n### **Answer 2**\n\nAs noted in the paper we present our experiments on a wide variety of tasks including classification and QA and other language generation tasks. The goal of our experiments was to study if prompting in pseudo-code could help improve the \"instruction-following\" abilities of models. As our experiments suggest, prompting in pseudo code instructions does indeed help models and will therefore also benefit when used in LLM Prompt chains for larger applications where LLMs are allowed to influence decisions. We are unsure if the reviewer meant something else by \"planning\" -- we'd be happy to further clarify during the discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. An interesting attempt to replace natural language prompts with more structured code-based prompts. This can benefit many practical tasks.\n2. Promising experimental results and solid analysis on a large collection of tasks.\n3. The collected dataset can be beneficial for the NLP community.", "reasons_to_reject": "1. The experimented LLMs are not very recent models. I understand that the area is moving forward rapidly, but it would be much more interesting if the paper can also experiment with recent models with larger size like LLAMA.\n2. Experiments tasks are all from Super-NaturalInstructions, which is still very different from current popular LLM tasks like planning and open generation.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposed to use pseudo-code as instructions of LLMs instead of natural language instructions. It created a collection of pseudo-code instructions comprising of 132 NLP tasks. Experiment are done with two LLMs: CodeGen and BLOOM. Experimental results show that pseudo-code instructions can improve task performance over natural language instructions on most tasks. It also did many ablation studies and analysis on model size and prompt structures.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rmYeLUr01c", "reviewer": "Reviewer_FjEd", "summary": "", "questions": "A. I am somewhat confused about the wording of \"structure\" in section 4.4.2. Does \"structure\" mean \"the pseudo-code is a structured type of input, compared to unstructured natural language\"?\n\nB. How are the 2-shot experiments conducted for natural language instructions in table 8? Can you give an example of the prompt? It is weird that 2-shot NL instructions are far worse than 0-shot NL instructions.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and we are pleased to note that the reviewer found our work innovative. \n\n---\n\n### **Question 1**\n\nThe fairness of the experiments is somewhat questionable. In general, vanilla LLMs (i.e., LLMs that are not instruction-tuned) are not good zero-shot learners. Particularly, they tend to generate excessive outputs because they are basically doing language completion instead of solving classification/QA problems. Given the significant improvements in the experiments (e.g., table 4) after using pseudo-code instructions, I am wondering whether this is because in some tasks, pseudo-code instructions have more detailed specifications or constraints on the format of the output. As far as I know, the specification of output format in the instruction may significantly impact the zero-shot performance of a vanilla LLM.\n\n### **Answer 1**\n\nWe agree it would indeed be interesting to assess the performance of instruction-tuned models  -- unfortunately, most instruction tuned models have seen SuperNatInstructions or versions of it and creating new instruction datasets and their pseudo-code prompts is beyond the scope of this work. The input format and the output formats of prompts can indeed affect zero-shot performance -- we include samples of prompts and all our other experimental settings for reference in this rebuttal. \n\nFor all prompts, we use new-line character as the delimiter and extract the text till the first occurrence of new line character as the generated text to ensure models prone to excessive text generation are not unfairly penalized.  \n\n---\n\n### **Question 2**\n\nAnother setting that I found questionable is 2-shot prompting. In section 4.4.2, the authors mentioned using 2-shot natural language examples without any natural language instructions on vanilla LLMs. Why are instructions not used? I think a consensus in LLMs is that few-shot instruction-based prompting is better than zero-shot prompting, at least this should be the case for the natural language instructions. However, in the reported results, 2-shot natural language prompting is far worse than its 0-shot counterpart.\n\n### **Answer 2**\n\nOur goal in Section 4.4.2 was to specifically assess the contribution made by structure in the input prompts -- for pseudo code prompts we have many elements to experiment with (function declaration, definition, with and without two-shot examples, etc). But for NL text, we could only consider an ablation where the instruction is removed and the structured 2-shot example is retained. \n\nFurther, we would like to highlight that we indeed experimented with full two-shot prompting as well  --  Please see Table 8 in the appendix of the paper. The reviewer has correctly highlighted that the lower performance of two-shot is a bit surprising - however as stated in lines 1298-1301 of the paper, lower performance with few-shot as compared to zero-shot has been noted by other works as well.\n\n---\n\n### **Question 3**\n\nThis paper lacks examples of (1) pseudo-code instructions, and (2) illustrations of different prompts used in section 4.4.2 (it is hard to imagine the difference between these ablation prompts without any examples). The only example is figure 1, but there is no comparison between the pseudo-code instruction and the natural language instruction.\n\n### **Answer 3**\n\nWe apologize for the oversight --- we will update the paper with the examples.\n\n**Example 1 (Pseudo-code instruction)**:\n```python\ndef generate_sentiment(tweet: str) -> str: \n    \"\"\"Given a text from tweet, the task in get the sentiment about the tweet. For a positive sentiment tweet return \"positive\" else return \"negative\". \n    Parameters: \n        tweet: text from tweet \n        Returns: \n                str:answer to the question \n    \"\"\" \n    # predict the sentiment (positive or negative) \n    if get_sentiment(tweet) == \"positive\": \n        return \"positive\" \n    else: \n        return \"negative\" \n\n>>> generate_sentiment(\"its sad that the rats are becoming aggressive against the guinea pigs. I have to seperate them. \") \n```\n\n**Example 2 (Pseudo-code instruction with 2-shot examples)**:\n```python\ndef generate_sentiment(tweet: str) -> str: \n    \"\"\"Given a text from tweet, the task in get the sentiment about the tweet. For a positive sentiment tweet return \"positive\" else return \"negative\". \n    Parameters: \n        tweet: text from tweet \n        Returns: \n                str:answer to the question \n    \"\"\"\n    # predict the sentiment (positive or negative) \n    if get_sentiment(tweet) == \"positive\": \n        return \"positive\" \n    else: \n        return \"negative\" \n\n>>> generate_sentiment(\"tormented by the quickened blood of rots\") \n\"negative\"\n\n>>> generate_sentiment(\"radiant as moses from the mount, he stood\") \n\"positive\"\n\n>>> generate_sentiment(\"its sad that the rats are becoming aggressive against the guinea pigs. I have to seperate them. \") \n```\n\n**Example 3 (Pseudo-code instruction with no docstring)**:\n ```python\ndef generate_sentiment(tweet: str) -> str: \n    # predict the sentiment (postive or ne \n    if get_sentiment(tweet) == \"positive\": \n        return \"positive\" \n    else: \n        return \"negative\" \n\n>>> generate_sentiment(\"its sad that the rats are becoming aggressive against the guinea pigs. I have to seperate them. \") \n```\n\n**Example 4 (Pseudo-code instruction with no docstring 2-shot examples)**:\n```python \ndef generate_sentiment(tweet: str) -> str: \n    # predict the sentiment (positive or negative) \n    if get_sentiment(tweet) == \"positive\": \n        return \"positive\" \n    else: \n        return \"negative\" \n\n>>> generate_sentiment(\"tormented by the quickened blood of rots\") \n\"negative\" \n\n>>> generate_sentiment(\"radiant as moses from the mount, he stood\") \n\"positive\"\n\n>>> generate_sentiment(\"its sad that the rats are becoming aggressive against the guinea pigs. I have to seperate them. \") \n```\n\n**Example 5 (Pseudo-code instruction with function definition only)**:\n ```python\ndef generate_sentiment(tweet: str) -> str: \n\n>>> generate_sentiment(\"its sad that the rats are becoming aggressive against the guinea pigs. I have to seperate them. \") \n```\n\n**Example 6 (Pseudo-code instruction with function definition only and 2-shot examples)**:\n```python\ndef generate_sentiment(tweet: str) -> str: \n \n>>> generate_sentiment(\"tormented by the quickened blood of rots\") \n\"negative\" \n \n>>> generate_sentiment(\"radiant as moses from the mount, he stood\") \n\"positive\"\n\n>>> generate_sentiment(\"its sad that the rats are becoming aggressive against the guinea pigs. I have to seperate them. \") \n```\n\n**Example 7 (Natural instruction)**:\n```text\nIn this task, you need to identify the sentiment of the given sentence as one of 'positive' or 'negative'. \n\nInput: its sad that the rats are becoming aggressive against the guinea pigs. I have to seperate them. \nOutput:  \n```\n\n**Example 8 (Natural instruction with 2-shot examples)**:\n```text\nIn this task, you need to identify the sentiment of the given sentence as one of 'positive' or 'negative'. \n\nInput: tormented by the quickened blood of rots \nOutput: negative \n\nInput: radiant as moses from the mount, he stood \nOutput: positive \n\nInput: its sad that the rats are becoming aggressive against the guinea pigs. I have to seperate them. \nOutput:\n```\n\n**Example 9 (Natural instruction with docstring)**:\n```text\nIn this task, you need to identify the sentiment of the given sentence as one of 'positive' or 'negative'. \n    \"\"\"Given a text from tweet, the task in get the sentiment about the tweet. For a positive sentiment tweet return \"positive\" else return \"negative\". \n    Parameters: \n        tweet: text from tweet \n        Returns: \n                str:answer to the question \n    \"\"\" \n \nInput: its sad that the rats are becoming aggressive against the guinea pigs. I have to seperate them. \nOutput:\n```\n\n**Example 10 (Natural instruction with docstring and 2-shot examples)**:\n```text\nIn this task, you need to identify the sentiment of the given sentence as one of 'positive' or 'negative'. \n    \"\"\"Given a text from tweet, the task in get the sentiment about the tweet. For a positive sentiment tweet return \"positive\" else return \"negative\". \n    Parameters: \n        tweet: text from tweet \n        Returns: \n                str:answer to the question \n    \"\"\" \n\nInput: tormented by the quickened blood of rots \nOutput: negative \n\nInput: radiant as moses from the mount, he stood \nOutput: positive \n\nInput: its sad that the rats are becoming aggressive against the guinea pigs. I have to seperate them. \nOutput: \n```\n \n**Example 11 (No instruction in pseudocode)**:\n```text\n>>> generate_sentiment(\"its sad that the rats are becoming aggressive against the guinea pigs. I have to seperate them. \") \n```\n\n**Example 12 (No instruction in pseudocode and 2-shot examples)**:\n```text\n>>> generate_sentiment(\"tormented by the quickened blood of rots\") \nnegative \n\n>>> generate_sentiment(\"radiant as moses from the mount, he stood\") \npositive \n\n>>> generate_sentiment(\"its sad that the rats are becoming aggressive against the guinea pigs. I have to seperate them. \") \n```\n\n**Example 13 (No instruction in pseudocode and generic function call)**:\n```text\n>>> func(\"its sad that the rats are becoming aggressive against the guinea pigs. I have to seperate them. \") \n```\n\n---\n\n### **Question 4**\n\nI am somewhat confused about the wording of \"structure\" in section 4.4.2. Does \"structure\" mean \"the pseudo-code is a structured type of input, compared to unstructured natural language\"?\n\n### **Answer 4**\n\nYes, by structure, we refer to programming language syntax. This can be seen in the examples listed above.\n\n---\n\n### **Question 5**\n\nHow are the 2-shot experiments conducted for natural language instructions in table 8? Can you give an example of the prompt? It is weird that 2-shot NL instructions are far worse than 0-shot NL instructions.\n\n### **Answer 5**\n\nPlease refer to example 8 above for 2-shot prompting with NL instructions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of leveraging pseudo-code is novel. The motivation is that pseudo-code are less ambiguous, more concise and clearer than natural language instructions. The motivation is clear and innovative.\n2. The authors provided a set of pseudo-code instructions on 132 tasks, which could be beneficial for future research.\n3. The authors proved the advantage of pseudo-code instructions in both code-based LLM (CodeGen) and text-based LLM (BLOOM).\n4. The authors analyzed the components in pseudo-code instructions and highlighted the critical role of task-indicative function names, docstrings and comments.", "reasons_to_reject": "1. The fairness of the experiments is somewhat questionable. In general, vanilla LLMs (i.e., LLMs that are not instruction-tuned) are not good zero-shot learners. Particularly, they tend to generate excessive outputs because they are basically doing language completion instead of solving classification/QA problems. Given the significant improvements in the experiments (e.g., table 4) after using pseudo-code instructions, I am wondering whether this is because in some tasks, pseudo-code instructions have more detailed specifications or constraints on the format of the output. As far as I know, the specification of output format in the instruction may significantly impact the zero-shot performance of a vanilla LLM.\n2. Another setting that I found questionable is 2-shot prompting. In section 4.4.2, the authors mentioned using 2-shot natural language examples without any natural language instructions on vanilla LLMs. Why are instructions not used? I think a consensus in LLMs is that few-shot instruction-based prompting is better than zero-shot prompting, at least this should be the case for the natural language instructions. However, in the reported results, 2-shot natural language prompting is far worse than its 0-shot counterpart.\n3. This paper lacks examples of (1) pseudo-code instructions, and (2) illustrations of different prompts used in section 4.4.2 (it is hard to imagine the difference between these ablation prompts without any examples). The only example is figure 1, but there is no comparison between the pseudo-code instruction and the natural language instruction.", "questions_to_authors": "A. I am somewhat confused about the wording of \"structure\" in section 4.4.2. Does \"structure\" mean \"the pseudo-code is a structured type of input, compared to unstructured natural language\"?\n\nB. How are the 2-shot experiments conducted for natural language instructions in table 8? Can you give an example of the prompt? It is weird that 2-shot NL instructions are far worse than 0-shot NL instructions.", "paper_topic_and_main_contributions": "This paper focused on using pseudo-code as instructions when prompting large language models (LLMs), instead of the traditional approach which used natural language prompts. The authors stated that using pseudo-code as instructions is much better than natural language under a zero-shot prompting setting.\n\nContributions:\n1. The authors proposed to use pseudo-code as instructions instead of traditional natural language instructions.\n2. They provided pseudo-code instructions on 132 tasks in the Super Natural Instructions benchmark.\n3. Experimental results on two LLM families (BLOOM & CodeGen) proved the advantage of using pseudo-code instructions over natural language instructions under the zero-shot setting. The authors also showed the importance of using docstrings, comments and function declarations.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates if you can write prompts for LLMs in pseudocode rather than natural language and investigates how well this works. The authors discussion period had a spirited discussion with the authors and the reviewers where the reviewers' concerns were mostly resolved; this discussion shows that the results are interesting and surprising and left the reviewers convinced that the results were indeed sound.
Decision: Accept-Main