TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Tasks
Authors: Shubhra Kanti Karmaker Santu
Dongji Feng
Venue: EMNLP 2023 Findings (2023)
Keywords: Large Language Models
Prompt Engineering
Prompt Taxonomy
Benchmarking
TL;DR: Taxonomy of LLM Prompts

Abstract: While LLMs have shown great success in understanding and generating text in traditional conversational settings, their potential for performing ill-defined complex tasks is largely under-studied and yet to be benchmarked.  However, conducting such benchmarking studies is challenging because of the large variations in LLMs' performance when different prompt types/styles are used and different degrees of detail are provided in the prompts. To address this issue, this paper proposes a general taxonomy that can be used to design prompts with specific properties in order to perform a wide range of complex tasks. This taxonomy will allow future benchmarking studies to report the specific categories of prompts used as part of the study, enabling meaningful comparisons across different studies. Also, by establishing a common standard through this taxonomy, researchers will be able to draw more accurate conclusions about LLMs' performance on a specific complex task.

TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Tasks
Shubhra Kanti Karmaker ("Santu")) Dongji Feng
Big Data Intelligence (BDI) Lab Department of Computer Science & Software Engineering Auburn University, Alabama, USA (sks0086, dz/00033 @ @auburn.edu
Abstract
In recent years, researchers have spent a signif- icant amount of effort proposing different ways of designing "appropriate" prompts. For example, Brown et al. (2020) showed a standard prompting technique with question-answer pairs that can re- sult in few-shot effect. Researchers also explored other prompt design techniques such as Chain-of- thought (CoT) (Wei et al., 2022), Reasoning and Acting (ReAct) (Yao al., 2022), and other tech- niques (Kojima al., 2022; Madaan and Yazdan- bakhsh, 2022: Press et al.. 2022) in terms of im- proving the reasoning and acting of LLMs in solv- ing Question-Answering tasks. Meanwhile, Kim et al. (2023) proposed prompting scheme where the agent recursively criticizes and improves its output (RCI) to solve task. However, these ex- periments primarily emphasized the utilization of diverse prompts to evaluate the ability of LLMs perform "well-defined" NLP tasks, while studies with diverse prompts for ill-defined complex tasks are still rare, if not nonexistent.
While LLMs have shown great success un- derstanding and generating text traditional conversational settings, their potential for per- forming ill-defined complex tasks largely under-studied and yet be benchmarked. How- ever, conducting such benchmarking studies challenging because in LLMs' performance when different prompt types/styles are used and different degrees of detail are provided in the prompts. To address this issue, this paper proposes a general taxon- omy that can be used to design prompts with specific properties in order perform wide range of complex tasks. This taxonomy will allow future benchmarking studies to report the specific categories of prompts used as part of the study, enabling meaningful comparisons across different studies. Also, by establishing a common standard through this taxonomy, re- searchers will be able to draw more accurate conclusions about LLMs' performance on specific complex task.
large variations
While conducting multiple benchmarking stud- ies with various LLMs for complex tasks seems interesting and compelling, conducting such stud- ies is challenging because of the large variations in LLMs' performance when different prompt types/styles are used and different degrees of detail are provided in the prompts, especially in case of complex tasks. In this paper, we exclusively focus on understanding LLMs' potential for performing complex tasks that are mostly: 1) ill-defined, 2) abstract goal-oriented, 3) highly dependent on sub- jective interpretation, and 4) very hard to evalu- ate quantitatively (Khot et al., 2022; Press et al., 2022). These complex tasks often involve mul- tiple steps/sub-tasks and designing "appropriate" prompts for such tasks indeed challenging there is no single rule book to follow in these cases (Zelikman al., 2022; Nye et al., 2021). )) A further complication arises if we want to compare two independent benchmarking studies targeted
1 Introduction
Recently, conversational Large Language Mod- els (LLMs) such GPT-3 (Brown et al., 2020), Bard (Thoppilan et al., 2022), LLaMA (Tou- vron et al., 2023), BLOOM (Scao al., 2022), PaLM (Chowdhery et al., 2022), etc. have demon- strated exceptional performance in wide range of popular natural language processing (NLP) tasks (Bubeck et al., 2023; Dai et al., 2022; Du et al., 2022; Smith et al., 2022). Prompt, as a stimu- lator, refers to textual input provided to the LLMs with the intention of guiding its output toward a specific task. Unsurprisingly, the quality and ef- fectiveness of the prompt can greatly influence the performance of the LLMs for a particular task, and therefore, designing appropriate prompts with the right amount of detail has become more important than ever (Liu et al., 2023; Han et al., 2022).

---

towards the same goal (complex) task. Such a com- plication arises because, for a given complex task and a particular LLM, the performance of the LLM can drastically vary when different types/styles of prompts are fed to it. Indeed, the exact details in- cluded in the prompt play a big role in how LLMs will perform in solving the goal complex task. This indeed creates problem for evaluation and bench- marking purposes if an apple-to-apple comparison is not made in terms of the prompts that are pro- vided to the L.Mss. In other words, just reporting accuracy numbers for LLMs without specifying the finer details of the prompts used in the experiments makes comparisons across LLMS meaningless.
tasks usually involve multiple steps/sub-tasks re- quiring higher levels of semantic understanding, planning, reasoning, and generation of natural lan- guage (Fu et al., 2022). These tasks often require the model to go beyond simple pattern recognition or retrieval of information and involve simulating more advanced cognitive abilities. In fact, differ- ences in prompis along several key factors can have significant impact the accuracy and perfor- mance of LLMs in complex tasks. Below, we list those key factors of prompt designing.
Level of Details in Task Specification: The prompt directive should define the task or ques- tion being asked in sufficient detail (White et al., 2023; Ouyang et al., 2022). For complex tasks, providing a detailed directive typically means following the general guidelines below.
Unfortunately, every complex task is different, and so are the prompts users can try to perform the task; therefore, a general taxonomy that can categorize these diverse kinds of prompts using a single standard/taxonony has now become a press- ing need. The main contribution of this paper is to introduce one such general taxonomy (we name it TELeR) that can be used by any benchmark- ing study that leverages LLMs to perform some complex task. The major benefit of adopting our proposed TELeR taxonomy is that will facili- tate more meaningful comparisons among multi- ple LLMs in terms of their performances across various complex tasks reported by multiple inde- pendent groups of rescarchers/developers and, thus, help derive more accurate conclusions. TELeR will achieve this goal by grounding different types of prompts into common standard and allowing an apple-to-apple comparison across different prompt categories using the same standard. As such, this taxonomy will serve as an important tool to es- tablish a common consensus around the ota the-art LLM performance for performing complex tasks.
Specifying clear goals helps guide the language model's understanding of the task or question hand, increasing the chances of receiving the desired information or output. Therefore, one should avoid vague or ambiguous terms that can lead to inaccurate or irrelevant responses (Jiang et al., 2022).
Clear Goal(s):
Associated Data: Some prompts require LLMs to perform a particular task on the data provided by the user in real-time, whereas some prompts do not provide any data and rely on the pre-trained model to generate sponse based on the background knowledge it has already learned. It is very important make explicit in LLM prompts whether the user is providing data as part of the prompt or not, and if yes, which part is data vs. directive.
Distinet Sub-Tasks: By definition, complex tasks consist of multiple steps/ sub-tasks. It is important to mention these distinct sub-tasks in the prompt clearly as separate bullet points or numbered items. This visual organization helps LLMs recognize the distinct sub-tasks and respond to each one individually.
2 Prompt Engineering for Complex Tasks
"Prompt Engineering" is crucial technique for maximizing the utility of LLMs in various tasks (Zhou et al., 2022). It involves crafting and revising the query or context in such a way that it elicits the desired response or behavior from LLMS (Brown et al.. 2022). In practice, prompt en- gineering is an iterative process requiring multiple trial and emrrr runs (Shao et al., 2023).
Evaluation Criteria/Few-Shor Examples: LLMs can benefit from example-basee learn- ing, where prompts include specific examples of the desired ouputt pairs (few-shot ex- amples) (Brown et al., 2020). By incorporating relevant examples, users can train the model to follow specific patterns or mimic desired behavior. In the absence of explicit few-shot examples, prompts may describe what consti-
Prompt Engineering becomes even more criti- cal and challenging in case of performing complex tasks (Tan et al., 2023) with LLMs, as complex

---

Prompt Directive Data
Directive-Type: Turn, Expression, Level of Details, Role>
Low Detail
Level 0
directive, Just Data
Level 1
Simple one-sentence directive expressing the high-level goal
(System) Role: Defined/Undefine
Multi-sentence (peragraph -ttyle directive expressing the high-level goal and the sub-tasks that need to be performed to achieve the goal
2
Moderate Detail
Complex (bulleteedlist--tyye)) directive expressing the high-level goal along with detailed bulleted list of sub-tasks to be performed
Expression: Instruction /Question
Level 3
Turn: Single Multiple
complex directive that includes the following: 1) Description of High-level goal, 2] detailed bulleted list of sub tasks, guideline how LLM output be evaluated/ Few-Shot Examples.
Level 4
complex directive that includes the following: goal, 2) detailed bulleted list sub-tasks, A guideline will evaluated/ Few-Shot Examples, 4) Additional relevant information gathered via retrieval-based techniques.
1) Description of High-level
output
Level 5
High Detail
complex directive that includes the following: 1) Description of High-level goal, 2] detailed bulleted list of sub-tasks, guideline how LLM output will evaluated/ Few-Shot Examples, 4) Additional relevant information gathered retrieval-based techniques, 5) An explicit statement asking LLM explain its own output.
Level
Figure 1: Proposed Prompt Taxonomy: TELeR (<Turn, Expression, Level of Details, Role>)
tutes "good" response versus what would make response "bad". For example, impos- ing word limits, specifying output formats, restricting particular data sources etc.
can help it make more informed and precise de- cisions. Different prompts may provide varying levels of context, which impact the accuracy of the model's responses.
Additional Information Fetched via Infor- mation Retrieval Techniques: Additional in- formation fetched via Information Retrieval Techniques enhances Large Language Mod- els (LLMMs) by providing them with real-time and contextually relevant data, improving their ability to generate up-to-date and accurate re- sponses. This helps LLMs stay current and adapt to evolving information, making them more valuable for various applications, such chatbots and search engines.
Expression Style: Directives can be expressed primarily in two styles: 1) Questions and 2) In- structions. For complex tasks, one may frame directives either a of questions or instruc- tions based on their preference/application need. Interaction Style: Prompts for complex tasks usually involve long text descriptions, often in- cluding details of associated sub-tasks be performed step-by-step. Therefore, some users may prefer to provide these step-by-step instruc- tions in a multi-turn fashion (like real dialog), whereas others may prefer to provide all the de- tails at a single turn. Such one-turn vs. multi-turn prompting can also impact the performance of an LLM significantly the dialog history becomes different in generation time for these two cases.
Explanation//Justiicatiii Seeking: LLMs are not only good at generating textual re- sponses, but they can also generate explana- tions for their output if an explanation is sought as part of the prompt explicitly (Rajani al., 2019). This is indeed valuable when a user wants understand why the LLM generated a particular output.
3 Proposed TELeR Taxonomy
Our proposed taxonomy is based on the key fac- tors diseussed in section 2, whose variations can lead to different outcomes while using LLMs to perform complex tasks. To begin with, we rep- resent each prompt as combination of Directive and Data. Assuming Data is fixed for a given goal task, the difference between two prompts essen-
Defining Context and Role: Including relevant context and background information as part of the prompt can provide the model with comple- mentary information in order to generate more accurate responses. For complex tasks, giving the model a clear understanding of the context

---

tially originates from the specifics/details of direc- tives they include. More specifically, we propose to categorize LLM prompts for complex tasks along the following four dimensions.
informed decisions and understanding the consen- sus of expert opinions on a submitted manuscript. Given the explosion the number of research manuscript submissions in recent years and the huge challenge in peer-reviewing timeline manage- ment (Bansal et al., 2022c; Karmaker Santu et al., 2018) 8), it is tempting to leverage LLMs to assist edi- tors (for journals)/ program chairs (for conferences) in preparing a first-cut draft of the meta-review for each manuscript from the individual review texts provided by relevant expert reviewers.
1. Turn: Based the number of turns used while prompting LLMs in order to perform a complex task, prompts can be either single or multi-turn.
2. Expression: Based on the expression style of the overall directive as well as the associated sub-tasks, prompts can be either question-style or instruction-style.
To demonstrate the applicability of the proposed TELeR taxonomy for categorizing different kinds of prompts for this complex task, we show some example prompts with varying levels of detail be- low. For simplicity, we show examples of only single-turn question-style prompts where the sys- tem role undefined. Other variations are left out due to lack of space. We also assume that three reviewers have reviewed the manuscript and pro- vided their comments (R1, R2, R3) as the data for the meta-review generation task.
3. Role: Based on whether proper system role is defined in the LLM system before providing the actual prompt. prompts can be categorized as either system-role defined or undefined.
4. Level of Details: Based on the degree of detail provided in the directive, we divided prompts into seven distinct levels (levels 0-6). Here, the degree of detail is determined by the presence or absence of different aspects like clear goals, sub-task division, explanation seeking, few-shot examples, etc. By definition, Level "0" means minimal details, i.e., no aspects/no directive, while Level "6" means the highest level of de- tails where the directive includes clear goals, distinct sub-tasks/steps, an explicit requirement of explamation/jusiffccttton, well-defined criteria for evaluation, additional information fetched via information retrieval techniques and/or few- shot examples. See Figure I for the exact defini- tions of each of these levels in our taxonomy.
Level 0 Prompt: <R1, R2, Ry>
Prepare a meta-review by sum- marizing the reviewer comments: <R1, R2, Ra> Prepare a mela-review by summarizing the following reviewer com- ments. The final output should highlight the core contributions of the manuscripi, common strengthshweakkeesee mentioned by multiple re- viewers, suggestions for improvement, and miss- ing references (if any). The review texts are pro- vided below: <B,, R2. R3>
Level Prompt:
Level 2 Prompt:
Because we used the following four factors, i.e., Turn, Expression, Level of Details and Role, to define our taxonomy, we name it as TELeR. The overall taxonomy is pictorially depicted in Figure 1.
Level 3 Prompt: Prepare a meta-review by answering the following questions from the re- viewer comments (provided after the questions).
4 Two Example Use Cases
1. Based on the reviewer's 's comments, what are the core contributions made by the authors?
this section, we present two interesting use cases of the proposed TELeR taxonomy that involve LLMs for performing a complex task: 1) gener- ating a meta-review from peer-reviewer comments on scholarly work, and 2) Combining multiple alternative narratives into single braided one.
2. What are the common strengths of this work, as mentioned by multiple reviewers?
3. What are the common weaknesses of this work, as highlighted by multiple reviewers?
4. What suggestions would you provide for im- proving this paper?
4.1 Use-Case 1: Meta-Review Generation
5. What are the missing references mentioned by the individual reviews?
Meta-reviewing is critical part of the scientific peer-review process and is generally a complex task that involves summarizing expert reviews from multiple reviewers (Shen et al., 2022, 2023). It is a very important and pertinent process for making
The review texts are below: <R1, R2, R3>
Level 4 Prompt: "Level Prompt" "A good output should be coherent, highlight major strengths/issues mentioned by multiple reviewers,

---

be less than 400 words in length, and finally, the response should be in English only".
Level 3: Braid a single coherent story from the following alternative narratives provided later by performing the following tasks.
Level 5 Prompt: "Level 4 Prompt" + "Below are additional information relevant your goal task. <Information Fetched using Information Retrieval Techniques>",
Extract overlapping clause pairs from both narratives and paraphrase them.
2. Extract unique clauses from each narrative and identify the interesting ones.
Level 6 Prompt: "Level 5 Prompt"+ + "Justify your response in detail by explaining why you made the choices you actually made".
3. Extract conflicting clause pairs conveyed in both narratives and resolve the conflict.
4. Generate paragraphs from overlapping- clauses and merge them into a single document.
4.2 Use Case 2: Narrative Braiding
Narrative braiding, also known as "interweaving" or "multi-perspective storytelling" is a literary tech- nique that involves the parallel telling of multi- ple storylines that eventually converge and inter- sect (Bancroft, 2018). This technique is often used in novels, short stories, films, and television shows create a complex and engaging narrative.
Reorder sentences of the merged document into a detailed, coherent story.
6. Summarize the detailed story into a concise, braided narrative.
The alternative narratives are below: <N1, N2>
Level 4 Prompt: "Level 3 Prompt" + "A good output should be coherent, highlight overlapping- unique-conflitting information provided by indi- vidual narratives, be less than 1000 words in length, and in English language only".
Narrative braiding is indeed a complex task to perform, even for humans, let alone computers, as it requires careful planning and execution to ensure that each storyline is fully developed and that the different strands of the narrative are balanced and complement each other. When done well, narrative braiding create rich and engaging story that keeps readers or viewers invested. From the recent promising results of language models generating high-quality controlled text (Bansal et al., 2022a,b), it is quite intuitive to test the LLM's performance in narrative braiding tasks.
Level 5 Prompt: "Level 4 Prompt" "Below are additional information relevant to your goal task. <Information Fetched using Information Retrieval Techniques>",
Level 6 Prompt: "Level 5 Prompt" + "Provide justification for your response in detail by ex- plaining why your response contains certain in- formation and discards other information of the inputs".
Now, we show how one can use the proposed TELeR taxonomy to categorize different types of prompts for the narrative braiding task. This time, we show examples of only single-turn instruction- style prompts with system roles undefined. Other variations are left out due to lack of space. We also assume two alternative narratives are available that describe the same event as our data for the braid- ing task, and the goal is to create final braided narrative from the two input narratives, N1 and N2.
Final Words
In this paper, we emphasize the importance of standardized taxonomy for LLM prompts targeted towards solving complex tasks and, subsequently, propose such a general taxonomy, i.e., TELeR, which can serve as unified standard for compar- ing and benchmarking LLMs' performances re- ported by muitiple independent research studies. We urge the community to use the TELeR taxon- omy for designing prompts in their future work and report the specific categories of prompts they exper- imented with in their manuscripts. This will enable more mcaningful comparisons among LLMs and, thereby, help to derive more accurate conclusions from multiple independent studies. This, in turn. will help the community to reach a consensus on state-of- thae-off-the-arl LLM performances more accurately and faster than otherwise.
Level 0: <N1. N₂>
Level 1: Braid single coherent story from the following alternative narratives: <N1, N₂>
Level 2: Braid single coherent story from the following alternative narratives. The final nar- rative should highlight the common information provided by both narratives, interesting, unique information provided by each individual narra- tive, and conflicting information (if any) con- veyed in these narratives. The input alternative narratives are provided below: <N1, N₂>

---

6 Limitations
pages 6193-6207, Gyeongju. Republic of Korea. In- ternational Committee on Computational Linguistics.
The proposed TeLER taxonomy is exclusively ap- plicable to LLM prompts targeted toward solving complex task. This taxonomy, especially the seven levels, does not apply to simple tasks. As such, TeLER taxonomy will be most useful to researchers and developers who are conducting applied LLM research and development that is focused on per- forming complex tasks.
Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. 2022. What does mean for a language model preserve privacy? In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 2280-2292.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Also, the TeLER taxonomy should not be con- sidered ultimate taxonomy for LLM prompts, and further extensions of this taxonomy tainly possible and actually desired. Having said that, the TeLER taxonomy is actually very general and can be easily extended by adding more dimen- sions for categorization as deemed necessary by the target application.
cer-
Sébastien Bubeck, Varun Chandrasekaran, Ronen EI- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lund- berg, et al. 2023. Sparks of artificial general intelli- gence: Early experiments with gpt-4. arXiv preprint arXiv:2303.122 12712.
7 Acknowledgements
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.0233 02.11.
This work has been partially supported by the Na- tional Science Foundation (NSF) Standard Grant Award #2302974 and Air Force Office of Scientific Research Grant/Cooperative Agreement Award 1FA9550--331-04266 We would also like to thank Auburn University College of Engineering and the Department of CSSE for their continuous support through Student Fellowships and Faculty Startup Grants.
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. 2022. Why can gpt learn context? language models secretly perform gradient descent as meta optimizers. arXiv preprint arXiv:22 12. 10559.
Nan Du, Yanping Huang. Andrew M Dai, Simon Tong, Dmitry Lepikhin. Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. 2022. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pages 5547-5566 PMLR.
References
Corinne Bancroft. 2018. The braided narrative, Narra- rive, 26(3):262-281.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2022. Complexity-basee prompt- ing for multi-step reasoning. arXiv preprint arXiv:2210.00720.
Naman Bansal, Mousumi Akter, and Shubhra Kanti Kar- maker Santu. 2022a. Learning to generate overlap summaries through noisy synthetic data. In Proceed- ings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 11765-11777 -1177, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2022. Ptr: Prompt tuning with rules for text classification. Al Open, 192...99..
Ellen Jiang, Edwin Toh, Alejandra Molina, Kristen Ol- son, Claire Kayacik, Aaron Donsbach, Carrie J Cai, and Michael Terry. 2022. Discovering the syntax and strategies of natural language programming with gen- erative language models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1-19.
Naman Bansal, Mousumi Akter. and Shubhra Kanti Karmaker Santu. 2022b. SEM-fl: automatic way for semantic evaluation of nulti-narrative overlap summaries at scale. In Proceedings of the 2022 Con- ference Empirical Methods in Natural Language Processing, pages 780-792, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Shubhra Kanti Karmaker Santu, Chase Geigle, Dun- can Ferguson, William Cope, Mary Kalantzis, Du- ane Searsmith, and Chengxiang Zhai. 2018. Sofsat: Towards setlike operator based framework for se- mantic analysis text. ACM SIGKDD Explorations Newsletter, 20(2):21-30
Naman Bansal, Mousumi Akter, and Shubhra Kanti Karmaker Santu. 2022c. Semantic overlap summa- rization among multiple altemative narratives: An exploratory study. In Proceedings of the 29th Inter- national Conference on Computational Linguistics,

---

Tushar Khot, Harsh Trivedi, Matthew Finlayson. Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sab- harwal, 2022. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406.
Chenhui Shen, Liying Cheng, Ran Zhou, Lidong Bing. Yang You, and Luo Si. 2022. Mred: A meta- review dataset for structure-controllaaeee text genera- tion. Findings of the Association for Computational Linguistics: ACL 2022, pages 2521-2535.
Geunwoo Kim, Pieme Baldi, and Stephen McAleer. 2023. Language models can solve computer tasks. arXiv preprint arXiv:2303. 17491.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022. Using deep- speed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201. 1990..
Takeshi Kojima, Shixiang Shane Gu. Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. arXiv preprint arXiv:2205. 11916.
Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin Qi. 2023. Evalu- ation of chatgpt as question answering system for answering complex questions. arXiv preprint mrXiv:2303.07992
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang. Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods natural language processing. ACM Computing Surveys, 1-35).
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Tze Cheng. Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applica- tions. arXiv preprint arXiv:2201.088 088399.
Aman Madaan and Amir Yazdanbakhsh. 2022. Text and patterns: For effective chain of thought, it takes two to tango. arXiv preprint arXiv:2209.00 07686.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz. Maarten Bosma, David Luan, et al. 2021. Show your work: Scratch- pads for intermediate computation with language models. arXiv preprint arXiv:2112.00
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13971.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances Neural Information Processing Systems, 35:27730-27744.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe L.e, and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11003.
Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olca, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smithe and Douglas C. Schmidt. 2023. A prompt pattern catalog enhance prompt engineer- ing with chatgpt. arXiv preprint arXiv:2302. 1382.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionallit gap language models. arxiv preprint 21..::2220.003555.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019, Explain your- self! leveraging language models for commonsense reasoning. arXiv preprint arXiv: 1906.02361.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran. Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arxiv:2210.03625
Teven Le Scao, Angela Fan, Christopher Akiki, EI- li Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-aceess multilingual language model. arXiv preprint arXiv:2211.085
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Good- man. 2022. Star: Bootstrapping reasoning with rea- soning. Advances in Neural Information Processing Systems, 35:15476-15488
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. arXiv preprint arXiv.2111.01110.
Nan Shao, Zefan Cai, Chonghua Liao, Yanan Zheng. Zhilin Yang, et al. 2023. Compositional task rep- resentations for large language models. In The Eleventh International Conference on Learning Rep- resentations.
Chenhui Shen, Liying Cheng, Yang You, and Li- dong Bing. 2023. A hierarchical encoding-decodiin scheme for abstractive multi-document summariza- tion. arXiv preprint 2305. 08503.


Reviews:
[{"review_id": "TOtF3rJiGO", "reviewer": "Reviewer_gD5n", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- \"It only showed the proposed Taxonomy. I wanted to see some sort of examples showing how the TELeR really helped evaluate performance across the LLMs with a complex question. The given example is about generating prompts. It would be great to see it is really helped by feeding these to LLMs with results reported.\"\n\n  - Thank you for your comment. We want to highlight that this is a short theme track \"position\" paper. Our \"position\" here is that the current practice of deploying prompt engineering techniques in LLMs without a structured taxonomy is problematic, and we need a standard prompting taxonomy to enable a fair comparison of LLMs and, thus, derive more accurate and meaningful conclusions from different independent studies. We agree that it would be great to see if the taxonomy is really helpful by creating prompts and feeding them to LLMs with results reported. However, a 4-page limit is impractical to report such studies with details of different instances of prompts created.  In fact, we have already used this taxonomy in multiple use cases and found it to be applicable to a wide range of complex tasks. We also found that Levels 3 and 4 are often performing better than other levels. However, we cannot point out/cite those studies as it will break the anonymity rules. \n\n- \"The coverage of proposed frameworks. It would be great to see some types of recall numbers of the proposed framework. Even though this paper admitted it only intended to cover complex problems, I wanted to see what types of complex problems the proposed framework can cover.\"\n\n  - The proposed TeLER taxonomy is actually very general and can be applied to any \"complex\" task. Our definition of a \"complex\" task is that the goal task must consist of multiple sub-tasks that need to be accomplished to reach the final output. The sub-tasks may be sequential or parallel in nature or a combination of both. In our paper, we have demonstrated the applicability of the TeLER taxonomy to two complex tasks, i.e., Meta-Review Generation (Section 4) and Narrative Braiding (Appendix). TeLER taxonomy can also be applied to other complex tasks like Comparative Analysis of Privacy Policy, Compiler Fuzzing, Social Media Bias Analysis, etc. Finally, the TeLER taxonomy can be easily extended by adding more dimensions for categorization as deemed necessary by the target application, and, therefore, the recall of covered tasks can improve as well, allowing maximal generality.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea itself is interesting and I believe it would be useful when we need to compare LLM performances. It can help the published LLM performance numbers are using the level of prompts, so that it's easier to compare\n2. The Taxonomy seems to reasonable to me with enough elements, so it could be useful.", "reasons_to_reject": "1. It only showed the proposed Taxonomy. I wanted to see some sort of examples showing how the TELeR really helped evaluate performance across the LLMs with a complex question. The given example is about generating prompts. It would be great to see it is really helped by feeding these to LLMs with results reported.\n2. The coverage of proposed frameworks. It would be great to see some types of recall numbers of the proposed framework. Even though this paper admitted it only intended to cover complex problems, I wanted to see what types of complex problems the proposed framework can cover.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposed a Taxonomy for LLM prompt for complex tasks with the intent to provide a standard to evaluate the performance of LLMs, with the hope to remove the difference caused by prompt designs.\n\nThe proposed Taxonomy defines Prompt as Directive + Data, with 6 levels of details, and turn type, instruction vs question, and defined vs undefined roles.\n\nThe paper gave an example of how to design prompts to write Meta-reviews based on the proposed Taxonomy to show the usefulness of the proposal.", "excitement": 2, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UNJ7GxkfWH", "reviewer": "Reviewer_YfVr", "summary": "", "questions": "1. Do all the tasks (or prompts) fall within the scope of this taxonomy? I believe that certain tasks which have specific requirements and evaluation procedures may not be practical to use, such as machine translations or reading comprehension. I am curious about the extent to which this taxonomy can be implemented.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- \"There are some questions related to the application of this taxonomy that have not been addressed. When designing a taxonomy, it's essential to consider how prompts will be categorized and how they can be transformed to different levels. These questions should be addressed systematically, besides case studies for a specific task.\"  \"The performance gaps of models when handling queries of different prompt levels are also worth investigating. One may naturally consider the more detailed a prompt, the better the performance model achieves. However, does this hold for all tasks and different models (closed and open-source) is unclear.\"\n\n  - Thank you for your comment. We want to highlight that this is a short theme track \"position\" paper. Our \"position\" here is that the current practice of deploying prompt engineering techniques in LLMs without a structured taxonomy is problematic, and we need a standard prompting taxonomy to enable a fair comparison of LLMs and, thus, derive more accurate and meaningful conclusions from different independent studies. We agree that it would be great to see if the taxonomy is really helpful by creating prompts and feeding them to LLMs with results reported. However, a 4-page limit is impractical to report such studies with details of different instances of prompts created. In fact, we have already used this taxonomy in multiple use cases and found it to be applicable to a wide range of complex tasks. We also found that Levels 3 and 4 are often performing better than other levels. However, we cannot point out/cite those studies as it will break the anonymity rules.\n\n\n- The paper writing needs improving, and sometimes the content is not easy to follow. For example, the term 'complex task' is somehow ambiguous. Formally defining what kind of complex is focused on (e.g., a task requires in-depth knowledge, or includes a long horizon of planning) is highly preferred.\n\n  - Thank you for your comment. The proposed TeLER taxonomy is actually very general and can be applied to any \"complex\" task. Our definition of a \"complex\" task is that the goal task must consist of multiple sub-tasks that need to be accomplished to reach the final output. The sub-tasks may be sequential or parallel in nature or a combination of both. In our paper, we have demonstrated the applicability of the TeLER taxonomy to two complex tasks, i.e., Meta-Review Generation (Section 4) and Narrative Braiding (Appendix). TeLER taxonomy can also be applied to other complex tasks like Comparative Analysis of Privacy Policy, Compiler Fuzzing, Social Media Bias Analysis, etc. Finally, the TeLER taxonomy can be easily extended by adding more dimensions for categorization as deemed necessary by the target application, and, therefore, the recall of covered tasks can improve as well allowing maximal generality.\n\n\n\n- \"Do all the tasks (or prompts) fall within the scope of this taxonomy? I believe that certain tasks which have specific requirements and evaluation procedures may not be practical to use, such as machine translations or reading comprehension. I am curious about the extent to which this taxonomy can be implemented.\"\n\n  - The proposed TeLER taxonomy is exclusively applicable to LLM prompts targeted toward solving a complex task. This taxonomy, especially the six levels, does not apply to simple tasks like sentiment classification or sentence-level inference tasks. As such, TeLER taxonomy will be most useful to researchers and developers who are conducting applied LLM research and development that is focused on performing complex tasks that are hard to evaluate. Also, the TeLER taxonomy should not be considered as an ultimate taxonomy for LLM prompts, and further extensions of this taxonomy are certainly possible and actually desired. Having said that, the TeLER taxonomy is actually very general and can be easily extended by adding more dimensions for categorization as deemed necessary by the target application.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Clarifying instructions is a crucial task, yet there is currently no standardized method for assessing their clarity. This paper touched on this important topic and proposed a taxonomy.\n\n2. The facets under consideration are thorough, exhibiting a strong resonance with the intuitions and experiences of querying LLM.\n", "reasons_to_reject": "1. There are some questions related to the application of this taxonomy that have not been addressed. When designing a taxonomy, it's essential to consider how prompts will be categorized and how they can be transformed to different levels. These questions should be addressed systematically, besides case studies for a specific task.\n\n2. The performance gaps of models when handling queries of different prompt levels are also worth investigating. One may naturally consider the more detailed a prompt, the better the performance model achieves. However, does this hold for all tasks and different models (closed and open-source) is unclear.\n\n3. The paper writing needs improving, and sometimes the content is not easy to follow. For example, the term 'complex task' is somehow ambiguous. Formally defining what kind of complex is focused on (e.g., a task requires in-depth knowledge, or includes a long horizon of planning) is highly preferred. ", "questions_to_authors": "1. Do all the tasks (or prompts) fall within the scope of this taxonomy? I believe that certain tasks which have specific requirements and evaluation procedures may not be practical to use, such as machine translations or reading comprehension. I am curious about the extent to which this taxonomy can be implemented.", "paper_topic_and_main_contributions": "The paper presents a general taxonomy for designing prompts, called TELeR, for querying large language models (LLMs) on complex tasks.\nThe main contributions include enabling meaningful comparisons across different benchmarking studies, establishing a common standard, and facilitating more accurate conclusions about LLMs' performance on specific complex tasks.\n\n", "excitement": 2, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mSRt8uh8PF", "reviewer": "Reviewer_SYHs", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comment. We want to highlight that this is a short theme track \"position\" paper. Our \"position\" here is that the current practice of deploying prompt engineering techniques in LLMs without a structured taxonomy is problematic, and we need a standard prompting taxonomy to enable a fair comparison of LLMs and, thus, derive more accurate and meaningful conclusions from different independent studies. We agree that it would be great to see if the taxonomy is really helpful by creating prompts and feeding them to LLMs with results reported. However, a 4-page limit is impractical to report such studies with details of different instances of prompts created. In fact, we have already used this taxonomy in multiple use cases and found it to be applicable to a wide range of complex tasks. In our paper, we have demonstrated the applicability of the TeLER taxonomy to two complex tasks, i.e., Meta-Review Generation (Section 4) and Narrative Braiding (Appendix). TeLER taxonomy can also be applied to other complex tasks like Comparative Analysis of Privacy Policy, Compiler Fuzzing, Social Media Bias Analysis, etc. We also found that Levels 3 and 4 are often performing better than other levels. However, we cannot point out/cite those studies as it will break the anonymity rules.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well-organized and easy to follow. \n2. Examples of using this taxonomy for prompt categorizing are provided for better understanding.\n", "reasons_to_reject": "1. The authors concentrate on presenting a taxonomy of prompts used in LLMs, but there is a need for a more comprehensive analysis of each category. For example, providing suggestions on which kinds of prompts are suitable for particular tasks or models will be meaningful and beneficial for future research.\n2. To achieve a comprehensive understanding of the differences of these classes, the authors should include quantitative comparisons using various datasets and baselines. Considering this, this paper is better suited as a long paper. \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "To address the challenge of varying LLM performance with different prompt types and details, the authors provide a general taxonomy to design prompts with specific properties for various tasks. Based on this taxonomy, future benchmarking studies may be able to draw more accurate conclusions about LLMs' performance on specific complex tasks.", "excitement": 2, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a taxonomy that can be used to design prompts for LLMs to perform "complex" tasks. The authors describe the taxonomy and argue for it's usefulness, including a use case of meta-review generation. There are some concerns about the definition of a complex task, which is somewhat ambiguous in the paper, which have been answered in the rebuttal. Furthermore, there are some concerns about the practical usefulness of the taxonomy, which the authors assert is useful but it cannot be proven due to anonymity, meaning that this cannot be considered for the purposes of the review.
Decision: Accept-Findings