Frugal Prompting for Dialog Models
Authors: Bishal Santra
Sakya Basak
Abhinandan De
Manish Gupta
Pawan Goyal
Venue: EMNLP 2023 Findings (2023)
Keywords: dialog
dialog generation
natural language generation
dialogue generation
dialogue
NLP
TL;DR: Understanding the behavior of large language models in different input scenarios is crucial for building efficient and cost-effective dialog systems.

Abstract: The use of large language models (LLMs) in natural language processing (NLP) tasks is rapidly increasing, leading to changes in how researchers approach problems in the field. To fully utilize these models' abilities, a better understanding of their behavior for different input protocols is required. With LLMs, users can directly interact with the models through a text-based interface to define and solve various tasks. Hence, understanding the conversational abilities of these LLMs, which may not have been specifically trained for dialog modeling, is also important. This study examines different approaches for building dialog systems using LLMs by considering various aspects of the prompt. As part of prompt tuning, we experiment with various ways of providing instructions, exemplars, current query and additional context.
The research also analyzes the representations of dialog history that have the optimal usable-information density. Based on the findings, the paper suggests more compact ways of providing dialog history information while ensuring good performance and reducing model's inference-API costs. The research contributes to a better understanding of how LLMs can be effectively used for building interactive systems.

Frugal Prompting for Dialog Models
Bishal Santra', 1x Sakya B Abhinandan De', Manish Gupta', Pawan Goyal
'ITT Kharagpur, Miccrsooft
(bishal. santral, abhinandan 31 60, p iitkkp...... .ac.gp.cc.... ac in Isakya.basale gmani h @microsoft.com
of LLMs remain unexplored. For instance, since The use large language models (LLMs) in
Abstract
we do not need to follow a fixed schema for the natural language processing (NLP) tasks is rapidly increasing, leading to changes in how researchers approach problems the field. To fully utilize these models" abilities, better understanding of their behavior for different input protocols required. With LLMs, users can directly interact with the models through a text-based interface to define and solve var- ious tasks. Hence, understanding the conver- sational abilities of these LLMs, which may not have been specifically trained for dialog modeling, is also important. This study exam- ines different approaches for building dialog systems using LLMs by considering various as- pects of the prompt. As part of prompt tuning, we experiment with various ways providing instructions, exemplars, current query and addi- tional context. The research also analyzes the representations of dialog history that have optimal usable-information density. Based on the findings. the paper suggests more compact ways of providing dialog history information while ensuring good performance and reducing model's inference-APe costs. The research con- tributes to better understanding of how L.LMs can be effectively used for building interactive systems.'
textual inputs anymore (like standard supervised learning for text), the ways in which input-text can be presented, and its impact on task performance is an essential aspect that needs to be investigated. Additionally, various LLM-inference APIs are becoming available for price, trade-off between performance gain and prompting (inference) cost another dimension that requires attention.
While efforts have been made reduce infer- encing costs of Transformer (Vaswani al., 2017) models, these contributions have mostly been the architecture level and require access to the model weights and source codes (Tay et al., 2022). As many models like GPT-3 (Brown et al., 2020), CODEX (Chen et al.. 2021a), LaMDA (Thoppilan et al., 2022), and PaLM (Chowdhery et al., 2022) are now closed source, it is not possible for the end-user to optimize the model's costs using these approaches.
In recent prompt engineering literature, the fo- cus has been optimizing the prompt to improve downstream task accuracy (Chung et al., 2022; Wei al., 2021), with the majority of past efforts target- ing single-turn tasks (e.g., classification, reading comprehension, question answering, etc.). How- ever, for longer inputs, another critical factor is the inferencing API cost, which has largely been ignored in prior works. This is especially true for interactive or dialog tasks.
1 Introduction
Large Language Models (LLMs) have rapidly trans- formed the landscape of natural language process- ing (NLP) research through emergent capabilities like prompt-based learning, in-context learning (ICL), and conversational capabilities (Wei al., 2022). While these novel approaches are being applied to various domains and tasks with an unre- alistic speed and effectiveness, many dimensions
This paper explores the trade-off between cost and performance for LLMs in prompt-based/in- context learning (ICL) setup. We propose the idea of frugal prompting in the context of dialog mod- cls, which involves input optimization methods maintain performance gains while minimizing costs. To compare effectiveness of input represen- tations for in-context learning based methods while considering both cost and task performance, we introduce a new metric called Usable Information
Equal contribution
We make the code publicly available on following two repos: 1) https::/// /githbb. com/bsanttaagil Prugal-Promer i igg 2) https:/ //pp:s///// hhub. con/ trantraigii/Frugal Proalll-riopiigggee Ana lysis.

---

Density (UID). Using this metric, we gain insights into the capabiliitiss of various ICL model families for understanding and accessing information from different input representations.
more complex model. (2) Efficient transformer ar- chitectures (Kitaev et al., 2020; Wang et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020) aim to reduce the quadratic complexity of the standard transformer architecture by using more efficient self-attention mechanisms. In this paper, we ex- amine the costs associated with the use of prompts in LLMs and suggest new method for assessing the cost-performance trade-offs involved, as well as strategies for optimizing the inference cost with regard to the inputs. Please refer to Appendix for more detailed literature review.
Overall, we make the following contributions in this paper. (1) We explore the effectiveness of various ICL models and input formats for dialog modeling. (2) We propose a new metric, UID, that captures the tradeoff between accuracy and length for various (input format, ICL model) combina- tions. (3) Extensive experiments on two benchmark dialog datasets (MSC and TC) and four ICL mod- els show that (a) Adding more context as part of the input does not necessarily improve UID by sim- ilar amounts across all ICL models. (b) For most ICL models, using the most semantically related utterance from dialog history is more cost effective compared to using full history, summarized dialog history or most recent utterance.
3 Prompting Methods for Dialog Systems
We first present the necessary ingredients of a prompt for dialog systems. Next, discuss recipes for manual and algorithmically optimized prompts. Lastly, we present ways of effectively including context information as part of prompts.
2 Literature Review
3.1 Prompt Ingredients for Dialog Systems
Large language models (LLMs) for Dialog mod- eling: A large number of recent dialog genera- tion models have been based on pretrained LLMs like DialoGPT (Zhang et al., 2019), Plato (Bao et al., 2019), Blenderbot (Roller et al., 2021: Shus- ter et al., 2022), Meena (Adiwardana et al., 2020), and LaMDA (Thoppilan et al., 2022) which use the transformer architecture. Although large scale pretrained models like 175B Blenderbot-3 (Shuster et al., 2022) or the 137B LaMDA (Thoppilan et al., 2022) lead to high accuracies across multiple dia- log datasets, these approaches can be prohibitively expensive due to the ever-increasing size of mod- els. In-context learning (ICL) (Brown et al., 2020) with prompt-based based models helps avoid expensive finetuning. Further, better accuracies have been ob- tained using instruction finetuning in models like TO (Sanh et al., 2022), FLAN (Chung et al., 2022), Ins (Wang et al., 2022), etc. But the in- creased inference costs due large prompts sizes remains an open challenge.
To build prompt-based dialog system using LLMs, the following components or information sources are an important part of the prompt template. (1) Task Instruction: The instruction is used to explain the task of dialog response generation model. We also assign a system-role for the LLM (also called Person2) to play through the instruction for example the role of "an automated chat system". (2) Dialog Context: As part of the dialog context, several components can be included like dialog history, persona information and Personl's 's latest utterance.
This refers to the past conver- sation between Personl and Person2 that provides the context for the current conversation.
(a) Dialog history:
(b) Background Information (BI): We also make use of some additional information like persona or knowledge sections when available. Persona is fictional representation of a user consisting of series of sentences describing their characteristics, events and opinions. This used to create a personal- ized experience during a conversation. Knowledge sections are short paragraphs from different data sources (Wikipedia, Reddit, and Washington Post) that are related to the topic of the conversation. We experiment with various combinations of different pieces of information understand their impact on the accuracy versus inference cost.
Ways to optimize computation for LLMs: Fol- lowing environmental impact discussion of the training process of these LLMS (Strubell et al., 2019), multiple studies have proposed these two main lines of work on optimizing costs of LLMs: (1) Model distillation-based (Hinton et al., 2015; Sanh et al., 2019; Gou et al., 2021; Gupta and Agrawal, 2022) methods train a smaller, simplified model to approximate the predictions of a larger,
(3) Personl's latest utterance: This is the most recent statement or question uttered by Person in

---

Template
Prompt
summary the conversation befween Personl and Person2: [5]
conversation between Personl Person2: Personl wants to back callege to learn more about accounting. Person2 wants to stady education so Person2 could teach art. Personl thinks it's never late for
Here summaay
change. Based on the dialog between the Personl and the Person2 so far, try to anticipate what the Person2's response might be the Personl's I'ss next statement.
Based on the diulog between the Personl and the Person2 far, try unticipate what the Person2's y respouse might be to the Personl's next statement.
Personl: [V]
years. FIVE long years, It's the most rewarding job but it's steady and reliable so never really looked for anything olse, but I'm starting to want change."
Personl:
Person2:
Person2:
Generation
[
think you should look into earcer change. It's never too lafo learn something new
Table I: Prompt template and instantiation with summary of dialog history dialog context and without exemplars or background information. This is perplexity optimized using FLAN-T5-XX X-L model. More examples are provided in Appendix G.
a dialog, that prompts the Person2's s response. (4) Exemplars: Although most recent LLMs are capable of solving tasks just using instructions (due to RLHF and instruction-finelee finetuning), providing ex- amples along with task description may help im- prove performance. We test our prompt-based mod- els in two configurations with respect to number of examples: zero-shot and few-shot.
were designed keeping in mind general principles of prompt design (Liu et al., 2023) like role based prompting (Schulhoff and Contributors, 2022), specifically adding requirements like "generate a consistent, diverse response" so as not to get repet- itive, dull responses and maintain consistency with respect to the current utterance and context. Table 2 illustrates one of our manually designed prompt template, with summary of dialog history, persona and current user utterance as dialog context and with one exemplar.
An example prompt is shown in Table 1. A full list of all the prompts used in our experiments can be found in the Appendix G.
Perplexity Optimized Prompts: We followed the strategy highlighted in Gonen et al. (2022) which claims that the performance of a prompt is cou- pled with the extent to which the model is familiar with its language, and this can be measured by the perplexity of the prompt. Given an LLM, we took the manually engineered prompt template, and created candidate prompt variants by using GPT3 and back translation. Further, we instantiated all such prompt templates using 100 instances (with full prompt sequence, including the input itself, and without the label), and computed average per- plexity per template using the L.M.. The lowest perplexity template was chosen.
Antomated Chat System:
generate consistent and di- verse responses between Personl and Person2 given buckground details along with summary. Example:
Learn from the below example
Here are some background details about Personl: [8T(P))
Here are some background details about Person2: [P)]
This is a summary of a dialog exchunge between Personl and Person2. [8x]
Given the background details and the summary of dialog exchange
between Personl and Person2, give consistent and diverse response to the following dialog by Personl.
Persont: [Ug]
Person::
Now try yourself:
Here
background details about Fersonl: [T(P)))
Here are some background details about Person2: [P)]
This is summary of dialig exchange between Person and Person
[S]
Given the summary the dialog evchange between Personi and Per- son2 and their background details, give a consistent and diverse re- sponse to the following dialog spoken by PersonI.
Personf: [U]
Person2:
Table 2: Manually engineered prompt template with summary of dialog history. persona and latest person utterance as dialog context and with one exemplar. [B))) is personl's 's persona, [P2) is person2's s persona, [S] is summary of the dialog history. [U] is the latest person I utterance, [/t] is the person2 response. "Ø implies corresponding elements are for the exemplar.
3.3 Optimizing the Dialog History Input
Redundancies in conversations: In conversa- tional agents, dialog history plays a crucial role in generating meaningful responses. It provides context and continuity, and enables the agent to re- member previous interactions with the user. How- ever, the dialog history can also be redundant, espe- clarifica-
cially when it contains tion, and mistake correction. While these elements are necessary for natural and useful conversa- tion, they increase the length of the dialog history without adding any new information. In addition, responses from some dialog models (like Instruct-
3.2 Manual versus Perplexity Prompts
We experimented with two ways to design prompt templates: manual and automatically optimized prompts using perplexit--based search method. Manually Designed Prompts: Manual prompts

---

GPT (Ouyang et al., 2022)-based models text- davinci-003) could be elaborate and long.
4 Experimental Setup
4.1 Datasets
Shortening Dialog Histories: To reduce the prompt length, we can compress the dialog his- tory by removing redundancies. The goal is to give the agent only the parts that are relevant and in- formative for generating the next response. Two possible approaches compress the dialog history into a shorter and more informative representation are selection and summarization.
We experiment with two dialog datasets for com- paring various methods on accuracy versus infer- ence cost for prompt-based dialog systems: Multi- session Chat (MSC) (Xu et al., 2022) and Topical Chat (TC) (Gopalakrishnan et al., 2019). We chose these datasets because of their varying characteris- tics and the length of the dialog history.
The MSC dataset consists of multiple chat ses- sions whereby the speaking partners learn about each other's interests and discuss the things they have learnt from past sessions. Each user partici- pating in these sessions (or conversations) asked to play a role (persona) while having the conver- sation. On the other hand, the TC dataset, each pair of users is assigned one or more topics along with some facts or knowledge about the topic, and the users are asked to have conversation about the topic. Users have no persona in the TC dataset but there knowledge sections associated with the conversations. The test set contains 16,299 and 7,512 context response pairs in the MSC and TC datasets, respectively. Also, there are 11.9 and 20.0 average number of utterances full conversations in the MSC and TC datasets, respectively.
Selection: Two possible ways to select parts of dialog history are as follows. (1) Recent-k: The simplest approach is to use fixed-length dialog history from the most recent utterances. However, this upproach may not be optimal, as users may refer back to context beyond the fixed length window and expect the system to under- stand. (2) Semantic-k: In this approach, the most relevant & utterances from the dialog his- tory are selected with respect to the current ut- terance. This method is simple, but its perfor- mance depends on the quality of the similarity measure used. We used the average of the simi- larity obtained using SimCSE model (Gao et al., 2021) and Sentence Transformers (Reimers and Gurevych, 2019) measure the overall similarity between utterances.
Since do not train finetune any specific models, we do use train splits of these datasets. For perplexity- based prompt optimization, we use validation splits of these datasets. We discuss de- tailed preprocessing steps in the Appendix A.
Summarization: An alternative approach to use a summary of the full dialog history. We con- encoder-decoder
sidered two Transformer-ban abstractive summarization models (BART (Lewis et al., 2019) and Pegasus (Zhang et al., 2020)) finetuned on generic as well as dialog datasets like CNNi (Hermann al., 2015), SAMSum (Gliwa et al., 2019) and Dialog- Sum (Chen et al., 2021b). These methods are more complex, but they can generate a summary that is more informative and short.
4.2 Summarization of Dialog and Background Information
We used BART and Pegasus models for summa- rization. However, in dialog summarization, the objective is to distill the most important informa- tion or key points from a conversation, which can be quite challenging because conversations tend to be more dynamic and context-dependent than normal documents. Unlike traditional summariza- tion, in dialog summarization, there a greater emphasis on preserving the coherence and con- text of the conversation. Hence, we used dialog summary datasets like DialogSum, SAMSum and CNNi to finetune abstractive summary models like Pegasus and BART and picked up the best model for use in terms of summarization per- formance by calculating ROUGE metric on dialog summarization data.
Shortening Background Information: Often dia- log datasets also include other background informa- tion like persona information (Xu et al., 2022), read- ing sets (Gopalakrishnan et al., 2019) and knowl- edge facts (Dinan al., 2018). Transformer-basee encoder-decoder abstractive summarization models (BART (Lewis al., 2019) and Pegasus (Zhang et al., 2020)) can be used to shorten such back- ground information as well.

---

We process the Dia and SAMSum dat- sets remove all conversation instances having more than two speakers and normalized the speaker names to P and Person2 so that the model does not hallucinate random names during sum- mary generation.
stance, the exemplar is different. For example, con- sider the Recent-4 few shot setting. Let ABCDEFG be the utterances in the conversation. Thus, the in- stance will have G as the target response, and input contains F Fas the current utterance and BCDE as the recent-4 dialog history. The input for this in- stance will also consist of an exemplar where the target response will be F and input for exemplar will contain E as current utterance and ABCD the recent-4 dialog history.
Overall, we train three models: (1) BART-D: facebook/bart-large model finetuned on Dialog- Sum, with 12 encoder and 12 decoder layers. (2) Pegasus-CD: google/pegasus-cmm_cooo a model (which has been finetumed on CNN-DailyMail cor- pus, with 16 encoder and 16 decoder layers. (3) Pegasus-DS: google/pegasus-cmm dailymail model further finetuned on both DialogSum and SAM- Sum data, with 16 encoder and 16 decoder layers. Training haper-parameters. are in Appendix B.
4.4 Metrics
Performance We evaluate the performance of the models using several popular metrics: ME- TEOR, BLEURT and DEB, METEOR (Baner- jee and Lavie, 2005) is widely used for various text-generation tasks (machine translation, dialog generation, etc.). It measures lexical be- tween n-grams of the predicted and ground truth response. BLEURT (Sellam et al., 2020) uses a pre-trained BERT model for evaluating text gener- ation models. DEB (Sai et al., 2020) is a BERT- based dialog metric further pre-trained on dialog data for next response classification using the next- sentence-predien prediction (NSP) loss.
4.3 Models and Prompt Design
For this study, we used GPT-3 (text-davinci-003), one of the most prominent models for prompt- based or ICL. Along with GPT-3, we also in- cluded other open-source models that are ca- pable of ICL: FLAN-T5 (google/fan-t5xx)), TO (bigscience/TO_ _B),, and Tk-Instruct (allenai/tk- instruct-3b-def for zero shot and allenaittk-instruct 3b-def-pos for few shot). These open-source mod- els are generally smaller in size compared to GPT- 3 (175B) and have the capability of ICL through based training.
Inference Cost To evaluate the effectiveness of different prompting methods for dialog systems, we need a metric that takes into account both the performance gain and the inference cost reduction. The cost is measured in terms of the length of the overall input, as longer inputs incur more inference- API costs and also slow down the inference.
We experiment with several input prompt set- tings: (1) Zero shot versus few shot. (2) Manually designed versus perplexity optimized prompts. (3) Settings based on usage of dialog history: (a) full history, (b) summarized dialog history (using any of the three summarization models), or (c) Recent- k semantic-k selection from history. (4) With and without summarized background-informaat
We propose a new benefit-cost based metric to simultaneously consider both model perfor- mance and the inference cost incurred: the usable- information-density (UID). UID with respect to metric M is defined as UID (a) (Mn)" / /Lu where Mu the average performance of the model as per metric M, LH the overall combined size of input and output averaged across all the test ex- amples, a is a metric-importance parameter. In the main paper, we present results using a=1, but show impact of varying a in the Appendix. With a=1, UID is defined as the ratio of performance to cost measured in terms of size of the input and output. The UID captures the amount of information, per token, usable by a model (Ethayarajh al., 2022) for a given input/prompt configuration. The UID metric can be used to evaluate the effectiveness of different prompting methods.
In case of few shot, we use only one exemplar since (I) previous work (Madotto et al., 2021) has shown that one exemplar is enough, and (2) we wish to find methods which retain good accuracy with short input lengths. The exemplar is also for- matted in the same way as the actual input. For example, if the actual input setting is to use persona with few shot, the exemplar also includes persona information. Similarly, the actual input setting is to use summarized dialog history with input, the exemplar also includes summarized dialog history.
The exemplar chosen based on the immedi- ately previous utterances if available, else it is ran- domly chosen from the dataset. Thus, for each in-

---

1100
shot-Manual
BLBURT
DEB
METBOR
FLAN-TS
0.357
0.765
0.139
1000
MSC
T0
0.347
0.912
0.153
900
Fe-Few shot-Manual
TK Instruet
0.355
0.832
0.133
800
Few shot Perpeeelttty--opiiiieedee
FLAN-TS
0.124
700
TV
0.321
0.868
0.133
TC
GPT3
0.342
[0,885
0.147
600
Ix- Ia-tuutruct
0.338
0.852
0.119
500
400
Table 3: Model comparison based on average perfor- mance over history, prompt-type and exemplar settings.
300
200
that the summary of background information in TC is much larger compared that in MSC. For example, Pegasus-DS BI for TC as large as full dialog history.
100
Semantics
Recent-1
BART-D
Pegarus-
Semantic-2
Recent-2
Pegasus-Co
Recent-4
Semantic-4
Semantice
Pagasux DS Bi
Recent-8
Semantlc 10
Recent-30 10 Full
5.2 Performance Results and Analysis
Figure 1: Comparison of average input length for var- ious representations of dialog prompts across the two datasets for the few shot setting. DH Dialog History, Bl = Background Information.
In Figs. 2 and 3, we analyze the absolute per- formances of various LLM model-families using prompts based on various input representations for TC and MSC, respectively. We show results for few shot (FFS as well as zero shot (ZS) cases across three popular metries BLEURT, DEB and ME- TEOR. We also show results for manually engi- neered as well as perplexity optimized prompts averaged across various models (FLAN-T5, TO, Ik-Instrct and GPT-3). Since we do not have access to logits from GPT-3 model, we cannot opti- mize prompts for GPT-3 using perplexity. Detailed model-win results are Appendix Figs. 8. For each of these combinations, we show results for different input prompt combinations: (1) Full dialog history, (2) Summary of dialog history us- ing BART-D Pegasus-DS or Pegasus-CD, (3) Pegasus-DS summary of dialog history as well Pegasus- CD summary background information (BI), (4) Recent-k selected dialog utterances, and (5) Semantic-k selected dialog utterances, where is varied as 1, 2, 4, 8, and 10. Note that we did not experiment with full background information since background information is very large in size, especially for the TC dataset.
5 Results and Analysis
5.1 Size Comparison across different Input Formats
Fig. shows the variation in the average input prompt size as we vary the prompt constituents, dia- log history (DH) and background information (BI), for the few shot. We show a similar figure (Fig. 4) for zero shot cases in the Appendix. We plot the variation for manually engineered as well as per- plexity optimized prompts for both the datasets (MSC and TC). -axis indicates the overall length of the input prompt, which is fed to the large lan- guage models (LLM) without further processing. We observe that the complete dialog history is sig- nificantly longer compared to summarized or se- lection forms. Since we use one demonstration ex- emplar in few shot cases, the few shot prompts are typically twice as long as their corresponding zero shot prompts. Perplexity optimized prompts are slightly shorter than manually engineered prompts on average. Pegasus-DS summarized dialog his- tory is almost 3 times shorter; Pegasus-DS DD sum- maries are shorter than Pegasus-CD summaries while BART-D summaries are shorter than Pegasus- DS summaries. Sizes of recent-2 (or semantic-2) are similar to the summarized dialog histories in terms of the final length of the input context to expect that the summa-
As shown in Table 3, GPT-3 generally outper- forms the other families of LLMS terms of abso- lute performance (DEB, BLEURT, and METEOR) and Tk-Instruct performs the worst. Also, gener- ally we observe the best results with full dialog history for TC in most cases for DEB and ME- TEOR. For MSC, even prompts with summarized history seem to do very well although they are much shorter. Averaged across metries, we observe that Semantic-k performs better than Recent-k for all values of &o (1, 2, 4, 8, 10) for both datasets. Further, while Semantic-k reaches peak perfor-
the model. However, rized dialog history will have more useful informa- tion stored in compressed form compared to the greedy choice of only the recent-2 or semantic-2 utterances. In case of Pegasus-De DS BI, we use the BI summarized using Pegasus-CI CD model. Note

---

Model beg. Mot-all
Model sog 10W1 Midel bng 179-PP1)
Miatel Any (25 Manrall (29 Aaarsal
Mocal Aug (25+113 (29-193)
Modi brg. (FS+Stznusll
bdodd log, Mounall
Nodi aug (39221)
9.466
0.5
0.000
03k
0.8
0.00
07-
0120
011
1387 3-
Pajaus 18 K.
Secantie Pill-
Pegune co
z-gon 26
Folaan ill
Vegese co
S
Comoon Sent
Ealur Sigal
Hige
Figure 2: Model averaged performance results for TC Dataset. DH = Dialog History, BI Background Information.
Modelloy, 23--Mtzea)
And Ave (s- Madel tog (X-FPE)
Monit Lng 125-itam
Mode Ang 125-FFU Mtddi Ang 135
Madd fog 12S MaxnaD
Modci PP5)
Model (3-terrea
30o0ht bng (15) Starast
Modd log ItS Mascal
Kode lvg, Ih- 2P1)
0.55
0.26
0.90
0.36
0.35
RUBURT
METIIR
0.80
0.91
0.34
0.75
0.32
0,3
BART Di
Pegress (D)
Pegonte D8 ni
Beceat
Recver 4
Becemts
Decemtt 10
gen cont
Scmastic
Semantic 10
Faill
BARTT-D
Pegprur
Vegunis DS
Pegens DS ai
Riceent1
Recent-22
Becent-4
Recent 6
Recent 10
Semaztic
Somantic 10 Fuil
BART-D -D)
Pegaru eD
Peptaro DS
Pegunns DS anl
Recenck
Rocest 2
Kecont
Rosen a
Bhe 30
Semantic 4
comman a
Semantic
Full
History Signal
Hlistory Signal
History Signal
Figure 3: Model averaged performance results for MSC Dataset.
DH =Dialog History, BI = Background
Information.
mance at k=4, Recent-k attains the best results at higher values of k (8 or 10), Adding background information (knowledge facts) to Pegasus-DS helps boost DEB and METEOR significantly but hurts BLEURT on average for both the datasets. In MSC, amongst different configurations for history signal, Recent-1 performs the worst on average. In TC, BART-D performs the worst on average. Surpris- ingly, even though zero shot prompts are almost half the size compared to few shot prompts, zero shot results are better than few shot results, except for perplexity prompts in MSC. Although recent prompt engineering based studies motivate using demonstration examples, it turns out that examples are not very useful for dialog modeling.
5.3 UID Results and Analysis
What is of our main interest is the fact that, in many cases, the summarized dialog-history input able to attain much of the performance, sometimes even better than the full dialog-history setting, which has a much longer input length. Thus, we are interested in comparing various input representation methods in terms of how much information, per token, particular LLM model can access, that is converted into better performance the response generation task. Hence, in this section, we discuss relative importance of different components of the input prompt using the UID as a metric that explains the input prompt size versus the performance tradeoff. We show the UID results averaged across models in Table 4. We also show model-wirg wise results in Tables 5, 6, 7, and 8 for FLAN-TS, TO, In- and GPT-3, respectively. We show results for few shot well zero shot cases, and for both the datasets (MSC and TC). We also show UID results across all dialog history, prompt-type and exemplar settings on the three different metrics BLEURT, DEB and METEOR.
Perplexity optimized prompts lead to shorter prompt sizes but not better accuracy values, ex- cept for DEB in TC. Since cannot compute perplexity optimized results for GPT-3, we show results for the remaining three models. We observe that TO is the best for DEB and METEOR while FLAN-T5 is the best for BLEURT. In both cases, zero shot results are better.

---

Manual Promutt
Perpieait Prompt
History
BLEURT
METEOR
BLFURT
METEOR
Z8
1S
28
E3
ZS
FS
ZS
13
28
F3
Z3
FS
BART-D
19.79
9.03
46.56
22.57
1.75
20:27
8.94
23.81
7:25
Pegasus CD
17.76
8.22
44.58
21.07
7.14
3.60
17.75
8.06
41.14
22.22
6.49
3.68
Pogasus-DS
19.72
46.49
7.78
8.9
45.40
17.69
7:25
Pegnsus DS + Bl
12:23
5.87
29.65
15.21
4.93
2.59
11.97
5.57
27.50
14.27
4.34
2.34
Revent-1 I
22.22
9.82
53.35
23.84
8.96
4.17
22.29
9.59
46.07
21.59
8.50
3.99
Recent-2
19.38
8.72
46.49
20.88
7,52
3.68
19:65
8.52
42.84
19.33
7.49
3.00
Recent 4
15.18
6.96
36.77
16.78
6.34
2.95
15.28
6.68
34.54
15.78
5.92
2.86
Revent-8
27.91
12.94
4.75
11:25
5:04
12.14
4.38
Revent- 10
10.05
4.85
25.08
11:92
4.08
2.08
10.14
4.59
24.40
11.19
3.94
1.99
Semantie-1
29:51
10.18
11.11
22:775
9.005
24.23
21.48
8.85
4.20
Semantic 2
19.86
8.73
44.19
19.85
7.88
3.78
20.42
8.68
43.63
19.34
7.56
3.60
Semanti-4
15.50
6.98
38.57
17.87
6.52
3.08
15.62
6.87
36.93
18.02
6.05
2.99
Semontic-8
11.07
5,00
25.58
12.002
4.56
4.85
25.06
11.91
4.14
2.06
Semantic 10
9.72
4,46
22.62
10:72
4.02
2.00
9.80
4.29
22.12
10.64
3.63
1.84
6.97
3.44
18.03
2.91
3:51
2.65
BART
24.36
11.11
57.75
24,46
8.67
4.02
25.64
11:00
59.05
22.42
8,46
3.71
Pogasus
8.77
20.14
16.31
6.63
3.00
Pegnsus-DS DS
24.37
11.00
57.61
39
8.66
3.99
25.47
10:96
58.82
22.49
18:43
3.70
Pegusus DS
7.67
3.88
20.75
9.09
2.94
1.57
7.59
3.63
20.76
8.27
2.68
1.43
Recent-1
24.17
10.62
60.36
26.08
9.009
4.13
24.98
10.83
58
28:06
8:83
4.05
Recent 2
20.35
9.14
51.54
22.58
7.89
3.59
21.25
9.27
51.71
24.58
7.72
3.67
Recent-4
15,99
7.35
40.57
17.98
6.31
16.66
7.29
41.21
19.82
6.15
TC
Revent &
11.92
5,54
30.56
13:65
4.73
2.19
12.24
5:47
30.85
14.93
4.52
2.23
Recent-10
5.04
27.94
12:53
4.26
10:95
28
13.64
4.08
Semantic
24.02
10.50
60.06
27.05
8.76
4.19
25.48
10:73
62.63
28.87
8.88
4.17
Semantic1
20.15
8,95
51.40
23:47
7.64
3.62
21.43
9.12
53.68
25.16
7.65
3.61
Semontic-4
15.34
7.23
41.23
18.90
6.41
2.93
16.31
7.51
42.70
20.07
6.36
2.99
Semantic &
11.57
5.29
30.20
13.92
4.49
2.15
12.01
5.30
31.14
14.94
4.34
2.14
Semumbe-11
10.38
27.45
12.67
4.03
10.68
13:57
3.86
Full
8.45
4,06
23.13
10.94
3.44
1.72
8.47
4.06
23.50
11.57
3,32
1.73
Table 4:
UID results across four models. Manual prompts are averaged over all 4 models; perplexity optimized prompts are averaged over all models except GPT-3.
Comparing manually engineered prompts ver- sus perplexity optimized prompts, we observe that manually engineered prompts are better on aver- age. We believe this is because perplexity and other metrics (BLEURT, DEB, METEOR) do not show similar correlation with dialog response quality as shown in (Liu et al., 2016).
This implies that few-shot UID is much smaller than zero-shot UID as can be seen in Table 4. Overall, we find that using full dialog history, or Semantic-k/Recent-t with large & are not very use- ful from a UID perspective. For both the datasets, it is clear that Semantic-1 1 and Recent-1 have very good UID values across all models and metries, with zero-shot being better than few-shot. This sug- gests that having smaller but more focused input is recommended for dialog model prompting.
Across the dialog history types, we make the fol- lowing observations which hold for both datasets: (1) For most metrics across datasets, we observe that using one semantically related utterance is the best. The UID decreases as increase k. (2) In terms of absolute metrics (Figs. 2 and 3), we observe that Recent-k typically increases with in- crease in while Semantic-k peaks at 4=4 and then drops. But in terms of UID, for both Recent-k and Semantic-k, UID reduces with increase in k. (3) Adding background information to Pegasus-DS DS does not help. (4) Amongst summarization meth- ods, Pegasus- DS and BART-D perform better than Pegasus- CDD. This is expected since Pegasus-Da and BART-D are both trained on dialog datasets. Using summaries of the dialog history provides better UID results than using the full dialog history. This suggests that models can work more efficiently with summarized input.
Conclusion
In conclusion, this paper has explored the trade- off between model performance and cost in inter- active tasks where dialog history plays a crucial role. Since recent large language models tend to produce longer dialog responses, using this long dialog history as context for next utterance pre- diction becomes more expensive. However, the experiments conducted in this study have demon- strated that compressing dialog history can improve model performance without significantly increas- ing cost. Our findings suggest that the optimal representation of dialog history is one that provides the highest amount of usable information per token. Summaries of dialog history are better than using full history itself. Recent utterance or best seman- tically similar utterance are both better than sum- maries. One best semantically similar utterance
As observed from Figs. and 3, few-shot accu- racy values are worse than zero-shot, although few- shot are almost twice the size of zhro-shot prompts.

---

is the best from both accuracy as well as usable information perspective. Overall, our results high- light the importance of carefully balancing model performance and cost in interactive tasks that rely on dialog history.
should use the outputs carefully. These models generate possible response candidates, and do not filter out any "problematic" candidates. Thus, for applications, where candidate responses could be problematic, (e.g., offensive, hateful, abusive, etc.), users should carefully filter them out before using the output from such models.
7 Acknowledgments
All the datasets used in this work are publicly available. We did not collect any new dataset part of this work.
This work was partially supported by Microsoft Academic Partnership Grant (MAPG) 2022. The first author was also supported by Prime Minister's Research Fellowship (PMRF), India.
MSC dataset: The dataset was downloaded from https: /tt///aal.... a ject.s /msc/. Xu et al. (Xu et al., 2022) describes details about cre- ation of the dataset. Parl.ai makes models and datasets available under MIT License.
8 Limitations
We experimented with datasets and models trained languages with limited morphology like English. While we hope that these results will generalize to models trained on multi-lingual datasets; empirical validation needs to be done.
TC dataset: The dataset was downloaded from
https: / / //g//gtuuutt com/ aalexa
Topi Topical-Chat The dataset is available under Community Data License Agreement.
While the study examines TC and MSC, these conclusions may only apply to these datasets and to general open-domain chit-chat dialogue. How- ever, there many more dialogue settings than just these two. For example. it needs to be val- idated if the conclusions would apply more information-erittcal dialogues (e.g. task-oriented dialogue datasets like MultiWOZ).
We used 4 models in this work: TO, Tk-Instruct, FLAN T5 and GPT-3 API. TO, Tk-Instruct and FLAN T5 are all provided under Apache 2.0 Li- cense Huggingface. We used the publicly avail- able GPT-3 API by signing up at OpenAI.
References
For task-oriented dialog systems with well- defined ontologies and belief states, the experimen- tal design would need to be reconsidered, includ- ing aspects like prompts, summarization methods, and evaluation metrics. Standard summarization techniques may need to be adapted to better re- tain key belief state information in the summary. Although we believe that the well-defined ontol- ogy could potentially allow further optimization of prompt lengths compared to open-domain dia- log. While the lower-level details would differ in applying frugal prompting notions to task-oriented dialogs, we are optimistic that similar beneficial findings around balancing model performance and computational costs could emerge.
Daniel Adiwardana, Minh-Thang Luong, David R So. Jamie Hall, Noah Fiedel, Romal Thoppilan. Zi Yang. Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards human- like open-domain chat- bot, arXiv preprint arXiv:2011009977.
Satanjeev Banerjee and Alon Lavie 2005. Meteor: An automatic metric for mt evaluation with improved cor- relation with human judgments. In Proceedings of the ael workshop on intrinsic and extrinsic evaluation measures for machine translation and/or sunmariza- tion, pages 65-72.
Siqi Bao, Huang He, Fan Wang, Hua Wu, and Haifeng Wang. 2019. Plato: Pre-trained dialogue generation model with discrete latent variable. arXiv preprint arXiv:1910.07931.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The dong-document transformer. ArXiv, abs/2004.05150.
9 Ethics Statement
In this paper, we studied how to efficiently use dialog generation models. Although we did not explicitly train our own dialog models, we would like to make the readers aware about potential risks in usage of such models. Many pretrained lan- guage representation models have learned patterns associated with exposure bias. Interpretability asso- ciated with the output is rather limited, hence users
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Hengyi Cai, Hongshen Chen, Yonghao Song. Xiao- fang Zhao and Dawei Yin. 2021. Exemplar guided

---

neural dialogue generation. In Proceedings of the Twenty-Ninte International Conference on Interna- rional Joint Conferences on Artificial Intelligence, pages 3601-3607.
Manish Gupta and Puneet Agrawal. 2022. Compression of deep learning models for text. A survey. ACM Transactions Knowledge Discovery from Data (TKDD), 16(4):1-55.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021a. Evaluating large lan- guage models trained on code. arXiv preprint urXiv:2107.03374
Prakhar Gupta, Jeffrey P Bigham, Yulia Tsvetkov, and Amy Pavel. 2020. Controlling dialogue gen- eration with semantic exemplars. arXiv preprint arXiv:2008.000 09075.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. Advances in neural information processing systems, 28.
Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. 2021b. Dialogsum: A real-life scenario dialogue summarization dataset. arXiv preprint arXiv:2105.06662.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in neural network. arXiv preprint arXiv:/503.0000 503302531.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. ArXiv, abs/2001. 1.0451..
Hyung Won Chung, Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. language models. arXiv preprint arXiv:2210. 1416.
Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2021. angerne-augmented augmented dialogue generation. ArXiv preprint, abs/2107.07566.
Internet-
2022. Scaling
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: De- noising sequence-to-sequeee language generation, translation, and comprehension. arXiv preprint arXiv:1900.1300 1346.1461..
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard conversational
pre-training for natural
of wikipedia: Knowledge-powee agents. arXiv preprint arXiv:181 1.000441.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. 2022. Understamding dataset difficulty with V-usable information. In Proceedings of the 39th International Conference on Machine Learning, volume 162 Proceedings of Machine Learning Research, pages 5988-60 6008. PMLR.
Chia- Wei Liu, Ryan Lowe, Iulian Serban, Mike Nose- worthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metries for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2122-2132. Austin, Texas. Association for Computational Linguistics.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. arXiv preprint arXiv:2104.083
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang. Hiroaki Hayashi, and Graham Neubig, 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35.
Bogdan Gliwa, Iona Mochol, Maciej Biesek, and Alek- sander Wawer 2019. Samsum corpus: A human- annotated dialogue dataset for abstractive summa- rization. arXiv preprint arXiv:1911. 1223.7
Hila Gonen, Srini Iyer, Terra Blevins, Noah A Smith, and Luke Zettlmoyer. 2022. Demystifying prompts language models via perplexity estimation. arXiv preprint arXiv:2212.040377
Andrea Madotto, Zhaojiang Lin, Genta Indra Winata, and Pascale Fung. 2021. Few-shot bot: Prompt- based learning for dialogue systems. arXiv preprint arXiv:21 10.0811.
Karthik Gopalakrishnan, Behnam Hedayatnia, Qin- lang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tinn Tür. 2019. Topical-chat: Towards knowledge- knowledge-grounded grounded open-domain conversations. Proc. Interspeech 2019, pages 1891-
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021. Knowledge distillation: A survey. International Journal of Computer Vision, 129: 189--199.
Nils Reimers and Tryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th

---

International Joint Conference on Natural Language Processing (EMNLP- (EMNLP-IJCNLP), pages 3982-3992.
Francisco, California, USA, pages 3295-3301. AAAI Press.
Stephen Roller, Emily Dinan. Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Out, Eric Michael Smith, Y-Lan Boureau, and Jason We- ston. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguistics: Main Volume, pages 300-325, Online. Association for Computational Linguisties.
Xiaoyu Shen, Hui Su, Yanran Li, Wenjie Li, Shuzi Niu, Yang Zhao, Akiko Aizawa, and Guoping Long. 2017. A conditional variational framework for dialog gener- ation. In Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics ( (Vol- ume 2: Short Papers), pages 504-509, Vancouver, Canada. Association for Computational Linguistics.
Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung. Moya Chen, Kushal Arora, Joshua Lane, et al. 2022. Blenderbot 3: deployed conversational agent that continually learns to responsibly engage. arXiv preprint arXt::2288.03)
Ananya B Sai, Akash Kumar Mohankumar, Siddhartha Arora, and Mitesh M Khapra. 2020. Improving di- alog evaluation with a multi-reference adversarial dataset and large scale pretraining. Transactions of the Association for Computational Linguistics, 8:810- 827.
Emma Strubell, Ananya Ganesh, and Andrew McCal- lum. 2019. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.0220
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108.
Yi Tay, Mostafa Dehghami, Dara Bahri, and Donald Met- zler. 2022. Efficient transformers: A survey. ACM Comput. Sure, 55(6).
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, al. 2022. Multitask prompted training enables zero-shot task generalization. In ICLR 2022-Tenth International Conference on Learning Representa- tions.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoory Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du et al. 2022. Lamda: Language models for dialog applica- tions. arXiv preprint arXiv:2201.08 008299.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.
Bishal Santra, Potnuru Anusha, and Pawan Goyal. 2021. Hierarchical transformer for task oriented dialog sys- tems. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 5649-5658.
Ben Wang and Aran Komatsuzaki. 2021.
GPT- J-6B: A 6 Billion Parameter Autoregressive Language Model. https:// kingoflolz mweeffool/messs-eeeeee weesh-transfomeee -ax....
Timo Schick and Hinrich Schütze. 2021a. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 255-269.
com/
Sinong Wang. Belinda 7. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. ArXiv, abs/2006.04766
Timo Schick and Hinrich Schütze. 2021b. It's not just size that matters: Small language models are also few- shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Techmologies, pages 2339-2352.
Yizhong Wang. Swaroop Mishra, Pegah Alipoormo- labashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasckaran, An- jana Arunkumar, David Stap, al. 2022. Super- naturalinstructions:: Generalization via declarative instructions on 1600+ nlp tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 5085-5109.
Sander Schulhoff and Community Contributors. 2022. Learn Prompting.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoc V Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint 1xXiv:2109.0166
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. Bleurt: Learning robust metrics for text gener- ation. arXiv preprint arXiv:2004, 04696.
lulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. 2017. A hierarchical latent vari- able encoder-decoder model for generating dialogues. Proceedings of the Thirty-First AAAI Conference on A Artificial Intelligence, February 4-9, 2017, San
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. arXiv preprint 22x::2006.76683

---

Yu Wu, Furu Wei, Shaohan Huang, Yunli Wang, Zhou- jun Li, and Ming Zhou. 2019. Response genera- tion by context-aware prototype editing. In The Thirty-Third AAAI Conference on Artificial Intelli- gence, AAAI 2019, The Thirty-Firsi Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Hon- olulu, Hawair, USA, January 27 February 1, 2019, pages 7281-7288, AAAI Press.
turns for conversation across sessions 1, 2, 3 and 4 by concatenating them in a temporal way. Fur- ther, context-response example pairs for our exper- iments have been created by considering (i) second utterance of each turn of sessions 2, 3 and 4 as a response and (ii) first utterance of corresponding turn and entire conversation history as context. We also use the persona information background information when constructing input for various dialog models.
Jing Xu, Arthur Szlam, and Jason Weston. 2022. Be- yond goldfish memory: Long-term open-domain con- versation. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5180-5197,
The test split of the TC dataset includes two sec- tions: frequent and rare. This is based on the fre- quency of the associated entities as observed in the training set. We combine these splits to create our test set and pursue our analysis. The conversations begin with preprocessed reading set which is trieved from Wikipedia. Further, context-responss example pairs for our experiments have been cre- ated by considering (i) second utterance of each turn as a response and (ii) first utterance of corre- sponding turn and entire conversation history as context.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang. and Amr Ahmed. 2020. Big bird: Transformers for longer sequences. ArXiv, abs/2007.14062.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In In- ternational Conference Machine Learning, pages 11328-11339. PMLR.
In both datasets, for each sample, we normalize the utterances by removing trailing whitespaces, and capitalizing first word of every sentence.
Yizhe Zhang, Siqi Sun, Michel Galley, Yea-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan, 2019, Dialogpt: Large-scale generative pre- training for conversational response generation. arXiv preprint arXiv: 1911: 1.00536.
B Hyper-parameters for training dialog summarization models
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-lewe diversity for neural dialog models using conditional variational autoencoders. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (oolume
We used a batch size of and finetuned the models for 10 epochs. We tried using various learning rates (le-5, 5e-5, le-4, 5e-4, Ie-3) and finally picked a learning rate of le-4 since that gave the most op- timal performance on the validation set. During training we limited the maximum length of gener- ated summary to 128 and set the number of beams to 5.
1: Long Popers). pages 654-666 664, Vancouver, Canada. Association for Computational Linguistics.
Qingfu Zhu, Lei Cui, Weinan Zhang, Furu Wei, and Ting Liu. 2018. Retrieval-enhancee adversarial train- ing for neural response generation. arXiv preprint NrXiv:180.00426.
C Overall input length
A Data Preprocessing
Refer to Fig. 4 for comparison of overall input length for various representations of dialog prompt across the two datasets for the zero shot setting.
The MSC dataset divided into multiple ses- sions, the first of which uses dialogs from the Per- sonaChat dataset. Each session has metadata in- formation such as time elapsed from the past con- versation and previous dialogs. Examples from Session I do not have enough context. Hence, experiment with examples from sessions 2, 3 and 4 are used, and the results are averaged across the three. As per the dataset construction, a single conversation has been conducted across multiple sessions. Hence, as a first step, we aggregate all
D Detailed Model-wise Performance Results
In Figs. 5 to 8, analyze the absolute per- formances of various LLM model-families using prompts based on various input representations for TC and MSC resp. We show model-wise results for few shot (FS) as well as zero shot (ZS) cases across three popular metrics BLEURT, DEB and

---

500
MSC-Zero shot-Manual
challenge. Several Seq2Seq models (Serban et al., 2017: Shen al., 2017; Zhao et al., 2017; Bao et al. 2019; Santra al., 2021) 1) have been proposed to address the specific properties of dialog model- ing. Recently, a significant amount of focus has been on pretraining large dialog generation models like DialoGPT (Zhang et al., 2019), Plato (Bao et al., 2019), Blenderbot (Roller et al., 2021), Meena (Adiwardana et al., 2020), Blenderbot-3 (Shuster et al., 2022) and LaMDA (Thoppilan al., 2022) using the transformer architecture. Retrieval augmented generation (RAG) has been another prominent approach to tackle the dialog generation task in both large and small-scale models (Wu et al.. 2019; Gupta et al., 2020; Cai et al., 2021; Komeili et al., 2021; Zhu et al., 2018). Although large scale pretrained models like 175B Blenderbot-3 (Shuster et al., 2022) or the 137B LaMDA (Thoppilan al., 2022) lead to high accuracies across multiple dia- log datasets, these approaches can be prohibitively expensive due to the ever- ever-increasing size of mod- els. Large model size makes finetuning difficult. Also, in-context learning (ICL) with prompt-base based models makes finetuning unnecessary.
450
MSC-Zero haot-Peplexity-optiiieee
400
Zero shot-Manual
350
T T--Zero shat-Perplextyyooptiizeee
300
250
200
150
100
0
Semantic-1
Recent-1
BART-D
Pegasus-Dr DS
Semantic-
Recent-2
Pegasus- CD
Recent-4
Semantic-4
Semantic-8
Pegasus-Dr DS BI
Recent-8
Semantic-10
Recent-10
Full
Figure 4: Comparison of average input length for var- ious representations dialog prompt across the two datasets for the zero shot setting. DH Dialog History, BI = Background Information.
METEOR. We also show results for manually en- gineered as well as perplexity optimized prompts averaged across various models (FLAN-T5, TO, Tk- Instruct and GPT-3), Since we do not have access to logits from GPT-3 model, we cannot optimize prompts for GPT-3 using perplexity.
F.2 Prompt-based models
Prompt-based usage of LLMs and in-context learn- ing was introduced by Brown et al. (2020). In prompt-based approach, an LM is adapted to per- form specific task by priming it with instruc- tions and/or examples. Following the success of the in-context learning approach towards gen- cralizing NLP models, various other equally or more capable models based on smaller LMs have also been introduced. Smaller-sized LMs capa- ble of in-context learning are created using meth- ods like pattern- explooiinx training (PET, Schick and Schütze, 2021a,b) and instruction-finetune (TO, Sanh et al., 2022; FLAN, Chung al., 2022; II-Instruct, Wang et al., 2022). OpenAl's text- davinci-0032 has been trained using RLHF (rein- forcement learning using human feedhack). In- context learning-based dialog systems (Madotto et al., 2021) using LLMs like GPT-J (Wang and Komatsuzaki, 2021) or GPT-3 (Brown et al., 2020) have also been investigated, but it is crucial to se- lect the right prompts and context to achieve the best results.
E Detailed Model-wise UID Results
We show the model-wise UID results in Ta- bles 5, 6, 7, and for FLAN-T5, TO, Tk-Instruct and GPT-3 respectively. We show results for few shot as well as zero shot cases, and for both the datasets (MSC and TC). We also show UID results across three different metrics BLEURT, DEB and METEOR. For each of these combinations, we show results for different input prompt combi- nations: (1) Full dialog history, (2) Summary of dialog history using BART-D Pegasus-DSS or Pegasus- CD, (3) Pegasus-DS summary of dialog history as well as Pegasus-CD summary of back- ground information (BI), (4) Recent-k & selected dia- log utterances, and (5) Semantic-k selected dialog utterances, where k is varied as 1, 2, 4, 8, and 10.
F
Detailed literature review
F1 Dialog modeling
The development of open-domain chatbot systems that possess long-term memory, generate engaging and coherent responses, and perform equally well on a variety of dialog tasks has been a longstanding
httpps: httpt/ptatttttt openal. com/ .com/docst docs/ models

---

Eac XL
TX. Suste uct (E5)
(E5)
Instrust Th Insemumt (88)
flon XL (750
fannt XL (25)
Th. Justuued (25)
Thi (P5)
GPT-3 (199)
To (BS)
GPT--
To CES)
GPT-3 (88)
GPT (225)
(25)
(25)
(2022))
(25)
0.38
0.16
0.36
015
RLEUST
0.34
0.16
013
0.52
Pegeessoi (D)
Pepaan DS B
Boxmee
Booor
Rossn
S
Scomunte 20
7nl.
BART
Iegane
Degone bs-
Poganta BI
Becenn
Bevert-
Dasint I-
Scaankk
Semmmm-
Semunte
Senamie
Sottantee X-
BAK
Pounn CD Pepinttess
Recom to
Semanee
Semures
Semamttio to
Eistory Signal
Iheerry Signal
Figure 5: Performance results for Topical-Chie -Chat Dataset, Manually designed prompts. DH = Dialog History, BI
Background Information.
faail' XL.(8)
(29)
Bar XL (FS)
(28)
(28)
EacTS XL
Bastruct (ES)
Da X2 (25)
Ik Instrust (E8)
Tk- Instront (18)
T0 (PS)
Ta Sastrrcct (25)
(0.(FS)
Bastruct
1.0
0115
0.36
0.16
0.36
0.32
MET000
0113
0.30
Tegromecco CO
Dngrese 38
Pepinaa 06 30
Rocent3
Rocunt 9
Reccur
Sumantic k
Semuntte d
Secauntte
Sanartie 10
Poll
BARV
Pegante
Prgore- 36
Popano-18 30
Becont
Boont
leven
Scmande:10 30-
BATT:
Pepucss
Socant
Semmanee
Semcator 3
Eistory Sigual
Atgaal
Hltory Signal
Figure 6: Performance results for Popical-Chat Dataset, Perplexity-optimizze Prompts. DH= Dialog History, BI
Background Information.
Ti-00-- CRO
SanTi-IL (15)
Thcrmman (75)
Dunli DxxTT5
Bani'sAt XL 125
2h thiintra (20
Ticleeran 08%
TO CRO
car-a IPS)
u22 a (es) carta (25)
(30)
TO 220
GPT-3 (28)
(25)
03388
0.18
0.9
D03
BLEURT
0.36
METBOR
0336
0.8
0.54
0.7
BART-D
Peganu-CC CD
Pepress 1S
Pegionss Bll
Recent-1-
Recent-2
Recemtt 4
Becent-8
Racent Io.
Semontte)
Semantr: 2.
Semsntle-4
Semantic--
Semantie- 10-
Pall.
BART D.
PagressCC
Peguns DS.
Peguaus--S DS Bt
Recont-1
Teeent-2
Recent-
Recontad
Semanti: J.
Semantie2
Semantec4
Semartic
Semantiir-p.
Full
BART D
Pegoous-DD CD
Pogasus Us.
Peguour 18 B1
Rocont-1
Rocent
Recent-4
Rocent
Becent-1 10
Semmonec)
Sermatttte
Senuestic
Semuntic 10
Tull
History Signa)
History Signal
History Signal
Figure 7: Performance results for Multi-Sessiin-- Seriissession-Chatt Dataset, Manually designed prompts. DH = Dialog History,
Bl = Background Information.

---

7o 100
Bants
10
BanTisik, 200 TO (8R)
Thoinerrn (15)
DonTtS-t (800
The teveron 125)
0.9
0.16
0.36
0.8
BLBUWT
METBOR
0.14
0.34
0.32
0.12
066
BART-D
Pegusss CD
Pegteus D8
Pegiomu Bi Recent-)
Rocent
Recent
Becertt
Somanna
Semamii
Sementice
Semsntics
Semantie 1o
Pall
Dair 9
Regonnsss
yegare DS
Pegursr- Douua-18- BI
3
Recont-
Becent-
Recent 10
Semannte 1
Semanticc
Semautti
Semuratic
Semeetteclo
Pall
DART-D
Peggaus-en D
pegonss DS:
Pegaru -S EE
Rovont-3
Rocent 2
Revent-
Recent t
Recent 30
samentic
Semantie-s
Semantic 30
Full
History Signal
History Signal
History Sugnal
Figure 8: Performance results for Multi-Session-Ciie Dataset, Perplexity-optimized Prompts. DH Dialog History.
Bl = Background Information.
Manual Promup
Perplexily Prompt DEB
History
BLEURT
DEB
METBOR
BLEURT
METEOR
7S
FS
75
ZS
FS
ZS
BART-D
18.99
9.04
40.95
20.01
6.82
3.53
19.12
9:00
39. 38
23.44
7.18
4.05
Pegasus-CC CD)
18:24
8.15
37.94
18.26
6.26
3.17
18.02
8.16
37.41
22.84
6.39
3.79
Pegasus DS
18.75
9.97
40.85
19.92
6.83
3.49
19.05
9.55
39.26
19.11
7.15
3.64
Pogasus-DS
26.76
14:99
14.45
4.38
Recent- I
22.61
9.93
47.60
19.60
8.37
3,70
21:47
9:79
41.68
20.07
8.33
3.88
Revent-1 2
19.90
8.88
41.56
16.25
7,31
3,10
19.50
8.61
40.78
17.73
7.45
3.54
Recent-4
15.59
7.12
33.25
13.34
3.80
2.47
15.47
6.59
33.2
13.07
5.89
2.70
MSC
Recent-8
11.51
5,39
25.23
10.71
4.27
1.92
11:44
4.86
25:51
10.35
4.34
1.97
Recent:
10.37
23.19
10:03
3.86
1.78
10.33
4.38
Semamtic- I
23.83
10.02
44.73
18.74
8.48
3.98
23.36
10.38
44.38
18.70
8.59
4.19
Semantie-2
20.23
8.44
39.25
16.30
7.38
3.44
20.07
8.85
17.10
7:43
3.52
Semamtic 4
15.74
6.64
34,75
14.86
5.87
2.71
15.64
6.90
33.89
16.96
5.85
2.90
Semaniic8
11.07
4.79
23:10
10:28
4.22
2.07
11.21
4.78
22.5%
10.78
4.22
1.92
Semanti-
4.27
20.20
9.28
3.66
4:21
9.73
3.68
1.70
Full
6.77
3.33
16.26
7.92
2.43
1.44
6.67
3.16
16.97
7.78
2.53
1.42
BART-D
25.68
8:44
5.50
53.
25,25
8.54
Fegasus CD
20.18
10.09
46.06
22.21
6.59
3.32
19.80
9.91
46.24
22.30
6.75
3.36
Pegasus-DS
8:46
53:12
25.37
18:52
Pegnsus- DS + Bl
7,85
4.28
20.87
9.67
2.86
1.58
7.51
3.95
19.79
9.22
2.87
1.51
Revent- I
25.81
11.24
57.44
18.38
8.87
3.45
24.16
11.12
$3.39
27.82
8.82
4.11
Recent22
21.76
9.55
48.
15.92
7.48
2.93
21.02
9:43
47.05 97.09
24.25
7.55
3.78
Recent-4
17.08
7.75
38.
12.76
5.97
2,30
16.83
7.30
38.
19.43
6.04
2.98
Recent-8
12.69
5.77
28.86
10.29
4.47
1.79
12.47
5:37
29.04
14.32
4,49
2.18
Revent 10
11.39
5,21
26.45
9.63
3.98
1.65
11.14
4.88
26.85
13.05
4.04
1.98
Semantie-1
22.52
5W
29.45
8.80
Semantic 2
21.60
8.91
49.54
19.28
7.50
3.18
21.45
9.46
50.46
25.7
7.62
3.67
Semmiic-4
15.35
7.35
40.89
16:03
6.42
2.62
16.40
7.50
39.65
20.53
6.12
3.13
Semantic-8
12.16
5.25
29.03
11:01
4.36
1.80
12.14
5:33
29.64
14,94
4.44
2.10
Semantic-10 10
1082
4.78
26.
10:00
3.88
1.63
10:73
4.85
27.07
13.46
3.93
1.91
4.03
22.50
3.26
1.56
4.01
22.69
11.70
Table 5: UID results for the FLAN-TS model.
F3 Compute Intensive LLMs
distillation-basee (Hinton et al., 2015; Sanh et al., 2019; Gou et al., 2021: Gupta and Agrawal, 2022) methods train a smaller, simplified model to ap- proximate the predictions of a larger, more com- plex model. Efficient transformer architectures, such as Reformer (Kitaev et al., 2020), Linformer (Wang et al., 2020), BigBird (Zaheer et al., 2020), and Longformer (Beltagy et al., 2020), aim to duce the quadratic complexity of the standard trans- former architecture by using more efficient self- attention mechanisms.
One of the most critical drawbacks of these LLMs is the training and inferencing cost, especially for long sequences. Other than the complexity of a single forward pass, there are other costs in- volved in training an effective transformer LLM,, e.g., amount of training data and compute needed (FLOP). Strubell et al. (2019) discusses the envi- ronmental impact that the training process of these LLMs has, in terms of total CO2 emissions. Op- timizing costs of LMs has mainly been explored from the perspective of incrcasing the efficiency of the inference step of a transformer. Model
In this paper, we examine the costs associated with the use of Large Language Model (LLMs)

---

Manual Prompt
Perpieiity Prompt
History
BLEURT
DEB
METFOR
BLEURT
DER
METEOR
28
FS
ZS
ES
ZS
1S
28
13
ZS
FS
ZS
T3
BART-D
3.81
21.18
23,85
Pegustss CD
16.26
7.96
46.51
22.45
7.34
3,80
18,93
7.85
49.86
21.97
7.71
3.74
Pogasus-DD DD
18.59
20.98
Pegasux- DS Hl
11.36
5.55
30.66
15.35
4.62
2.36
11.73
5.33
29.48
14.21
4.59
2:24
Revent-
20.94
9.34
55.82
25,49
9.24
4.50
25.80
8.90
58.37
22.72
9.77
4.28
Recent-2
18
48.24
22.80
3,56
201,99
51.00 08
20.54
Recent
13.13
6.65
37,46
18.30
6.37
3.15
15.53
6.41
39.72
16.64
6.52
3.07
MSC
Recent-8
2.40
11.30
Recent- 10
9.63
4.61
26.07
12.88
4.31
2.20
10.15
4.52
26.
11,93
4.34
2.21
Semantic11
22.40
57.11
25,449
4.59
25.37
24.99
4:25
Semantic
18.85
8.49
48.01
2.1.82
7.82
3,90
20.93
8.12
$2.31
21.94
8.35
3.70
Semanbi-4
14.40
6:88
39.18
19.26
6.60
3.21
15.92
6.62
42.33
18.51
6.75
3.17
Semantic-8
10:45
27.34
12:78
2.332
11.16
12.52
Semambe- 10
9.10
4,25
24.23
11:37
3.90
2.07
9.76
4.17
24.74
11.03
3.82
2.03
Full
9.34
8.67
BART
23.04
9.68
62.76
22.55
18.79
3.91
26.50
8.79
65.90
15.87
9.39
3.47
Pogasus- -CD
15,58
3.21
8.91
Pegasus-DD DS
22.81
9.56
62.377
22,40
18:75
3.86
266.23
8.78
65.44
16.13
9.33
3.42
Pegasus DS DT
7.74
3.44
21.39
7.87
2.64
1,46
7.78
3.23
21.27
7.51
2.66
1.40
Recent-
22. 22.30
10:17
63.52
29.15
9:666
4.64
25.66
9:55
27.83
9.96
4.18
Recent-2 2
18.68
8.86
53.001
25.42
8.28
3.92
2.1.7
8.34
55.82
24.20
8,48
3.66
Recent-4
40.75
20.37
3.14
16.39
43.15
19,53
2.98
TC
Recent-8
10.95
5.27
30.666
15.36
4.72
2.37
11.95
5.14
31.75
14.87
4.72
2.31
Rocent-10
9.90
2:16
Semantic
22.63
10.17
63.22
28.76
9:25
4,60
25.988
9:53
66.90
27.78
9.67
4.12
Semmmbee2
18.83
8.83
52.98
24.89
7.87
3.91
21.39
8.23
56,92
24.03
8.25
3.60
Somantic-4
14.62
6.96
42.36
20,10
(.50
3.07
15.9%
44.00
19.15
6.84
2.93
Semambc
10.91
5.15
30.98
14.91
4.44
2.29
11.80
4.98
32.35
14,50
4.55
2:21
Semanix-TO
9.70
28.12
Full
7.76
3,86
23.18
11.52
3.36
1.76
8.08
3.89
23.56
11.28
3.53
1.71
Table 6: UID results for the TO model.
Manual Procmp
Perpiesity Pronipt
History
BLEURT
DEB
METEOR
BLEURT
DEB
METEOR
28
F8
23
25
F8
BART-D
21.99
8,39
47.38
22.25
7.26
3.64
21.51
9.35
44
24,13
6.48
3.78
Fegusus
17.95
7,89
47.15
20.99
6.22
16.30
8.19
36.13
21.84
5.38
3.50
PSgasus-DS
22.15
9.27
47.58
19.32
7:32
3.26
20.62
9:32
44.55
19.75
6.54
3.25
Pegnous DS
1282
5.83
29.94
14.71
4.32
2.39
12.23
5.66
28.13
14.17
4.05
2.30
Revent-1
22.64
9.52
54.86
24.34
8.34
3.71
21.61
10.09
38.
21.97
7.39
3.82
Recent-2
19.48
8,30
48.21
21
7.24
8.37
18.95
8.95
36.66
19.73
6.58
3.47
Recent
15.24
6.57
38.59
17.05
5.78
14.85
7.05
16.74
5.35
MSC
Recem-8
11.32
5.01
29.30
12.82
4.28
2.03
11:02
5.33
24:85
13:15
4.00
2.17
Recent
10.26
4.58
27.01
11.75
3.84
4.87
12.17
Semantic-1
2403
9.92
50.24
21.50
9:30
3.86
23.96
10:62
47:26
20.77
8:03
4.15
Semanlic 1
20.14
8.47
44.33
19.32
7.19
3,38
20.25
9:07
39.46
18.99
6.90
3.57
Semumlic-4
15.63
6.70
40.74
18.08
3.98
2.77
15.31
7.09
34.57
18.58
5.54
2.92
Semantic 8
11.11
4,79
25.74
11:93
3.96
1.91
11:21
5.06
24:27
12.45
3.87
2.01
Semantic-
9.71
4.27
22.65
10.54
3.48
4.49
21.68
11.16
3.38
1.78
Fall
6.86
3.29
18.84
8.60
2.47
1.42
6.78
3.44
17.
9,23
2.41
1.50
D
56.80
23:75
8.15
26,
7.45
3.53
Pogasus-CI
19.09
8.52
$1.04
18.85
6.34
3.02
17.83
8.81
47.84
17.71
5.57
2.79
Pegnuus-1D5 DS
26.97
11.18
56.85
23:70
8.08
3.78
25.88
12.17
57.89
25.97
7.44
3:52
Pegusus DS + BI
7.66
4.01
20.04
8.81
2.49
1.43
7,49
3.70
21.22
8.09
2.53
1.39
Recent-1
25.13
10.07
61.05
28.29
8.57
3.99
25.13
11.81
56.07
28.54
7.70
3.88
Revent
21.15
8.66
53.67
24.08
7.52
3.60
21.35
10.04
25,29
7.12
3.58
Recent 4
16.50
6.98
43.09
18.67
6.03
2.82
16.78
7.88
42.15
20.48
5.83
2.90
Recent-8
12.19
5.32
32.51
13.67
4.47
12.31
5.90
31.76
15.59
4.36
Recent-10
10.96
4.87
29.58
12.42
4.01
1.91
11:03
5.35
29:00
14.22
3.94
2.01
Semantic
24.85
10.30
60.19
28.42
8.20
4.01
25.29
11.48
62.78
29.42
8.16
4.09
Semmitic-2
20.68
8.66
52.68
24.96
7.12
3.51
21,46
9.67
53.67
25.73
7.09
3.54
Semantic 4
15.54
7.10
43.10
19.57
6.13
2.89
16.54
3.78
43.75
20.52
6.12
2.22
Semantic
11.77
5,13
31.43
14.93
4.12
12.10
5.61
31.43
15.38
4.04
2.10
Semantic 10
10.58
4.68
28.52
13.60
3.67
1.89
10.84
5.10
28.55
13.97
3.60
1.90
8.45
24.00
11.17
3.20
11.73
Table 7: UID results for the In-Instr model.
and suggest new metrics for assessing the cost- performance trade-offs involved, as well as strate- gies for optimizing the inference cost with regard to the inputs.
G Full list of prompts
G.1 Manually engineering prompts
In this section, we provide full list of manually engineering prompts. Tables 9 to 14 show prompt instances for six different settings: zero-shot versus few-shot, and passing persona versus dialog history summary versus both as context. The generations

---

Manual Promps
bottleneck. We analyzed the accuracy-length trade- off using different values for the parameter "a" capture various types of user requirements terms of the allowed expenses towards the inference pro- cess. The average UID values (for zero-shot man- ual prompts) across all the models are shown in Tables 20 and 21. Based on these experiments, we found the following insightful observations. These tables show that for both MSC and TC, for DEB and METEOR, as the value of "a" is increased, summary- based dialog history variants tend to be- come better in terms of UID while Recent-k and Semantic-k variants tend to become less impressive. Although, in terms of BLEURT (UID), the ranking is in favour of Semantic-1 I or 2 and Recent-1 I or 2 throughout the complete range of "a" that we have explored. This might be because BLEURT mea- sures normal sentence semantic similarity but not context- context-response relevance measured by DEB.
History
BLEURT
METEOR
ZS
FS
25
ES
Z8
13
49.10
Pegusus CD
18.59
8,86
46.71
22.60
8.74
4.24
Popasur-DS
48.84
Pegasuss DS v BI
12.86
6.22
31.24
15.78
6.37
3.07
Revent-
22.70
10.51
55.12
25,94
9.89
4.78
Recont-2
19.97
9.36
47.94
22.99
9.01
4.29
Recent 4
15.77
7.52
31.78
18.45
7.41
3.50
Recent-3
28.60
Recent 10
9.91
5.31
24.05
13.03
4.31
2.49
Semantic-1
10.98
52.83
Semantic
20.22
9.51
45.19
21.97
9.10
4.39
Semantic-4
16.21
7.71
39.61
19.29
7.63
3.63
Semantic-8
5.06
26.16
133008
5.63
Semantic 10
10.42
5.06
23.38
11.69
5.03
2.41
7.92
9.48
BART D
22.34
10.92
56.84
26.67
9.28
4.34
PC
Pegasus--Dp
22:27
10.85
56.57
26.50
9:28
4.33
Pegasus US 61
7.45
3.77
20.72
10.00
3.78
1.82
Recent-
23.43
11:001
59.42
28.51
9.27
4.47
Recent
19.81
9.48
50.90
24.90
8.26
3.93
Recent-4
40.23
6.79
TC
Revent
11.85
5.79
30.22
15.29
5:25
2.49
Recent-10
27.63
Semantic
27
10:95
59.43
28.51
8.98
4.44
Semantsc-
19.48
9.41
50.39
24.74
8.07
3.89
Semantic-4
7.52
40.58
19.88
Semantic
11.43
5.63
29.37
14.83
5.06
2.43
Semantic-U
26.76
Fall
8.99
4.34
22.82
11.36
3.95
1,87
MSC (DEB Rank)
Table 8: UID results for the GPT-3 model. Manual prompt only, since prompts for GPT-3 cannot be per- plexity optimized.
Iecent-1
Vegesme-C3
Scmantic:
Semanticc
BART-Q
PegeanDD
are from the GPT3 model. Rather than the per- sona, when use knowledge facts, the prompt templates remain the same. When the dialog con- text consists of components other than summary, e.g., full history or recent-k utterances or semantic- k utterances, "summary" in prompt templates is replaced with "full history" or "list of recent-k ut- terances" or "list of semantic-k utterances" respec- tively.
PegasusoS
BARTIO
Recent-2
Recent
Semantic 2
Recent
Negasus-CD
Recent-
Semantic
Pegaoustts
Recent-4
Semantic-1
Pip
Recern-10
Recent
Semantic-2
Sementic:
Recent-1
Recent 10
Ful
Sementic-10
Semantic-8
G.2 Perplexity optimized prompts
Ful
Samantic-10
Since we have only API access the GPT3 model, we could perform perplexity optimization for only Flan T5 XL, TO and Tk-Instruct models. Tables 15 to 19 show perplexity optimized prompts (tem- plates as well as instance) for Flan TSXI,, TO and Tn-Instruct models under various settings like (a) zero shot versus few shot, and (b) persona, sum- mary, knowledge section combinations as dialog context.
Figure Trend Ranks of History Signal Types for Different Values of the Metric- Index (for MSC dataset, DEB metric)
To fully understand how the rank of various history signals vary over the value of the metric- importance "a", we plot the rank-order of all his- tory signal types vs. the value of "a" (increased from 0.5 to 10) as show in Fig. 9. This rank or- der dynamics helps clearly understand, we give more and more importance the model per- formance and ignore the cost of inference, how the choices over the history signal change. For example, terms of the UID (DEB) metric on the MSC dataset, the average trend across mod- els is that Recent-1 and Semantic-1 are the recom-
Impact of varying metric-importance index (a)
We vary a [0.5, 1, 2, 5, 10). This updated for- mulation of the UID metric, with MH raised to an exponent "a", can be used capture the im- portance assigned by the user on the model perfor- mance MH. e.g., when inference cost is less of a

---

Automafed Chat System:
This is summaryy of a dialog exchange between Personl and Person2: Person1 and Person2 talk about the lelt handed QB in the NEL and the origins of the huddle. They also talk ahout Rush and the law that makes people listen to 40% Canadian music. Given the sumuaury of the dialug eachange between Personl and Personz, give a consistent and diverse respouse to the following dialog by Personi. He was terrible bockey player, so he took up music. Would you listen to news broadcast in Latin?
Persond:
Person2:
Generation: Sure, I'dl be happy to listen to a news broadcast in Latin. Latin is a great language with a rich history and culture, so would he very interesting beur broudeust in that language. Additionally, I'm always open to exploring new vullures and linguage, this would be great upporiunity leam more.
Table 9: Manually engineered prompt with summary of dialog history as dialog context and without exemplars or background information.
Automated Chat System:
Learn from the below example on to genernte consistent diverse responses hetween Personl and Person2 given summary. Example: This is summary of a dinlog exchange hetween Personi and Person2: Pansoni and Penson2 are talking about Nintenda. They didn't 1 know Nintendo was over years old that they started out as a playing card company. Person2 also tells Personl the Amy uses Xbox 360 controllers and Personl says homans play about 3 billion hours of video games a week.
Given the summury
dialug exchange between Personl and Person2, give consistent and diverse respouse following dialog by Person PersonI: Yeah that would have been a problem for me as a kid baba. Do you know who Stephen bry is? He did the uarration for lot of movies and video
games like the Hanry Potter video games.
Person2: Yeah. He also did the narration for Litlle Big Planet.
Now it yourself:
This is summary of dialog exchange between Persont and Person2: Personl tells Person2 about a Notre Dame quartomack named lan Book.
Given the summary the dialog exchange between Personl and Person2, give consistent and diverse response the following dialog by Persont.
Personf: the chance to play for notre dame must be great. they love foothall there.
Person2:
Generation: Absolutely! Notre
football historr and huge base. a huge huncur for any player to chance to play for them.
Table 10: Manually engineered prompt with summary of dialog history but no background information as dialog context and with one exemplar.
Automated Chat System:
Here are some background details about Personl: Personl likes the Lord of Rings movies and riding a bike, while Person] travel agent, listening to music to relax, and Personl could not be a vegetarian.
Here are some background detuils about Person2: Persen2 likes to talk about Personl's 's favorite books, characters, and places. Person2 runs charity and PersonZ's parents are doctors.
Given the background details of Personl and Person2, give consistent and diverse response to the Following dialog spoken Personl.
Personl: WHAT kind of music do you listen to when you are relaxing?
Person2:
Generation: ussaalyy listen to variety of music when I' relaxing, ranging from classical pop.jarz, and hip hop. I'm always looking
discover something new and interesting.
Table II: Mamually engineered prompt with persona as dialog context but no summary of dialog history and with no exemplars.
Automated Chat System:
Learu from the below example on how to use background details to generate a consistent and diverse response by Personz on what Personl says. Example:
Here are some backgroend dotails about Personl: Personi loves ice cream and works a fist food place. Porsonl in high school and plans become
eagle scout. Personl would love to volunteer at the church.
Here are some background details about Person2: Person2 recemlly gave to and wants many more children. Person2 is stay-al-homee mom who plays tennis and goes to the charch.
Given the background details Personl and Person2. give consistent and diverse response to the following dialog spoken Personl,
Persont: THAT's a lot of kids! BUT since you love the church so much having enough for a charch choir sounds like good idea! HAVE you had any lock finding valunteers? WHEN is your retreat?
Person2: WE've ve gollen some volunteers, buil always need more! THANK you for getling your scout troop involved. WE are having another retreal next mooth. II'll he youth retreat you and your friends would he interested.
Now try it yourself:
Here some background details about Personl: Personl revenily gave birth to twiss and wants many children. Personl is stay homee
who plays tennis and goes to the church.
Here are some background details about Person2: Person2 loves ice cream and works a fast food place. Person2 in high school and plans to become
eugle scout. Persun2 would love to volunteer the church.
Given the background details of Personl and Person2, give a consistent and diverse response to the following dialog spuken by Personl. THANK you for potting your scout troop involved. Wis having another
Persont: WE've potten some volunteers, but always month. II'll be a youth retreat if you and your friends would be interested.
Persom2:
Generation: That's is great sure your scont trop will he excined to help out at the retrear. I'd lo see If can get some of my friends involved too, I'm sure they 'I be interested in altending the retreat as well.
Table 12: Manually engineered prompt with persona as dialog context but no summary of dialog history and with one exemplar.
mended ways to summarize the context information if cost is an important factor to the user. Whereas, if cost is of less importance, then longer dialog summaries such as Pegasus-Ch and Semantic-4
are recommended approaches for MSC. Addition- ally, we observe that some models like Recent- 10, Semantic-8, Semantic- 10 and Full are always bad choices, while some (BART-D, Pegasus-DS) are

---

Automafed Chat System:
Here are some background details about Personl: Personl likes Blying kiles and works as zoo keeper. Person has a bearded dragon and pet raccoon. Personl recommends that people adope raccoons.
ure some background details about Person2: Person2's s self employed artist. Penson's looking for animals adops. They like of getting a pel raccoon, Penson2 has dog.
summary dialog exchange between Personl Person2: Person2 taking his deg a work sometimes. Personl bearded dragon and racoon as zoo keepets. Personl recormends pet racoon to Person2.
Given the background defails and summarry dialog exchange between Personl PersonZ, give consistent and response the following dialog by Persont.
Persont: MY pet raceoon just learned trik!!
Person2:
Generation: Wow! What kind of trick did your pet raccoon learn? I'm always interested in learning new things that my pet can do!
Table 13: Manually engineered prompt with persona and summary of dialog history dialog context and with no exemplars.
Automated Chat Learn from the below example on how to generate consistent and diverse responses between Person] and Person2 given background details along summary. Example:
Here are some background details about Personl: Personl lowas ice cream and works at a fastt food place. Penson in high school and plans to become
eagle
Personl I would love volunteer the church.
Here are some background details about Person2: Person2 revenily gave birth to twins and wants many more children. Person2 is stay atl homne mom who plays tennis and goes to the church.
This is summary of a dialog exchange between Personl and Person2; Person2 plays tennis every week and goes to church. Person] works a fiist Food place and loves ice cream. Person2 his twins and wants start choir. Persoal hopes to become cagle scout.
Given the background details and the summary of the dialog exchange between Personl and Person2, give a consistent and diverse respouse to the following dinling by Person
Personl: THATTss a kids! since you the church so much having enough for a church sounds like good HAVE you any luck
linding volunteers! WHEN is your next retreat?
volunteers, but always need more! THANK you for getting your scout troop involved. WE having another retreat mooth. IT'll he youth retreat you and your friends would he interested.
Person2: WF ve gotten
Now yourself:
Here are some background details about Persont: Personl recently gave hirth to twins and wants many more children. Person is stny-at-homm mom
who plays lennis and gues to the church.
Here are some background details about Person2: Person2 loves ice cream and works a fast food place. Penson2 in high school and plans become
eagle scour. Person2 would love to volunteer the church.
This is summary of a dialog exchange between Personl and Person2: Person2 works al a fast food place and loves ice cream. Personl is a slay-at-home mom and wants to start her own choir, Persne2 an cagle soour and manages school and work. Person] is starting to get back to tennis after having twins. Given the sammary the dialog exchange between Personl and Person2 and their background details, give a consistent and diverse response to the following dialog spoken by Personl.
Personl: WE've ve gollen some volunteers, but always mooth. IT'll he youth retreat you and your friends would he interestad.
THANK you for getting your scout trop involved. WB having another retreat next
Person2:
Generation:
'hat sounds great! would love to volunteor the retreat and got my friends involved as well, How can help? What kind of activ do you have planned?
Table 14: Manually engineered prompt with persona and summary of dialog history dialog context and with one exemplar.
quite robust across the whole range of values of "a"

---

Serting
Prompe
Only Summary Few Shot
Take look at the following example for guidance. Example: Here is conversation summary between Personl and Person2: Person2 and Persont ahout their hobbies. Person2 has hig family and Person2 Personl has wwis and enjoys playing tennis. They also talk about other hubbies they might want to lake up for their kids. Bused on the summary conversation
hang out with Person2's mom on her days from fedex,
Person1 and Person2, what do you think Person2 will say next? Persont: WELL gasss can try swimming. WHAT are your tips when you want to try something but are afraid of it? Person2: think the most important thing to remember is that it's okay if you fail. YOU can always get back up and try again. I that when play tennis but I do it with everything else I do, too. I like cooking 1 like busketbull. I even like playing video games. I always just ty to luve fun and if fail, I Ity again. I know you like lennis but what else do you like to do? Now try yourself. Here a conversation summary between Person and Person2: Person2 and Personl about their hubbies. Person| the movies, and playing tonnis. Then they talk about other hobbics they might want to take up for their kids. Based on the summary of conversation between Personl and Person2, what do you think Person2 will say next? Persont: I think the
2 beothers
2 sisters. Person2 has big family and Person2 enjoys shopping, going
important thing remember is that it's okay if you fail. YOU can always get back up and try again. I do that when play teamis but do it with every everything ehe do, too, like cooking. I like basketball. even like playing video games. I always just try have fon and if fail. I try again. know you like tennis but what else do you like to do! Person2: like cooking. I like basketball. I even like playing video games. I always just try to have fun and I fail, I try again. I know you like tennis but what else do you like do?
Only Summary
Here a summary of the conversation between Personi and Person2: Person2
Personl talk about their jobs the food industry. Personl works at car deulership in sales and likes music. Person2 likes 21 pilots and Person2 goi the new 2021 subaru as Person2's is dream Based on the dialogue between the Personl the Person2
Zero Shot
try anticipate what the Person2's response might be the Personl 's next statement. Personl: YEAH they are really nice cars. HOW much did you pay for it after? Person2: I hope to over the school.
summer. just
Only Persona
Learn from the example first. Example: Here is some information about the Personi and Ferson:: Person2 has been to Baston hat grew up San Francisco. Person2 works lot and fishs in his spare time. Person2 married with children. Personl an English teacher in Boston. Pesson] likes drawing and Hish 'n Chips. Person] and Personl like Batman and Fish "n Chips. Based on the information provided about the Personl and the Person2, predict what might have been suid following this dinlogue by the Person Personl: I west fishigg yesterday. I pulled enough
Few Shot
have dimner. Personl: WOW!! THAT's great! WHO cooked? Now tey yourself, Here some information about the Person1 1 and Porson2: Person! and Person] are loners, Personi likes grilling and hiking along the occan, water, while Personl
water, Personz's thinking fieding new job. Person2 tells Person2 about Person2's 's personality. Person2's 2ss nervous about hearing back on Wednesday. Baxed on the information provided about the Personl and the Person2, predict what might have been said following this dialogue by the Persoul. Personl: CONGRATI ATUINSI BF sure show them you should have hoan their first choice. SO more family time the heach now? Person2: MY kids have been asking for a cat. I don't 1 know cats and water go along. is some information about the Fersonl
Personl is scared swimming
swimming
Only Persona
Fersonl: Fersonl and Personl talk about their hobbies. Personl likes to work computers, read, and ride bike. Personl is vegetarian. Person2 is a car salesman and Person2 is fan of the hand 5 Finger Death Punch and Slipknol song Duality and its music video Based on the information provided about
Zero Shot
the Person2, predict what the Person2 might have said in response to the Persont's i's dialogne. Persont: THAT's a huge namber, especially given the commissions you get per car? Person2: WT get 25% of the gross profit of the HOW is your job? ARE you
Persoul
you must get. WIIA kind of looking for another?
Persona Sum-
Take a look the following example for guidance. Example: Here some information about Personl and Person2: Person2 and Person2 are talking about their personal churacteristics. Person2 is single and works from borne duing poyprmmyinng and web design. Person2 doesn't like hirds and wants in watch North by Northwest. Personl single and Porson likes conspiney theories. Personl" 's favorite birds. Here conversie conversation summury between Personl and Person2: Person2 is single and doesn't want to got married. Persen2 prefers to read outdoors alone. Person2 is on introvert and warries ahout running wild animals. Person? just bought north by nonthwes. In the light of the conversation summary and the information provided about the Personl and the Person2, predict what might have been suid following this diulogue by the Person I. Person LET me know
mary Few Shot
llitchock movies are Vertigo, Rear Window, and North by Northwest. Personl doesn't
good onc. watched city of lics lest night. IT was pretty good. I want to see wanderer next. jerk. Now try yourself. Here some information about the Personi and Person2: Personl and Personl are talking about their personal characteristics. Person| is single and works from home doing programming and web design. Personl doesn't like birds and wants to watch North by Northwest. Person2 is single and Porson2 likes conspiracy theories. Person2's a favorite Hitchoock movies are Veriigo, Rear Window, and North by Northwest. Person2 docsn't like birds. Here conversation summary between Personl and Person2. Personl single and doesn't I want get married. Person! prefers read outdoors alone. Pensonl worties about running into wild animals. Personl bought north by northwest. Person2 watched city of lies last night. In the light of the conversation summary and the information provided shout the Person and the Person2, predict what might have been
Person2: think most hitchcock's hold up man he was
following this dialogue by the Personl. Personl: think most hitcheuck's 's hold up man was kind of a jerk. Person2: LET me know if its good one. I watched city of lies last night. IT was pretty good. I want to see wanderer next.
Persona Sum-
hobbies. Personl likes camping. leisure aciivities coffee, and Corona beer. Personi went hiking on memorial day. Person2 loves the beach. Person2 likes to cook and make coffee. Person2 lives in California and has kids. Person2 thirry years old. Here is conversation summary between Persont and Person2: Personl iss family asked her to go camping for the holiday weekend. Person2 visiting her family over the summer. Person2 is going Las Vegns with her mom and sisters for weekend, Based on the persona information of the Person1 and the Person2 and their conversation summary so far, anticipate what the Person2's l response might be the Personl's iss next statement. Personl: ARE you a gambler or are you going there to people watch and to shows? Person2: I'm a gambler I'm going to the vegas
are some personu details about
Personl and Person2: Personl and Fersonl talk about
mary Zero Shot
Only
Knowl-
Here is some data on the topics the Personi and Person2 are discussing ahout: Spnder-aan first appcared in the anthology comic book Amazing Fantasy 15 (August 1962) in the Silver Age of Comic Incks.. the stories, Spider Man is the alias of Peter Parker, an orphan raised by his Aunt May and Unele story has him acquiring spider-related ahilities after a hite from a mddractive spider, these include clinging to surfaces, shouting spider wehs from wrist mounied devices. Marvel Comics the brand name and primury imprint of Marvel Worldwide Inc. In 2009, The Walt Disncy Company aequired Marvel Entertainment Marvel Worldwide's 's pureni company. carly 1950s, had generally become known Allas Comics. Comic books were first popularized in the United States and the United Kingdom during 1930s .The Marvel Cinematic Universe lakes place in Earth- 199999, mulliverse different from the original comic book universe. Earth-616 Rased the data about the toples being discussed the Personl and the Person2, anticipate what the Person2 might have sald next in respouse to the Personl's dialogue. Personl: NAHI usually eat a yogurt berry smoothie for breakfast. YOU? Person2: I usually eat a bowl of outmeal. What do cat for breakfist? I'm curious.
edge Zero Shot
New York City after
parents Richard and Mary
were killed
plane crash. origin
Marvel Marvel started in 1939 as Timely Publications, and by the
term comic book derives from American comic books once heing compilation of comic strips of a humorous
Table 15: Perplexity Optimized Prompt Examples for F5XL. Template part is shown in bold. The last output for the Person2 is the generated output using FlanT5XL. Part 1

---

Setting Only odge How Shot
Procupt
Knowl-
Learn
example first, Example: Here some information on the tupics being discussed by Personl and Person2: The name Rotien Tomatocs derives from the practico of audieness throwing rotton tomatres when disapproving of poor stage performance. Michael Bay's average Rotien Tomatoes film rating 38%. film based video gume has achieved above 44% rotten tomatnes. Netflix has almost 150 movies available with 100% rating on Ratten Tomatoes. Incredibles 2 is 2018 American computer animated superhero film .It the sequel to The Incredibles (2004) and the second fuil-length installmeni of the frunchise .Craig Nelson, Hully Hunter. Surah Vowell and Samuel Jackson reprise their roles from the first film... newcomers the cast include Hackleherry Milner, Bob Odonkirk, Catherine Keener and Jonathan Banks Box office business can be measured in the terms of the number of tickets sold or the amount of money raised
sales (rovenue). With over 5 billion worldwide film carnings, .Teee Silence of the Lambs came out on Valentine's is day in 1991 and had a hox office of over $270 million. Rased on the data about the topics being discussed by the Personl and the Person2, anticipate what the Person2 might have said next in respomse to the Personl's I dialogue. Personl: MARS is also considered lerrestrial planet and the fourth planet from the sun. MUST be hot there. Person2: BUT it his polar icecups like us. Now try yourself. Here some data on the topics the Person1 and Person2 are discussing about: A planet is an actronomical body orhiting a star stellar remmant
Hanks
highest all-time box office star
massive enough to rouded by own gravity .he term planet is ancient. with ties to history, ustrology. science, my mytholoyy and religion. .IIn 2006 the International Astronomical Union (IAU) oflicially adopied resolution defining plancts within Solar System planet astronomical body orbiting star or stellar remnant that massive enough to be rounded by its own gravity... . TThe term planet ancient, with ties to history, astrology, science, mythology, and religion .IIn 2006, the Interstionnal Astronenical Union (IAU) officially adopted resolation defining plancts within the Solar System. planet astroaomical body orbiling a siar or stellar remmant that is massive enough to be rounded by its own gravity . The term planet uncient. with ties to history, astrologg, seience, mythology, and religion. In 2006, the International Astronomical Union (IAU) officially adopted resolution defining planets within the Solar System. Based on data about topics being discussed
Personl auticipate hhat the Person2 might have said next in response to the Personl's iss dialogue, Personl: OH man. Units sad for him. like his movies usually, Person2: don't recall many his mevies, I've sean couple and they are alright. I did like him in that 70s show!
the Person2.
Knowledge
Learn from the example first. Example: Here is some data on the toples the Personl and Persom2 are diseuesing about: The Walt Disney Company
Summary Few
founded October 16. 1923 brothers Walt and Ray O Disney the Disney Brothers Cartoon Studio .he company established itself as leader in the American animation industry before diversifying into live action film production. television. and theme parks II the world's largett independent media conglemerate in temms of revenue. The film a live-action reinagining of Disney's x 1991 animated film the same name, itself an adaptation of Jeanne-Mai Marie Leprince de Reaumont's 18th-century fairy tale. film features an ensemble cast that includes Emma Watson and Dan Stevens as the eponymous characters with Luke Evans, Kevin Kline. Josh Gad, Morgan Freeman won an Academy Award in 2005 for Best Suppotting Actor with Million Dollar Raby .ee has received Occar nominations for performances in Street Smart (1987), Driving Miss Daisy (1989). The shank Redemption (1994), and Invictus 2009 .Morgan Freeman played a singing vampire obseased with vegetables on The Elecaric Company in the 70's Here conversation summary between Persont and Person2: Person2 and Porsont are talking about the movic Deadpool 2. They also talk about Marvel comics and the X X-men. Person? likes
Shot
McGregor, Stanley Tacci. Audra McDonald, Gugu Mbatha-Raw
McKellen, Emma Thompson in supporting roles.
men and Personl likes Spider Man. the light the chat history and information about topics that the Personl and the Person2 are chatting on, predict what might have been suid following this dislogue by the Personl. Personl: DID yom know marvel successfullyy argues court that them in any way? Now try it yourself, Here is some data on the tupics the Personl and Person2 are discussing about: It is the eleventh installment in the X-Men film series, and direct sequel to the 2016 film Deadpool II the film. Deadpool forms the team X-Foree protect a young couaant from the time-travlling soldier Cable Deadpool the highest grossing R-mated film of all time, the highest grossing X-Men film, and the highest grossing 20th Century Fox film not directed by James Cameron or Gerege lacas It is the eleventh instal mment in the series, and direct sequel
hamane? Person2: distnt and did winning
argument helped
X
2016 Deadpoal IIn the film, Deadpool forms the leam X-Force protect a young mutani from the lime-traveling soldier Cable Deadpool is the highost gressing R-rated film of all time, the highest grossing X-Men film, and the highest groesing 20th Centary Fox film directed by James Cameron George Lucas. It the eleventh installment in the X-Men film series, and a direct sequel to the 2016 film Deadpool II the film. Deadpool form the team X- X-Force to protect a young musaat from the timo-traveling soldier Cable Deadpool the highest grossing R-rated film of all time the highest grossing X-Men film, and the highest grossing 20th Contury Fox film not directed by James Cameron or George Lacas. Here a conversation summary between Personl and Person2. Person2h Personl and Person2 love Disney. Personl loved the canooon beauty and the beast, the little mermaid, lion king. some of those icenic cartoons from when Personl was growing up. Person2 really likes little memmadd lion king, aladdin. frozen mouna. Personl didn't know the disney channel dresn't accopt outside walt disney was fired from his newquper joh for not heing more crcative. In the light of the chut history and information about topics that the Personl and the Person2
chutting on, predict what might huve been said following this dialogue by the Personl. Personl: IT has boen a good for me. bui I live an hour away, so its not huge trip. Person2: AH cool! YOU should go more often then lol did you see the most recent heauty and the heast?
Knowledge
Here is some information on the tupics being discussed by Personl and Person2; Goll is a club-and-han bull sport which players use various clibs to hit halls into a series of holes on course . average American golf cotase consames around 3 312,000 gallons of water per day ..aaee Ruth was once America's is most famous Sammel Jackson puts a golf clause his contracts that allows goll twiss a week during production. The University of lowa's locker room for visitng football teams is completely painted pink The foothall game occurred 1916 when Geargia Tech defeated Cumberland 222-0 Formerr Partices RB BenJarvus Green Ellis has never fumbled the football in his NFL career. The NFL is one of the four major professional sports leugues in North America .The NHI his no written rule against female players; wonen would fact be allowed they met the lesgac's eligibility requirements. .New Orkeans Saints chcerleader are forbidden from eating in the same restaurant as any NEL player Here is conversation summary between P and Person2: The averge golf course consumes around 3 312,000 gallons of water per day. There is golf course in Duhai that needs 4 million gallens water per day. top bowler made twice as much as uop football stars. The average engineer makes more his lifetime that the average NHL and mih player. The highest seoring foothall game ever was 222-0 In the light of the conversation history and information about the topics heing discussed by the Personl and the Person2. predict what might have been said following this dialogue by the Personl. Personl: THAT soems like the oddest rule. II's been fun challing with you! Person2: The average bowler makes swice as much as the top foothall stars. The avemge enginee neer makes more in his lifetime that the avenge NEI. and mih player.
Sammary Zoto
Shot
play
highest
Table 16: Perplexity Optimized Prompt Examples for Flan-T5-XL Template part shown in bold. The last output for the Person2 is the generated output using Flan-T5-XL, Part 2

---

Serting
Prompe
Summary Shot
following example for guidance. Example: Here conversation summarry between Persoal Persou2: Person is sales manager for premium mattress retailer. Person2 works in the tech industry working on software solutions. They talk nhour their hubbies and work. Based on the conversation between the Personl and the Person2, predict what the Person2 might have said response to the Personl's iss dialogue. Personl: am glad too. THE online compettiion was killinng us. THEN my boss docided to also seil online to we been very basy since then. Person2: IF you can't 1 heat them, join them. SOME people prefer the convenience of heing able to stay homee and buy. Now Ley yourself. Here conver cnnversation summary between Personl Person2; Person2 is sales munager premiuu mattress retailer. Personl works in the tech industry working on soflware solutions. They talk about their lubbies, week, and their current situation. Based the conversation between the Personl and the Person2, predict what Person2 might have said response to the Personl* I's dialogne. Personl: IF you can't it heat them, join them. SOME people prefer the convenience being able to stay and buy. Person2: IF you can't beat Ihem, join Uivm SOME people prefer the convenience of being able stay home and buy. Here is a summary of the conversation between Personl and Person2: Person2 and Personl talk about the reasons why Personl makes terrible choices. Person1 tries to love Person2 but it's hard. Person2 tells Person| Person2's mom loves Person] and Person2 is plaoning surpsise pany Pensont ''s mom who mental health issues. Based on the summury of conversation between Personl
Take
Summury Zero
Shot
Person2.
what do you think Person2 will say next? Personl: OH dkay, so have you done anything fan lately? Person2: I hope to over the summer.
I just got out school.
Only
Persona
Learn from the exumple first. Example: Here is some information about the Personl and Person2: Persun2 is in a bund and plays jazz music. Person2 is high school and plays jazz music froquently. Person2 thinks hass casiest instrument leam. Person] likes juzz, snowborrding, and cating steak. Personl wants to attend a juzz concert, and Personl wants to learn to play an instrument. Based on the information provided about the Personl and the Person2, predict whal the Person2 might have said respouse to the Personl's Iss dialogue. Personl: WOW that's seriously su awesome! I really coming logether! IF you need anyone to play some music on your opening day, let me know lol, Porson2: WOULD LD you he down play show on opening day? THAT would actually be awesome and would draw a huge crowd. NOTHING beats ice cream and a show! Now try it yourself. Here some information about the Personl and Person2: Personl is a band and plays jazz mussc Personl is in high school and plays jazz music frequenily Person] thinks hass the easiest instrament leam. Person2 likes ja27, snowboarding. and eating steak Person2 Person2 wants to learn to play un instrument Baved on the information provided about the Personl and the Person2, predict what the Person2 might have said response to the Personl's I's dialogue. Personl: WOULD you be down to play show opening day" ? THAT w would actually be gwesooe and I think would draw a huge crowd. NUTHING beats iee cream and a show" Person2: YEAH for sure! I'd love to do that, AS long as I get some steak haha I love that you re selling steak out of an ice cream truck. SLCH a unique idea!
New Shot
attend a jazz concert, and
Only Zero Shot
Persona
Here are some persona details about the Personi and Person2: Person] is landlord and needs help with Personl iss business. Person has good kids and good relationhhip with moon. Personl bates rumning. Person2 and Penson2 are talking about their plans for high school. Borkeles
on the given persona details about the Personl and the Person2. predict what Person2 would say after the Personl said this dialogne. Persont: FOR sure! WHAT tyne of music have you downloudedd WHO your favorite musician? Person2: It's good idea. I've downlouded
Person2 wants go Stanford and Ponson2 wants to go
of masic, I like the Beatles.
Persona + Sum-
Learn from the example first. Example: Here is some information about Personl and Person2: Porson? has been to Boston but grew in San Fruncisco. Person2 works a lot and fishs his spare time. Person2 married with children. Personl is English teacher Boston. Person] likes drawing and Fish 'n Chips, Personl and Person] like Batman and Fish 'n Chips Here is a conversation summary between Person and Person2: Person2 would love to draw San and volunteer for the homeless. Person| spends rime fishing and traveling. Person2 loves the dark knight movie and often dresses up cosplay. Personl's Iss wile makes Personl home- made french fries. Person2's favorite movic is the dark knight. Persont 'Il watch the next one Based on the persona information of the Person1 and the Person2 and their conversation summary so far, auticipate it all, but any parts with the joker are awesome. WHAT was your favorite part? Person2: SAME, I loved the jokerr! DID you watch the movie joker with jouquin phoenix? Now try yourself. Here are
mary Few Shot
Person2's respouse might to the Personl's next statement. Personl:
persona details about the Personl and Person2: Personl hus heen to Roston hut grew up in San Francisco. Person] works a lot and fishs his spare time. Personl is married with children. Porson2 an English teacher in Doston, Person2 likes drawing and Fish 'n Chips. Person2 and Person2 like Batman and Fish 'n Chips. Here is a conversation summary between Personl homeless and walks through parks. Personl loves the dark knight mavie and often dresses up for cosplay. Person2's i wife makes Person? home-made french fries. Person2 going to watch the next one in the series. Rased the persona information of the Person| and the Person2 and their conversation summary so far, anticipate what the Person2's 's response might be to the Personl's I next statement. Personl: SAME, loved the joker! DID you watch the movic joker with jonquin phoenix! Person2: did. IT was the darkest movic I've soon and very sad.
Person2. Person2 spends time fishing whon not basy
work Personl voluntcers for the
Persona e Sum-
Here are some persona details about the Personl and Person2: Parsonl I's cat just gave birth to a kitten. Penson lives in the city and doesn't i the country. Personl likes leu and Personl going to zoo. Penson2 a Senior in llggh School. Person2 moving wook. Porson2 likes cats and riding a mountain bike. Penson2 speaks Spanish. Here is a conver sation summary between Personl and color
mary Zero Shot
sty list. Person2 laves the 700 and listening to Spanish people. Person? will help Personl unpack at her now place. Based on the persona information of the Personl and the Person2 and their conversation summmary so far, anticipate what the Person2's 2's response might be to the Personl's I's next statement. Person1: YOU're right about that, and thanks so much for offering to help me unpack I'll definitely have some yummy tea ready! Person2: SWEET! LITERALLY. I think this move is going to be a great move for you. HAHH get it?
Person2: Person] moving into Sponish community next week. Person2 has just got a new haircut
Only
Knowl
Here is some on the topics Person and Person2 are discussing about: Thomas Cruise Mapother IV (born July 1962) is American actor and producer. He started his carcer at age 19 in the film Fndless Love (1981), before making his hreakthrough in the comedy Risky Business Allr starring
edge Zero Shot
Color of Money (1986) and Cocktail (1988). Cruise starred opposite Dustin Hoffinan the Academy Award for Best Pietare-winning drama Rain Man. For his role as anti-war activist Ron Kovic in the drama Bom on the Fourth of July (1989), Cruise received the Golden Globe The series is co prodaced by and agent of the Impossible Missions Force (IMF).The films follow the missions of the IMF's main field team under the lendership of Hunt Same characters, such as l Lather Stickell (played by Ving Rhames) and Benji Dann (played by Simon Poge) have recurring roles in the Wonder Woman 2017 American superbero film bused on DC Comics churacter of same name II is fourth installment in the DC Extended Univense (DCEU) is the second live action theatrical film featuring Wonder Woman following her debut in 2016's light
Tom Cruise. whose character Ethan Hunt, special
information about topics being discussed by Personl predict what might have been said following this dinlogue by the Person I. Persont: YEAH, did. THE movie isn't 't had either. WEL.I. nice chatling with you! Person2: LEONARD nimoy played role in that tv show. NICE chatting with you too!
Batman Superman: Dawn of Justice.
the Person2,
Table 17: Perplexity Optimized Prompt Examples for TO. Template part is shown in bold. The last output for the Person2 is the generated output using TO. Part 1

---

Setting Only
Frompt
Knowl
Learn from the example first. Example: Here is some information the toplcs being discussed by Personl and Person2: The NLL is one of the four major professional sports laggue in North America .The NEL has no written niee against female players; women would lact by allowed they met the league's eligibiiit requirements.... .New Orleans Suints cheerieakess
edge Few Shot
furbidden from coting the sume restaurant as any player The University lowa's locker room visiting foothall teams is complerely painted pink The highest soore ever in a foothall game nccurred in 1916 when Georgia Tech defeated Cumberland 222 0 o..Fommer Partiots RB BenJarvus Green Ellis has never fumbled range in length from 8.5 millionths of a metre to 33.6 metres (110 ftj In the light of the information about the toples being diseussed by the Personl and the Person2, prediet what might have been suid following this diulogue the Personl. Personl: HAAVE you
football his NHL career Animals are multicelluler organisms that
biological Kingdom. Animalia ..Animalss
seen the movie julie & julia with meryl streep? IT's is so great and makes me very bungry each time watch it. Person2: don't I think I've seen it but will add yourself. Here some data
my list ANOTHER good actress is emmma watson. I thoughh she was good in the harry potiie series Now try it topics Personl and Person2 are discussing about: Academy Awards given annually by the Acaulemy of Motion Picture Arts and Sciences The veremmnny was lirst broadcast on radio in 1930 and televised for the litst time in 1953 .. total of 5,072 Oscars been awarded from inception by the Academy of Motion Picture Arts and Sciences Thee ceremony was first broadcast on radio in 1930 and televised for the first time in 1953 .AA lotal of 3,072 Oscars have been awarded from the inception of the award through the 90th The Academy Awards
through the The Academy Awards given annually
given annually by the Acauleyy of Motion Picture Arts and Sciences TThe veremonn was first brondcast on rudso in 1930 and televised for the first time in 1953 ..A total of 3,072 Oscars have been swarded from the inception of the sward through the 90th. In the light of the information about topics being discussed by the Personl and Person2. predict what might have been said following this dialogue by the Personi. Personl: have to admit do like watching the best of the nlc and ale compete against each other! SPEAKING of animals, did you know there is lawyer in swizerland that represents them court? Person The Univerrity
lowa's locker for viciting frothall reams is completely painted pink . The highesi score ever a foothall gume accurred in 1916 when Georgia Tech defested Cumberland 222 O Former Partiots RB BeaJarvus Green-Eilli has never fumblld the football in his NHL carcer.
Knowledge
Take a look at the following esample for guidunce. Example: Here is some data on the topics the Personl and Person2 are discussing English poet. playwight and actor He is widely regarded as the greatest writer in the English language .His plays are performed more than those any other playwright Comedy a gonre film in which main emphasis humour. Cemouu..Demetri Demetri Martin was accepted into Harvard Law, but left out of beredom to pursus a career in comedy. Ryan Stiles dropped out of high school pursuc carezr comedy, Osear Fingal O' O'Fisheric Wills Wilde October 1854 2013 November 1900) an poet and p playwight..He is hest remembered for his epigrams and plays, his novel The Picture of Darian Gray, and the circumitances of his imprisomment and curly death. Here is conversation summary between Personl and Person2: Personl and Person2 football lans. Person2 likes the 49ers because Person2 used to live in San Francisco. Persun] is a hawks Jan. Person2 doesa't like brady or belichick. Personl docan't know what brady has on facebook. Person2 might not watch him on the facchook watch, Based on the chat history and the data topics by the Personl. Personl: SO was I bui when you slop and think, about iL il makes sense. MOST plays only lake seconds to complete. gameplay makes was sincerely nice chatting with you. Now try it yourself. Here is some data on the topics the Personl and Person2 are discussing about: The University of lowa's 's locker room for visiting football teams completely painted pink .Tee highest score ever in football game occurred in 1916 when Georgia Tech defeated Cumbetland 222-0 Former Partiots RB BenJarvus Green- Ellis has never lumbled the foothall in his NEI, carcer The University of lown's is locker room for visiring foothall teams is completely painted pink. .. The highest score ever a fontball game uccurred in 1916 when Georgia Tech defeated Cumberland 222 Former Furtiots RB BenJarvus Green Ellis has never fumbled the football in his NHL caroar The University of lowa's locker room for visiting football trams is completely painted pink foothall game accureed 1916 when Georgia Tech defented Cumberland 222.0 O ..omerr Partinis RB RenJarvus
Summary
Few
about: William Shakespeare
Shot
Person1 and the Person2 are chatting about, predict what might
been following
dialogue
THF the time ads and huddles. Person2: YOU right. guess
I've gor to
IT
.The highest Green-Elt Ellis has never fumblee the football in his NFL curver. Here is a conversation summary between Personl and Person2. Person2 fan of William Shokspeare. Person| a fan of Shakespeare' 's works. Porson2 and Person both like comedies. Person2 likes old school stuff like monty python. Personl likes black comedics, including black comedics. and bromances too. Based on chat history and the data on topics that the Persont and the Person2 are chatting about, predict what might have been suid following this dialogue by the Personl. Persoul: YEAH. jack black is natural comeeiaan never saw green lantern ANYWAY, great chat! Person2: been great talking with you too. I'm big fan of Monty Python and I think their homor still stands the test of time.
Knowledge
visiting foothall teams is completely painted pink The highest score ever in a football game occurred in 1916 when Georgia Tech defeated Cumberland 222-0 ..oomerr Partions RB BenJarvus Green- Ellis has
Here
data
toples the Persont and Person2
disenssing about: The University
lown's locker
Summary Zero
Shot
of the four mujor professsioaal spoets lougues in North America .The NEL has no wrillen rule against female players; w women would in fact allowed if they met the lasgne's eligibility requirements New Orlcans Saines cheerleaders are frebidden from cating in the same restaurant any NEI. player The Bible a collection sacred texts or scriptures that Jews Christians consider be a product of divine inspiration and a record of the relationship betwee God and humans. The Christian Old Testoment overlaps with the Hebrew Bible and the Greek Septungint; the Hehrew Rible is known Judaism the Tanakh. The New Testament is collection of writings by early Christians, believed to he mosily Jewish disciples of Chrisi, writlen in first century Koine Greek. Here a conversation summury between Personl and Person2: Porson2 and Personl are talking about sports. They talk about Brady's 's success, the highest scoring football game ever, the II minutes of live game play, and the tracking chips on the foothulls. In the light of the chat history and information about topies that the Personl and the Person2 are chatting on, predict what might have heen said following this dinlogue by the Person Persont: YES. I think we'l 'll see weman get in there al some point soon. THERE's no written rule against having Temale players in the nd. Person2: The NFL has no written rule against tomale players; women would in fact be allowed they met the ieague's i oligibility requirements. .Nw Orleans Soints cheerleaders forbidden from eating the same restaurant any NFL player.
fumbled the football his NFL curver The NFL
Table 18: Perplexity Optimized Prompt Examples for TO. Template part is shown in bold. The last output for the Person2 is the generated output using TO. Part 2

---

Seuing
Procupt
Summary
Few
Learn from the example first. Example: Here is a conversation summary between Personi and Person2: Porson2 prefers to watch movies and cook while Personl works al seafood restaurant. Person2's favorite artist Katy Penry and Personl's favorite
Shot
21 pilots. They plan to go rock climbing and watch movies lnterr In the light of the provided conversation summary between the Personl and the Person2, predict what might have heen said following this dialogue by the Person Personl: THAT one really good. I like mar. ?
Person2: OII that's a great one! I also love 21 pilots too. THEI songs are so caichy. HLAVE you ever seen them live? Now try it yoursell. Here is conversation summary between Person1 and Person2: Person2 got call from place where she
job application.
Person2 has an interview on Monday. Personl has been working the govermeent for years.
light of the provided conversation
summarry between the Personl and the Person2, predict what might have been said following this dialogue by the Personl. Personl:
WELL not sure enjoy lots, its not but enjoy is strong word. JUST think though your searches may alteady be done.. Person2:
HAHA yes, enjoy is probably a tare word to use about a job. I really hopel I get the role, It'll be nico to settle into the new city knowing! I
have money coming in
Sammary Zero
Here is a summary of the conversation hetween Personl and Person2: Personl tells Person2 about Persont i's favorite band, three dog night, und Personl's 's favorite song. Personl a huge fan of rock metal. Person has seen three dog night once bui would love to again. Rased on given summary of the conversation between the Persont and the Person2, predict what Person2 would say after the Person1 said this dialogue. Person1: really enjoy mmaa told me" one of their lesser known songs, but really like it. SOMETTMES when I get a chance. HOW's ' sae paula like, I want to visit this summer am
Shot
those are best songs. Person2: I'll give it
unsure of what to see or do in the city.
Only Persona
Learn from the example first. Example: Here some infocrattinn about the Personl and Persun2: Person2 Person2 used lo work for a govermment paper pasher. Penson2 laves books and has pet lizard. Porson2 worked as government paper pusher. Penson| works as assistant at doctor's office and tells Personl about the joh. Personl thinks it's great job. Based the information provided about the Personl and the Person2. predict what the Person2 might have said in response to the Personl's I's dialogue, Personl: HOW is it working in doctor's ' oflice? WORKING but all a good joh. Now try It yourself. Here is some information about Persoal likes horror movies. pizza and ramen, and Person| likes rock climbing, zombie movies, and zombie movies. P Persun2 and Person2 like horror movics, rock climbing, and zombic movies. Person2 broke up with Heather, and Person2
Few Shot
paper pustaer a bittl slow but like it so far. Person2: JUST same as other ollice setting Persont and Person2: Person1 listens Bruno Mars,
restaurant. Based on the information provided about the Personl and the Person2, prodict what the Person2 might have said in response to the Person! 's Lhrough my skylight the
dialogue. Personl: HLAVE you painted unything injeresting recenly? Person2: I have. I saw Lhis sun beam other day and it inspired me to paint a new picce. IT's like a sunburst.
Only Zero Shot
Persona
veterinarian but quit. Person I'' favorite travel destination is Maldives, Personl's I band is planning to create music next week. Person2 tells Person2 Person2 might quit PersonZ's job bocause Person2 doesn't 't have time to spend time with family and go to vacations. Based on the given persona details about the Person and the Person2. predict
Here some persona details about the Personl and Person2: Persunl
Person2 would suy after the Personl said dialugue. Personl: remember when nirvana get big! TILT was an exciting time for sure. HAVE you beon able to see any acts in concert lately? Person2: NO. have not recently, but I would love to go 500 some more live music again. WHAT about you?
Persona + Sum-
Take a look at the following example for guidance. Example: Here is some informattion about Personl and Person2: Porson2 tells Person2 Person2 has red hair and
mary Few Shot
quit Person2's job. Person2 likes old-school punk rock and cooking. Personl and Persont talk about their favorite food, music, and purents. Personl's Iss mot in commilled relationship. while Personl's I's parenis luught Personl to care for others. Here is a conversation summary between Person1 and Person2: Person2 wants support Person2's family. Personi picked up the groceries make enchiladas for Person2. Person2 loves Personl I's family and other food. In the light of the conversation summary and the information provided about the Personl and the Person2, predict what might luve been following home, think have some lemon and apole flavors left, and I can also use those if you want Person2: HAHA how shout all three" IT'll make for memorable meal. Now try it yourself. Here some information about the Personl and Person2: Person] tells Person I Personl has red hair und wants to quil Personl's is job. Penson likes old-school punk rock and cooking. Person2 and Person2 talk about their favorite food, music, and parents. Person2's not in a commined relarionship. while Person2's parents taught Parson2 to care for others. Here is conversation summury between Personl and Person2. Personl wants to quit job because Person] needs support Person family. Ponson2 picks up all the proeeties Person] noeds to make those enchiladas for Persen] Person2 has used 2'sson2's grandma's 's secret recipe in it and both Person2's mom approved of it. Person| loves Person2's family and will make strawherry jell-o for Person2, In the light of the conversation summary and the information provided about the Persent and the Person2, predict what might have heen suid following this dialogue by the Personl. Personl: HAIHA how about all ihree? IT make for memorable meal. Person2: SURE thing! I'll araagge giant cakes made from all these jellos, that II be such sight to behold!
Personi'ss job because Person2 needs
dialogue by the Personl. Persoul:
do you like the berry flavor? also have some other flaver
Persona Sum-
Here are some persona details about the Personl and Person2: Daria's name Daria and sbe works landscaper. Pensont Isss name
mary Zero Shot
Person Personl plays baskethall and likes shows about lawyers. John edits videos for a living and Person2 wants huild a keyboard. Persen2's 's popular in lown, and John's brother is popular too. Here is conversation summary between Personl and Person2: Persenl's 'ss landscaping business is really bus work on his landscaping at this time of yoar Person2 has started working on building that keybourd Personl would like to have a keyboard with talk to text feature built into it Based summary su far, anticipale what the Personl's is response might be to the Personl's nexl statement. Persoul: I can never lake vacation in the summer, but I'm looking forward some time off in the fall before the leaves fall. THE heach is calling my name. Person2: I can never take vacation in summer. I'm looking forward to some time off the before leaves
time of year. Everyone wants their beds cleuned and mulched. Person2 usally gets motivated
the persona information of the Personl and the Person2 and their conversation
Table 19: Perplexity Optimized Prompt Examples for Tk- Ik-snuttt Template part is shown in bold. The last output for the Person2 is the generated output using Tk-Instruct.
BLKURT
DEB
METROK
History Signal
a=0.5
a=1
am2
am5
a=10
a=10.5
a=1
Rm2
a=5
8=10
A=0,5
am1
Rm2
Rm5
a=10
BART-D
3.21
19.79
752.7
4.164407
3,388+15
4.92
46.56
4192.7
188-409
2.23-199
2.00
7.75
1180
4,774405
7,59E+11
PDgasus-CD
2.94
17.76
649.6
3,2264477
2,366+115
4.65
44,58
4107.0
3 3,315-00
2,496419
1.56
7.14
107.2
4,158405
5.57H+11
Pogasus-DS
3,20
19.72
749.8
4.144407
3.368+15
4.91
46.49
4194.1
3.26409
2
2.00
7.78
119.5
5,058405
8
Pegasus DS
2.01
12.23
453.6
2.340+07
1,72E+15
3.13
29.65
2673.8
2.01+099
1335+19
1.27
4.93
76.0
3.40E+05
6.83E+11
II
Recent-
3.82
22.23
753.0
2.9415077
1.352+15
5.92
33.35
4351.5
2411=09
9.445-18
2.42
8.96
125.0
3. 3.30E408
1911=11
Recent-2 2
3,16
19.38
727.7
337755077
15
4,50
46.49
4197.5
155
2.0755
2.02
7.92
122.6
5.57E+11
Recent-4
1.48
15.18
530.1
3.044.007
1.
3,86
36.77
3354.3
2.59
1.775419
1.60
6.34
100.7
4.37H448
6.42F.11
Recent-S
11.35
2.20E+07
1.65E+15
2.92
2563.8
2:03E009
144E=119
1:20
4.75
75.3
3.35E+05
5.48E+11
Recent- 10
1.68
10.05
359.4
1.65E+07
9.82E+14
2.65
25:08
2249.8
1667-009
1.1E+19
1.07
4,08
59.5
1990+05
1.4E+11
Semantic-1
23.51
4.71E207
3.59E+15
5.65
51.11
4208.7
2.46E+09
1.16E+19
2.38
9.03
131.3
4,20E+05
3.31+11
Semantic 2
3,25
19.86
743.8
3,93E+07
2.96E++15
4.84
44.19
3700.9
2 1224-099
1007:+19
2.04
7.88
118.2
4,28E+05
4,57E+11 11
Semantig-4
2,53
15.50
580.4
340TE407
2.358415
4.00
38.57
3602.3
2.998-00
29794119
1.64
6.52
105.0
4,588405
6885-11
Semantic-8
183
11:07
406.8
2,04E:477
1,466+15
2.78
25,58
2178.7
3815400
6.956+18
1.17
4,56
70.5
3.0566055
5.25E+11
Semantie-10 10
1.61
9.72
353.8
1.74E407
1.228+15
2.46
22.62
1922.5
226-09
6.144188
1.03
4.02
61.9
2.6784455
4.70E+11
Full
1.18
6.97
245.5
1.11E407
7.27E+14
1.89
18:03
1642.9
1.27-009
8.74E+18
0.76
2.91
44.3
1.926405
3.51E+11
Table 20: UID versus the Metric-Importance Index (for MSC Dataset)

---

BLEURT
METEOR
History Signal
n-1.5
n=
B=2
a=5
2=10
a-115
n-5
8-10
0-0
a-1
m=2
n=5
10
2905+09
3.19
18.59
632.5
2333407
1,258+15
5.17
48.79
43485
1.44409
1819419
1.95
6.96
39.2
9.00F-10
PS
3.91
48480
2.00400
1.404499
100. 8.64
2,328405
9.37P+10
DS
1,36
7.67
243.1
7.80F4446
2.60
2.24
30.75
1782.1
1115-000
5.045418
0.84
2.54
36.9
8.926404
6.40F-10
Recent
4.10
24.17
839.9
3.58E+07
1.95E+15
6.49
60.36
5233.5
3.446-09
1.77E+19
2.52
9.09
118.9
2.716-05
1.12E+11
Recent
3.45
20.35
707.3
3,01E+07
1.63++15
5.50
51.54
4533.9
3.11E-09
1.71=119
2.15
7.89
106.4
2.68EH05
1.345-1
Recent
2.71
13.99
558.8
2.411+07
1.32E+15
4.31
40.57
3598.5
2.330-09
1445=19
1.70
6.31
2.40E+05
1.420-11
Recent
2.02
11.92
415.3
1,770407
9.46E+
3.23
30.56
2731.8
1.97E=09
111E-19
1.27
4.73
65.7
1,84605
1.18-111
Rocent-10
1.83
10.77
372.2
1,55E+07
7.89E+14
2.95
27.94
2505.9
1.835009
1111=499
1.15
4.26
58.7
1,63+005
1.06+11
5194.7
1.395-09
Semantse-2
3.43
20.15
696.0
2,90E+07
1,49E+15
5.48
51.40
4528.7
3.12E-09
1.71E+19
2.11
7.64
100.4
2.34E+05
1.04+11
Semumisc-4
1.96E+07
3811.9
2.915+09
Semantic8
137
1157
198.0
1.63P447
8.06F11
3.39
30.20
2714.2
1.98--00
1.294-19
1:23
4.49
6406
9.45H+10
Semanris-
1.414407
6.648414
2474.3
82P-400
1.12H419
1.11
1.38H405
Pull
144
845
290.6
1,3084077
7.348+14
2.36
23.13
2218.7
1776-000
1.634419
0.91
3.44
49.5
1.566405
1,258+11
Table 21: UID the Metric-Impportance Index (for TC Dataset)


Reviews:
[{"review_id": "a2D3FMT2g2", "reviewer": "Reviewer_RHz2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer providing thoughtful and constructive feedback on our work. Your positive outlook about our work encourages us. We are pleased that you found our extensive experiments on prompt optimization impactful for understanding the trade-off between length and performance. Given the widespread use of large language models across domains, we are delighted that you consider the research question addressed in this paper to be relevant and meaningful.\n\n> **Q1. \"(573-574) It is important to clarify what efficiency means in this context.\"**\n\n\nThanks for identifying this lack of clarity in usage of the term “efficiency” in the draft.\nIn this work, by the term “efficiency” in L573-574 and all other occurrences, we refer to the temporal efficiency of the inference step of transformer-based language generation models. The decoding time during inference depends on the length of the input prompt and model scale. For a model of given scale, the more the length of the input, the more the compute requirements and hence the increased time and API cost. We will clarify this in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper describes extensive experiments on prompt optimization for the trade-off between length and performance. This is a relevant problem, considering that the costs increase with length, especially when using the paid APIs of many recent large language models.", "reasons_to_reject": "Considering the extent of the experiments, as revealed by the length of the supplementary material, it could be more appropriate to report them in a journal article.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses the problem of prompt optimization in the context of large language models. More specifically, it explores the trade-off between prompt length and performance when such models are used as dialog systems. The paper describes extensive experiments that compare multiple approaches to reduce prompt length while maintaining sufficient performance. The experiments mainly focus on different ways to compress the dialog history and background information, as well as on the comparison of zero-shot and few-shot scenarios. The results in terms of usable information density, a new metric defined by the authors, on two datasets using four different large language models suggest that the highest efficiency can be achieved using zero-shot scenarios and by reducing the dialog history to the most semantically similar utterance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "(573-574) It is important to clarify what efficiency means in this context."}, "extra_text": {}}, {"review_id": "wqSVXmJROc", "reviewer": "Reviewer_6RFn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful and constructive comments on our paper. Their feedback has helped us better contextualize and articulate the contributions of our work. We are grateful for your perspective on the practical usefulness of our work in providing guidance for efficient model design when using LLMs via API. We acknowledge your point on needing more clarity about takeaways and impacts of our findings in the NLP community, and will revise our draft to incorporate your suggestions.\n\n> **Q1: \"In section 5, they just state the experiment's results without much consideration of the reasons for such results. Therefore, the analysis lacks generality and may or may not be useful.\"**\n\nWe do have several important takeaways that are worth considering during practical dialog systems development and future research. Reviewers RHz2 and MCUv also point out the comprehensiveness of the experimental findings and lack of a similar study in literature looking at the important matter of optimizing inference costs while developing dialog systems. Overall, we find that using full dialog history, or Semantic-k/Recent-k with large k are not very useful from a UID perspective. Thus, our results have direct practical implications - lower UID configurations like Semantic-1 can be employed to build real-world dialog systems under budget constraints. (L558). Given the empirical nature of the work, we provide reasons wherever possible.\n\nWe present several generally applicable analyses for absolute performance and UID in Sections 5.2 and 5.3 resp.\nGeneral analysis from Section 5.2: (1) GPT3 is best; Tk-Instruct performs the worst -- across all metrics and datasets. (2) Semantic-k performs better than Recent-k -- across all metrics, datasets and models. (3) zero shot results are better than few shot results across metrics, models and datasets. This is such a non-intuitive finding. (4) Perplexity optimized prompts lead to shorter prompt sizes but not better accuracy values, across metrics, models and datasets.\n\nGeneral analysis from Section 5.3 for UID: (1) Surprisingly, manually engineered prompts are better on average (2) using one semantically related utterance is the best way of using dialog history. (3) for both Recent-k and Semantic-k, UID reduces with increase in k. (4) Using summaries of the dialog history provides better UID results than using the full dialog history.\n\nLastly, as evident from Fig. 5-8 and UIDs in Tab. 5-6 (Appendix), not all models are equally capable of interpreting dialog history in various summarized forms. In terms of UID (DEB) FLAN-T5 has difficulty in understanding Semantic-k and Recent-k summarized dialog histories, but works fine with Pegasus-DS. Whereas T0 is much more performant with all forms of summarized dialog histories in both datasets (L531-535). We apologize if the significance of our findings were not clear in Section 5, and gladly accept this suggestion for improving our paper further.\n\n> **Q2: \"Their argument needs to have verification of the usefulness of the UID. The UID can be interpreted as the average of the performance values normalized by the length of the prompt, but since the scale is different, using the values as they are, the length effect is likely to be significant.\"**\n\nWe appreciate the reviewer's concern regarding verification of the usefulness of the UID metric. To address this, we have conducted additional experiments analyzing the effect of using different exponents \"a\" in the following UID formulation\n\n$$UID(a)= \\frac{M^a}{L}$$ \n\nfor different values of $a$ (metric-importance index). We vary a as [0.5, 1, 2, 5, 10]. This updated formulation of the UID metric, with M raised to an exponent “a”, can be used to capture the importance assigned by the user on the model performance M (e.g. when inference cost is less of a bottleneck). We analyzed the accuracy-length tradeoff using different values for the parameter “a” to capture various types of user requirements in terms of the allowed expenses towards the inference process. Based on these experiments, we found some very insightful observations. The average UID values (for zeroshot manual prompts) across all the models are shown in Table 1 and 2 below (**for *better visualization* please check**: [https://anonymous.4open.science/api/repo/Frugal-Prompting-Evaluation/file/results/rebuttal_plots/UID_vs_a_table.png](https://anonymous.4open.science/api/repo/Frugal-Prompting-Evaluation/file/results/rebuttal_plots/UID_vs_a_table.png)). These tables show that for both MSC and TC, for DEB and METEOR, as the value of \"a\" is increased, summary-based dialog history variants tend to become better in terms of UID while recent-k and semantic-k variants tend to become less impressive. Although, in terms of BLEURT (UID), the ranking is in favour of Semantic-1 or 2 and Recent-1 or 2 throughout the complete range of \"a\" that we've explored. This might be because BLEURT measures normal sentence semantic similarity but not context-response relevance as measured by DEB. \n\n**Table 1: UID vs. the Metric-Importance Index $a$ (for MSC Dataset)**\n\n|  | History Signal | BLEURT |  |  |  |  | DEB |  |  |  |  | METEOR |  |  |  |  |\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n|  |  | a=0.5 | a=1 | a=2 | a=5 | a=10 | a=0.5 | a=1 | a=2 | a=5 | a=10 | a=0.5 | a=1 | a=2 | a=5 | a=10 |\n| **MSC** | BART-D | 3.21 | 19.79 | 752.7 | 4.16E+07 | 3.37E+15| 4.92 | 46.56 | 4192.7 | 3.17E+9| 2.22E+019 | 2.00 | 7.75 | 118.0 | 4.82E+05 | 7.58E+11|\n|  | Pegasus-CD | 2.94 | 17.76 | 649.6 | 3.22E+07 | 2.25E+15| 4.65 | 44.58 | 4107.0 | 3.31E+9| 2.48E+019 | 1.86 | 7.14 | 107.2 | 4.15E+05 | 5.57E+11|\n|  | Pegasus-DS | 3.20 | 19.72 | 749.8 | 4.14E+07 | 3.36E+15| 4.91 | 46.49 | 4194.1 | 3.20E+9| 2.27E+019 | 2.00 | 7.78 | 119.5 | 5.05E+05 | 8.57E+11|\n|  | Pegasus-DS+BI | 2.01 | 12.23 | 453.6 | 2.34E+07 | 1.71E+15| 3.13 | 29.65 | 2673.8 | 2.00E+9| 1.32E+019 | 1.27 | 4.93 | 76.0 | 3.40E+05 | 6.83E+11|\n|  | Recent-1 | 3.82 | 22.22 | 753.0 | 2.94E+07 | 1.34E+15| 5.92 | 53.35 | 4351.5 | 2.40E+9| 9.44E+018 | 2.42 | 8.96 | 123.0 | 3.30E+05 | 1.91E+11|\n|  | Recent-2 | 3.16 | 19.38 | 727.7 | 3.87E+07 | 2.97E+15| 4.90 | 46.49 | 4197.5 | 3.15E+9| 2.06E+019 | 2.02 | 7.92 | 122.6 | 4.81E+05 | 5.56E+11|\n|  | Recent-4 | 2.48 | 15.18 | 570.1 | 3.04E+07 | 2.35E+15| 3.86 | 36.77 | 3353.3 | 2.59E+9| 1.76E+019 | 1.60 | 6.34 | 100.7 | 4.37E+05 | 6.41E+11|\n|  | Recent-8 | 1.86 | 11.35 | 422.7 | 2.20E+07 | 1.65E+15| 2.92 | 27.91 | 2563.8 | 2.02E+9| 1.44E+019 | 1.20 | 4.75 | 75.3 | 3.35E+05 | 5.47E+11|\n|  | Recent-10 | 1.68 | 10.05 | 359.4 | 1.65E+07 | 9.81E+14| 2.65 | 25.08 | 2249.8 | 1.66E+9| 1.10E+019 | 1.07 | 4.08 | 59.5 | 1.90E+05 | 1.43E+11|\n|  | Semantic-1 | 3.83 | 23.51 | 883.8 | 4.71E+07 | 3.59E+15| 5.65 | 51.11 | 4208.7 | 2.46E+9| 1.15E+019 | 2.38 | 9.03 | 131.3 | 4.20E+05 | 3.29E+11|\n|  | Semantic-2 | 3.25 | 19.86 | 743.8 | 3.93E+07 | 2.96E+15| 4.84 | 44.19 | 3700.9 | 2.24E+9| 1.06E+019 | 2.04 | 7.88 | 118.2 | 4.28E+05 | 4.57E+11|\n|  | Semantic-4 | 2.53 | 15.50 | 580.4 | 3.07E+07 | 2.34E+15| 4.00 | 38.57 | 3602.3 | 2.98E+9| 2.28E+019 | 1.64 | 6.52 | 104.0 | 4.58E+05 | 6.84E+11|\n|  | Semantic-8 | 1.83 | 11.07 | 406.8 | 2.04E+07 | 1.46E+15| 2.78 | 25.58 | 2178.7 | 1.38E+9| 6.94E+018 | 1.17 | 4.56 | 70.5 | 3.03E+05 | 5.24E+11|\n|  | Semantic-10 | 1.61 | 9.72 | 353.8 | 1.74E+07 | 1.21E+15| 2.46 | 22.62 | 1922.5 | 1.21E+9| 6.16E+018 | 1.03 | 4.02 | 61.9 | 2.67E+05 | 4.76E+11|\n|  | Full | 1.18 | 6.97 | 245.5 | 1.11E+07 | 7.27E+14| 1.89 | 18.03 | 1642.9 | 1.27E+9| 8.73E+018 | 0.76 | 2.91 | 44.3 | 1.92E+05 | 3.50E+11|\n\n**Table 2: UID vs. the Metric-Importance Index $a$ (for TC Dataset)**\n\n|  | History Signal | BLEURT |  |  |  |  | DEB |  |  |  |  | METEOR |  |  |  |  |\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n|  |  | a=0.5 | a=1 | a=2 | a=5 | a=10 | a=0.5 | a=1 | a=2 | a=5 | a=10 | a=0.5 | a=1 | a=2 | a=5 | a=10 |\n| **TC** | BART-D | 4.12 | 24.56 | 873.8 | 3.98E+07 | 2.41E+15| 6.32 | 57.75 | 4848.1 | 2.96E+9| 1.44E+019 | 2.45 | 8.67 | 109.6 | 2.31E+05 | 9.19E+10|\n|  | Pegasus-CD | 3.19 | 18.59 | 632.3 | 2.53E+07 | 1.25E+15| 5.17 | 48.79 | 4348.5 | 3.10E+9| 1.80E+019 | 1.95 | 6.96 | 89.2 | 2.00E+05 | 9.06E+10|\n|  | Pegasus-DS | 4.09 | 24.37 | 864.9 | 3.91E+07 | 2.32E+15| 6.29 | 57.61 | 4848.0 | 2.97E+9| 1.45E+019 | 2.44 | 8.64 | 109.3 | 2.32E+05 | 9.37E+10|\n|  | Pegasus-DS+BI | 1.36 | 7.67 | 243.1 | 7.80E+06 | 2.60E+14| 2.24 | 20.75 | 1782.1 | 1.15E+9| 5.94E+018 | 0.84 | 2.94 | 36.9 | 8.92E+04 | 6.46E+10|\n|  | Recent-1 | 4.10 | 24.17 | 839.9 | 3.58E+07 | 1.95E+15| 6.49 | 60.36 | 5233.5 | 3.44E+9| 1.77E+019 | 2.52 | 9.09 | 118.9 | 2.71E+05 | 1.12E+11|\n|  | Recent-2 | 3.45 | 20.35 | 707.3 | 3.01E+07 | 1.63E+15| 5.50 | 51.54 | 4533.9 | 3.11E+9| 1.70E+019 | 2.15 | 7.89 | 106.4 | 2.68E+05 | 1.34E+11|\n|  | Recent-4 | 2.71 | 15.99 | 558.8 | 2.41E+07 | 1.31E+15| 4.31 | 40.57 | 3598.5 | 2.53E+9| 1.45E+019 | 1.70 | 6.31 | 87.4 | 2.40E+05 | 1.41E+11|\n|  | Recent-8 | 2.02 | 11.92 | 415.3 | 1.77E+07 | 9.45E+14| 3.23 | 30.56 | 2731.8 | 1.97E+9| 1.17E+019 | 1.27 | 4.73 | 65.7 | 1.84E+05 | 1.18E+11|\n|  | Recent-10 | 1.83 | 10.77 | 372.2 | 1.55E+07 | 7.88E+14| 2.95 | 27.94 | 2505.9 | 1.82E+9| 1.10E+019 | 1.15 | 4.26 | 58.7 | 1.63E+05 | 1.06E+11|\n|  | Semantic-1 | 4.09 | 24.02 | 831.3 | 3.48E+07 | 1.81E+15| 6.46 | 60.06 | 5194.7 | 3.39E+9| 1.71E+019 | 2.47 | 8.76 | 110.6 | 2.26E+05 | 7.79E+10|\n|  | Semantic-2 | 3.43 | 20.15 | 696.0 | 2.90E+07 | 1.48E+15| 5.48 | 51.40 | 4528.7 | 3.11E+9| 1.71E+019 | 2.11 | 7.64 | 100.4 | 2.34E+05 | 1.03E+11|\n|  | Semantic-4 | 2.65 | 15.34 | 515.2 | 1.96E+07 | 8.60E+14| 4.37 | 41.73 | 3811.9 | 2.90E+9| 1.86E+019 | 1.71 | 6.41 | 90.2 | 2.52E+05 | 1.43E+11|\n|  | Semantic-8 | 1.97 | 11.57 | 398.0 | 1.63E+07 | 8.05E+14| 3.19 | 30.20 | 2714.2 | 1.98E+9| 1.19E+019 | 1.23 | 4.49 | 60.6 | 1.58E+05 | 9.44E+10|\n|  | Semantic-10 | 1.78 | 10.38 | 353.6 | 1.41E+07 | 6.64E+14| 2.89 | 27.45 | 2474.3 | 1.82E+9| 1.12E+019 | 1.11 | 4.03 | 53.9 | 1.38E+05 | 8.35E+10|\n|  | Full | 1.43 | 8.45 | 296.6 | 1.30E+07 | 7.33E+14| 2.36 | 23.13 | 2218.7 | 1.96E+9| 1.62E+019 | 0.91 | 3.44 | 49.5 | 1.56E+05 | 1.25E+11|\n\n\nTo fully understand how the rank of various history signals vary over the value of the metric-importance “a”, we plot the rank-order of all history signal types vs. the value of \"a\" (increased from 0.5 to 10) as show in the figure [here (https://anonymous.4open.science/api/repo/Frugal-Prompting-Evaluation/file/results/rebuttal_plots/bumpplot_MSC_DEB.png)](https://anonymous.4open.science/api/repo/Frugal-Prompting-Evaluation/file/results/rebuttal_plots/bumpplot_MSC_DEB.png). We have reported the rank-order of the history signal types in **Table 3*** as well. This rank order dynamics helps us clearly understand, as we give more and more importance to the model performance and ignore the cost of inference, how the choices over the history signal change. For example, in terms of the UID (DEB) metric on the MSC dataset, the average trend across models is that Recent-1 and Semantic-1 are the recommended ways to summarize the context information if cost is an important factor to the user. Whereas, if cost is of less importance, then longer dialog summaries such as Pegasus-CD and Semantic-4 are recommended approaches for MSC. Additionally, we observe that some models like Recent-10, Semantic-8, Semantic-10 and Full are always bad choices, while some (BART-D, Pegasus-DS) are quite robust across the whole range of values of \"a\".\n\n(*P.S. - *We had to share these results in tabular format as there is no provision for adding images (plots/graphs) in Openreview.*)\n\n**Table 3: Trend in Ranks of History Signal Types for Different Values of the Metric-Importance Index $a$ (for MSC dataset, DEB metric)**\n\n| MSC | DEB |  |  |  |  |\n|---|---|---|---|---|---|\n|  | a=0.5 | a=1 | a=2 | a=5 | a=10 |\n| BART-D | 3 | 3 | 5 | 3 | 4 |\n| Pegasus-CD | 7 | 6 | 6 | 1 | 1 |\n| Pegasus-DS | 4 | 4 | 4 | 2 | 3 |\n| Pegasus-DS+BI | 10 | 10 | 10 | 11 | 8 |\n| Recent-1 | 1 | 1 | 1 | 8 | 12 |\n| Recent-2 | 5 | 5 | 3 | 4 | 5 |\n| Recent-4 | 9 | 9 | 9 | 6 | 6 |\n| Recent-8 | 11 | 11 | 11 | 10 | 7 |\n| Recent-10 | 13 | 13 | 12 | 12 | 10 |\n| Semantic-1 | 2 | 2 | 2 | 7 | 9 |\n| Semantic-2 | 6 | 7 | 7 | 9 | 11 |\n| Semantic-4 | 8 | 8 | 8 | 5 | 2 |\n| Semantic-8 | 12 | 12 | 13 | 13 | 14 |\n| Semantic-10 | 14 | 14 | 14 | 15 | 15 |\n| Full | 15 | 15 | 15 | 14 | 13 |\n\nWe will add this in-depth analysis of the effect from the chosen value of “a” in the updated draft. Once again, we sincerely appreciate your time and effort to provide constructive feedback on our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- They focus on prompt length from the perspective of practical use, not only from that of performance, and propose a metric that considers it. When using LLM via API, it is necessary to consider the cost. Since the cost of many APIs is determined by the number of tokens, this metric is useful in practical use.\n- They investigate what prompts are good from the efficiency perspective and provide valuable guidelines for model design.", "reasons_to_reject": "- In section 5, they just state the experiment's results without much consideration of the reasons for such results. Therefore, the analysis lacks generality and may or may not be useful.\n- Their argument needs to have verification of the usefulness of the UID. The UID can be interpreted as the average of the performance values normalized by the length of the prompt, but since the scale is different, using the values as they are, the length effect is likely to be significant.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Given the recent prominence of LLM, the authors investigate the impact of prompts in dialogue tasks using two dialogue datasets and four models in this paper.\nThey compare manually engineered prompts and perplexity-based prompts in terms of what kind of prompts should be better. It also examines how dialogue history can be included in the prompts.\nAssuming that LLM is used via an API, the authors propose the concept of a frugal prompt because the length of a prompt affects its cost, and they propose a measure called UID to measure it.\nFrom experimental results, the authors show that it is not effective to include many contexts, and that using the most semantically relevant utterances is useful in terms of UID.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "h0XcrVQELz", "reviewer": "Reviewer_MCUv", "summary": "", "questions": "1) Can the UID metric be redefined with a scale parameter M^a / L? To quantify the value of the tradeoff between quality and inference cost.\n\n2) PEGASUS-DI + BI has a drastically worse UID vs. PEGASUS-DI in Table 4, on both MSC and TC. Is there a reason why? From what I can tell in Section 4, I don’t think the length of the BI is very long.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful feedback on our paper. We appreciate you highlighting the comprehensive nature of our study and raising important questions about our proposed UID metric. We are grateful that you took the time to provide such insightful and constructive critiques. We will revise our draft to incorporate the valuable suggestions provided by you.\n\n> **Q1: \"However, there is an implicit assumption that the desired tradeoff between M and L is linear. … In this case, a better metric would be M^2 / L, or in general M^a / L for some a. … This paper would benefit from a deeper analysis on how to quantify the tradeoff between M and L, e.g. for a = 0.5, 1, 2, 5, 10…. At high enough a, I would assume that full prompting on GPT-3 would have the best UID.\"**\n\n> **\"Can the UID metric be redefined with a scale parameter M^a / L? To quantify the value of the tradeoff between quality and inference cost.\"**\n\nThanks for this very interesting suggestion. We agree with the reviewer that our definition of the UID metric assumes a linear trade-off relation between metric and length. An updated formulation of the UID metric, with M raised to an exponent “a”, can be used to capture the importance assigned by the user on the model performance M (e.g. when inference cost is less of a bottleneck). \n\n$UID(a) = \\frac{M^a}{L}$\n\nAs suggested, we analyzed the accuracy-length trade-off using different values for the parameter “a” to capture various types of user requirements in terms of allowed expenses for the inference process. These UID values are reported below in Table 1 and Table 2. Based on these experiments, we found some very insightful observations. The average UID values (for zero-shot manual prompts) across all the models are shown in Table 1 and 2 below (**for *better visualization* please check**: [https://anonymous.4open.science/api/repo/Frugal-Prompting-Evaluation/file/results/rebuttal_plots/UID_vs_a_table.png](https://anonymous.4open.science/api/repo/Frugal-Prompting-Evaluation/file/results/rebuttal_plots/UID_vs_a_table.png)). These tables show that for both MSC and TC, for DEB and METEOR, as the value of \"a\" is increased, summary-based dialog history variants tend to become better in terms of UID while recent-k and semantic-k variants tend to become less impressive. Although, in terms of BLEURT (UID), the ranking is in favour of Semantic-1 or 2 and Recent-1 or 2 throughout the complete range of \"a\" that we've explored. This might be because BLEURT measures normal sentence semantic similarity but not context-response relevance as measured by DEB. \n\n**Table 1: UID vs. the Metric-Importance Index $a$ (for MSC Dataset)**\n\n|  | History Signal | BLEURT |  |  |  |  | DEB |  |  |  |  | METEOR |  |  |  |  |\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n|  |  | a=0.5 | a=1 | a=2 | a=5 | a=10 | a=0.5 | a=1 | a=2 | a=5 | a=10 | a=0.5 | a=1 | a=2 | a=5 | a=10 |\n| **MSC** | BART-D | 3.21 | 19.79 | 752.7 | 4.16E+07 | 3.37E+15| 4.92 | 46.56 | 4192.7 | 3.17E+9| 2.22E+019 | 2.00 | 7.75 | 118.0 | 4.82E+05 | 7.58E+11|\n|  | Pegasus-CD | 2.94 | 17.76 | 649.6 | 3.22E+07 | 2.25E+15| 4.65 | 44.58 | 4107.0 | 3.31E+9| 2.48E+019 | 1.86 | 7.14 | 107.2 | 4.15E+05 | 5.57E+11|\n|  | Pegasus-DS | 3.20 | 19.72 | 749.8 | 4.14E+07 | 3.36E+15| 4.91 | 46.49 | 4194.1 | 3.20E+9| 2.27E+019 | 2.00 | 7.78 | 119.5 | 5.05E+05 | 8.57E+11|\n|  | Pegasus-DS+BI | 2.01 | 12.23 | 453.6 | 2.34E+07 | 1.71E+15| 3.13 | 29.65 | 2673.8 | 2.00E+9| 1.32E+019 | 1.27 | 4.93 | 76.0 | 3.40E+05 | 6.83E+11|\n|  | Recent-1 | 3.82 | 22.22 | 753.0 | 2.94E+07 | 1.34E+15| 5.92 | 53.35 | 4351.5 | 2.40E+9| 9.44E+018 | 2.42 | 8.96 | 123.0 | 3.30E+05 | 1.91E+11|\n|  | Recent-2 | 3.16 | 19.38 | 727.7 | 3.87E+07 | 2.97E+15| 4.90 | 46.49 | 4197.5 | 3.15E+9| 2.06E+019 | 2.02 | 7.92 | 122.6 | 4.81E+05 | 5.56E+11|\n|  | Recent-4 | 2.48 | 15.18 | 570.1 | 3.04E+07 | 2.35E+15| 3.86 | 36.77 | 3353.3 | 2.59E+9| 1.76E+019 | 1.60 | 6.34 | 100.7 | 4.37E+05 | 6.41E+11|\n|  | Recent-8 | 1.86 | 11.35 | 422.7 | 2.20E+07 | 1.65E+15| 2.92 | 27.91 | 2563.8 | 2.02E+9| 1.44E+019 | 1.20 | 4.75 | 75.3 | 3.35E+05 | 5.47E+11|\n|  | Recent-10 | 1.68 | 10.05 | 359.4 | 1.65E+07 | 9.81E+14| 2.65 | 25.08 | 2249.8 | 1.66E+9| 1.10E+019 | 1.07 | 4.08 | 59.5 | 1.90E+05 | 1.43E+11|\n|  | Semantic-1 | 3.83 | 23.51 | 883.8 | 4.71E+07 | 3.59E+15| 5.65 | 51.11 | 4208.7 | 2.46E+9| 1.15E+019 | 2.38 | 9.03 | 131.3 | 4.20E+05 | 3.29E+11|\n|  | Semantic-2 | 3.25 | 19.86 | 743.8 | 3.93E+07 | 2.96E+15| 4.84 | 44.19 | 3700.9 | 2.24E+9| 1.06E+019 | 2.04 | 7.88 | 118.2 | 4.28E+05 | 4.57E+11|\n|  | Semantic-4 | 2.53 | 15.50 | 580.4 | 3.07E+07 | 2.34E+15| 4.00 | 38.57 | 3602.3 | 2.98E+9| 2.28E+019 | 1.64 | 6.52 | 104.0 | 4.58E+05 | 6.84E+11|\n|  | Semantic-8 | 1.83 | 11.07 | 406.8 | 2.04E+07 | 1.46E+15| 2.78 | 25.58 | 2178.7 | 1.38E+9| 6.94E+018 | 1.17 | 4.56 | 70.5 | 3.03E+05 | 5.24E+11|\n|  | Semantic-10 | 1.61 | 9.72 | 353.8 | 1.74E+07 | 1.21E+15| 2.46 | 22.62 | 1922.5 | 1.21E+9| 6.16E+018 | 1.03 | 4.02 | 61.9 | 2.67E+05 | 4.76E+11|\n|  | Full | 1.18 | 6.97 | 245.5 | 1.11E+07 | 7.27E+14| 1.89 | 18.03 | 1642.9 | 1.27E+9| 8.73E+018 | 0.76 | 2.91 | 44.3 | 1.92E+05 | 3.50E+11|\n\n**Table 2: UID vs. the Metric-Importance Index $a$ (for TC Dataset)**\n\n|  | History Signal | BLEURT |  |  |  |  | DEB |  |  |  |  | METEOR |  |  |  |  |\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n|  |  | a=0.5 | a=1 | a=2 | a=5 | a=10 | a=0.5 | a=1 | a=2 | a=5 | a=10 | a=0.5 | a=1 | a=2 | a=5 | a=10 |\n| **TC** | BART-D | 4.12 | 24.56 | 873.8 | 3.98E+07 | 2.41E+15| 6.32 | 57.75 | 4848.1 | 2.96E+9| 1.44E+019 | 2.45 | 8.67 | 109.6 | 2.31E+05 | 9.19E+10|\n|  | Pegasus-CD | 3.19 | 18.59 | 632.3 | 2.53E+07 | 1.25E+15| 5.17 | 48.79 | 4348.5 | 3.10E+9| 1.80E+019 | 1.95 | 6.96 | 89.2 | 2.00E+05 | 9.06E+10|\n|  | Pegasus-DS | 4.09 | 24.37 | 864.9 | 3.91E+07 | 2.32E+15| 6.29 | 57.61 | 4848.0 | 2.97E+9| 1.45E+019 | 2.44 | 8.64 | 109.3 | 2.32E+05 | 9.37E+10|\n|  | Pegasus-DS+BI | 1.36 | 7.67 | 243.1 | 7.80E+06 | 2.60E+14| 2.24 | 20.75 | 1782.1 | 1.15E+9| 5.94E+018 | 0.84 | 2.94 | 36.9 | 8.92E+04 | 6.46E+10|\n|  | Recent-1 | 4.10 | 24.17 | 839.9 | 3.58E+07 | 1.95E+15| 6.49 | 60.36 | 5233.5 | 3.44E+9| 1.77E+019 | 2.52 | 9.09 | 118.9 | 2.71E+05 | 1.12E+11|\n|  | Recent-2 | 3.45 | 20.35 | 707.3 | 3.01E+07 | 1.63E+15| 5.50 | 51.54 | 4533.9 | 3.11E+9| 1.70E+019 | 2.15 | 7.89 | 106.4 | 2.68E+05 | 1.34E+11|\n|  | Recent-4 | 2.71 | 15.99 | 558.8 | 2.41E+07 | 1.31E+15| 4.31 | 40.57 | 3598.5 | 2.53E+9| 1.45E+019 | 1.70 | 6.31 | 87.4 | 2.40E+05 | 1.41E+11|\n|  | Recent-8 | 2.02 | 11.92 | 415.3 | 1.77E+07 | 9.45E+14| 3.23 | 30.56 | 2731.8 | 1.97E+9| 1.17E+019 | 1.27 | 4.73 | 65.7 | 1.84E+05 | 1.18E+11|\n|  | Recent-10 | 1.83 | 10.77 | 372.2 | 1.55E+07 | 7.88E+14| 2.95 | 27.94 | 2505.9 | 1.82E+9| 1.10E+019 | 1.15 | 4.26 | 58.7 | 1.63E+05 | 1.06E+11|\n|  | Semantic-1 | 4.09 | 24.02 | 831.3 | 3.48E+07 | 1.81E+15| 6.46 | 60.06 | 5194.7 | 3.39E+9| 1.71E+019 | 2.47 | 8.76 | 110.6 | 2.26E+05 | 7.79E+10|\n|  | Semantic-2 | 3.43 | 20.15 | 696.0 | 2.90E+07 | 1.48E+15| 5.48 | 51.40 | 4528.7 | 3.11E+9| 1.71E+019 | 2.11 | 7.64 | 100.4 | 2.34E+05 | 1.03E+11|\n|  | Semantic-4 | 2.65 | 15.34 | 515.2 | 1.96E+07 | 8.60E+14| 4.37 | 41.73 | 3811.9 | 2.90E+9| 1.86E+019 | 1.71 | 6.41 | 90.2 | 2.52E+05 | 1.43E+11|\n|  | Semantic-8 | 1.97 | 11.57 | 398.0 | 1.63E+07 | 8.05E+14| 3.19 | 30.20 | 2714.2 | 1.98E+9| 1.19E+019 | 1.23 | 4.49 | 60.6 | 1.58E+05 | 9.44E+10|\n|  | Semantic-10 | 1.78 | 10.38 | 353.6 | 1.41E+07 | 6.64E+14| 2.89 | 27.45 | 2474.3 | 1.82E+9| 1.12E+019 | 1.11 | 4.03 | 53.9 | 1.38E+05 | 8.35E+10|\n|  | Full | 1.43 | 8.45 | 296.6 | 1.30E+07 | 7.33E+14| 2.36 | 23.13 | 2218.7 | 1.96E+9| 1.62E+019 | 0.91 | 3.44 | 49.5 | 1.56E+05 | 1.25E+11|\n\nTo fully understand how the rank of various history signals vary over the value of the metric-importance “a”, we plot the rank-order of all history signal types vs. the value of \"a\" (increased from 0.5 to 10) as show in the figure [here (https://anonymous.4open.science/api/repo/Frugal-Prompting-Evaluation/file/results/rebuttal_plots/bumpplot_MSC_DEB.png)](https://anonymous.4open.science/api/repo/Frugal-Prompting-Evaluation/file/results/rebuttal_plots/bumpplot_MSC_DEB.png). We have reported the rank-order of the history signal types in **Table 3*** as well. This rank order dynamics helps us clearly understand, as we give more and more importance to the model performance and ignore the cost of inference, how the choices over the history signal change. For example, in terms of the UID (DEB) metric on the MSC dataset, the average trend across models is that Recent-1 and Semantic-1 are the recommended ways to summarize the context information if cost is an important factor to the user. Whereas, if cost is of less importance, then longer dialog summaries such as Pegasus-CD and Semantic-4 are recommended approaches for MSC. Additionally, we observe that some models like Recent-10, Semantic-8, Semantic-10 and Full are always bad choices, while some (BART-D, Pegasus-DS) are quite robust across the whole range of values of \"a\".\n\n(*P.S. - *We had to share these results in tabular format as there is no provision for adding images (plots/graphs) in Openreview.*)\n\n**Table 3: Trend in Ranks of History Signal Types for Different Values of the Metric-Importance Index $a$ (for MSC dataset, DEB metric)**\n\n| MSC | DEB |  |  |  |  |\n|---|---|---|---|---|---|\n|  | a=0.5 | a=1 | a=2 | a=5 | a=10 |\n| BART-D | 3 | 3 | 5 | 3 | 4 |\n| Pegasus-CD | 7 | 6 | 6 | 1 | 1 |\n| Pegasus-DS | 4 | 4 | 4 | 2 | 3 |\n| Pegasus-DS+BI | 10 | 10 | 10 | 11 | 8 |\n| Recent-1 | 1 | 1 | 1 | 8 | 12 |\n| Recent-2 | 5 | 5 | 3 | 4 | 5 |\n| Recent-4 | 9 | 9 | 9 | 6 | 6 |\n| Recent-8 | 11 | 11 | 11 | 10 | 7 |\n| Recent-10 | 13 | 13 | 12 | 12 | 10 |\n| Semantic-1 | 2 | 2 | 2 | 7 | 9 |\n| Semantic-2 | 6 | 7 | 7 | 9 | 11 |\n| Semantic-4 | 8 | 8 | 8 | 5 | 2 |\n| Semantic-8 | 12 | 12 | 13 | 13 | 14 |\n| Semantic-10 | 14 | 14 | 14 | 15 | 15 |\n| Full | 15 | 15 | 15 | 14 | 13 |\n\nWe will add this in-depth analysis of the effect from the chosen value of “a” in the updated draft.\n\n> **Q2: \"PEGASUS-DI + BI has a drastically worse UID vs. PEGASUS-DI in Table 4, on both MSC and TC. Is there a reason why? From what I can tell in Section 4, I don’t think the length of the BI is very long.\"**\n\nActually, the increase in the length of the prompt due to BI is quite significant for both datasets. The difference in length can be observed in Fig 1. and Fig 4. for few shot and zero shot settings, respectively. The increase in prompt length would be more prominent for the few shot scenario as examples of BI have to be added to the prompt also. For example, BI adds 200 and 500 tokens in average to the prompt for MSC and TC datasets, respectively.\nNow, since the performance of PegasusDS+BI is generally comparable to PegasusDS (Fig. 2 and Fig. 3), the increase in prompt input length (between 1.5-2x) causes a significant drop in the UID values.\n\n> **Q3: \"The conclusion presented by the authors ... I’m not sure what readers can take away from this when doing future dialogue research. It’s very intuitive that conversations can have a lot of redundant information between utterances that can be condensed in order to save on sequence length (e.g. in both DialogSum and SAMSum).\"**\n\nWe do have several important takeaways that are worth considering during practical dialog systems development and future research. Overall, we find that using full dialog history, or Semantic-k/Recent-k with large k are not very useful from a UID perspective. Thus, our results have direct practical implications - lower UID configurations like Semantic-1 can be employed to build real-world dialog systems under budget constraints. (L558). Given the empirical nature of the work, we provide reasons wherever possible.\n\nWe present several generally applicable analyses for absolute performance and UID in Sections 5.2 and 5.3 resp.\nGeneral analysis from Section 5.2: (1) GPT3 is best; Tk-Instruct performs the worst -- across all metrics and datasets. (2) Semantic-k performs better than Recent-k -- across all metrics, datasets and models. (3) zero shot results are better than few shot results across metrics, models and datasets. This is such a non-intuitive finding. (4) Perplexity optimized prompts lead to shorter prompt sizes but not better accuracy values, across metrics, models and datasets.\n\nGeneral analysis from Section 5.3 for UID: (1) Surprisingly, manually engineered prompts are better on average (2) using one semantically related utterance is the best way of using dialog history. (3) for both Recent-k and Semantic-k, UID reduces with increase in k. (4) Using summaries of the dialog history provides better UID results than using the full dialog history.\n\nLastly, as evident from Fig. 5-8 and UIDs in Tab. 5-6 (Appendix), not all models are equally capable of interpreting dialog history in various summarized forms. In terms of UID (DEB) FLAN-T5 has difficulty in understanding Semantic-k and Recent-k summarized dialog histories, but works fine with Pegasus-DS. Whereas T0 is much more performant with all forms of summarized dialog histories in both datasets (L531-535). We apologize if the significance of our findings were not clear in Section 5, and gladly accept this suggestion for improving our paper further. \n\nOnce again, we sincerely appreciate your time and effort to provide constructive feedback on our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a comprehensive study on how to prompt LLMs for open-domain dialogue tasks with a focus on frugality. There are a wide variety of parameters that are studied: 0-shot vs 1-shot, manual vs perplexity-chosen prompts, dialog history (full, summarized by BART or PEGASUS, recent-k, semantic-k), and also whether to add summarized background information (BI). To the best of my knowledge, this is the only study to look at prompting methods for open-domain dialogue.", "reasons_to_reject": "1) I am not sure if I am persuaded by the authors’ design of UID = M / L, where M is some metric, and L is the combined length of inputs and outputs averaged over the dataset. The design of the metric quantifies the tradeoff between model quality and inference cost through a ratio. However, there is an implicit assumption that the desired tradeoff between M and L is linear. It may be that the model quality M can be weighted quadratically more than the sequence length L (e.g. I am a startup with a lot of money to burn and am willing to spend more for a better quality chatbot). In this case, a better metric would be M^2 / L, or in general M^a / L for some a. If there is a scaling parameter involved, then the authors’ conclusions on which models perform best on UID are thrown into question. This paper would benefit from a deeper analysis on how to quantify the tradeoff between M and L, e.g. for a = 0.5, 1, 2, 5, 10…. At high enough a, I would assume that full prompting on GPT-3 would have the best UID.\n\n2) The conclusion presented by the authors (that summarized dialogues or Semantic/Recent-1 is the best approach) is not very impactful. While having this shown empirically is important, I’m not sure what readers can take away from this when doing future dialogue research. It’s very intuitive that conversations can have a lot of redundant information between utterances that can be condensed in order to save on sequence length (e.g. in both DialogSum and SAMSum).", "questions_to_authors": "1) Can the UID metric be redefined with a scale parameter M^a / L? To quantify the value of the tradeoff between quality and inference cost.\n\n2) PEGASUS-DI + BI has a drastically worse UID vs. PEGASUS-DI in Table 4, on both MSC and TC. Is there a reason why? From what I can tell in Section 4, I don’t think the length of the BI is very long.", "paper_topic_and_main_contributions": "The paper does a comprehensive study on prompt engineering GPT-3 and other LLMs for open-domain dialogue datasets MSC and TC. The authors wish to focus on frugality when prompting, and to quanitfy the trade-off between model quality and inference cost, introduct Usable Information Density (UID) as a metric to measure it.\n\nThey find that:\n - Semantic-1 is the best for UID. Higher k on Recent-k and Semantic-k can improve the core metric but reduce UID.\n - Using summaries has better UID vs. full dialog history.\n - Few-shot accuracy does worse than zero-shot.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper explores prompt optimization for dialogue applications (focusing on open-domain chit chat).

**Pros**: All reviewers agree the study is fairly comprehensive, and most mention that the problem is important/relevant. There are limited technical concerns overall, and most concerns raised appear to be resolved during the rebuttal.

**Cons**: There is some disagreement among reviewers on the impact of the work. While reviewers all agree on the comprehensiveness of the study, some reviewers also feel there are issues in interpretation of the results, indicating a lack of generalizable takeaways and concerns on the proposed UID metric. Authors provide a detailed rebuttal with new results and highlight relevant takeaways from the current iteration - both of which are acknowledged by reviewers - but excitement is still marginal.
Decision: Accept-Findings