Is ChatGPT a Good Multi-Party Conversation Solver?
Authors: Chao-Hong Tan
Jia-Chen Gu
Zhen-Hua Ling
Venue: EMNLP 2023 Findings (2023)
Keywords: large language models
multi-party conversations
zero-shot
in-context learning
TL;DR: Exploring the ability of Large Language Models (LLMs)  on multi-party conversations (MPCs) and apply MPC Structure Incorporation to further improve.

Abstract: Large Language Models (LLMs) have emerged as influential instruments within the realm of natural language processing; nevertheless, their capacity to handle multi-party conversations (MPCs) – a scenario marked by the presence of multiple interlocutors involved in intricate information exchanges – remains uncharted. In this paper, we delve into the potential of generative LLMs such as ChatGPT and GPT-4 within the context of MPCs. An empirical analysis is conducted to assess the zero-shot learning capabilities of ChatGPT and GPT-4 by subjecting them to evaluation across three MPC datasets that encompass five representative tasks. The findings reveal that ChatGPT’s performance on a number of evaluated MPC tasks leaves much to be desired, whilst GPT-4’s results portend a promising future. Additionally, we endeavor to bolster performance through the incorporation of MPC structures, encompassing both speaker and addressee architecture. This study provides an exhaustive evaluation and analysis of applying generative LLMs to MPCs, casting a light upon the conception and creation of increasingly effective and robust MPC agents. Concurrently, this work underscores the challenges implicit in the utilization of LLMs for MPCs, such as deciphering graphical information flows and generating stylistically consistent responses.

Is ChatGPT a Good Multi-Party Conversation Solver?
Chao-Hong Tan, Jia-Chen Gu, Zhen-Hua Ling" National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, China edu. en, {gujc, zh1 chling] @ustc. edu. cn
chtangmail
ustc. usteeaaiiiuss....oooo.oooo
Abstract
Peng et al., 2023), has significantly amplified their proficiency in language comprehension, generation, interaction, and reasoning. The impressively high performance exhibited by these models has been extensively documented in related works (Qin et al., 2023; Wang et al., 2023a; Bubeck et al., 2023).
Large Language Models (LLMs) have emerged influential instruments within the realm of natural language processing; nevertheless, their capacity to handle multi-party conversations (MPCs) a scenario marked by the presence multiple interlocutors involved in intricate information exchanges remains uncharted. In this paper, we delve into the potential of generative LLMs such as ChatGPT and GPT-4 within the context of MPCs. An empirical analysis is conducted to assess the zero-shot learning capabilities of ChatGPT and GPT-4 by subjecting them to evaluation across three MPC datasets that encompass five representative tasks. The findings reveal number of evaluated MPC tasks leaves much to be desired, whilst GPT-4'ss results portend promising future. Additionally, we endeavor to holster performance through the incorporation of MPC structures, encompassing both speaker and addressee architecture. This study provides an exhaustive evaluation and analysis of applying generative LLMs to MPCs, casting a light upon the conception and creation of increasingly effective and robust MPC agents. Concurrently, this work underscores the challenges implicit utilization of L.MSs for MPCs, such as deciphering graphical information flows and generating stylistically consistent responses.
Multi-Party Conversations (MPCs) represent prevalent and natural facet of human commu- nication. In these exchanges, more than two participants engage in interactive discourse variety of topics. Such dynamic introduces new challenges and opportunities for dialogue systems. Here, the key requirement isn't merely generating coherent and relevant utterances, but also making strategic determinations about when to intervene and whom to address in the conversation. We queried ChatGPT about its potential strategies for addressing the inherent challenges of MPCs with " "Can you solve multi-party conversation tasks?". ChatGPT's response was as follows: "I do not have built-in mechanisms to keep track of indi- vidual participants in a conversation. Therefore, 's important to explicitly mention the name or identifier of the participant you are addressing when providing instructions or asking questions.", ", which intriguing and touched upon some of the critical aspects we are investigating in this paper.
that ChatGPT's performance
Considerable efforts have been made to explore the capabilities of LLMs across a variety of NLP tasks (Qin al., 2023; Sun al., 2023). Despite these advancements. the effectiveness of LLMs handling MPCs remains largely underexplored. this study, we scrutinize the potential of LLMs such as ChatGPT and GPT-4 in managing MPCs by implementing five distinct tasks including Emo- tion Detection, Addressee Recognition, Speaker Identification, Response Selection, and Response Generation, across three different MPC datasets. Our experiments reveal that both ChatGPT and GPT-4 can achieve performance on par with super- vised methods when evaluated on the EmoryNLP
1 Introduction
Large Language Models (LLMs), including notable instances such as ChatGPT (OpenAI, 2022) and GPT-4 (OpenAI, 2023), are ushering in a new era in the field of natural language processing (NLP), showeasing remarkable zero-shot and few- shot generalization capabilities. The methods of pretraining language models extensive text corpora (Brown et al., 2020; Touvron al., 2023), followed by aligmment fine-tuning to ensure adher- ence to human instructions (Wang et al., 2023b;
"Corresponding author.

---

and MELD datasets. However, the performance of these models on the more complex Ubuntu IRC dataset remains less than satisfactory. To address this, we introduce a strategy known as MPC Structure Incorporation, which weaves the speaker and addressee structure information of MPCs into the LLMs. This addition leads to significant performance boost of ChatGPT and GPT-4 across all three datasets, underpinning the value of this approach for improving LLM effectiveness MPC scenarios.
and Zhang al. (2018) proposed update speaker embeddings with conversation streams dynamically and role-sensitively. Wang et al. (2020) proposed to track the dynamic topic in a conversation. Gu et al. (2021) proposed jointly learning "who says what to whom" in a unified framework by designing self- supervised tasks. Gu et al. (2023) present graph- induced fine-tuning which adapts Transformer- based LMs by integrating four types of edges into attention mechanisms. On the other hand, Hu al. (2019) explored generation based approaches by proposing graph suap--ttuctured network (GSN) that encoded the conversation context using homoge- neous GNNs. Gu et al. (2022) proposed HeterMPC to model the complicated interactions between utterances and interlocutors with a heterogeneous graph. Li and Zhao (2023) proposed to iteratively generate missing addressees and optimize the generative model via the EM algorithm.
In summary, our contributions in this paper are three-fold: 1) An exploratory study to examine the performance of ChatGPT and GPT-4 in handling MPCs within zero-shot context is carried out. This is the first study of its kind to investigate how these LLMs perform in MPC scenarios. 2) An MPC structure incorporation approach is proposed, which enhances the performance of ChatGPT and GPT-4 in managing MPCs, This strategy integrates the speaker and addressee structure information of MPCs into LLMs, leading to substantial per- formance improvements. 3) We delve into the potential of LLMS in handling MPCs and shed light on the challenges that need to be tackled in future research. This discussion forms the basis for continued investigations into improving LLM effectiveness in MPC contexts.
LLMs for NLP Tasks Several contemporaneous papers present empirical studies of leveraging LLMs for various NLP tasks to explore whether LLMs can achieve competitive performance. For example, Qin et al. (2023) study the zero-shot learning cupability of ChatGPT by evaluating it on representative task categories, such as reasoning, natural language inference, and summarization. Liu et al. (2023) and Wang et al. (2023a) explore whether ChatGPT is good evaluator of natural language generation. Sun et al. (2023) investigate LLMs for relevance ranking in information trieval. Zheng et al. (2023) seek to understand why ChatGPT falls short in providing truthful answers and provide a guideline towards truthfulness in question answering via LLMs.
2
Related Work
Large Language Models Recently, the develop- ment of large language models (LLMs) has made tremendous progress, as evidenced by models such as GPT-3 (Brown et al., 2020), PaLM (Chowdhery et al., 2022), LLaMA (Touvron et al., 2023), Chat- GPT (OpenAI, 2022), and GPT-4 (OpenAT, 2023). These LLMs exhibit emergent abilities, including in-context learning, mathematical reasoning, and commonsense reasoning (Wei et al., 2022). A recent line of work focuses instruction learn- ing, either via generating high-quality instructions data (Wang al., 2023b; Peng al., 2023) or by boosting LLMs with instructions (Chung et al., 2022; Taori et al., 2023).
To the best of our knowledge, this paper makes the first attempt to empirically analyze the zero- shot lcarning ability of LLMs on the MPC tasks, providing comprehensive evaluation and analysis to inspire the development of more effective and robust MPC agents.
Approach
To explore an out-of- MPC solver, our emphasis lies in the zero-shot setting. Instruction for each task is shown in Figure 1. For each task, LLMs are first instructed with the prompt "You have been presented with a sequence of multi-party conversational turns, organized in chronological order.". Then, LLMs are instructed to complete the task with task-specific prompts. To stabilize
Multi-Party Conversations Existing methods on building MPC systems can be generally cat- egorized into retrieval-based approaches (Ouchi and Tsuboi, 2016; Zhang et al., 2018; Wang et al., 2020; Gu et al., 2021, 2023) or generation based (Hu et al., 2019; Gu et al., 2022; Li and Zhao, 2023). On the one hand, Ouchi and Tsuboi (2016)

---

Instruction:
task instruction "Please ideniffy the speaker of the last sentence." and output template "The output format should be only one speaker.". Dialogue history with speaker is formalized as "#(num] (speaker]: [utterance|".
You have been preseiated with a sequence of padi--party conversational
organized chrowological onder.
$ T(Takk Instruction)
S/Ouppt Template)
ExExtra Description)
Use tomperatueeml), minimize ummocsstory words not confised
Input:
Response Selection (RS) LLMs are tasked to select the most appropriate response from the candidates with the task instruction "Your task is select the most appropriate response from the candidate set." and output template "The oulput format must be: #[num) - /utterance]". The candidate set formalized as "#(num) [utterance]". Thus the history with input template is formalized as "Dialogue History: (conversation turns/ Candidates: /candidates|".
HtHisooy with Input Template)
S(Extend Instruction]
Response:
Figure I: The prompt template to complete the tasks.
the output of LLMs, the effect of temperature is amplified with the prompt "Use temperature=0, minimize wunecessary words to not get confused.", For more comprehensive explication, certain tasks require an Extended Instruction as shown in Ta- ble 5.
Response Generation (RG) LLMs tasked to generate response with the task instruction "Your task is to generate the most appropriate response.". There no need to provide the output template since the generation task is free-form.
3.1 Task-Specifie Prompts
Different instructions for each task are designed to guide I.LMs to complete the task, namely task- specific prompts.
3.2 MPC Structure Incorporation
Considering that the complicated interactions be- tween interlocutors, between utterances and be- tween an interlocutor and an utterance naturally increase the difficulty of fully understanding MPCs, it might be helpful to incorporate the MPC structure information.
Emotion Detection (ED) LLMs are tasked to predict the emotion of each utterance with the task instruction "Please evaluate the emotions of each utterance in the dialogue using the following n labels: (...)." and the output template "The output format must be: #[nmm]-[uterancee // [emotion]" Here, n is the number of emotion labels, and (...) is the list of emotion labels. Dialogue history is formalized as "[uta]_[utterrmee"""""
-
Speaker Structure The speaker structure is in- corporated into LLMs to help understand utter- ances. Specifically, the prompts with "[utterance]" is replaced with "/speaker]: /utterance/" to inform LLMs of the speaker of each utterance. An example is shown in Figure 2.
Addressee Recognition (AR) LLMs are tasked predict the addressee of each utterance with the task instruction "Your task is to find the addressee of each utterance." " and the output template "The output format must be: #(num) - (utterance) // Reply to #(replyjif" __"". And we can get the addressee information if we know the reply-to utterance of each utterance. Since the addressee of the first utterance is unknown, add the extra description "Please start from #1 since #0 is the first ufterance that has no reply-to utterance. You should not leave any utterance unattended." " to inform LLMs of the addressce of the first utterance. Dialogue history is formalized as the same as emotion detection.
Addressee Structure The addressee structure of MPCs is constructed with adding sentence "Reply to # #(reply_uid uid]
into prompt.
Speaker-Addresser Structure The speaker- addressee structure of MPCs denotes the information flow from a speaker to an addressee, which is constructed with the prompt "Reply to #(reply_uit _id] Speaker (reply_spie (reply_ "uppy_uuterance""" and "[utterance]" is replaced with "[speaker]: /utterance]".
4 Experiments
Speaker Identification (SI) LLMs are tasked to predict the speaker of the last utterance with the
4.1 Datasets
EmoryNLP EmoryNLP (Zahiri and Choi, 2018) is a collection of the TV show Friends, where
We found that this prompt can improve the performance, even if the temperature is already set to O in API.

---

Datasets
Train
Valid
Test
Instruction:
EmoryNLP (Zahiri and Choi, 2018)
659
89
79
You have been presenied with sequence of multi party conversationa turns, orgonized chronological order. Please esaluote the emotious of each utterances in the dioiogue wsnn the following 7 labels: l'eerr, 'nuurall' Juy', "llguss, 'anger. 'aanness 'nuprisee). The ouppt format #/mum] (speaker): (utterance) il [emotionn Use comperature-G, O, minimize unnocessory words to not get confased.
MELD Poria al. (2019)
1.039
114
280
Ubuntu IRC (Hu et al., 2019)
311,725
5,000
5,000
Table I: Statistics of the datasets evaluated in this paper.
Input:
4.2 Baselines
All: Push. push!
We're here!
(1) BERT (Devlin et al., 2019) is a bidirectional language representation model and can be fine- tuned for various NLP tasks. (2) GPT-2 (Radford et al., 2019) is a uni-directional pre-trained lan- guage model. (3) BART (Lewis et al., 2020) is de- noising autoencoder using standard Tranformer- based architecture, trained by corrupting text with an arbitrary noising function and learning to re- construct the original text. (4) SPCL-CL-ERC (Song et al., 2022) introduced a novel Supervised Prototypical Contrastive Learning loss function for the Emotion Recognition in Conversation task, addressing the issues stemming from imbalanced classification through the medium of contrastive learning, obviating the necessity for large batch sizes. It is the SOTA of the EmoryNLP and MELD dataset on ED task. (5) BART w/. EM (Li and Zhao, 2023) introduce an Expectation- Maximization (EM) method that alternately per- forms the expectation steps to infer addressee labels, and the maximization steps to fine-tune a response generation model. It is the SOTA of the Ubuntu IRC datasets on RG task. (6) MPC-BERT w/. GIFT (Gu et al., 2021, 2023) is pre-trained model for MPCs that learns 'who says what to whom' with self-supervised tasks (MPC-BERT) and a graph- based fine-tuning method (GIFT). It is the SOTA of the Ubuntu IRC datasets on AR, SI and RS tasks. (7) ChatGPT (OpenAI, 2022), ingeniously enriched by the infusion of Supervised Fine- (SFT) and Reinforcement Learning from Human Feedback (RLHF) methodologies, ensuring seamless synchronization between the model and human directives. (8) GPT-4 (OpenAI, 2023) is a large-scale, multimodal model which can accept image and text inputs and produce text outputs, exhibiting human-level performance on various professional and academic benchmarks.
R2 Carol: Where have you been?
Ross: Long story, honey.
#4 Dr. Franzblau: All right, Carol, need you to keep pushing. need
Excuse me. could I have this?
Response:
80 All: Pash. push! // neutral
here! joy
A2 Carol: Where have you bocn? noutral
Ross: Long story, honey. neutral
Franzblau: All right, Carol, need you to keep pushing. need
Excuse me, could I have this? !f noutral
Figure 2: The prompt and output of emotion detection with speaker structure.
each utterance is annotated with one of the seven emotions borrowed from the six primary emotions in the Willcox (1982)'s feeling wheel, including sad, mad, scared, powerful, peaceful, joyful, and default emotion of neutral.
The Multimodal Emotion Dataset (MELD) (Poria et al., 2019) was developed by im- proving and expanding the original EmotionLines dataset (Hsu et al., 2018). MELD includes the same multi-party dialogue instances as EmotionLines but incorporates additional audio and visual modalities alongside text. Each utterance within dialogue is tagged with one of the seven emotions, including: Anger, Disgust, Sadness, Joy, Neutral, Surprise, Fear. This comprehensive tagging facilitates deep emotion-focused analysis.
MELD
Ubuntu IRC benchmark Ubuntu IRC (Hu et al., 2019) represents substantial, Ubuntu Internet Relay Chat (IRC) channel corpus, replete with annotations for multiple interlocutors. Furthermore, it is distinguished by the provision of addressce labels accompanying each individual utterance.
The EmoryNLP and MELD datasets proffer emotion labels, hence enabling the execution of ED tasks. Conversely, the Ubuntu IRC dataset offers Addressee tags, thereby serving as resource for AR tasks. Table presents the statistics of the three datasets evaluated in our experiments.
4.3 Implementation Details
All supervised models were trained with the AdamW method (Loshchilov and Hutter, 2019).
CChatGPT and GPT-4 are recognized representatives of LLMss, so we only consider them to evaluate MPCs.

---

The learning rate was initialized 6.25e-5 and was decayed linearly down O. The batch size was set to 128 gradient accumulation steps. Models were trained in 10 epochs. For ChatGPT and GPT-4, we used the API endpoints gpr-3,5- turbo-0301 and gpt-4-0314 provided by OpenAI respectively.
task, it is essential note that speaker information detection is unattainable without the provision of explicit speaker information. Consequently, our evaluation of CharGPT and GPT-4 in the SI task is limited to instances where speaker information is provided. Results demonstrate that both models outperform BERT on both EmoryNLP and MELD datasets. Specifically, CharGPT and GPT-4 exhibit superior performance BERT by 3.60% and 16.46% on EmoryNLP, and by 13.49% and 34.42% on MELD, respectively. However, it is worth noting that ChatGPT andGPT-4 significantly trail behind supervised models on the Ubuntu IRC dataset. Regarding the RS task, only in the context of the MELD dataset does GPT-4 outshine BERT. Specifically, on the SI task of the Ubuntu IRC dataset, CharGPT and GPT-4 lag behind BERT by 25.58% and 12.31 %, respectively. This can be attributed to the fact that Ubuntu IRC leans towards a more technical and specialized domain, which is also difficult for humans to understand.
4.4 Metries
To evaluate ED task, we employed the weight-Fi score, which is the harmonic mean of precision and recall. To evaluate SI and AR tasks, employed accuracy. To evaluate RS task, we employed R10@1 I, which is the percentage of the first correct response selected from 10 candidates. To evaluate the quality of the generated text, we employed the standard triin-simiilrity-baass metrics SacreBLEU (Post, 2018), ROUGE (Lin, 2004) and METEOR (Banerjee and Lavie, 2005). Higher is better for all metrics. All metrics were calculated by the evaluate toolkit !
Speaker Information Enhancement To delve into the significance of the interlocutor within the MPC understanding, we incorporate speaker information into three distinctive tasks across three datasets, disregarding the ED task. Comparing the line of ChatGPT (GPT-4) and CharGPT (GPT-4) w/. Speaker, we can find that the incorporation of speaker information can improve the performance of ChaiGPT (GPT-4) on all five tasks except for the RS task of Ubuntu IRC. This improvement is not surprising. In the context of MPCs, the speaker doesn't 't adhere to the rigid alternation characteristic of two-party dialogues, hence the incorporation of speaker information can enhance the lucidity of the conversation, rendering it more readily comprehensible. Note that the substantial advancement observed in the AR task partly stems from our modification of the task from identifying the response sentences to directly recognizing the addressee. Nonetheless, compared with ChaIGT, the subpar performance of CharGPT w/. Speaker on the RS task of Ubuntu IRC suggests that the imparted speaker information could not be opti- mally assimilated and deployed by ChatGPT. On the contrary, it appeared to disrupt the process of response selection. However, GPT-4's performance on this task was substantially superior, enhancing the effectiveness of the RS task by a margin of
4.5 Evaluation Results of MPC Understanding
As shown in Table 2, we evaluated the dialogue understanding performance of supervised language models and LLMs on three test sets. Here, the SOTA results of EmoryNLP and MELD on ED are copied from Song et al. (2022). The SOTA results of Ubuntu IRC of all three tasks are copied from Gu et al. (2023).
Supervised Language Models Versus LLMs When one juxtaposes the outcomes associated with supervised language models and LLMs, becomes apparent that the LLMS demonstrate parity in per- formance with their supervised counterparts on the EmoryNLP and MELD datasets. However, their performance on Ubuntu IRC falls short of the mark. It is unsurprising to discern that the capability of GPT-4 surpasses its predecessor, ChaGGT across all four understanding tasks. In the ED task, both CharGPT and GPT-4 outperform BERT but fall short of the -fate-of-the-eee (SOTA) on EmoryNLP. Additionally, ChatGPT lags behind BERT on the MELD dataset. For the AR task, both ChatGPT and GPT-4 trail behind BERT and SOTA on the Ubuntu IRC dataset, respectively, Regarding the SI
*Cude available an/ MCC. 'https:// //g thub. com/hug i ing face/ lu ate
1xcht
*This syntax denotes that the conclusion is applicable to both ChaiGPT and GPT-4

---

Tasks
EmoryNLP
MELD
Ubuntu IRC
Models
ED (F1) SI (ACC) RS (RI0@1) 1)
ED (FI)
(ACC)
RS (R10@1)
AR (ACC)
SI (ACC) RS (R10@1) l)
Supervised
BERT (Devlin 2019)
34.76
47.82
48.68
44.18
48.13
82.88
71.81 90.50
73.42
SOTA
40.94
67.25
90.18
80.74
ChaiGPT (OpenAI, 2022)
37.16
29.11
58.32
36.42
54.11
54.68
Speaker
38.50
51.42
34.21
60.90
57.67
39.17
67.19
46.23
49.52
Addressce
37.50
Speaker & Addressee
38.50
29.50
GPT-4 (OpenA1, 2023)
66.02
61.50
Speaker
64.28
50.00
57.46
82.50
58.00
71.00
Addressee
68.50
Speaker & Addressee
59.50
72.00
Table 2: Evaluation results of MPC Understanding. Numbers in bold denoted that the results achieved the best performance. The vacant cells signify their incalculability. The SOTA of ED task is SPCI-CI-ERCC (Song et al., 2022), and the SOTA of Ubuntu IRC tasks is MPC-BERT w/. GIFT (Gu al., 2023)
Metrics (RG)
EMOryNLP
MELD
Ubuntu IRC
Models
S-BLEU ROUGE, METEOR S-BLBU ROUGE, METHOR S-BLEU ROUGE, METBOR
Supervised
GPT-2 (Radford al., 2019)
0.6175
7,90
10.26
0.5160
6.01
7.74
0.93
4.01
BART (Lewis al., 2020)
0.7009
8.63
11.86
1.0757
8.64
10.37
0.95
4.46
SOTA
2.45
11.71
5.52
1.I.ss
ChatGPT (OpenAI, 2022)
0.5358
9.03
11.30
0.9059
8.63
0.5145
8.64
8.85
w/. Speaker
0.3082
8.95
11.45
9.86
0.8512
9.45
w/. Addressoe
0,3332
10.73
11.49
w/. Speaker & Addressce
0.6101
10.60
12.07
GPT-4 (OpenAI, 2023)
0.4608
8.60
12.22
0.9301
8.93
12.43
0,2206
7,86
12.03
w/. Speaker
0,9049
9.99
14.61
0.9666
8,89
13.06
0.2324
9.18
12.65
w/. Addressee
0.2458
9.52
12.78
w/. Speaker & Addressec
0.2856
9.24
13.30
Table 3: Evaluation results of MPC Generation. Numbers in bold denoted that the results achieved the best performance. S-BLEU is the short of SacreBLEEU. The vacant cells signify their incalculability." The SOTA of Ubuntu IRC is BART w/. EM (Li and Zhao, 2023).
9.50%. This result suggests that GPT-4 is better at fusing speaker information than CharGPT.
the SI and RS tasks, well between the results of CharGPT and ChatGPT w/. Addressee on the RS task, we find, to our surprise, that the integration of addressee information has led to a diminution in performance. When comparing the results of GPT- 4 w/. Speaker and GPT-4 w/. Speaker & Addressee on SI and RS tasks, we find that the incorporation of addressee information can slightly improve the performance. The only marginal improvement of approximately 1.00% observed on the and RS tasks of Ubuntu IRC can be predominantly attributed to the proficiency GPT-4 w/. Speaker in correctly inferring addressee information, given its noteworthy accuracy of 82.50% on AR tasks.
Addressee Information Enhancement To probe the import of the addressee within the context of MPC comprehension, we integrate addressee information into the SI and RS tasks of Ubuntu IRC. This is primarily due to the absence of addressee information within the other AR task is excluded from this process, given that addressee information is unavailable within the corresponding Ubuntu IRC task. When drawing comparisons between the results of CharGPT w/. Speaker and ChaiGPT w/. Speaker & Addressee on
datasets. The
'For SI task, the speakers in dialogue history are needed. It means that we cannot lack SI information, so these cells are empty. For EmoryNLP and MELD datasets lacking the addressee information, therefore corresponding rows are empty. For the Ubuntu IRC AR task, it asked not to add the addressce information, so these cells are empty.
Speaker and Addressee Information Enhance- ment When contrasting the results of CharGPT (GPT-4) w/. Speaker & Addressee with the Chat- GPT (GPT-4) on RS tasks, we discern that the

---

##* Instruction:
You have been presented with sequence of multi-purty conversational Please identify the speaker of the last sentence. The output format should be only one speaker. Use temperature=d,, minimize unnecessury words to not get confused.
organized in chronological order.
Note that the speaker is one of ['Speaker I', 'Speaker 2", "Speaker 3", "Speaker 4'].
### Input:
#0 Speaker I: let me look at this post-meeting
|Reply ... 1 Speaker 2: are you doing promotions main if when good time to ping you about xfce lot
[Reply #2 Speaker after we have anastacia again so that do break stuff
[Reply #5 ] #6 have FILEPATH checked for reverse-dependencis? the archive tools do that check do n't work yet
### Response:
Answer Speuker
GPT-4 w/. Speaker Addressee: Speaker
CharGPT w/. Speaker Addressee: Speuker
GPT-4 w/. Speaker: Speaker 4
ChaiGP w/. Speaker: Speaker
Table 4: One selected example of the SI task Ubuntu IRC with speaker and addressee structure. Here also list the results of ChatGPT (GPT-4) w/. Speaker for comparison.
integration of speaker and addressee information precipitates performance decrement of 25.18% for CharGPT, whilst conversely, it elicits aug- mentation of 1 1..00 in the performance of GPT- 4. Indeed, both categories of information enhance the comprehension of the conversation, yet the infusion of additional data concurrently am- plifics the complexity of understanding. Empirical outcomes indicate that ChatGPT grapples with the processing of this surplus information, whereas its more potent successor, GPT-4, exhibits the capacity to assimilate it effectively.
this particular set. In terms of the METEOR metric,
LLMs outdistance supervised models across all test sets, thereby affirming the formidable aptitude of LLMs in generating responses.
to
MPC Structure Incorporation Similar to the results in MPC Understanding, the inclusion of interlocutors' information emerges as beneficial to the generation of dialogue responses. This is substantiated by the marked superiority Chat- GPT (GPT-4) w/. Speaker over CharGPT (GPT- 4) across all three test sets, with the exception of ChatGPT's performance on the EmoryNLP dataset. Nonetheless, addressee information doesn't seem to confer discernible advantage in the genera- tion of dialogue responses when comparing the performance of CharGPT w/. Addressee to the CharGPT, as well as the performance of CharGPT w/. Speaker & Addressee to ChatGPT w/. Speaker. When our focus shifts to GPT-4, we observe that the performance of GPT-4 w/. Addressee does not exhibit the anticipated enhancement. However, with the addition of interlocutor information, there is noticeable improvement in the performance of GPT-4 w/. Speaker & Addressee.
4.6 Evaluation Results of MPC Generation
As shown in Table 3, evaluated the dialogue response generation performance of supervised language models and L.Ms on three test sets. The SOTA results of Ubuntu IRC of all three tasks are copied from Li and Zhao (2023).
Supervised Language Models Versus LLMs It becomes apparent that the SacreBLEU scores of supervised models consistently eclipse those of LLMs in a significant number of cases across all three evaluative subsets. This phenomenon is largely a byproduct of ChatGPT and GPT-4's inclination to generate more prolix responses, an approach inherently detrimental to the calculation of SacreBLEEU. Regarding the ROUGE, metrics, LLMs yield superior results compared to super- vised models on the EmoryNLP and MELD test sets. In contrast, within the ambit of the Ubuntu IRC dataset, supervised models command a greater presence-aad an outcome likely driven by the am- plified comprehension complexity associated with
4.7 Case Study
To analyze the performance of LLMs the tasks of MPC understanding and generation specifically, case studies were conducted by presenting ran- domly selected examples for further illustration.
MPC Understanding As shown in Table 4, the absence of addressce information, the model exhibits inadequacies comprehending MPC,

---

#** Instruction:
You have been presented with a sequence of multi-purty conversational tums, organized in chronological order. Your task is generate the most appropriate response.
The output format is "[Reply to #(reply_uid] _uid] Speaker (reply_spk]; [reply_utterance]] Speaker (rsp_spk): (1sp)". Use temperature=0, minimize unnecessury words to not get contused.
#*# Input:
Dialogue History:
#0- Speaker 1: yep- only primaries.. but ive never needed more then 4
IReply #0 | #1 Speaker 2: but ca n't see your other partitions right?
[Reply to #3 ...] #4 Speaker 2: i see there 's some confusion, that 's my mistake guess.. with active meant that windows ca n't see' the other partitions.. but i can see that 's only a historical issuc..
Please give response on behalf of Speaker for Uttenrance Speaker see there some confusion, that my mistake guess., with active i
The part of response [Reply meant that windows n't 'see" the other partition... but that only historical issue..] Speaker 1.
Please finish the response generation.
### Response:
BART
i'm sure, but i'm sure there is way do that
You can automate the postfix sctup routine and use one machine as the relay for all the others.
ChatGPT
This will save you time and effort.
Yes, you cun automate the setup process by creating script that answers the questions for you.
w/. Speaker & Addressee
This will save you time and ensure that the setup consistent across machines.
If you're trying to automate the process and distribute to multiple machines, you might consider creating shell script or Ansible playbook that sets everything nocessary for node.js development, including postfix. This way, you can handle the postfix setup with pre-defined configurations, and then package all the necessary files and tools into an AMI for EC2 or an Ubuntu image. This would be more maintainable and scalable. Yes. you can automate the process. You
GPT4
tools like 'expect' or 'auloexpect" in Linux handle interactive command line sessions. These tools can learn from your inputs and can generate script that automates the process, Alternatiely, you might ulso consider using configuration management tool like Ansible or Puppet.
w/. Speaker & Addressee
Ground Truth
untar your default config all your client machines
Table 5: One selected example the RG task on Uhuntu IRC with speaker and addressee structure. Here we also list the results of ChatGPT, GPT-4 and BART for comparison.
resulting in inaccurate responses from both Chat- GPT and GPT-4. However, when furnished with addressee information, GPT-4 shows improved aptitude to comprehend the dialogue, leading to a correct response. Conversely, CharGPT appears to be confounded by the addressce information, yielding an incorrect answer that lies beyond the purview of the candidates.
zero-shot learning ability of ChatGPT and GPT-4 by evaluating them on three popular MPC datasets covering five representative tasks. On EmoryNLP and MELD datasets, ChatGPT and GPT4 achieve comparable performance to the supervised training models. However, ChatGPT performs poorly on the evaluated Ubuntu IRC tasks, while GPT-4 shows promising results. However, there is still large gap relative to the supervised training on SI task of Ubuntu IRC. Taking into account the structure of the MPC, it is evident that both ChatGPT and GPT-4 enhncedd performance across nearly all tasks when equipped with speaker information. However, in the context of addressee information, ChatGPT's performance may decline if it is encumbered with extraneous data. Con- versely, GPT-4 aptly leverages this information to accomplish the task with heightened proficiency. Devoting efforts towards efficaciously intertwining the graphical structure inherent in MPCs with LLMs through prompts even supervised fine- tuning constitutes one of future work.
MPC Generation As shown in Table 5, the response generated by BART is conspicuously devoid of substantive content. All the LLMs, espe- cially GPT-4, exhibit the ability to produce lengthy and meaningful responses. Although ChalGPT and GPT-4 are capable of crafting responses with greater pertinence, their propensity to yield verbose responses hampers the SacreBLEU score.
5 Conclusion
this paper, we have explored the abilities of generative LLMs for MPCs, which have been largely underexplored. We empirically analyze the

---

Limitations
Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi. Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, et al. 2022. PaI.M: Scaling language modeling with pathways. CoRR, abs/2204.02311
The design of prompts plays a crucial role in determining the final results, as it holds significant sway over the outcome. The way in which prompts are constructed and structured can greatl impact the performance and effectiveness of ChatGPT and GPT-4 when it comes to handling MPC tasks. However, our current prompt architecture might not fully encompass the ideal potential and capabílities of these advanced language models in tackling MPC tasks. There is a possibility that further improvements and refinements in the prompt de- sign can unlock even greater performance and unleash the full potential of ChatGPT and GPT- 4 in handling MPC tasks. It is essential to explore and enhance the prompt architecture to ensure optimal results and leverage the capabilities of these powerful language models to their fullest extent in the realm of MPC tasks.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghami. Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang. Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling instruction- finetuned language models. CoRR, abs/22 14
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 4171 4186.
Acknowledgements
This work was supported by the Opening Founda- tion of State Key Laboratory of Cognitive Intel- ligence, iFLYTEK COGOS-2022003 We thank anonymous reviewers for their valuable comments.
Jia-Chen Gu, Zuen-Hua Ling, Quan Liu, Cong Liu, and Guoping Hu. 2023. GIFT: graph-induced fine- tuning for parti-pprtt conversation understanding. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, ACI. 2023. (Volume 1: Long Papers), Toronto, Canada, July 9-14, 2023. Association for Computational Linguisties.
References
Satanjeev Banerjee and Alon Lavie. 2005. METEOR:
automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the Workshop on Intrinsic and Extrinsic Eval- uation Measures for Machine Translation and/or Summarization@Al @CL 2005, Ann Arbor, Michigan, USA, June 29, 2005, pages 65-72. Association for Computational Linguistics.
Jia Tia-Chen Gu, Chao-Hong Tan, Chongyang Tao, Zhen- Hua Ling, Huang Hu, Xiubo Geng, and Daxin Jiang. 2022. HeterMPC: A heterogeneous graph neural network for response generation in multi- party conversations. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 5086-5097. Association for Computational Linguistics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sustry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference Neu- ral Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual.
Jia-Chen Gu, Chongyang Tao, Zhen-Hua Ling, Can Xu, Xiubo Geng, and Daxin Jiang. 2021. MPC- BERT: A pre-trained language model multi- party conversation understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1ith International Joini Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 3682-3692. Association for Computational Linguistics.
Sébastien Bubeck, Varun Chandrasckaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li. Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with GPT4. CoRR, abs/2303.15 122112
Chao-Chun Hsu, Sheng- Chen, Chuan-Chun Kuo, Ting-Hoo K. Huang, and Lun-Wei Ku. 2018. Emotionlines: An emotion corpus of multi-party conversations. In Proceedings of the Eleventh
Aakanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts,

---

International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018. European Language Resources Association (ELRA).
dataset for emotion recognition in conversations. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL. 2019, Florence, Italy, July 28- August 2019, Volume I: Long Pupers, pages 527-536. Association for Computational Linguistics.
Wenpeng Hu, Zhangming Chan, Bing Liu, Dongyan Zhao, Jinwen Ma, and Rui Yan. 2019. GSN: A graph-structured network for multi-party dialogues. In Proceedings of the Twenty-Fighth International Joint Conference on Artificial Intelligence, IJCA 2019, Macuo, China, August 10-16, 2019, pages 5010-5016.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, WMT 2018. Belgium, Brussels, October 31 November I, 2018, pages 186-191. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to--sequeee pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7871-7880 Association for Computational Linguistics.
Chengwei Qin, Aston Zhang. Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023, Is ChatGPT a general-purpose natural language processing task solver? CoRR, abs/2302.06476.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):
Yiyang Li and Hai Zhao. 2023. EM pre-training for multi-party callti-party dialogue response generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Lingnistics, ACL 2023, (Volume I: Long Papers). Toronio, Canada, July 9-14, 2023. Association for Computational Linguistics.
Xiaohui Song, Longtao Huang, Hui Xue, and Songlin Hu. 2022. Supervised prototypical contrastive learning for emotion recognition in conversation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 5197-5206. Association for Computational Linguistics
Chin-Yew Lin. 2004. Rouge: package for automatic evaluation of summaries. In Text summarization branches ouf.
Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023. chatgpt good at search? investigating large language models as re-ranking agent. CoRR, abs/2304.09542..
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using GPT4 with better human alignment. CoRR, abs/2303. 16634.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instrucionnfolloviin Ilama model. ht tps: / htg.tb.t.bbbc.///g///ggg wtw/ lab/ stanfor d d_alpaca.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971.
OpenAl. 2022. Introducing chatgpt.
OpenAl. 2023. GPT-4 technical report. CoRR, abs/2303.08774.
Hiroki Ouchi and Yuta Tsuboi. 2016. Addressee and response selection for multi-party conversation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2133-2143.
Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi. Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023a. Is chatgpt good NLG evaluator? A preliminary study. CoRR, abs/2303.04045.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with GPT-4. CoRR, abx/2304.03277.
Weishi Wang, Steven C. H. Hoi, and Shafiq R. Joty. 2020. Response selection for multi-party conversations with dynamic topic tracking. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020. Online, November 16-20, 2020, pages 6581- 6591.
Soujanya Poria, Devamanyu Hazarika, Navonil Ma- jumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. MELD: A multimodal multi-party

---

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra. Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023b. Self-instruct: Aligning language model with self generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguisticx, ACL 2023, (Volume I: Long Papers), Toronto, Canada, July 9-14, 2023. Association for Computational Linguistics.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud. Dani Yogatama. Maarten Bosma, Denny Zhou, Donald Metzler, Bd H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent abilities of large language models. Trans. Mach. Learn. Res., 2022.
Gloria Willcox. 1982. The feeling wheel: A tool for expanding awareness of emotions and increasing spontaneity and intimacy. Transactional Analysis Journal, 1244)2274-
Sayyed M. Zahiri and Jinho D. Choi. 2018. Emotion de- tection on TV show transcripts with sequence-based convolutional neural networks. In The Workshops of the The Shirty-Second AAAI Conference Intelligence, New Orleans, Louisiana, USA, February 2-7, 2018, volume WS-18 18 of AAAI Technical Report, pages 44-52. AAAI Press.
Artificial
Rui Zhang. Honglak Lee, Lazaros Polymenakos, and Dragomir R. Radev. 2018. Addressee and response selection in multi-party conversations with speaker interaction rnns. In Proceedings the Secooy- AAAI Conference on Artificial Intelligence, (AAAI- 18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 5690-5697.
Shen Zheng. Jie Huang, and Kevin Chen-Chuan Chang. 2023. Why does chatgpt short answering questions faithfully? CoRR, 105123


Reviews:
[{"review_id": "t8UQgn4Eh1", "reviewer": "Reviewer_pXrH", "summary": "", "questions": "- is there a reason for not providing results with few-shots learning? This would be reasonably easy to test, by including e.g. a few examples into the prompt.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and insights.\n\n#### **Re** *Reason To Reject 1*:\n\n*\"the contribution is very limited: it essentially boils down to providing zero-shot results with GPT 3.5 and 4 on existing tasks and datasets. The presented approach simply adds speaker and/or addressee into the formatting of the dialogue, but this is relatively trivial.\"*\n\n1. Since ChatGPT is a closed source model, we only have access to its inputs as well as outputs. So what we can do is change its inputs (Prompts) to get better results.\n2. The main purpose of this paper is to explore whether the most powerful language models available can be good tools for solving MPC problems, which have not yet been explored. As the reviewer 2d2w agrees, the application of LLMs in MPC is expected to expand in the near future, and we believe that this paper can provide good inspiration for follow-up work. \n3. MPCs do not maintain sequential information flows since the third utterance can reply to the first one, which brings a challenge for autoregressive serialized LLMs, especially for ChatGPT, which is carefully designed for two-person conversations.\n4. As for how to improve the performance of LLMs in MPC, inspired by MPC--BERT [1], we utilize additional information including the speaker and the addressee. How these two types of information are utilized and how they affect the performance of LLMs in MPC tasks are also our contributions. In addition, we discuss the experimental results in detail. This discussion forms the basis for continued investigations into improving LLM effectiveness in MPC contexts.\n\n#### **Re** *Reason To Reject 2*:\n\n*\"the experiments only test for GPT 3.5 and GPT4 along with a few baselines based on BERT, BART and a few others. Why not include a few other large LLMs, such as Llama 2?\"*\n\n1. ChatGPT and GPT-4 are well-recognized representatives of LLMs, so we only consider them as evaluation objects. \n2. We wish to explore whether LLMs with zero-shot can perform as well as supervised trained models in the MPC task, so we do not consider other LLMs as evaluation objects. \n3. For a reference, we provide the performance of Vicuna-1.5 (*w/. Speaker*) which is finetuned from Llama-2 on the MELD dataset as below. We found that it's much more difficult to control the output for Vicuna comparing with ChatGPT.\n\n|                                | MELD    |          |            |             |\n| ------------------------------ | ------- | -------- | ---------- | ----------- |\n| Vicuna-1.5-13b (*w/. Speaker*) | ED (F1) | SI (ACC) | RS (R10@1) | RG (METEOR) |\n| **Zero-Shot**                  | 54.89   | 42.79    | 11.29      | 12.43       |\n| **2-shots (Retrival)**         | 56.32   | 33.48    | 15.63      | 13.33       |\n| **2-shots (Random)**           | 55.26   | 25.58    | 25.08      | 11.82       |\n\n\n\n#### **Re** *Question:*\n\n*is there a reason for not providing results with few-shots learning? This would be reasonably easy to test, by including e.g. a few examples into the prompt.*\n\n1. The scenario we consider is an out-of-the-box MPC solver, so we focus more on the performance of LLMs with zero-shot. \n2. Few-shots increase the prompt tokens leading to more cost.\n3. For comparison purposes, we also provide some results of the few-shots experiments as below. Example selection includes random-select and retrieval by similarity (calculated by Model sentence-transformers/all-MiniLM-L6-v2). \n   Note that *w/. All* means that *w/. Speaker & Addressee* if *Addressee* is accessable else *w/. Speaker*.\n\n|                       | MELD    |          |            |             | Ubuntu   IRC |          |            |             |\n| --------------------- | ------- | -------- | ---------- | ----------- | ------------ | -------- | ---------- | ----------- |\n|                       | ED (F1) | SI (ACC) | RS (R10@1) | RG (METEOR) | ED (F1)      | SI (ACC) | RS (R10@1) | RG (METEOR) |\n| **Zero-Shot**         |         |          |            |             |              |          |            |             |\n| GPT-3.5-Turbo w/. All | 60.90   | 57.67    | 39.17      | 9.86        | 67.19        | 38.50    | 29.50      | 12.07       |\n| GPT-4 w/. All         | 64.18   | 78.60    | 57.46      | 13.06       | 82.50        | 58.00    | 72.00      | 13.30       |\n| **2-shot (Retrival)** |         |          |            |             |              |          |            |             |\n| GPT-3.5-Turbo w/. All | 59.98   | 60.00    | 40.00      | 11.09       | 65.37        | 47.50    | 40.00      | 17.42       |\n| GPT-4 w/. All         | 64.84   | 80.00    | 54.00      | 13.42       | 81.00        | 86.00    | 78.00      | 20.40       |\n| **2-shot (Random)**   |         |          |            |             |              |          |            |             |\n| GPT-3.5-Turbo w/. All | 61.88   | 61.39    | 38.21      | 8.16        | 65.35        | 41.50    | 24.00      | 12.26       |\n| GPT-4 w/. All         | 63.65   | 86.00    | 54.00      | 13.24       | 79.76        | 68.00    | 70.00      | 9.99        |\n\n\n\n#### **Re** *Typos 1:*\n\n*\"line 22: treatise --> study\"*\n\nThank you for your correction.\n\n#### **Re** *Typos 2:*\n\n*\"line 29: graphical information flows? \"*\n\nMPCs do not maintain sequential information flows since the third utterance can reply to the first one. The same term also was used in HeterMPC[2].\n\n#### **Re** *Typos 3:*\n\n*\"figure 1 can be dropped\"*\n\nThank you for your advice. Figure 1 presents a vivid example of human-machine dialogue that demonstrates the power of current LLMs. In terms of content, ChatGPT's response was intriguing and touched upon some of the critical aspects we are investigating. \n\nWe will remove figure 1 and put the example in the main body in our revision.\n\n#### **Re** *Typos 4:*\n\n*\"I would suggest another than than \"MPC structure incorporation\" for the approach\"*\n\nThank you for your advice. As stated in *Re Typos 2*, MPC is with graph information flow. \"Speaker to Utterance\" and \"Utterance to addressee\" can both be considered as “graph structure\".\n\n#### **Re** *Typos 5:*\n\n*\"lines 254-256: I assume the last part (with \"[utterance]\" ...) should be removed?\"*\n\nI apologize for the lack of clarity. The meaning of this sentence is to add speaker information to utterances. We will correct the expression of (with \"[utterance]\" ...) to (and \"[utterance]\" …).\n\n#### **References**\n\n[1] MPC-BERT: A Pre-Trained Language Model for Multi-Party Conversation Understanding. Gu et al. ACL 2021.\n\n[2] HeterMPC: A Heterogeneous Graph Neural Network for Response Generation in Multi-Party Conversations. Gu et al. ACL 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- the paper is well-written\n- the problem of understanding multi-party conversation is both interesting and useful\n- the experimental design is generally well described", "reasons_to_reject": "- the contribution is very limited: it essentially boils down to providing zero-shot results with GPT 3.5 and 4 on existing tasks and datasets. The presented approach simply adds speaker and/or addressee into the formatting of the dialogue, but this is relatively trivial.  \n- the experiments only test for GPT 3.5 and GPT4 along with a few baselines based on BERT, BART and a few others. Why not include a few other large LLMs, such as Llama 2?", "questions_to_authors": "- is there a reason for not providing results with few-shots learning? This would be reasonably easy to test, by including e.g. a few examples into the prompt.", "paper_topic_and_main_contributions": "The paper presents experimental results with the use of ChatGPT / GPT-4 on multi-party conversation data. More precisely, it investigates the zero-shot performance of GPT3.5 and GPT-4 on various MPC tasks such as emotion detection, addresse recognition, speaker identification, response selection and generation, using three distinct datasets. The paper also shows that incorporating speaker and addressee information to the formatting of the dialogue history leads to performance improvements.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 22: treatise --> study\n- line 29: graphical information flows?\n- figure 1 can be dropped\n- I would suggest another than than \"MPC structure incorporation\" for the approach\n- lines 254-256:  I assume the last part (with \"[utterance]\" ...) should be removed?"}}, {"review_id": "DgDKqi5bwt", "reviewer": "Reviewer_CWvj", "summary": "", "questions": "A: In Figure 2 and in Section 3.1, you mention \"extend instruction\" and I do not understand what that means, especially since none of the 5 tasks seems to need this item. Do you maybe mean \"extended instruction\" as in \"more details about the task\"? How does this differ from \"task instruction\" and \"extra description\"?\n\nB: I am having trouble reading Table 2, especially all the empty cells: For the base LLM models, there are no results reported for SI, why is that? Why are there no results for adding speaker and adding speaker and addressee information?\n\nC: The prompt does not contain any explanation of the input format (e.g. it does not explain what the #1, #2 etc stands for or what the variables like rsp mean). Have you checked that the models can interpret the format correctly?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and insights.\n\n#### **Re** *Reason To Reject 1*:\n\n*\"The main reason to reject is that we do not know anything about the training data of the models that are tested here. It is unclear whether the evaluated datasets have been in the training data of ChatGPT or GPT-4 and therefore the results are unreliable and not generalizable.\"*\n\nYour apprehensions are warranted, yet we proffer the following perspectives. \n\n1. Most datasets have been accessed by LLMs in pretraining stage.\n2. Most of the existing research works[1-2] on LLMs performance do not consider this problem. \n3. Since LLMs has seen too much data, the probability of public datasets being accessed is large. However, even though the LLMs pre-training process has access to this data, there are no supervised labels to train the task on, and the task is considered unseen.\n4. We test on multiple MPC datasets to reduce the bias introduced by a single dataset. \n\n#### **Re** *Reason To Reject 2*:\n\n*\"There is no in-depth error analysis of failure cases. It would be interesting to see whether errors are systematic and how they differ from the failure and success cases of the supervised models.\"*\n\n1. I appreciate your interest in a more in-depth error analysis of failure cases in our study. \n\n2. While we value the feedback, I would like to provide a rebuttal by explaining our rationale for not including such an analysis in the current study. Our study was primarily focused on evaluating the performance of the LLMs in solving MPC problems. Due to the scope and focus of our study, we believe that such an analysis would be more suitable as a separate research endeavor. We hope that our study provides a valuable contribution to the field and encourages further research in this area, including more comprehensive error analyses in future investigations.\n\n#### **Re** *Reason To Reject 3*:\n\n*\"The framing that the title suggests (\"to solve multi-party conversation\") is misleading. ...\"*\n\nWe adopt the above statement due to the following considerations.\n\n1. The difficulty with MPC is that the stream of conversations has complex relationships, and determining who speaks (Speaker) and speaker to whom (Addressee) is key to recognizing the relationships.\n2. In terms of tasks coverage, we study two major parts including MPC understand and MPC response generation. For the MPC understanding task, we refer to some papers[3-4] and select representative ED, AR, SI, RS as evaluation. Therefore, we believe that our evaluation is sufficient to cover most of the problems of MPC.\n\n*\"The latter two tasks give the impression of the LMs actively ...\"*\n\nWe evaluated the performance of response generation using metrics that are common in response generation tasks, which are also used in papers on MPC.\n\n#### **Re** *Question 1:*\n\n*\"In Figure 2 and in Section 3.1, you mention \"extend instruction\" and I do not understand what that means, especially since none of the 5 tasks seems to need this item. Do you maybe mean \"extended instruction\" as in \"more details about the task\"? How does this differ from \"task instruction\" and \"extra description\"?\"*\n\nSorry for the unclear presentations. \"extend instruction\" is used for incorporating additional information into MPCs. For example, as shown in Table 5, the paragraph “Please give a response on behalf of Speaker …” is \"extend instruction\".\n\n#### **Re** *Question 2:*\n\n*\"I am having trouble reading Table 2, especially all the empty cells: For the base LLM models, there are no results reported for SI, why is that? Why are there no results for adding speaker and adding speaker and addressee information?\"*\n\nThank you for your suggestion, we will add detailed notes to Table 2 in our revision. Here's why empty cells exist.\n\n1. For the speaker identification task, we need to determine which speaker has spoken the current sentence from the history of speakers and what they have said. This means that we cannot lack SI information, so these cells are empty. \n2. For EmoryNLP and MELD datasets, they did not provide the Addressee information, therefore *w /. Addressee* and *w /. Speaker & Addressee* two rows are empty. \n3. For the Ubuntu IRC AR task, we need to estimate the Addressee, so we can't add the Addressee information, resulting in the cells with Addressee information are empty.\n\n#### **Re** *Question 3:*\n\n*\"The prompt does not contain any explanation of the input format (e.g. it does not explain what the #1, #2 etc stands for or what the variables like rsp mean). Have you checked that the models can interpret the format correctly?\"*\n\nThe models are able to understand these special symbols. For example, in the ED task of MELD dataset, we have an output \"#0 -- Hi! // joy\\n#1 -- Hi. // neutral\". We will add more cases to the appendix of the revised version to show the raw output of the model.\n\n#### **Re** *Typos 1:*\n\n*\"Section 4 \"Speaker Information Enhancement\": I do not understand this syntax \"GPT-3.5-Turbo (GPT-4)\". There are several instances of these parentheses, it is unclear to me what this means.\"*\n\nIt means that both the GPT-3.5-Turbo and GPT-4 have the same conclusions.\n\n#### **Re** *Typos 2:*\n\n*\"The actual tasks are not mentioned until Section 3 ...\"*\n\nThank you for your advice. We will add them in Section Introduction in the revision.\n\n#### **Re** *Typos 3:*\n\n*\"Table 3: Please spell out the abbreviated tasks in the caption for easier mapping. Also please mention what exactly SOTA refers to (i.e. what paper).\"*\n\nWe explained all the SOTA in Section 4.2 Baselines. Thank you for your advice. Different tasks have different SOTA methods, and we will add instructions in Table 2 to make it clearer.\n\n#### **Re** *Typos 4:*\n\n*\"Section 4.1, EmoryNLP: \"iis\" -> \"is\", MELD: \"enrichadding to the richness\"\"*\n\nThank you for your correction.\n\n#### **Re** *Typos 5:*\n\n*\"Table 2: It is confusing that the third column for Ubuntu doesn't match the other two (instead ED it reports AR), please highlight this in the caption or the header. Also please spell out what SOTA refers to, e.g. in the caption.\"*\n\nIt depends on the dataset. EmoryNLP and MELD datasets provide emotion labels and can therefore be used to do ED tasks. The opposite Ubuntu IRC dataset provides Addressee tags and can therefore be used to do AR tasks.\n\n#### **Re** *Typos 6:*\n\n*\"I would find it easier if the settings had some kind of special syntax...\"*\n\nThank you for your advice. We will modify them in the revision.\n\n#### **Re** *Typos 7:*\n\n*\"In the related work section you mention a number of LLMs but I did not understand why you picked only GPT-4 and ChatGPT to test.\"*\n\n1. ChatGPT and GPT-4 are recognized representatives of LLMs, so we only consider them to evaluate solving MPC problems. \n2. For a reference, we provide the performance of Vicuna-1.5 (*w/. Speaker*) which is finetuned from Llama-2 on the MELD dataset as below. We found that it's much more difficult to control the output for Vicuna comparing with ChatGPT.\n\n|                                | MELD    |          |            |             |\n| ------------------------------ | ------- | -------- | ---------- | ----------- |\n| Vicuna-1.5-13b (*w/. Speaker*) | ED (F1) | SI (ACC) | RS (R10@1) | RG (METEOR) |\n| **Zero-Shot**                  | 54.89   | 42.79    | 11.29      | 12.43       |\n| **2-shots (Retrival)**         | 56.32   | 33.48    | 15.63      | 13.33       |\n| **2-shots (Random)**           | 55.26   | 25.58    | 25.08      | 11.82       |\n\n#### **Re** *Typos 8:*\n\n*\"I found \"addressee recognition\" to be confusing... I was initially confused about the first utterance not having an addressee ...\"*\n\n1. This expression is from MPC-BERT [4].\n2. In the Ubuntu scenario, since the first person posting doesn't target a specific person, it can be considered to not have an addressee.\n\n#### **References**\n\n[1] Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent. Sun et al. Arxiv 2023.\n\n[2] Is ChatGPT a General-Purpose Natural Language Processing Task Solver? Qin et al. Arxiv 2023.\n\n[3] MELD: A multimodal multi-party dataset for emotion recognition in conversations. Poria et al. ACL 2019.\n\n[4] MPC-BERT: A Pre-Trained Language Model for Multi-Party Conversation Understanding. Gu et al. ACL 2021.\n\n[5] GSN: A Graph-Structured Network for Multi-Party Dialogues. Hu et al. IJCAI 2019.\n\n[6] HeterMPC: A Heterogeneous Graph Neural Network for Response Generation in Multi-Party Conversations. Gu et al. ACL 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The work tests an interesting ability of LMs – their ability to keep track of a multi-party conversation.", "reasons_to_reject": "- The main reason to reject is that we do not know anything about the training data of the models that are tested here. It is unclear whether the evaluated datasets have been in the training data of ChatGPT or GPT-4 and therefore the results are unreliable and not generalizable.\n\n- There is no in-depth error analysis of failure cases. It would be interesting to see whether errors are systematic and how they differ from the failure and success cases of the supervised models.\n\n- The framing that the title suggests (\"to solve multi-party conversation\") is misleading. All tasks aim at being able to follow a conversation (speaker and addressee identification), judge contributions (emotion recognition), or take the position of one of the participants (response selection and generation). The latter two tasks give the impression of the LMs actively participating in the conversation but the evaluation aims at their ability to reproduce the ground truth. (i.e. a generated contribution could be acceptable in a dialog even if counted as wrong here).\n\n**Update after rebuttal**\n\nI thank the authors for their comments and additional explanations and for providing answers to my questions. I have updated my scores. While I appreciate the methodology and chosen evaluation tasks as well as the decision to leave more in-depth analysis for later, my main concern remains the first reason to reject above.", "questions_to_authors": "A: In Figure 2 and in Section 3.1, you mention \"extend instruction\" and I do not understand what that means, especially since none of the 5 tasks seems to need this item. Do you maybe mean \"extended instruction\" as in \"more details about the task\"? How does this differ from \"task instruction\" and \"extra description\"?\n\nB: I am having trouble reading Table 2, especially all the empty cells: For the base LLM models, there are no results reported for SI, why is that? Why are there no results for adding speaker and adding speaker and addressee information?\n\nC: The prompt does not contain any explanation of the input format (e.g. it does not explain what the #1, #2 etc stands for or what the variables like rsp mean). Have you checked that the models can interpret the format correctly?", "paper_topic_and_main_contributions": "This paper investigates the performance of large language models on 3 multi-party dialog datasets for 5 tasks: emotion detection, addressee recognition, speaker identification, response selection, response generation. The models are put into a position of overhearing and queried for their ability to follow the conversation (speaker and addressee identification), judge contributions (emotion recognition), or take the position of one of the participants (response selection and generation).", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Section 4 \"Speaker Information Enhancement\": I do not understand this syntax \"GPT-3.5-Turbo (GPT-4)\". There are several instances of these parentheses, it is unclear to me what this means.\n- The actual tasks are not mentioned until Section 3 which made reading the related work section odd bcecause I didn't know what the target for the relation was. Please mention the tasks in the introduction (naming them is sufficient in the introduction).\n- Table 3: Please spell out the abbreviated tasks in the caption for easier mapping. Also please mention what exactly SOTA refers to (i.e. what paper).\n- Section 4.1, EmoryNLP: \"iis\" -> \"is\", MELD: \"enrichadding to the richness\"\n- Table 2: It is confusing that the third column for Ubuntu doesn't match the other two (instead ED it reports AR), please highlight this in the caption or the header. Also please spell out what SOTA refers to, e.g. in the caption.\n- I would find it easier if the settings had some kind of special syntax like using italics or subscript instead of regular text because the names get quite long and often span line breaks, e.g. in section \"MPC Structure Incorporation\" (\"GPT-3.5-Turbo (GPT-4) w/. Speaker\", \"GPT-3.5-Turbo w/. Addressee\", etc.)\n- In the related work section you mention a number of LLMs but I did not understand why you picked only GPT-4 and ChatGPT to test.\n- I found \"addressee recognition\" to be confusing: addressee in dialog can also be interpreted in the forward-looking sense. I was initially confused about the first utterance not having an addressee because surely the first utterance is addressed to someone. it is just not replying to anyone. this is better phrased differently"}}, {"review_id": "FUdXrjGesi", "reviewer": "Reviewer_2d2w", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and insights. \n\n#### **Re** *Reason To Reject 1*:\n\n*\"Although the main focus of this paper is on evaluation, it does not offer much in the way of technical novelty. It just designs the prompt given to the LLM.\"*\n\n1. Since ChatGPT is a closed source model, we only have access to its inputs as well as outputs. So what we can do is change its inputs (Prompts) to get better results.\n2. The main purpose of this paper is to explore whether the most powerful language models available can be good tools for solving MPC problems, which have not yet been explored. We believe that this paper can provide good inspiration for follow-up work. \n3. MPCs do not maintain sequential information flows since the third utterance can reply to the first one, which brings a challenge for autoregressive serialized LLMs, especially for ChatGPT, which is carefully designed for two-person conversations.\n4. As for how to improve the performance of LLMs in MPC, inspired by MPC--BERT [1], we utilize additional information including the speaker and the addressee. How these two types of information are utilized and how they affect the performance of LLMs in MPC tasks are also our contributions. In addition, we discuss the experimental results in detail. This discussion forms the basis for continued investigations into improving LLM effectiveness in MPC contexts.\n\n#### **Re** *Reason To Reject 2*:\n\n*\"Evaluations are conducted using closed models like ChatGPT and GPT-4 ...\"*\n\n1. Since NLP research is constantly evolving, the increasing performance of LLMs is obvious. Our evaluation is for the strongest LLMs of the day.\n\n2. We use *gpt-4-0314* and *gpt-3.5-turbo-0301*APIs provided by OpenAI. As stated in OpenAI docs, these models will not receive updates, and will be maintained to June 13th 2024 at the earliest, which means that we can get the consistency of results in about 1 year. \n\n#### **Re** *Typos*:\n\nThank you and we will correct them in the revision.\n\n#### **References**\n\n[1] MPC-BERT: A Pre-Trained Language Model for Multi-Party Conversation Understanding. Gu et al. ACL 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Since the use of LLMs in MPC is expected to expand in the near future, quantitatively evaluating their dialogue capabilities is important.\n- The current research appropriately compares these capabilities with existing supervised methods.", "reasons_to_reject": "- Although the main focus of this paper is on evaluation, it does not offer much in the way of technical novelty. It just designs the prompt given to the LLM.\n- Evaluations are conducted using closed models like ChatGPT and GPT-4, so even if the temperature is lowered, there's a possibility that the models themselves could be updated frequently, which means that the consistency of results cannot be guaranteed.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper investigates how well zero-shot methods using ChatGPT and GPT-4 can perform on multi-party conversation (MPC) tasks compared to supervised models.\nThe authors design prompts for zero-shot learning and evaluate tasks such as dialogue understanding and response generation. The authors also examine the effects of including speaker and addressee information in the prompt.\nThe corpora used are EmoryNLP, MELD, and Ubuntu IRC, which were not newly constructed for this research.\nThe contribution of this paper lies in examining the dialogue capabilities of recently introduced LLMs in situations such as MPC, which are thought to be more challenging than regular one-on-one conversations.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Section 4.1 \"iis\" -> \"is\"\nTable 3 \"Evaluation results of MPC Understanding.\" -> \"Evaluation results of MPC Generation.\""}, "extra_text": {}}]

Author rebuttal:

Meta-review: *Summary*: This work evaluates the zero-shot capabilities of ChatGPT and GPT-4 on multi-party conversation (MPC) tasks. Five tasks including emotion detection, accurately detecting the addresses in a conversation, speaker identification, response selection and generation have been explored. They find that including speaker and addressee information in the prompt yields performance improvements.

*Evaluation*:  R1 and R3 rated this work as medium to strong on soundness (3/4), while excitement ratings were lower (2/3). R2 rated this low on soundness (2) and stated two main concerns: A) Since training data of ChatGPT, GPT-4 is unknown, we cannot be sure there is no test leakage. B) No in-depth analysis of failure cases has been presented. Regarding A), this is a broader concern that holds for all closed model evaluations and not specific to this work. B) is indeed a valid concern. Given this is a first investigation of the capabilities of ChatGPT/GPT-4 on MPC tasks, demanding an in-depth analysis of the failure cases (along with anecdotal examples in an Appendix) is warranted and would have been useful for the reader.
Decision: Accept-Findings