Finding Common Ground: Annotating and Predicting Common Ground in Spoken Conversations
Authors: Magdalena Markowska
Mohammad Taghizadeh
Adil Soubki
Seyed Abolghasem Mirroshandel
Owen Rambow
Venue: EMNLP 2023 Findings (2023)
Keywords: Common Ground
Belief Extraction
Corpus Construction
Cognitive state
T5 Language Model
TL;DR: We introduce a new corpus for common ground in conversations, and describe initial experiments.

Abstract: When we communicate with other humans, we do not simply generate a sequence of words. Rather, we use our cognitive state (beliefs, desires, intentions) and our model of the audience’s cognitive state to create utterances that affect the audience’s cognitive state in the intended manner. An important part of cognitive state is the common ground, which is the content the speaker believes, and the speaker believes the audience believes, and so on. While much attention has been paid to common ground in cognitive science, there has not been much work in natural language processing. In this paper, we introduce a new annotation and corpus to capture common ground. We then describe some initial experiments extracting propositions from dialog and tracking their status in the common ground from the perspective of each speaker.

Finding Common Ground:
Annotating and Predicting Common Ground in Spoken Conversations
Magdalena Markowska 04, Mohammad Taghizadeh",, Adil Soubki
Seyed Abolghasem Mirroshandel Owen Rambow
Department of Computer Science & Department of Linguistics Institute for Advanced Computational Science Stony Brook University, Stony Brook, NY, USA University of Guilan, Rasht, Guilan, Iran markowska stonybrook.cdu
Corresponding Author:
Abstract
updates from the perspective of the two speakers in dialog.
When we communicate with other humans, do not simply generate a sequence of words. Rather, we use our cogmitive state (beliefs, desires, intentions) and model of the au- dience's cognitive state to create utterances that affect the audience's 's cognitive state in the intended manner. An important part of cognitive state is the common ground, which is the content the speaker believes, and the speaker believes the audience believes, and so on. While much attention has been paid to com- mon ground in cognitive science, there has been much work in natural language processing. this paper, we introduce new annotation and corpus capture common ground. We then describe some initial experiments extract- ing propositions from dialog and tracking their status in the common ground from the perspec- tive of each speaker.
The concept of common ground has been widely studied across various disciplines including linguis- tics and cognitive science (Kiparsky and Kiparsky, 1968; Karttunen, 1971a,b; Clark, 1996; Horton and Gerrig, 2016), computer science, artificial intelli- gence, and natural language processing (Grosz and Sidner, 1990; Cohen and Levesque, 1990; Traum, 1994; Del Tredici et al., 2022), and philosophy (Lewis. 1969; Stalnaker, 2002). CG can be de- scribed in simple terms as a set of beliefs which interlocutors believe they share, and which they be- lieve other discourse participants also believe they share. However, when people communicate, they do not only exchange their beliefs, but also other components of cognitive state, such desires, in- tentions, goals, etc. Consequently, the full defini- tion of CG should also contain sets of shared de- sires, intentions, and so on. Because this is the first corpus created for that purpose, we focus solely on the belief component of CG. We do acknowledge, however, that future work will need to extend the notion of CG.
1 Introduction
Expressions such as "finding common ground" or "establishing common ground" are familiar ev- cryone. They are often used with reference to two or more people trying to reach some sort of mutual understanding about various topics; for example, politicians during parliamentary meeting. Most often, however, common ground does not involve conscious effort. Establishing and updating com- mon ground (CG) between interlocutors is the key successful conversation. is therefore cru- cial to develop methods of CG representation and learning. This work focuses on modelling the con- cept of common ground from the cognitive per- spective. Specifically, we introduce the first (to the best of our knowledge) corpus annotated for common ground. The procedure developed for the annotation is designed such that reflects what we think happens when people update their common grounds. We further conduct preliminary machine learning experiments focused on extracting events subject to CG updates, and on predicting the CG
This paper is organized follows. We summa- rize the NLP literature in Section 2. We present our new corpus in Section In Sections 4, 5, and we discuss our baseline experiments for predict- ing the events, beliefs about events, and achieving common ground.
2 Related Work
The concepts of belief and factivity are closely re- lated, see (Prabhakaran et al., 2015, Section 2) for discussion. They have been annotated in multiple language corpora, such as LU (Diab al., 2009), FactBank (Saurí and Pustejovsky, 2009), UW (Lee et al., 2015), LDC CB (Prabhakaran al., 2015). MegaVeridicality (White et al., 2018), Commit- mentBank (de Marneffe et al., 2019), RP (Ross and Paylick, 2019), and BeSt (Tracey et al., 2022),

---

among others. While all of those datasets differ in terms of identification of events and the anno- tation procedure, they target one common goal. Given some text, they identify the level of commit- ment to the events expressed in the text, whether that be from the perspective of the author solely (e.g. LU, Mega MegaVeridicality, RP, UW) or from the author along with other mentioned sources (Fact- Bank, BeSt).
concept first discussed in Premack and Woodruff (1978), which refers to the capacity in humans (Valle et al., 2015), as well as chimpanzees and orangutans (Call and Tomasello, 1998), to infer information about others' mental states including beliefs, desires, and emotions. CG is an aspect of ToM, since beliefs in the CG are beliefs speaker models as being held by the addressee. This means that humans (as well as other primates) are able infer some "hidden" information about what other people think, believe, and perceive particular situations. A natural question for NLP is whether contemporary advanced language models exhibit knowledge of ToM. Furthermore, if not, what are some possible approaches that will allow machines to infer information about their conversational part- ner's 's cognitive states?
What is considered an event usually tailored to the specific question that a corpus is address- ing. For example, CB targets solely final clausal complements as it examines the role of the ma- trix predicate in determining the speaker's 's belief, while LU and LDC CB consider all full lexical verbs in a sentence, and FactBank and BeSt tar- get verbs and nouns referring events and states, The genre of the data is also correlated with the task. FactBack uses WSJ newswire, and as a result the author is limited in the expressiveness of their cognitive state, as news-writing requires following certain standards. RP, LU, and CB, on the other hand, use mixed genres, including discussion fo- rums and blogs, which allow for more creative and expressive freedom. All of these corpora also ex- hibit various scales identifying the level of belief. Those are either represented numerically, averaging over annotators' judgements, on a [-3,3] scale (UW, CB, RP), or with categorical labels (LU, FactBank, LDC CB, BcSt). One last crucial factor affecting the outcome of annotation are the annotators them- selves. RP and UW use crowdsourced annotators, while FactBank, LU, LCD CB, and BeSt opt for trained annotators.
Kosinski (2023) tested current large language models (LLMs) by having them complete narra- tives inspired by the Sally-Anne (Wimmer and Perner, 1983; Baron-Cohen et al., 1985) and Smar- ties (Perner et al., 1987) tests, both of which were originally designed to determine if children are aware of the cognitive state of others. He focused specifically on one subcomponent of the ToM scale (Wellman and Liu, 2004), namely false belief. The results found that the most advanced GPT models correctly complete both true and false beliefs for characters in various scenarios. For example, if there is a bag of popcorn with a "chocolate" label on it, the models accurately predict that when Sam finds the bag she will believe that the bag is full of chocolate. Kosinksi then concludes that this is either (1) evidence of the inadequacy of these tests for evaluating ToM or (2) evidence of ToM emerg- ing as byproduct of learning on different tasks. Ullman (2023) disagrees with this hypothesis and states that it is indeed possible that ToM tests are in fact accurate and useful for studying human cog- nition but not LLMs. His claim is backed by a series of ToM experiments, slightly different from the ones in Kosinski (2023), that show that LLMs fail to make correct false-belief predictions. For bag scenario
Belief/factivity prediction tasks have attracted a lot of attention in the past two decades. In the early 2000s, rule-based systems were used to detect au- thors' beliefs (Nair et al., 2006). Diab et al. (2009), Prabhakaran al. (2010), and Lee et al. (2015) use SVMs together with dependency trees and lexical based features. Newer, neural network approaches include bidirectional LSTMs for both single and multi-task setups (Rudinger et al., 2018). Pouran Ben Veyseh et al. (2019) use BERT representations with graph convolutional neural networks. Jiang and de Marneffe (2021) use fine-tuned BERT for belief detection on various corpora and closely ex- amine shortcomings of the model. Murzaku et al. (2022) show that combination of specific corpora can improve the model's 's performance.
example, in the popeorn-chocokee they added additional information stating that Sam cannot read. GPT 3.5 still guessed with 98% confi- dence that Sam believes the bag is full of chocolate. Similarly, Sileo and Lernould (2023) develop a dataset of "mind games" using dynamic epistemic logic and target aspects of ToM to show that con- temporary LLMs fail to make correct inferences in
CG is closely related to Theory of Mind (ToM),

---

other scenarios as well.
Example 1: A: I thought was going to get see everybody.
The experiments conducted by Kosinski (2023),
Sileo and Lernould (2023), and Ullman (2023)
e1: thought was going to get to see everybody.
show that while LLMs can produce accurate com- pletions for false-belief in some settings, the results are highly sensitive perturbation and far from ro- bust. This its own should be sufficient evidence that ToM cannot possibly be an emergent property of L.LM learning, at least not yet, and points to a need for higher quality, naturalistic data on how humans reason about aspects of ToM, like CG.
e2: was going to get to see everybody.
ey A got to see everybody.
There two cases where create events that are not explicitly mentioned in speaker's utter- ance. First, we form speech act events signal question asking or giving orders. Second, if one speaker elliptically negates an event, or gives a negative answer to an event under question, we cre- ate fully fledged event out of it. Both cases are illustrated in the first three columns of Table 1.
3 The CG Corpus
The Common Ground Corpus is annotated on the top of the LDC CALLHOME American Speech corpus (Canavan et al., 1997), which consists of collections of 120 unscripted dialogs between close friends or family members. The dialogs are avail- able in both written form and audio. Since our dataset is the first attempt at annotating CG in discourse, we chose start with conversations be- tween two people. Another crucial aspect of the CALLHOME corpus is that the interlocutors know each other very well, which allows us to assume that their CG set is non-emply at the start of the conversation. Finally, the speakers were allowed to speak freely about any topic, which allows us to capture the speakers' cognitive states in natural settings.
3.2 Belief and CG annotation
As was discussed in Section 1, we limit the defi- nition of CG to beliefs in this paper. In order to infer the interlocutors CG, we first identify their beliefs towards the formulated events. We follow the belief types proposed in FactBank (Saurí and Pustejovsky, 2009). Since the main goal of our corpus to represent the CG updates, as opposed to the granularity of speakers' beliefs, we simplify the types and limit them to different levels of be- lief. An event e will be marked: CT+: if speaker certainly believes that e; CT-: if speaker certainly believes that not PS: if speaker possibly be- lieves that e; NB: if speaker expresses no belief about c.
In dialog, each of the interlocutors form their own cognitive states, which includes model of other speakers' cognitive state. Since CG forms a substantial component of a speaker's cognitive state, we model CG for each speaker separately: the CG is not shared. In the majority of cases, the CGs of the two interlocutors converge. However, we also scenarios in which the CGs for speakers diverge and in which we observe miscommunica- tions.
Once belief values are established for speakers A and B, the annotators determine the CG for each speaker. We differentiate between three distinct updates:
JA: an event e is mutually believed by both in- terlocutors and is added to CG in the moment e was uttered. The level of belief in the CG (which may differ from individual beliefs) is left implicit, since it is always the less certain degree of the two interlocutor's beliefs.
The two main parts of the annotation procedure are event extraction, and belief and CG annotation.
IN: an event e has already been a part of the inter- locutors' CGs before the time of the event. In other words, it must be true that at some point in the past, CG(A)=JA and CG(B)=JA for e.
3.1 Event extraction
CALLHOME dialogs are divided into utterances. The events are formed from the main predicates in each clause forming the utterance. To make the events as informative as possible, we resolve any pronominal anaphors, i.c. we spell out any refer- ents of pronouns, including first and second person (sometimes resulting in intentionally unnatural sen- tences). Here is an example from the corpus.
RT: an event that has been presented by speaker has been entertained but rejected by the addressee. In order to reflect the dynamic nature of a dialog, we must allow changes to the interlocutors' beliefs and, consequently, their CGs. Our annotation inter- face allows to conduct such updates and register the history. Table shows how beliefs can change. Our annotation procedure requires looking

---

Nb
Utterance
edd
Event
Bel(A)
BelßB
CG(A)
CG(B)
So you've leading life Reilyy
usks B if B has been lending the life of Reily
C
CT+a
has been leading the life of Reilly
es
CT-ei
2
B: No. Not really.
has not been leading the life of Reilly
CT ey
RT
RT es
CT+
CT+
eg
eA
Table 1: An annotation sample.
ahead. An annotator, overhearer of the con- versation, needs to have access to the interlocutor's responses to be able determine the interlocutor's cognitive states. This most evident in the case of questions. the example I, order to de- termine whether B believes e2, the annotator must look to the next utterance to see that B's attitude towards is in fact CT-, already at the time of the question utterance (though A is not yet aware of B's belief towards e2).
3.3 Inter-Annotator Agreement
Computing agreement for the event extraction task challenging, because we need to penalize an an- notation if the number of annotated events differs between annotators, We call this method EMBERT. Given similarity measure f(-) which takes two events (strings) and returns score [0, 1], we compute event-matchen score between the set of events extracted by an annotator taken to be the reference E, and the set of events extracted by an annotator for comparison Eg. We compute. among the possible mappings between E, and Ee, the mapping which maximizes the mean pairwise similarity as produced by f(-). In the event that E-| A |Ee any unmapped event from the annota- tor being compared receives score of zero. Simi- larly, the annotator being compared receives zero for each reference event which cannot be mapped. This penalty imparts high emphasis on annotators first finding the same number of events from each utterance and then producing events which have similar semantic meaning. The mean similarity for the maximal mapping is returned to produce an event-matched score. EMBERT uses cosine sim- ilarity between SBERT (Reimers and Gurevych, 2019) embeddings as the similarity measure f()). We report EMBERT among the four annotators for event extraction in Table Scores for EM- BERT exceed 0.7 among all pairs of annotators, with the exception of Annotator 2. We use these studies to decide which annotators to retain.
The event that is being questioned is e2. Be- lief of a speaker asking about e2 is determined by the structure of the question. In the example, A chooses to inquire about e2 by using affirmative sentence structure and rising intonation. We iden- tify expressing possible belief towards e2. To determine the addressce's (B's) belief status, an an- notator needs to continue reading the conversation until they see B's response in the second utterance.
In Example 2, B negates 62- This is marked as Bel(B)=CT- at the level of the event because B always believed certainly not e2. Speaker A, on the other hand, needs wait for B's response to potentially update their belief about e2. A's belief about e2 (Bel(A)=CT-) is recorded the time of the second utterance. Now that both interlocutors' beliefs about e2 are established, an annotator can deduce that this event will be rejected from CG from the perspectives of both A and B.
An obvious question to ask would be why one cannot determine the status of B's CG at the time of utterance since both Bel(A) and Bel(B) are avail- able. This is because establishing common ground is an iterative process, and it is not sufficient for B to have access to B's beliefs and A's beliefs about event. B also needs to believe that A believes that B believes e2 and the other way around, at minimum. At the stage of e2, in fact certain that A does not know that and needs to wait for B's response.
We evaluate inter-annotator agreement among the four annotators on the belief and CG annota- tions for both speakers using a pairwise Cohen's 's kappa (Cohen, 1960). We use gold events for this study. The mean of these results across the four tasks are reported in Table 3. According to Co- hen, anything above 0.6 indicates substantial agree- ment, while values above 0.8 are considered "al- most perfect agreement". We also computed com- puted Fleiss' kappa. which was 0.70 across all tasks (Fleiss, 1971). Based on the interpretation guide- lines provided Landis and Koch (1977) these numbers also indicate substantial agreement among the four annotators.
Finally, with B's negative response A's ques- tion, do not only update the speaker belief about 62 but we also create negated version of 02 that will then enter CG in both interlocutors' mind un-
less it is further explicitly denied.

---

Anno. 2
Anno. 3
Anno. 4
Annotation Type
Train
Test
Anno. 1
0.56
0.73
0.79
Utterance
415
Anno. 2
0.60
0.61
Event
970
325
Anno. 3
0.84
CT+
1,576
523
CT-
107
71
Bel(A) + Bel(B)
Table EMBERT antertannttato agreement scores for
PS
150
event extraction.
NB
81
8
26
14
Anno.
Anno. 3
Anno. 4
JA
1,540
490
CG(A) CG(B)
IN
124
58
Anno. 0.58
0.76
0.77
Anno.
0.63
0.60
RT
115
73
Anno.
0.87
161
29
Table 3: Mean pairwise Cohen's kappa for Belief and CG judgments.
Table 4:
The distribution of different annotation types
in the annotated dataset.
4 Experiments: Events
presented in Table 5, using the EMBERT measure introduced in Section 3.3.
For generating events, use the base version of FLAN-T5, which is an instruction-fine finetuned version the T5 LLM (Chung et al., 2022). has demonstrated impressive few-shot performance across various tasks, outperforming even larger models. In our experiments, the model receives utterances input and it generates the correspond- ing event(s) for each utterance. We have fine-tuned FLAN-T5 the training set, which is a small dataset (see Table 4), and evaluated the model on the test set.
Models
EMBERT
FLAN-T5 No Context Event Generation
45.94
PLAN-TS Speaker-Basel Window (preceding)
48.65
FLAN-TS Fixed Window Size (preceding)
48.51
FLAN-TS Fixed Window Size (preceding)
48.69
Table 5:
Experimental results of Event Generation.
As it is shown in Table 5, we have examined FLAN-T5 with four different input formats: 1) No context: the model only receives the target utter- ance and generates its event(s), 2) Speaker-based window: in addition to the target utterance, the model also receives speaker- ppeaker-based utterances, 3) Fixed window size (preceding): in addition the target utterance, the model also receives the 2 preceding utterances, and 4) Fixed window size 4 (preceding): in addition to the target utterance, the model also receives the 4 preceding utterances. As can be seen, the model using four preceding utter- ances context performed best by both metrics. We have explored alternative combinations, such using the following context. However, these com- binations yielded unsatisfactory results, which is why they are not reported.
Alongside the input utterance, the model can also receive contextual information as input. In this paper, the contextual information is provided the following modes: 1) Fixed window: A pre- determined number of utterances preceding and/or following the target utterances events will be included as input for the model. And 2) Speaker- based window: The model will receive all pre- ceding and/or following utterances until encoun- ters an utterance from speaker different from the speaker in the target utterance or event. The input format of the model is as follows:
"Preceding :ontext": (Preceding Utterances)
"Events": (Target Utterance)
"Following Context": (Following Utterances)
4.2 Error analysis
4.1 Experimental Setup
We performed preliminary error analysis on the first 60 utterances from the test sets in three condi- tions: without context, and with the context win- dow of size and four. We identified seven types of errors. Senseless errors are those which make no reference the actual utterance. For instance, for the gold event Jill's 's' situation sounds like a real mess, the system generated The baby's 's birth mom 's
This version of the dataset consists of four dialogs. To assess the performance of our model, we se- lected three dialogs as the training set, while the remaining dialog
designated the test set. The distribution of all annotation types available in the training and test partitions are shown Table 4.
The experimental results for event generation are

---

mom mom's 's mom J. If an event was match- ing some part of what was said but still failed to provide sufficient information, was marked Intel- ligible. Anaphora resolution error combines unre- solved pronoun references and wrongly identified references, e.g. proper names in the test sets were mistaken with the ones from training sets. Uninfor- mative emrors were the results of arbitrary division of utterances per speaker elided information in speakers' turns, e.g. B says same car instead of has the same car. The remaining types are Missing events, i.c not generated by the system, events wrongly generated by the system (Gen), and Gold errors.
5.1 Data Augmentation
To mitigate the issue of data imbalance, particu- larly for the CT-, PS, and NB classes, we use a data augmentation technique. Given FLAN-T5's ability to handle multiple languages, opt for machine translation-based data augmentation ap- proach. For the training set, we employ the FLAN- '5 model to translate events associated with the minority classes. These newly translated events are subsequently incorporated into the training set, fol- lowed by additional model fine-tuning. We utilize French, German, and Spanish translations of the events in this process.
Table 6 presents counts of each error type and the total number of errors given particular con- text size. As expected, the more context our sys- tem receives, the better performs. The number of missing events does not improve with context. This is somewhat expected given that most missed events were caused by some failure of pragmatic inference. Anaphora resolution errors are signifi- cantly reduced given more context, which was also expected behavior.
5.2 Experimental Setup
The experimental results for belief classification are shown in Table 7. Measures of classification performance are computed for both Bel(A) and Bel(B) and this table reports the average. The uti- lized metrics include Fl for each possible label and overall accuracy. As the label distribution is highly imbalanced, we also report macro FI.
In Table 7, we have evaluated systems using BERT, FLAN-T5, and augmented FLAN-T5 with different contextual information. Except for one model (i.e., "FLAN-T5 Fixed Window (2 preced- ing 2 following)"), all models utilize preceding fixed or speaker-based windows with different sizes and the events of corresponding utterances are fed to the model as context. In cases where the con- text is provided the form of "Utterance Context". the model receives the context as raw utterances without the inclusion of events.
5 Experiments: Beliefs
For belief classification, we designed compre- hensive set of models utilizing BERT-base-casee (Devlin et al., 2019) and FLAN-T5. The system is trained to annotate beliefs for each speaker (c.g. Bel(A)=CT+) given a gold target event and any additional context. As with event extraction, we experiment with both fixed and speaker- win- dowing methods and additionally investigate in- cluding forward context. In these experiments, ut- terances are presented the model either as raw sentences or as the corresponding events of those utterances.
The results reported in Table 7 highlight several important findings. (1) The problem at hand proves to be challenging due the small number of exam- ples for the minority classes, resulting in low macro FI values. (2) terms of macro F1, the FLAN- T5 models generally outperform the BERT-based models. In terms of accuracy, however, the best result is achieved by the speaker-based window us- ing BERT. This is because BERT does particularly well with the very high frequency category CT+, and less well with the other categories.
We fine-tune BERT and FLAN-TS on the train- ing set using the following input format:
"Preceding Context": (Preceding Events Utterances)
"Target Event": (Target Event)
"Following Context": (Following Events Utterances)
(3) Interest- ingly, the table reveals that the best results of fixed
Input
Senseless
Intelligible
Missing
Anaphora
Uninformative
Gold
Gen:
Total
No context
15
34
12
112
Context
12
7
40
32
5
108
Context
11
2
36
27
10
5
92
Table 6: Error analysis; counts per error category shown by context size of system

---

Bel(A.B) AVG
Models
CT+ F1
CT- F1
PS FI
NB F1
Macro F1
Accuracy
BERTI Fixed Window Size
89.67
14.83
32.33
12.67
37.38
79.17
BERT2 Fixed Window Size
90.00
17.33
29.83
37.17
79.50
BERT3 Fixed Window Size
89.50
16.00
22.50
0.00
32,00
79.00
BERT3 Speaker--used Window
91.50
23.33
19.50
9.17
35.88
81.83
FLAN-T5 T5 Fixed Window Size (Utterance Context)
87.00
20.25
20.25
10.50
34.50
76.75
FLAN-T5 T5 Speaker-based Window (Utterance Context)
86,
11.25
18.67
5.50
30.40
73.75
FLAN-T5 Fixed Window Size
87,00
35,00
28.50
13.33
40.96
74.67
FLAN-T5 Fixed Window Size
86.83
34.67
28.67
17.50
41.92
74.67
FLAN-T5 Fixed Window Size
85,50
27.00
26.00
20.00
39.63
72.00
FLAN-T5 Fixed Window Size
87.50
27.50
25.50
11.00
37.88
74.00
FLAN-T5 Fixed Window (2 preceding following)
87.75
27.50
30.75
12.75
39.69
76.25
FLAN-T5 Speaker-hased Window
87.67
32.67
30.67
17.83
42.21
76.33
Augmented
FLAN-T5 T5 Fixed Window Size (French)
87.50
26.50
26.50
16.17
39.17
75.00
FLAN-TS Fixed Window Size FrrnnhhGGemann
88.33
34.00
32.67
43.50
77.83
FLAN-T5 Fixed Window Size (French-Gente BeenchhGerman-ee SSaamnsh
88.50
32.00
31.75
8.25
40.13
77.00
FLAN-TS Speaker-based Window (F'rench)
87,25
24.25
28.50
19.25
39.81
74.25
FLAN-T5 Speaker-based Window ('rench-German))
88,50
15.67
26.50
8.50
34.79
75.83
FLAN-T5 Speaker-based Window
88.67
32.33
28.67
11.83
40.38
76.00
Table 7: Experimental results of Belief Classification
window based contexts are achieved when utilizing a window size of 2. Morcover, increasing the win- dow size negatively impacts the macro Fl scores. Also, using the following context does not help the model. (4) Generally, the speaker-based window contexts exhibit superior performance compared the fixed window approach and the utterance context approach. (5) The results also demonstrate the effectiveness of data augmentation, particularly when using French and German for augmentation. The augmented data contributes to achieving the best macro-averaged Fl results because of the mi- nority classes.
both speakers, when in fact this event is a presup- position that has already been part of the speakers' common ground, and should therefore be annotated as CT+. Jokes, sarcasm and hypotheticals form an- other error type that we named hard inferences (HI). There were also cases of wrong predictions that could not be linguistically motivated. Those form fourth group called other. Finally, there were also few gold errors that were the result of data formatting. Table presents counts of errors within each type.
KW
Question
HI
Other
Gold
38
21
12
11
10
5.3 Error analysis
Table 8: Error types in the belief prediction task
We conducted error analysis of FLAN-TS in the Speaker-based window condition. There were 92 errors in belief prediction, given the gold events. We differentiated between five types of errors. The most common errors were observed in events em- bedded under matrix predicates, such as say, tell, hope, think, or conjunctions such as before or be- cause. For example, an event A sons never treat one another like and B' mom and dad treat A em- bedded under hope (A hopes that e), had predicted CT+ values for both speakers, while it should have been marked only possible belief (PS). We call those type of errors key words (KW). Another type of event that was problematic were events em- bedded in questions (Question). For example, the event sleeps which is a part of a speech act A asks B when B sleeps has predicted values PS for
Experiments: Common Ground
We implemented two strategies to predict common ground. The first strategy based on heuristics and the second strategy is based on fine-tuning of the FLAN-T5 model.
Heuristic based approach In this strategy, we have utilized the following rules. We always update CG for both speakers with these simple heuristics.
1) If Bel(A) CT- Bel(B) CT-, then CG RT.
2) If Bel(A) CT+ and Bel(B) CT+,
then CG = JA or CG= IN.
3) Bel(A) = PS and Bel(B) = CT+,
then CG = JA(SS) or CG=

---

CG(A,B) AVG
Models
JA FI
IN F1
RT FI
Macro Fl
Accuracy
FLAN-T5 (No Livent No Context)
94.50
0.00
99.50
64.67
90.50
FLAN-TS Fixed Window (No Context)
94.50
33.75
99.50
75.92
91.00
FLAN-T5 TS Fixed Window 2
95.00
50.17
99.50
81.56
91.83
FLAN-TS Fixed Window
48.83
99.50
80.89
90.67
FLAN-TS Fixed Window 8
95.00
54.00
99.50
82.83
91.83
FLAN-TS T5 Fixed Window (2previous-2next)
92.50
41.00
99.50
77.67
88.50
Table 9: Experimental results of FLAN-TS based Common Ground classification utilizing Gold belief.
CG(A,B) AVG
Models
JA F1
IN F1
RT F1
Macro F1
Accuracy
FLAN-T' Fixed Window 0 (No Context)
88.00
28.25
0.00
38.75
78.00
FLAN-T5 TS Fixed Window 2
88.67
55.83
15.83
53.44
79.17
FLAN-TS Fixed Window
87.00
42.75
23.00
50.92
76.25
FLAN-TS Fixed Window 8
87.00
46.00
21.50
51.50
76.50
FLAN-TS TS Fixed Window (2previous-2next)
87.00
38.50
27.25
50.92
76.50
Table 10: Experimental results of FLAN-TS based Common Ground classification without utilizing Gold belief.
4) If Bel(A) CT+ and Bel(B) PS,
Threshold
CG(A,B) AYG
JA F1
IN F1
RT F1
Macro
Accurncy
then CG JA(SS) CG
98.59
62.70
5) If Bel(A) NB or Bel(B) NB, then NULL.
0.4
70.00
19.05
98.59
62.55
61.17
0.6
73.48
20.59
98.59
64.22
64.72
Rules 2-4 under-determine whether the belief al- ready in the CG or newly added. In this context, the crucial task is to determine whether the target event had already been present in the common ground of the speakers (i.e., CG IN) not (i.e, CG JA).
85.03
20.93
98.59
77.67
67.98
0.92
93.10
15.00
98.59
68.90
88.67
0.95
93.39
0.00
98.59
63,99
89.00
94.41
90.61
Table 11:
Experimental results of heuristic based Com- mon Ground classification utilizing Gold belief.
To address this problem, we first create a data structure called "dialog memory". which combines each event, its related (gold) beliefs, and extracted common grounds. When processing the "target event", all preceding events, beliefs, and common grounds are presented in the dialog memory. Then, within the dialog memory, the heuristic algorithm searches for the "target event" based the SBERT similarity measure. For similarities more than threshold, the "target event" will be regarded as an event that is already in common ground. Note that in this approach, the heuristic utilizes gold annotated beliefs.
FLAN-T5 model operates in two distinct modes. The first mode incorporates gold annotated beliefs, while the second mode solely focuses the events without utilizing belief information. In either ap- proach uses gold events.
6.1 Experimental Setup
The experimental results for CG classification us- ing heuristics approach, based on different SBERT thresholds, are shown in Table 11. The experimen- tal results for CG classification using FLAN-T5 with and without gold belief information shown in Tables 9 and 10, respectively. In these tables, different contextual information has been studied. Furthermore, as demonstrated in Table we con- ducted experiments on CG classification by exclu- sively utilizing gold belief information, excluding event and context information (i.e., "No Event, No Context" case). Additionally, we explored another scenario where the model operates without access context information (i.e., "Fixed Window 0 (No Context)" case).
Common Ground classification with FLAN-TS model In the next approach, we used the FLAN- T5 model to classify common ground. We fine- tuned FLAN-T5 on the training set. The model takes the following input prompt format:
["Bel (A)": [Gold Bel())) "Bel(B)": (Gold Bel (B))]
"Input Event with Context:"
"Preceding Context": (Preceding events)
"Target Event": (Target Event)
"Following Context": (Following events)
this approach, is important to note that the
The results presented Table 11 indicate the

---

inherent difficulty in designing a heuristic even with the inclusion of gold belicfs, as they show relatively low performance across the evaluated metrics. Comparing the performance of the FLAN- T5 model with gold beliefs and the heuristic, is evident that the FLAN-T5 model outperforms the heuristic in all metrics. This highlights the superi- ority of the FLAN-TS model when leveraging gold beliefs for CG classification.
also intend experiment with a multi-task learning (MTL) setup, where we jointl model belief and common ground updates.
Graph Approaches Our data can naturally be rep- resented as a graph structure where speaker A and 's utterances are nodes, and the changes in belief and common ground being edge labels. We intend to leverage graph neural network approaches in- spired by neural architecture and general insights from Zhang al. (2023), who model the emotion recognition in conversation (ERC) task.
Interestingly, when considering the FLAN-TS model with gold beliefs, increasing the context window size yields better results. However, this trend is not observed in the case of the FLAN- T5 model without gold beliefs. Furthermore, it is worth noting that following utterances do not have positive impact context generation classification in both versions of the FLAN-T5 model.
Higher-Ordee Order Beliefs We explicitly model first- order beliefs and the CG. Many higher-ordet beliefs follow from the combination of individual belief and CG for example, if A says that Mary is com- ing to dinner but B rejects it, it does not enter A's model of the CG. As result, we can infer higher- order belief, namely that A believes B does not believe the event (since if believed it, it would be in the CG). However, there may be cases in which higher-order beliefs cannot be inferred from the CG. We intend to annotate and model higher-order beliefs in dialog, i.e. beliefs held about other peo- ple's 's belief which are not derivable from CG. Error Analysis Finally, we intend to conduct more detailed error analysis that could give us more insight to specific issues that need to be addressed in future work.
As anticipated, the task of CG classification with- out gold beliefs proves challenging due to class imbalance.
6.2 Error analysis
Error analysis for CG prediction was conducted on FLAN-T5 with window size of 8. Out of 29 mistakes (311 events in total). 27 were of one kind, namely IN was mistaken for JA and the other way around (13:14 14 ratio). Such errors are expected as the IN condition requires either keeping track of events that have entered CG during the span of the conversation, or recognizing certain lexical items that indicate that those event have been already part of CG. The remaining two errors involved cases of speaker mishearing the other and CG updates that were ignored for the moment.
A Acknowledgments
Markowska's, Rambow's, and Soubki's contri- butions to this work were supported in part by funding from the Defense Advanced Research Projects Agency (DARPA): Markowska and Ram- bow under No. HR001 120C0037 and PR No. HR001 1 154158, and Rambow and Soubki under No. HR001 02:001122C02 122C0034 (CCU). The views, opin- ions and/or findings expressed are those of the au- thor and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government. Mirroshandel participated in this research while a visitor to the Institute for Advanced Computational Science at Stony Brook University, and gratefully acknowl- edges its support. We thank: Lee Cohen, Alana Gill, Lawrence Ma, Erica Solis, the annotators for their help in creating this corpus; Ying Ou for creat- ing the interface; Susan Brennan and John Murzaku for useful suggestions and encouragement; and four anonymous reviewers for useful and constructive feedback on the submission.
7 Future work
There are many obvious extensions of our work to date. We list few.
Audio Modality In our modeling, we only use the CALLHOME transcripts. However, intonation English is related to what the speaker believes the hearers already know. We will incorporate the audio signal into future CG predictions,and will explore multimodal neural architectures.
Existing Belief Corpora and Systems We intend to incorporate existing belief corpora and systems into the belief prediction task, which could im- prove the system performance. Specificallyy, we intend use the state-of-the-art FactBank-only be- lief system from Murzaku et al. (2023) as a belief' tagger, and give those outputs to our system. We

---

Limitations
mind"? Cognition, 21(1):37-46.
Since this work introduces a novel language corpus and provides base line system results, we acknowl- edge several limitations. First, our event formu- lation procedure does not account for every event that can actually be inferred. For example, if A said Now we have a king in England, Charles III, the only event that would be recognized is Charles III is now the king of England. Other inferences such as presupposition that Charles' predecessor was not a king will be omitted. Furthermore, we limited anaphora resolutions to personal pronouns only, disregarding other types such as temporal or event anaphora.
Josep Call and Michael Tomasello, 1998. Distinguish- ing intentional from accidental actions in orangutans (pongo pygmaeus), chimpanzees (pan troglodytes), and human children (homo sapiens). Journal of com- parative psychology. 112(2):192-2066
Alexandra Canavan, David Graff, and George Zipperlen. 1997. CALLHOME American English Speech LDC97S42. Web Download. Philadelphia: Linguis- tic Data Consortium.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, al. 2022. Scaling instruction-fnetuneed language models, arXiv preprint arXiv:2210.1144
One erucial component of our corpus is the rep- resentation of updates of CG in the mind of inter- locutors. That often resulted in multiple annota- tions associated with one event, some of which are updates of belief and CG about previous events. Because of the complexity of this procedure, we ignored the updates in experiments. Finally, reported only preliminary error analyses for each task that may not provide very detailed insight into the systems performances, and consequently may not be informative enough to clearly understand what needs to be improved.
Herbert H. Clark. 1996. Common ground, 'Using' Lin- guistic Books, page 92-122. Cambridge University Press.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Mea- surement, 20(1):37-44)
Philip R. Cohen and Hector J. Levesque. 1990. Ratio- nal interaction as the basis for communication. In Jerry Morgan Philip Cohen and James Allen, editors, Intentions in Communication. MIT Press, Cambridge, MA.
Carri-Cathheine de Marneffe, Mandy Simons, and Ju- dith Tonhauser. 2019. The commitmentbank: Inves- tigating projection in naturally occurring discourse. Proceedings of Sinn und Bedeutung. 23(2):107-124.
We have only worked on Linglish; other lan- guages have different strategies for manipulating the CG and we intend to extend this work to more languages.
Marco Del Tredici, Xiaoyu Shen, Gianni Barlacchi, Bill Byrne, and Adrià de Gispert. 2022. From rewriting to remembering: Common ground for conversational qa models.
Ethics Statement
The annotation procedure developed for the pur- pose of CG annotation is the first one are aware of. It reflects on our assumption on how speakers establish and update their common grounds in a dialog, however, we acknowledge that this may not necessarily reflect what actually happens in our mind. As a result, we do not recommend using our procedure to make any important decision about regarding people's 'ss (mutual) beliefs. The data we used for the experiments and the inter-annotator agreement evaluations are publicly available. All annotation was done in-house with trained under- graduate students who received credit and/or pay- ment.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186. Minncapolis, Minnesota. Association for Computational Linguistics.
Mona Diah, Lori Levin, Teruko Mitamura, Owen Ram- bow, Vinodkumar Prabhakaran, and Weiwei Guo. 2009. Committed belief annotation and tagging. In Proceedings of the Third Linguistic Annotation Work- shop (LAW III), pages 68-73.
Joseph Fleiss. 1971. Measuring nominal scale agree- ment among many raters. Psychological bulletin, 76(5):378.
References
Barbara J. Grosz and Candace Sidner. 1990. Plans for discourse. In Intentions in Communication. MIT Press, Cambridge, MA.
Simon Baron-Cohen, Alan M. Leslie, and Uta Frith. 1985. Does the autistic child have a "theory of

---

William S. Horton and Richard J. Gerrig. 2016. Revisit- ing the memory-base based processing approach to com- mon ground. Topics in Cognitive Science, 8(4):780- 795.
Meeting of the Association for Computational Lin- guistics, pages 2393-4399 Florence, Italy, Associa- tion for Computational I Linguistics.
Vinodkumar Prabhakaran, Tomas By, Julia Hirschberg. Owen Rambow. Samira Shaikh, Tomek Strzalkowski, Jennifer Tracey, Michael Arrigo, Rupayan Basu, Micah Clark, Adam Dalton, Mona Diab, Louise Guthrie, Anna Prokofieva, Stephanie Strassel, Gre- gory Werner, Yorick Wilks, and Janyce Wiebe. 2015. A new dataset and evaluation for belief/factualityy In Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics, pages 82-91, Denver, Colorado. Association for Computational Linguisties.
Nanjiang Jiang and Marie-Catherie de Marneffe. 2021. He thinks he knows better than the doctors: BERT for event factuality fails pragmatics. Transac- tions of the Association for Computational Linguis- tics, 9:1081-1097 1099
Lauri Karttunen. 1971a. Implicative verbs. Language, pages 340-358.
Lauri Karttunen. 1971b. Some observations on factivity, Research on Language & Social Interaction, 4:55- 69.
Vinodkumar Prabhakaran, Owen Rambow, and Mona Diab, 2010. Automatic committed belief tagging. In Coling 2010: Posters, pages 1014-1022, Beijing, China. Coling 2010 Organizing Committee.
Paul Kiparsky and Carol Kiparsky. 1968. Fact. Linguis- tics Club, Indiana University.
Michal Kosinski. 2023. Theory of mind might have spontaneously emerged in large language models.
David Premack and Guy Woodruff. 1978. Does the chimpanzee have a theory mind? Behavioral and Brain Sciences, 1(4):515-526.
J. Richard Landis and Gary G. Koch. 1977. The mea- surement of observer agreement for categorical data. Biometrics, 33(1):
Nils Reimers and Tryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese nert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics.
Kenton Lee, Yoav Artzi, Yejin Choi, and Luke Zettle- moyer. 2015. Event detection and factuality assess- Proceedings of
ment with non-expers supervision. the 2015 Conference on Empirical Methods in Natu- ral Language Processing, pages 1643-1648, Lisbon, Portugal. Association for Computational Linguistics.
Alexis Ross and Ellie Pavlick. 2019. How well do NLI models capture verb veridicality? In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNNPP) pages 2230-2240, Hong Kong, China. Association for Computational Linguistics.
David Lewis. 1969. Convention: A philosophical study. Cambridge MA: Harvard University Press.
John Murzaku, Tyler Oshorne, Amittai Aviram, and Owen Rambow. 2023. Towards generative event factuality prediction. In Findings of the Association for Computational Linguistics; ACL2023, pages 701- 715, Toronto, Canada, Association for Computational Linguistics.
Rachel Rudinger, Aaron Steven White, and Benjamin Van Durme. 2018 Neural models of factuality. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long Papers), pages 731-744, New Or- leans, Louisiana. Association for Computational Lin- guistics.
John Murzaku, Peter 7eng, Magdalena Markowska, and Owen Rambow. 2022. Re-examining factbank: Pre- dicting the author's 's presentation of factuality. In Proceedings of the 29th International Conference on Computational Linguistics, pages 786-796.
Roser Saurí and James Pustejovsky. 2009. FactBank: corpus annotated with event factuality, Language Resources and Evaluation, 43:227-268.
Rowan Nairn, Cleo Condoravdi, and Lauri Karttunen. 2006. Computing relative polarity for textual in- ference. In Proceedings of the fifth international workshop on inference computational semantics (icos-5).
Damien Silco and Antoine Lernould. 2023. Mindgames: Targeting theory of mind in large language models with dynamic epistemic modal logic.
Robert C. Stalnaker. 2002. Common ground. Linguis- rics and Philosophy, 2515-6)
Josef Perner, Susan R Leekam, and Heinz Wimmer. 1987. Three-year-olds difficulty with false belief: The case for a conceptual deficit. British journal of developmental psychology, 5(2):125-13 137.
Jennifer Tracey, Owen Rambow, Claire Cardie, Adam Dalton, Hoa Trang Dang, Mona Diab, Bonnie Dorr, Louise Guthrie, Magdalena Markowska, Smaranda Muresan, et al. 2022. Best: The belief and sentiment corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2460- 2467.
Amir Pouran Ben Veyseh, Thien Huu Nguyen, and De- jing Dou. 2019. Graph based neural networks for event factuality prediction using syntactic and seman- tie structures. In Proceedings of the 57th Annual

---

David Traum. 1994. A Computational Theory of Grounding in Natural Language Conversation. Ph.D. thesis, University of Rochester.
as a were calculated and presented in Ta- ble 12. To analyze the stability of the results, their average and standard deviation were determined Table 13.
Tomer Ullman. 2023. Large language models fail on trivial alterations to theory-of-mind tasks.
Annalisa Valle, Davide Massaro, Ilaria Castelli, and Antonella Marchetti. 2015. Theory of mind devel- opment in adolescence and early adulthood: The growing complexity of recursive thinking ability. Eu- rope's 's Journal of Psychology. 11(1):11-1124
Henry Wellman David 2004. Scaling of theory-of-mind tasks. Child Development, 75(2):523- 541.
Aaron Steven White, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme, 2018. Lexicosyntactic inference in neural models. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pages Belgium. Association for Computational Linguistics.
Brussels,
Heinz Wimmer and Josef Perner. 1983. Beliefs about beliefs: Representation and constraining function of wrong beliefs in young children's 's understanding of deception. Cognition, 13(1):103-1) 128.
Duzhen Zhang, Feilong Chen, and Xiuyi Chen. 2023. DualGATs: Dual graph attention networks for emo- tion recognition conversations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 7395-7408 Toronto, Canada. Association for Computational Linguistics.
A Access to Data and Code
The corpus and code related to experiments are available in the GitHub repository.' All implemen- tations use the Python programming language.
B Belief: different train test splits
The results of the belief classification tests, con- sidering each corpus conversation as the test set. As we mentioned before, four conversations have been used for the tests and these conversations are numbered as "T". "II", "III" and "TV". In the main tests presented in Table7, conversation "IV" is con- sidered the test set. To check the stability of the results obtained by the models trained the main test of the CG body, we also conducted experiments with different train test splits. These experiments done for, FLAN-T5 Fixed Window FLAN-T5 Fixed Window 4, FLAN-TS Speaker- Based Window. The results of Macro Fl and ac- curacy of the model considering each conversation
thupps.//github.... http://gggiuuubcc/ooooteteeee/ee com/cogstates/ 2823-ead emnlp- cumnpp-comon--aee common-grome

---

B)l(A, AVG
Models
CT+ FI
CT- FI
PS
FI
NB FI
Macro FI
Accuracy
Lis test set
FLANNTS Fixed Window
92.00
26.50
6.25
14.25
34.75
82.00
FLAN-TS Fixed Window
93.00
33.75
7.50
11.75
36.50
84.00
FLAN-TS Speaker-based window
90.50
27.25
8.25
18.75
36.19
80.50
II is test set
FLAN-TS Fixed Window
87.00
15.00
32.75
10.75
36.38
73.00
FLAN-T5 Window
23.50
33.75
11.00
38.75
73.00
FLAN-TS Speaker-hased window
87.75
19.50
37,25
6.75
37.81
73.75
FLAN-T5 Window
91.00
27.25
54.25
23.00
48.88
81.50
FLAN-TS Fixed Window
91.00
25.00
43.50
20.50
45.00
80.00
FLAN-TS Speaker-based window
91.50
27.00
46.25
28.75
48.38
81.25
main test set
FLAN-TS Fixed Window
86.83
34.67
28.67
17.50
41,92
74.67
FLAN-TS Fixed Window
87.50
27.50
25.50
11.00
37.88
74.00
FLAN-T5 Speaker-based window
87.67
32.67
30.67
17.83
42.21
76.33
Table 12: Experimental results of the belief classification with different train test split.
Models
CT Fl
CT- Fl
PS
NB FI
Macro F1
Accuracy
AYG, STD
AVG, STD
STD
AVG, STD
AVG, STD
AVG, STD
FLAN-TS Fixed
FLAN-TS Fixed Window 4
89.56, 2.552
27.44, 3.915
21.56, 13.220
13,56, 4.017
39.53, 3.258
77.75, 4.493
FLAN-T5 Speaker-based window
89.35, 1.684
26.60, 4.685
30.60, 14.042
18.02, 7.790
41.15. 4.719
77.96. 3.068
Table 13: Average and standard deviation of the final results on belief classification task using different train test split.


Reviews:
[{"review_id": "OscptKZ3tO", "reviewer": "Reviewer_tfoe", "summary": "", "questions": "Finding a common ground between two people is not a trivial task. The difficulty can arise from ambiguous syntax structures and/or assumed background knowledge. For example, in the following conversation:\nA: \"I think the king of France is bald.\"\nB: \"That's simply not true!\"\nPossible events include:\n\"There is no king of France, so the statement is trivially false.\"\n\"There is a king of France, but he's not bald.\"\nIn this regard, have the authors thought about the different levels of difficulty in the task?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and constructive suggestions.\n\n>> Reasons to reject: \n\n> “The main weakness of the paper is the size of the dataset.”\n\nWhile the dataset may be small by modern standards it has strong IAA between multiple trained annotators for every utterance. The annotation is of very high quality.\n\n> “the sheer size of it is too small to be useful, and this is reflected in the experimental results”\n\nWe do not totally follow how the experimental results demonstrate this. The models trained achieve reasonable performance that may be of use in bootstrapping a larger release as you suggest.  While more data would of course be useful, we also see this as a challenge to machine learning in future work.\n\n> “Initial experiments seem limited. An obvious baseline would be modeling via the Rational Speech Act (RSA) framework.”\n\nThank you for your valuable suggestion. We will consider incorporating your proposed framework as a further baseline in the final version of the paper. It's worth noting that our decision to utilize Flan-T5 for tackling these challenges is based on thorough testing across comparable tasks, along with a careful review of current research literature. Our findings are consistent with the prevailing consensus that adopting a generative approach with a powerful and advanced language model leads to significantly improved outcomes in contrast to traditional methods.\n\n\n\n>> Questions for the authors: \n\nour approach to annotation should be able to disambiguate such cases. This is because we annotate continuous text, using the cues from the whole dialogue when making annotation decisions. Information gained from surrounding utterances will most likely help the annotator to determine whether “that’s simply not true” refers to the king of France not being bald or France not having a king. Note that our dialogues are among actual people.  If in your example B thought there is no king of France, we sincerely doubt they would simply say “That is not true” but instead (or in addition) perhaps “There is no king of France”.  What is an instructive corner case in philosophical semantics may not be relevant to empirical pragmatics: our corpus consists of everyday conversations. (Also remember that “truth in the world” is not part of this research, we are solely focused on speakers' internal beliefs about events.)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "As large language models have become quite powerful, any new pragmatic task (and dataset) that requires \"reading between the lines\" is welcomed by the NLP community. The notion of \"common ground\" is closely related to pragmatics, especially the rational speech act framework. Currently, not many dataset resources exist in this domain, albeit with its growing importance in the field. The CG corpus addresses this missing gap.", "reasons_to_reject": "The main weakness of the paper is the size of the dataset; it builds upon 4 dialogs (561 utterances) which are very small in size, despite acknowledging the difficulty of annotation. Perhaps the authors could bootstrap from the human-annotated datasets to produce a silver-standard but much larger dataset? Orthogonally, authors could leverage the power of instruction-tuned models and provide a step-by-step guideline (rationale) for the task instead of the current input prompt setting. While the main contribution of the paper is the creation of the dataset, the sheer size of it is too small to be useful, and this is reflected in the experimental results.\n\nA more detailed taxonomy of the CG framework is warranted.\n\nInitial experiments seem limited. An obvious baseline would be modeling via the Rational Speech Act (RSA) framework.", "questions_to_authors": "Finding a common ground between two people is not a trivial task. The difficulty can arise from ambiguous syntax structures and/or assumed background knowledge. For example, in the following conversation:\nA: \"I think the king of France is bald.\"\nB: \"That's simply not true!\"\nPossible events include:\n\"There is no king of France, so the statement is trivially false.\"\n\"There is a king of France, but he's not bald.\"\nIn this regard, have the authors thought about the different levels of difficulty in the task?", "paper_topic_and_main_contributions": "This paper introduces the Common Ground corpus which annotates the existing LDC CALLHOME American Speech corpus with events, speakers' beliefs, and their shared common beliefs. The paper's main contributions include (1) the release of new data resources, namely, the CG corpus, and (2) initial experimental analyses on the task of common belief classification.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "References on Rational Speech Act (RSA) framework.", "Typos_Grammar_Style_and_Presentation_Improvements": "Line #208: \"me\" --> \"mean\"\nLine #360, 361: talks about \"X\" and \"{Y}\", but they are not presented in the format."}, "extra_text": {}}, {"review_id": "rnpKeiyJVl", "reviewer": "Reviewer_Vosj", "summary": "", "questions": "1. Ambiguity in mind annotation:\n    a. How to distinguish between certainly believe and possibly believe?  \n2. Experiment set-up clarification: \n\n    a. For experiments in 4.1, what are the results for the speaker-based window?\n\n    b. For experiments in Table 7, is it based on events input or utterance input? For events input, is it the ground truth events or the predicted events?\n3. Common ground prediction: \n\n    a. How do context and belief contribute to the common ground prediction? What are the results if sole belief is given as the input? \n\n    c. In Lines 602-605, CG updates depend on the dialogue history. To distinguish between JA and IN, would it be beneficial to add CG history to the input? \n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed comments and many useful suggestions, and for the thought and care they put into t​​he review.\n\n>> Reasons to reject: \n\n> 1a. “For CG updates, if e is RT, does that mean not e will definitely be JA? Is there a significant dependency between certain annotations?” \n\nThe only time where we allow not e to enter the CC after e was annotated RT is in explicit negations of events, e.g. A: “Did he go to Paris last night?” B: “No.” or A: “He went to Paris last night.” B: “No, he didn’t”. In both cases the event of A going to Paris last night will be rejected, and its negation will be added to CG. This solely depends on B’s answer and not on the fact that e was RT, i.e. B’s response “no” is interpreted as “A did not go to Paris last night”. There are cases where we reject an event and do not add its negation, for example, hypotheticals. A: “If I had gone to Paris, I would have visited the Louvre”. While the conditional relation between the events is entered in the CG, both events (that A goes to Paris, and that A will visit the Louvre) are rejected, but their negations will not be added to the speakers CGs.  In general, our approach is to add what is said explicitly, but not what can be inferred (there can be a large number of inferences in general), which however can at times be hard to pin down.  In such cases, we make explicit choices and explain them in the manual.\n\n> 1b. “It also looks like certain words are more correlated to certain belief categories. How do you address the potential bias or its influence on data statistics?”  \n\nThere definitely exists a set of vocabulary that helps determine the level of belief. For example, the predicate “know” or the adverb “certainly” indicate certain belief in an event. However, our annotation procedure strictly states that while it is allowed to use such semantic cues, we also need to consider other factors, such as syntax, surrounding context, and our world knowledge about humans’ interactions. \n\n\n> 1c. “A statistical analysis of the CG and beliefs dependency would be interesting besides the heuristics designed in L527.” \n\nYes, this is a good suggestion (thank you!) and we will consider performing such analysis for the final version of the paper.\n\n\n> 1d. “Second-order beliefs: as mentioned in L257-266, the common ground depends on not only the first-order belief of the events but also the second-order beliefs of others’ minds. Additional studies can be conducted regarding the relations between common ground and second-order beliefs.” \n\nThe CG annotation allows us to deduce second order beliefs (when there is CG, there is also a second-order belief, and third-order, etc).  Of course there are cases in which we have second order beliefs but NOT common ground (A knows that B does not know that A knows that B knows that A is a murderer, and so on), but we chose for now not to annotate these.  We acknowledge that there can be cases where non-CG second-order beliefs arise, and that we will need to include them, but we have not encountered them in our dialogues yet.  \n\n>> Questions for the authors: \n\n> Q1: “Ambiguity in mind annotation: a. How to distinguish between certainly believe and possibly believe?”  \n\nThere is extensive literature (on belief and factuality) on how to differentiate between these two. It mostly depends on certain lexical items, i.e. predicates in complex clauses that imply uncertainty about the embedded event. For example, “know” vs. “think”. The literature also shows that while distinguishing between different types of such predicates is useful, we also need to consider the context. That means that in our annotations, we take into consideration both the semantic meaning encoded in the main predicates, as well as the annotator’s judgements of the level of belief. \n\n\n> Q2.a:  For experiments in 4.1, what are the results for the speaker-based window?\n\nWe conducted a range of diverse experiments; however, due to space constraints, we have focused on presenting the most compelling findings. We did do the experiment you suggest, and its results are provided below. Notably, the results indicate that it does not outperform the fixed window methods with window sizes 2 and 4.\n\n*No Context Event Generation: SBERT = 69.96%\n\n*Speaker Based Window Event Generation: SBERT = 71.53% (not in paper)  \n\n*Fixed Window Size 2 Event Generation: SBERT = 72.56%\n\n*Fixed Window Size 4 Event Generation: SBERT = 74%\n\nThe Speaker Based approach yields a marginal enhancement when contrasted with the Context-Excluded strategy. However, there was no observable progress compared to the Fixed Window context selection methodologies for this specific task. We would incorporate this result in any final version. \n\n> Q2.b:   For experiments in Table 7, is it based on events input or utterance input? For events input, is it the ground truth events or the predicted events?\n\nAs referenced in line 456 of the paper, Table 7 showcases outcomes derived from Events (not utterances) utilizing various contexts as inputs. These events represent ground truth events. We also experimented with utterances as inputs, ultimately concluding that using events yields higher accuracy. For example, the results of Macro F1 of two utterance-based methods are shown in the following, compared to the results of using the events approach:\n\n*Context with Fixed Window size 2, employing the events approach: Macro F1 = 41.92%\n\n*Context with Fixed Window size 2, utilizing the utterances approach: Macro F1 = 34.50%  (not in paper)\n\n*Context with Speaker-based Window, employing the events approach: Macro F1 = 42.21%\n\n*Context with Speaker-based Window, utilizing the utterances approach: Macro F1 = 30.40%  (not in paper)\n\nThese results would be integrated into either the appendix or the main text of any camera-ready version of the paper.\n\nNeedless to say, if we used predicted events, we would see a reduction in performance.  Furthermore, evaluation is tricky in this case.  We will turn to end-to-end systems and their evaluation in future work. \n\n> Q3.a: How do context and belief contribute to the common ground prediction? What are the results if sole belief is given as the input?\n\nBelief stands as a pivotal determinant in the prediction of Common Ground (CG). Put differently, when interlocutors share identical beliefs, both their CGs tend to be JA or IN; conversely, when beliefs diverge, the CGs tend to be RT.  Neither relation is logically necessary, however.\n\nAs the reviewer notes, one of the main challenges in CG prediction is distinguishing between IN and JA categories, whether a given proposition is newly introduced or already exists in the common ground.  The results of Tables 9 (using context and belief), 10 (context only) show that for JA classification, belief is important.  For IN classification, the interpretation of the results is more complex: it seems that if Gold belief is used, more context is better, but if Gold belief is not used, less context is better.  We have no explanation for this pattern.  We note that small changes in results should not be over-interpreted as there are 58 cases of IN in the test data. \n\nTo answer the second part of this question, we agree that we should have provided results for a beliefs-only FLAN-T5 experiment, and the results will be documented in any final version of the paper. \n\n> Q3.c:  (No Q3b.)  In Lines 602-605, CG updates depend on the dialogue history. To distinguish between JA and IN, would it be beneficial to add CG history to the input?\n\nThe efficiency of distinguishing between \"JA\" and \"IN\" through the utilization of CG history is important. In the lines 543 to 554 of the paper, we mention that our heuristic approach introduces a \"dialog memory\" concept that explicitly incorporates CG history in CG prediction. Conversely, in the T5-based experiments, the inclusion of context in the input implies a more implicit consideration of CG history. Nevertheless, the relatively low performance on IN suggests future experiments where CG history is directly employed as an input feature in the final version of the paper, as you suggest.\n\n>> Missing references:  Thank you for the recommended literature. We will definitely mention it in the related work in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is the first to approach sentence-level belief prediction and study common ground based on speakers’ beliefs in daily conversation. \n2. The corpus annotation involves intensive procedure design and sanity check. The authors clearly presented the definition and designed detailed similarity measures to filter annotators. \n3. Comprehensive experiments and analyses have been conducted on each subtask. \n", "reasons_to_reject": "1. The corpus covers a detailed annotation of events and interlocutors' beliefs. There are interesting several analyses can be made by taking advantage of these annotations. It can give insights into the model design:\n\n    a. For CG updates, if e is RT, does that mean not e will definitely be JA? Is there a significant dependency between certain annotations?\n \n    b. It also looks like certain words are more correlated to certain belief categories. How do you address the potential bias or its influence on data statistics?\n\n    c. A statistical analysis of the CG and beliefs dependency would be interesting besides the heuristics designed in L527.\n\n    d. Second-order beliefs: as mentioned in L257-266, the common ground depends on not only the first-order belief of the events but also the second-order beliefs of others’ minds. Additional studies can be conducted regarding the relations between common ground and second-order beliefs.\n\n\n2. I have several clarification questions below.\n", "questions_to_authors": "1. Ambiguity in mind annotation:\n    a. How to distinguish between certainly believe and possibly believe?  \n2. Experiment set-up clarification: \n\n    a. For experiments in 4.1, what are the results for the speaker-based window?\n\n    b. For experiments in Table 7, is it based on events input or utterance input? For events input, is it the ground truth events or the predicted events?\n3. Common ground prediction: \n\n    a. How do context and belief contribute to the common ground prediction? What are the results if sole belief is given as the input? \n\n    c. In Lines 602-605, CG updates depend on the dialogue history. To distinguish between JA and IN, would it be beneficial to add CG history to the input? \n", "paper_topic_and_main_contributions": "This paper introduced a new corpus annotated with sentence-level events, beliefs, and common ground regarding the events of the speakers involved in the dialogue. The annotation procedure and inter-annotator agreement are carefully presented. Preliminary experiments are conducted on event extraction, belief classification, and common ground updates type.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The difference between the previous works on Theory-of-Mind and common ground in situated dialogues and this work can be discussed under Related work. \n1. Theory-of-Mind: \n\n    [a] Michal Kosinski. Theory of mind may have spontaneously emerged in large language models. arXiv preprint arXiv:2302.02083, 2023.364\n\n    [b] Tomer Ullman. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08399, 2023.366\n\n    [c] Damien Sileo and Antoine Lernould. Mindgames: Targeting theory of mind in large language models with dynamic epistemic modal logic. arXiv preprint arXiv:2305.03353, 2023\n2. Common ground: \n\n    [a] He He, Anusha Balakrishnan, Mihail Eric, and Percy Liang. Learning symmetric collaborative dialogue agents with dynamic knowledge graph embeddings. Annual Meeting of the Association for Computational Linguistics (ACL), 2017.\n\n    [b] Janosch Haber, Tim Baumgärtner, Ece Takmaz, Lieke Gelderloos, Elia Bruni, and Raquel Fernández. The PhotoBook dataset: Building common ground through visually-grounded dialogue. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1895–1910, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1184.\n\n"}}, {"review_id": "IC5ThlrXHI", "reviewer": "Reviewer_R4T2", "summary": "", "questions": "(1) l.121: \"Another crucial aspect of the CALLHOME corpus is that the interlocutors know each other very well, which allows us to assume that their CG set is non-empty.\" Can it be empty at all? There must be something there as soon as interaction begins. Confusing common ground with background knowledge?\n\nWhy precisely the CALLHOME corpus? There are several candidates that could be used instead and would full-fill the same criteria given in the explanation or be even better. In l.123 it is stated that it contains 120 dialogues but in Table 4 we see that only 415 + 146 turns have been annotated which come from 4 dialogues (stated somewhere else). Hence, the dialogues are also quite short?\n\n(2) p4, footnote 1: Description of why this is the case?\n\n(3) l.360: If I understand correctly description at this point, the output of the model to detect events is a sequence of generated text? Why not IOB labels?\n\nSee also discussion in l.391 and following. These errors would be avoided with the IOB labels.\n\n(4) l.366 taking 4 dialogues for training and 1 dialogue for testing is extremely risky for a discourse task, especially on categories such as beliefs and common ground. While the labels for common ground are objective, the dialogues will be very different in terms of language between individuals and hence the test results might not be representative. You mention that participants were familiar with each other which here presents a challenge since one cannot control for the degree of background knowledge the dyads will be possession prior to establishing the common ground leading to very different language.\n\nHave you tried running the experiment with different dialogue as a test set? I expect this would give very different results.\n\n(5) l.438 Why do you expect that data-augmentation based on translation works best (and not some other method of dialogue augmentation)? It is unintuitive as one would expect that common ground is overtly signalled very differently across different languages.\n\nl.492: Why do you think this method of data augmentation works best?\n\n(6) The section on Future work is quite vague. \n\nWhy continue straight away with the audio signal, given that semantic information from text still poses several challenges (described in this paper)? Would that really solve them?\n\n\"We also intend to conduct a more detailed error analysis that could give us more insight to specific issues that need to be addressed in future work.\" What in particular?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments, suggestions, and questions. \n\n> Q(1) “Confusing common ground with background knowledge?” \n\nWe acknowledge that one can distinguish between different types of common knowledge in discourse, with background knowledge (BK) including common knowledge based on common cultural, social, or education background. In addition, there is common knowledge shared because of previous conversations, or previous non-verbal experiences, or acquired in the current conversation only.  We have decided to not distinguish among these types of common knowledge (as it can be difficult given our dataset), and refer to all as t​​he common ground (CG).  So in this sense it is true, that it is safe to assume that the CG cannot ever be empty, and we chose our words badly and thank the reviewer for pointing this out.  What we mean this is that there is a distinction between conversation between two strangers and two close friends, such that the CG between the former is likely empty in the sense that it does not contain any non-BK information, and, consequently, hinders and limits freedom of conversation since it does not allow for, for example, proper name resolutions. We will make the distinction between those two concepts clearer. Thank you for pointing that out. \n\n> “Why precisely the CALLHOME corpus? There are several candidates that could be used instead and would full-fill the same criteria \ngiven in the explanation or be even better. In l.123 it is stated that it contains 120 dialogues but in Table 4 we see that only 415 + 146 turns have been annotated which come from 4 dialogues (stated somewhere else). Hence, the dialogues are also quite short?” \n\n We specifically chose CALLHOME because it was crucial for us to examine dialogues, as opposed to conversations among multiple interlocutors. Additionally, we wanted the interlocutors to be familiar with each other, which allows for more diversity in linguistic expressions. Finally, it was crucial for us that the conversations were not scripted, and in English. We are not aware of any other corpora that meet all of those requirements.  120 dialogues is the entire CALLHOME corpus, we annotated only 4 dialogues so far, as explained in lines 364.  Furthermore, the dialogues often do not consist of the entire conversation between the interlocutors.\n\n> Q2:  p4, footnote 1: Description of why this is the case? \n\nThis footnote is a mistake, thanks for noticing.  We will remove it.\n\n\n> Q.3:  l.360: If I understand correctly, description at this point, the output of the model to detect events is a sequence of generated text? Why not IOB labels?\nSee also discussion in l.391 and following. These errors would be avoided with the IOB labels. \n\nAs can be seen from Table 1, our event descriptions are not in general substrings of the text of the conversations.  There are different reasons for this (expansion of co-reference, introduction of speech act verbs in certain cases, expansion of short answers – all illustrated in Table 1).  We chose this representation of events to make them both expressive as standalone events, and easy to annotate.  We do not wish to give up our explicit notion of event in order to be able to use IOB, and generative techniques using LLMs are ideally suited for this purpose.  Of course, LLMs may hallucinate, and IOB techniques cannot.  But if we used IOB, we would need to introduce additional complex machinery to address the other aspects of our event descriptions we just mentioned. \nEven if we did want to identify substrings of the input text, extensive testing on tasks of this nature, including those addressed in current research literature, show that LLM-based generative approaches work better.  One interesting idea for future work inspired by this question could be to first use an IOB approach and then an LLM-approach to further expand and transform the extracted passages.\n\n\n> Q.4: Have you tried running the experiment with different dialogue as a test set? I expect this would give very different results.\n\nActually, it is three training dialogs and one test dialog.  Due to space constraints, in the paper, we purposefully reported the results on a dialogue as a test set with a balanced representation of the minority classes. \nWe provide here the results of the two most effective event classification methods (namely, those achieving the highest Macro F1 measure without data augmentation in table 7) in a cross-validation setting, using each of the four conversations as the test dataset in turn. We provide the average F1 and Accuracy scores, and their corresponding standard deviations, across all four folds.\n\n\nBelief Classification with Context using Fixed Window Size 2 method (all numbers are based on %):\n\nClass/Measure:           CT+ F1    |   CT- F1   |   PS F1   |   NB F1   |    Macro F1   |   Accuracy\n\nAverage:       89.21      |    25.85    |   30.48    |   16.38    |      77.79       |    40.48\n\nSTD:                2.32     |     7.03     |   17.03    |     4.51    |       4.01        |      5.53\n\n\nBelief Classification with Context using Speaker-based Window method:\n\nClass/Measure:           CT+ F1    |   CT- F1   |   PS F1   |   NB F1   |    Macro F1   |   Accuracy\n\nAverage:       89.35      |    26.60    |   30.60    |   18.02    |      77.96       |    41.15\n\nSTD:                1.68     |     4.69     |   14.04    |     7.79    |       3.07        |      4.72\n\nOur approach yields satisfactory outcomes across these varied train-test partitions.  However, due to the limited occurrences of minority classes (CT-, PS, and NB), the effects of result fluctuations across different train-test splits are pronounced.  We will include these results in any final version.\n\n\n> Q.5: l.438 Why do you expect that data-augmentation based on translation works best (and not some other method of dialogue augmentation)? It is unintuitive as one would expect that common ground is overtly signaled very differently across different languages.\n\nConsidering the size of our corpus, we embarked on exploring various data augmentation strategies. Among these, paraphrasing emerged as one approach we adopted. We used GPT-3.5 to generate sentence paraphrases. However, the best outcomes materialized through a distinct avenue: data augmentation via translation; due to the space limit, we have  only reported the results of data augmentation via translation. Notably, instances where we employed translations in French and German, as highlighted in Table 7, exhibited noteworthy performance enhancements.\nThis intriguing result might stem from the fact that multilingual language models, like Flan-T5, possess a unique proficiency in comprehending concepts across different languages. Yet, the question at hand necessitates further inquiry, a pursuit we intend to delve into during our forthcoming studies.\n\n\n> Q. 6: The section on Future work is quite vague.\n\nWhy continue straight away with the audio signal, given that semantic information from text still poses several challenges (described in this paper)? Would that really solve them?\n\"We also intend to conduct a more detailed error analysis that could give us more insight to specific issues that need to be addressed in future work.\" What in particular? \n We believe that audio signals will allow for resolution of many semantic and pragmatic ambiguities, especially when it comes to short responses, such as “mhm”, “okay”, “i mean”, “i see”, etc. The error analysis conducted for the purpose of this paper looked at all the errors across all 4 annotated dialogues. It would be interesting to conduct dialogue-internal error analysis to find any potential correlations between the type of vocabulary, syntax, communication style used in a dialogues and types of errors observed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The annotation is carefully done and well-motivated and as such the corpus presents a valuable contribution to discourse-annotated corpora that can be used with ML models dealing with text generation (as opposed to dialogue systems).\n\nThe approach to dialogue as text opens interesting questions how research on textual models and dialogue systems can learn from each other (but this is not explicitly addressed in the paper).", "reasons_to_reject": "The authors claim (l.56) that this is the first corpus of common ground which in a very narrow sense is probably true but common ground has been modelled extensively, in different ways, and to different degrees of depth in rule-based and hybrid dialogue systems that the paper does not compare with or refer to (with the exception of work by Traum).\n\nThe authors do not seem to be aware of the work on dialogue systems as they state \"While there has been much attention paid to the common ground in cognitive science, there has not been much work in natural language processing\" which is simply not true as there this a very prominent ACL track.\n\nThe main contribution of the paper is therefore the annotation itself.", "questions_to_authors": "(1) l.121: \"Another crucial aspect of the CALLHOME corpus is that the interlocutors know each other very well, which allows us to assume that their CG set is non-empty.\" Can it be empty at all? There must be something there as soon as interaction begins. Confusing common ground with background knowledge?\n\nWhy precisely the CALLHOME corpus? There are several candidates that could be used instead and would full-fill the same criteria given in the explanation or be even better. In l.123 it is stated that it contains 120 dialogues but in Table 4 we see that only 415 + 146 turns have been annotated which come from 4 dialogues (stated somewhere else). Hence, the dialogues are also quite short?\n\n(2) p4, footnote 1: Description of why this is the case?\n\n(3) l.360: If I understand correctly description at this point, the output of the model to detect events is a sequence of generated text? Why not IOB labels?\n\nSee also discussion in l.391 and following. These errors would be avoided with the IOB labels.\n\n(4) l.366 taking 4 dialogues for training and 1 dialogue for testing is extremely risky for a discourse task, especially on categories such as beliefs and common ground. While the labels for common ground are objective, the dialogues will be very different in terms of language between individuals and hence the test results might not be representative. You mention that participants were familiar with each other which here presents a challenge since one cannot control for the degree of background knowledge the dyads will be possession prior to establishing the common ground leading to very different language.\n\nHave you tried running the experiment with different dialogue as a test set? I expect this would give very different results.\n\n(5) l.438 Why do you expect that data-augmentation based on translation works best (and not some other method of dialogue augmentation)? It is unintuitive as one would expect that common ground is overtly signalled very differently across different languages.\n\nl.492: Why do you think this method of data augmentation works best?\n\n(6) The section on Future work is quite vague. \n\nWhy continue straight away with the audio signal, given that semantic information from text still poses several challenges (described in this paper)? Would that really solve them?\n\n\"We also intend to conduct a more detailed error analysis that could give us more insight to specific issues that need to be addressed in future work.\" What in particular?", "paper_topic_and_main_contributions": "The paper describes annotation of an existing LDC CALLHOME American Speech corpus with speaker and hearer beliefs and their common ground information states with labels that are described and motivated in the paper. The corpus is validated with ML models based on BERT and FLAN-T5 pre-trained models.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Work on dialogue systems that deals with modelling BDIs and information state of agents."}, "extra_text": {}}, {"review_id": "ajULmeW6iL", "reviewer": "Reviewer_vhRH", "summary": "", "questions": "1. Why can't you view CG, in the way to define it, as fact-checking (or belief checking) where the bank of facts if being updated? If so, why not use a Fact Checking pipeline as a baseline\n\n2. the tasks of predicting humor and satire seem promising avenues to check CG. People understand satire only when they share a broad CG. \n\n3. What doe the numbers in Table 5 represent? It's better to add this to the caption (in all tables) to read tables as stand-alone.\n\n4. For event detection and belief detection, you could have used previous work since these tasks are not new. Why didn't you?\n", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments, suggestions, and questions. \n\n\n> Q1: Why can't you view CG, in the way to define it, as fact-checking (or belief checking) where the bank of facts is being updated? If so, why not use a Fact Checking pipeline as a baseline \n\nThe task of choosing between JA and IN is indeed similar to fact-checking, as long as we have a complete representation of the CG.  However, in our case some shared beliefs come from prior conversations or outside knowledge so there is nothing to check against. Items which are marked as IN need not appear explicitly in the prior text. For the cases where evidence for a belief can be found from the text, a fact-checking inspired approach might work.  In addition, the detection of rejection by the discourse partner (RT) does not appear to have any equivalent in fact-checking. \n\n\n> Q2: the tasks of predicting humor and satire seem promising avenues to check CG. People understand satire only when they share a broad CG. \n\nYes, those are definitely exciting avenues to extend our work. Thank you for the comment! \n\n\n> Q.3: What do the numbers in Table 5 represent? It's better to add this to the caption (in all tables) to read tables as stand-alone.\n\nAs explained in line 351, these numbers denote the length of context employed. In the event generation process, we input the target utterance along with a structured context, as detailed in line 359, into the Flan T5 language model. To illustrate, consider the \"FLAN-T5 Fixed Window Size 4\" method for event generation, where the context appended to the target utterance encompasses all preceding utterances up to a length of 4. This approach facilitates a comprehensive contextual input for generating events.\nWe will take your suggestion into account in any final version of the paper.\n\n\n> Q.4: For event detection and belief detection, you could have used previous work since these tasks are not new. Why didn't you?\n\nAs can be seen from Table 1, our event descriptions are not in general substrings of the text of the conversations.  There are different reasons for this (expansion of co-reference, introduction of speech act verbs in certain cases, expansion of short answers – all illustrated in Table 1).  We chose this representation of events to make them both expressive as standalone events, and easy to annotate.  We do not wish to give up our explicit notion of event in order to be able to use previous existing methods, and generative techniques using LLMs are ideally suited for this purpose.  On the other hand, we would need to introduce additional complex machinery to address the other aspects of our event descriptions we just mentioned. \nEven if we did want to identify substrings of the input text, extensive testing on tasks of this nature, including those addressed in current research literature, show that LLM-based generative approaches work better.  \n\n\nFor belief detection, we will be using existing research in future work, both the corpora and the actual research on predicting belief/factuality.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is very well written and easy to follow. The CG dataset is indeed unique and would be used by other researchers to study the CG problem.", "reasons_to_reject": "The perspective of CG that the authors study is limited (only to beliefs) as they themselves attest. Therefore one can think of their work as an extension of fact-checking where the fact database is being updated with new facts or facts re removed.", "questions_to_authors": "1. Why can't you view CG, in the way to define it, as fact-checking (or belief checking) where the bank of facts if being updated? If so, why not use a Fact Checking pipeline as a baseline\n\n2. the tasks of predicting humor and satire seem promising avenues to check CG. People understand satire only when they share a broad CG. \n\n3. What doe the numbers in Table 5 represent? It's better to add this to the caption (in all tables) to read tables as stand-alone.\n\n4. For event detection and belief detection, you could have used previous work since these tasks are not new. Why didn't you?\n", "paper_topic_and_main_contributions": "The paper continues the line of work of factual checking and belief checking in the direction of predicting common grounds. The main two contributions are making available an annotated dataset for the task of CG  precision (they re-annotate an existing dataset). The second contribution is to evaluate the hardness of the CG prediction task by training a classifier and showing its performance.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main contribution of this paper is to annotate the Callhome corpus with speaker and hearer beliefs and their common ground. The dataset is used for the tasks of event extraction, belief classification  and common ground update classification, and provides baseline performances for each of these tasks. 

The reviewers appreciate the dataset as timely to allow for evaluation of a pragmatic task, but note that more comparison to related tasks in dialog systems could be discussed. The reviewers also note the small size of the dataset as the main drawback of this work (4 dialogs, 561 utterances); they also suggest modelling the task using a model that fits the task better, such as a rational speech act model. 

Overall, the task is timely and important, but the dataset is small and the modelling not fully convincing, so this seems like a piece of ongoing work that might achieve higher impact if extended.
Decision: Accept-Findings