Identifying Statements Crucial for Awareness of Interpretive Nonsense to Prevent Communication Breakdowns
Authors: Tomoyuki Maekawa
Michita Imai
Venue: EMNLP 2023 Main (2023)
Keywords: Communication breakdowns
dialogue
context
rephrasing
language model
TL;DR: Crucial statements are identified by processing text with omissions.

Abstract: During remote conversations, communication breakdowns often occur when a listener misses certain statements. Our objective is to prevent such breakdowns by identifying Statements Crucial for Awareness of Interpretive Nonsense (SCAINs). If a listener misses a SCAIN, s/he may interpret subsequent statements differently from the speaker's intended meaning. To identify SCAINs, we adopt a unique approach where we create a dialogue by omitting two consecutive statements from the original dialogue and then generate text to make the following statement more specific. The novelty of the proposed method lies in simulating missing information by processing text with omissions. We validate the effectiveness of SCAINs through evaluation using a dialogue dataset. Furthermore, we demonstrate that SCAINs cannot be identified as merely important statements, highlighting the uniqueness of our proposed method.

Identifying Statements Crucial for Awareness of Interpretive Nonsense to Prevent Communication Breakdowns
Tomoyuki Maekawa and Michita Imai
Keio University
Yokohama, Japan (maekawa, michita)@an taaitataaallll kes.keei..c...
Abstract
Table Example dialogue. Retrieved from JPes-
onaChat (Sugiyama al., 2023). Originally in Japanese; translated into English for inclusion in this paper.
During remote conversations, communication breakdowns often occur when listener misses certain statements. Our objective is to pre- vent such breakdowns by identifying State- ments Crucial for Awareness of Interpretive Nonsense (SCAINs). If listener misses SCAIN, s/ee may interpret subsequent state- ments differently from the speaker's intended meaning. To identify SCAINs, we adopt unique approach where we create a dialogue by omitting two consecutive statements from the original dialogue and then generate text to make the following statement more specific. The novelty of the proposed method lies in simulating missing information by processing text with omissions. We validate the effective- ness of SCAINs through evaluation using dialogue dataset. Furthermore, we demonstrate that SCAINs cannot be identified merely im- portant statements, highlighting the uniqueness of our proposed method.
Statement
A: Hello. How are you?
2 B:
I'm fine. But I'm busy with my work at the advertising agency.
3 A: Thank you. I'm
worker.
B: You work as care worker? That's a tough job, it?
A: Exactly. Working at an advertising agency must be hard too. By the way, live in one-story house in Hokkaido. 6 B: You live in Hokkaido? envy you be- cause that's place with good food. I'm from Aomori and I like mountain- ous areas, so live near the mountains. There are many delicious foods in Hokkaido. You're 'ou're from Aomori, so you're close by. But I'd like to live Tokyo one day.
1 Introduction
8 B: Tokyo? Tokyo has many gorgeous places, doesn't it?
Communication brcakdowns are sometimes caused
by listeners missing speaker's statements con- versations. Listeners tend to miss statements when they are absent-minded or lost in thought. In on- line conversations. there are other obstacles such network failures and background noise, which may cause more statements to be missed than in face-to-face conversations.
A: Because I was born in the countryside,
it's very attractive place.
speaker A, which contradicts the primary intent of statement 9. Importantly, the listener might not be aware of the missing statements because state- ment 9 was still interpretable without statements 7 and 8, which could lead to breakdown in the communication.
Problems associated with listeners missing state- ments are not just the information the missing statements not being conveyed. The listener's in- terpretation of other statements may also change because they missed previous statements. Table shows example of dialogue between two speak- ers. With statement 9, speaker A wants to say that Tokyo is a very attractive place for her. Suppose a listener missed statements and 8, as shown in Table 2. The listener might interpret statement to mean that mountainous areas are attractive to
Online conversations can be supported by exist- ing techniques, such dialogue summarization (Chen al., 2021) and discourse structure visual- ization (Holmer, 2008). However, previous studies have not considered the change in the interpreta- tion caused by missing statements. Speakers need to be aware that the listener's 's interpretation may

---

Table 2: Example of a dialogue with two statements omitted. The interpretation of statement 9 may differ from that in Table 1.
SCAIN
SCAIN extractor
Transcript
#
Statement
Online meeting server
1 A: Hello. How are you?
2 B: I'm fine. But I'm busy with my work
Speech
Speech
at the advertising ageney.
3 A: Thank you. I'm m care worker.
4 B: You work a worker? That's tough job, it?
Speaker
Speaker
5 A: Exactly. Working at an advertising agency must be hard too. By the way, I live in a one-story house in Hokkaido. 6 B: You live in Hokkaido? I envy you be- cause that's place with good food. I'm from Aomori and like mountain- areas, I live near the mountains. 9 A: Because born in the countryside. a very attractive place.
SCAIN:
would Takyo.
Notification of SCAIN on the online meeting application
Figure I: Diagram of the support system for online conversation using SCAINs.
differ from their intended meaning, though it is difficult to detect that the listener has missed statement.
the statements when mishearing may change the interpretation of the subsequent statement, but the listener is not aware of the change, so the conversa- tion support system will alert the speakers to pay attention to the different interpretations.
Therefore, statements crucial for interpreting other statements need to be detected to prevent com- munication breakdowns. Specifically, our goal is to identify the Statements Crucial for Awareness of Interpretive Nonsense (SCAINs). The "nonsense" abbreviated in SCAIN has special meaning. A listener who has not heard SCAINs may interpret subsequent statement differently from the speaker's primary intent. However, the listener is likely to believe that his/her interpretation is correct.
This paper proposes the concept of SCAIN as novel idea that aims at mutual understanding through conversations. We explain the method for identifying SCAINs, which uses dialogue with some statements omitted from the original. Then, we evaluate the validity and uniqueness of SCAINs using a dialogue dataset.
We assume that SCAINs will be used to develop a conversation support system that is implemented online meeting tools. Figure I shows a diagram of the assumed system. Speech from the speakers is sent the online meeting server. The transcript of the speech fed into the SCAIN extractor. The SCAIN reported the speakers in the online meeting application so that the speakers are aware of possible interpretations that differ from their intended meaning.
The contributions this paper are two-fold:
We first propose the method of extracting SCAINs from dialogue by creating another dialogue assuming that some statements have been missed.
We validate that SCAINs are indeed crucial to the interpretation of the dialogue and that SCAINs cannot be identified as merely impor- tant statements.
The SCAIN extractor computes the change in interpretation by listener who missed hearing certain statements. In this sense, SCAINs are dif- ferent from locally ambiguous statements. Liven if a statement is locally ambiguous, the statement is not a SCAIN unless the interpretation changes due to mishearing. The SCAIN extractor picks up
2
Related Work
2.1 Important statements extraction
Conversations can be supported by using tech- niques to extract important statements. Sentence

---

importance scoring has been integrated into the process of generating summaries (Moratanch and Chitrakala, 2017). the context of dialogue sum- marization, important statements are typically re- garded as those containing salient information, named entities, and speaker emotion and intent (Chen et al., 2021). However, in previous meth- ods, scores have been assessed after reading all statements, without considering the possibility of overlooked or missed statements.
to the statement (Li al., 2016). Shibata et al. (2023) proposed method transform an arbitrary search query into another search query that fits the dialogue context. However, there no existing method to generate a natural language text that rep- resents the interpretation of statement from the perspective of particular context.
SCAIN extraction architecture
3.1 Overview
2.2 Topic shift detection
The key idea the SCAIN extraction architec- ture is to simulate the interpretation of a statement from two different perspectives: a listener who has missed some previous statements and speaker who has not missed any statements. The listener's perspective is simulated by constructing dialogue with omissions, while the speaker's perspective is simulated by constructing a dialogue without omis- sions.
A conversation can contain multiple topics, and sometimes the topic shifts during the conversation. Detecting the topic shifts is important to promote smooth discussion in online communities (Sun and Loparo, 2019). Xie et al. (2021) introduced a Topic- shIft Aware dialoG datasEt (TIAGE) and proposed dialogue models that detect and generate topic-shift responses. However, it is unclear whether topic shifts change the interpretation of the subsequent statement.
We extract SCAINs by judging whether a pair of consecutive statements are SCAINs not. Fig- ure shows the overall architecture for extracting SCAINs from dialogue. We refer to particu- lar pair of consecutive statements as the candidate statements, except for the first and last statements in the dialogue. The statement following the can- didate statements called the core statement. We take several statements from the original dialogue to construct two shorter dialogues: the complete dialogue and the omitted dialogue. We build a prompt with each dialogue by using prompt tem- plate. A large language model (LLM) generates more specific statement by rephrasing the core statement in each prompt. Each rephrased state- ment embedded into a vector. We compute the similarity score between two embedding vectors. Finally, we compare the similarity score with a predetermined threshold to determine whether the candidate statements are SCAINs or not.
2.3 Discourse structure and cohesion analyses
Discourse structure and cohesion analyses can also be used for dialog comprehension, as means of providing the listener with useful information to overcome the missing statements. Discourse struc- ture analysis extracts various kinds of relationships between sentences in structured way, which al- lows one to find out the sentences that are crucial for understanding a particular sentence (Webber et al., 2012). Holmer (2008) proposed ChatLine, an application for analyzing the discourse structure of chat transcripts. Connections between statements can be represented visually by using ChatLine. Co- hesion analysis refers the resolution of various types of relationships between words and phrases in a text, such as coreference and bridging anaphora (Ueda et al., 2020). Coreference resolution is useful for dialogue comprehension because pronouns and ellipses are often observed in dialogues (Lin et al., 2016). However, discourse structure and cohesion analyses are not sufficient to simulate the change in interpretation caused by missing statements.
3.2 Constructing complete and omitted dialogues
We construct the complete and omitted dialogues for each pair of statements. The complete dialogue consists of the statements from the first statement to the statement. For example, if statements 7 and 8 are the candidate statements, the complete dialogue is created by concatenating statements 1, 2, 3, 4, 5, 6, 7, and 9. On the other hand, the omitted dialogue consists of the statements from the first statement to the core statement, except
2.4 Context dependence
Our approach is track the change in the inter- pretation of statements depending on the dialogue context. Some statements a dialogue are context- dependent the sense that a listener must con- sider the previous statements in order to respond

---

Pairs of statements
A. Hello, How you?
#2 #3
to bay my adrcitinng aucrics.
[A Tihani you I'm worker
#4
webk as care worker? a tough joh un'tr?
#4 #5
AA Lsaccity Werking adventiing agensy be
the way, live in one stoy house in lookaado you grodd food. I'm from Asmoni and like mocnteitoon snce, so
Holkatddl
N6 47
A: There many delicious foos Hokiaião You're too
Anmoci, you'e by. places soein' ai!
Tokyo day.
88 & A9
3. Tokyo! Telyoo has
Candidate statements:
Besssoe honu couttrands it's vaty stiustne paase
3.
place
Wher
#9 & #10
#7 & #8
def?.ceff! enjoy photogmphy
have been withor boythend for long time, spend
#10 & #1l
days off selosing by myself Postogmphy is nioe.
#11 #12
Spending
important
Lofien
spead time relasing home.
#13
I'm hazy
B: That's right is also impociant upead time alone relax from wark.
Core statement:
Complete dialogue
Omitted dialogue
are you?
How you?
B: I'm linc. Bat I'm bay wilh work the udrettising
2 B: I'm fixe. But I'n with wotk et Bie advettiong
you. I'ms case
Thank you. I'm worker? "That's tough isn't
B: You work
worker" That's
ion't
4 B: You work
A: Disetly: Working ot adventsiigg agency must hard too. By Hikkaaddo
A: Diactty. Working at an adventising agency must be hand con, By
live
live B: You live is Hokkaidor I ency you banare thet place with
HoMMaido
B:
You live Hokknids? ansy you bacause thet' 's place with
food from and like
from and like
liee near the mouniine
live norr the mounttine.
mony
counerydee
Autoci, you're cime But I'd like to live Tokyo day.
Tokyo? Tokyo masy gurgoous places. docalt
Becaue I was bom the countesale, a'' slinative place
Prompt template
Prompl
Prompl
Large language
Large language model
Rephrused statement
Rephrased statement
Tolse has so mann uands cales, aottusatt, and stoces thas voty appsalin
I like nouttinee arces, and living noa dhe moustaits vay sttnetive to me
Embedding model
Embedding model
Embedding of rephrased statement
Similarity
Embedding of rephrased statement
calculation
Similarity score
Statements & 8 are not SCAINs.
Lower than threshold?
Yes
Statements 7 & are SCAINs.
Figure Diagram of the architecture to extract SCAINs from a dialogue.

---

Table Prompt template to rephrase the core statement. [dialogue] is filled in with either the complete omitted dialogue, {core statement) is filled in with the core statement.
3.4 Calculating similarity score
We obtain the embeddings of the rephrased state- ments by using embedding model. The embed- dings vectors in the embedding space, where the shorter distance between two vectors indicates that the corresponding texts are more similar. The simi- larity score between the embeddings is computed as the cosine similarity. We set a threshold and compare the similarity score to the threshold. If the similarity score is lower than the threshold, which means that the interpretation of the core statement from the perspective of the missing listener is sig- nificantly different from the primary intent of the speaker, the candidate statements are determined to be SCAINs. Otherwise, missing the candidate statements does not significantly change the inter- pretation of the core statement, so the candidate statements are determined to be non-SCAINs.
Rephrase particular statement a dialogue into a more specific statement.
A dialogue between A and B is written and one of the statements is assigned to be rephrased. Rephrase the assigned statement into more specific statement using the words in the dia- logue.
# Dialogue (dialogue)
# Statement to be rephrased {core statement]
# Rephrased statement
Evaluation
4.1 Overview
for the candidate statements, If we focus on state- ments and the omitted dialogue created by concatenating statements 1, 2, 3,4,4 4, 5, 6, and 9. The omitted dialogue simulates the situation where a listener misses the candidate statements, while the complete dialogue simulates the situation where no statements are missed.
We performed two types of evaluations: validity evaluation and uniqueness evaluation. The pur- pose of the validity evaluation was to show that the SCAINs extracted by our proposed method are significantly crucial for interpreting the subsequent statement. The purpose of the uniqueness eval- uation was to show that the SCAINs cannot be extracted by another method, particularly impor- tant statements topic shifts.
3.3 Rephrasing the core statement
We obtain the rephrased statements by prompting LLM. Table 3 shows the prompt template used build the prompt. The LLM generates more specific statement under the interpretation on the basis of the context. When the complete dialogue provided, the LLM takes into account the complete context to rephrase the core statement, so that the rephrased statement represents the primary intent of the speaker. On the other hand, when the omit- ted dialogue is provided, the LLM does not have access the candidate statements. In this case, the rephrased statement represents the interpretation of the core statement from the perspective of listener who has missed the candidate statements.
4.2 Validity evaluation
4.2.1 Setup
We used 100 dialogues in JPersonaChat (Sugiyama et al., 2023). We applied our extraction method to total of 952 pairs of statements. Since the dialogues written in Japanese, we used the Japanese translation of the prompt template. We used gpt- t.r-o (Brown et al., 2020) as the LLM and entered the prompt as content for the "user" role. Table 4 lists the parameters set for the LLM. We used texx-embedding-ada-uu 2022) the embedding model. We set the thresh- old for the similarity score at 0.85 on the basis of the observation of the rephrased statements. Fig- ure shows the histogram of the similarity scores computed for the pairs of statements. We identified 65 pairs of statements as SCAINs that ended up with similarity scores lower than 0.85. Here, we considered the SCAINs as pairs because we wanted
(Neelakantan et al.,
Our method takes advantage of the LLM's abil- ity to translate text in the manner specified in the prompt (Brown et al., 2020). It is noteworthy that the LLM never ignores any statements in the prompt, so we need the proposed architecture to simulate the situation where certain statements are missed.

---

Table 4: Parameters set for the LLM in the evaluations.
and omitted dialogues. Table lists the details of the metrics.
Parameter
Value
4.2.4 Procedure
Model
" "gpt-3.5- 3.5-ttubb""
We crowdsourced 30 raters. We created 10 online questionnaires and assigned three raters to each questionnaire. Each questionnaire contained total of 10 sets of dialogues: five in the SCAIN condi- tion and the other five in the non-SCAIN condition. There was no duplication of dialogues within any of the questionnaires. The dialogues were presented in a random order, and the raters were blinded to the conditions. For each set of dialogues, the raters first read the omitted dialogue and answered the questions for the omitted dialogue. They then read the complete dialogue and answered the questions for the complete dialogue. At this point, we asked the raters not change the answers for the omitted dialogue to ensure that they were rating the omitted dialogue without reading the complete dialogue. Finally, they answered the question for the compar- ison between the complete and omitted dialogues.
Max tokens
200
Temperature
0
Stop words
"In"
200
150
Frequency
100
50
0
0.80
0.85
0,90
0.95
1.00
Similarity score
Figure 3: Histogram of the similarity scores.
4,2.5 Prediction of the results
to analyze the tendency of the dialogues containing the SCAINs.
We hypothesized that the SCAINs would signifi- cantly affect the interpretation of the subsequent statement. If the SCAINs were omitted from the dialogue, the intent of the core statement would be less clear, the core statement would be perceived as more sudden, and the meaning of the core state- ment would be different from that of the complete dialogue. In summary, we predicted the following results:
4.2.2 Conditions
The validity evaluation conducted with human raters in within-participan design. In particular, there were two conditions for the pairs of state- ments to be rated:
SCAIN condition: We selected SCAINs.
Non-SCAIN condition: We selected state- ments that ended up with similarity scores higher than 0.9897. This was done to ob- tain the same fraction of the statements as the SCAINs with the highest similarity scores.
Clarity in the omitted dialogue would be lower in the SCAIN condition than in the non- SCAIN condition.
Suddenness in the omitted dialogue would be higher in the SCAIN condition than in the non-SCAIN condition.
We set these extreme conditions for the basic analy- sis to find out tendencies in the dialogues extracted by the proposed method.
Difference would be higher in the SCAIN con- dition than in the non-SCAIN condition.
4.2.3 Metrics
For each pair of statements, both the complete and the omitted dialogues were presented to the raters. The omitted dialogue was presented as "dia- presented
4.2.6 Results
Figure 4 shows the averages and standard devi- ations of the ratings. Mann-Whitner U tests vealed that clarity in the complete dialogue is higher in the SCAIN condition (U 9785.0, p = 0.030), suddenness in the complete dialogue is lower in the SCAIN condition (U = 12926.0,
logue 1", and the complete dialogue "dialogue 2". The raters assessed the interpreta- tion of the core statement in the dialogues by using the following metries: clarity and suddenness were assessed for each dialogue and difference was as- sessed for the comparison between the complete
0.015), clarity in the omitted dialogue is

---

Table 5: Metrics used in the validity evaluation. These metrics were assessed with 5-point Likert seales. Placcholders marked with [core statement) were filled in with the text of the statement. The questions were originally in Japanese and were translated into English for inclusion in this paper.
Metrics for each of the complete and the omitted dailogues
Metric
Question
Clarity
the intent of the statement "[core statement)" clear this context?
(I: not at all clear, 5: very clear)
Suddenness
Is the statement "(core statement)" sudden in this context?
(I: not at all sudden, 5: very sudden)
Metric for the comparison between the complete and omitted dialogues
Metric
Question
Difference
Are the meaning of the statement "(core statement)" dialogue and the meaning of the statement "(core statement)" in dialogue different from each other?
(I: completely equal, 5: completely different)
B.S.
*0*
5-
Rating
4.28
4-
3.87
3.75
3,63
94
2.55
2.65
1.94
2-
1.68
SCAIN
R00000
Non-SCAIN
Clarity (complete)
Suddenness (complete)
Clarity (omittedd)
Suddenness (omitted)
Difterence
Figure 4: Results of the validity evaluation. Bars represent averages and eтrоr bars represent standard deviations.
*p < 0.05, *$p < 0.01, s*sp < 0.001.
lower in the SCAIN condition (U = 13155.5, p 0.009), suddenness in the omitted dialogue is not significantly different between the conditions (U 10735.0, 0.47), and difference is higher in the SCAIN condition (U 6721.5, p 0.001).
in each dialogue and inserted the dialogue into the prompt template shown in Table 6. See Appendix A for other prompt templates that were considered before the evaluation.
For the comparison between SCAINs and topic shifts, we used the TIAGE test dataset (Xie et al., 2021) annotated on PersonaChat (Zhang et al., 2018). For each statement in the TIAGE dataset, the annotators annotated whether or not there was topic shift. We extracted SCAINs using the En- glish version of the prompt template while keeping the other settings the same in the validity experi- ment.
4.3
Uniqueness evaluation
4.3.1 Setup
The uniqueness experiment consists of the follow-
ing two comparisons:
Comparison between SCAINs and important statements
Comparison between SCAINs and topic shifts
4.3.2 Conditions
The uniqueness evaluation was performed by clas- sifying the statements in the dialogues from three different aspects. One of the classification aspects was whether the statements were SCAINs or non- SCAINs:
For the comparison between SCAINs and impor- tant statements, used the same 100 dialogues as in the validity evaluation. We extracted SCAINs using the same LLM, embedding model, and pa- rameters as in the validity evaluation. In addition, we extracted important statements by using another prompt template. We numbered the statements
SCAINS: A statement was identified as a SCAIN if at least one pair containing the state-

---

Table 6; Prompt template to extract important state- ments. [dialogue] is filled in with the dialogue.
Table Results the uniqueness evaluation.
Non- important
Select the most important statements from the dialogue. A sequence of dialogue written with numbered statements. Select the two most important statements from the dialogue and write their numbers.
Important
Total
SCAIN Non-SCAIN Total
25
100
125
173
954
1127
198
1054
1252
# Format of output [statement number 1, statement number 2]
For the comparison between SCAINs and topic shifts, we classified the statements into topic shifts and non-topic shifts:
# Dialogue (dialogue)
Topic shifts: A total of 315 statements in the PersonaChat dataset were annotated as having topic shifts.
# Important statements
Non-topic shifts: The remaining 1149 state- ments in the PersonaChat dataset were anno- tated as having no topic shifts.
ment ended up with similarity score lower than 0.85. We obtained total of 125 SCAINs of 1252 statements from the JPersonaChat dataset and 84 SCAINs of 1464 statements from the PersonaChat dataset.
4.3.3 Prediction of the results
We hypothesized that the SCAINs would capture a unique aspect different from the importance of the statements. Consequently, we predicted that the classification based on the SCAINs would be inde- pendent of the classification based on importance and topic shifts.
Non-SCAINs: A statement otherwise identified as non-SCAIN. We identified total of 127 statements as non-SCAINs in the JPersonaChat dataset and 1380 statements in the PersonaChat dataset as non-SCAINs.
4.3.4 Results
Here, considered the SCAINs as individual statements because we wanted analyze the re- lationship between the classifications of the state- ments in terms of being SCAINs, important state- ments, and topic shifts. The number of SCAINs may be less than twice the number of pairs because there are overlapping pairs
Table 7 shows the frequencies of the classified statements for the comparison between SCAINs and important statements. a result, 25 SCAINs were classified as important statements, while the remaining 100 SCAINs were classified as non- important statements. A chi-square test showed that the classification based on SCAINs is inde- pendent of the classification based on importance (1) 1.49, p 0.22). Some qualitative results are shown in Appendix B.
The other classification aspects were the impor- tance and topic shifts. For the comparison between SCAINs and important statements, we classified the statements into important statements and non- important statements:
Table 8 shows the frequencies of the classified statements for the comparison between SCAINs and topic shifts. Similarly, 18 SCAINs were classi- fied as having topic shifts, while the remaining 66 SCAINs were classified as not having topic shifts. A chi-square test showed that the classifica- tion based SCAINs was also independent of the classification based topic shifts (x²(1) 0.0, p 1.0).
Important statements: A statement was identi- fied as important if the statement number was listed in the output of the LLM. We obtained total of 198 important statements because the LLM did not generate any statement numbers for one the dialogues.
Non-important statements: A statement was not otherwise identified important. We identified a total 1054 statements as non- important statements.
Discussion
The validity of the SCAINs was confirmed that when the SCAINs were omitted, the intent

---

Table Results uniqueness experiment for compari- son between SCAINs and topic shifts.
The important statements tended contain information about the speaker's personal tributes.
Topic shift
Non-topic
shift
Total
The important statements tended to contain the speaker's opinions and/or suggestions.
SCAIN
18
66
84
Non-SCAIN
297
1083
1380
Note that JPersonaChat consisted of dialogues in which the speakers tried to get to know each other, so most statements contained some kind of informa- tion about the speaker's personal attributes. There- fore. it may have been difficult select only two important statements from a dialogue.
Total
315
1149
1464
of the subsequent statements became unclear and the meaning of the subsequent statements changed. On the other hand, the statements became sudden when the preceding statements were omitted, re- gardless of whether the preceding statements were the SCAINs not. The statements may have been perceived sudden because they were inconsis- tent with previous statements, not just because the intent of the statement was unclear. In contrast, the intent of the statements became clearer and the statements became less sudden when the SCAINs were provided before the statements. There may have been context effect caused by the order in which the dialogues were presented. In the SCAIN condition, the raters may have found the complete dialogues relatively clearer and less sudden after rating the omitted dialogues, which were less clear.
We speculate that there is no significant relation- ship between the SCAINs and topic shifts because the topic shifts do not necessarily change the inter- pretation of the subsequent statement. Even if there is topic shift, the listener may correctly interpret statement that contains sufficient information about the new topic.
The SCAIN extraction architecture can be ap- plied to a conversation support system. The system will inform the sender of SCAINs which statements are SCAINs during conversations. The sender will pay close attention to verbal and nonverbal re- sponses from the listener. The sender will be able to identify the communication breakdowns earlier and recover from the breakdowns by supplementing the SCAINs with more information. If the speakers do not use the system, the sender will not care if his/her intent conveyed to the listener and will continue with a one-way conversation. The system will prevent such situations and let the speakers be aware of communication breakdowns.
If the LLM is informed that some statements have been omitted, it can generate possible inter- pretation of the core statement by predicting the content of the omitted statements. However, the generated interpretation is likely to be different from the intended meaning because crucial infor- mation for interpreting the core statement has been lost.
Conclusion
The uniqueness of the SCAINs was confirmed by the fact that the SCAINs were identified regard- less of the importance of the statements and topic shifts. We found different characteristics for the SCAINs and for the important statements through qualitative analysis. Particularly, the SCAINs had the following characteristics:
In this paper, we proposed the concept of State- ments Crucial for Awareness of Interpretive Non- sense (SCAIN) and method for extracting SCAINs from dialogue. We confirmed the valid- ity of SCAINs with the results showing that clarity in the omitted dialogue was significantly lower the SCAIN condition and that difference was sig- nificantly higher in the SCAIN condition. We also confirmed the uniqueness of the SCAINs with the results showing that there was no significant depen- dence between the classification based on SCAINs and the classification based on importance. In fu- ture work, we will develop a conversation support system using SCAINs and evaluate the effective- ness of the system with an experiment in a real environment.
The SCAINs tended to introduce new topics not previously mentioned in the dialogues.
The statements following the SCAINs tended to contain pronouns and null anaphoras refer- ring to the terms in the preceding SCAINs.
On the other hand, the important statements had the following characteristics:

---

Limitations
expect dialogues with intermediate similarities be- tween the SCAINs and Non-SCAINs conditions will fall into intermediate results. The evaluation results show that the proposed method can extract dialogues with significantly different tendencies, at least for extreme conditions. However, the thresh- old needs to be determined on the basis of more detailed experiment that examines the relationship between the similarity score and the change in in- terpretation of the core statement.
The current research has a limitation in that the re- sults are highly dependent on the LLM used in the architecture. The outputs of LLMs are often biased (OpenAI, 2023), meaning that the rephrased state- ments do not cover all possible interpretations by the listener, especially if the omitted dialogue does not provide sufficient context. Furthermore, there may be individual differences in the interpretation of statement, depending the listener's back- ground and knowledge, which cannot be simulated with single prompt.
Finally, the uniqueness of the SCAINs needs to be evaluated in more diverse ways. In this pa- per, compared the classification based on the SCAINs with the classification based the im- portance of the statements evaluated by the LLM. There are other approaches, such as discourse struc- ture and cohesion analysis, that need to be com- pared with SCAIN extraction.
As shown in (OpenAI, 2023), the performance of GPT-4 in Japanese language processing is infe- rior to that of English. We speculate that GPT-3.5- turbo has similar tendency. However, we have confirmed that GPT-3.5-turbo shows sufficient ac- curacy for the task of rephrasing a sentence into more specific one.
Ethics Statement
Moreover, the SCAIN extractor requires four forward passes of the LLM for each utterance. Cur- rently, the computation time of the LLM cannot be ignored. However, the inference process of the LLM is becoming more efficient thanks to the progress in computer hardware and LLM research. Therefore, we expect that the SCAINs will be ex- tracted in much shorter time for real-time appli- cations.
We believe that there little concern about ethi- cal issues related to our research since we assume that the support system will only display some of the speakers' statements, not private information or harmful messages. Instead, we expect that peo- ple will understand each other if they recognize the possibility of different interpretations of their language.
Another limitation is that the current upproach to extracting SCAINs focuses only on the pair of statements immediately preceding the state- ment. We recognize that there need to identify more distant SCAINs. The current study aimed to establish the concept of SCAIN, so we defined the candidate statements as the immediately preceding statements for the first step. We will extend the method to identify more distant SCAINs as an im- portant future work. In addition, there will be an issuc of computational cost increase the num- ber of patterns of candidate statements. We will be able predict the change in interpretation of the core statement caused by the omission of more distant statements if we provide sufficient computa- tional resources. This extension will be done when the inference process of the LLM becomes more efficient thanks to research progress.
Acknowledgements
We would like to thank Mr. Aoto Tsuchiya at Keio University for the fundamental analysis of the chitecture.
This work was supported by JST CREST Grant number JPMICRI99 19A1, Japan.
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford. 1lya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates. Inc.
We predetermined the threshold for identifying SCAINs in the evaluation because we wanted to capture the tendency of SCAINs. The results of the validity experiment do not guarantee that the predetermined value of the threshold valid. We

---

Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. 2021. DialogSum: real-life scenario dialogue sum- marization dataset. In Findings of the Association for Computational Linguistics: ACL-UJCNLP 2021, pages 5062-5074, Online. Association for Computa- tional Linguisties.
Japanese texts, In Proceedings of the 28th Inter- national Conference on Computational Linguistics. pages 1323-1333 Barcelona, Spain (Online). Inter- national Committee on Computational Linguistics.
Bonnie Webber, Markus Egg. and Valia Kordoni. 2012. Discourse structure and language technology. Natu- ral Language Engineering, 18(4):437-49) 490.
Torsten Holmer. 2008. Discourse structure analysis of chat communication. Language Internet, 5(9).
Huiyuan Xie, Zhenghao Liu, Chenyan Xiong, Zhiyuan Liu, and Ann Copestake. 2021. TIAGE: A bench- mark for topic-shift aware dialog modeling, In Find- ings of the Association for Computational Linguis- rics: EMNLP 2021, pages 1684-1690, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Chaozhuo Li, Yu Wu, Wei Wu, Chen Xing, Zhoujun Li, and Ming Zhou. 2.16. Detecting context dependent messages in a conversational environment. In Pro- ceedings of COLING 2016, the 26th International Conference on Compulational Linguistics: Technical Papers, pages 1990-1999, Osaka, Japan. The COL- ING 2016 Organizing Committee.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: have dog, you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Pupers), pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics.
Chuan-Jie Lin, Chien-Wei Pao, Yen-Heng Chen, Chi- Ting Liu, and Hui-Huang Hsu. 2016. Ellipsis and coreference resolution in a computerized virtual pa- tient dialogue system. Journal of medical systems, 40:1-15.
N. Moratanch and S. Chitrakala. 2017. A survey on extractive text summarization, In 2017 International Conference on Computer, Communication and Signal Processing (ICCCSP), pages 1-6.
A Other prompt templates
We tested the following prompt templates to extract important statements:
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Rad- ford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hal- lacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, Madeleine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, Peter Welin- der, and Lilian Weng. 2022. Text and code em- beddings by contrastive pre-training arXiv preprint arXiv:220i.10005.
Select the two most important state- ments from the dialogue and write their numbers: this prompt template was used for the uniqueness evaluation.
Write down the two most important statements from the dialogue: when we tested this prompt template, the LLM did not always write down the statements accurately, which made the analysis difficult.
OpenAL. 2023. GPT-4 technical report.
Ryoichi Shibata, Shoya Matsumori, Yosuke Fukuchi, Tomoyuki Mackawa, Mitsuhiko Kimoto, and Michita Imai. 2023, Conversational context-sensitie ad gen- eration with few core cure-queries. ACM Transactions on Interactive Intelligent Systems.
Which statements are important?? when we tested this prompt template, the format of the output was unstable, which made the analysis difficult.
Hiroaki Sugiyama, Masahiro Mizukami, Tsunehiro Ari- moto, Hiromi Narimatsu, Yuya Chiba, Hideharu Nakajima, and Toyomi Meguro. 2023. Empirical analysis training strategies of buudd Japanese chit-chat systems. In 2022 IEEE Spoken Language Technology Workshop (SI.T), pages 685- 191. IREE.
Note that most statements in JPersonaChat con- tain some kind of information about the speaker's personal attributes. Therefore, is difficult to sub- jectively judge whether important statements are accurately extracted.
Yingcheng Sun and Kenneth Loparo. 2019. Topic shift detection in online discussions using structural con- text. In 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC), volume 1, pages 948-99 949.
B Qualitative results
Tables 9, 10, 11, 12, and 13 show five example dia- logues with the lowest similarity scores. Although there may be more than one pair of SCAINs in a dialogue, we only show the SCAINs with the lowest similarity score in the dialogue. We also
Nobuhiro Ueda, Daisuke Kawahara, and Sadao Kuro- hashi. 2020. baRT- cohesion analysis of

---

show the rephrased statements from the complete and omitted dialogues, as well the similarity scores between the rephrased statements. The di- alogues and rephrased statements were originally in Japanese and were translated into English for inclusion in this paper.
C Codes
The codes used extract the SCAINs and analyze the results are available at ht //gi thub. com/ imai- latoor htttoorrtttrysmmryysoooooo summary_scain.

---

Table 9: Example dialogue 1. Italic text represents the SCAINs. Underlined represents the statement following the SCAINs. Bold text represents the important statements.
A: Hey, can I talk to you for second?
B: What's up?
A: My boyfriend hates my Kansai dialect. He says sound like old lady.
B: live in the Kinki region, so I don't agree at all, but that's a bit harsh!
A: Definitely! But I'm a high school graduate, and there aren't many men who will go out with me because I'm moody.
B: I don't think that matters. I'm a college graduate, but one of my classmates married a
high school graduate. Anyway, cheer up!
A: Thanks! Oh, by the way, I just remembered something even worse that was said to me!
B: Even worse!
A: When wore leopard print tank top match my brown hair date, he asked me if was
a leopard.
B: Well, don't 't know if I can agree with that. But he's 's not so bad for a guy who can make a joke, is he?
A: Really? Is he? Do you think we' 'Il continue to get along as a couple?
B: Maybe you're a surprisingly good match!
A: see! That's s relief. I'm glad talked to you about it!
Rephrased statement from the complete dialogue
You guys understand each other well and respect each other so I am sure
you will get along together.
Rephrased statement from the omitted dialogue
Maybe the leopard print tank top to match your brown hair would look good!
Similarity score
0.775

---

Table 10: Example dialogue 2. Italic text represents the SCAINs. Underlined text represents the statement following the SCAINs. Bold text represents the important statements.
A: Hello. What you doing these days?
B: Lately, I've been working lot at my current company, which joined as a graduate hire. Because of the work, I'm always short of sleep. How about you?
A: It's tough being an employee. After changing jobs many times, I've found that being a freelancer suits me better.
B: Freelance. That's cool. I'm only good at being patient, so I'm going to stay with my current company for while.
A: You are respectable. I'm the kind of person who makes enemies. I'm sure that's my problem.
B: Do you have that harsh personality?
A: Well, think I'm good taking the lead, but many people think I'm too forward for college dropout.
B: That's bit of prejudice. But there are people like that everywhere. I'm glad you found something that fits your personality!
A: Thank you! That makes me feel better. By the way, have you been traveling lately?
B: Not at all. Have you?
A: Every day is like a trip for me. I travel all over Japan without settling down anywhere.
B: I'd like to live like that just once!
Rephrased statement from the complete dialogue
I've been so busy with work lately that haven't had time travel. Have you traveled lately?
Rephrased statement from the omitted dialogue
think am good at taking the lead, but actually, am still a work in progress and have many areas for improvement.
Similarity score
0.778

---

Table 11: Example dialogue 3. Italic text represents the SCAINs. Underlined text represents the core statement following the SCAINs. Bold text represents the important statements.
A: Hello. How you?
B: I've been well. How about you?
A: have been well too. On my days off, enjoy my hobby, photography.
B: Photography nice. live near the sea, so I often go for a walk.
A: So you live near the sea. must be beautiful. live near the mountains.
B: I see. like mountains, but used to live in Okinawa and really wanted to live near the sea.
A: You feel at home in place similar to where were born and raised. I am from Aomori, so understand how you feel.
B: That's right. also tend to get angry, seeing the ocean is good for calming me down.
A: Nature is very calming, isn' 't it? want go take pictures of the seaside next time.
B: Yes, I think the sea is also very nice because has many different faces.
A: Please let me know if you have any other recommendations for places to go.
B: Sure. By the way. am good at dancing, how about you?
A: have hardly ever danced. I like to watch dancing, though.
B: I see. But if you like to watch dancing, will show you next time.
A: would love to.
Rephrased statement from the complete dialogue
Could you please let know if there any other recommended seaside photo spots? Thank you very much in advance.
Rephrased statement from the omitted dialogue
Let's dance together sometime. like dancing too, I am looking forward to
Similarity score
0.783

---

Table 12: Example dialogue 4. Italic text represents the SCAINs. Underlined text represents the statement
following the SCAINs. Bold text represents the important statements.
A: Nice to meet you. Hello.
B: Hello. I'm member of the track team at my high school.
A: So you high school student! I am esthetician.
B: That's cool. Actually, I'm not good at getting up early, morning practice is hard for me.
A: feel for you. When was a club member, it was hard for me to get up early, too.
B: see. How did you manage to get up in the morning?
A: I set many alarms. My mother used to wake me up to0.
B: I live with my grandmother, so she wakes me up, but she wakes me up too softly.
A: I can imagine. By the way, my favorite author is Kenji Miyazawa, What books do you like
read?
B: I don't read many books, but know one of his books. It's a fantasy, isn't 't it?
A: You'r right. Please try reading
B: Okay. I'll look for
A: By all means! I'm also working hard to become a manicurist, so please do your best in your club activities.
Rephrased statement from the complete dialogue
Kenji Miyazawa's "Night on the Galactic Railroad" is very moving fantasy novel. Please read it.
Rephrased statement from the omitted dialogue
used have trouble getting up in the morning too, but had several alarms and even had my mother wake me up. It's sweet of your grandmother to wake you up, but often she wakes you up too softly. If that's the case, please try the alarm app use. It may be an effective way to wake up.
Similarity score
0.784

---

Table 13: Example dialogue 5. Italic text represents the SCAINs. Underlined text represents the statement
following the SCAINs. Bold text represents the important statements.
A: Long time no see. How is your work going?
B: Well, I'm leave of absence. I have the problem that I don't have enough money!
A: You don't have much money saved? 1 quit my job and now I don't have job, but did save some money, so guess I'll be okay for now.
B: envy you! I don't 't have any savings at all, and since I took a leave of absence, I can't even eat my favorite sushi, feel like crying.
A: That's ss too bad. Next time you pick out an outfit that looks good on me with short hair, I'll treat you!
B: Really? I'll pick one! have dark hair, so I'm often told I look good in Gothic Lolita, but
what kind of clothes do you like?
A: I don't think I'm Gothic Lolita at least. But do like skirts, I'll go with skirt dress that goes with short hair!
B: Wow, one-piece dresses are cute! Do you think short hair looks good on me, even though have a lot of hair?
A: I'm sociable person, but I'm not a flatterer and don't lic!
B: Then believe you! When shall we go?
A: Anytime! don't 't have a job.
B: Yeah, yeah, we're re both unemployed!
Rephrased statement from the complete dialogue
What kind of clothes do you like to wear? People often say look good in Gothic Lolita because have dark hair, but what styles do you think suit you?
Rephrased statement from the omitted dialogue
"Yes, but it's tough worrying about money. I'm unemployed right now, but did save some money, so I'm relieved for now."
Similarity score
0.785


Reviews:
[{"review_id": "RbuobCj68T", "reviewer": "Reviewer_g4QE", "summary": "", "questions": "1) Please illustrate details of applying the extractor in online conversation tools. When communication breakdowns occur, does the extractor have access to the complete dialogues and who will the extractor notify? senders and listeners? It would be much better if a detailed example could be included.  \n\n2) I am curious about the difference between SCAINs and utterances eliciting new topics (where a topic shift occurs). ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments and questions. We will address them comprehensively.\n\n> The work relies on LLM too much and the computational efficiency seems not high. Firstly, for each turn of dialogues, the LLM is used to paraphrase two core statements with two forward passes. Then the embedding model is employed to obtain the embeddings of the two core statements, two forward passes too. Four forward passes of LLM are required for each turn of dialogues.\n\nAs you pointed out, the SCAIN extractor requires four forward passes of the LLM for each utterance. Currently, the computation time of the LLM cannot be ignored. However, the inference process of the LLM is becoming more efficient thanks to the progress in computer hardware and LLM research. Therefore, we expect that the SCAINs will be extracted in a much shorter time for real-time applications. We will add this limitation of the current method to the limitation section.\n\n> Some experimental results seem not much significant and convincing. E.g., In line 264, the SCAINs condition is defined as the similarity being lower than 0.85 but the Non-SCAINs condition is defined as 0.9897. While the inconsistent of condition threshold is explained by authors: obtaining the same fraction of the statements as SCAINs, it can be replaced by randomly sampled from the statements with the similarity being higher than 0.85, at the meanwhile, keeping the fraction of statements the same.\n\nAs you pointed out, the results of the validity experiment do not guarantee that the predetermined value of the threshold is valid. We expect dialogues with intermediate similarities between the SCAINs and Non-SCAINs conditions will fall into intermediate results. However, it is important to set the extreme conditions for the basic analysis. The evaluation results show that the proposed method can extract dialogues with significantly different tendencies, at least for extreme conditions. We will add explanations to clarify that the paper describes the basic analysis to find out the tendencies with the extreme conditions.\n\n> Please illustrate details of applying the extractor in online conversation tools. When communication breakdowns occur, does the extractor have access to the complete dialogues and who will the extractor notify? senders and listeners? It would be much better if a detailed example could be included.\n\nWe apologize for the lack of explanation. The SCAINs do not indicate that there are communication breakdowns. Instead, if a listener missed hearing the SCAINs, there would be communication breakdowns. Therefore, the tool is designed to alert speakers to the possibility of communication breakdowns, not to detect actual communication breakdowns.\n\nThe tool will be useful for improving interaction between speakers. We assume that the tool will inform the sender of SCAINs which statements are SCAINs during conversations. Note that the SCAINs only indicate the possibility of communication breakdowns, so it is up to the sender how to use this information. Because the SCAINs are presented to the sender, the sender will pay close attention to verbal and nonverbal responses from the listener. The sender will be able to identify the communication breakdowns earlier and recover from the breakdowns by supplementing the SCAINs with more information. If the speakers do not use the tool, the sender will not care if his/her intent is conveyed to the listener and will continue with a one-way conversation. The tool will prevent such situations and let the speakers be aware of communication breakdowns.\n\nWe will add these details of the application tool to the discussion section. \n\n> I am curious about the difference between SCAINs and utterances eliciting new topics (where a topic shift occurs).\n\nThank you for the suggestion. We tested whether the SCAINs predict topic shifts using the TIAGE dataset. As a result, there is no significant relationship between the SCAINs and topic shifts. We show the cross-table and the results of a Chi-square test.\n\n|    |Topic shift|Non-topic shift|Total|\n|----|----|----|----|\n|SCAIN|18|66|84|\n|Non-SCAIN|297|1083|1380|\n|Total|315|1149|1464|\n\n$\\chi ^2 (1) = 0.0$, $p=1.0$.\n\nWe speculate that there is no significant relationship between the SCAINs and topic shifts because the topic shifts do not necessarily change the interpretation of the subsequent statement. Even if there is a topic shift, the listener may correctly interpret a statement that contains sufficient information about the new topic. We will add this evaluation and discussion to the appropriate sections.\n\n> Missing References:\n\n> Huiyuan Xie, Zhenghao Liu, Chenyan Xiong, Zhiyuan Liu, and Ann Copestake. 2021. TIAGE: A Benchmark for Topic-Shift Aware Dialog Modeling. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1684–1690, Punta Cana, Dominican Republic. Association for Computational Linguistics.\n\nBased on your suggestion, we will add a reference to the paper on topic-shift aware dialog modeling and expand the Related Work section to present research in this area.\n\n> Line 295: The second “omitted statements” should be “complete statements”.\n\nWe apologize for the confusion. We will revise the sentence to make it clearer.\n\nBefore revision: \n\nAt this point, we asked the raters not to change the answers for the omitted dialogue to ensure that they were rating the omitted dialogue without reading the **omitted statements**.\n\nAfter revision: \n\nAt this point, we asked the raters not to change the answers for the omitted dialogue to ensure that they were rating the omitted dialogue without reading the **complete dialogue**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) This work defines a new kind of utterance related to the interpretation of other statements, which can indicate whether a communication breakdown occurs within the current dialogue.  \n\n2) The experimental results and statistical analysis are comprehensive and clear.  ", "reasons_to_reject": "1) The work relies on LLM too much and the computational efficiency seems not high. Firstly, for each turn of dialogues, the LLM is used to paraphrase two core statements with two forward passes. Then the embedding model is employed to obtain the embeddings of the two core statements, two forward passes too. Four forward passes of LLM are required for each turn of dialogues.  \n\n2) Some experimental results seem not much significant and convincing. E.g., In line 264, the SCAINs condition is defined as the similarity being lower than 0.85 but the Non-SCAINs condition is defined as 0.9897. While the inconsistent of condition threshold is explained by authors: obtaining the same fraction of the statements as SCAINs, it can be replaced by randomly sampled from the statements with the similarity being higher than 0.85, at the meanwhile, keeping the fraction of statements the same.  ", "questions_to_authors": "1) Please illustrate details of applying the extractor in online conversation tools. When communication breakdowns occur, does the extractor have access to the complete dialogues and who will the extractor notify? senders and listeners? It would be much better if a detailed example could be included.  \n\n2) I am curious about the difference between SCAINs and utterances eliciting new topics (where a topic shift occurs). ", "paper_topic_and_main_contributions": "This paper studies the communication breakdowns problem which often occurs when a listener misses certain statements during online conversation interactions. Authors define the statements crucial for interpreting other statements as SCAINs (Statements Crucial for Awareness of Interpretive Nonsense). The contributions of it are two-fold: 1) proposes the method of extracting SCAINs. 2) validates that the SCAINs are crucial to the interpretation of dialogues and shows the relations and difference between SCAINs and important statements.  ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 295: The second “omitted statements” should be “complete statements”.  "}, "extra_text": {"Missing_References": "Huiyuan Xie, Zhenghao Liu, Chenyan Xiong, Zhiyuan Liu, and Ann Copestake. 2021. TIAGE: A Benchmark for Topic-Shift Aware Dialog Modeling. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1684–1690, Punta Cana, Dominican Republic. Association for Computational Linguistics. "}}, {"review_id": "Syl7f5L5hF", "reviewer": "Reviewer_qqvW", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and giving insightful comments. We will explain our motivation in detail.\n\n> They call these statements \"SCAINs\", explained in the title. A simple example of such a thing would be a sentence introducing a referent for a pronoun which was missed out, forcing the pronoun to refer to some eligible entity earlier in the conversation. The title of the paper mentions \"interpretive nonsense\" but all the examples are more to do with contextual effects on locally ambiguous utterances rather than sense/nonsense.\n\nThe \"nonsense\" abbreviated in SCAIN has a special meaning. A listener who has not heard SCAINs may interpret a subsequent statement differently from the speaker's primary intent. However, the listener is likely to believe that his/her interpretation is correct. The SCAIN extractor computes the change in interpretation by a listener who missed hearing the candidate statements. In this sense, SCAINs are different from locally ambiguous utterances. Even if a statement is locally ambiguous, the statement is not a SCAIN unless the interpretation changes due to mishearing. The SCAIN extractor picks up the statements when mishearing may change the interpretation of the subsequent statement, but the listener is not aware of the change, so the conversation support system will alert the speakers to pay attention to the different interpretations. We will add this explanation of our motivation to the Introduction section.\n\n> The paper lacks any convincing motivation for the need for such a system. \n\nWe assume that the conversation support system will present SCAINs to the speakers in a conversation, not to the listener who has misinterpreted the subsequent statement. The system is designed to prevent the speaker from continuing a one-way conversation when the speaker's utterance has been misinterpreted. When presented with the SCAINs, the speaker will pay close attention to the verbal and nonverbal responses of the listener. The speaker will be able to identify the communication breakdowns earlier and recover from the breakdowns by supplementing the SCAINs with more information. The system will prevent such situations and let the speakers be aware of communication breakdowns. \n\nNote that the SCAINs do not indicate that there are communication breakdowns. Instead, if a listener did not hear the SCAINs, there would be communication breakdowns. Therefore, the system is designed to alert speakers to the possibility of communication breakdowns, not to detect actual communication breakdowns. \n\nWe will add these details of the application system to the discussion section. \n\n> It sounds like a rather artificial problem, a perception enhanced by the weirdness of the dialogues used for illustration. Even allowing for translation effects, they sound like two bots talking, and on following up the references I found that they are indeed completely artificial.\n\nAs you pointed out, the dialogue for illustration was created artificially. However, communication breakdowns due to mishearing can occur in a real environment. There is no existing study that represents the change in interpretation caused by missing statements in the format of natural language. We believe that the significance of our paper lies in proposing the concept of SCAIN as an idea that can be useful for communication support.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is clearly written and the method for locating SCAINs might have applications elsewhere, such as in assembling a corpus of contextually-dependent ambiguities. ", "reasons_to_reject": "The paper lacks any convincing motivation for the need for such a system. It sounds like a rather artificial problem, a perception enhanced by the weirdness of the dialogues used for illustration. Even allowing for translation effects, they sound like two bots talking, and on following up the references I found that they are indeed completely artificial.", "questions_to_authors": "", "paper_topic_and_main_contributions": "\nThe aim of this paper is to identify statements in a dialogue such that if a listener missed them, the interpretation of\nsome subsequent utterances might change. They call these statements \"SCAINs\", explained in the title. A simple example\nof such a thing would be a sentence introducing a referent for a pronoun which was missed out, forcing the pronoun to\nrefer to some eligible entity earlier in the conversation.  The title of the paper mentions \"interpretive nonsense\" but\nall the examples are more to do with contextual effects on locally ambiguous utterances rather than sense/nonsense.\n\nThe paper proposes a method of locating such statements with the eventual aim of incorporating a mechanism into a\nconversational support system of some kind.\n\nTo do this they take a dialogue from an existing dataset and omit a statement at position j. This gives two versions of\nthe dialogue. Then an LLM is prompted to paraphrase the statement at the position j+1 in both versions, one with the\noriginal j position statement present and one without. Next the embeddings of the two paraphrased statements are\ncompared and if they are sufficiently far apart a SCAIN is hypothesised.\n\nFinally, human raters are used to validate the decision. Raters are also asked to judge the importance of the various\nstatements, and the results show that importance is a distinct dimension to the property of being a SCAIN.\n", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vsJzdzANy0", "reviewer": "Reviewer_e24W", "summary": "", "questions": "- page 3 \"we compute the similarity score between 2 vectors \" - which vectors do you mean? rephrased and original? candidate vs core?\n- The prompt for 'important statements' seems weak/unclear (what is an important statement) - were multiple prompt templates tested?\n-  are there limitations in the quality of gpt 3.5-turbo for japanese?\n- Would it be possible for the LLM to guess the correct interpretation in the omitted scenario even it was actually a SCAIN that was missing? For example because the correct interpretation is the 'regular' interpretation and the LLM did not pick up on the alternative interpretation from the dialogue?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for insightful comments and questions. We will address them comprehensively.\n\n> now only short distance SCAINS are considered (pair of consecutive statements), longer term SCAINs may be interesting as well, which the authors mention as a limitation as well\n\nAs you pointed out, the current method identifies SCAINs from the statements immediately preceding the core statement. We recognize that there is a need to identify more distant SCAINs. The current study aimed to establish the concept of SCAIN, so we defined the candidate statements as the immediately preceding statements for the first step. We will extend the method to identify more distant SCAINs as an important future work. In addition, there will be an issue of computational cost as we increase the number of patterns of candidate statements. We will be able to predict the change in interpretation of the core statement caused by the omission of more distant statements if we provide sufficient computational resources. This extension will be done when the inference process of the LLM becomes more efficient thanks to research progress. \n\nWe will add this explanation of our perspective to the limitation section.\n\n> how was the threshold determined - only \"on the basis of the observation of the rephrased statements\" but this is a bit vague - which is mentioned in the limitations as well\n\nAs you pointed out, we predetermined the threshold based on the observation of the rephrased statements in the current method. However, we will be able to determine a strict threshold by analyzing the relationship between the similarity score and the interpretation of the core statement in more detail.\n\n> As I understand it, the LLMs determine which are SCAINs (below 0.85) and these vary depending on the prompt (validity test: 65 SCAINs and uniqueness 125 SCAINs) for the same set. So what's the 'ground truth' then? Or is this difference because of the pairs (but why is this different in the uniqueness condition then)\n\nWe apologize for the lack of explanation. The SCAINs were identified using the same procedure for both validity and uniqueness evaluation. However, the ways of counting the SCAINs are different between the validity and the uniqueness evaluation. During the validity evaluation, we counted the number of pairs of SCAINs and identified 65 pairs. We considered the SCAINs as pairs because, for the validity evaluation, we wanted to analyze the tendency of the dialogues containing the SCAINs. On the other hand, during the uniqueness evaluation, we counted the number of SCAINs and identified 125 SCAINs. We considered the SCAINs as individual statements because, for the uniqueness evaluation, we wanted to analyze the relationship between the classifications of the statements in terms of being SCAINs and important statements. The number of SCAINs is less than twice the number of pairs because there are five overlapping pairs. We will add a more detailed explanation.\n\n> page 3 \"we compute the similarity score between 2 vectors \" - which vectors do you mean? rephrased and original? candidate vs core?\n\nWe apologize for the lack of explanation. We compared how similar the embedding vector of the rephrased statement is when using the complete dialogue prompt versus when using the omitted dialogue prompt. We will add a more detailed explanation.\n\n> The prompt for 'important statements' seems weak/unclear (what is an important statement) - were multiple prompt templates tested?\n\nWe tested the following prompt templates to extract important statements: \n\n1. Select the two most important statements from the dialogue and write their numbers (used for uniqueness evaluation). \n\n2. Write down the two most important statements from the dialogue (the LLM did not always write down the statements accurately, which made the analysis difficult). \n\n3. Which statements are important? (the format of the output was unstable, which made the analysis difficult). \n\nNote that most statements in JPersonaChat contain some kind of information about the speaker's personal attributes. Therefore, it is difficult to subjectively judge whether important statements are accurately extracted. We will add this explanation of prompt templates as an appendix.\n\n> are there limitations in the quality of gpt 3.5-turbo for japanese?\n\nAs shown in the following report, the performance of GPT-4 in Japanese language processing is inferior to that of English. We speculate that GPT-3.5-turbo has a similar tendency. However, we have confirmed that GPT-3.5-turbo shows sufficient accuracy for the task of rephrasing a sentence into a more specific one. We will add this explanation to the Limitations section.\n\n* OpenAI. 2023. GPT-4 technical report.\n\n> Would it be possible for the LLM to guess the correct interpretation in the omitted scenario even it was actually a SCAIN that was missing? For example because the correct interpretation is the 'regular' interpretation and the LLM did not pick up on the alternative interpretation from the dialogue?\n\nWhen SCAINs are omitted from the dialogue, the interpretation of the core statement changes significantly, making it difficult to predict the primary intent of the core statement. If the LLM is informed that some statements have been omitted, it can generate a possible interpretation of the core statement by predicting the content of the omitted statements. However, the generated interpretation is likely to be different from the primary intent because crucial information for interpreting the core statement has been lost. We will add this concern to the discussion section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- very interesting and unique topic that can be very relevant also for quality of summarization tasks\n- Clear explanation of the approach with mostly sufficient details for reproduction\n- Good reflection on limitations.", "reasons_to_reject": "- now only short distance SCAINS are considered (pair of consecutive statements), longer term SCAINs may be interesting as well, which the authors mention as a limitation as well\n- how was the threshold determined - only \"on the basis of the observation of the rephrased statements\" but this is a bit vague - which is mentioned in the limitations as well\n- As I understand it, the LLMs determine which are SCAINs (below 0.85) and these vary depending on the prompt (validity test: 65 SCAINs and uniqueness 125 SCAINs) for the same set. So what's the 'ground truth' then? Or is this difference because of the pairs (but why is this different in the uniqueness condition then)\n", "questions_to_authors": "- page 3 \"we compute the similarity score between 2 vectors \" - which vectors do you mean? rephrased and original? candidate vs core?\n- The prompt for 'important statements' seems weak/unclear (what is an important statement) - were multiple prompt templates tested?\n-  are there limitations in the quality of gpt 3.5-turbo for japanese?\n- Would it be possible for the LLM to guess the correct interpretation in the omitted scenario even it was actually a SCAIN that was missing? For example because the correct interpretation is the 'regular' interpretation and the LLM did not pick up on the alternative interpretation from the dialogue?", "paper_topic_and_main_contributions": "The authors introduce the concept of Statements Crucial for Awareness of Interpretive Nonsense (SCAINs) which are missed statements that lead to misunderstandings later in a dialogue. To determine which are these crucial statements they simulate missing information and generate texts with/without the missing information. SCAINs are not only important statements, but statements that may impact the interpretation. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces the task of identifying missed statements that lead to misunderstandings later in a dialogue. This is an interesting task; the proposed solution is novel and might have applications in other tasks. The paper is well-written, and the experiments are comprehensive. The problem statement needs further elaboration to provide convincing motivation for the need for such a system. The practical applications must be clarified.

The paper will benefit from all the modifications mentioned in the authors' rebuttals. Specifically, the authors need to add:

- Better motivation and problem statement, along with examples 
- Clarifications on the threshold setting
- Discussions on the computational cost 
- Explaining the differences between validity and uniqueness evaluations
Decision: Accept-Main