Def-DTS: Deductive Reasoning for Open-domain Dialogue Topic Segmentation
Authors: Seungmin Lee, Yongsang Yoo, Minhwa Jung, Min Song
Accepted at: acl25

Def-DTS: Deductive Reasoning for Open-domain Dialogue Topic Segmentation

Seungmin Lee1,4, Yongsang Yoo1,2, Minhwa Jung1,3, Min Song1,4*

1Yonsei University, 2LOTTE INNOVATE, 3LG Electronics, 4Onoma AI
1{elplaguister, 4n3mone, minalang, min.song}@yonsei.ac.kr
2yongsang.yoo@lotte.net, 3minalang.jung@lge.com, 4min.song@onomaaai.com

Abstract
Dialogue Topic Segmentation (DTS) aims to divide dialogues into coherent segments. DTS plays a crucial role in various NLP downstream tasks, but suffers from chronic problems: data shortage, labeling ambiguity, and incremental complexity of recently proposed solutions. On the other hand, despite advances in Large Language Models (LLMs) and reasoning strategies, these have rarely been applied to DTS. This paper introduces Def-DTS: Deductive Reasoning for Open-domain Dialogue Topic Segmentation, which utilizes LLM-based multi-step deductive reasoning to enhance DTS performance and enable case study using intermediate result. Our method employs a structured prompting approach for bidirectional context summarization, utterance intent classification, and deductive topic shift detection. In the intent classification process, we propose the generalizable intent list for domain-agnostic dialogue intent classification. Experiments in various dialogue settings demonstrate that Def-DTS consistently outperforms traditional and state-of-the-art approaches, with each subtask contributing to improved performance, particularly in reducing type 2 error. We also explore the potential for autolabeling, emphasizing the importance of LLM reasoning techniques in DTS.1

1 Introduction

Dialogue Topic Segmentation (DTS) is a task that aims to divide a dialogue into segments where each segment focuses on a coherent topic. Figure 1 shows an example of topic shift within a single dialogue. DTS is crucial for various natural language processing (NLP) tasks, including response prediction (Lin et al., 2020; Xu et al., 2021b; He et al., 2022), response generation (Li et al., 2016; Xu et al., 2021a; Liu et al., 2022), dialogue state tracking (Das et al., 2024), summarization (Bokaei et al., 2016; Chen and Yang, 2020; Qi et al., 2021; Zhong et al., 2022), question answering (Yoon et al., 2018; Zhang et al., 2022), and machine reading comprehension (Ma et al., 2024).

Despite growing interest, DTS suffers from several chronic challenges. First, the shortage of annotated data has led most recent DTS studies to an unsupervised way, which generally yields suboptimal performance. Second, ambiguity in segment labeling has hindered the development of effective approaches. Lastly, recent studies: DialSTART (Gao et al., 2023), UR-DTS (Hou et al., 2024) have proposed incremental approaches that require more parameter and complexity, enhancing previous studies: CSM (Xing and Carenini, 2021) and DialSTART (Gao et al., 2023), respectively. This progression indicates that DTS is a challenging and often underestimated problem.

While DTS struggles with its complexities, NLP has witnessed significant advancements with the rise of Large Language Models (LLMs) and reasoning methodologies. However, even considering the robust problem-solving skills of these LLMs and the challenges posed by DTS, reasoning strategies are rarely applied in the DTS area. This is because

![An example of a topic shift in a conversation. The cues for a topic shift are highlighted in red.](page_1012_370_393_312.png)

Figure 1: An example of a topic shift in a conversation. The cues for a topic shift are highlighted in red.
DTS has been treated largely as a lightweight sub-task in NLP. Nonetheless, with the rise of AI-driven chat services, the demand for more advanced DTS modules is growing. LLMs with reasoning capabilities are well suited to meet this need, making LLM-based DTS a viable solution.

To establish clear criteria for topic shifts and simplify complex subtasks, we propose Def-DTS: a deductive reasoning approach for open-domain dialogue topic segmentation using LLM-based multi-step reasoning. Def-DTS employs structured prompting to guide LLMs through bidirectional context summarization, utterance intent classification, and deductive topic shift detection at the utterance level, with an emphasis on domain-agnostic intent classification.

To evaluate Def-DTS, we test it on three dialogue datasets spanning open-domain and task-oriented settings across three key metrics. Our method consistently outperforms traditional and state-of-the-art unsupervised, supervised, and prompt-based techniques by a significant margin. Ablation studies confirm that each subtask improves overall performance, with intermediate intent classification particularly improving true-positive detection. Finally, we explore LLM-based auto-labeling for DTS. Our contributions are fourfold:

• We introduce LLM reasoning techniques to DTS for the first time, consolidating insights from previous methodologies into a coherent and deductive prompt design.

• We reformulate DTS as an utterance-level intent classification task by implementing intent classification as a core component of a multi-step reasoning process, enabling flexible and task-agnostic prompting.

• Our method empirically demonstrates superior performance across nearly all comparative baselines, underscoring the efficacy of prompt engineering in DTS.

• Through an in-depth analysis of our approach’s reasoning results, we shed light on the challenges LLM reasoning faces in DTS and discuss the possibility of using LLM as a DTS auto-labeler.

2 Related Works

2.1 Dialogue Topic Segmentation

Dialogue topic segmentation divides dialogues into coherent topic units. Due to limited annotated datasets, researchers have largely relied on unsupervised approaches despite their complexity (Xing and Carenini, 2021). Early methods like TextTiling (Hearst, 1997) detected topic shifts via lexical similarity, later improved with embedding-based methods (Song et al., 2016).

Recent research emphasizes topical coherence and similarity scoring. CSM (Xing and Carenini, 2021) leverages BERT-based coherence, while Dial-START (Gao et al., 2023) incorporates SimCSE (Gao et al., 2021) for topic similarity. SumSeg (Artemiev et al., 2024) extracts key information via summaries and applies smoothing to handle topic variations. UR-DTS (Hou et al., 2024) enhances segmentation by rewriting utterances to recover missing references. Despite these advances, data scarcity and performance limitations persist.

To address this, SuperDialSeg (Jiang et al., 2023) introduces a supervised approach using large-scale DGDS datasets (Feng et al., 2020, 2021). However, the available datasets remain limited for open-domain conversations. LLMs are also influencing DTS. S3-DST (Das et al., 2024) applies structured prompting for dialogue state tracking and segmentation, but lacks general applicability to diverse DTS settings.

2.2 Reasoning Strategy at LLM Inference

The advancement of LLMs (Brown et al., 2020) has led to research on the integration of System 2 reasoning (Kahneman, 2011), including in-context learning (Brown et al., 2020) and chain-of-thought prompting (Wei et al., 2022). These techniques enable LLMs to tackle complex tasks, such as symbolic mathematics (Yang et al., 2024), retrieval-augmented generation (Lewis et al., 2020), and data generation (Adler et al., 2024). Studies show that intermediate reasoning steps significantly enhance performance in areas like multi-hop reasoning (Wang et al., 2023) and math problem-solving (Imani et al., 2023). Building on this, we integrate LLM-based reasoning into DTS, leveraging its potential to improve segmentation accuracy in this inherently complex task.
Figure 2: Prompt configuration and overall flow of our method: Def-DTS. (a) We utilize general intent list including the intent-specific examples to enable domain-agnostic categorization. (b) We employ the xml structured input-output format to stably provide the dialogue. (c) We instruct LLM to process the multi-step reasoning for each utterance in a inference.

3 Def-DTS

3.1 Overall Flow

Our method (Figure 2) applies multiple subtasks to each utterance using a structured prompt format. The prompt template (Figure 2b) includes four parts: valid label list, task description, output format, and input. As shown in Figure 2c, it comprises three main subtasks: (i) bidirectional context extraction, (ii) utterance intent classification, and (iii) deductive topic shift classification. Each subtask is executed deductively, with further details in Algorithm 1.

3.2 Structured Format

We benchmark previous DST research (Das et al., 2024) by using a structured template in XML format. Prompt formatting improves adherence to labeling instructions by providing structured and well-defined formats, which improve the alignment with the task and facilitate easier parsing, thereby reducing the need for post-processing (Das et al., 2024). We believe that our deductive, multi-step method can similarly benefit from these advantages, enhancing alignment with labeling instructions and reducing postprocessing efforts.

To this end, we utilize an XML-based structured prompt to standardize LLM output, enhancing parsing efficiency, and minimizing post-processing. The input template (Figure 2b) organizes utterances in <UX> elements, each containing the speaker’s information and the content of the utterance. The out-

Algorithm 1 Def-DTS

Require: CLASSIFYINTENT, to classify the intent of an utterance given its context and intent pool
Require: SUMMARIZE, to summarize a given dialogue context
Require: IntentPool \( X = \{x_1, x_2, \ldots, x_m\} \)
Require: StructuredDialogue \( D = \{d_i\}_{i=1}^N \), where \( d_i = \{u_i, s_i\} \) (utterance \( u_i \) and speaker \( s_i \))
Ensure: Results \( R = \{r_i\}_{i=1}^N \), where \( r_i = \{P_i, Q_i, X_i, T_i\} \) (context \( P_i \), \( Q_i \), intent \( X_i \), topic shift \( T_i \))
1: \( R \leftarrow \emptyset, S \leftarrow \text{STRUCTUREDIALOGUE}(D) \)
2: for \( i \leftarrow 1 \) to \( N \) do
3:     \( P_i \leftarrow \text{EXTRACTCONTEXT}(D, \max(1, i - 2), i) \)
4:     \( Q_i \leftarrow \text{EXTRACTCONTEXT}(D, i + 1, \min(i + 3, N)) \)
5:     \( X_i \leftarrow \text{CLASSIFYINTENT}(D[i], P_i, Q_i, X) \)
6:     \( T_i \leftarrow \text{CLASSIFYTOPICSHIFT}(X_i) \)
7:     \( r_i \leftarrow \{P_i, Q_i, X_i, T_i\} \)
8:     \( R \leftarrow R \cup \{r_i\} \)
9: end for
10: return \( R \)

11: function CLASSIFYTOPICSHIFT(x)
12:     return \( x \in \{"introduce\_topic", "change\_topic"\} ? "YES" : "NO" \)
13: end function

14: function EXTRACTCONTEXT(D, start, end)
15:     context \( \leftarrow \text{SUMMARIZE}(D[start : end]) \)
16:     return "\(U_{start-U_{end}}\)", context
17: end function
put (Figure 2c) follows the same structured format, ensuring consistent labeling. A complete template is available in Appendix A.1.

3.3 Bidirectional Context Extraction
In the first stage of Def-DTS, we instruct the LLM to summarize both the preceding and subsequent dialogues for each utterance. Considering bidirectional context is commonly used in many methods such as the BERT architecture (Devlin et al., 2019) and frequently employed in unsupervised settings (Gao et al., 2023; Hou et al., 2024), this strategy has proven effective for understanding context in dialogue. Although Das et al., 2024 only extracts the preceding context to prevent contextual forgetting, we improve upon this by extracting both the preceding and subsequent contexts to enable context-aware dialogue topic segmentation and prevent contextual forgetting.

We opt for a fixed window size to ensure applicability in unsupervised and real-world environments without predefined segments, as it offers a robust approach when segment boundaries are not available. As shown in Figure 2c(i), we instruct the model to summarize no more than two preceding turns using the <preceding_context> tag and no more than three subsequent turns using the <subsequent_context> tag. This window size of -2:-1 and 1:3 was chosen to balance context informativeness and token efficiency, an approach that is similar to the method used in (Gao et al., 2023). By summarizing each context range, we aim to conserve tokens while maintaining nuanced topic relationships.

Each element is composed of <range> . . . </range> for defining the summary scope and <context>. . . </context> for the actual summary. Summarizing in this manner ensures that we capture relevant dialogue while keeping the context concise. Our experiments reveal that this approach effectively facilitates the observation of nuanced topic relationships.

<table>
  <tr>
    <th>Intent</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>JUST COMMENT</td>
    <td>Commenting on the preceding context without any asking. Not a topic shift</td>
  </tr>
  <tr>
    <td>JUST ANSWER</td>
    <td>Answering preceding utterance. Not a topic shift</td>
  </tr>
  <tr>
    <td>DEVELOP TOPIC</td>
    <td>Developing the conversation to similar and inclusive sub-topics. Not a topic shift</td>
  </tr>
  <tr>
    <td>INTRODUCE TOPIC</td>
    <td>Introducing a relevant but different topic. A topic shift</td>
  </tr>
  <tr>
    <td>CHANGE TOPIC</td>
    <td>Completely changing the topic. A topic shift</td>
  </tr>
</table>

Table 1: Utterance intent list for open-domain dialogue. Using this list, we categorize the utterance and deductively classify the topic shift label.

the context of Task Oriented Dialogue (TOD) (Liu and Lane, 2016; Chen and Luo, 2023; Fong and Ong, 2023). TOD typically involves datasets that are specific to a particular domain, making generalization to other domains or more open-ended conversations challenging.

As a way to address this issue, we find that Xie et al., 2021 identified five patterns of conversational responses that are utilized in their annotation guidelines. These patterns reflect the natural characteristics of the utterances, enabling intent classification for various forms of dialogue. Detailed intent patterns and descriptions are in Table 1.

Inspired by this research, we instruct model to detect topic change through utterance intent classification. Specifically, after the bidirectional context extraction, as shown in Figure 2c(ii), model classifies the utterance into an intent of the predefined general intent pool (upper box of Figure 2a), considering previously generated bidirectional context.

Additionally, we enhance the model’s understanding of the intents by providing example dialogue for each intent, as shown in the General intent pool in Figure 2a. The intent-specific examples provide the model with a helpful guideline detecting topic changes, allowing it to derive results in the subsequent subtask, deductive topic shift. Also in order to prove the significance of our intent labels through statistical frequency analysis from the traditional text segmentation method, which can be found in Section 5.5. We observed significant performance improvements across various dialogue settings using this technique, enabling a detailed analysis of each intent. The details of intent pool construction are presented in Appendices A.3–A.5.
3.5 Deductive Topic Shift Classification

Finally, the model predicts whether the topic is changing based on the deductive guidelines from the previous intent classification result, as shown in Figure 2c(iii). This task is processed in an enforced manner, and the model deductively outputs the pre-determined label based on the intent classification results of the previous step, according to the tail of description (i.e. Not a topic shift or A topic shift) for each intent described in Table 1.

We explicitly instruct the model to process this step for two reasons. First, it simplifies the parsing process, making the task easy to handle. Second, it ensures that the model explicitly outputs the result of primary goal, allowing it to stay focused on the main objective(DTS), while working through the subtasks.

4 Experiments

4.1 Datasets

<table>
  <tr>
    <th>Dataset</th>
    <th># Sample</th>
    <th colspan="3">Utterance per Dial.</th>
    <th colspan="2">Segment per Dial.</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>avg</th>
    <th>min</th>
    <th>max</th>
    <th>avg</th>
    <th>avg.len</th>
  </tr>
  <tr>
    <td>TIAGE</td>
    <td>100</td>
    <td>15.6</td>
    <td>14</td>
    <td>16</td>
    <td>4.2</td>
    <td>3.8</td>
  </tr>
  <tr>
    <td>SuperDialseg</td>
    <td>1322</td>
    <td>12.1</td>
    <td>7</td>
    <td>19</td>
    <td>4.0</td>
    <td>3.0</td>
  </tr>
  <tr>
    <td>Dialseg711</td>
    <td>711</td>
    <td>26.2</td>
    <td>7</td>
    <td>47</td>
    <td>4.9</td>
    <td>5.4</td>
  </tr>
</table>

Table 2: Statistics of datasets for dialogue topic segmentation.

We evaluated our method on three datasets: TIAGE, SuperDialseg, Dialseg711 to verify the performance of our method in both open-domain and task-oriented settings. Dataset statistics are presented in Table 2.

TIAGE (Xie et al., 2021) is the only publicly available dataset with topically segmented daily conversations, derived from PersonaChat (Zhang et al., 2018) and designed to model topic shifts in open-domain dialogue. SuperDialseg (Jiang et al., 2023) is a large-scale dialogue segmentation dataset based on document-grounded corpora, offering a framework for identifying segmentation points in document-based dialogues. Dialseg711 (Xu et al., 2021b) is a real-world dialogue dataset auto-labeled from MultiWOZ (Budzianowski et al., 2018) and Stanford Dialog Dataset (Eric et al., 2017), created by joining dialogues with distinct topics, resulting in clear topical differences and low coherency at segment boundaries due to its synthetic nature.

4.2 Evaluation Metrics

As early studies (Xing and Carenini, 2021; Jiang et al., 2023; Artemiev et al., 2024) did, we leverage \( P_k \) error (Beeferman et al., 1997), WindowDiff (WD) error (Pevzner and Hearst, 2002), and the f1 score. The \( P_k \) error is calculated by counting the existence of a misallocated segment with a sliding window of predictions. The WD error is calculated by comparing the number of boundaries within the sliding window of gold labels and predictions. Note that the lower \( P_k \) or WD means the higher performance.

4.3 Comparison Methods

We propose a sophisticated DTS method based on prompt engineering way and compare its performance with various unsupervised, supervised, and LLM-based methodologies. First, we compare our method with a random baseline that arbitrarily assigns segment boundaries based on a randomly chosen number of segments. Next, we compare it to notable unsupervised learning methods based on Text-Tiling like Coherence Scoring Model (CSM) (Xing and Carenini, 2021), DialSTART (Gao et al., 2023), SumSeg (Artemiev et al., 2024). We also compare supervised learning methodologies. We selected the basic BERT model(Devlin et al., 2019), the advanced and high-performing RoBERTa model(Liu, 2019), RetroTS-T5(Xie et al., 2021) system for our comparative analysis. Finally, we compare our approach with recently introduced LLM-based methods. We applied these methods using gpt-4o.\footnote{https://platform.openai.com/docs/models/gpt-4o} We selected the PlainText prompts performed in SuperDialseg(Jiang et al., 2023) and the prompt of S3-DST(Das et al., 2024). The details of the methodology utilized for each of the actual comparisons are discussed in the Appendix D.1.

4.4 Experimental Results

The experimental results are presented in Table 3. Def-DTS consistently demonstrated superior performance among LLM-based methods and achieved state-of-the-art results on the TIAGE and Dialseg711 datasets, which are closely aligned with our objective of analyzing general open-domain dialogues. In contrast, other LLM-based methods showed lower performance not only compared to supervised learning and some unsupervised methods, indicating that simply using an LLM is not a
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">TIAGE</th>
    <th colspan="3">SuperDialseg</th>
    <th colspan="3">Dialseg711</th>
  </tr>
  <tr>
    <th>P<sub>k</sub>↓</th>
    <th>WD↓</th>
    <th>F1↑</th>
    <th>P<sub>k</sub>↓</th>
    <th>WD↓</th>
    <th>F1↑</th>
    <th>P<sub>k</sub>↓</th>
    <th>WD↓</th>
    <th>F1↑</th>
  </tr>
  <tr>
    <th colspan="10">Unsupervised Learning Methods</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>0.526</td>
    <td>0.664</td>
    <td>0.237</td>
    <td>0.494</td>
    <td>0.649</td>
    <td>0.266</td>
    <td>0.533</td>
    <td>0.714</td>
    <td>0.204</td>
  </tr>
  <tr>
    <td>TextTiling</td>
    <td>0.469</td>
    <td>0.488</td>
    <td>0.204</td>
    <td>0.441</td>
    <td>0.453</td>
    <td>0.388</td>
    <td>0.470</td>
    <td>0.493</td>
    <td>0.245</td>
  </tr>
  <tr>
    <td>TextTiling+Glove</td>
    <td>0.486</td>
    <td>0.511</td>
    <td>0.236</td>
    <td>0.519</td>
    <td>0.524</td>
    <td>0.353</td>
    <td>0.399</td>
    <td>0.438</td>
    <td>0.436</td>
  </tr>
  <tr>
    <td>CSM</td>
    <td><b>0.400</b></td>
    <td><b>0.420</b></td>
    <td><b>0.427</b></td>
    <td>0.462</td>
    <td>0.467</td>
    <td>0.381</td>
    <td>0.278</td>
    <td>0.302</td>
    <td>0.610</td>
  </tr>
  <tr>
    <td>DialSTART</td>
    <td>0.482</td>
    <td>0.528</td>
    <td>0.378</td>
    <td><b>0.373</b></td>
    <td><b>0.412</b></td>
    <td><b>0.627</b></td>
    <td><b>0.179</b></td>
    <td><b>0.198</b></td>
    <td><b>0.733</b></td>
  </tr>
  <tr>
    <td>SumSeg</td>
    <td>0.482</td>
    <td>0.496</td>
    <td>0.075</td>
    <td>0.479</td>
    <td>0.485</td>
    <td>0.119</td>
    <td>0.477</td>
    <td>0.483</td>
    <td>0.070</td>
  </tr>
  <tr>
    <th colspan="10">Supervised Learning Methods</th>
  </tr>
  <tr>
    <td>BERT</td>
    <td>0.418</td>
    <td>0.435</td>
    <td>0.124</td>
    <td>0.214</td>
    <td>0.225</td>
    <td>0.725</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>RoBERTa</td>
    <td><b>0.265</b></td>
    <td><b>0.287</b></td>
    <td>0.572</td>
    <td><b>0.185</b></td>
    <td><b>0.192</b></td>
    <td><b>0.784</b></td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>RetroTS-T5</td>
    <td>0.280</td>
    <td>0.317</td>
    <td><b>0.576</b></td>
    <td>0.227</td>
    <td>0.237</td>
    <td>0.733</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <th colspan="10">LLM-based Methods</th>
  </tr>
  <tr>
    <td>Plain Text</td>
    <td>0.445</td>
    <td>0.485</td>
    <td>0.185</td>
    <td>0.412</td>
    <td>0.427</td>
    <td>0.048</td>
    <td>0.333</td>
    <td>0.353</td>
    <td>0.010</td>
  </tr>
  <tr>
    <td>S3-DST<sub>uttr</sub></td>
    <td>0.439</td>
    <td>0.498</td>
    <td>0.265</td>
    <td>0.442</td>
    <td>0.469</td>
    <td>0.404</td>
    <td>0.087</td>
    <td>0.109</td>
    <td>0.790</td>
  </tr>
  <tr>
    <td>Def-DTS (Ours)</td>
    <td><b>0.232</b></td>
    <td><b>0.256</b></td>
    <td><b>0.699</b></td>
    <td><b>0.315</b></td>
    <td><b>0.324</b></td>
    <td><b>0.686</b></td>
    <td><b>0.015</b></td>
    <td><b>0.018</b></td>
    <td><b>0.979</b></td>
  </tr>
</table>

Table 3: Performances on three datasets. Due to absence of train and validation split for Dialseg711 dataset. There are no report in dialseg711’s supervised learning part. The best results for each method group are highlighted in bold. The best performances around all method are indicated as red colored text.

guarantee of success.

TIAGE Compared to S3-DST<sub>uttr</sub>, the recent LLM-based method, our method achieved reductions of more than 0.2 in both P<sub>k</sub> and WD errors, along with an impressive increase of more than 0.4 in the F1 score. Furthermore, Def-DTS outperformed even the supervised approaches in TIAGE, surpassing them in all metrics by over 10%, thus highlighting the effectiveness of our approach in achieving high performance across various dialogue environments without additional training, even in domain-agnostic settings.

SuperDialseg Our method outperformed all unsupervised methods. Although it showed lower performance compared to models trained using supervised learning, it achieved the best results among unsupervised methods that use prompt-based techniques.

Dialseg711 Our approach also delivered superior performance. Notably, it surpassed the strongest LLM-based approach, S3-DST<sub>uttr</sub>, underscoring the general applicability and robustness of our method.

Overall, these consistent improvements across all tested datasets confirm the robust effectiveness of our approach for diverse open-domain dialogue scenarios, demonstrating that it not only excels in unsupervised settings, but also surpasses previously leading LLM-based methods. This further establishes potential of our method for delivering high performance even under challenging, domain-agnostic conditions.

5 Analysis and Discussion

5.1 Ablation Study

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">TIAGE</th>
  </tr>
  <tr>
    <th>P<sub>k</sub>↓</th>
    <th>WD↓</th>
    <th>F1↑</th>
  </tr>
  <tr>
    <td>w/o all</td>
    <td>0.295</td>
    <td>0.333</td>
    <td>0.605</td>
  </tr>
  <tr>
    <td>w/o intent</td>
    <td>0.316</td>
    <td>0.342</td>
    <td>0.524</td>
  </tr>
  <tr>
    <td>w/o examples</td>
    <td>0.287</td>
    <td>0.308</td>
    <td>0.617</td>
  </tr>
  <tr>
    <td>w/o context</td>
    <td>0.263</td>
    <td>0.296</td>
    <td>0.682</td>
  </tr>
  <tr>
    <td>w/o bidirectional</td>
    <td>0.269</td>
    <td>0.301</td>
    <td>0.659</td>
  </tr>
  <tr>
    <td>Def-DTS</td>
    <td><b>0.232</b></td>
    <td><b>0.256</b></td>
    <td><b>0.699</b></td>
  </tr>
</table>

Table 4: Ablation study.

To assess the contributions of each part of our approach, we performed an ablation study. The results are shown in Table 4. In the w/o all components case, the model is instructed to detect topic
shifts without context extraction or intent classification. In the w/o intent case, the model detects topic shifts after context extraction for each utterance. We observe that w/o intent performs worse than w/o all components, indicating that relying solely on dialogue context for topic shift prediction does not yield optimal performance. In the w/o examples case, this is essentially Def-DTS but without examples for intent. w/o examples performed better than w/o intent, showing that processing context into intent before using it for topic shift prediction provides a significant advantage. In the w/o context case, the model is instructed to detect topic shifts after intent classification for each utterance, which is the opposite of the w/o intent case. This result demonstrates that intent classification, when supported by appropriate examples, has a significant impact on topic shift prediction for individual utterances. In the w/o bidirectional context case, the subsequent context is not considered at the context extraction step. Compared to both the Def-DTS and w/o context case, this case showed lower performance, highlighting that considering bidirectional context is crucial for intent classification and topic shift detection. In summary, each module of Def-DTS contributes to performance improvement, and when all modules are applied, they work synergistically to yield a substantial increase in performance.

5.2 Comparative Study for Structured Format

<table>
  <tr>
    <th rowspan="2">I/O Format</th>
    <th colspan="3">TIAGE</th>
  </tr>
  <tr>
    <th>P<sub>k</sub>↓</th>
    <th>WD↓</th>
    <th>F1↑</th>
  </tr>
  <tr>
    <td>NL</td>
    <td>0.274</td>
    <td>0.302</td>
    <td>0.640</td>
  </tr>
  <tr>
    <td>JSON</td>
    <td>0.259</td>
    <td>0.292</td>
    <td>0.658</td>
  </tr>
  <tr>
    <td>XML</td>
    <td>0.232</td>
    <td>0.256</td>
    <td>0.699</td>
  </tr>
</table>

Table 5: Comparative study for structured format.

To examine the impact of structured I/O formats not covered in the ablation study, we represent identical prompts in three different formats—Natural Language (NL), JSON, and XML—and compare their performance in Table 5. The results show that the structured formats, XML and JSON, not only offer parsing advantages but also outperform NL in terms of task performance. These findings empirically support the hypothesis proposed by Das et al., 2024 that XML can provide structural benefits in dialogue processing.

5.3 Intent Classification Accuracy

<table>
  <tr>
    <th>Intent</th>
    <th>TP</th>
    <th>FP</th>
    <th>TN</th>
    <th>FN</th>
    <th>Acc</th>
  </tr>
  <tr>
    <td>JUST_COMMENT</td>
    <td>0</td>
    <td>1</td>
    <td>498</td>
    <td>35</td>
    <td>0.93</td>
  </tr>
  <tr>
    <td>JUST_ANSWER</td>
    <td>0</td>
    <td>1</td>
    <td>456</td>
    <td>23</td>
    <td>0.95</td>
  </tr>
  <tr>
    <td>DEVELOP_TOPIC</td>
    <td>0</td>
    <td>0</td>
    <td>119</td>
    <td>47</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>INTRODUCE_TOPIC</td>
    <td>189</td>
    <td>68</td>
    <td>0</td>
    <td>0</td>
    <td>0.73</td>
  </tr>
  <tr>
    <td>CHANGE_TOPIC</td>
    <td>21</td>
    <td>6</td>
    <td>0</td>
    <td>0</td>
    <td>0.78</td>
  </tr>
</table>

Table 6: Intent-level confusion matrix for TIAGE benchmark.

As our method deduces topic shifts directly from utterance intent, analyzing the intent classification results is crucial. We examine the confusion matrix (Table 6) to identify the utterance types that the model struggles with. Since there is no ground truth about intent and only topic shift labels are provided, correctness is determined by whether the predicted intent aligns with the topic shift label. For instance, if an utterance is a topic shift but classified as JUST_COMMENT for intent and NO for topic shift, it counts as a False Negative (FN).

Most results performed well in topic shift classification, except for two cases: positives in JUST_COMMENT and JUST_ANSWER. These findings indicate that the model’s primary challenge lies in distinguishing subtle topic differences as actual shifts, a more significant factor in performance degradation than other utterance types. Analysis of additional datasets (SuperDialseg, Dialseg711) is in Appendix B.

5.4 Intent Level Comparison

![Two scatter plots labeled (a) MATCHED INTENT and (b) MISMATCHED CASE, showing accuracy and co-error count for different methodologies.](page_1012_1342_388_388.png)

Figure 3: (a) MATCHED INTENT indicates the accuracy of the other methodologies for grouped utterances by our intent classification process only in the true cases of our method. (b) MISMATCHED CASE indicates the co-error count of the other methodologies with our methods for only in the false cases of our method.
We compared the performance of various approaches across different intent categories predicted by our method, as shown in Figure 3. In (a) MATCHED INTENT, for utterances without a topic shift, other methodologies achieved approximately 80-85% accuracy when our method was also correct. However, for utterances that involve a topic shift, the accuracy of other methods dropped to around 20-50% for the correct case of our method. In (b) MISMATCHED CASE, for utterances without a topic shift, other methods correctly classified 50% of the cases where our method was incorrect. However, for utterances with a topic shift, other methods failed to classify 80% of the cases that our method also missed. This demonstrates that detecting utterances with actual topic shifts is considerably more challenging than detecting those without topic shifts. Our method outperforms the other methodologies by roughly 20% in cases without topic shifts, and by over 40% in cases with topic shifts. In summary, while our approach improves accuracy across all cases, it shows even greater improvement when handling utterances with topic shifts.

5.5 Linguistic Test for Intent Labels

To demonstrate the impact of intent labels on topic shifts, we adopted methods from statistical linguistics. Traditional text segmentation uses pauses, cue words, and referential noun phrases to identify boundaries (Passonneau and Litman, 1997). Galley et al., 2003 found a significant correlation between cue phrases and topic segmentation. Building on this, we hypothesized that cue words in labels like "introduce topic" and "change topic" correlate with their overall frequency in the data. A \( \chi^2 \) test yielded \( \chi^2(32) = 76.2263, p < 0.001 \), confirming a significant relationship. This validates our labels as linguistically rich markers of discourse boundaries and provides a criterion for selecting topic-shift data in new datasets.

5.6 Performance Comparison for Local LLMs

We conducted experiments using local LLMs instead of GPT-4, specifically using Lama 3.1 and Qwen 2.5, with the exact model names listed in Table 7. For the experiments, we tested three prompts: Plain Text, S3-DST\(_{utt}\), and Def-DTS. To ensure efficiency, we randomly sampled 100 examples from each dataset. Def-DTS achieved the highest performance in all datasets in this experiment. In the case of Qwen, formatting errors were observed in all datasets. Although plain is an unstructured method, it did not have errors, but its performance remained comparatively lower. These results demonstrate that Def-DTS maintains high accuracy in different LLMs. Experiments with more LLMs are presented in the Appendices C.1–C.2.

5.7 Discussion for Possibility of Auto-Labeling

As various auto-labeling methodologies have been proposed to date, we assess prompt engineering could potentially serve as a viable auto-labeling methodology. We conducted a preliminary experiment to assess the feasibility of using prompt engineering for DTS. We compared the segment labels generated by GPT-4 for Def-DTS with the correct labels using Cohen’s Kappa score. The results showed Kappa scores of 0.485 for TIAGE, 0.429 for SuperDialseg, and 0.975 for Dialseg711, indicating moderate agreement for TIAGE and SuperDialseg, and almost perfect agreement for Dialseg711. Notably, our labeling result for TIAGE exceeded the 0.479 agreement score observed between actual human annotators. While improvements are needed given the moderate agreement, these findings suggest that our approach can still function as a minimal annotator.

6 Conclusion

Previous approaches to DTS have been constrained by several challenges, including data shortage, ambiguity of segment labeling, and increasingly complex model architectures. Concurrently, The promising approach of reasoning with LLMs has yet to be explored in the context of DTS. To address these issues, we propose Def-DTS, that leverages LLMs in conjunction with sophisticated reasoning strategies. Def-DTS incorporates bidirectional context extraction, a crucial component in previous research, along with the novel task of utterance intent classification. This approach demonstrates significant performance improvements in both the open-domain dialogue setting and the task-oriented dialogue setting. Its efficacy across diverse datasets is enhanced through the provision of dataset-specific examples in the utterance intent classification task, enabling adaptable performance in varied dialogue contexts. Through its primary findings and diverse analysis, we demonstrate the efficacy of LLM-reasoning as a promising approach to DTS. It not only highlights the potential of our method, but it also statistically delineates
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">TIAGE</th>
    <th colspan="3">SuperDialseg</th>
    <th colspan="3">Dialseg711</th>
  </tr>
  <tr>
    <th>P<sub>k↓</sub></th>
    <th>WD↓</th>
    <th>F1↑</th>
    <th>P<sub>k↓</sub></th>
    <th>WD↓</th>
    <th>F1↑</th>
    <th>P<sub>k↓</sub></th>
    <th>WD↓</th>
    <th>F1↑</th>
  </tr>
  <tr>
    <td>Plain Text + Llama</td>
    <td>0.472</td>
    <td>0.515</td>
    <td>0.215</td>
    <td>0.492</td>
    <td>0.495</td>
    <td>0.026</td>
    <td>0.350</td>
    <td>0.373</td>
    <td>0.032</td>
  </tr>
  <tr>
    <td>Plain Text + Qwen</td>
    <td>0.495</td>
    <td>0.533</td>
    <td>0.162</td>
    <td>0.485</td>
    <td>0.487</td>
    <td>0.059</td>
    <td>0.422</td>
    <td>0.434</td>
    <td>0.012</td>
  </tr>
  <tr>
    <td>S3-DST<sub>uttr</sub> + Llama</td>
    <td>0.456</td>
    <td>0.474</td>
    <td>0.143</td>
    <td>0.490</td>
    <td>0.512</td>
    <td>0.072</td>
    <td>0.158</td>
    <td>0.190</td>
    <td>0.553</td>
  </tr>
  <tr>
    <td>S3-DST<sub>uttr</sub> + Qwen</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Def-DTS + Llama</td>
    <td><b>0.307</b></td>
    <td><b>0.339</b></td>
    <td><b>0.552</b></td>
    <td><b>0.384</b></td>
    <td><b>0.385</b></td>
    <td><b>0.432</b></td>
    <td><b>0.029</b></td>
    <td><b>0.039</b></td>
    <td><b>0.941</b></td>
  </tr>
  <tr>
    <td>Def-DTS + Qwen</td>
    <td>0.327</td>
    <td>0.345</td>
    <td>0.530</td>
    <td>0.433</td>
    <td>0.434</td>
    <td>0.171</td>
    <td>0.102</td>
    <td>0.208</td>
    <td>0.729</td>
  </tr>
</table>

Table 7: Performances on the local LLMs. We employed Llama-3.1-70B-Instruct and Qwen2.5-72B-Instruct for Llama and Qwen, respectively. The case of best performance across all method are highlighted in bold.

the challenges to be addressed in future research. In subsequent investigations, we intend to explore the feasibility of automated labeling for DTS and examine the potential integration of DTS with other NLP downstream tasks through LLM reasoning.

7 Limitations

Firstly, though we have demonstrated the significance of the current intent labels by statistical linguistic experiments, we cannot entirely rule out the possibility that more suitable intent labels exist. Additionally, as we provide a first approach for selecting representative examples, a more in-depth exploration of methodologies for selecting optimal examples remains a future step of this research. In order to improve the quality of intents and examples in a variety of dialogue settings, the fundamental problem must be addressed first, namely the provision of quality datasets for DTS. Dialogue should include thorough labeling criteria and realistic dialogue domains to address our limitations. However, human labeling is not only still expensive but also carries the risk of inconsistent or ambiguous labeling. We believe that automated labeling using LLMs with a sophisticated guideline will play a crucial role in creating a more sustainable and reliable DTS environment.

Acknowledgments

This research was supported by the Culture, Sports and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism in 2024 (Project Name: Developing a generative AI story platform for Fanfiction, Project Number: RS-2024-00442270). This work was partly supported by an IITP grant funded by the Korean Government (MSIT) (No. RS-2020-II201361 , Artificial Intelligence Graduate School Program (Yonsei University))

References

Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. 2024. Nemotron-4 340b technical report. arXiv preprint arXiv:2406.11704.

Aleksei Artemiev, Daniil Parinov, Alexey Grishanov, Ivan Borisov, Alexey Vasilev, Daniil Muravetskii, Aleksey Rezykh, Aleksei Goncharov, and Andrey Savchenko. 2024. Leveraging summarization for unsupervised dialogue topic segmentation. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 4697–4704, Mexico City, Mexico. Association for Computational Linguistics.

Doug Beeferman, Adam Berger, and John Lafferty. 1997. Text segmentation using exponential models. In Second Conference on Empirical Methods in Natural Language Processing.

Mohammad Hadi Bokaei, Hossein Sameti, and Yang Liu. 2016. Extractive summarization of multi-party meetings through discourse segmentation. Natural Language Engineering, 22(1):41–72.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language
models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.

Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Ultes Stefan, Ramadan Osman, and Milica Gašić. 2018. Multiwoz - a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP).

Jiaao Chen and Diyi Yang. 2020. Multi-view sequence-to-sequence models with conversational structure for abstractive dialogue summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4106–4118, Online. Association for Computational Linguistics.

Yan Chen and Zhenghang Luo. 2023. Pre-trained joint model for intent classification and slot filling with semantic feature fusion. Sensors, 23(5).

Sarkar Snigdha Sarathi Das, Chirag Shah, Mengting Wan, Jennifer Neville, Longqi Yang, Reid Andersen, Georg Buscher, and Tara Safavi. 2024. S3-DST: Structured open-domain dialogue segmentation and state tracking in the era of LLMs. In Findings of the Association for Computational Linguistics ACL 2024, pages 14996–15014, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Mihail Eric, Lakshmi Krishnan, Francois Charette, and Christopher D. Manning. 2017. Key-value retrieval networks for task-oriented dialogue. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 37–49, Saarbrücken, Germany. Association for Computational Linguistics.

Song Feng, Siva Sankalp Patel, Hui Wan, and Sachindra Joshi. 2021. MultiDoc2Dial: Modeling dialogues grounded in multiple documents. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6162–6176, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Song Feng, Hui Wan, Chulaka Gunasekara, Siva Patel, Sachindra Joshi, and Luis Lastras. 2020. doc2dial: A goal-oriented document-grounded dialogue dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8118–8128, Online. Association for Computational Linguistics.

Hannah Fong and Ethel Ong. 2023. Evaluating Chat-GPT for joint intent detection and slot filling: Zero-shot vs. few-shot prompting. In Proceedings of the 23rd Philippine Computing Science Congress (PCSC).

Michel Galley, Kathleen R. McKeown, Eric Fosler-Lussier, and Hongyan Jing. 2003. Discourse segmentation of multi-party conversation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 562–569, Sapporo, Japan. Association for Computational Linguistics.

Haoyu Gao, Rui Wang, Ting-En Lin, Yuchuan Wu, Min Yang, Fei Huang, and Yongbin Li. 2023. Unsupervised dialogue topic segmentation with topic-aware contrastive learning. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23, page 2481–2485, New York, NY, USA. Association for Computing Machinery.

Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseck-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.

Wanwei He, Yinpei Dai, Yinhe Zheng, Yuchuan Wu, Zheng Cao, Dermot Liu, Peng Jiang, Min Yang, Fei Huang, Luo Si, Jian Sun, and Yongbin Li. 2022. Galaxy: A generative pre-trained model for task-oriented dialog with semi-supervised learning and explicit policy injection. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):10749–10757.

Marti A. Hearst. 1997. Text tiling: Segmenting text into multi-paragraph subtopic passages. Computational Linguistics, 23(1):33–64.

Xia Hou, Qifeng Li, and Tongliang Li. 2024. An unsupervised dialogue topic segmentation model based on utterance rewriting. arXiv preprint arXiv:2409.07672.

Shima Imani, Liang Du, and Harsh Shrivastava. 2023. MathPrompter: Mathematical reasoning using large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 37–42, Toronto, Canada. Association for Computational Linguistics.

Junfeng Jiang, Chengzhang Dong, Sadao Kurohashi, and Akiko Aizawa. 2023. SuperDialseg: A large-scale dataset for supervised dialogue segmentation.
In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4086–4101, Singapore. Association for Computational Linguistics.

Daniel Kahneman. 2011. Thinking, fast and slow. macmillan.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474. Curran Associates, Inc.

Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Austin, Texas. Association for Computational Linguistics.

Ting-En Lin, Hua Xu, and Hanlei Zhang. 2020. Discovering new intents via constrained deep adaptive clustering with cluster refinement. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8360–8367.

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437.

Bing Liu and Ian Lane. 2016. Attention-based recurrent neural network models for joint intent detection and slot filling. arXiv preprint arXiv:1609.01454.

Che Liu, Rui Wang, Junfeng Jiang, Yongbin Li, and Fei Huang. 2022. Dial2vec: Self-guided contrastive learning of unsupervised dialogue embeddings. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7272–7282, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Yinhan Liu. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.

Xinbei Ma, Yi Xu, Hai Zhao, and Zhuosheng Zhang. 2024. Multi-turn dialogue comprehension from a topic-aware perspective. Neurocomputing, 578:127385.

Rebecca J. Passonneau and Diane J. Litman. 1997. Discourse segmentation by human and automated means. Computational Linguistics, 23(1):103–139.

Lev Pevzner and Marti A. Hearst. 2002. A critique and improvement of an evaluation metric for text segmentation. Computational Linguistics, 28(1):19–36.

MengNan Qi, Hao Liu, YuZhuo Fu, and Ting Liu. 2021. Improving abstractive dialogue summarization with hierarchical pretraining and topic segment. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1121–1130, Punta Cana, Dominican Republic. Association for Computational Linguistics.

Yiping Song, Lili Mou, Rui Yan, Li Yi, Zinan Zhu, Xi-aohua Hu, and Ming Zhang. 2016. Dialogue session segmentation by embedding-enhanced texttiling. Interspeech 2016, pages 2706–2710.

Jinyuan Wang, Junlong Li, and Hai Zhao. 2023. Self-prompted chain-of-thought on large language models for open-domain multi-hop reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2717–2731, Singapore. Association for Computational Linguistics.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc.

Huiyuan Xie, Zhenghao Liu, Chenyan Xiong, Zhiyuan Liu, and Ann Copestake. 2021. TIAGE: A benchmark for topic-shift aware dialog modeling. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1684–1690, Punta Cana, Dominican Republic. Association for Computational Linguistics.

Linzi Xing and Giuseppe Carenini. 2021. Improving unsupervised dialogue topic segmentation with utterance-pair coherence scoring. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 167–177, Singapore and Online. Association for Computational Linguistics.

Jun Xu, Zeyang Lei, Haifeng Wang, Zheng-Yu Niu, Hua Wu, and Wanxiang Che. 2021a. Discovering dialog structure graph for coherent dialog generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1726–1739, Online. Association for Computational Linguistics.

Yi Xu, Hai Zhao, and Zhuosheng Zhang. 2021b. Topic-aware multi-turn dialogue modeling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14176–14184.

An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 2024. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122.
Seunghyun Yoon, Joongbo Shin, and Kyomin Jung. 2018. Learning to rank question-answer pairs using hierarchical recurrent encoder with latent topic clustering. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1575–1584, New Orleans, Louisiana. Association for Computational Linguistics.

Sai Zhang, Yuwei Hu, Yuchuan Wu, Jiaman Wu, Yongbin Li, Jian Sun, Caixia Yuan, and Xiaojie Wang. 2022. A slot is not built in one utterance: Spoken language dialogs with sub-slots. In Findings of the Association for Computational Linguistics: ACL 2022, pages 309–321, Dublin, Ireland. Association for Computational Linguistics.

Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204–2213, Melbourne, Australia. Association for Computational Linguistics.

Ming Zhong, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2022. Dialoglm: Pre-trained model for long dialogue understanding and summarization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11765–11773.
A  Prompts

A.1  Prompt Template

<table>
  <tr>
    <th>Prompt Template</th>
  </tr>
  <tr>
    <td>&lt;valid_utterance_intent&gt; &lt;item&gt;<br>&lt;name&gt;JUST_COMMENT&lt;/name&gt;<br>&lt;desc&gt;Commenting on the preceding context without any asking. Not a topic shift&lt;/desc&gt;<br>&lt;example&gt;<br>&lt;speaker1&gt;My dad works for the New York Times.&lt;/speaker1&gt;<br>&lt;speaker2&gt;Oh wow! You know, I dabble in photography; maybe you can introduce us sometime.&lt;/speaker2&gt;<br>&lt;speaker1&gt;Photography is the greatest art out there. (not a topic shift)&lt;/speaker1&gt;<br>&lt;/example&gt;<br>&lt;/item&gt;<br>&lt;item&gt;<br>&lt;name&gt;JUST_ANSWER&lt;/name&gt;<br>&lt;desc&gt;Answering preceding utterance. Not a topic shift&lt;/desc&gt;<br>&lt;example&gt;<br>&lt;speaker1&gt;Do you teach cooking? &lt;/speaker1&gt;<br>&lt;speaker2&gt;No, since I’m a native of Mexico, I teach Spanish. (not a topic shift)&lt;/speaker2&gt;<br>&lt;/example&gt;<br>&lt;/item&gt;<br>&lt;item&gt;<br>&lt;name&gt;DEVELOP_TOPIC&lt;/name&gt;<br>&lt;desc&gt;Developing the conversation to similar and inclusive sub-topics. Not a topic shift&lt;/desc&gt;<br>&lt;example&gt;<br>&lt;speaker1&gt;Pets are cute!&lt;/speaker1&gt;<br>&lt;speaker2&gt;I heard that Huskies are difficult dogs to take care of. (not a topic shift)&lt;/speaker2&gt;<br>&lt;/example&gt;<br>&lt;/item&gt;<br>&lt;item&gt;<br>&lt;name&gt;INTRODUCE_TOPIC&lt;/name&gt;<br>&lt;desc&gt;Introducing a relevant but different topic. A topic shift&lt;/desc&gt;<br>&lt;example&gt;<br>&lt;speaker1&gt;You are an artist? What kind of art, I do American Indian stuff.&lt;/speaker1&gt;<br>&lt;speaker2&gt; I love to eat too, sometimes too much. (a topic shift)&lt;/speaker2&gt;<br>&lt;/example&gt;<br>&lt;/item&gt;<br>&lt;item&gt;<br>&lt;name&gt;CHANGE_TOPIC&lt;/name&gt;<br>&lt;desc&gt;Completely changing the topic. A topic shift&lt;/desc&gt;<br>&lt;example&gt;<br>&lt;speaker1&gt;What do you do for fun?&lt;/speaker1&gt;<br>&lt;speaker2&gt;I drive trucks so me and my buds go truckin in the mud.&lt;/speaker2&gt;<br>&lt;speaker1&gt;Must be fun! My version of that’s running around a library!&lt;/speaker1&gt;<br>&lt;speaker2&gt;That’s cool! I love that too. Do you have a favourite animal? Chickens are my favourite. I love them. (topic shift)&lt;/speaker2&gt;<br>&lt;/example&gt;<br>&lt;/item&gt;<br>&lt;/table&gt;<br>&lt;/valid_utterance_intent&gt;<br>&lt;valid_topic_shift_label&gt;
Prompt Template
<item>
<name>YES</name>
<desc>The current utterance has **weak OR no topical** relation to the preceding conversation context OR is the first utterance in the conversation, marking the beginning of a new dialogue segment.</desc>
</item>
<item>
<name>NO</name>
<desc>The current utterance has **relevant OR equal** topic to the preceding conversation context.</desc>
</item>
</valid_topic_shift_label>

## TASK ##
You are given a dialogue starting with U. From utterance number 0, you have to answer the following sub-tasks for each utterance.
1. Summarize the preceding and subsequent context in <=3 sentences seperately
The range of the context should be previous or next 1-3 utterances except for the case of the first or last utterance.
For example, given current utterance number is 2, preceding range is 0-1, subsequent range is 3-5.
2. Output the utterance_intent
Use the list <valid_utterance_intent> ... </valid_utterance_intent> to categorize utterance.
Consider topical difference between preceding and subsequent context.
3. Output the topic_shift_label
Use the list <valid_topic_shift_label> ... </valid_topic_shift_label>.

## OUTPUT FORMAT ##
<U{utterance number}>
<preceding_context>
<range>{range of utterances referred in context}</range>
<context>{context of the previous 1-3 utterances}</context>
</preceding_context>
<subsequent_context>
<range>{range of utterances referred in context}</range>
<context>{context of the next 1-3 utterances}</context>
</subsequent_context>
<utterance_intent>{valid utterance intent}</utterance_intent>
<topic_shift_label>{valid topic shift label}</topic_shift_label>
</U{utterance number}>

## INPUT ##
{XML-structured dialogue}

## OUTPUT ##

Table 8: We provide prompt template for main dataset: TIAGE. Each dataset has different characteristics to other datasets, so we modified intent pool from original template for each dataset.
A.2 Intent Labels for other datasets

<table>
  <tr>
    <th>Intent</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>DIFFERENT QUESTION</td>
    <td>Questioning about something that is not similar or topically different to preceding context. A topic shift.</td>
  </tr>
  <tr>
    <td>RELEVANT QUESTION</td>
    <td>Questioning about something that is similar or topically coherent to preceding context. Not a topic shift.</td>
  </tr>
  <tr>
    <td>ANSWERING</td>
    <td>Answering preceding utterance. Not a topic shift</td>
  </tr>
  <tr>
    <td>ADDITIONAL COMMENT</td>
    <td>An additional comment from the same speaker in addition to a previous utterance. Not a topic shift</td>
  </tr>
</table>

Table 9: Utterance intent list for SuperDialseg dataset.

For the Dialseg711 dataset, we delete an intent named INTRODUCE_TOPIC from the original intent list. For the SuperDialseg dataset, the topic shift occurs when the utterance refers to a different document to previous utterance. As shown in Table 9, we completely change the intent from the original to fit to document grounded dialogue setting based topic transition.

A.3 Modification of TIAGE Response Patterns

<table>
  <tr>
    <th>Scenario</th>
    <th>Topic Shift</th>
  </tr>
  <tr>
    <td>Commenting on the previous context</td>
    <td>No</td>
  </tr>
  <tr>
    <td>Question answering</td>
    <td>No</td>
  </tr>
  <tr>
    <td>Developing the conversation to sub-topics</td>
    <td>No</td>
  </tr>
  <tr>
    <td>Introducing a relevant but different topic</td>
    <td>Yes</td>
  </tr>
  <tr>
    <td>Completely changing the topic</td>
    <td>Yes</td>
  </tr>
</table>

Table 10: Intent labels in different dialog scenarios.

TIAGE (Xie et al., 2021) is originally designed for real-time topic shift detection without using a subsequent context. Consequently, its conversation response pattern list cannot be applied as is for our full-dialogue segmentation task. To address this, we propose two methods. First, instead of classifying each utterance with only its immediate context, we retrieve both preceding and subsequent context to inform our decisions. This bidirectional view captures how an utterance relates to what was said before and what follows, enabling more precise intent classification. Second, we adapt TIAGE’s original response pattern list to our classification objectives by reorganizing patterns (e.g., Asking) and refining the description of each intent (e.g., Relevant, Inclusive). This transformation ensures a more granular detection of whether an utterance continues the topic or shifts in a subtle way. By employing these enhancements, we achieve significantly better performance in segmenting topics across entire dialogues, surpassing the results obtained by using TIAGE’s list unaltered.

A.4 Construction of Intent Examples

For the TIAGE dataset, we used the examples directly from their paper. For other datasets, we randomly selected parts of the conversation from the Train split that adhere to the following rules:

• Select 2–3 consecutive utterances for each example.

• Ensure that the final utterance in the example corresponds to the target utterance intent.

• Extract all the examples from a single dialogue.

• Keep the utterance lengths concise (within 100 characters).

This domain-independent guideline can be an initial pathfinder to tailor the best examples in dialogue, though it may not be perfect.

A.5 Intent Pool Construction Principles

In constructing the intent labels, descriptions, and illustrative examples, we found that prompts yield the most generalizable and effective performance when they exhibit the following two key characteristics:

• The intent list allows for mutually exclusive categorization of utterances.

• The explanations and examples clearly differentiate the degree of topic shifts induced by each utterance.

The intent specification adopted in Appendix A.1 satisfies these conditions, and the intents tailored for other datasets (Appendix A.2) were likewise designed to adhere to these principles.

B Analysis on other datasets

B.1 Ablation Study

As shown in Table 11, we found that the absence of some module leads to performance degradation. For efficient evaluation, we used 100 randomly sampled dialogue for the ablation study for SuperDialseg and Dialseg711 each. This data is the same as we used in Section 5.6. Especially on the SuperDialseg dataset, we observed a consistent improvement by adding any subtask. However, with
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">SuperDialseg</th>
    <th colspan="3">Dialseg711</th>
  </tr>
  <tr>
    <th>P.</th>
    <th>WD.</th>
    <th>F1</th>
    <th>P.</th>
    <th>WD.</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>w/o all</td>
    <td>0.378</td>
    <td>0.382</td>
    <td>0.467</td>
    <td>0.007210</td>
    <td>0.009416</td>
    <td>0.987245</td>
  </tr>
  <tr>
    <td>w/o intent</td>
    <td>0.363</td>
    <td>0.364</td>
    <td>0.448</td>
    <td>0.0093486</td>
    <td>0.127211</td>
    <td>0.701330</td>
  </tr>
  <tr>
    <td>w/o examples</td>
    <td>0.338</td>
    <td>0.341</td>
    <td>0.646</td>
    <td>0.005826</td>
    <td>0.012322</td>
    <td>0.984733</td>
  </tr>
  <tr>
    <td>w/o context</td>
    <td>0.327</td>
    <td>0.331</td>
    <td>0.635</td>
    <td><b>0.005800</b></td>
    <td><b>0.008464</b></td>
    <td><b>0.989770</b></td>
  </tr>
  <tr>
    <td>Def-DTS</td>
    <td><b>0.317</b></td>
    <td><b>0.322</b></td>
    <td><b>0.674</b></td>
    <td>0.009024</td>
    <td>0.013738</td>
    <td>0.982143</td>
  </tr>
</table>

Table 11: Ablation study for identifying effectiveness of each subtask within our method.

respect to Dialseg711 dataset, the existence of context extraction module is crucial to performance improvement. Even the w/o case surpasses our full-attached method. we conjecture that the issue is due to over-concentration for local context, same as the result of ablation study for TIAGE, moreover, on the case of dialseg711 having clear topic shift signal, just predicting label is enough to solve the problem. After all, our intent classification module elevates the performance across all datasets.

B.2 Intent Classification Accuracy

<table>
  <tr>
    <th rowspan="2">Intent</th>
    <th colspan="4">SuperDialseg</th>
    <th rowspan="2">Acc</th>
  </tr>
  <tr>
    <th>TP</th>
    <th>FP</th>
    <th>TN</th>
    <th>FN</th>
  </tr>
  <tr>
    <td>DIFFERENT_QUESTION</td>
    <td>2456</td>
    <td>688</td>
    <td>83</td>
    <td>192</td>
    <td>0.74</td>
  </tr>
  <tr>
    <td>RELEVANT_QUESTION</td>
    <td>1</td>
    <td>0</td>
    <td>811</td>
    <td>989</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>ANSWERING</td>
    <td>0</td>
    <td>2</td>
    <td>7819</td>
    <td>168</td>
    <td>0.98</td>
  </tr>
  <tr>
    <td>ADDITIONAL_COMMENT</td>
    <td>0</td>
    <td>0</td>
    <td>1264</td>
    <td>211</td>
    <td>0.86</td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Intent</th>
    <th colspan="4">Dialseg711</th>
    <th rowspan="2">Acc</th>
  </tr>
  <tr>
    <th>TP</th>
    <th>FP</th>
    <th>TN</th>
    <th>FN</th>
  </tr>
  <tr>
    <td>JUST_COMMENT</td>
    <td>0</td>
    <td>6</td>
    <td>5067</td>
    <td>14</td>
    <td>0.996</td>
  </tr>
  <tr>
    <td>JUST_ANSWER</td>
    <td>0</td>
    <td>6</td>
    <td>7675</td>
    <td>8</td>
    <td>0.998</td>
  </tr>
  <tr>
    <td>DEVELOP_TOPIC</td>
    <td>0</td>
    <td>3</td>
    <td>2359</td>
    <td>13</td>
    <td>0.993</td>
  </tr>
  <tr>
    <td>CHANGE_TOPIC</td>
    <td>2708</td>
    <td>66</td>
    <td>60</td>
    <td>0</td>
    <td>0.976</td>
  </tr>
</table>

Table 12: Intent-level confusion matrix for other datasets.

We conducted a detailed accuracy analysis for the other datasets and the results are presented in Table 12.

For SuperDialseg, as shown in Table 9, four new intent pools were applied. ADDITIONAL_COMMENT, ANSWERING, and RELEVANT_QUESTION are classified as non-topic shift cases, whereas the case of DIFFERENT_QUESTION is classified as a topic shift case. However, the instruction following was not well executed for the case of DIFFERENT_QUESTION. For the case of RELEVANT_QUESTION, the following instruction was well executed with one exception, but its accuracy was relatively low. The difference in explanations between the case of RELEVANT_QUESTION and DIFFERENT_QUESTION could be linked to the actual dataset characteristics and topic changes. In contrast, the cases of ANSWERING and ADDITIONAL_COMMENT showed significantly high classification accuracy. This comparison suggest that improving Question-type intents will lead to an overall improvement in performance.

For Dialseg711, overall accuracy was higher compared to other datasets where the following deductive instruction not executed for less than 1% of utterances. For the results of the three intents, excluding the case of CHANGE_TOPIC, 18% of DEVELOP_TOPIC, 30% of JUST_COMMENT, and 42% of JUST_ANSWER cases among all false cases were misclassified due to errors in instruction following. It is believed that this issue can be resolved through additional instructions or prompt modifications for instruction following.

C Additional Experiments

C.1 Experiments for Local LLMs

Def-DTS leverages LLM reasoning capabilities, so model size significantly affects performance. Table 7 shows that S3-DST on Qwen 70B had formatting errors, which discouraged us from testing smaller LLMs initially. However, considering the growing capabilities and applications of sLLMs, we conducted additional experiments on: Llama 8B, Qwen 7B, Qwen 32B.

The experimental result is shown in table 13. Although Def-DTS struggled with smaller models and dialseg711, it showed greater improvements with larger models by leveraging LLM reasoning capability. However, we acknowledge that applying Def-DTS to smaller LLMs would require adjustments such as additional parameter modification.

C.2 Performance for Closed-source LLMs

We evaluate the performance of Def-DTS with additional closed-source LLMs, including Deepseek-R1(Guo et al., 2025) and Deepseek-V3(Liu et al., 2024), as shown in Table 14. All three models achieve better performance than the LLM-based approaches reported in Section 4.4, and—with the exception of R1—even outperform the supervised baseline. This demonstrates the general applicability of Def-DTS across a variety of closed-source LLMs. Interestingly, although R1 is specialized for reasoning, it underperforms compared to GPT-4o and V3, which are not explicitly designed for such capabilities. We attribute this to our prompting
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Model</th>
    <th colspan="4">TIAGE</th>
    <th colspan="4">SuperDialseg</th>
    <th colspan="4">Dialseg711</th>
  </tr>
  <tr>
    <th>P<sub>k</sub>↓</th>
    <th>WD↓</th>
    <th>F1↑</th>
    <th>Error</th>
    <th>P<sub>k</sub>↓</th>
    <th>WD↓</th>
    <th>F1↑</th>
    <th>Error</th>
    <th>P<sub>k</sub>↓</th>
    <th>WD↓</th>
    <th>F1↑</th>
    <th>Error</th>
  </tr>
  <tr>
    <td rowspan="3">Plain Text</td>
    <td>Llama 8B</td>
    <td>0.529</td>
    <td>0.604</td>
    <td>0.303</td>
    <td>1</td>
    <td>0.497</td>
    <td>0.504</td>
    <td>0.036</td>
    <td>0</td>
    <td>0.350</td>
    <td>0.373</td>
    <td>0.032</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Qwen 7B</td>
    <td>0.509</td>
    <td>0.563</td>
    <td>0.249</td>
    <td>2</td>
    <td>0.517</td>
    <td>0.522</td>
    <td>0.132</td>
    <td>1</td>
    <td>0.486</td>
    <td>0.513</td>
    <td>0.069</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Qwen 32B</td>
    <td>0.476</td>
    <td>0.515</td>
    <td>0.221</td>
    <td>0</td>
    <td>0.466</td>
    <td>0.471</td>
    <td>0.083</td>
    <td>0</td>
    <td>0.391</td>
    <td>0.415</td>
    <td>0.015</td>
    <td>0</td>
  </tr>
  <tr>
    <td rowspan="3">S3-DST<sub>utr</sub></td>
    <td>Llama 8B</td>
    <td>0.460</td>
    <td>0.460</td>
    <td>0.018</td>
    <td>0</td>
    <td>0.472</td>
    <td>0.494</td>
    <td>0.076</td>
    <td>0</td>
    <td>0.188</td>
    <td>0.196</td>
    <td>0.705</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Qwen 7B</td>
    <td>0.563</td>
    <td>0.860</td>
    <td>0.299</td>
    <td>57</td>
    <td>0.578</td>
    <td>0.952</td>
    <td>0.351</td>
    <td>47</td>
    <td>0.582</td>
    <td>0.759</td>
    <td>0.093</td>
    <td>71</td>
  </tr>
  <tr>
    <td>Qwen 32B</td>
    <td>0.430</td>
    <td>0.455</td>
    <td>0.211</td>
    <td>22</td>
    <td>0.431</td>
    <td>0.443</td>
    <td>0.106</td>
    <td>57</td>
    <td>0.237</td>
    <td>0.270</td>
    <td>0.497</td>
    <td>83</td>
  </tr>
  <tr>
    <td rowspan="3">Def-DTS</td>
    <td>Llama 8B</td>
    <td>0.474</td>
    <td>0.525</td>
    <td>0.218</td>
    <td>1</td>
    <td>0.473</td>
    <td>0.527</td>
    <td>0.398</td>
    <td>0</td>
    <td>0.246</td>
    <td>0.268</td>
    <td>0.464</td>
    <td>10</td>
  </tr>
  <tr>
    <td>Qwen 7B</td>
    <td>0.462</td>
    <td>0.465</td>
    <td>0.022</td>
    <td>20</td>
    <td>0.473</td>
    <td>0.474</td>
    <td>0.071</td>
    <td>14</td>
    <td>0.162</td>
    <td>0.170</td>
    <td>0.715</td>
    <td>44</td>
  </tr>
  <tr>
    <td>Qwen 32B</td>
    <td>0.338</td>
    <td>0.374</td>
    <td>0.501</td>
    <td>38</td>
    <td>0.392</td>
    <td>0.400</td>
    <td>0.429</td>
    <td>1</td>
    <td>0.221</td>
    <td>0.343</td>
    <td>0.435</td>
    <td>18</td>
  </tr>
</table>

Table 13: Performances on the local LLMs. We employed Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct and Qwen2.5-32B-Instruct for Llama 8B, Qwen 7b and Qwen 32B, respectively. P<sub>k</sub>, WD, F1 were calculated only for correctly formatted outputs.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">TIAGE</th>
  </tr>
  <tr>
    <th>P<sub>k</sub>↓</th>
    <th>WD↓</th>
    <th>F1↑</th>
  </tr>
  <tr>
    <td>R1</td>
    <td>0.286</td>
    <td>0.331</td>
    <td>0.644</td>
  </tr>
  <tr>
    <td>V3</td>
    <td>0.259</td>
    <td>0.204</td>
    <td>0.674</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>0.232</td>
    <td>0.256</td>
    <td>0.699</td>
  </tr>
</table>

Table 14: Performance for additional closed-source LLMs.

strategy: by providing explicit topic-shift criteria, illustrative examples, and a clearly defined reasoning path, the task is structured in a way that reduces the need for complex, proactive reasoning.

D Details for Experiment

D.1 Details for Implementation

The model used for our experiments is gpt-4o for closed LLM, Llama-3.1-70B-Instruct and Qwen2.5-72B-Instruct for open-source LLM. At first, we considered two closed models: gpt-4o and Claude-3.5-sonnet. But Claude was excluded due to poor accuracy compared to gpt-4o at a preliminary evaluations and there are no prior studies applied their method to the claude family. For the inference of open-source LLMs, we utilized a computational infrastructure consisting of 4*NVIDIA A100 80GB GPU. We conducted our experiments without employing any model-specific tuning or quantization techniques, thus maintaining the original model architecture and parameters. We kept the hyperparameters initially stated, except for the temperature that we set to 0 for reproducibility of our experiments.

D.2 Details for Reproduce

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">TIAGE</th>
    <th colspan="4">SuperDialseg</th>
    <th colspan="4">Dialseg711</th>
  </tr>
  <tr>
    <th>P<sub>k</sub></th>
    <th>WD</th>
    <th>F1</th>
    <th>Error</th>
    <th>P<sub>k</sub></th>
    <th>WD</th>
    <th>F1</th>
    <th>Error</th>
    <th>P<sub>k</sub></th>
    <th>WD</th>
    <th>F1</th>
    <th>Error</th>
  </tr>
  <tr>
    <td colspan="12"><b>Unsupervised Learning Methods</b></td>
  </tr>
  <tr>
    <td>DialSTART</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>0.179</td>
    <td>0.198</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>SumSeg</td>
    <td>0.438</td>
    <td>0.455</td>
    <td>-</td>
    <td>-</td>
    <td>0.469</td>
    <td>0.480</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td colspan="12"><b>LLM-based Methods</b></td>
  </tr>
  <tr>
    <td>Plain Text (GPT-3.5)</td>
    <td>0.496</td>
    <td>0.560</td>
    <td>0.362</td>
    <td>0.318</td>
    <td>0.347</td>
    <td>0.658</td>
    <td>0.290</td>
    <td>0.355</td>
    <td>0.690</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>S3-DST<sub>turn</sub></td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>0.009</td>
    <td>0.008</td>
    <td>-</td>
    <td>-</td>
  </tr>
</table>

Table 15: Performances reported in their original papers. Denoted as DialSTART indicates main result of Gao et al., 2023, SumSeg indicates main result of Artemiev et al., 2024, Plain Text indicates ChatGPT variant of Jiang et al., 2023’s main result and S3-DST<sub>turn</sub> indicates main result of Das et al., 2024.

For the SuperDialseg paper(Jiang et al., 2023), we obtained experimental results for the Random, TexTiling, TexTiling+Glove, CSM, BERT, RoBERTa, and RetroTS-T5 methods.

We reproduced the experimental result for PlainText(Jiang et al., 2023), S3-DST(Das et al., 2024), DialSTART(Gao et al., 2023), and SumSeg(Artemiev et al., 2024), which either lacked F1 scores or did not perform experiments on certain datasets.

During reproduction, we maintained all settings, including seeds, without any parameter modifications. However, We observed results that differed from the original experiments. Their original experimental results are presented in Table 15.

For the reproduction of LLM-based methods, we made the necessary modifications in the following cases.
Plain Text (Jiang et al., 2023): As Plain Text was the only methodology that disclosed the system prompt, we equitably refrained from using system prompts. We compared results with and without system prompts, finding nearly identical performance aside from parsing inconveniences.

S3-DST (Das et al., 2024): S3-DST constructs prompts on a turn basis, while we performed on an utterance basis. Their approach is not suitable for our approach when dialogue has consecutive utterance or odd numbers of utterances. Therefore, we modified turn-based inference to utterance-based.

All final prompts used are attached to our repository.

Reviewer paper summaries: The paper introduces Def-DTS, a method for Dialogue Topic Segmentation that leverages LLM-based multi-step reasoning to improve performance by using structured prompting for context summarization, intent classification, and topic shift detection.

This paper proposed a structured prompting approach for dialogue topic segmentation (DTS), including three steps: (1) bidirectional context extraction (i.e., summarizing the preceding context and the subsequent context) (2) utterance intent classification (3) final deductive topic shift classification. Empirical studies show that their method achieves the best performance on TIAGE and Dialseg711, as well as a comparative performance on SuperDialseg. Besides, they also employed an ablation study to confirm the necessity of each part of their proposed prompting approach.
Summary of strengths: Def-DTS leverages LLM-based reasoning to significantly improve dialogue topic segmentation, especially in reducing false negatives.

1. The proposed structured prompting approach is helpful for dialogue topic segmentation (DTS), achieving the best performance on TIAGE and Dialseg711. The ablation study also confirms the significance of different components within the design of the prompt.
Summary of weaknesses: - I wonder if there is still any application for the task of dialogue topic segmentation in the era of large models. It seems that this task has become less relevant to the dialogue itself.
- Furthermore, I believe that the failure of the large model baseline is primarily due to the fact that XML is not well-suited for large models. It may be more effective to use descriptions in natural language and JSON-style outputs, which could yield better results [1]. In fact, the ablation study, specifically the "w/o all" experiment, already demonstrates this situation.
- How does this work compare to R1? If a direct natural language prompt can achieve excellent performance, it seems that such complex prompting may not be necessary.
- Moreover, I seem to have missed the Deductive Reasoning section entirely. There appears to be no reasoning process presented?

[1] A preliminary evaluation of chatgpt for zero-shot dialogue understanding.

1. Prompt template is sensitive to different data and different large language models. In the current experiment, the authors only do experiments with GPT-4o, which may not be sufficiently reliable. 

2. The second contribution doesn't stand, since reformulation of DTS as an utterance-level intent classification is well-studied.

3. How did they decide the pre-defined utterance intent list wasn't explained. Is it the complete set of utterance intents? If not, I think this part should be explored further to make this work solid enough for publication.
Comments, suggestions and typos: Missing Reference:

- A preliminary evaluation of chatgpt for zero-shot dialogue understanding.
- End-to-end Task-oriented Dialogue: A Survey of Tasks, Methods, and Future Directions.
- Synergizing large language models and pre-trained smaller models for conversational intent discovery
- Evaluating ChatGPT for Joint Intent Detection and Slot Filling: Zero-Shot vs. Few-Shot Prompting
- Zero-shot spoken language understanding via large language models: A preliminary study

## Questions
- In Line 480, why did you sample only 100 examples for evaluation there?

## Grammar Issues and Typos
- In Line 186, 'extracts' -> 'extract'.

## Presentation Improvement
- in Lines 025-026, what is 'type 2 error'? You should explain it before you use this term.
- In Algorithm 1, you don't need to define the functions CLASSIFYINTENT and SUMMARIZE like this, without any procedure inside the definition. You can just claim that at the beginning of the algorithm. For example, you can state 'Require: SUMMARIZE, to summarize a given context'.
Overall assessment scores: 3

2.5
Metareview score: 3
Metareview: The paper introduces a new method for Dialogue Topic Segmentation Def-DTS, which performs multi-step reasoning with LLMs for Open-domain Dialogue Topic Segmentation. It employs structured prompting for bidirectional context extraction, utterance intent classification and deductive topic shift classification. Results on three benchmarks show that the proposed method can achieve competing or superior performance.
Summary of reasons to publish: This paper proposed a deductive reasoning framework for DTS based on  structured prompting with LLMs. It achieves good results on several widely-used benchmarks for open-domain dialogues. The authors provide additional experiments and analysis to address the concerns from the reviewers, including comparing to more baseline etc.