Improving Long Document Topic Segmentation Models With Enhanced Coherence Modeling
Authors: Hai Yu
Chong Deng
Qinglin Zhang
Jiaqing Liu
Qian Chen
Wen Wang
Venue: EMNLP 2023 Main (2023)
Keywords: topic segmentation
text coherence
semantic similarity
sentence structure
contrastive learning

Abstract: Topic segmentation is critical for obtaining structured documents and improving down- stream tasks such as information retrieval. Due to its ability of automatically exploring clues of topic shift from abundant labeled data, recent supervised neural models have greatly promoted the development of long document topic segmentation, but leaving the deeper relationship between coherence and topic segmentation underexplored. Therefore, this paper enhances the ability of supervised models to capture coherence from both logical structure and semantic similarity perspectives to further improve the topic segmentation performance, proposing Topic-aware Sentence Structure Prediction (TSSP) and Contrastive Semantic Similarity Learning (CSSL). Specifically, the TSSP task is proposed to force the model to comprehend structural information by learning the original relations between adjacent sentences in a disarrayed document, which is constructed by jointly disrupting the original document at topic and sentence levels. Moreover, we utilize inter- and intra-topic information to construct contrastive samples and design the CSSL objective to ensure that the sentences representations in the same topic have higher similarity, while those in different topics are less similar. Extensive experiments show that the Longformer with our approach significantly outperforms old state-of-the-art (SOTA) methods. Our approach improve $F_{1}$ of old SOTA by 3.42 (73.74 → 77.16) and reduces $P_{k}$ by 1.11 points (15.0 → 13.89) on WIKI-727K and achieves an average relative reduction of 4.3\% on $P_{k}$  on WikiSection. The average relative $P_{k}$ drop of 8.38\% on two out-of-domain datasets also demonstrates the robustness of our approach.

Improving Long Document Topic Segmentation Models With Enhanced Coherence Modeling
Hai Yu, Chong Deng, Qinglin Zhang, Jiaqing Liu, Qian Chen, Wen Wang
Speech Lab, Alibaba Group
{yuhai.yu, w. w.wang)@aliib wwwpaallaaaainnnn... com
Abstract
parts (Hearst, 1994). Topic segmentation makes documents easier read and understand, and also plays a key role in many downstream tasks such as information extraction (Prince and Labadié, 2007;
Topic segmentation is critical for obtaining structured documents and improving down- stream tasks such as information retrieval. Due its ability automatically exploring clues of topic shift from abundant labeled data, re- cent supervised neural models have greatly promoted the development of long document topic segmentation, but leaving the deeper re- lationship between coherence and topic seg- mentation underexplored. Therefore, this paper enhances the ability of supervised models to capture coherence from both logical structure and semantic similarity perspectives further improve the topic segmentation performance, proposing Topic-aware Sentence Structure Prediction (TSSP) and Contrastive Seman- tic Similarity Learning (CSSL). Specifically, the TSSP task proposed force the model to comprehend structural information by learn- ing the original relations between adjacent sen- tences in disarrayed document, which is con- structed by jointly disrupting the original doc- ument at topic and sentence levels. Moreover, we utilize inter- and intra-topic information to construct contrastive samples and design the CSSL objective ensure that the sentences rep- resentations in the same topie have higher sim- ilarity, while those in different topics are less similar. Extensive experiments show that Long- former with our approach significantly outper- forms state-of-the-art (SOTA) methods. Our ap- proach improves Fj of SOTA by 3.42 (73.74 77.16) and improves Ps by 1.1 points (15.0 13.89) on WIKI-727K and achieves an average relative reduction of 4.3% on Pa WikiSec- tion. The average relative Ps drop of 8.38% on out-of-domain datasets also demonstrates the robustness of our approach'.
Shtekh et al., 2018) and document summariza- tion (Xiao and Carenini, 2019; Liu et al., 2022). Topic segmentation methods can be categorized into linear segmentation (Hearst, 1997), which yields linear sequence of topic segments, and hier- archical segmentation (Bayomi and Lawless, 2018; Hazem et al., 2020), which produces a hierarchical structure with top-level segments divided into sub- segments. We focus on linear topic segmentation in this work, especially for long documents.
Based on the definition of topics, each sentence in a topic relates to the central idea of the topic, and topics should be discriminative. Hence, two adjacent sentences from the same topic are more similar than those from different topics. Exploring this idea, prior unsupervised models mainly infer topic boundaries through computing text similar- ity (Riedl and Biemann, 2012b; Glavaš et al., 2016) or exploring topic representation of text (Misra et al., 2009; Du et al., 2013). Different from the shallow features carefully designed and used by un- supervised methods, supervised neural models can model deeper semantic information and explore clues of topic shift from labeled data (Badjatiya et al., 2018: Koshorek al., 2018). Supervised models have achieved large gains on topic seg- mentation through pre-training language models (PLMs) (e.g., BERT) and fine-tuning on large-scale supervised datasets (Kenton and Toutanova, 2019; Lukasik et al., 2020; Zhang et al., 2021; Inan et al., 2022). Recently, (Arnold et al., 2019; Xing et al., 2020: Somasundaran et al., 2020; Lo et al., 2021) improve topic segmentation performance by ex- plicitly modeling text coherence. However, these approaches either neglect context modeling beyond adjacent sentences (Wang et al., 2017), or require additional label information (Arnold et al., 2019;
1 Introduction
Topic segmentation aims to automatically segment the text into non-overlapping topically coherent
OOur code is publicly available at https: http:://gttttuuueee com/ alibaba- damo- academy/ /SpeeemLSSSo

---

Barrow al., 2020; Lo al., 2021; Inan al., 2022), or impede learning sentence-pair coherence without considering both coherent and incoherent pairs (Xing et al., 2020). Moreover, compared to short documents, topic segmentation becomes more critical for understanding long documents, and coherence modeling for long document topic segmentation is more crucial.
improve topic segmentation performance and they also improve generalizability of the model.
2 Related Work
2.1 Topic Segmentation Models
Both unsupervised and supervised approaches have been proposed before to solve topic segmenta- tion. Unsupervised methods typically design fea- tures based on the assumption that segments the same topic are more coherent than those that belong to different topics, such as lexical cohe- sion (Hearst, 1997; Choi, 2000; Riedl and Bie- mann, 2012b), topic models (Misra et al., 2009; Riedl and Biemann, 2012a; Jameel and Lam, 2013; Du et al., 2013) and semantic embedding (Glavaš et al., 2016 Solbiati al., 2021: Xing and Carenini. 2021). contrast, supervised models can achieve more precise predictions by automatically min- ing clues of topic shift from large amounts la- beled data, either by classification on the pairs of sentences or chunks (Wang et al., 2017: Lukasik et al., 2020) or sequence labeling on the whole input sequence (Koshorek et al., 2018; Badjatiya et al., 2018; Xing et al., 2020; Zhang et al., 2021). However, the memory consumption and efficiency of neural models such as BERT (Kenton and Toutanova, 2019) can be limiting factors for model- ing long documents as their length increases. Some approaches (Amoll et al., 2019; Lukasik et al., 2020; Lo et al., 2021: Somasundaran et al., 2020) use hicrarchical modeling from tokens to sentences, while others (Somasundaran et al., 2020; Zhang al., 2021) use sliding windows reduce resource consumption. However, both directions of methods may not be adequate for capturing the full context of long documents, which is critical for accurate topic segmentation.
Coherence plays key role understanding both logical structures and text semantics. Con- sequently, to enhance coherence modeling in super- vised topic segmentation methods, we propose two auxiliary coherence-related tasks, namely, Topic- aware Sentence Structure Prediction (TSSP) and Contrastive Semantic Similarity Learning (CSSL). We create disordered incoherent docu- ments, then the TSSP task utilizes these documents and enhances learning sentence-pair structure infor- mation. The CSSL task regulates sentence repre- sentations and ensures sentences in the same topic have higher semantic similarity while sentences in different topics are less similar. Experimental results demonstrate that both TSSP and CSSL im- prove topic segmentation performance and their combination achieves further gains. Moreover, per- formance gains on out-of-domain data from the proposed approaches demonstrate that they also significantly improve generalizability of the model.
Large Language Models such ChatGPT have achieved impressive performance on a wide variety of NLP tasks. We adopt the prompts proposed by Fan and Jiang (2023) and evaluate ChatGPT on the WIKI-50 dataset (Koshorek et al., 2018). We find ChatGPT performs considerably worse than fine-tuning BERT-sized PLMs on long document topic segmentation (as shown in Appendix A). Our contributions can be summarized as follows.
We investigate supervised topic segmentation on long documents and confirm the necessity of ex- ploiting longer context information.
2.2 Coherence Modeling
The NLP community has developed models for comprehending text coherence and tasks to mea- sure their effectiveness, such as predicting the co- herence score of documents (Barzilay and Lap- ata, 2008), predicting the position where the re- moved sentence was originally located (Elsner and Charniak, 2011) and restoring out-of-order tences (Logeswaran et al., 201 8; Chowdhury al., 2021). Some researchers have aimed improve topic segmentation models by explicitly modeling text coherence. However, all of prior works con- sider coherence modeling for topic segmenta-
We propose two novel auxiliary tasks TSSP and CSSL for coherence modeling from the perspec- tives of both logical structure and semantic sim- ilarity, thereby improving the performance of topic segmentation.
Our proposed approaches set new state-of-the- (SOTA) performance on topic segmentation benchmarks, including long documents. Abla- tion study shows that both new tasks effectively

---

tion only from a single perspective. For exam- plc, Wang et al. (2017) ranked sentence pairs based on their semantic coherence to segment documents within the Learning-to- to-Rank framework, but they did not consider contextual information beyond two sentences. CATS (Somasundaran et al., 2020) cre- ated corrupted text by randomly shuffling or replac- ing sentences to force the model to produce higher coherence score for the correct document than for its corrupt counterpart. However the fluency of the constructed document is too low so that the seman- tic information is basically lost. Xing et al. (2020) proposed to add the Consecutive Sentence-purt Co- herence (CSC) task by computing the cosine simi- larity as coherence score. But no more incoherent sentence pairs are considered in CSC, except for those located at segment boundaries. Other meth- ods (Amold al., 2019; Barrow et al., 2020; Lo al., 2021; Inan al., 2022) have used topic la- bels to constrain sentence representations within the same topic, but they require additional topic label information. In contrast to these works, our work is the first to consider topical coherence as both text semantic similarity and logical struc- ture (flow) of sentences.
Lox
positive
aegative
Ts Predictor
TSSP Predictor
Shared Encoder
Shared Encoder
Shared Embeddingg Layer
Shared Emhedding Layer
The overall architecture of our model. 8; is i-th sentence in document d. of is the augmented data we construct corresponding to document dl (Section 3.2). TS denotes Topic Segmentation, TSSP denotes Topic- aware Sentence Structure Prediction (Section 3.2) and CSSL denotes Contrastive Semantic Similarity Learning (Section 3.3). Lts. Lasp and Lessl denote the losses we describe in Section 3.
Figure 1:
The token sequence for the document is embedded through the embedding layer and then fed into the encoder to obtain its contextual representations. We take the representation of each BOS h4 the sentence representation, shown in Eq. Then we apply softmax binary classifier g as Eq. 3 on top of h, to compute the topic segmentation probability of each sentence. We use the standard binary cross-entropy loss function as Eq. to train the model.
3 Methodology
this section, we first describe our baseline model for topic segmentation (Section 3.1), then intro- duce our proposed Topic-aware Sentence Structure Prediction (TSSP) module (Section 3.2) and Con- trastive Semantic Similarity Learning (CSSL) mod- ule (Section 3.3). Figure 1 illustrates the overall architecture of our topic segmentation model.
n-1
Lis =
[ys Iaps + (1 221) In(1 (a[1-p:)] (2)
P ghis)
(3)
h Encoder(tr.))
(4)
3.1 Baseline Model for Topic Segmentation
Our supervised baseline model formulates topic segmentation sentence-level sequence labeling task (Zhang al., 2021)). Given a document repre- sented as a sequence sentences [s1, 82, 83, .... 8n] (where n is the number sentences), the model predicts binary labels [y/ V2...., .... Vn-1] correspond- ing to each sentence except for the last sentence, where W = 1, E {1,-... last sentence of topic and means not. Following prior works (Somasundaran et al., 2020: Zhang et al., 2021), we prepend a special token BOS before each sentence and the updated sentence shown in Eq. 1, where 1ú is BOS and Bsl is the number of tokens in 8i.
3.2 Topic-aware Sentence Structure Prediction
Learning sentence representations that reflect inter- sentence coherence (inter-sentence relations) is crit- ical for topic segmentation. Several tasks have been proposed in prior works for modeling sentence-pai relations. The Next Sentence Prediction (NSP) task in BERT (Kenton and Toutanova, 2019) predicts whether two segments appear consecutively in the same document come from different documents, hence it fuses topic prediction and coherence pre- diction in one task. In order to better model inter- sentence coherence, the Binary Sentence Ordering (BSO) task in ALBERT (Lan et al., 2019) con- structs input as two consecutive segments from the
}} means s; is the
of = [t&1 12,, [1.1
(1)

---

ument, then randomly replace some topics with topics from other documents increase diversity. Specifically, for a randomly selected subset of pl percent of the documents, we replace each topic in them with a topic snippet from other documents with probability of P2 and keep the same with the probability of 1 p2- The default values of and P2 both 0.5. Finally, we shuffle the sentences in each topic further increase the difficulty of the TSSP task.
shurffle topics
F
Pg
replace topics randomly
shufffle sentences within each topic
Sentence-pair Relations After constructing the augmented document d', we define the TSSP task auxiliary objective assist the model to cap- ture inter-sentence coherence, by learning the orig- inal structural relations between adjacent sentence pair a and b in the augmented incoherent document d', We define three types of sentence-pai cantence--pir relations. The first type (label 0) is when a and belong to different topics, indicating a topic shift. The sec- ond type (label 1) is when a and b are in the same topic and b is the next sentence of a. The third type (label 2) is when and b belong the same topic but & is not the next sentence of a. For example, the sequence of sentences Figure 2 will be assigned with the TSSP labels as [1, O, 2, 2, O, 2, 2]. We use ÿ - [vo. V1, 72] to represent the one-hot encod- ing of the TSSP labels, where ūoj = 1,j E {0, 1, if the sentence pair belongs to the j-th category; otherwise, V/ O. For the TSSP task, we use the cross entropy loss function defined in Eq. 5, where Vig denotes the label of the i-th sentence pair, and pīj denotes the probability of the i-th sentence pair belonging to the j-th category.
Sa
The process of constructing the augmented document (the bottom line) from the original document (the top line), denotes i-th sentence of the document. The sentences with the same colors are in the same topic. Sentences light purple are topics from another document.
Figure 2:
same document but 50% the time their order re- versed. The ternary Sentence Structural Objective (SSO) in StructBERT (Wang et al., 2019) further increases the task difficulty of BSO by adding class of sentence pairs from different documents.
All these tasks for learning inter-sentence co- herence have not explored topic structures. Dif- ferent from them, we propose a Topic-aware Sen- tence Structure Prediction (TSSP) task to help the model learn sentence representations with struc- tural information that explore topie structures and hence are more suitable for topic segmentation.
Data Augmentation We tailor data augmenta- tion techniques for topic segmentation. As depicted in the right half of Figure 1, we create an aug- mented document d' from the original document d and feed d' into the shared encoder after the em- bedding layer to enhance inter-sentence coherence modeling. Different from the auxiliary coherence modeling approach proposed by Somasundaran et al. (2020) which merely forces the model to predict a lower coherence for the corrupted docu- ment than for the original document, we simultane- ously perturb d at both topic and sentence levels, constructing the augmented document to force the model to learn topic-aware inter-sentence structure information. Hence, task is more challeng- ing and the learned sentence representations more suitable for topic segmentation. Figure il- lustrates the process of constructing an augmented document. We first shuffle topics within the doc-
Lreep ΣΣ yhs In(pis)
(5)
i=1 1=0
3.3 Contrastive Semantic Similarity Learning
We assume that sentences segments from the same topic are inherently more coherent than those from different topics. Therefore, the motivation of our Contrastive Semantic Similarity Learning (CSSL) module is to adjust the sentence represen- tation learning to grasp the relative high and low coherence relationship. Different from (Xing et al., 2020) which only calculates the cosine similarity of two adjacent sentences, CSSL takes into count both similar and dissimilar sentence pairs improve sentence representation learning. Construct Positive and Negative Samples The upper left part of Figure 1 illustrates how CSSL

---

leverages contrastive learning regulate sentence representations, which in turn influences the pre- dictions of topic segmentation. To construct posi- tive and negative sample pairs in the contrastive learning framework, prior works in NLP pro- pose several data augmentation techniques such word deletion and substitution (Wu et al., 2020), dropout (Gao et al.. 2021), and adversarial at- tack (Yan al., 2021). However, different from prior works that synthesize positive and negative pairs, we explore natural positive and negative pairs for topic segmentation, as two sentences or segments from the same topic are inherently more coherent than those from different topics. Accord- ingly, regarding each sentence in the document as the anchor sentence, we choose k sentences in the same topic to constitute positive pairs and l2 sen- tences from different topics as negative pairs based on the ordering of the distance of a sentence from the anchor sentence, starting from the nearest to farthest. Recently, Gao et al. (2023) proposes a con- trastive learning method for unsupervised topic seg- mentation. However, in their unsupervised method, similar and dissimilar sample pairs could be noisy due to lack of ground truth topic labels, which would not occur in our supervised settings.
c )
T cssl
(7)
ki
sim(heik
x 22
sim(21,22)
(8)
| |
Combining Eq. 2, 5 and 6, we form the final loss function of our topic segmentation model as Eq. 9, where α1 and 02 are hyper-parameters used to ad- just the loss weights.
Liotal = Lis + an Lesup + Oz Lesel
(9)
Dataset
Docs
#Topies
#Sentences
#Tokens
WTKI-727K
727,746
6.18
52.65
1356
WikiSection
23,129
6.98
57.69
1321
WIKI-50
50
7.68
61.40
1544
Elements
118
7.71
23.81
1654
Table I: Statistics of Intra-domain Out-of- domain datasets. #X denotes the average number of per document.
Experiments
4.1 Experimental Setup
Loss Function As illustrated in Figure 1, uti- lize the following loss function train our model and learn contrastive semantic representations of inter-topic and intra-topic sentences. b and k2 are hyperparameters that determine the number of sentences used to form positive and negative pairs, respectively. For each sentence representation hi bJ denotes the j-th similar sentence in the same topic as sentence i, while hi denotes the j-th dis- similar sentence different topic from sentence i. We select sentences to form sentence pairs based on their distances to the anchor sentence, from clos- est to farthest. The objective of our loss function is to bring semantically similar neighbors closer and push away negative sentence pairs, in Eq. 6. in Eq. a temperature super-parameter the cosine similarity of two vectors, with a default value 0.1. In future work, in order to avoid pushing away sentence pairs in different topics but covering similar topical semantics, we plan to consider refin- ing the loss, such assigning loss weights based on their semantic similarity.
Datasets We conduct
of experiments evaluate the effectiveness of our method, includ- ing intra-domain and out-of-domain settings. The details the datasets are summarized in Table Intra-domain Datasets We use WIKI- 727K (Koshorek et al., 2018) and English WikiSection (Amoold et al., 2019), which are widely used as benchmarks to evaluate the text segmentation performance of models. WIKI-727K is large corpus with segmentation annotations, created by leveraging the manual structures of about 727K Wikipedia pages and automatically organizing them into sections. WikiSection consists of 38K English and German Wikipedia articles from the domains of disease and city, with the topic labeled for each section of the text. We use en_city and disease throughout the paper to denote the English subsets of disease and city domains, and use WikiSection to represent the collection of these two subsets. Each section is divided into sentences using the PUNKT tokenizer of the NLTK library Additionally, we utilize the newline information in WikiSection to only predict whether sentences with line breaks are
scale
12
Lessl
log(Lesd)
(6)
Phttps://www http:///www.ltt.orr.....

---

Model
city
disease
Fi f
Pk 4
Fi f
Ps 1
WD +
SEC*T+bloom (Aroold et al., 2019)
71.6
14.4
56.6
26.8
S-LSTM (Barrow et al., 2020)
76.1
9.1
59.3
20.0
BiLSTM-BERT (Xing ul., 2020)
9.3
21.1
Transformer 2gger (l.o al., 2021)
9.1
18.8
Tipster (Gong 2022)
79.8
8.3
62.2
14.2
PEN-NS (Xia al., 2022)
80.0
Naive LongTTS-Basee
(Inan al., 2022)
33.5
Naive LongT5-Base- (Inan et al., 2022)
73.1
9.2
38.8
24.8
BERT-Base
78.99
8.94
11.34
67.34
19.69
24.83
+TSSP+CSSL (ours)
80.16
8.22
10,19
68.26
18.29
22.06
BigBird Base
80.49
8.21
10.24
70.61
16.73
20.44
+TSSP+CSSL (ours)
81.89
7.71
9.70
72.14
16.62
20.55
LongformerSim
79.75
9.85
11.75
66.66
19.60
22.74
Longformer- Base
82, 22.19
7.76
9.74g.us
72. .2.29
16.66u.44
20.624.79
+CATS (Somasundaran et al., 2020)!
82,30
7.84
9.80
72.00
16.21
19.95
+CSC (Xing et al., 2020y*
82.40
7.74
9.69
72.84
16.18
19.74
+TSSP (ours)
83,12
7.39
9.31
73.74
16.03
19.71
+CSSI (ours)
82.67
7.57
9.54
73.07
15.88
19.25
+TSSP&CSSL (ours)
83.19
7.38
9.29
74.17%
15.44
19.05
Pre-traines LongTT5-Bars T5-BBse---ee S--Bsc--DS DS (Inan et al., 2022)
6.8
15.3
Pre-trained Long T5-Base-SS Basc-SS (Inan al., 2022)
82.3
7.1
68.3
15.0
Pre-trained Longformer-Bus
84.70
6.83
8.52
76.02
14.15
17.28
+TSSP+CSSL, (ours)
85.14
8.26
77.33
13.66
16.70
Table 2: Performance of baselines and w/ our methods on en_city and en mn_disease test sets of WikiSection. LongformerSin denotes Longformer-Di Base uses cosine similarity of neighbor sentences as the predictor. Pre- trained Longformer-Be Base denotes further pre-training Longformer-Bac with WIKI-727K training set and then fine-tuning with WikiSection training set. i denotes training Longformer-Ban with the corresponding auxiliary task described Section 2.2. Max sequence length for BigBird-Baa Base and Longformer- Base 2048. x and y Ty denote mean and standard deviation from three runs with different random seeds. indicates the gains from +TSSP+CSSL over Longformer Base are statistically significant with < 0.05.
topic boundaries, which is beneficial to alleviate the class imbalance problem.
tive F1 and WindowDiff. F1 is calculated based on precision and recall of correctly predicted topic seg- mentation boundaries. The Pş metric is introduced to address some limitations of positive F1, such the inherent trade-off between precision and recall as well as its insensitivity to near-misses. WD is proposed by Pevzner and Hearst (2002) as supple- ment to Ps to avoid being sensitive to variations in segment size distribution and over-penalizing near- misses. By default, the window size for both Pa and WD is equal to half the average length of tual segments. Lower Ps and WD scores indicate better algorithm performance.
Out-of-domain Datasets Following prior work (Xing et al., 2020), evaluate the domain our model, we fine-tune
transfer capability the model on the training set of WiKiSection dataset (union of en_city and en_ (r_disease) due to its distinct domain characteristics. Then we evaluate its performance on two other datasets, including WIKI-50 (Koshorek al., 2018) and Elements (Chen al., 2009), which have different domain distributions from WikiScction. Specifi- cally, WIKI-50 consists of 50 samples randomly selected from Wikipedia, while Elements consists 118 samples which are also extracted from Wikipedia but focuses on chemical elements.
Baseline Models Although Transformer (Vaswani et al., 2017) has become the SOTA architecture for sequence modeling on a wide variety of NLP tasks and transformer-basee PLMs such BERT (De- vlin et al., 2019) become dominant in NLP, the self-attention mechanism has quadratic time and memory complexity the input sequence length (Vaswani al., 2017), limiting the max quence length during pre-training (e.g., 512 for BERT) for balance between performance and memory usage. As shown in Table 1, the avg. num-
Evaluation Metries Following prior works, we three standard evaluation metrics for topic seg- mentation, namely, positive F1, Pk (Beeferman al., 1999), and WindowDiff (WD) (Pevzner and Hearst, 2002) 4, To simplify notations, we use F, and WD throughout the paper to denote posi-
Wo use https: bttps://seege httt::///eeeeaeeeeeeeeee
htwpeeewwll.eeaeeedeeoeeoo readthedocs. io/ to com-
pute Pt and WD

---

ber of tokens per document of each dataset exceeds 512 and hence these datasets contain long docu- ments. We tailor the backbone model selection for long documents. Prior models using BERT- like PLMs for topic segmentation either truncate long documents into the max sequence length or use sliding window. These approaches may de- grade performance due to losing contextual infor- first evaluate BERT- Base (Devlin al., 2019) and several competitive efficient transformers on the WikiSection dataset, including BigBird-Ban Base (Zaheer et al., 2020) and Longformer-Bei (Beltagy et al., 2020). As shown in Table 2, Longformer- Base achieves 82.19 and 72.29 Fi, greatly outperforming BERT- (78.99 and 67.34 F1) by (+3.2, +4.95) F1 and BigBird- Base (80.49 and 70.61 Fj). Hence we select Longformer-Bass as the encoder for the main ex- periments. To compare with our coherence-related auxiliary tasks, we evaluate Longformer-Base on WikiSection with the prior auxiliary CATS or CSC task in Section 2.2. In addition, following Inan et al. (2022), we evaluate the pre-trained settings where we first pre-train Longformer on WIKI-727K and then fine-tune on WikiSection. Under the domain transfer setting, we cite the results in (Xing al., 2020). Note that all the baselines we include for comparisons are well-established and exhibit top performances on these benchmarks.
Model
WIKI-727K
1
PL
WD 1
BHLSTM (Koshorck el., 2018)
22.13
segment BERT (Lukasik et al., 2020)
66.0
Hier. BERT (1 uksskk ct al. 3020)
66.5
CATS (Somsonddaaa et sl., 2020)
15.95
Be--BERT-Bas (Zhang
2021>1
70.39
17:35
18.50
Seq-ELE Seq-ELECTRA- Base (22ang
2021)
15,83
17.01
Naive Long't U.S. Base DS (Iean et al. 2022)
15.4
Naive LongTS SongT-Ruun Rase SS (Iran ol., 2022)
15.0
Longfomner-B Rase
76.27
14.40uus
15 15,50uss
-TSSP (omrs)
76.57
14.13
15.20
CCSLL (ours)
76.30
14,28
15,40
mation. Consequently,
TSSPPCSSLL (ours)
77.36
13,8%
14.9% ne
Table 3: Performance of baselines and w/ our methods on the WIKI-727K test set. represents our reproduced results. x and in 2g denote mean and standard devia- tion from three runs with different random seeds. in- dicates the gains from +TSSP+CSSL over Longformer- Base are statistically sigmificant with < 0.05.
Base already outperforms all baselines the first group of Table 2 and Table 3 and BERT-Bas Base and BigBird-Base. Training BERT-Base, BigBird-Base and Longformer-Bas Base with our TSSP or CSSL task achieves further gains. Table 2 shows that our TSSP and CSSL both outperform CATS CSC auxiliary tasks. More importantly, combin- ing TSSP and CSSL exhibits complementary ef- fects, as it achieves further gains and sets the new SOTA, confirming the necessity of modeling both sentence structure and text semantic similarity for modeling text coherence and in turn for topic seg- mentation. On WikiSection, +TSSP+CSSL im- proves BERT-Base by (+1.17, +0.92) F1, Big Base by (+1.4, +1.53) F1, and Longformer- by (+1.0, +1.88) Fj. On WIKI-727K, +RTIK+TTSSST CSSS improves Longformer -asereer--BBaeeeeee by +0.89 F,. In addi- tion, utilizing pre-training data also improves the performance on WikiSection, by (+1.95, +3.16) F,. Finally, our new SOTA reduces P& of old SOTA by 1.11 points on WIKI-727K (15.0~> 13.89) and achieves an average relative reduction of 4.3% on P& WikiSection. It is also important note that our proposed TSSP and CSSL are agnostic to docu- ment lengths and are also applicable to models and datasets for short documents, which is verified by their gains on both short and long document subsets of WIKI-727K test set (as shown in Appendix C).
Implementation Details To investigate the efficacy of exploring longer context for topic segmentation, we conducted additional evaluations on WikiSec- tion using maximum sequence lengths of 512, 1024, and 4096, alongside the default 2048. For docu- ments longer than the max sequence length, we use a sliding window to take the last sentence of the prior sample as the start sentence of the next sam- ple. We the baseline Longformer-Ban Base and w/ our model (i.e., three times with different random seeds and report means and standard deviations of the metrics. De- tails of hyperparameters are in Appendix B.
4.2 Main Results
Intra-domain Performance Table 2 and Table 3 show the performance of baselines and w/ our approaches on WikiSection and WIKI-727K test sets, respectively. The results of Longformer-Bass and LongformerSim in Table show that using cosine similarity alone is insufficient to predict the topic segmentation boundary. Longformer-
Domain Transfer Performance Table 4 shows the performance of the basclines and w/ our method on the out-of-domain WIKI-50 and Elements test sets. Longformer-Bar already achieves 5.51 point reduction on P& on Elements over the prior best performance from supervised models, and our ap- proach further improves Ps. by 2.83 points. While

---

Model
WIKI-50
Elements
F1 T
Pk 4
WD 4
F
Pk +
WD 4
BayesSeg (Eisenstein and Burzilay, 2008)
49.2
35.6
GraphSeg (Glavaš 2016)
63.6
49.1
Sector (Amold et al., 2019)
28.6
42.8
CATS "Somasundaran et al., 2020)
29.3
45.2
BiLSTM+BERT (Xing ul., 2020)
26.8
39,4
Longlormer--uur
46,45
28. 26..2
31.166
15.132.16
3
44.99
+CATS (Somasundaran et al., 2020)
43.96
28.11
29,82
57.48
31.45
41.75
+CSC (Xing et al., 2020)*
48.18
28.88
31.20
55.18
33.26
42,51
+TSSP=CSST. (ours)
52.29
25.70
27.69
58.542.01
31.063.05
43.91
Table 4: Performance baselines and w/ our methods under domain transfer setting. BERT and Longformer are Base size. The training set of WikiSection is used for fine-tuning. denotes fine-tuning Longformer with the corresponding auxiliary task described in Section 2.2. r and in By denote mean and standard deviation from three runs with different random seeds. indicates the gains from +TSSP+CSSL over Longformer are statistically significant with < 0.05.
Longformer-Band Base does not perform best on WIKI- 50, incorporating our method achieves +5.84 Fj gain and 3.06 point gain on Px. setting new SOTA on WIKI-50 and Elements for both unsupervised and supervised methods. Overall, the results demonstrate that our proposed method not only greatly improves the performance of model un- der the intra-domain setting, but also remarkably improves the generalizability of the model.
82.5-
KLay
$1.42.
K2.18
XILO-
2Ki88
77.5-
75.0
74.08
22.
72.5
23--5
21.66.......
Inference Speed Our proposed TSSP and CSSL do not bring any additional computational cost to in- ference and do not change the inference speed. We randomly sample KK documents from WIKI-727K test set and measure the inference speed on single Tesla V100 GPU with batch_size 1. On aver- age, BERT-Base with max sequence length 512 pro- cesses 19.5K tokens/sec while Longformer-Bas with max sequence length 2048 processes 15.9K tokens/sec. This observation consistent with the findings in (Tay et al., 2020) that Longformer does not show a speed advantage over BERT until the input length exceeds 3K tokens.
70.0-
Our Longformer
Baseline Longformer on city
Longformer on, discase
Riseline Longformer discusa
context length
Figure 3: Fl of baseline Longformer and our Long- former (i.e., Longtorner++SSSPCSSSL on Wiki- Section en ot_city and en_disease test sets the context length increases.
+4.41 respectively. Considering that the average document length of WikiSection is 1321, we infer that capturing more context information is more beneficial to topic segmentation on long documents. We also observe that compared to Longformer- Base, Longformer:+-SSPPCSSS yields consistent improvements across different input lengths on both en_city and en en_ disease test sets. These results suggest that our methods are effective at enhancing topic segmentation across various context sizes and can be applied to a wide range of data sets. Ab- lation Study of TSSP and CSSL Figure 4 shows ablation study of TSSP and CSSL on the WikiSec- tion dev set, respectively. Figure 4(a) demonstrates effectiveness of the three classification tasks in the TSSP task (Section 2.1). Compared SSO and CATS, TSSP helps the model learn better inter- sentence relations and both intra- and inter-topic
5 Analysis
Effect of Context Size To study the effect of the context size, we evaluate Longformer with max sequence length of 512, 1024, 2048 and 4096 on WikiSection. We evaluate the effectiveness of our proposed methods using the corresponding max sequence length to investigate whether they re- main effective with different context sizes. As be seen from Figure 3, the topic segmenta- tion F1 gradually improves the context length increases. Among them, the effect of increasing the input length from 512 to 1024 is the largest with Fi on en city and en disease improved by +2.54 and

---

K0
312
1146
Baseline
CSSL
$1.21
$12
$3.85
80.71
80.70
302
50.4
80.00
60.35
204
102
8043
300
CATS
S8O
TSSP intr--upiic
The number of uogaativee pairs
(a) TSSP
(b) CSSL
Figure 4: Ablation study of Longformer with our method on the WikiSection dev set with (a) TSSP and (b) CSSL separately. CATS and SSO in Figure (a) are previously proposed auxiliary tasks where CATS denotes Coherence- Aware Text Segmentation (Section 2.2) and SSO denotes Sentence Structural Objective (Section 3.2). In Figure (a). TSSP w/o inter-topic means without category and w/o intra-topic means category and in Section 3.2. In Figure (b), 0 negative pairs represents fine-tuning with just the CSC task (Section 2.2).
structure labels are needed to improve performance. Figure 4(b) illustrates the impact of varying num- bers of negative sample pairs for CSSL on Long- former. We find that adding similarity-related aux- iliary tasks improves the performance. Compared with CSC, CSSL focuses sentences of the same and different topics when learning sentence rep- resentations. As the number of negative samples increases, the model performance improves and optimizes at =2=3. Gain from TSSP is slightly larger than that from CSSL, indicating that compre- hending structural information contributes more to coherence modeling. We speculate that encoding the entire topic segment into semantic space learn contrastive representation may help detect- ing topic boundaries, which we plan explore in future work.
Similarity of Sentence-paii Representations To investigate impact of coherence-related auxiliary tasks on sentence representation learning, we cal- culate cosine similarity of adjacent sentence rep- resentations for predicting topic boundaries. We compute F1 of baselines and Longformer (i.e., Bongformer-BBseetSSSP----
on en en_city and en en_ rr_disease dev sets. As shown in Figure 5, com- pared to Longformer-Busee our model achieves higher F,, indicating that sentence representations learned with our methods are more relevant to topic segmentation and are better at distinguishing sen- tences from different topics. We also explore com- bining probability and similarity predict topic boundaries in Appendix D but find further gain. The results suggest that the model trained with TSSP+CSSL covers more features than similarity.
Conclusion
Comprehending text coherence is erucial for topic segmentation, especially on long documents. We propose the Topic-aware Sentence Structure Predic- tion and Contrastive Semantic Similarity Learning auxiliary tasks to enhance coherence modeling. Ex- perimental results show that Longformer trained with our methods significantly outperforms SOTAs two English long document benchmarks. Our methods also significantly improve generalizability of the model. Future work includes extending our approach to spoken document topie segmentation and other segmentation tasks at various levels granularity and with modalities beyond text.
22
Longfonns Oue Loneleens sio Baudine Langliema discae
engfomeer
81
02
63
6.6
05 ds Irrateld
33 0.
Figure F1 from using cosine similarity of two ad- jacent sentence representations to predict topic bound- aries, with different thresholds on WikiSection dev set.

---

Limitations
Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre- pre-training of deep bidirectional transformers for language under- standing. In NAACL-HLT, pages 4171-4186. Associ- ation for Computational Linguisties.
Although our approach has achieved SOTA results on long document topic segmentation, further re- search is required on how to more efficiently model even longer context. In addition, our method needs to construct augmented data for the TSSP task, which will take twice the training time.
Lan Du, Wray Buntine, and Mark Johnson. 2013. Topic segmentation with a structured topic model. In Pro- ceedings of the 2013 conference of the North Amer- ican chapter of the Association for Computational Linguistics: Human language technologies. pages 190-200.
References
Sebastian Arnold, Rudolf Schneider, Philippe Cudré- Mauroux. Felix A Gers, and Alexander Löser. 2019, Sector: A neural model for coherent topic segmenta- tion and classification. Transactions of the Associa- tion for Computational Linguistics, 1:169-184.
Jacob Eisenstein and Regina Barzilay, 2008, Bayesian unsupervised topic segmentation. In Proceedings of the 2008 Conference Empirical Methods in Natural Language Processing, pages 334-343.
Micha Elsner and Eugene Charniak. 2011. Extend- ing the entity grid with entity features. Proceedings of the 49th Annual Meeting of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 125-129.
Pinkesh Badjatiya, Litton J Kurisinkel, Manish Gupta, and Vasudeva Varma. 2018. Attention-based neural text segmentation. In Advances in Information Re- trieval: 40th European Conference on IR Research, ECIR 2018, Grenoble, France, March 26-29, 2018, Proceedings, pages 180-193. Springer.
Yaxin Fan and Feng Jiang. 2023. Uncovering the poten- tial of chatgpt for discourse analysis in dialogue: An empirical study. arXiv preprint arXiv:2305. 08809....
Joe Barrow, Rajiv Jain, Vlad Morariu, Varun Manju- natha, Douglas W Oard, and Philip Resnik. 2020. A joint model for document segmentation and segment labeling. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pages 313-322.
Haoyu Gao, Rui Wang. Ting-En En Lin, Yuchuan Wu, Min Yang. Fei Huang, and Yongbin Li. 2023. Unsuper- vised dialogue topic segmentation with topic-aware utterance representation. arXiv e-prints, pages arXiv 2305.
Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An basee approach. Compu- lational Linguistics, 34(1):1-34.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 6894-6910.
Mostafa Bayomi and Séamus Lawless. 2018 C-hts: A concept- based bierarchical text segmentation ap- proach. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
Goran Glavaš, Federico Nanni, and Simone Paolo Ponzetto. 2016. Unsupervised text segmentation us- ing semantic relatedness graphs. In Proceedings of the Fifth Joint Conference on Lexical and Computa- tional Semantics, pages 125-1300
Doug Beeferman, Adam Berger, and John Lafferty. 1999. Statistical models for text segmentation. Ma- chine learning, 34:177-210.
Iz Beltagy, Matthew E Peters, and Arman Cohan, 2020. Longformer: The long-document transformer. arXiv preprint 2004.0000
Zheng Gong, Shiwei Tong. Han Wu, Qi Liu, Hanqing Tao, Wei Huang, and Runlong Yu. 2022. Tipster: A topic-guided language model for topic-aware text segmentation. In Database Systems for Advanced Ap- plications: 27th International Conference, DASFAA 2022, Virtual Event, April 11-14, 2022, Proceedings, Part III, pages 213-221. Springer.
Harr Chen, SRK Branavan, Regina Barzilay, and David R Karger. 2009. Global models of document structure using latent permutations. Association for Computational Linguistics.
Freddy YY Choi. 2000. Advances in domain indepen- dent linear text segmentation. In 1st Meeting of the North American Chapter of the Association for Com- putational Linguistics.
Amir Hazem, Béatrice Daille, Dominique Stutz- mann, Christopher Kermorvant, and Louis Cheva- lier. 2020. Hierarchical text segmentation for me- dieval manuscripts. In Proceedings of the 28th Inter- national Conference Computational Linguistics. pages 6240-6251.
Somnath Basu Roy Chowdhury, Faeze Brahman, and Snigdha Chaturvedi. 2021. Is everything in order? a simple way to order sentences. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10769-10779.
Marti A Hearst. 1994, Multi-paragraph segmentation expository text. In 32nd Annual Meeting of the Asso- ciation for Computational Linguistics, pages 9-16.

---

Marti Hearst. 1997. Text tiling: Segmenting text into multi-pen paragraph subtopic passages. Computational linguistics, 23(1) 3))333644.
Lev Pevzner and Marti Hearst. 2002. A critique and improvement of an evaluation metric for text segmentation. Computational Linguistics, 28(1):19- 36.
Hakan Inan, Rashi Rungta, and Yashar Mehdad. 2022. Structured summarization: Unified text segmentation and segment labeling as a generation task. arXiv preprint arXiv:2209.0 13759.
Violaine Prince and Alexandre Labadié. 2007. Text seg- mentation based on document understanding for in- formation retrieval. In Natural Language Processing and Information Systems: 12th International Confer- ence on Applications of Natural Language to Infor- mation Systems, NLDB 2007, Paris, France, June 27- 29, 2007. Proceedings 12, pages 295-304. Springer.
Shoaib Jameel and Wai Lam. 2013. An unsupervised topic segmentation model incorporating word order. In Proceedings of the 36th international ACM SIGIR conference on Research and development informa- lion retrieval, pages 203-212.
Martin Riedl and Chris Biemann. 201 2a. Text segmen- tation with topic models. Journal for Language Tech- nology and Computational Linguisties, 27(1):47-66
Jacob Devlin Ming-Wai Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirec- tional transformers for language understanding, In Proceedings of NAACL-HLT, pages 4171-4186
Martin Riedl and Chris Biemann. 2012b. Topictiling: text segmentation algorithm based on Ida. In Pro- ceedings of ACL 2012 student research workshop. pages 37-42.
Omri Koshorek, Adir Cohen, Noam Mor, Michael Rot- man, and Jonathan Berant. 2018. Text segmentation as a supervised learning task. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies, Volume 2 (Short Pa- pers), pages 0693473..
Gennady Shtekh, Polina Kazakova, Nikita Nikitinsky, and Nikolay Skachkov. 2018. Exploring influence of topic segmentation on information retrieval quality. In Internet Science: 5th International Conference, INSCI 2018, St. Petersburg, Russia, October 24-26, 2018, Proceedings 5, pages 131-140. Springer.
Zhenzhong Lan. Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite hert for self-supervised learn- ing of language representations. arXiv preprint urXiv:1909.0 1et..1999191111 1999.11942. 11942.
Alessandro Solbiati, Kevin Heffernan. Georgios Damaskinos, Shivani Poddar, Shubham Modi, and Jacques Cali. 2021. Unsupervised topic segmentation of meetings with bert embeddings. arXiv preprint 1.Xiv:2106.1297:
Yang Liu, Chenguang Zhu, and Michael Zeng. 2022. End-to-end segmentation- news summarization. In Findings of the Association for Computational Linguistics: ACL 2022. pages 544-554.
Swapna Somasundaran et al. 2020. Two-level trans- former and auxiliary coherence modeling for im- proved text segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence vol- ume 34, pages 7797-7804.
Kelvin Lo. Yuan Jin, Weicong Tan, Ming Liu, Lan Du, and Wray Buntine. 2021. Transformer over pre- trained transformer for neural text segmentation with enhanced topic coherence. In Findings of the Associ- ation for Computational Linguisticx: EMNLP 2021, pages 3334-3340.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang. Sebastian Ruder, and Donald Metzler. 2020. Long range arena: A benchmark for efficient transformers. arXiv preprint 20X::2011.040000
Lajanugen Logeswaran, Honglak Lee, and Dragomir Radev, 2018. Sentence ordering and coherence mod- eling using recurrent neural networks. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 32.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS, pages 5998-6008.
Tlya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint arXiv:/711.05/0..
Liang Wang. Sujian Li, Yajuan Lü, and Houfeng Wang. 2017. Learning to rank semantic coherence for topic segmentation. In Proceedings of the 2017 Confer- ence Empirical Methods in Natural Language Processing, pages 1340-1344.
Michal Lukasik, Boris Dadachev. Kishore Papineni, and Gonçalo Simóes. 2020. Text segmentation by cross segment attention. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 4707-4716.
Hemant Misra, François Y von, Joemon M Jose, and Olivier Cappé. 2009. Text segmentation via topic modeling: an analytical study. In Proceedings of the 18th ACM conference on Information and knowledge management, pages 1553-1556.
Wei Wang, Bin Bi, Ming Yan, Chen Wu, Zuyi Bao. Jiangnan Xia, Liwei Peng, and Luo Si. 2019. Struct- bert: Incorporating language structures into pre- training for deep language understanding. arXiv e- prints, pages XrXiv-1908.

---

Zhuofeng Wu, Sinong Wang, Jiatao Gu. Madian Khabsa, Fei Sun, and Hao Ma. 2020. Clear: Contrastive learning for sentence representation. arXiv preprint 2tl::2012.114666
evaluate ChatGPT on the WIKI-50 test set. The prompts are shown in Table 5. We set tempera- ture as 0 to ensure the consistency of ChatGPT. The post-procsssing strategy for ChatGPT remains unchanged to obtain the formatted output. Differ- ent from Fan and Jiang (2023), we change the key word dialogue to document in the prompt and try to use one-shot prompt to see if it can improve per- formance.
Jinxiong Xia, Cao Liu, Jiansong Chen, Yuchen Li, Fan Yang, Xunliang Cai. Guanglu Wan, and Houfeng Wang. 2022. Dialogue topic segmentation via paral- lel extraction network with neighbor smoothing. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Infor- mution Retrieval, pages 1136-2131.
Table shows the results of ChatGPT with differ- ent prompts and Longformer with supervised data. Firstly, the results show that the generative prompt can achieve higher performance than the discrim- inative prompt, which is consistent with the con- clusion of Fan and Jiang (2023) that representing structure directly can better leverage the generation ability of ChatGPT. Additionally, incorporating single example in the generative prompt can fur- ther improve 2-point F1, which indicates one-shot prompt can better stimulate the in-context learning ability of Large Language Models. However, while ChatGPT-GP one achieves an F1 metric that is 4.6 points higher than LongformeToot: its' P&: and WD metric are significantly worse due to the high false recall rate of topic boundaries. This suggests that how fully apply the ability of Large Language Models to the topic segmentation in long docu- ments remains be further explored. Finally, com- pared with ChatGPT, the significant improvement of Longformer shows the key role supervised data plays in the topic segmentation task.
Wen Xiao and Giuseppe Carenini. 2019. Extractive summarization of long documents by combining global and local context. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing TEMNLP-IJCNLP) pages 3011-3021.
Linzi Xing and Giuseppe Carenini. 2021. Improv- ing unsupervised dialogue topic segmentation with utterance-pair coherence scoring. Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 167-177.
Linzi Xing, Brad Hackinen, Giuseppe Carenini, and Francesco Trebbi. 2020. Improving context mod- eling in neural topic segmentation. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 626-636.
Yuanmeng Yan, Rumei Li, Sirui Wang. Fuzheng Zhang, Wci Wu, and Weiran Xu. 2021. Consert: A con- trastive framework for self-supervised sentence repre- sentation transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 1ith International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 5065-5075.
B Training Details
Our experiments are implemented with transform- ers package The model parameters are initialized with corresponding pre-trained parameters. The initial learning rate is 5e 5 and the dropout prob- ability is 0.1. AdamW (Loshchilov and Hutter, 2017) is used for optimization. The batch size for WIKI-727K and WikiSection is 4 and 8, and the epoch for WIKI-727K and WikiSection is 3 and 5 respectively. We set the number of positive pair in CSSL as k = and carry out grid-search of loss weight a1, 02 E [0.5, 1.0], k E [1, 2, 3, 4] on dev set. The final configuration in the two bench- marks is 22 3, aj 0.5. 02 performs best on WikiSection when set 1.0 while on WIKI-727K 02 0.5 performs best.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:1 111177788-79797
Qinglin Zhang, Qian Chen, Yali Li, Jiaqing Liu, and Wen Wang. 2021. Sequence model with adaptie sliding window for efficient spoken document seg- mentation, In 202] IEEE Automatic Speech Recog- nition and Understanding Workshop (ASRU), pages 411 411-418. IEEE
A ChatGPT for Topic Segmentation in Long Document
To investigate the performance of ChatGPT on long document topic segmentation, we adopt the prompts proposed by Fan and Jiang (2023) and

---

Type
Prompts for Document Topic Segmentation
The following document. Give each utterance binary label, where indicates that the utterance starts a new topic. please output the result of the sequence annotation python list.
Discriminative
0: $1
1: 8:
- 1: 1: 8n
Please identify several topic boundaries for the following docu- ment and each topic consists of several consecutive utterances. please output in the form of [topic i:[] ,opicc j:(]), where the elements in the list are the index of the consecutive utterances within the topic, and output even if there is only one topic.
Generative
O: 81
1:
1: 8n
Table 5: Prompts for Document Topic Segmentation. denotes the number of sentences and s; denotes i-th sentence in the document.
Model
WIKI-50
Model
WIKI-737K ....
WIKE-72KK...
Ft
PL L
WD
R.7
PLJ
WD
F1 +
Py +
WD 4
LonglorneeBBus +SSPP=CSSLoorn
83.
11.5ca
1176ca
15:4
KAM9
1481;
1622
ChatGPT-DP zero
23.28
52.83
66.63
ChatGPT-GP sero
49.59
37.79
46.18
Table 7: The performance of Longformer Base and our approach on short and long document subsets of WIKI- 727K test set,* indicates the gains from +TSSP+CSSL over Longformer-Base are statistically significant with
ChatGPT-DP one
21.91
54.86
71.40
ChatGPT-GP ome
51.61
37.19
45.21
Longformerood
47.01
29.04
31.41
p < 0.05.
Longformer,ad
76.32
11.01
11.82
Scort
<n.xdit
Table 6: Comparison of ChaiGPT and Longformer on the WIKI-50 test set. Longformer denotes fine- tuning Longformer WikiSection and Longformer,ud denotes fine-tuning Longformer on WIKI-727K. DP and GP are short for discriminative prompt and generative prompt, respectively. zero and one denote zero-shot and one-shot prompting settings.
F, 1
7V. L
WD
F1
1
Ph. J
WD
Longformer
82.18
82.05
77.60
16,89
+SSP=CSSSL
Preb Only
X3.66
7.33
9.34
74.19
16.29
20.29
Table 8: The results of combing the probability and co- sine similarity to predict topic boundary on en_ciry and en disease. Prob Only denotes only using probability. Prob and Sim denotes compute
Formula 10.
C Performance of the Proposed Approach on Short and Long Documents
(+TSSP+CSSL) are shown in Table 7. Our ap- proach significantly improves the baselinc on both short and long documents. Notably, the gains from our approach larger long documents, sug- gesting that our coherence modeling benefits long document topic segmentation even more. This is consistent with our hypothesis.
It is important to note that our proposed TSSP and CSSL approaches are agnostic to document lengths and are applicable to models and datasets for short documents. In order to evaluate the performance of our proposed approach on various document lengths, partition the WIKI-727K test set into short document subset (18310 samples) and long document subset (54922 samples) according whether the number of tokens document less than 512 or not. The results from the baseline Longformer and Longformer with our approaches
D Ensemble Probability and Similarity
As shown in Formula 10, we try combine the co- sine similarity of neighbor sentence representations (Sim) and model probability (Prob) to get the final score to infer topic boundary. Specifically, we get

---

different thresholds intervals of 0.05 from 0 to 1. Then we choose the threshold when Fi reaches the best in the dev set, and finally use this thresh- old to obtain the effect on the test set. The results shown Table suggest that the score combining similarity of neighbor sentence representations and model probability lower the performance slightly, We speculate that the boundary probability predic- tion and auxiliary coherence tasks performed during training simultaneously, therefore the model has incorporated more features than just similarity.
1 Score -igmoidd-
(10)
1
Sigmoid(z)+ = 1+e =
(11)


Reviews:
[{"review_id": "MDpH0H7oMM", "reviewer": "Reviewer_b6Np", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful reviews and valuable feedback.  Below are our responses to all of your questions and concerns.\n\n**Q1: The motivation is not solid. Detailed explanation of the underlying mechanism of this method is needed.**\n\n**Response:** The motivation of our work stems from the findings in prior research works that **coherence plays a key role in understanding both logical structures and text semantics**. As summarized in Line 174-200 of our paper, there have been prior works that develop models for comprehending text coherence and tasks to measure effectiveness of the models, such as predicting the coherence score of documents [1], predicting the position where the removed sentence was originally located [2] and restoring out-of-order sentences [3]. Prior works explored incorporating coherence modeling into topic segmentation and achieved better topic segmentation performance. However, **all of prior works consider coherence modeling for topic segmentation only from a single perspective**. [4] only models coherence for understanding only logical structures and forces the model to differentiate coherent sequences of sentences from corrupt ones. [5] models coherence for understanding only text semantics and reduces the semantic coherence scores of the two sentences across topic boundaries. \n\nIn contrast to these works, **our work is the first to consider topical coherence as both text semantic similarity and logical structure (flow) of sentences**. As described in Line 092-112 of our paper, we design the two auxiliary coherence-related tasks, Topic-aware Sentence Structure Prediction (TSSP) and Contrastive Semantic Similarity Learning (CSSL), in order to regulate the topic segmentation model to grasp text coherence from both perspectives of logical structures with the TSSP task and text semantic similarity with CSSL. We conducted extensive ablation study to analyze the effect of each module. The ablation results clearly demonstrate that both TSSP and CSSL contribute to the improvements of topic segmentation performance.  Importantly, **these two approaches exhibit complementary effects when combined**, confirming the necessity of modeling both sentence structure and text semantic similarity for identifying topic boundaries.\n\n\n**Q2: Method do not have a substantial difference from conventional approaches**\n\n**Response:**   While it is acknowledged that data augmentation and contrastive learning are well-established techniques in NLP, they have been explored quite insufficiently for topic segmentation. Also, we carefully customized data segmentation and contrastive learning to specifically address the requirements of the topic segmentation task. As discussed in Line 184-200 of our paper, all prior researches exploring coherence for topic segmentation inadequately model the intricate relationship between text coherence and topic segmentation. \nIn contrast, **our work is the first to consider both perspectives of text coherence for topic segmentation**, that is, modeling logical structures with our proposed TSSP task and modeling text semantic similarity with our proposed CSSL task. As described in Line 271-291 of our paper, our data augmentation is substantially different from prior data augmentation approaches, as we customize data augmentation for topic segmentation by **simultaneously perturbing the document at both the topic and sentence levels**. We further introduce the TSSP task based on these augmented data to enhance the understanding of logical structures. Also, as described in Line 317-326 of our paper, our proposed CSSL task is also substantially different from conventional contrastive learning approaches. **We customizes constrative learning in CSSL to takes into account both *similar* and *dissimilar* sentence pairs to regulate sentence representations for topic segmentation. Specifically, we design constrative learning in CSSL to promote higher semantic similarity between sentences belonging to the same topic and lower semantic similarity between sentences across different topics**.  The integration of our TSSP and CSSL tasks results in substantial improvements over prior approaches for topic segmentation. \n\n\n**Q3: Any design that is specifically tailored for long documents?**\n\n**Response:** Our work indeed focuses on topic segmentation on long documents due to the reason that all the evaluation datasets used in our work are consisted of long documents. As shown in Section 4.1 of our paper, the datasets used in this paper include In-domain Datasets and Out-of-domain Datasets. As described in Line 386-390, the In-domain Datasets WIKI-727K and English WikiSection datasets are widely used as benchmarks to evaluate the text segmentation performance of models. As described in Line 413-421, we followed prior works and adopted WIKI-50 and Elements datasets as Out-of-domain Datasets. As shown in Table 1, the average number of tokens per document for each of these four evaluation datasets ranges from 1321 to 1654, which exceeds the maximum sequence length 512 of the classic BERT-based pre-trained models. Hence, these datasets are considered long-document datasets. \n\nWe specially tailor the backbone model selection for long documents, as elaborated in Line 441-466. We model topic segmentation as a sequence labeling task since prior work [6] shows that sequence labeling is more effective and efficient than paired sentence or chunk modeling for topic segmentation. On this basis, as shown in Table 2, we compare the topic segmentation performance using the BERT-base model with sliding window and several competitive efficient transformers which are specially designed for long sequence modeling, including BigBird-base and Longformer-base, on the WikiSection dataset. As discussed in Line 456-466 and Line 499-502, Longformer-Base outperforms all baselines in the first group of Table 2 and Table 3 and outperforms BERT-Base and BigBird-Base in Table 2. As shown in Table 2, Longformer-Base achieves **82.19 and 72.29 F1**, greatly outperforming BERT-Base (78.99 and 67.34 F1) by **(+3.2, +4.95) F1** and BigBird-Base (80.49 and 70.61 F1). **Hence we select Longformer-Base as the encoder for the main experiments.**  We also show in Figure 3 that the topic segmentation F1 gradually improves as the context length increases, demonstrating the importance of context lengths. \n\nConsequently, on the long-document evaluation datasets, with backbone selected as Longformer to effectively model long documents, we apply our proposed TSSP and CSSL to Longformer, which significantly improves Longformer (and BigBird and BERT) and leads to the SOTA results (Line 510-520). \n\n**It is also important to note that our proposed TSSP and CSSL approaches are agnostic to document lengths and are applicable to models and datasets for short documents**. In future work, we plan to evaluate effectiveness of our methods on short documents or dialogue datasets.\n\n\n**Q4: Lack strong and recent baselines for fair comparison**\n\n**Response:**\nWe respectfully disagree on the reviewer's comment that our paper lacks strong and recent baselines for comparison. Firstly, we would like to emphasize that all the evaluation datasets in our experiments are widely used in prior works on text segmentation (the first paragraph in Response to Q3 includes more details). Secondly, **all the baselines we chose for comparison are well-established and exhibit top performance on these benchmarks**. We cited results from all strong and recent baselines and reproduced the best performing prior works that open-sourced their code.\nAs noted in Line 467-470, in Table 2, we reproduced results of Longformer-Base on WikiSection with the auxiliary CATS or CSC task described in Section 2.2, which is clearly the strongest baseline. In Table 3, we also reproduced the strong baseline with open-sourced code, Seq-BERT-Base and Seq-ELECTRA-Base. In our experiments, to ensure fair comparisons, we adopted the same partitioning and preprocessing of the datasets as used by the baseline models. \n\nIn addition, in order to show that our methods does not require additional label information, we also compare to complex baselines that utilize descriptive labels for each topic, such as Transformer$^{2}_{BERT}$ and Naive LongT5-Base-SS in Table 2. Here we apologize for a typo in Table 2: The last row of group 1 should be Naive LongT5-Base-SS instead of Naive LongT5-Base-DS.  Still, our Longformer-Base baseline already outperforms all baselines of BERT-Base and BigBird-Base. \n\nUpon acceptance of the paper, we plan to make public available all the baseline models we build and reproduce, including Longformer-Base with different context lengths and adding auxiliary coherence tasks from prior works, as well as other backbone models including BERT-Base and BigBird-Base.\n\n\n**Reproducibility: 3**\n\n**Response:** To ensure the transparency and reproducibility of our experiments, we have already provided all the relevant code and experimental results in the supplementary materials. All the training/evaluation datasets are publicly available and references of all datasets are provided in our paper. The code we provided can reproduce the experimental results. We have also provided implementation details in Section 4.1 Experimental Setup and details of hyperparameters in Appendix B (Line 490-491).\n\n\n[1] Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An entity-based approach. Computational Linguistics, 34(1):1–34.\n\n[2] Micha Elsner and Eugene Charniak. 2011. Extending the entity grid with entity-specific features. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 125–129.\n\n[3] Somnath Basu Roy Chowdhury, Faeze Brahman, and Snigdha Chaturvedi. 2021. Is everything in order? a simple way to order sentences. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10769–10779.\n\n[4] Swapna Somasundaran et al. 2020. Two-level transformer and auxiliary coherence modeling for improved text segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7797–7804.\n\n[5] Linzi Xing, Brad Hackinen, Giuseppe Carenini, and Francesco Trebbi. 2020. Improving context mod- eling in neural topic segmentation. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 626–636.\n\n[6] Qinglin Zhang, Qian Chen, Yali Li, Jiaqing Liu, and Wen Wang. 2021. Sequence model with self-adaptive sliding window for efficient spoken document segmentation. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 411–418. IEEE.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The introduction of TSSP and CSSL offers fresh perspectives and methods to improve topic segmentation.\n- Experiments on multiple datasets are provided to demonstrate the gain of the proposed method.\n- This paper generally is well-structured and easy to follow.", "reasons_to_reject": "- The motivation is not solid. While sentence-level coherence is demonstrated to be effective in the experiments, the authors lack a detailed explanation of the underlying mechanism of this method.\n- The novelty is limited. I understand that data augmentation and contrastive learning are new to the task of topic segmentation. However, the methods used by the authors do not have a substantial difference from conventional approaches.\n- The proposed method is inconsistent with the paper's description. The authors repeatedly mention 'long documents' at the beginning, but in the methodology section, I do not see any design that is specifically tailored for long documents.\n- The paper lacks strong and recent baselines for comparison. In the experimental section, the authors compare their approach with multiple baseline models, but these models are either designed for more complex scenarios or have not been published. Such comparisons do not seem to be fair.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on the topic segmentation of long documents, a critical task for structuring documents and improving information retrieval. The authors argue that existing supervised neural models for this task have largely overlooked the intricate relationship between semantic coherence and topic segmentation. To address this, the paper introduces two methods: Topic-aware Sentence Structure Prediction (TSSP) and Contrastive Semantic Similarity Learning (CSSL). TSSP is designed to enable the model to understand the structural relationships between sentences in a document, while CSSL aims to ensure that sentence representations within the same topic are semantically similar and distinct from those in different topics. The experimental results show improvements over previous state-of-the-art methods.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The text flows smoothly, with very few minor typos or grammar errors."}, "extra_text": {}}, {"review_id": "rDjRXfOncY", "reviewer": "Reviewer_5wiE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for all valuable feedback. Below we address all your questions and concerns.\n\n**Q1: Additional computational cost brought by TSSP and CSSL is unmeasured**\n\n**Response:** The CSSL module does not introduce any additional parameters. It is used to adjust the sentence representations learned by the encoder and can be computed in parallel to the TSSP module. The TSSP module requires a linear transformation layer with only $768*3$ parameters to predict the structural relationship between sentences. The computational cost of this linear transformation layer is negligible compared to that of the encoder. As mentioned in Section Limitations, the TSSP module works on augmented data, hence it requires twice the training time compared to the baseline model. However, our proposed TSSP and CSSL modules do not change the inference time since we still utilize the same output of encoder and predictor to obtain results. We would like to clarify that Line 537-547 of our paper compares inference speed of the two baselines Longformer-Base and BERT-Base; however, what we should have clarified in the paper is that **our proposed TSSP and CSSL methods do not bring any additional computational cost to inference and do not change inference speed**. We will clarify this point in the revised version.\n\n\n**Q2: The way to sample negative samples may be questionable. The blue and yellow sentences in Figure 1 may cover similar topical semantics. How to avoid this should be investigated.**\n\n**Response:** It is possible that two sentences in different topics could cover similar topical semantics. However, we assume that **two sentences or segments from the same topic are inherently more coherent than those from different topics**. Therefore, the motivation of our CSSL module is to adjust the sentence representation learning to grasp the **relative** high and low coherence relationship. In future work, in order to avoid pulling away sentence pairs in different topics covering similar topical semantics, we plan to consider refining the loss, such as assigning loss weights based on their semantic similarity.\n\n\n**Q3:  Method details are unclear including how to replace topics randomly in TSSP and sample negative pairs in CSSL.**\n\n**Response:** Thank you very much for this question. We apologize for insufficient information on these two details. In Step 2 of Figure 2, for a randomly selected subset of $p_1$ percent of the documents, we replace each topic in them with a topic snippet from other documents with a probability of $p_2$ and keep the same with the probability of $1-p_2$. The default values of $p_1$ and $p_2$ are both 0.5. As for selecting the negative pairs in CSSL, we select $k_2$ sentences from different topics as negative samples, based on the ordering of the distance of a sentence from the anchor sentence, starting from the nearest to the farthest.\n\n\n**Q4: Some tables are a bit too small, like Tables 2 and 3**\n\n**Response:** We will adjust the sizes of Table 2 and 3 so that they are better presented.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well-organized and easy to follow.\n2. The proposed TSSP and CSSL methods are overall reasonable and easy to understand.\n3. The experiments in this paper are extensive.\n", "reasons_to_reject": "1. The additional computational cost brought by TSSP and CSSL is unmeasured.\n\n    The cost seems to be high especially because this paper focuses on the topic segmentation of long documents.\n\n\n2. The way to sample negative samples may be questionable.\n\n    How to obtain negative samples is not reported in detail.\n    For example, the blue and yellow sentences in Figure 1 may cover similar topical semantics.\n    But CSSL considers them as negative pairs. This will hinder sentence representation learning.\n    How to avoid this should be investigated.\n\n\n3. Some method details are unclear.\n\n    How exactly to replace topics randomly in step 2 of TSSP?\n\n    How exactly to sample negative pairs in CSSL?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on improving topic segmentation with two auxiliary tasks, topic-aware sentence structure prediction (TSSP) and contrastive semantic similarity learning (CSSL).\nTSSP augments a document and then predicts its sentence-pair relations to facilitate sentence representation learning.\nCSSL applies contrastive learning to regulate sentence representations by sampling contrastively positive and negative sentence pairs.\nThis paper conducts extensive experiments to evaluate the performance of the proposed methods.\n", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Some tables are a bit too small, like Tables 2 and 3.\n"}, "extra_text": {}}, {"review_id": "y3NShhfbd1", "reviewer": "Reviewer_wwEk", "summary": "", "questions": "1. The paper does not seem to reference recent advancements in topic contrastive learning, as outlined in [1][2]. Could you explain why these were not considered in your study? Would incorporating these advancements impact your methods and results?\n2. Could you discuss the potential impact of your research on the broader community? How do these methods contribute to the advancement of the field?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful reviews and valuable feedback. Below we address all of your questions and concerns.\n\n**Q1: What's the potential impact of this research on the broader community.**\n\n**Response**: First of all, topic segmentation is a critical task for structuring documents and topic segmentation plays a key role in many downstream tasks such information extraction [1] and document summarization [2] (Line 042-047 of our paper). Our methods significantly improve topic segmentation performance, hence our work will significantly benefit document understanding and many downstream tasks. \n\nFurthermore, it is worth noting that our proposed TSSP and CSSL methods, with adjustments, can be applied to a broad spectrum of segmentation tasks at various levels of granularity and with modalities other than text. For example, our methods can be applied to models for paragraph segmentation or section segmentation on text or scene segmentation on narrative texts [3]. Our methods can also be applied to the computer vision field such as scene boundary detection of video [4]. We believe that our work holds great potential on the broader community. These topics can be explored in future directions to demonstrate broader impacts of our work.\n\n\n**Q2: Missing the recent advancements in topic contrastive learning. Would incorporating these advancements impact the methods and results?**\n\n**Response**: Thank you for recommending paper [5] and paper [6]. Paper [5] was published on arxiv on July 5, 2023, which is after EMNLP 2023 direct paper submission deadline June 23, 2023. Consequently, we did not cite this work. Paper [6] focuses on producing high-quality topic distributions of short texts by utilizing contrastive learning to model the relations among samples. This work is different from identifying topic boundaries within the document, as done by our work and the related works we discussed and compared to in the paper. Nevertheless, we speculate that encoding the entire topic segment into a semantic space and obtaining a topic-level contrastive representation may help detecting topic boundaries, which we plan to explore in future work.\n\n\n**Q3: Table 2,3,4 are too small to read comfortably**\n\n**Response:** We will adjust the sizes of the tables so that they could be better presented.\n\n\n**Reproducibility: 3**\n\n**Response:** To ensure the transparency and reproducibility of our experiments, we have already provided all the relevant code and experimental results in the supplementary materials. All the training/evaluation datasets are publicly available and references of all datasets are provided in our paper. The code we provided can reproduce the experimental results. We have also provided implementation details in Section 4.1 Experimental Setup and details of hyperparameters in Appendix B (Line 490-491).\n\n[1] Gennady Shtekh, Polina Kazakova, Nikita Nikitinsky and Nikolay Skachkov. 2018. Exploring influence of topic segmentation on information retrieval quality. In Internet Science: 5th International Conference, INSCI 2018, St. Petersburg, Russia, October 24–26, 2018, Proceedings 5, pages 131–140. Springer.\n\n[2] Yang Liu, Chenguang Zhu, and Michael Zeng. 2022. End-to-end segmentation-based news summarization. In Findings of the Association for Computational Linguistics: ACL 2022, pages 544–554.\n\n[3] Zehe A, Konle L, Dümpelmann L K, et al. Detecting scenes in fiction: A new segmentation task[C]//Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 2021: 3167-3177.\n\n[4] Rao A, Xu L, Xiong Y, et al. A local-to-global approach to multi-modal movie scene segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 10146-10155.\n\n[5] Luo, Zheheng, Lei Liu, Qianqian Xie and Sophia Ananiadou. “Graph Contrastive Topic Model.” ArXiv abs/2307.02078 (2023): n. pag.\n\n[6] Wu, Xiaobao, Anh Tuan Luu and Xinshuai Dong. “Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning.” Conference on Empirical Methods in Natural Language Processing (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Writing: The paper is well-structured and clearly presents the problem statement, its importance, and the proposed solutions.\n2. Idea: The paper proposes innovative methods (TSSP and CSSL) to enhance topic segmentation in long documents. These methods address existing limitations in supervised neural models, particularly in exploring the deeper relationship between semantic coherence and topic segmentation.\n3. Experiment: Experimental results demonstrate the improving of the performance of topic segmentation in long documents.", "reasons_to_reject": "- Limited Impact: The proposed methods are limited in their application to topic segmentation, which have little impact on the broader research or application community.\n- Missing references: The authors ignore the recent advancements in topic contrastive learning, such as [1][2].", "questions_to_authors": "1. The paper does not seem to reference recent advancements in topic contrastive learning, as outlined in [1][2]. Could you explain why these were not considered in your study? Would incorporating these advancements impact your methods and results?\n2. Could you discuss the potential impact of your research on the broader community? How do these methods contribute to the advancement of the field?", "paper_topic_and_main_contributions": "This paper proposes an innovative method to enhance the process of topic segmentation in long documents, improving their structure and the effectiveness of information retrieval tasks. The approach focuses on enhancing semantic coherence in topic segmentation by introducing two new methods. The first, Topic-aware Sentence Structure Prediction (TSSP), is designed to improve the model's understanding of the sentence structure in relation to the topic. The second, Contrastive Semantic Similarity Learning (CSSL), focuses on learning the semantic similarity or difference between sentences. The experimental results suggests that the implementation of these methods could lead to significant improvements in the topic segmentation process.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Table 2,3,4 are too small to read comfortably; consider enlarging the table, increasing the font size, and improving the contrast for better readability."}, "extra_text": {"Missing_References": "[1] Wu, Xiaobao, Anh Tuan Luu and Xinshuai Dong. “Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning.” Conference on Empirical Methods in Natural Language Processing (2022).\n[2] Luo, Zheheng, Lei Liu, Qianqian Xie and Sophia Ananiadou. “Graph Contrastive Topic Model.” ArXiv abs/2307.02078 (2023): n. pag."}}]

Author rebuttal:

Meta-review: This paper introduces Topic-aware Sentence Structure Prediction (TSSP) and Contrastive Semantic Similarity Learning (CSSL) into modeling long document topic segmentation. The experimental results show their model based on longformer outperforms the existing SOTA methods. 

All three reviewers believe that the paper has a good structure and clear organization, and the proposed solution has a certain novelty. However, there are certain concerns about motivation, increasing the cost and impact of using TSSP and CSSL, and the details still need to be further improved. I suggest the author refine the paper based on the valuable feedback from the reviewer, whether accepted or not.
Decision: Accept-Main