A Reference-free Segmentation Quality Index (SegReFree)
Authors: Evan Lucas
Dylan Kangas
Timothy Havens
Venue: EMNLP 2023 Findings (2023)
Keywords: segmentation
metrics
TL;DR: We propose the first (to the best of our knowledge) text segmentation metric that doesn't require a reference segmentation set.

Abstract: Topic segmentation, in the context of natural language processing, is the process of finding boundaries in a sequence of sentences that separate groups of adjacent sentences at shifts in semantic meaning.  Currently, assessing the quality of a segmentation is done by comparing segmentation boundaries selected by a human or algorithm to those selected by a known good reference. This means that it is not possible to quantify the quality of a segmentation without a human annotator, which can be costly and time consuming. This work seeks to improve assessment of segmentation by proposing a reference-free segmentation quality index (SegReFree). The metric takes advantage of the fact that segmentation at a sentence level generally seeks to identify segment boundaries at semantic boundaries within the text. The proposed metric uses a modified cluster validity metric with semantic embeddings of the sentences to determine the quality of the segmentation. Multiple segmentation data sets are used to compare our proposed metric with existing reference-based segmentation metrics by progressively degrading the reference segmentation while computing all possible metrics; through this process, a strong correlation with existing segmentation metrics is shown. A Python library implementing the metric is released under the GNU General Public License and the repository is available at \url{https://github.com/evan-person/reference_free_segmentation_metric}.

A Reference-free Segmentation Quality Index (SegReFree)
Evan Lucas and Dylan Kangas and Timothy C Havens Michigan Technological University / 1400 Townsend Drive Houghton, Michigan, United States of America (eglucas, dylank, thavens} @mtu.edu
Abstract
topic segmentation, which has many applications from information retrieval (Dias al., 2007)
Topic segmentation, in the context of natural language processing, is the process of finding boundarics in a sequence of sentences that sep- arate groups of adjacent sentences shifts in semantic meaning. Currently, assessing the quality a segmentation is done by comparing segmentation boundaries selected or algorithm to those selected by a known good reference. This means that it is possible to quantify the quality of segmentation without human annotator, which can be costly and time consuming. This work seeks to improve ment of segmentation by proposing reference- free segmentation quality index (SegReFree). The metric takes advantage of the fact that seg- mentation at sentence level generally seeks to identify segment boundaries semantic bound- aries within the text. The proposed metric uses a modified cluster validity metric with seman- tic embeddings of the sentences to determine the quality of the segmentation. Multiple seg- mentation data sets are used to compare our proposed metric with existing reference-basee segmentation metrics by progressively degrad- ing the reference segmentation while comput- ing all possible metrics; through this process, strong correlation with existing segmentation metrics is shown. A Python library implement- ing the metric is released under the GNU Gen- eral Public License and the repository is avail- able https://gii thub. com/evan- person/ weweeerrrreeeeettttiiiiie
summarization of long documents (Gidiotis and Tsoumakas, 2020; Zhang et al., 2022).
Segmentation is generally evaluated by compar- ing reference set of possible segment boundaries with the boundaries chosen by given segmen- tation algorithm. This means that segmentation evaluation can only be performed on data that has already been segmented by human evaluator (or other accepted-as-trume segmentation). A vari- ety of methods have been proposed for evaluating segmentation, all of which focus on the reference and candidate sets, rather than the content being segmented. A review of these methods is included in Section 2. We significantly push the state-of- the-art segmentation evaluation by proposing reference- frre segmentation quality metric.
human
The rest of this paper is organized in the follow- ing manner. A brief history of modern segmenta- tion metrics and other related work is described in Section 2. The proposed reference-fre free segmenta- tion quality metric is developed, alongside design choice explanations, in Section 3. Validation of the method, by comparing results existing segmen- tation metrics on popular segmentation data sets are described in Section 4 along with an ablation study to demonstrate the behavior of the method in several scenarios. Potential uses and impacts of this work are presented alongside a summary in Section 5.
1 Introduction
Related Work
Text segmentation is a common task in language processing, in which a text is split into segments. Text segmentation is performed at different scales: when it is used to split sentences into their parts, it is referred to elementary discourse unit (EDU) separation, and when used to split larger documents into topic based groups of sentences, it is known as topic segmentation (Marcu and Echihabi, 2002: Beeferman et al., 1999). This work focuses on
Existing segmentation evaluation metrics, proposed by Beeferman et al. (1999); Pevzner and Hearst (2002); Fournier and Inkpen (2012): Fournier (2013) reviewed, along with a brief introduc- tion to cluster validity metrics. We include re- view of the Davies-Bouldin Index (Davies and Bouldin, 1979), which was used as a starting point for this work. A brief review of some unsupervised

---

segmentation methods that use Sentence-BERR (Reimers and Gurevych, 2019) is also presented.
penalize transposed boundaries (known as near misses) too heavily, and is impacted by segment size variation. WindowDiff is simple to implement and compares the number of boundaries within a moving window. This metric is also widely used in the segmentation literature (Mota et al., 2019; Zhong et al., 2022). Similar to Pk. a score of zero is seen desirable. Again, using Figure as a reference, with the window in the example position shown, the number of boundaries present match and this window position would return score of zero.
2.1 Existing Segmentation Metries
A variety of segmentation evaluation metrics have been proposed (Beeferman al., 1999; Pevzner and Hearst, 2002; Fournier and Inkpen, 2012; Fournier, 2013). To our knowledge, all currently used metrics focus on comparing the set bound- aries generated by the segmentation algorithm to some other set of boundaries, either a true reference or another hypothesis boundary set. Additionally, we not aware of any existing segmentation qual- ity evaluations that based cluster validity metries or the semantic representations of the seg- mented text.
2.1.3 Segmentation Similarity
Fournier and Inkpen (2012) noted that WindowDiff and P3 are both window-based metrics that depend on a reference segmentation, which introduces some issues. They noted that neither metric penal- izes error types (missing or additional boundaries) equally and that window size influences the met- rics' outcome greatly. They proposed new metric called segmentation similarity, which attempted improve upon existing metrics by approximating the distance between segmentation sets in terms of the number of edits required to make them the same. This method has the advantage of being symmetric (neither of the compared segmentation sets are treated as the true reference) and can be ex- tended to compare segmentations between multiple annotators. Unlike PA and WindowDiff segmen- tation similarity similarity measurement and score of considered to be a perfect match.
Existing segmentation metrics can be split into two categories, window-based and mass-based, with P& and WindowDiff being window-based and Segmentation Similarity and Boundary Similarity being mass- mass-based. A simple visualization of these two ways of considering segmentation boundaries is presented in Appendix C. The SegEval (Fournier, 2013) implementation was used for the four classic segmentation metrics used in this work.
2.1.1 Ps
The Ps metric was first proposed by Beeferman et al. (1999) and is widely used for evaluating seg- mentation quality (Amool et al., 2019; Xia et al., 2022). Pse is the conditional probability of a bound- ary classification eтror occurring within a window of & chunks (making this a window based metric). given the reference and hypothesis boundary sets. Errors can be either missing boundary or added boundary within a moving window that goes across the segmentation set. As an error rate, a score of zero is the desirable score. Using Figure 8 as a reference, with the example window in the position shown, Px would not consider there to be an error present. The window is then transposed across the entire segmentation set, with errors calculated at each transposition position, and combined score is created to represent the full segmentation.
2.1.4 Boundary Similarity
Fournier (2013) observed that their proposed seg- mentation similarity metric suffered from opti- mistic values and proposed a new set of metrics derived from the same motivations as segmentation similarity in Fournier (2013). The primary metric proposed is called boundary similarity, although boundary edit distance based confusion matrix and metrics based on that confusion matrix are also proposed in this work. Similar segmentation similarity, a score of is considered optimal for boundary similarity.
2.1.2 WindowDiff
2.2 Alignment based similarity
The Ps metric, although widely used, has some noted shortcomings that were raised by Pevzner and Hearst (2002) in the development of their pro- posed metric, DindowDiff (Pevzner and Hearst, 2002). They observed that Ps penalizes false neg- atives more highly than false positives, tends to
Diaz and Ouyang (2022) propose new method of segmentation metries based on the alignment of segments between a reference and hypothesis segmentation set. This method was not used as a comparison method for this paper.

---

2.3 Cluster validity metries
where N is the number of labels. The maximum term in R2 means that the most similar cluster (in other words, the worse case clustering) to label is included in the final final score can be thought of as the ratio of intra- cluster distance to inter-cluster distance, averaged across the worse case pair of clusters for each label. A low score indicates that clusters are relatively compact and well-separated, whereas high score indicates that clusters are large and/or overlap.
Cluster validity metrics are more generalized ap- proach for assessing quality of segmentation that they were designed to assess the quality of parti- tions found by unsupervised learning algorithms. A variety of cluster validity metrics exist, but in order meet the design goal of not requiring reference, only cluster validity metrics that are reference-free (known in clustering literature as internal evalua- tion schemes) were considered. Three such metrics considered were the Duvies-Bouldin Index (Davies and Bouldin, 1979), the Calinski-Harabasz Index (Caliñski and Harabasz, 1974), and the Silhouette Score (Rousseeuw, 1987). All three of these met- rics attempt to compare the relative distance (or dissimilarity) of the members of cluster to the distance (or dissimilarity) between clusters, with varying differences how they are computed. Sil- houette Score was rejected due to a lack of sensi- tivity when overlapping clusters are present, which is a common situation when considering groups of textual semantic representations. Adaptations of the
index. The
2.4 Unsupervised segmentation methods
Although not a metric, unsupervised segmentation methods are strongly related to the SegReFree met- ric and could be used in some similar ways. For example, an unsupervised segmentation method could be used to create reference segmentation set, which could be then used with any of the seg- mentation metrics previously discussed. It should be noted that this would be a comparison between two different segmentation methods and still sub- ject to the shortcomings whatever segmentation metric was utilized.
Index and Davies-Boulle Index were both used in initial trials, but the adap- tation of the Ravies-Bouldiy Index was found to better demonstrate differences segmentation per- formance.
Two specific segmentation methods that are somewhat similar to our proposed method are published by Solbiati et al. (2021) and Ghinassi (2021). Solbiati al. (2021) uses Sentence-BER (Reimers and Gurevych, 2019) to generate embed- dings as an input for a modified TextTiling algo- rithm that attempts to detect topic changes based on variations in segment similarity. Ghinassi (2021) uses variety of different sentence embedding mod- els along with modified version of TextTiling that is also based on segment similarity. In both cases, they rely on Sentence- BERT embeddings to create semantic representation of the sentences to be segmented and attempt to use that semantic infor- mation to derive similarity based boundaries.
2.3.1
Index
The
Index (Davies and Bouldin, 1979) is computed on a set of labeled points in some n-dimensional space in the following way. For each label i, a centroid of the members of that label is computed and the average distance between the segment centroid and the members of the seg- ment computed and stored as Sj. Any distance measure can be used, but Euclidian distance is seen frequently in the literature.
The distances between all pairs of centroids are computed and stored as Mij- A ratio of pairwise intra-cluster distances and centroid distances is then computed as the following.
Method
Conceptually, our proposed metric seeks quan- tify segmentation quality by assuming that the de- sired segmentation for a given text occurs at topical boundaries. If the segmentation boundary separates two different semantically different groups of text, the semantic vector representations of the sentences in those segments should generally be close to the other sentences within the segment and separated from the sentences in adjacent text segments.
S + S3
Rij =
Mij
The maximum value of Rig for each label i is taken,
R max{ Ray).
and the average of these over all segments is re- ported as the
index value,
1 N DB
At a high level, the proposed metric works by splitting text into sentences or utterances (the term
Ra
N

---

This helps preserve the temporal aspect of segmen- tation, as assessing the quality of boundary is best done by analyzing the adjacent segments to that boundary. Initial experiments found that the adjacent segment restriction improved sensitivity of the final SegReFree score. This effect can be conceptually explained quite easily: the quality of boundary only depends on the segments being separated; topically similar segment that isn't ad- jacent should not influence a descriptive score of segmentation quality. The impact of this change is most apparent when considering the error mode missing boundaries, where adjacent segments get combined, which is presented in Section 4.1.1 and the impact of this modification is presented in the ablation study in Appendix D.
segrertation Algotthim
chumne regmenteiion houndary
semtios embecclinge
mosifiee Dasdee auu ftom
dass 1Dxx to each arrteddinng
Figure 1: Metric computation flowchart
chunks is used as a synonym for both), generat- ing embeddings of the chunks being segmented, and using modified cluster validity metric where we treat segments clusters. A block diagram is shown in Figure 1. Explanations of each step are elaborated upon the rest of this section.
3.1 Text splitting
For the purposes of this paper and without loss of generality, all text was split at the sentence level following the previous usage of the data sets used for validation. This work could be extended to dialogue segmentation evaluation by splitting at the utterance level or it could be used for multi- document segmentation by splitting at a paragraph level.
The second modification required was adding an exponential penalty term to the intra-segment dis- tance computation. This necessary due to the likelihood of adjacent chunks having higher mantic similarity than non-adjacent chunks within segment. Additionally, a segment that has a sin- gle sentence will have an intra-cluster distance of zero (ie. the centroid is the single point). This was found to be problematic during initial inves- tigations where we created artificially low (good) scores by segmenting very frequently. The expo- nential penalty term was empirically created as way to counteract this effect. We present this error mode in Section 4.1.2 and show the impact of this modification on wewlller-thn--desiin segments the ablation study in D. It is still possible to get a score of zero by treating each sentence as an inde- pendent segment, however this is an edge case and not expected segmentation result. Our imple- mentation of SegReFree will allow user to score segmentation with single sentence segments, but will warn them that may cause an artificially low score. Multiple distance measures were considered (Euclidian, LI, and Cosine), with Euclidian being selected. Data supporting our choice of Euclid- ian distance is presented in the ablation study in Appendix D.
3.2 Sentence embeddings
Sentence embeddings were generated using Sentence-BERT using the pre-trained 'all-mpnet- base-v2' model, which was trained on over one billion pairs of sentences from variety of sources (Reimers and Gurevych, 2019). Centence-BERY is modification of the pretrained BERT network, which derives sentence embedding from semantic meanings and seeks to improve the vector space em- bedding over other methods of generating sentence level BERT embeddings. The goal of Sentence- BERT is to improve semantic textual similarity comparisons between sentences, which provides a necessary input for our proposed method. Sentence- BERT passed sentences, from the text splitting, and returns 768 dimension embeddings, which used in conjunction with labels as the input to the modified Davies-Bouldin Index discussed 3.3. Other sentence embedding models could be used with the proposed method, as the SegReFree qual- ity index can be used with any embedding that pro- vides semantic representation of the content being segmented.
The SegReFree Index algorithm works as fol- lows. Embeddings of all sentences and correspond- ing segment labels for all sentences are used as the input. For each segment, centroid of the embed- dings of the members of that segment is computed and the average Fuclidian distance between the seg- ment centroid and the members of the segment are computed; this average intra-cluster distance for
3.3 Modifications to the Davies-Bouldin Index
Two main modifications were required for the Davies-Bouldin Index be used for assessment of segmentation quality. The first was restricting centroid distance calculations to adjacent segments.

---

segment i is referred to as dispersion S,. These intra-cluster distances are then modified by
(Francis and Kucera, 1979), which creates substan- tial topical shifts between boundaries. The '3-11' subset of the Choi data set was used for this work, which includes paragraphs with three to eleven sen- tences each. Two files that included repeating para- graphs were excluded, as they show a very poor (high) score on the given segmentation boundaries between them and throw off average results. It should be noted that this is artificially created data set that has strong topical shifts between seg- ments and the results reported from this data set can interpreted as close to ideal behavior our proposed metric.
S
S
(2)
for all segments containing more than one mem- ber, where n, is the number of members in the ith cluster. To prevent infinite values, the correction factor is set 4 for segments where n, is equal to 1. The distances between centroids of temporally adjacent segments (e.g., between segment and both 1 and i + 1) are computed and stored as Mij. E {i 1, + 1}. Non-temporally adjacent segment distances are not computed or compared, as they are not relevant to the segmentation bound- aries. A ratio of pairwise intra-cluster distances and centroid distances is then computed
The second data set used is the newer Wiki-50 data set, which is scraped from Wikipedia and uses sections and subsections topical segment bound- aries (Koshorek et al., 2018). A larger Wiki-727k data set also available from the same paper, how- ever the Wiki-50 data set sufficient to demon- strate the strengths and limitations of our proposed metric. Both data sets used in this work are freely shared for research purposes and the authors do not anticipate any issues with their inclusion in this work. Three other data sets derived from Wikipedia were also considered: the WikiSection (Arnold et al., 2019) and the Cities and Elements data sets (Chen al., 2009). However, due to the similarity to the Wiki-50 data set they were not included this work.
S; S My
Rij
jE 11
(3)
for both adjacent segments to i. The maximum value of Ray for each segment i taken,
R max R 1, Rag11).
and the average of these over all segments is re- ported as the quality index value,
DB
Σ R.
3.5 Computational infrastructure and budget
Sentence- BERT models were evaluated on either one or two A100 40GB GPUs and cluster metric computations were performed on CPU. The pre- trained aal-mmpettbaseevv2 Sentence-Biir BERT model contains 109.5 million parameters. No models were trained for this work. Total computation time for this paper and related experimentation was less than 100 GPU hours.
where N is the number of segments. Essentially, Rij in (3) is the ratio of the added distances of segments and j and Mij, the distance between centroids of segments i and j. If Raj has a low value then the segments are small compared to the distance between them. If Rt is large then the intra-cluster distances are large compared to the distance between them. Hence, the average of the maximum Rays for each segment represents a measure of the relative semantic dissimilarity of the the temporally adjacent segments are in a given segmentation.
Experiments
A variety of tests were devised to evaluate the per- formance of our metric. Tests were designed to mimic segmentation errors that could be encoun- tered in realistic scenarios.
3.4 Data sets
Two data sets commonly used in segmentation work were used to validate this metric. The first is the Choi data set, introduced by Choi (2000) and used to demonstrate the C99 segmentation algo- rithm. The Choi data set is constructed by taking paragraphs from different files in the Brown corpus
4.1 Metric performance on degraded segmentation sets
This series of trials all follow the same philosophy: to demonstrate how the metric reflects changes in an existing segmentation set as it is degraded

---

SegRefree
0.6
SegRefree
Claster based seg regmentation metric
S WindowDif
1
Cousli Shamm Shomentoo www.mmaaaaoo mestr
Clatte segmentation
2.8-
Windowif!
1-8oundSim
0.6
1-8oundSim
2.a
0
4.5
22
2.2
0.0
boundary
boundary
Figure 2: Proposed metric and classic metrics as a func- tion of number of random boundary removals using the Choi 3-11 data set
Figure 3: Proposed metric and classic metrics as a func- tion of random boundary removals using the Wiki-50 data set
through different operations. For comparison, the four segmentation metrics described in Section 2 are included. All of these trials for each data set were performed in the same way, with the entire corpus being used and the results reported are the mean of all trials across the entire corpus. The Choi 3-11 corpus experiments were repeated five times and averaged to better sample the possible random outcomes and the Wiki-50 corpus exper- iments were repeated 30 times due to the smaller number of files. This brought the total number of averaged files for each corpus to be roughly the same (1500 for Wiki-50 and 1490 for Choi 3-11 due to the two excluded files).
poorer segmentation quality. The strong correlation between existing metric values and SegReFree, function of increasingly degraded segmentation, can be clearly seen. The vertical-axis scales for the SegReFree index are set make the most most effi- cient use space and are therefore using different scales between Figures 2 and 3. The SegReFree in- dex is not bounded to a consistent maximum value and should not be used to compare between models and datasets simultaneously. We suggest using it primarily for comparing segmentation methods on consistent data set, although there may be appli- cations where is appropriate data sets. Conceptually, this can be explained by considering how different data sets use different parts of the embedding space.
compare
4.1.1 Boundary removal
The first test performed the removal of existing boundaries from a segmentation The existing boundaries were randomly selected and removed from each segmentation set. Classic reference- based metrics were computed using the original 'truth' and altered segmentation sets and the pro- posed metric was computed using only the altered segment labels (as ours is reference-free Results for all 30 trials for the entire data set were averaged for each set of boundary deletions. If the metrics are effective, they should show that boundaries are deleted and the segmentation changes from that the 'truth' segmentation that the metric shows this degradation in segmentation quality.
Pearson correlation coefficients computed be- tween classic reference-based metries and our pro- posed reference-free metric are presented in Ta- ble 1. Note that the similarity metries were con- verted to dissimiliarity metrics for consistency: low score indicates good segmentation quality. It can be seen that there is a very strong agreement be- tween the classic metrics and our reference- free metric, with correlation coefficients all 0.95. The trend is stronger with the Choi 3-11 data set than the Wiki-50, but this is expected due to the consis- tently larger topical shifts found in the Choi 3-11 data set.
4.1.2 Segment splitting
Plots of mean proposed and classic metric values function of boundary removals are shown in Figures and for the Choi 3-11 and Wiki-50 data sets, respectively. To maintain vertical-axit consis- tency, the two similarity metrics are presented as one minus the metrie; hence, higher values indicate
For our second experiment, we demonstrate the behavior of the segmentation quality indices when segments are randomly split, thus degrading the segmentation quality by adding boundaries. Seg- ment splitting was performed by randomly select-

---

Data
Pk
Window-
Seg.
Bound.
Set
Diff
Sim.
Sim.
4.36
SegRefree
Choi
0.998
1.000
0.984
0.984
Claster based segmentatinn mebri
1 Windowiff
Casslc Segim www.eeeeeeettommmmmmm ewt.enaaoooo mesri
3-11
Wiki-
0.982
0.987
0.970
0.968
4.34
50
Table Correlation Between Proposed Reference-Free Index and Reference-Band Indices in Boundary Re- moval Experiment
4.30
6.28
'Seg. Sim. and Bound. Sim, are converted dissimilarity metrics consistency.
segments split
Figure 5: Proposed metric and classic metrics as a func- tion of random segment splits using the Wiki-50 data set
4.0
Segietree
Pk
28-
WindowDif
1-Segsim
Data Set
Ps
Window-
Seg.
Bound.
188und5im
Diff
Sim."
Sim.
0.25
3.-
0.20
Choi
0.998
0.999
0.989
1.000
Chistr baved
2.4
3-11
Wiki-
0.939
0.924
0.946
0.902
3.2
0.05
50
0.00
0
Number of segments spitt
Table 2: Correlation Between Proposed Reference-
Index and Reference-Basee Indices in Segment Splitting Experiment
Figure 4: Proposed metric and classic metries func- tion of random segment splits using the Choi 3-11 data set
'Seg. Sim. and Bound. Sim. converted to dissimilarity metrics for consistency.
ing existing segments and splitting at the midpoint (rounding down in cases of odd numbers of sen- tences). As with the boundary removal experiment, the segmentation set was altered and the reference- based metrics were computed by comparing the altered segmentation with the original 'truth' seg- mentation. Only the altered segmentation was used to compute the proposed reference-free metric. The results of each were averaged and shown in Figures 4 and 5 for the Choi 3-11 and Wiki-50 data sets, respectively. Again, the two similarity metrics presented minus the metric to maintain verti- cal consistency; where higher scores for all metries indicate a lower quality segmentation. As with the boundary removal experiment, a strong correlation between existing metrics and SegReFree, as func- tion of increasingly degraded segmentation, can be clearly seen.
with the Wiki-50 data set is also good, although it more inconsistent and almost completely insen- sitive to a single segment split. Thus, the overall correlation is not as strong in the Wiki-50 segment splitting experiment.
4.1.3 Boundary transposition
To simulate the condition of 'near miss", as described in the segmentation metric literature, boundary transposition was also performed. In this experiment, an existing boundary was randomly se-
Data Set
Pk
Window- Diff 0.904
Seg. Sim.* 0.785
Bound. Sim.* x
Choi
0.906
0.800
3-11 Wiki-
-0.320
-0.316
0.423
0.361
50
A table of Pearson correlation coefficients com- puted between the proposed metric and each of the classic metrics presented in Table 2. very strong correlation is observed for the Choi 3-11 data set: all > 0.99. The correlation between our proposed metric and the reference- based metrics
Table 3: Correlation Between Proposed Reference-Fres
Index and Reference-Bartee Indices in Boundary Trans- position Experiment
SSg. Sim. and Bound. Sim. converted to dissimilarity metrics for consistency.

---

Data Set
SegReF 'truth'
SegReF-TT
Pk
Win. Diff
Seg. Sim.
B. Sim.
% correct
Choi 3-11
3.02
4.72
0.582
0.719
0.856
0.071
(98%)
Wiki-50
4.27
4.84
0.485
0.494
0.819
0.161
(68%)
Table Comparison of Segmentation Quality Metrics for Segmentation Found By the TexTTiling Algorithm. Pa. WindowDiff, Segmentation Similarity, and Boundary Similarity are all presented between reference segmentation sets and segmentation sets chosen by TextTiling. The column labeled '% correct' is the percentage of files where SegReFree is directionally correct.
lected and shifted by varying numbers of sentences. As described in Section 2, this is more challeng- ing problem and is handled in different ways by different classic segmentation metrics. Diaz and Ouyang (2022) also notes that boundary transpo- sition is poorly handled by other existing segmen- tation metrics they treat all boundary shifts as equally bad. The mean metrics as a function of transposition distance are displayed Figures 6 and 7 for the Choi 3-11 and Wiki-50 data sets, re- spectively.
311
SegRefree
0,200
3.12
-X-
0.175
WindowDiff
0.150
0.125
0.100
3.06
0.075
Claster
3.04
0.025
A table of Pearson correlation coefficients com- puted between the proposed metric and classic met- rics is displayed Table 3. Our proposed met- ric shows strong agreement with classic metrics with Choi 3-11 data set and directionally cor- rect agreement for the majority of transposition distances tested on the Wiki-50 data set. We hy- pothesize that the degraded performance on this experiment with the Wiki-50 data set comes from the inconsistent size of segments. Because sec- tion and subsection headings are given their own segments, there many single sentence even single word segments within this data set. The se- mantic mcaning of a section and an adjacent subsce- tion heading is often quite similar and when these are combined, it would likely improve (lower) the SegReFree score.
0.000
boundary transpoutionss
Figure 6: Proposed metric and classic metrics as a func- tion of single random boundary transposition using the Choi 3-11 data set
0.40
4.38
Segftereee
Ik
0.35
1-SegSim
WindowDff
0.30
4.34
18oundSim
0.25
4.2
Using existing segmentation methods
0.20
To provide an additional demonstration the met- rics use, an older text segmentation method, Text- Tiling (Hearst, 1997) is used to segment Choi 3-11 and Wiki-50 data sets. This method inten- tionally chosen as lower quality, but still purpose- ful method of segmentation. The NLTK (Bird et al., 2009) implementation of TextTiling was used for this section. Window sizes for TextTiling were chosen arbitrarily ensure least two segments per file, with value of 100 words used for the Choi 3-11 data set and window of 20 words used for the Wiki-50 data set. Results of this test are presented in Table along with classic metrics
4.30
0.15
Clsder
4.28
0.10
0
Number of boundary transpostions
Figure 7: Proposed metric and classic metrics func- tion of single random boundary transposition using the
Wi data set

---

provided for reference. To interpret these results, we assume that the quality of segmentation per- formed by TextTiling is lower than that of the orig- inal source segmentation, and therefore, we should expect a higher (worse) SegReFree score for the TexiTiling segmentation. It can be observed that our proposed metric is directionally correct the ma- jority of the time, showing higher (worse) score for the lower quality segmentation 98% of the time for the Choi 3-11 data set and 68% of the time for the Wiki-50 data set.
meaning, the SegRef'ree score between two dif- ferent texts has no particular meaning or use. It should be used as a way to compare segmentations between consistent text or sets of texts rather than comparing segmentations on dissimilar texts.
Ethics Statement
As this is not a generative task, the authors have no concerns about the content created by this metric. Misuse of the metric is still possibility, such blind reliance upon it when used for important de- cisions. Additionally, there may be ways for the metric to introduce bias in its usage, particularly if the training of the embedding model is not ap- propriate for the domain. It has been shown that contextual word representations, such as BERT, can carry biases (Kurita et al., 2019); so it is also likely that bias present in Centence-BERY would propagate into models dependent on it, such as this one.
5 Conclusions
this work, a new topical text segmentation eval- uation metric that requires no reference segmenta- tion set is proposed and evaluated. To our knowl- edge, this is the first reference- free segmentation quality metric that has been proposed. Strong cor- relation to existing reference-based metrics demonstrated for the worst case errors of missing boundaries and boundary insertion in the middle of a coherent segment. weaker, yet present correla- tion demonstrated for the harder case of bound- ary transposition (so-called "near misses"). The use of our reference-free metric can help enable future segmentation efforts on data that do not have existing human annotation.
References
Sebastian Arnold, Rudolf Schneider, Philippe Cudré- Mauroux, Felix A Gers, and Alexander Löser. 2019, Sector: A neural model for coherent topic segmenta- tion and classification. Transactions of the Associa- rion for Computational Linguistics, 7:169-184.
Limitations
David Baker. 1990. Stargazers look for life. South Magazime, 117:76-77.
This work was performed using a somewhat limited set of data, as the majority of text segmentation data sets found also derived from Wikipedia and were very similar the Wiki-50 data set. likely that this metric may not be the best possible metric for certain segmentation use cases, such as situations where the 'ideal' segmentation has boundaries that do not correlate to semantic shifts in topic.
Doug Beeferman, Adam Berger, and John Lafferty. 1999. Statistical models for text segmentation. Ma- chine learning. 34(1):177-210.
Steven Bird, Ewan Klein, and Edward Loper. 2009, Nat- ural language processing with Python: analyzing text with the natural language toolkit. O'Reilly Media, Inc.".
It is also likely that for data sets that have seg- ments including many chunks with varying seman- tic meaning, such transcription that includes every small interjection, that this metric will lose some sensitivity and be less useful. The Wiki-50 data set is closest to this, with the inclusion of short section titles, which is why we believe we see worse performance with that data set. How- ever, due to the lack of data sets clearly represent- ing these specific cases, the exact impact of non- existant semantic shifts and semantically-scatters inclusions not quantified.
Tadeusz Caliñski and Jerzy Harabasz. 1974. A den- drite method for cluster analysis. Communications in Staristics-hherry and Methods, 3(1):1-27.
Harr Chen, SRK Branavan, Regina Barzilay, and David R Karger. 2009. Global models of document structure using latent permutations. Association for Computational Linguistics.
Freddy YY Choi. 2000. Advances domain indepen- dent linear text segmentation. In 1st Meeting of the North American Chapter of the Association for Com- putational Linguistics.
David L Davies and Donald W Bouldin. 1979. A cluster separation measure. IEEE transuctions on patlern analysis and machine intelligence, (2y:244-221
should also be noted that because this metric relies on computing distances based on semantic

---

Gaël Dias, Elsa Alves, and José Gabriel Pereira Lopes. 2007. Toopic segmentation algorithms for text sum- marization and passage retrieval: An exhaustive eval- uation. In AAAI, volume 7, pages 1334-1340.
Lev Pevzner and Marti Hearst. 2002. A critique and improvement of an evaluation metric for text segmentation. Computational Linguistics, 28(1): 19- 36.
Gerardo Ocampo Diaz and Jessica Ouyang. 2022. An alignment- -ased approach to text segmentation sim- ilarity scoring. In Proceedings of the 26th Confer- ence on Computational Natural Language Learning (CoNLL), pages 374-383.
Nils Reimers and Iryna Gurevych. 2019, Sentence-bert, Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNE)
pages 3982-3992.
Chris Fournier. 2013, Evaluating text segmentation us- ing boundary edit distance. In Proceedings of the 51 Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1702-1712.
Peter Rousseeuw. 1987. Silhouettes: graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53-65.
Chris Fournier and Diana Inkpen. 2012. Segmentation similarity and agreement. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 152-161.
Alessandro Solbiati, Kevin Heffernan, Georgios Damaskinos, Shivani Poddar, Shubham Modi, and Jacques Cali. 2021. Unsupervised topic segmentation of meetings with bert embeddings. arXiv preprint arXiv:2106.1223
W Nelson Francis and Henry Kucera. 1979. Brown corpus manual. Letters to the Editor, 5(2)77.
Jinxiong Xia, Cao Liu, Jiansong Chen, Yuchen Li, Fan Yang, Xunliang Cai, Guanglu Wan, and Houfeng Wang. 2022. Dialogue topic segmentation via paral- lel extraction network with neighbor smoothing. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, pages 2126 2131.
Iacopo Ghinassi. 2021.
Unsupervised text segmenta- tion via deep sentence encoders: a first step towards a common framework for text-based segmentation, summarization and indexing of media content. In ACM International Conference on Interactive Media Experiences (IMX 2021).
Yusen Zhang, Ansong Ni, Ziming Mao, Chen Henry Wu, Chenguang Zhu, Budhaditya Deb, Ahmed Awadal- lah, Dragomir Radev, and Rui Zhang. 2022. Summn: A multi-stage summarization framework for long in- put dialogues and documents: A multi-stage sum- marization framework for long input dialogues and documents. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), ), pages 1592-1604.
Alexios Gidiotis and Grigorios Tsoumakas. 2020. A divide-and and-conquer approach to the summarization of long documents. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:3029-3040.
Marti A Hearst. 1997. Text tiling: Segmenting text into multi-pen paragraph subtopic passages. Computational linguistics, 23(1):33-64 34.
Omri Koshorek, Adir Cohen, Noam Mor, Michael Rot- man, and Jonathan Berant, 2018 Text segmentation as a supervised learning task. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Compulational Linguistics: Human Language Technologies, Volume 2 (Short Pa- pers). pages 469-40 473.
Ming Zhong. Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2022. Dialoglm: Pre-trained model for long dialogue understanding and summa- rization. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 11765- 11773.
Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Quantifying social bi- ases in contextual word representations. In 1st ACL
A Appendix A: Human segmentation comparison with Stargazer Data Set
Workshop on Gender Bias for Natural Language Pro- cessing.
To provide an additional demonstration of the util- ity of the proposed metric, the Stargazer segmen- tation data set is evaluated. The Stargazer article (Baker, 1990) was selected as a challenging seg- mentation task for human annotators and (Hearst, 1997) collected seven different human segmenta- tion's with the intent of developing segmentation agreement metrics. Our proposed metric can be used to evaluate the topical segmentation of the dif- ferent individual metrics. To provide a reference, the best scoring annotation found with SegReFree
Daniel Marcu and Abdessamad Echihabi. 2002. An unsupervised approach to recognizing discourse rela- tions. In Proceedings of the 40th annual meeting of the association for computational linguistics. pages 368-375.
Pedro Mota, Maxine Eskenazi, and Luísa Coheur. 2019. Beamseg: A joint model for multi-document seg- mentation and topic identification. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 582-592.

---

Annotator
SegReFree
Pk relative to annot. 4
Score
Reforence Boundary
Hypotessi Boundary
0
0
3.15
0.42
2
3.82
0.47
3
2.32
0.26
4
2.23
0
Figure 8: Handow-based segmentation comparison
5
3.66
0.37
6
3.67
0.47
7
3.52
0.32
Boundary
Hypotheats Boundary
1
0
0
Table 5: Proposed metric computed for Stargazer seg-
mentation data set
Figure 9; Mass-based segmentation comparison
is chosen hypothesis and compared with each the other annotations using the Py metric. The two metrics correlate fairly well, with the highest Py score aligned with the two largest SegReFree scores found.
D Appendix D: Ablation study
To validate the design choices used in our proposed metric, small ablation study was performed using both data sets and both of the synthetic degraded segmentation experiments. As with previous exper- iments, random selection of boundaries was per- formed several times (five for each file the Choi 3-11 data set and 30 for each file in the Wiki-50 data set) and results were averaged. The two mod- ifications made to the original Davies-Bouldin In- dex are described in detail in Section 3.3 and are the inclusion of size penalty term well adjacency requirement for the computation of the metric. Table 6 shows the results of the removal of these individual alterations by computing the Pearson correlation coefficient between the Davies- Bouldin based metric and the Ps metric. The Ps. metric was chosen as a comparison due to it's posi- tion as the oldest and most common segmentation metric. It can be seen that the original Davies- Bouldin Index does not correlate with Pe as well our proposed metric does, and for the some experi- ments has negative correlation. The size penalty appears be helpful for improving performance the presence of segment boundary additions (simu- lated as segment splits), which causes smaller-than- desired segments. The adjacency appears to be most necessary for penalizing missed (simulated as deleted) segment boundaries, which conceptually makes sense missed boundary would cause different topical segments to become combined.
B Appendix B: Justification for data set selection
Our choice to only use two segmentation data sets is based similar numbers of real language data (or fewer) used in the literature that cite in this work. Fournier and Inkpen (2012) 2012 used one natural language data set for one subsection, Pevzner and Hearst (2002) did not include any nat- ural language data sets; Both Solbiati et al. (2021) and Ghinassi (2021) used two data sets for evalu- ation of their segmentation methods. Instead of wide diversity of data sets may be found in other research topics, focus presenting diverse set of experiments in Section
C Appendix C: Visualization of window and mass based segmentation
A simple visualization of window-based segmen- tation comparison is shown in Figure 8. Here, the start of a new segment represented by a /. The hindow-based metrics use a sliding window that compares segment boundaries between reference and hypothesis sets within the window. The same example segmentation set is presented in mass- based format in Figure 9, where the segments are represented as alternating shaded boxes. In a mass- based metric, the sizes of the segments in terms of chunks are used as the basis for comparison.
Multiple distance metrics were also tested, in- cluding L1, L2, and Cosine distance. Although L1 distance slightly outperformed L.2 distance a significant way on one experiment, L.2 distance was more used in Davies-Bouldin Index literature and was chosen for our proposed metric.

---

Ablation
Choi Split
Choi Del.
Wiki- 50
Wiki- 50 Del.
Split
Proposed
0.999
0.998
0.694
0.985
-Size
0.988
0.999
-0.994
0.994
Penalty
-Adjacen.
0.816
-0.844
0.940
0.960
Req.
Orig.
0.763
-0.844
-0.998
0.993
DBI
Dist. 0.999
0.998
0.732
0.986
Cos.
-0.871
-0.832
-0.806
0.987
Dist.
Table
Correlation Between Ablated Versions of Pro- posed Metric and Pk-
Original DBI includes neither adjacency requirement size penalty Proposed metric uses L2 distance.


Reviews:
[{"review_id": "HojEigtInH", "reviewer": "Reviewer_ZTcw", "summary": "", "questions": "1. Does the metric contain any bias for segmentation using SentenceTransformers, since it is used as part of the metric?\n2. Have you explored incorporating this metric to improve segmentation quality?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review.\n\nWe apologize for the missing reference links to figures, that will be fixed in the camera ready version. We will also update Figure 1, add periods after captions, and try combining some tables and figures.\n\nRegarding the reasons to reject:\n\n1.1 \"It is sometimes hard to understand what to make of the experiments.\"\n\nThe goal of the experiments is to demonstrate the strong correlation between our proposed metric and the existing metrics as they respond to realistic segmentation errors. This is articulated in Section 4.1 and 4.2. We can also add language to make this more explicit when referencing Figures 2-7.\n\n1.2 \"For example, what is 1-seg-sim and 1-bound-sim?\"\n\nIn Section 2.1, where existing metrics are introduced, we discuss how Pk and WindowDiff have a zero as an optimal score and Segmentation Similarity and Boundary Similarity have optimal scores of 1; because of this we plot 1 minus the Segmentation Similarity and Boundary Similarity to make the trends consistent. This is explicitly stated in the sub-caption of Tables 1-3, but we can add a sentence to the camera ready version to make this more clear when referencing the figures.\n\n1.3 \"What is the takeaway of all the figures? Should any one metric achieve low scores throughout?\"\n\nThe takeaway of the figures is that we show that our proposed segmentation metric (SegReFree) responds to segmentation errors in a way that is similar to existing segmentation metrics. Ideally, all segmentation metrics should perform similarly.\n\n2. The paper would benefit more from polishing the presentation of the paper.\n\nThank you for your input. We feel that after updating the missing reference links, adding clarifying language, and adding the reference suggested by Reviewer 3, we will have a nicely polished paper. \n\n\nRegarding your Questions For The Authors:\n\n1. Does the metric contain any bias for segmentation using SentenceTransformers, since it is used as part of the metric?\n\nIt is likely that there are very abstract influences of bias on the segmentation performance because of the use of SentenceTransformers. It is known that contextual word embeddings show different social biases [1], so it is likely that a sentence level embedding would learn bias from its training data. How this bias would affect segmentation is a harder question - figuring this out is a question probably spanning multiple papers. Thank you for asking this question, we will add some of this discussion to our Ethics Statement.\n\n\n2. Have you explored incorporating this metric to improve segmentation quality?\n\nIn internal experiments, we have used this metric to tune segmentation algorithms for use in a summarization task. Due to the use of the internal dataset, we have not published these experiments. We feel that the use of this algorithm for summarization would fit into another paper, as optimal segmentation for summarization is an area of open research, such as [2].\n\n[1] Kurita, Keita, et al. \"Quantifying social biases in contextual word representations.\" 1st ACL Workshop on Gender Bias for Natural Language Processing. 2019.\n\n[2] Zhang, Yusen, et al. \"SummN: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents.\" Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is clearly written and provides a comprehensive introduction and related work on the topic of segmentation.\n2. The authors provide strong results indicating the effectiveness of their evaluation metric which highly correlates with reference-based indices.", "reasons_to_reject": "1. It is sometimes hard to understand what to make of the experiments. For example, what is 1-seg-sim and 1-bound-sim? What is the takeaway of all the figures? Should any one metric achieve low scores throughout?\n2. The paper would benefit more from polishing the presentation of the paper.", "questions_to_authors": "1. Does the metric contain any bias for segmentation using SentenceTransformers, since it is used as part of the metric?\n2. Have you explored incorporating this metric to improve segmentation quality?", "paper_topic_and_main_contributions": "This work proposes a new reference-free segmentation metric that does not need to rely on reference or human-labeled segmentation to evaluate the quality by using sentence embedding and modifying a cluster validity metric. On two datasets, the authors show high correlations with classical reference-based segmentation metrics in a variety of settings.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. References to figure and appendices are incorrect (indicated with ??):\n- Line 103\n- Line 117\n- Line 138\n\n2. Figure 1 is a little bit blurry, try using pdf or higher resolution.\n3. I think there should be periods after each captions.\n4. There are too many figures and tables that are quite similar. For example, Tables 2 and 3 can be combined."}}, {"review_id": "PYZaAz1KH0", "reviewer": "Reviewer_Cxez", "summary": "", "questions": "1/ How Solbiati et al. (2021) and Ghinassi (2021) is different from what you proposed\n2/ Why don't you use your model to segment a text from scratch?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review.\n\nWe will fix the cross-references for the camera ready version, thank you for pointing them out.\n\nRegarding the Questions For The Authors:\n\n1. \"How Solbiati et al. (2021) and Ghinassi (2021) is different from what you proposed\"\n\nThis is explained on lines 243-257 - they use sentence embeddings to create a segmentation algorithm, we propose a segmentation metric agnostic of method. Our proposed method could be used for comparing human annotators, for example, which is not doable with a segmentation algorithm.\n\n2. Why don't you use your model to segment a text from scratch?\n\nOur goal is to create a segmentation metric, not a segmentation algorithm. We could utilize it with a greedy algorithm to segment a text, but that is not the design goal.\n\nRegarding the Reasons To Reject:\n\n1. The model is not compared with newer techniques using embeddings \n\nWe assume you are referring to Solbiati (2021) and Ghinassi (2021). Neither is a segmentation metric (both are segmentation algorithms) and therefore not comparable, like WindowDiff/Pk/etc..\n\n2. The authors do not write how Solbiati et al. (2021) and Ghinassi (2021) is different from what they proposed \n\nThis is explained on lines 243-257 - they use sentence embeddings to create a segmentation algorithm, we propose a segmentation metric agnostic of method. Neither is a segmentation metric (both are segmentation algorithms) and therefore not comparable, like WindowDiff/Pk/etc..\n\n3. A few issues in the formatting of the submitted version: missing figures and crossreferences\n\nThank you for pointing these out - they will be fixed in the camera ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1/ A simple model that measures a text segmentation quality\n2/ Convincing experiments showing that the index can reflect well the degradation of a segmentation   \n2/ An evaluation on two datasets and a comparison with relatively old baseline algorithms where the index proved better that other baseline techniques", "reasons_to_reject": "1/ The model is not compared with newer techniques using embeddings\n2/ The authors do not write how Solbiati et al. (2021) and Ghinassi (2021) is different from what they proposed\n3/ A few issues in the formatting of the submitted version: missing figures and crossreferences", "questions_to_authors": "1/ How Solbiati et al. (2021) and Ghinassi (2021) is different from what you proposed\n2/ Why don't you use your model to segment a text from scratch?", "paper_topic_and_main_contributions": "This paper describes a metric to assess the segmentation quality of a text. As input, the metric reads a topic-segmented text and as output, it returns a quality index. The lower the index, the better the quality. The index is unsupervised and uses the SBERT transformer to vectorize the sentences in a segment. The rationale is that that sentences in a segment should be close to each other and two sentences in two different segments should be away from each other. For each segment, the algorithm computes the centroid of the SBERT embeddings of the sentences in this segment. Then the algorithm computes a ratio of pairwise intracluster segments for adjacent segments. This method is a variant of the Davies-Boulding Index.\nThe authors assess their metric by degrading already segmented texts. The degradation was obtained by either removing boundaries or splitting segments. On the two evaluation datasets, the proposed metric revealed a consistent behavior reflecting the degradation and proved better than baseline techniques.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 1 is missing from the paper and some cross-references are replaced by ??."}, "extra_text": {}}, {"review_id": "ZEzPx2rYGq", "reviewer": "Reviewer_8kNW", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed metric shows strong correlation with existing segmentation metrics for most error types and can be used without a reference segmentation, which existing metrics require. The motivation behind the metric is clearly explained, its design is sensible, and it will be useful to the community, especially for researchers wanting to evaluate segmentation as an upstream task, who are unlikely to have reference segmentations available.", "reasons_to_reject": "The modifications made to the Davies-Bouldin Index are relatively minor and straightforward.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents a reference-free topic segmentation metric. Existing metrics require a reference and so cannot be used on data without gold segmentations. The proposed \"SegReFree\" metric is based on the Davies-Bouldin Index for cluster validity. \n\nAn embedding is computed for each sentence in the document using Sentence-BERT, and all embeddings are labeled with the segment (cluster) their sentence belongs to in the candidate segmentation. The authors make two modifications to the Davies-Bouldin Index to make it more suitable for sequence segmentation. First, they only compare segments that are adjacent in the document, since topic segmentation is only concerned with placing boundaries between adjacent segments. Second, the average intra-segment similarity score is penalized exponentially based on the number of sentences in the segment to prevent small segments with very few sentences from receiving significantly better scores than larger segments; the authors note that adjacent sentences can be expected to be more similar than sentences farther apart, even within a segment.\n\nThe authors perform experiments on two topic segmentation datasets, demonstrating strong correlations between SegReFree and existing metrics for boundary removal and segment splitting errors. The results on boundary transposition errors are more mixed. Prior work has noted that transpositions can be treated very differently among the existing metrics (see missing references), so these results seem reasonable.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Gerardo Ocampo Diaz and Jessica Ouyang. An alignment-based approach to text segmentation similarity scoring. CONLL 2022."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Section 2.1, Line 103: The visualization mentioned in this line is missing from the Appendix (reference link is also broken).\nSection 2.1.1, Line 117: Figure is missing, and reference link is broken.\nSection 2.1.2, Line 138: Same as above."}}]

Author rebuttal:

Meta-review: This paper proposes a new metric for summarization, which does not require manual annotations. The reviewers agree about the soundness of the submission but are split between "strong" and "ambivalent" on excitement. There is some debate about the relationship between reference-free segmentation *evaluation* and segmentation *algorithms*, since the former seems to imply the latter. Another excitement issue is the degree of innovation over the Davies-Bouldin segmentation algorithm, which is the foundation for the approach.
Decision: Accept-Findings