XLS-R fine-tuning on noisy word boundaries for unsupervised speech segmentation into words
Authors: Robin Jonathan Algayres
Pablo J. Diego Simon
Benoît Sagot
Emmanuel Dupoux
Venue: EMNLP 2023 Findings (2023)
Keywords: unsupervised speech segmentation into words
self-supervised Learning
self-training
TL;DR: Fine-tuning SSL models with DP-Parse boundaries

Abstract: Due to the absence of explicit word boundaries in the speech stream, the task of segmenting spoken sentences into word units without text supervision is particularly challenging. In this work, we leverage the most recent self-supervised speech models that have proved to quickly adapt to new tasks through fine-tuning, even in low resource conditions. Taking inspiration from semi-supervised learning, we fine-tune an XLS-R model to predict word boundaries themselves produced by top-tier speech segmentation systems: DPDP, VG-HuBERT and DP-Parse. Once XLS-R is fine-tuned, it is used to infer new word boundary labels that are used in turn for another fine-tuning step. Our method consistently improves the performance of each system and set a new state-of-the-art that is, on average 130% higher than the previous one as measured by the F1 score on correctly discovered word tokens on five corpora featuring different languages. Finally, our system can segment speech from languages unseen during fine-tuning in a zero-shot fashion.

XLS-R fine-tuning on noisy word boundaries for unsupervised speech segmentation into words
Robin Algayres 12 Pablo Diego-Simon's Benoit Sagot', Emmanuel Dupoux
ENS, INSERM, UPEC, PSL IInii, Paris, "Meta Al robinalgayreséinri..f
Abstract
in the last couple of years that speech segmenta- tion systems (Bhati al., 2021; Peng and Harwath, 2022; Kamper, 2023; Algayres et al., 2022b) have successfully done better than a uniform baseline. These progress have enabled the use of discovered spoken words as inputs to spoken language models to learn high-level semantic and syntactic represen- lations(Algan)) Algayres al., 2022b) well generate intelligible and meaningful spoken sentences (Al- gayres et al., 2023). The authors highlight that the discovery of boundaries that are aligned with real word boundaries strongly benefits downstream spoken language models.
Due to the absence of explicit word bound- aries in the speech stream, the task of segment- ing spoken sentences into word units without text supervision is particularly challenging. In this work, we leverage the recent self- supervised speech models that have proved to quickly adapt to new tasks through fine-tuning, even in low resource conditions. Taking inspi- ration from semi-supervised learning, we fine- tune an XLS-R model predict word bound- aries themselves produced by top-tier speech segmentation systems: DPDP, VG- H--BBERRT, GradSeg and DP-Parse. Once XLS-R is fine- tuned, it is used to infer new word boundary labels that are used turn for another fine- tuning step. Our method consistently improves the performance of each system and sets a new state-of of off-the-at art that is, on average 130% higher than the previous one measured by the F1 score on correctly discovered word tokens on five corpora featuring different languages. Fi- nally, our system can segment speech from lan- guages unseen during fine-tuning a zero-shot fashion!.
We present a speech segmentation method in- spired by a phoneme segmentation model Strgar and Harwath (2022) that leverages pseudo labelling and recent progress in self-supervised learning (SSL) speech models (Baevski et al., 2020; Hsu et al., 2021; den Oord et al., 2018; Chen et al., 2021; Babu al., 2021: Conneau al., 2020). SSL models are trained large speech datasets to predict masked parts of the input speech signal. Such pre-training methods yield speech represen- tations that can be quickly adapted through fine- tuning to a variety of downstream tasks ranging from ASR to speaker recognition, keyword spot- ting, intent classification and emotion recognition (Yang al., 2021). 1). We exploit the ability of SSL models to learn new tasks quickly and fine-tune pre-trained XLS-R model (Babu al., 2021) predict the word boundaries produced by an off- the-shelf unsupervised speech segmentation sys- tem. Our method is inspired by the semi-supervised learning literature (Xie et al., 2019; Yalniz et al., 2019; Scudder, 1965; Hinton et al., 2015; Grill et al., 2020; Chen and He, 2020), that have ex- plored how a model can bootstrap itself by provid- ing its own labels. We applied our method on the three wate-offthe-ane speech segmentation system: VG-HuBERT, DPDP and DP-Parse (Kamper, 2023; Peng and Harwath, 2022; Algayres et al., 2022b)
Introduction
In an attempt to model infant ability to segment speech into words, researchers have aimed at un- veiling word boundaries directly from the speech signal without prior knowledge of the language and, of course, without relying on textual annotations. Even though the notion of 'word' does not obey a set of strict rules, the goal is to bridge the exist- ing large performance gap between speech-based and text-based segmentation systems (Dunbar et al., 2.22) Indeed, after decade of work since the first publications (Jansen and Van Durme, 2011; 1; Lee and Glass, 2012; Lee et al., 2015), it is only
'Code is available ml. wtl.f//allayrrs//aawoodddmm
https:// gitlab.cognitiee
"Text segmentation into words is the task of finding word boundaries in a phonemicised text where spaces between words have been removed

---

and consistently improves their segmentation per- formances. Our method works particularly well with DP-Parse: on average, over five corpora fea- turing different languages, an XLS-R fine-tuned on DP-Parse boundaries increases by 130% the seg- mentation performances compared to the previous state-of-the-art, Finally, our method gets results above the ofte--ofthheart even in zero-shot set- ting where XI.S-R is not fine-tuned and sometimes not even pre-trained on the target language.
augmentation and loss selection) whereas Fuchs and Hoshen (2023) train single fully connected layer on top of a frozen Wav2vec2.0 (Baevski al., 2020) without iterative self-labelling. Also, our method is tested across different languages whereas their model only focuses on English speech.
1.2 Wav2vec2.0 and XLS-R
Speech Self-Supervised Learning (SSL) is a paradigm that enables to train deep neural net- works directly on the speech stream, typically by predicting masked parts of the input speech sig- nal. Wav2vec2.0 (Baevski al., 2020) is par- ticularly performant SSL model that is composed of a convolutional front-end and a stack of trans- former layers. Even though other SSL models have outperformed Wav2vec2.0 (Chen et al., 2021; Hsu et al., 2021; Chung et al., 2021) on down- stream tasks, Wav2vec2.0 has recently been trained in multilingual settings with XLSR53 ((Conneau et al., 2020), 53 languages), and XLS-R ((Babu et al.. 2021), 128 languages). These multilingual SSL models are excellent candidates for our work as we wish to perform speech segmentation into words in different languages. We carry out experi- ments with XLS-R, one of the latest multilingual Wav2vec2.0 model?. We provide in Appendix our experiments with other mono-lingual and multi- lingual Wav2vec2.0 models to analyse the effect of the amount of pre-training data.
1 Related works
1.1 Speech Segmentation
A particularly successful approach that has been applied to the problem of text segmentation is non-parametric Bayesian models (Goldwater et al., 2009; Johnson al., 2007). This approach has inspired two recent speech segmentation systems: DP-Parse and DPDP (Algayres et al., 2022b; Kam- per, 2023). These models segment a spoken sen- tence by first assigning every speech fragment probability to be word. Then, using dynamic programming beam search, one of the most prob- able segmentation for the whole spoken sentence is sampled. DPDP assigns probability scores us- ing the loss value of an RNN auto-encoder that has been trained to reconstruct random speech se- quences. DP-Parse computes probabilities by es- timating the frequency of speech fragments using density estimation on speech fragments encoded into Speech Sequence Embeddings (SSE). SSE models are trained with contrastive learning (AI- gayres et al., 2022a; Settle and Livescu, 2016) or auto-encoders (Kamper, 2018; Peng et al., 2020) to embed variable-size speech segments into fixed- size vectors. DP-Parse authors have shown that using better SSEs (that can be obtained with weak textual supervision) leads to higher segmentation performances.
2 Method
Let us use speech dataset C, pre-trained speech SSL model W, and an off-the-shelf speech segmen- tation system S. On top of W is added a random feed-forward layer with one neuron and a sigmoid activation. Here is our method to train W on bound- aries produced by S.
First, S is used to infer word boundaries for every spoken sentence in C. In addition, spoken sentences are data-augmented with random quan- tity of reverb, pitch, time-stretch and time-drop and then encoded into a series of frames by W. For each output frame, we create a label that is either 1 if the frame aligns with a word boundary or O if not. Because word boundaries are, by nature, not clearly defined in the time domain, we label as I the left and right neighbouring frames of ev- ery frame that has been already tagged 1. W is
A second type of model has reached the state- of-the-art in speech segmentation: VG-HuBERT (Peng and Harwath, 2022; Peng et al., 2023). This multimodal model fine-tunes the CLS tokens of a pre-trained HuBERT (Peng and Harwath, 2022) and a pre-trained ViT (Dosovitskiy et al., 2020) on aligned pairs of Engish utterances and images. Lastly, Fuchs and Hoshen (2023) also tackles speech segmentation into words using pseudo label- ing and SSL fine-tuning. Yet, our method finetunes the full XLS-R R model using various optimization methods (iterative labellan,,,,gg, Ir scheduler, data
'A Wav2Vec2.0 model pre-trained on almost 4000 lan- guages has recently heen released hut we did not have time to include it in our analysis (Pratap et al., 2023)

---

fine-tuned with back-propagation by minimizing the negative cross entropy between W's output and the labels. In a sentence, most of the frames are la- belled as zeros (for 'not a boundary'). ) and the loss is particularly low on those frames. To force the model to focus on harder sections of the input, we only backpropagate the loss on the top 50% frames with the highest loss.
performed grid-search on hyper-parameters on the development datasets maximise the token-Fl.
During our fine-tuning step, we use backpropa- gation on batches that contain 12 spoken sentences of a maximum of 20 seconds each. The XLS-R is fine-tuned a single 32Go GPU for maximum of 2000 updates, after which we keep the model with the lowest loss on the development set. Op- timization is done with Adam optimizer (Kingma and Ba, 2017), the learning rate warmed up from to 10 and then decayed back to with a co- sine annealing (Loshchilov and Hutter, 2016) with period 10 We freeze the convolutional front end and use 10% dropout, 15% layer-drop, and 15% masked frames. Data augmentation is done mainly with the WavAugment library. We use the values of parameters advised by the authors Kharitonov et al. (2020): for reverb, we sampled the room scale uniformly in [0,100] while keeping the other pa- rameters unchanged and for pitch, we pick a value uniformly in the range [-300,300]. Time-stretch co- efficients are uniformly sampled between 0.8 and 1.
At inference, for a given sentence, W produces at each frame the probability of discovering a boundary. To decide which frames should be la- belled as a boundary, we apply a peak-detection method that finds local maxima by comparison of neighbouring probability values. This function has two hyperparameters: maximal height of peaks and minimal distance between two peaks. We fit these hyperparameters on the development set by maximizing the Fl scores between the boundaries produced by S and the new boundaries produced by W.
At this stage, W be used infer the dataset C' set of new word boundary labels. W is set back to its initial unfine-tuned state and is fine-tuned again on the new word boundaries. This process is iterated until segmentation performances start to decrease.
4 Results
3 Datasets, Evaluation and Hyperparameters
4.1 F1 scores for different noisy boundaries
In Table I, we show the comparison of token-F1 scores for different speech segmentation systems and the token-Fl scores after iterative fine-tuning of XLS-R initialised by these systems. Instead of fine- tuning a different XLS-R model on each dataset, realised that we always get equal or higher per- formances if we fine-tune only one XLS-R on the boundaries of the five datasets at once*. We argue that there no overfitting possible as our method is completely unsupervised, and no true word bound- ary labels are used to train those models. In partic- ular, Germand and Wolof datasets have never been used to tweak hyper-parameee provide the main scores obtained when XLS-R only fine-tuned on each dataset separately. Even in this setting, our method produces an average to- ken Fl score that is twice higher than the previous state-of-thee
The metric that use evaluate performance is the token-Iil score which is the F'l score on correctly discovered tokens. token correctly discovered when both its boundaries correspond to the boundaries of a word in the time-aligned transcription. This metric was introduced by the ZeroSpeech Challenge 2017 (Dunbar et al., 2017) and is computed with the TDEv2 library. The datasets used in this work are the five corpora introduced in the ZeroSpeech Challenge 2017 (REF): Mandarin (2h), French (20h), English (30h), German (19h30min), Wolof (2h43min). The spoken sentences are split into voice activity detections (VAD) (i.e. sequences that only contain speech). The datasets are not split into train/test because they are meant for unsupervised learning. Yet. in order to fit syper-parameters, the five corpora are split into two sets: Mandarin, French and English are development datasets where hyper-parameters are tuned and German and Wolof are test datasets where the model is tested to show generalization to new languages. We
In Appendix 5,
The first two segmentation systems are baselines models. For the first, speech is segmented along the VAD timestamps. For the second one, in addition to the VAD timestamps, we added random boundaries
"w did not use language tags nor language-speciific heads.

---

Mandarin
French
English
German
Wolof
average
init
init
init
init
init
init
VADs
4.5
2.0
4.4
1.8
4.5
2.0
3.3
1.3
1.5
1.2
3.6
1.7
Random
12.1
3.1
7.7
2.0
8.1
2.5
6.7
1.8
11.4
2.5
9.2
2.4
VG-Hubert' (Puy22)
20.0
17.0
15.0
17.7
23.2
34,0
19.9
28.8
7.0
13.5
17.0
22.2
DPDP" (Kam22)
26.3
26.0
12.2
17.4
19.5
26.6
15.2
14.8
17.3
17.6
GradSeg! (Fush23)
12.2
27.5
18.2
31.3
19.4
30.2
12.5
21.4
11.8
24.9
14.8
27.1
DP-Parse" (Alg22)
16.0
32.0
15.3
41.8
21.9
42.5
13.4
49.5
17.5
37.8
16.8
40.7
weak-sap sup DP-Parse"
28.2
31.9
30.9
52.9
31.3
55.9
34.4
61.9
39.2
40.7
32.8
48.6
Gold
100.0
100.0
72.2
DP-Parse on text
50
n/a
68.1
n/a
78.5
n/a
67.4
D/a
69.1
I/a
66.6
n/a
Table I: Token-Fl obtained by different segmentation systems ('init' in the table) and after iterative fine-tuning XLS-R on datasets once ('ft' in the table). weak-sup DP-Parse is topline SSEs instead of unsupervised SSEs. Gold is supervised topline where XLS-R is fine-tuned with the true word boundaries. DP-Parse on text obtained by replacing the speech stream by text without spaces between words. (:22)x:Algaa Algayres al. (2022b)
uses weakly-supervised
t:Fuchs and Hoshen (2023), K:Kamper (2023)/Peng and Harwath (2022) x
average token-Fl
The last topline, DP-Parse on text gets 66.6% token-F1. The scores on text data show that our fine-tuning strategy has significantly narrowed the performance gap between speech and text. The scores Gold being higher than DP-Parse text show that our strategy has the potential to bridge the gap between speech and text segmentation com- pletely. To complete our analysis, we provide Appendix Table the Boundary-FI scores, which are the Fl scores on correctly discovered bound- aries instead of correctly discovered tokens.
full model
40.7
without loss selection without data augmentation without tuning the peak detection
38.5
37.7
37.4
Table 2: Ablation table: average token-Fi F1 score of seg- mentation over the five corpora. Each ablation compared to the row above itself.
that token durations have the same mean and standard deviation as the true word tokens. As each VAD give two true word boundaries, we thought that this information could be enough to kick-start our self-training method. The results show that it is not the case, our method leads to a drop in segmentation scores.
Overall, the results show great discrepancies be- tween the initial FI scores of model and the per- formances of fine-tuned XLS-R. We are not yet able to explain precisely why the fine-tuning of XLS-R on DP-Parse works better than on DPDP and VG-HuBERT. The main reason is certainly that we initially chose hyperparameters to maximise performances of XLSS R when it is fine-tuned on Parse. Then, we tried other hypee-rarameters to try to boost DPDP and VG-HuBERT scores (dif- ferent learning rates and data augmentations) but did manage improve over the scores reported in Table Figure 2 is visual presentation of the performances compared to the speech segmenta- tion systems submitted to the Zerospeech challenge (Dunbar et al., 2017) since 2017. The increase in performance obtained by fine-tuning XLS- R appear in deep blue.
Then, evaluate the three speech segmentation systems that were presented in the review On average, over the five datasets, our method consis- tently improves their FI scores with a clear advan- tage for DP-Parse which gets an average token FI score of 40.7.
Finally, we present topline models. For weak- sup DP-Parse, we segmented speech with the help of weakly-supervised SSEs instead unsuper- vised ones (as explained in Section 1). weaksup DP-Parse present higher performances than DP- Parse, which shows that our fine-tuning method could yield even better results, providing better SSE models. For Gold, we used the true word boundaries, and XLS-R degrades the segmentation performances on average from 100% 5
To analyse the importance of each of the tricks that we have used in our fine-tuning strategy, we provide Table the average token-Fl scores over the five datasets by successively ablating each trick. Overall, this table shows that the main gain
As the datascts do not have a validation sct, when we train XI.S-R to predict true word boundaries (and only in this case), we keep a held-out development set to compute the Fl scores.
*DP-Parse is applied on the transcriptions of the 7c- roSpeech datascts instead of the specch signal

---

DP-Parse text
120ms baseline
Mandarin
French
English
German
Wolof'
average
with XLS-R
original
DP-Parse
35.6
38.0
22.3
46.4
23.8
33.1
Table 3: Zeroshot performances: for each corpus, report the token-FI scores after finetuning XLSSRR to predict the boundaries of all other corpora except itself. Scores are presented for DP-Parse boundaries and true word boundaries (Gold).
60
token-F1
40
word segmentation performances by 130% com- pared to the previous statc-of-the-art. Our method also shows high performances in the zero-shot set- ting, which suggest that universal segmentation features exist in the speech Regarding inter- pretation, our results are sometimes hard to explain. Even though we proved that high initial Fl scores (obtained with weak supervision) do lead to better performances, more work is needed on the evalu- ation of the speech segmentation model to under- stand what characteristics are beneficial to XLS-R finetuning but that are not captured by Fl scores.
20
Garl7a
Ras20a
Bhazoa
Kam17
Kam22
Puy22
Fush23
Alo22
(w))2(ws)
Figure I: A general view of the performances of speech segmentation models so far. The figure shows the aver- age token-Fl FI scores (Mandarin, French, English, Wolof, German) obtained by different systems. In light blue are the original scores and in deep blue the increase in performances after XLS-R R finetuning. The baseline is segmentation every 120ms and the topline is DP-Parse applied text data. In green is the performance ob- tained by the weakly supervised DP-Parse model.
Limitations
Our method has only been tested on the Ze- roSpeech corpora, which come pre-segmented into Voice Activity Detection (VAD). These VADs have been obtained by first force-aligning audio and tran- scriptions and then by excluding all audio sections that were aligned to silences noise. If our model is used to segment other audio files, it is important to properly remove beforehand silent sections well any other non-speech sections (we advise using Pyannote (Bredin et al., 2019) or Brouhaha (Lavechin et al., 2023) if you cannot rely on force- alignment). Also, the audio from the ZeroSpeech corpora are studio recorded, which means the level of noise is extremely low. The performances of our model with noisier recording conditions would be much lower than those reported in this paper.
our method obtained by the simple finetuning XLSSR on noisy boundaries.
4.2
Zeroshot performances
We show in Table that our method can be used to segment languages that are unseen during the fine- tuning stage and also unseen during the pre-training stage of XLSS- (which is the case of Wolof). In turn, we selected four out of the five datasets for fine-tuning and used the remaining dataset for test- ing. We did this experiment using DP-Parse bound- aries and the true word boundaries. The results of the zero-shot DP-Parse are sometimes as high as when all datasets are included in the fine-tuning stage. This result echoes the intuition from Peng et al. (2023) that these models can learn universal segmentation features, which appear in their study to coincide with syllables.
Ethics Statement
Our model inherits from all the biases of audio models pre-trained on a large amount of data. In particular, languages and accentuations that were not present in the original pre- pre-training dataset may be less well encoded by XLS-R, which could result in impaired performances. The reader can refer to the list of pre-training languages Babu et al. (2021).
5 Conclusion
In this work, we propose an unsupervised speech segmentation system that fine-tunes XI.S-R on boundaries provided by an external off-the-shelf speech segmentation system. Our method increases

---

References
Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2020. An image
Robin Algayres, Yossi Adi, Tu Anh Nguyen, Jade Copet, Gabriel Synnaeve. Benoît Sagot, and Emmanuel Dupoux. 2023. Generative spoken language model based on continuous word-sized audio tokens.
is worth 16x 16 words: Transformers for image recognition at scale. CoRR, abs/2010.11929.
Ewan Dunbar, Xuan-Nga Cao. Juan Benjumea, Julien Karadayi, Mathieu Bernard, Laurent Besacier, Xavier Anguera, and Emmanuel Dupoux. 2017. The zero resource speech challenge 2017. CoRR. abs/17 12.04313.
Robin Algayres, Adel Nabli, Benoît Sagot, and Em- manuel Dupoux. 2022a. Speech sequence embed- dings using nearest neighbors contrastive learning.
Robin Algayres. Tristan Ricoul, Julien Karadayi, Hugo Laurençon, Salah Zaiem, Abdelrahman Mohamed, Benoît Sagot, and Emmanuel Dupoux. 2022b. DP- Parse: Finding Word Boundaries from Raw Speech with an Instance Lexicon. Transactions of the Associ- ation for Computational Linguistics. 10:1051-1065.
Ewan Dunbar, Nicolas Hamilakis, and Emmanuel Dupoux. 2022. Self-supervised language learning from audio: Lessons from the resource speech challenge. IEEE Journal of Selected Topics in Signal Processing, 16(6):1111-126.
Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf. Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. 2021. Xls-r: Self-supervised cross-lingual speech represen- tation learning at scale. arXiv, abs/2111.nn 109296.
Tzeviya Sylvia Fuchs and Yedid Hoshen. 2023. Unsu- pervised word segmentation using temporal gradient pseudo-labels.
Sharon Goldwater, Thomas Griffiths, and Mark Johnson. 2009. A bayesian framework for word segmentation: Exploring the effects of context. Cognition, 11::21 54.
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self superviedd learning of speech representations. CoRR, abs/2006. 1477.
man-BBstien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. 2020. Bootstrap your own latent: A new approach to self-supervised learning. CoRR, abs/2006.0777 07733.
Saurabhchand Bhati, Jesús Villalba, Piotr Zelasko, Lau- reano Moro- velo-Veláz, and Najim Dehak. 2021. Seg- mental contrastive predictive coding for unsupervised word segmentation. ArXiv, abs/2206.02210.0
Hervé Bredin, Ruiqing Yin, Juan Manuel Coria, Gre- gory Gelly, Pavel Korshunov, Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, and Marie-Philipee Phillppe Gill. 2019. pyannote. audio: neural building blocks for speaker diarization.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge neural network.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdel- rahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. CoRR, abs/2106.07447.
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, and Furu Wei. 2021. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. CoRR, abs/2110. 139000.
Aren Jansen and Benjamin Van Durme. 2011. Efficient spoken term discovery using randomized algorithms. In 201 IEEE Workshop on Automatic Speech Recog- nition Understanding, pages 401-406.
Xinlei Chen and Kaiming He. 2020. Exploring simple siamese representation learning. CoRR, abs/2011.10566.
Mark Johnson, Thomas L. Griffiths, and Sharon Gold- water. 2007. Adaptor grammars: A framework for specifying compositional nonparametric bayesian models. B. Schölkopf, C. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems 19, pages 641-648. MIT Press.
Yu-An Chung, Yu Zhang. Wei Han, Chung-Chie Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. 2021. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training CoRR, 0as/2108.06209
Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Emmrnuel Mazaré, Julien Karadayi. Vitaliy Liptchinsky. Ronan Col- lobert, Christian Fuegen, Tatiana Likhomanenko. Gabriel Synnaeve, Armand Joulin, Abdelrahman Mo- hamed, and Emmanuel Dupoux. 2019. Libri-light: A benchmark for ASR with limited or no supervision. CoRR,
Alexis Conneau, Alexei Baevski, Ronan Collobert, Ab- delrahman Mohamed, and Michael Auli. 2020. Un- supervised cross- -insull representation learning for speech recognition. CoRR, 139999
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai,

---

Herman Kamper. 2018. Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder models. CoRR, abs/181 100403.
H. Scudder. 1965. Probability of error of some adaptive rattern-recognition machines. JEEE Transactions on Information Theory, 11(3):633-31.
Herman Kamper. 2023. Word segmentation on discov- ered phone units with dynamic programming and self-supervised scoring. IEFF/ACM Transactions on Audio, Speech, and Language Processing, 31 31:684- 694.
Shane Settle and Karen Livescu. 2016. Discrimina- tive acoustic word embeddings: Recurrent neural network-based approaches. CoRR, abs/161 1.02550.
Luke Strgar and David Harwath. 2022. Phoneme seg- mentation using self self-supervised speech models.
Bugene Kharitonov, Morgane Rivière, Gabriel Syn- naeve, Lior Wolf, Pierre-Emmamud' Mazaré, Matthijs Douze, and Emmanuel Dupoux. 2020. Data augment- ing contrastive learning of speech representations in the time domain.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. CoRR, abs/1807.013 0877007748.
Qizhe Xie, Eduard H. Hovy. Minh-Thang Luong. and Quae V. Le. 2019. Self-training with noisy student improves imagenet classification. CoRR. abs/1911.04252
Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization.
Marvin Lavechin, Marianne Métais, Hadrien Titeux, Alodie Boissonnet, Jade Copet, Morgane Rivière, Filika Bergelson, Alejandrina Cristia, Emmanuel Dupoux, and Hervé Bredin. 2023. Brouhaha: multi- task training for voice activity detection, speech-to to- noise ratio, and c50 room acoustics estimation.
1. Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. 2019. Billion-scale semi- supervised learning for image classification. CoRR, abs/1905.000 0000000046.
Shuwen Yang. Po-Han Chi, Yung-Sung Chuang, Cheng- Jeff Lai. Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko tik Lee, Da- Rong Liu, Zili Huang, Shuyan Dong, Shang- Wen Li, Shinji Watanabe, AbdeIrahman Mohamed, and Hung yi Lee. 2021. SUPERB: Speech Processing Univer- sal PERformance Benchmark. In Proc. Interspeech 2021, pages 1194-1198.
Chia-ying Lee and James Glass. 2012. A nonparamet- ric Bayesian approach to acoustic model discovery. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics Volume
1: Long Papers), pages 40-49, Jeju Island, Korea. Association for Computational Linguistics.
Chia-ying Lee, Timothy J. O' Donnell, and James Glass. 2015. Unsupervised Lexicon Discovery from Acous- tic Input. Transactions of the Association for Compu- tational Linguistics, 3:389 403..
A Study on the types of pre-training
Ilya Loshchilov and Frank Hutter. 2016. stochastic gradient descent with restarts. CoRR,
SGDR:
Wav2vec2.0 comes in several mono-lingual and multi-lingual pre-trained versions. The monolin- gual Wav2vec2.0 are W2V2-LS trained on Lib- riSpeech (Panayotov et al., 2015) (960 hours of English speech), and W2V2-LV trained on Libri- Light (Kahn et al., 2019) (58k hours of English speech). The multilingual versions are XISSR33 trained on 56k hours of speech from 53 languages, with 82% of the total being English speech and XLS-R, trained on 436k hours from 128 languages, with 15% of the total being English speech. Thanks to those pre-training variations we will be able to study the impact of pre-training for the task of speech segmentation into words. All models have the same architecture (convolutions and 24 trans- former layers), number of parameters (300M), and training loss (NTXent), they only differ by their pre-training dataset.
tbs//108.03983,
Vassil Panayotov, Guoguo Chen, Daniel Povey, and San- jeev Khudanpur. 2015. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pages
Puyuan Peng and David Harwath. 2022. Word dis- covery in visually grounded, self-supervised speech
models.
Puyuan Peng. Herman Kamper, and Karen Livescu. 2020. A correspondence variational autoencoder for unsupervised acoustic word embeddings.
Puyuan Peng, Shang- Wen Li, Okko Räsänen, AbdeIrah- man Mohamed, and David Harwath. 2023. Syllable discovery and cross-lingual generalization in visu- ally grounded, self-supervised speech mode.
Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello. Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoory Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. 2023. Scal- ing speech technology to I 1,000+ languages. arXiv.
Table 6 presents our analysis of the amount of pre-training data required to reach the XI.S- high segmentation performances. We provide the Fl scores after fine-tuning either on DP-Parse or on

---

Mandarin
French
English
German
Wolof
average
init
ft
init
ft
init
ft
init
ft
init
ft
init
VADs
47.8
45.4
47.7
46.5
39.7
47.6
37.4
47.6
Random
54.3
48.1
47.0
41.0
46.3
42.2
43.3
39.5
52.9
49.4
48.9
44,0
VG-Hubert'
63.9
63.2
55.3
57.7
59.4
65.6
58.1
64.6
51.9
54.6
57.7
61.4
DPDP
68.3
70.2
53.5
58.6
$7.5
63.7
55.6
68.5
59.6
59.2
58.9
64.1
DP-Parse"
59.9
75.5
55.9
75.4
60.0
74.1
51.5
79.8
59.0
74.1
57.3
75.8
zero-shot DP-Purse
59.9
62,9
55.9
71,3
60.0
69.7
51.5
75.6
59.0
72,9
57.3
70.48
weak-sup DP-Purse"
69.9
75.4
70.0
77.6
68.4
80.8
64.4
83.2
75.5
79.4
69.6
79.7
Gold
100
84.4
100
93.1
100
95.2
100
82.3
89.0
DP-Parse (text)
76
n/a
84.3
n/a
89,8
n/a
83,5
n/a
84.1
n/a
83,5
n/a
Table 4: Boundary-F1 obtained by different segmentation systems ('init' in the table) and after iterative fine-tuning XLS-R on all datasets at once( 't' in the table). seak-sup DP-Parse is topline that uses weakly-supervised SSEs instead of unsupervised SSEs. Finally, Gold supervised topline where XLS-R R is fine-tuned with the true word boundaries. t:Kamper P2ng3) and Harwath (2022) (022)x:Algayrr) :Alaayres et al. (2022b)
Mandarin
French
English
German
Wolof
average
init
init
ft
init
ft
init
ft
init
ft
init
ft
To
VG-Huben
20.0
15.3
15.0
16.7
23.2
34.3
19.9
27.6
1.2
17.0
19.1
DPDP
26.3
30.2
12.2
16.1
19.5
27.0
15.2
28.9
14.8
19.1
17.6
24.2
16.0
30.4
15.3
28.8
21.9
47.3
13.4
38.2
17.5
30.2
16.8
35.1
Boundary-F1
VG-Huberr
63.9
64.1
55.3
59.4
58.1
63.2
51.9
35,9
55.9
DPDPf
68.3
73.1
53.5
57.3
57.5
63.8
55.6
68.3
59.6
59.9
58.9
62.3
DP-Parse
59.9
73.7
55.9
68.2
60.0
76.6
51.5
73.3
59.0
70.8
57.3
72.5
Table 5: Token-F1 and Boundary-Fl obtained by different segmentation systems ('init' in the table) and after iterative fine-tuning of XI.S-R on each dataset separately ('ft' in the table). {:Kamper (2023)/Peng and Harwath x et al. (20226)

---

true boundaries. For visualization of these results, Figure 2 shows the average token-Fl per model. As expected, pre-training Wav2vec2.0 strongly bene- ficial for learning word boundaries. Yet, the amount of speech data available for pre-training does not correlate well with segmentation performances. In spite of being trained on much fewer data than W2V2-LV and XLSR53, W2V2-LS reaches high segmentation performances. These Preliminary re- sults on other SSL models have not been included in this section for lack of time. In particular, Hu- BERT (Hsu et al., 2021) has slightly lower perfor- mances than Wav2vec2.0 models. Also, recent Wav2vec2.0 model came to our attention (Pratap et al., 2023), pretrained on nearly 4000 different languages but we did not have time to include this model in our work.
on DP-Parse
on Gold
DP-Parse
20
W2V2-V
pro-tsiinin))
(438k
Average token-F1 scores (Mandarin, French, English, Wolof, German) obtained by differ- Wav2vec2.0 models pre- pre-trained on different amount of speech and fine-tuned on either DP-Parse boundaries or Gold (i.e. true word boundaries). The average token- Fl score of DP-Parse on the five datasets is represented by a black line.
Figure 2:
Study on the type of input boundaries
and tokens per type ratios closer to the ratios ob- tained with true segmentation.
We are not yet able to explain precisely why the fine-tuning of XLS-R on DP-Parse works better than on DPDP and VG-HuBERT. As said in the main paper, the main reason is certainly that the optimisation hyperparameters have been tuned to maximise performances when XLS-R is being fine- tumed on DP-Parse. Yet, we think there could be another reason: the difference tokens per type ratios. This ratio is obtained by first transcribing the discovered speech tokens and then by dividing the number of discovered tokens by the number of different types of transcriptions. Indeed, as XI.S-R needs to (at least partially) memorise the differ- ent word types that it trained segment, XLS-R will more easily learn to segment a small number of types than a large number of types. In Table we show that DP-Parse has a higher tokens per type ra- tio (i.e. fewer types) than DPDP and VG-HuBERT. For that reason, we think that, compared to its com- petitors, DP-Parse provides better kind of input for XLS-R finetuning. This higher tokens per type could come from DP-Parse higher tokens per sec- ond, shown in Table 7. More work is needed know if XLSS- simply favours segmentation systems that tend to oversegment (and therefore have higher tokens per second and higher tokens per type).
For completeness, also provide in Table 7 the tokens per second and tokens per type after fine- tuning XI.S-R on unsupervised boundaries. As expected from the token-Fl and boundary Fl anal- ysis, fine-tuning XLSSR pushes token per second

---

Mandarin
French
English
German
Wolof
avg
W2V2 (no pre- pre-training)
pre-training hours
0
0
0
0
0
ft DP-Parse ft on Gold W2V2-LS
1.2
0.7
7.8
1.3
27.6
6.2
12.5
13.2
22.6
16.3
pre-training hours on DP-Parse ft on Gold W2V2-LV
0
0
960
0
21.8
24.3
44.2
18.1
24.2
26.5
57.8
52.0
85.5
63.67
51.2
62.0
pre-training hours ft on DP-Parse It Gold
57700
18.5
20.2
35.2
16.8
23.4
19.7
51.1
49.5
76.0
$7.2
52.3
57.2
XLSR53
pre-training hours ft on DP-Parse
32
1424
46009
1966
28.9
24.4
37.7
20.3
24.0
27.1
Gold
49.1
70.1
85.5
86.3
58.9
70.0
XLS-R
pre-training hours ft DP-Parse ft on Gold
90
23900
69500
25300
0
32.0
41.8
42.4
49.7
37.8
40.7
53.8
78.3
85.6
88.5
54.8
72.2
Table Token-Fl scores obtained by different Wav2vec2.0 models pre-trained on different amounts of speech and fine-tuned on either DP-Parse boundaries or Gold (i.e. true word boundaries)
Mandarin
French init
English
German
Wolof
average
init
init
init
Tokens per type
VG-HuBERT
2.08
2.43
4.40
7.38
4.03
6.56
4.05
6.62
1.51
1.08
3.21
4.81
DPDP
1.52
2.38
3.21
4.36
4.54
6.52
2.83
4.20
1.93
2.62
2.81
4.02
DP-Parse*
2.97
4.41
5.18
3.95
3.95
5.00
3.63
4.11
3.32
6.61
3.81
4.82
weak-sup DP-Parse
2.38
5.51
6.20
6.30
6.08
6.21
5.91
6.45
5.26
4.70
5.20
5.80
Gold
2.50
2.91
14.02
7.89
17.8
8.35
8.04
6.09
14.86
5.40
11.44
6.13
Token per seconds
VG-HuBERT
3.75
5.42
3.57
3.90
3.34
3.56
3.51
3.80
0.51
3.24
3.44
DPDPf
2.64
2.92
3.22
3.18
3,70
3.73
3.24
3.09
2.96
3.27
3.15
3.24
DP-Parse
4.44
4.03
3.74
2.68
3.23
2.97
3.47
2.91
3.88
4.39
3.75
3.40
weak-sup sup DP-Parse"
3.30
3.31
3.01
3.09
3.69
3.50
Gold
2.78
3.19
3.10
3.07
3.34
3.33
2.73
2.79
3.67
3.88
3.12
3.25
Table 7: Token per type and token per seconds obtained by different segmentation systems ('init' in the table) and after iterative fine-tuning XLS-R on datasets once( 'i'' in the table). weak-sup DP-Parse topline that SSEs instead of unsupervised SSEs. Finally, Gold is supervised topline where XLS R is fine-tuned with the true word boundaries. t:Kamper (2023)/Peng and Harwath (202))xAAgayyee et al. (2022b)


Reviews:
[{"review_id": "60X75uO91q", "reviewer": "Reviewer_oKq6", "summary": "", "questions": "A. The proposed method is simple but efficient, however, the generalization of the fine-tuning approach is not convincing, as shown in Table 1 that the performance of DPDP and Hubert based ones are worse than the baseline for Mandarin. The discrepancy between different SSL speech models should be investigated to improve the quality of this work. For instance, the authors could dive deep to compare the precision and recall, the effectiveness of data augmentation, etc.\n\nB. Is the method robust to the accuracy/performance of the initial pseud-label from an off-the-shell speech segmentation system?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer very much for their time and effort into reviewing our paper. Here are the responses to your questions and comments.\n\n1 - “The generalization of the fine-tuning approach is not convincing, as shown in Table 1 that the performance of DPDP and Hubert based ones are worse than the baseline for Mandarin. The discrepancy between different SSL speech models should be investigated to improve the quality of this work. For instance, the authors could dive deep to compare the precision and recall, the effectiveness of data augmentation, etc. Is the method robust to the accuracy/performance of the initial pseud-label from an off-the-shell speech segmentation system?”\n\nWe think there is a misunderstanding on the objective of the paper, which we will make clearer in the camera -ready version. In the unsupervised speech segmentation literature, people are trying to find any method to segment a given corpus without prior knowledge of the given language. The goal being to top the leaderboard on the token-F1 metric. We provide such a method by fine-tuning XLS-R on DP-Parse boundaries, and get very high token-F1 scores. Then, we show generalization of this method to 5 different corpora and even show that this method can generalize to languages unseen during pretraining and fine-tuning (see zero-shot dpparse in table1). Without being too presumptuous, the scores obtained in our paper are extremely high, and we honestly believe this work is going to be an important paper in the unsupervised speech segmentation literature. The reported gains have never been that high since people have started working on speech segmentation, a decade ago. \n\nThen, for completeness, we also provide the scores of our method applied to DPDP and VGHubert and indeed these latter do not show consistent improvements. Yet, we do not believe that this observation shows that our method does not generalize. For us this shows that the word boundaries produced by DPDP and VG-HuBERT are not good enough to kickstart the self-training procedure. Finding why DP-Parse boundaries help fine-tuning and not DPDP and VG-HuBERT  is the subject of a whole new paper that would compare precision, recall, number of implicit types, average length and duration of words, … this project is still ongoing research in our lab. We will add this clarification in the final paper.\n\n2 - “ In addition to the main approach, there are many extra tricks used during training and evaluation, such as post-processing, loss sample selection, and data augmentation steps. It would be clearer to add some analysis or ablation study to convince the readers and show the distinct contributions of these different steps.”\n\nEach one of these tricks provide a little bit of improvements. We agree that an ablation table would be appreciated, and we will add one in the appendix for the camera ready version.\n\nThank you very much for all your comments that will make this paper much clearer and much easier to read.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea of the proposed method is simple yet yields good results. By fine-tuning the added random feed-forward layer, the paper demonstrates that the pre-trained speech SSL system can be effective for the downstream task of speech segmentation. Additionally, the paper utilizes speech from different languages, which provides a valuable exploration into the understanding of SSL speech models on multilingual speech segmentation tasks.", "reasons_to_reject": "The proposed method does not convincingly demonstrate its generalization and novelty, mostly showing the effectiveness of fine-tuning a certain speech SSL model on speech segmentation tasks. In addition to the main approach, there are many extra tricks used during training and evaluation, such as post-processing, loss sample selection, and data augmentation steps. It would be clearer to add some analysis or ablation study to convince the readers and show the distinct contributions of these different steps.", "questions_to_authors": "A. The proposed method is simple but efficient, however, the generalization of the fine-tuning approach is not convincing, as shown in Table 1 that the performance of DPDP and Hubert based ones are worse than the baseline for Mandarin. The discrepancy between different SSL speech models should be investigated to improve the quality of this work. For instance, the authors could dive deep to compare the precision and recall, the effectiveness of data augmentation, etc.\n\nB. Is the method robust to the accuracy/performance of the initial pseud-label from an off-the-shell speech segmentation system?", "paper_topic_and_main_contributions": "The paper proposes an unsupervised speech segmentation system by fine-tuning the XLS-R speech representation model. Through an iterative training strategy, starting with an off-the-shelf speech segmentation system, the finetuned model shows improved results on the speech segmentation downstream task.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2d9hcEjXjY", "reviewer": "Reviewer_gXyg", "summary": "", "questions": "See RR.\nAlso, I am not convinced why through the described training method, we can get those improvements in token-F1. We did not get any further supervision signal, and there are no new mechanisms introduced. Can you explain why we get that improvement clearly?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer very much for their time and effort into reviewing our paper. We think there has been a misunderstanding on the objective of this paper. We will make some points clearer in the camera-ready version. Here are the responses to your questions and comments.\n\n1 - \"I wonder if this is unsupervised, as it still receives supervision signals from a trained system. Training that system requires supervised labels, and it is uncertain where we can get those labels if the experiments are unsupervised.\"\n\nOur model is indeed totally unsupervised, thank you for bringing this up, we will make it clearer in the camera-ready version. As a reminder, the model is a pretrained XLS-R that is fine-tuned on word boundaries produced by an off the shelf speech segmentation system. XLS-R is unsupervised because it is trained with by masking and predicting random part of the input speech signal. The word boundaries are unsupervised labels because they are produced by an unsupervised speech segmentation system (DP-Parse, DPDP, VG-HuBERT). The pipeline is therefore completely unsupervised: no labels obtained with textual supervision were used at any point of the whole process. (Except for the ‘Gold’ topline in Table 1 that serves to show maximal potential performances)\n\n2 - \"This paper does not compare to any force-alignment systems. The task, in my understanding, can be achieved reasonably easily using force alignment with a GMM-based system.\"\n\nIndeed, force-alignment system would make a very good supervised topline system. Yet, as stated in the introduction, our goal is to mimic infant language acquisition, who are known to learn speech segmentation during their first year of life without explicitly using prior linguistic knowledge. Therefore, we need to avoid using text at any point in the pipeline, which makes it not relevant to us to compare to force-alignment systems that leverage text data. \n\n3 - “Why is the system trained on Gold reference achieved 100.0 token-F1? So is that system already perfect? What on earth is the evaluation metric? How is it computed?”\n\nI think there is a misunderstanding on what the scores in table 1 are. We will make this clearer in the updated version.  As stated in the description of the table 1, the metric is the token-F1. This metric is indeed not explained enough in section 3. Token-F1 is computed for each corpus and each model. A given model is used to segment a complete corpus into a list of pseudo-word tokens. Then the token-F1 is obtained by comparing all segmented tokens with the word alignment reference. A pseudo-word token is correctly predicted if only both of its boundaries match with the reference.\nThen, in table 1, for each corpus, ‘init’ means the token-F1 score obtained by each off-the-shelf segmentation system, and ‘ft’ is the token-F1 score after fine-tuning XLS-R on the boundary produced by such speech segmentation system.\nFinally, it should be clearer now what the ‘Gold’ scores are. First, ‘init’ is 100% because it is the token-F1 of the true word segmentation, then ‘ft’ is the token-F1 obtained by fine-tuning XLS-R on the gold boundaries (which, as expected, decrease performances from 100% to 72% in average)\n\n4 - “Also, I am not convinced why through the described training method, we can get those improvements in token-F1. We did not get any further supervision signal, and there are no new mechanisms introduced. Can you explain why we get that improvement clearly?”\n\nThis method is common in the weakly-supervised and self-supervised literature. A model can bootstrap itself using a simple pipeline: first produce labels with a given model, then filter some of the labels out, and finally retrain the original model on the labels. This method is the base principle of models like  BYOL https://arxiv.org/abs/2006.07733 . Because of the constraint on the number of pages, we had to remove the related works on weakly supervised learning which, we agree, should be added in the appendix for the camera ready version.\n\n5- “Reproducibility: 3”\n\nWe have created a github that will be made public upon acceptance that ensures full reproducibility of our results.\n\nLastly, without being too presumptuous, we believe this work is going to be an important paper in the unsupervised speech segmentation literature as the reported gains have never been that high since people have started working on this problem, a decade ago. We are confident that it will attract the attention of the whole field of unsupervised speech segmentation.\n\nThank you very much for all your comments that will make this paper much clearer and much easier to read.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Exploring unsupervised ways to perform word segmentation in speech is an interesting direction", "reasons_to_reject": "1. I wonder if this is unsupervised, as it still receives supervision signals from a trained system. Training that system requires supervised labels, and it is uncertain where we can get those labels if the experiments are unsupervised. \n2. This paper does not compare to any force-alignment systems. The task, in my understanding, can be achieved reasonably easily using force alignment with a GMM-based system. \n3. Why is the system trained on Gold reference achieved 100.0 token-F1? So is that system already perfect? What on earth is the evaluation metric? How is it computed?", "questions_to_authors": "See RR.\nAlso, I am not convinced why through the described training method, we can get those improvements in token-F1. We did not get any further supervision signal, and there are no new mechanisms introduced. Can you explain why we get that improvement clearly?", "paper_topic_and_main_contributions": "This paper proposes to finetune the XLS-R model in an \"unsupervised\" way to get better performance on word boundary detection. It compares several systems using their methods.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QPRPCShLCt", "reviewer": "Reviewer_j6sr", "summary": "", "questions": "A. I'd like to see some ablation studies on how different tricks affect the performance (augmentation, smoothing, loss selection, peak detection)\nB. When using the time stretch augmentation, does the word boundary label also get stretched? \nC. How many iterations of self-training are needed before the model performance start to decrease?\n\n", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer very much for their time and effort into reviewing our paper. Here are the responses to your questions and comments.\n\n1- With regard to the impressive performance zero-shot DP-Parse in Table 1, why is zero-shot working so well? What kind of words are being predicted? Or is there a pattern?;\n\nThe zero-shot performance is a difficult to understand effect that was first pointed out by the authors of VG-HuBERT. Our interpretation is that by simply segmenting speech into syllables (which are more universal then words) the model can reach high segmentation scores. Yet this has not been proved yet and further research is needed.\n\n2- What leads to the discrepancy between using pseudo-labels from DPDP, VG-HuBERT, and DP-Parse\n\nWe have studied the difference in F1, precision, recall, number of types of discovered words, differences in word durations,... and still did not find anything that clearly explain the difference. After speaking about this with colleagues, we will certainly dedicate more time to this question in the future.\n\n3- I'd like to see some ablation studies on how different tricks affect the performance (augmentation, smoothing, loss selection, peak detection) B. When using the time stretch augmentation, does the word boundary label also get stretched? C. How many iterations of self-training are needed before the model performance start to decrease?\n\nWe will add an ablation table in the appendix for the camera ready version. Also, yes when applying time stretch we also stretch the labels to keep the match between speech and labels. And finally, the self-training takes usually 3 iterations before starting to degrade. We will include this in the final paper.\n\n4 - Citations,\nIndeed we forgot to cite [1] and [2] that are very related to our work. Thank you very much for pointing that out.\n \nThank you very much for all your comments that will make this paper much clearer and much easier to read.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. the fact that simple finetuning on pseudo labels via the general cross-entropy loss can bring such significant gain is very interestingly, which reveals interesting properties of the pretrained self-supervised speech models.\n\n2. this work examined using pseudo labels generated by a range of very different state-of-the-art unsupervised systems, namely DPDP, VG-HuBERT, and DP-Parse, and observed universal improvement. This indicate the robustness of the proposed approach.\n\n3. Training one model on all languages leads to better performance than training language specific models. Although this has been observed in large scale speech models (e.g. OpenAI's Whisper), it's the first time I observe this phenomenon in small scale studies (the total amount of data is below 80 hours.)\n\n4. a few tricks has being proposed to (potentially) make their approach works better, namely augmentation, smoothing, loss selection, peak detection. These tricks are not novel, but the usage is new and they make sense intuitively. These tricks could be valuable for researchers working on speech segmentation\n", "reasons_to_reject": "Although significant improvements are shown, more explanations are desired: \n\n1. with regard to the impressive performance zero-shot DP-Parse in Table 1, why is zero-shot working so well? What kind of words are being predicted? Or is there a pattern?; \n2. what leads to the discrepancy between using pseudo-labels from DPDP, VG-HuBERT, and DP-Parse\n\n", "questions_to_authors": "A. I'd like to see some ablation studies on how different tricks affect the performance (augmentation, smoothing, loss selection, peak detection)\nB. When using the time stretch augmentation, does the word boundary label also get stretched? \nC. How many iterations of self-training are needed before the model performance start to decrease?\n\n", "paper_topic_and_main_contributions": "this work finetuned a multilingual self-supervised speech model using labels generated from off-the-shelf unsupervised word-level speech segmentation systems, namely DPDP, VG-HuBERT, and DP-Parse, for improved word segmentation performance. \nSignificant performance gain is observed for all three types of pseudo-labels. In particular, using pseudo labels from DP-Parse, finetuning XLS-R brings an improvement of 130%. \n\nThis work advanced the state-of-the-art of unsupervised word-level speech segmentation by a large margin, which is important for bridging the gap between text-based and speech-based language systems. This work leads to a deeper understanding of multilingual self-supervised speech models\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 228:  32Go GPU -> 32GB GPU\nline 257: Germand -> German"}, "extra_text": {"Missing_References": "Citations to two relevant papers are missing: [1] and [2]. But it's worth mentioning that [2] is published within 3 months of this paper. \n\nBoth two papers and this work use pseudo-labels to finetune self-supervised speech models for speech segmentation.\n\n[1] focused on unsupervised phoneme segmentation, by finetuning w2v2 and hubert on pseudo labels generated by off-the-shelf unsup phoneme segmentation models.\n\n[2] showed that using the features of the SSL model itself to produce pseudo label can already bring significant improvement.\n\n[1] L. Strgar and D. Harwath, \"Phoneme Segmentation Using Self-Supervised Speech Models,\" 2022 IEEE Spoken Language Technology Workshop (SLT), Doha, Qatar, 2023, pp. 1067-1073, doi: 10.1109/SLT54892.2023.10022827.\n\n[2] T. S. Fuchs and Y. Hoshen, \"Unsupervised Word Segmentation Using Temporal Gradient Pseudo-Labels,\" ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece, 2023, pp. 1-5, doi: 10.1109/ICASSP49357.2023.10095363."}}]

Author rebuttal:

Meta-review: This paper presents a method for improving unsupervised speech segmentation by leveraging the large pre-trained self-supervised model XLS-R and off-the-shelf unsupervised word segmentation models to produce labels for self-training. The paper studies three existing approaches, namely DP-PARSE, VG-HUBERT and DPDP, and finds that pseudo labels obtained with DP-PARSE can be successfully used to fine-tune the XLS-R model for unsupervised word segmentation, resulting in an improvement of 130%. The authors report discrepancies between different off-the-shelf word segmentation methods and leave the analysis for future work.

The reviewers liked that this simple idea improves results by a large margin and that the paper evaluates the method in multilingual settings. The reviewers asked for several ablations experiments and clarification of some results. The authors ran the requested ablation experiments and tried to clarify the results, noting that some observations will probably need a more thorough analysis in future papers. After reading the paper and reviews, I base my decision on the review by the reviewer j6sr, who works on unsupervised word segmentation and is excited about this paper.
Decision: Accept-Findings