Generative Spoken Language Model based on continuous word-sized audio tokens
Authors: Robin Jonathan Algayres
Yossi Adi
Tu Anh Nguyen
Jade Copet
Gabriel Synnaeve
Benoît Sagot
Emmanuel Dupoux
Venue: EMNLP 2023 Main (2023)
Keywords: spoken language models
speech generation
zerospeech
textless nlp
TL;DR: A spoken language models that trained on word-size continuous acoustic embeddings

Abstract: In NLP, text language models based on words or subwords are known to outperform their character-based counterparts. Yet, in the speech community, the standard input of spoken LMs are 20ms or 40ms-long discrete units (shorter than a phoneme). Taking inspiration from word-based LM, we introduce a Generative Spoken Language Model (GSLM) based on word-size continuous-valued audio tokens that can generate diverse and expressive language output. This is obtained by replacing lookup table for lexical types with a Lexical Embedding function, the cross entropy loss by a contrastive loss, and multinomial sampling by k-NN sampling. The resulting model is the first generative language model based on word-size continuous tokens. Its performance is on par with discrete unit GSLMs regarding generation quality as measured by automatic metrics and subjective human judgements. Moreover, it is five times more memory efficient thanks to its large 200ms units. In addition, the embeddings before and after the Lexical Embedder are phonetically and semantically interpretable.

Generative Spoken Language Model based on continuous word-sized audio tokens
Robin Algayres!.', Yossi Adi Tu Anh Nguyen', .ade Copet Gabriel Synnaeve", Benoit Sagot', Emmanuel Dupoux' 1.3 'ENS, INRIA, INSERM, UPEC, PSL Research University
The Hebrew University of Jerusalem MMeta AI
(robinalgayres, bet.tesssbbbittttttsaeseee @inria. fr (adiyoss, edpx @meta...... com
Abstract
Kharitonov et al., 2021b). In addition, ASR may not be available in many languages that have in- sufficient textual resources and can make errors, which may then perturb the learning of the LM.
In NLP, text language models based on words subwords are known to outperform their character-based counterparts. Yet, in the speech community. the standard input of spoken LMs are 20ms 40ms-long discrete units (shorter than phoneme), Taking inspiration from word- based L.M, we introduce Generative Spoken Language Model (GSLM) based on word-size contimuous-valued audio embeddings that can generate diverse and expressive language out- put. This is obtained by replacing lookup ta- ble for lexical types with Lexical Embedding function, the cross entropy loss by a contrastive loss, and multinomial sampling by k-NN sam- pling. The resulting model the first genera- tive language model based word-size con- tinuous embeddings. Its performance is on par with discrete unit GSLMs regarding generation quality measured by automatic metrics and subjective human judgements. Moreover, it is five times more memory efficient thanks its large 200ms units. In addition, the embeddings before and after the Lexical Embedder are pho- netically and semantically interpretable.
The problem of using self-discovered units, how- ever, is that these units are typically very small, in fact, usually smaller than phonemes (Lakho- tia et al., 2021; Borsos et al., 2022). We think that increasing the size of the units will favourably impact the semantic capabilittiss of a downstream spoken LM. This intuition comes from the NLP literature. Among others, Graves (2013); Mikolov et al. (2011); Bojanowski et al. (2015); Nguyen et al. (2022a) have shown a performance gap be- tween character-based LM and word-based LM. The main reason is that at the level of characters, it is difficult for text LM to extract long-range syntactic and semantic relationships. This is one of the reasons why recent state-of-the-art text-based LM (Radford et al., 2019) typically use tokenizer representing word or subword units (Byte Pair En- coding (Gage, 1994), WordPiece (Wu et al., 2016), Unigram (Kudo, 2018)). Another advantage of large units is to save GPU memory at training time that enables to use both larger batches and longer sequences.
1 Introduction
Recent work has opened up the possibility of learn- ing generative language models directly from the raw audio signals, without using either text or Automatic Speech Recognition (ASR) (Lakhotia et al., 2021; Kharitonov et al., 2021b; Nguyen et al., 2022b; Borsos et al., 2022). The basic idea of these model is to rely on traditional text-based language models (LM),, but replace the text input with some other discrete tokens directly learned from audio in an unsupervised fashion. The advantage of learn- ing units from speech instead of relying on ASR is that this procedure can capture nen-verbal vocal- izations (like laughter) or intonation and rhythm, which are typically not transcribed, resulting in more expressive generations (Kreuk et al., 2021;
In speech, building the equivalent of a text-based tokenizer is hampered by two difficulties. First, the boundary problem that contrary to text in most orthographic systems, speech does not have spaces and punctuation to delimit between word units. Finding word boundaries from raw audio is itself a difficult challenge (Dunbar et al., 2022a). Second, the clustering problem, is that even if boundaries were available, the clustering of speech fragments is challenging because the same word may surface in variety of forms depending on speaker, accent, speech rate, etc. This problem may be even more difficult to solve than the first one (Dunbar et al., 2022a) because of the highly skewed distribution of word frequencies (Algayres et al., 2022b). Here,
Audio examples are available at our website.

---

we investigate the possibility of building continu- ous tokenizer that sidesteps these two problems by using tokens that have neither perfect boundaries nor require a clustering step. In Appendix B, we explain in more detail why we wish to avoid the clustering of speech fragments and what methods have been applied to tackle this problem so far.
outperform traditional features, like mel-filterbanks and MFCCs, in many applications (Yang et al., 2021). In parallel to these works, there is a grow- ing literature on variable-lengle acoustic encoding called speech sequence embeddings (SSE) (Peng et al., 2020; Algayres et al., 2022a; Jacobs et al., 2021; Kamper, 2018; Settle and Livescu, 2016). SSE models take a sequence of speech of any length and return fixed-size vector. These models encode speech by maximizing phonetic informa- tion while minimizing speaker identity and record- ing conditions. SSEs are used for spoken term discovery (Thual et al., 2018), speech segmenta- tion into phones or words (Kamper, 2022; Algayres et al., 2022b) but also as input to BERT model (Algayres et al., 2022b) for spoken language mod- elling.
Having a continuous tokenizer instead of dis- crete one results in drastic changes from the point of view of the downstream LM. With a discrete tokenizer, one can define a finite list of tokens over which the LM can learn a lookup embedding ta- ble at the input of the model and use softmax layer at the output of the model. The softmax is used in training mode to compute the loss func- tion through a cross-entropy with the target token and at inference time to sample sentences. With continuous representations. the list of tokens un- bounded, making these computations intractable. We tackle this problem with a Lexical Embedder, a semi- Meai-llernable function that maps continuous to- kens to a practically infinite list of embeddings.
Speech generation is often performed with neural vocoder conditioned mel-filterbanks (van den Oord et al., 2016; Kumar al., 2019; Kong et al., 2020; Prenger et al., 2018). In a text-to-speech pipeline, the mel-filterbanks are ob- tained with another neural network, which is condi- tioned on text (Ping et al., 2017: Shen et al., 2018), In the next step, the mel-filterbanks are decoded into natural-sounding speech by neural vocoder (van den Oord et al., 2016; Kumar et al., 2019; Kong al., 2020; Prenger et al., 2018). For the Zerospecch Challenge 2019, Dunbar et al. (2019) proposed to remove text and replace with un- supervised discrete units. This challenge has fu- eled a large body of works on learning low bi- trate speech representations for speech compres- sion, voice conversion and spoken language mod- elling (Chen and Hain, 2020; Liu al., 2019; Feng et al., 2019; Baevski al., 2019; Tjandra et al., 2019; Kharitonov al.. 2021b: Lakhotia al., 2021; Nguyen et al., 2020). For evaluation, the Resoure challenge used bitrate and human evaluation.
The key question addressed in this paper is whether it is possible to generate speech using large (word-size) continuous units instead of short discrete ones. Our major technical contribution is to replace the three standard elements of a text-based LM (lookup table, cross-entropy loss function, multinomial sampling) with elements adapted to virtually infinite list of continuous embeddings. We show that with these changes, is possible to generate speech of the same quality as discrete unit models. This is interesting because our units are 200ms long which amounts to a 5-time memory reduction compared to regular discrete units (Lakhotia et al., 2021; Borsos et al., 2022), opening up the possibility to train spoken LMs longer speech sequences. In addition, our model builds interpretable representations thanks the Lexical Embedder which learns a mapping between an acoustic space, with phonetic properties, to a lexical space, with semantic and syntactic properties. We call the resulting model (GSLM (token-based GSLM).
Spoken Language Model are neural networks trained to predict missing parts of a spoken sen- tence with predictive or contrastive losses. GSLM (Lakhotia et al., 2021) is the first spoken LM able to generate expressive and consistent spo- ken sentences in a pure textless fashion. It uses causal transformer LM trained with NLL loss on sequences of discrete units obtained with k- means clustering (with k= 1=00 of HuBERT frames. Once trained, GSLM can generate a sequence of discrete units by multinomial sampling that is de-
2 Related work
Unsupervised speech representations like CPC, Wav2vec2.0 and HuBERT (van den Oord et al., 2018; Baevski et al., 2020; Hsu et al., 2021) are fixed-size representations (10 to 20ms long) that

---

3
Approach
autput:
P
h
NCF loss
3.1 tGSLM: training
The general structure of (GSLM is presented Figure 1. It composed of an encoder which seg- ments the input speech into sequences of possibly varying size, and computes fixed-sized Speech Sequence Embedding (SSE), which call acous- tic tokens (Section 3.1.1). These tokens turned into lexical tokens through learnable Lexical Em- bedder (Section 3.2) and fed into a causal Lan- guage Model that has been modified to deal with continuous inputs (Section 3.1.3).
transformer
lexical
tokens:
Lex Emb
Lex
Lex
Lex Emb
Lex Emb
acoustic
Emb
Emil
Emiy
tokens:
segmentation
SSE
negative samples
input:
Figure Speech is encoded into Wav2vec2.0 frames and segmented into chunks. These latter are converted into acoustic tokens with an SSE model and turned into lexical tokens by applying the function LexEmb. Finally, lexical tokens are fed to causal transformer LM which attempts to predict the first, second, and third following tokens using parallel output heads. The acoustic tokens are pre-extracted before training the learnable modules Ene,, the transformer and final fully connected layers) with the NCE loss. The negative samples are chosen randomly from other utterances of the same speaker.
3.1.1 Acoustic tokens
In Figure I, speech sequence, S, is turned into n acoustic tokens, (oo, an), after applying speech segmentation and an SSE model.
Speech segmentation consists in finding word boundaries in speech sentence (Algayres et al., 2022b; Kamper, 2022; Kreuk et al., 2020). In this work, we rely on a naive method by placing a boundary every 200 ms, regardless of the content of the speech signal. In the Appendix A. we show that this method leads to better results than recent, more complex speech segmentation systems.
The acoustic tokens (a:)ign built by first en- coding the speech sentence S into series of n frames (fi)ign' with the 8th layer of Wav2vec2.0 Base from Baevski et al. (2020). For any two boundaries (k, a; = SS. fil) where SSE is a self-supervised system from Algayres et al. (2022a) trained with contrastive learning. This model has state-of-the-ar performances on phonetic representation of pre- meessegmented words as measured by the Mean-Average- Precision met- ric. The acoustic tokens are extracted in a prepro- cessing step and stored before the training of the subsequent LM.
coded into speech with a separate vocoder. Specif- ically, the sampled HuBERT units are mapped to mel-filterbanks with Tacotron2.0 and decoded into speech with WaveGlow (Prenger al., 2018), neural vocoder. Lakhotia al. (2021) also pro- vides a way to evaluate their spoken LM using an ASR to transcribe their spoken generations and an external LM to compute the perplexity of the re- sulting transcriptions. In addition, the Zerospeech Challenge 2021 (Nguyen et al., 2020) designed set of zero-shot metrics to probe what spoken LMs learn. A recent paper (Borsos et al., 2022), audi- oLM, came to our attention, which we did not have the time to include in our experiments. AudioLM works similarly to GSLM yet with the ability to generate speech that preserves the identity of the speaker. In another line of work, Algayres et al. (2022b) trained BERT model with a contrastive loss function on sentences represented as series of SSEs. They showed the resulting BERT is able model semanties and syntax. This work suggests that discrete tokenizers and the NLL loss are not necessary to tackle language modelling on speech. We take inspiration on their work to design our
3.1.2 Lexical tokens
In a text-based transformer LM, there is often an embedding lookup table before the transformer, that has the size of the vocabulary and that maps discrete word tokens to lexical tokens (Vaswani et al., 2017). These lexical tokens, also known word embeddings (Mikolov et al., 2013), Iearn dur- ing training semantic and syntactic properties that have been studied extensively in the NLP litera- ture. our case, the situation is different. First, instead of discrete word tokens, our LM takes as input continuous acoustic tokens which latent vo-
approach.

---

cabulary size is unknown. Second, the mapping between acoustic and lexical space cannot be lin- ear, as two speech segments may sound the same, i.e. be close in the acoustic space, while being semanticaly/syntatiiaall different, i.e. far in the lexical space. This highly non-lime non-linear function be- tween acoustic and lexical space is learned by our lexical embedder: LexEEb = Log q function. L is stack of non-linear fully connected layers learned jointly with the LM. q is an information bottleneck quantization function that we had to introduce to minimize the presence of low-level non- non-linguistic acoustic information. For a speech sequence S composed of n acoustic tokens (az)iga we note the sequence of lexical tokens (4))gn such as Vi S n, 4 Leb(a)(()).
(Vasuki and Vanathi, 2006) where several VQ code- books are learned in parallel as in Baevski et al. (2020); Zeghidour et al. (2021). The PCA and the d-k-means are trained over the whole training set as a preprocessing step, before the transformer LM. We ablate the use of in Appendix A.2 and show that it is necessary for the LM to generate sentences".
3.1.3 Causal language model
The LM is a standard causal transformer with two modifications: the loss function and the prediction heads. First, in standard LM, the number of possi- ble types is fixed beforehand and remains tractable even for a very large corpus (10k to 100k). Here, because the number of different lexical tokens is virtually infinite, we cannot use a standard softmax and cross-entropy loss. We first tried a simple L2 reconstruction loss with an additional decoder but it did not work for us in practice. Instead, we use contrastive loss: the Noice Contrastive Estima- tion (NCE) loss (Gutmann and Hyvärinen, 2010). This loss works by maximizing the similarity be- tween a pair of positive samples while minimizing the similarity between the positive samples and various negative samples. However, even though the SSE model from Algayres et al. (2022a) has learned to be speaker invariant, there is still lot of speaker-related information encoded into the acous- tic tokens. This is a problem already encountered in Algayres al. (2022a); van den Oord al. (2018) that dealt with by sampling the negative tokens from the same speaker as the positive tokens.
To understand why we need we have to go back to the LexEimb function input: the acous- tic tokens. The acoustic tokens are derived from Wav2vec2.0, which is a transformer architecture whose attention mechanism covers the whole sen- tence. Each wav2vec2 frame, therefore, con- tains potential information about relative positions (through the transformer's positional embeddings), adjacent acoustic materials (through self-attention) or global properties like speaker. What we've 've found in preliminary experiments is that this information may leak into the acoustic tokens and be ampli- fied by the prediction or contrastive loss of the downstream causal LM. Fortunately, it turns out that this information has low variance and can be partially removed by slightly degrading the qual- ity of the acoustic tokens. The degradation of the acoustic tokens is the role of the function q. is composed of a PCA reduction and a quantization step that we call d-k-means, which stands for per- dimension k-means. Specifically, given speech database that has been segmented and encoded into N acoustic tokens, (aN) we reduce their dimen- sions d with a PCA. Then, we train d different k-means, one for each dimension of the PCA. In other words, for each j I d, we train a k-means on (PCA(a))))EeN- We chose the number of centroids per k- k-means to be proportional to the ex- plained variance of each of the PCA dimensions. Once the k-means each acoustic token is mapped to its cluster id. Fi- nally, the cluster ids are turned into one-hot vectors and concatenated into one vector (see Appendix A.2 for more detailed explanations). d-k-mcans is inspired from multi-stage vector quantizer (VQ)
Second, in a standard LM, the output head typ- ically predicts the next word. However, in the case of speech, the boundary between individual phonemes is blurred by coarticulation. fore easy to predict the next word by just attending to very local acoustic information at the end of the last word (something impossible to do with char- acters which are sequentially disentangled). We, therefore, introduce three prediction heads (three linear fully connected layers: h1.h2,)b)) which do not only predict the first next token, but also the second and third as they cannot be co-articulated with the last token encoded by the LM. These pre- diction layers are trained jointly with the LM. We
there-
trained, each dimension of
"Due to this quantization step. the resulting vectors (PCA+ d-k-means) could in principle be mapped to a finite dictionary of tokens, but, in practice, there is little or no collision and the number of classes remains identical to the number of tokens, i.c., way too high to apply softmax.

---

neighbours in the lexical space. We apply softmax on the vector of cosine similarities and treat it as a multinomial distribution to sample one element: 1++1- The softmax function contains a temperature parameter that controls the range of the sampling area. The acoustic tokens 41+1 that correspond 4+1 is retrieved from the stored database and appended to (do.... ..... at). Once the desired length is reached, the sequence of acoustic tokens is decoded into spoken sentence as explained the next section.
speech generation
lexical space
Tacofron2
transformer
denoder to HuBERT units
Lcx
LCX
database
Emb
input:
Figure 2:
Our sampling procedure. Given list of au- dio files unseen during training, N random speech seg- ments are stored their acoustic and lexical forms: (01,4)g... In addition, lexical space is created by indexing (4)agN into k-NN graph. Given a speech prompt. segmented and encoded into (ao, .... at), we do forward pass in IGSLM and search for the near- est neighbors of h output in the lexical space. le+1 is sampled and its corresponding 04+1 is appended to (ao...., 02). When a final or token is sampled, (ao, ..., aT) decoded into HuBERT units and speech is generated with Tacotron2.
3.2.2 Speech generation
Lakhotia et al. (2021); Kharitonov et al. (2022) trained a Tacotron2.0 decoder (Shen et al., 2018) to map deduplicated HuBERT units into mel fil- terbanks. Then, speech is generated from the mel filterbanks by WaveGlow vocoder (Prenger et al., 2018). In order make use of this pre- trained Tacotron2.0 decoder, we trained an encoder- decoder transformer model to map series of acous- tic tokens to series of HuBERT umits. During train- ing, the encoder computes an attention over a series of acoustic tokens while the decoder predicts Hu- BERT units auto-regressively. At inference, given a series of acoustic tokens, corresponding sequence of HuBERT units is obtained by taking the argmax of the decoder softmax function. Finally, the Hu- BERT units are given as input the pre- trained Tacotron2.0 to be decoded into spoken utterances.
justify the choice of three prediction heads with a grid-scarch available in Appendix Table 5.
3.2 tGSLM: generation
Once (GSLM training is done, we use to generate spoken sentences. We do that in two steps: we generate sequence of acoustic tokens (Section 3.2.1) and then decode this sequence into speech (Section 3.2.2).
Evaluation and datasets
3.2.1 Sampling
4.1 Datasets and settings
To generate a spoken sentence, we take inspiration of the popular top-k sampling method used in NLP to generate text sentences. This method requires sampling series of word tokens by sampling among the most probable word types. In our case, do not have access to types so we are going to sample among the most probable lexical tokens. Our sampling method is summarized Figure 2. We start by collecting a few dozen hours of speech that have not been seen during tGSLM training. The utterances are segmented and encoded into N speech segments and stored in their acoustic and lexical forms: (a,,4) 4)vEN- Using the FAISS library (Johnson et al., 2017), we index (I))ss into a k-NN graph called the lexical space. Given a prompt of t acoustic tokens (dg..., 4t), we do a forward pass into 1GSLM. Then, we compute the cosine similarity of hn output and its closest
LJ Speech (LJ), LibriSpeech (LS), Libri-light 6k clean (LL6k-clean) are three corpora of studio recordings of read English of respectively 24, Ik and 6k hours (Ito and Johnson, 2017: Panayotov et al., 2015; Rivière and Dupoux, 2021). These corpora are used to train the different parts of the pipeline. The training details and specific model architectures can be found in Appendix Section A.3.
4.2 Generation metrics
Perplexity (PPX) is text-based metrics used by Lakhotia al. (2021) evaluate the overall qual- ity of generated spoken sentences. The authors propose to transcribe the spoken generations with an external ASR system and to compute the mean perplexity score over batches of transcribed speech

---

with an external transformer LM The spoken gen- eration process is guided by a temperature parame- ter that controls how diverse generated sentences are. The diversity of a batch of sentences can be computed as in Lakhotia et al. (2021) with the VERT score that is an average of self-BLEU (Zhu et al., 2018) and auto-BLEU (Lakhotia et al., 2021) scores. Typically, low temperatures produce high diversity and low perplexity, whereas high temper- atures produce low diversity and high perplexity.
the correct element in each pair. Probabilities are computed by applying the spoken LM training loss directly on the test items.
A X and ABXpoS are additional zero- shot tasks introduced in Algayres et al. (2022b) to evaluate the semantic encoding and Part-Of-Speece (POS) tagging. this time not based on probabilities but distances between embeddings. An ABX task is list of triplets A,B and X where A and B belong to the same category and X is distractor. The task is to encode the triplet with a distance d and show that d(AA, B) < ((.A.)). In this case, A,B, and X are spoken words given in the context of a sentence. For ABX A and B are close semantically, and X is random. For A ABXPOS A and B share the same POS tag, and X has different POS tags.
Finally, the perplexity of spoken generation is a metric that presents high variance, therefore, as a compromise between acceptable generation time and low variance, we compute perplexity over batches of 100 generated utterances whose tran- scriptions each exactly 30 words (around 10 seconds of audio).
Subjective judgements are computed with the meaningful Mean Opinion Scores (MMOS) in which human raters were asked to evaluate how natural (considering both grammar and meaning) a given spoken generation is. For both subjective tests, raters evaluate the samples on a scale of 1- 5 with an increment of 1. We follow the method from Lakhotia et al. (2021) where they evaluated 100 samples from each of the evaluated methods while enforcing at least 15 raters for each sam- ple. The CrowdMOS package (Ribeiro et al., 2011) was used with the recommended recipes for detect- ing and discarding inaccurate scores. As for the perplexity measures, the sentences are generated without conditioning on prompt.
Normalised Edit Distance (NED) introduced in Versteegh et al. (2016) term discovery task that consists finding clusters or pairs of speech seg- ments from unsegmented audio that have the same phonetic transcription. For each discovered pair, the NED is computed as the edit distance normal- ized by the length of the longest item in the pair. As for ABX tasks, the NED is also based on the distance between embeddings. To compute NED score, we take inspiration of the procedure intro- duced in Thual et al. (2018). Given a segmentation of the LibriSpeech dev-clean subset, all speech seg- ments are embedded into fixed-size vectors. With a k-NN, we search for the pairs of closest embed- dings and sort them by cosine similarity. Starting from the higher similarities, we retrieve as much pair as necessary to cover the whole dev-clean set. With the phoneme-level transcription of the dev- clean set, all pairs can be transcribed into series of phonemes. The final NED score obtained by averaging the NED over all pairs transcriptions. NED and ABX tasks both rely on embeddings that can be extracted at any level of a multi-layer neural model.
4.3 Zero-shot metrics
s SUGGY and sBLIMP are zero-shot tasks evaluate spoken language models introduced the Zerospeech Challenge 2021 (Nguyen et al., 2020):. These metrics are inspired by psycholin- guistics and are used for interpreting what spo- ken LM learns. sWUGGY is list of pairs of word/non-worts synthesized with the Google TTS API and filtered for the words that are in the Lib- riSpeech training set. sBLIMP P is list of pairs of syntactically correct/incorrec, synthesized sen- tences. Both sWUGGY and sBLI M P require the spoken LM to attribute a higher probability to
5 Results
5.1 Generation performances
5.1.1 Perplexity and diversity
Figure 3 provides comparison of the original dis- crete unit-based GSLM with two versions of our continuous unit model: 200ms-1GSLM, trained on speech segmented every 200ms and gold-IGSLM, trained on speech segmented on the true word boundaries. GSLM and 200ms-tGSLM are trained
*ASR transcripts are obtained with a pretrained large Wav2Vec 2.0 model, trained on LibriSpeech-960h combined with a standard KenLM 4-gram LM. The external LM used for perplexity is trained on the En- glish NewsCrawl dataset and accessible at https: //g hub. cot.cooo//aaaaaaa racebook resea meesearch/f reneeaichhharrreee haa//riiregg tree/mainn m /eciin/exxmpppppl / Language model wttp:/leeeeaaggggaaeeaegaaeaeeaa

---

Zero-shol metricst
Generation PPX.
Generation MMOS?
sWUGGY
SBLIMP
ABX
ABXPos
LS-VERT
LJ-VERT
LS-VERT
LJ-VERT
GSLM
70.36
56.31
55.85
59.03
503.254-12.3
387.45+-11.2
3.76 + 0.035
3.78 +- 0.023
200ms-GGLM
68.53
55.31
55.89
60.3
532.87+-10.1
356.24e-15.7
4,09 0.016
4.04 0,020
gold-IGSLM
86,37
-f
65.6
73.59
361
255.324-14.2"
n/a
n/a
claracter-gold
n/a
n/a
180.2
J42.6
4.12 +- 0.016 4.11 0.023
Table 1: Results on zero-shots and generation tasks 200ms-1GSLM and GSLM, trained on LL clean, and gold-IGSI.M, trained on LibriSpeech. ABX is computed on M lexical tokens and on GSLM 9th layer. The last line is a topline that is composed of true sentences from both LibriSpeech and LJ those scores are obtained without using a speech decoder. time-aligned word boundaries for sBLIMP are available
Regarding the perplexity scores from Table 1, compared to GSLM, 200ms-1GSLM is slightly bet- ter at LJ-VERT and slightly worse at LS-VERT. The measure of perplexities being very noisy, these scores show that both models have similar perfor- mances. Some examples of transcribed spoken generations are available Appendix Tables 8,9 and 10.
700
-0SLM
go0 1GSLH
020
15 PEX/EER LYPI/VEHT
500
300
The topline gold-IGSLM produces much lower perplexities than GSLM and 200ms-1GSLM. Yet. have experienced problem with the speech de- coder (described in Section 3.2.2) of gold-IGSLM. The scores of our topline are obtained by retriev- ing the exact transcriptions of the sampled SSES instead of decoding them with the speech decoder. We had to do this because our speech decoder makes a lot of decoding mistakes when it tries to decode SSEs of variable-size speech fragments. It seems to generate fully intelligible speech only when it is trained decode SSEs of same-size speech chunks, is the case for 200ms-1GSLM. We think this happened because, for lack of time and resources, we chose a poor decoding strategy (decoder from SSEs to HuBERT frames and Hu- BERT frames to speech). In our future works, we will focus on training model to decode the SSEs directly into speech, using, for instance, recent dif- fusion models Hi-Fi Gan (Polyak et al., 2021; Huang al., 2022). As consequence of the poor performances of our speech decoder, we have not been able to leverage recent progress in speech segmentation into words (Algayres et al., 2022b; Kamper, 2022; Peng and Harwath, 2023) that pro- vide word boundaries more aligned with real words than our 200ms chunks. In Appendix A. 1.1 are the re- sults of our attempts using speech segmentation systems.
PPX and VERT scores for GSLM, 200ms- (GSLM and gold-IGSIM. Each dot obtained by gener- ating sentences with fixed temperature parameter. The curves are 3rd-degree polynomial interpolations of the dots. The green dashed lines are the oracle PPX/VERT obtained on the LibriSpeech and LJ corpus.
Figure 3:
LL6k-clean' while the topline, gold-tGSLM, is trained only on LibriSpeech corpus.. The dots in Figure represent batches of generated sen- tences conditioned on different temperatures. Color curves are the 3rd-degre Brd-degree polynomial interpola- tion of the dots. In green dashed lines appear two VERT anchor points LJ-VERT(=0.113) and LS- VERT(=0.189), These points are the mean VERT scores obtained on batches of sentences from, re- spectively IJ and LibriSpeech datasets. The inter- section of the dashed lines and the curves gives the scores PPX @ VERT and PPX (PX@LJ-VERT that are reported in Table 16.
'Training (OSMs on Libri-light 60k (Kahn al., 2019), larger but noisier corpus, slightly undermined the performance.
'word boundaries cannot computed LL6k-clean because sentence-level speech text alignments are missing "For a given spoken LM, its PPX @LS-VERT score the perplexity score obtained hy that spoken LM when condi- tioned on a temperature that makes it generate spoken sen-
tences with a VERT equal to the VERT of the LibriSpecch.

---

In order analyze what is learned by Lex Emb we measure the ABX and NED of lexical tokens and acoustic tokens. In Table 2, the ABX scores show that the acoustic tokens are at chance level on se- 5.1.2 Subjective judgements
models
tokens
NED
ABX..m T
A83xps
20LM
34.51
50.14
49.87
As for perplexity, we report in Table I, the MMOS for batches of spoken generations that have a diver- sity score equal to the VERT of cither LibriSpeech (MMOS @ LS- VEERR or LJ (MMOS @ LJ-VERT). In addition to 200ms-1GSLM and GSLM we evaluate topline called character-gold that speech utterances obtained with Text-To-Speech (Tacotron2.0 from Shen al. (2017)) taking in input the transcriptions of LJ and LibriSpeech utterances. From Table I, for the high-temperature regime that leads to diversity scores in the range of LJ and Librispeech, 200ms-IGSLM is slightly better than GSLM and gets close scores with the topline. MMOS scores not available for gold-IGSLM has the speech decoder did not work properly. Nonetheless, our table of results does not show the performances tGSLM much lower temperature regime. When conditioned on very low temperature, GSLM can generate very simple and intelligible sentences, whereas 200ms-1GSLM start to produce gibberish. Therefore, both models have their strengths and weaknesses.
lexical
47.98
55.08
60.24
guld-IGSLM
50.20
50.12
lexical
22.70
65.60
75.59
Table 2: NED and ABX scores on acoustic and lex- ical tokens for 200ms-tGSLM and gold-IGSLM both trained on LibriSpeech. ABX and NED are computed on tGSLM lexical tokens
5.4 Memory consumption
GSLM model (Lakhotia al., 2021) and 200ms- tGSLM use the same transformer LM but with dif- ferent types of inputs. Compared to the 200ms-long units of our model, GSLM trained on discrete units that are 40ms long on average (when con- tiguous duplicates are removed). Therefore, we expected our model to be more memory efficient than GSLM² which can be observed by the maxi- mal batch size that both models can handle. Indeed, on the one hand, we managed to train GSLM with 34 60-seconds-long sentences on a 32G V100 GPU without OOM error. On the other hand, 200ms- tGSLM can fit as many as 162 sentences, which shows almost a 5-time reduction (#2 4.76) of mem- ory use.
5.2 Zero-shot performances
Training spoken LMs on long sequences of audio will become necessary in order to learn long-term semantic relations. The usage of very short acoustic units can become bottleneck which our method helps to alleviate. To complete our analysis, we provide in Appendix A.5 a theoretical analysis memory reduction.
To complete our analysis, we provide Table I, performances on the zero-shot tasks scores that are comparable for GSLM and 200ms-1GSLM. GSLM has a little advantage on sWUGGY and sBIIMP and 200ms-tGSLM slight advan- tage on ABX and PBX The topline gold- (GSLM, once again gets much stronger results. ABX scores are obtained, for GSLM at the 9th layer of the transformer and for tGSLM with the lexical tokens.
Conclusion
We introduced a generative spoken LM based on continuous word-sized acoustic tokens. Our model able to generate speech with the same level diversity and accuracy as a model based on discrete units. This shows that building a lexicon of types is not necessary for spoken language modelling,
5.3 Interpretability
which is encouraging considering the difficulty of clustering large segments of speech without de- grading the representation (see Appendix B). In addition, this performance was obtained with seg- ments that were not very well aligned with word boundaries (200ms segments). The good result obtained with gold word boundaries indicates that there is room for improvement by using segments
mantic and syntactic encoding. After the Lex Emb function, the lexical tokens lose a bit of their pho- netic encoding (NED increases) but gain the ability to represent semantics and syntax. However, the NED is chance level, meaning that bit of acoustic information has leaked into the lexical tokens. To visualize the difference between acous- tic and lexical spaces, we provide t-SNE maps in Appendix Section A.4.
"The acoustic tokens that are the input of 200ms-tGSLM are extracted in a preprocessing step. They do not impact memory usage at training time.

---

better aligned with word boundaries and of course a better speech decoder. Further work is also needed to better limit the leakage of low-level acoustic in- formation into the LM through continuous units, which our analysis has shown is detrimental to the performance of the generative model (see also Nguyen et al. (2022c)). Finally, the fact that the units are about 5 times larger than standard GSLM units aligns with the NLP literature that is in favour of word- bard-bbaed LMs. opens the possibility fit larger spans of audio in GPUs and capture long- distance relationships.
synthesizers. It is of paramount importance to safe- guard against these issues.
First, generative text-based LMs are known to repeat stereotypes and biases that belong to the training corpus which can cause a series of harms Chowdhery et al. (2022); Bender et al. (2021). One way to mitigate this is to apply post-processing on generated sentences to detect harmful content. Yet. from what we have heard, tGSLM still struggles to generate sentences that fully make sense, so we do not think that procpssoes is required at the moment.
Second, if (GSLM is used to continue speech prompt, the continuation might be inconsistent for accents of underrepresented groups in the training data. Indeed, speech systems are known encode poorly accents and dialects out of the training dis- tribution (Riviere et al., 2021).
7 Limitations
Our method has some limitations that range from GPU consumption, potential overfitting on the En- glish language and sub-optimal decoding method. First, (GSLM is trained on 32 Nvidia V100-32Go GPUs for 30 hours. Due to the several modules work in (GSLM (SSE model, LexEmb func- tion, transformer decoder and seq2seq decoder), a large grid-search on hyper-parameters has been necessary which makes this work quite resource- consuming. Secondly, during the grid-search we chose hyper-parameters to optimize the semantic and syntactic ABX scores on English. By doing so, might have overfitted the English language and made tGSLM specifieally good generat- ing English specch. Further analysis is required to see if our method generalizes well to syntacti- cally and morphologically different languages, like French or Mandarin. Finally, our decoding method is based on seq2seq transformer that produces HuBERT frames which are decoded into speech with combination of Tacotron2.0 and WaveGlow. We chose that method as this later speech synthe- siser comes pre-trained in the lextlesslib Python library (Kharitonov et al., 2022). Yet, recent work on textless speech synthesis Kreuk et al. (2021); Kharitonov et al. (2021a) skip the spectrogram pre- diction of Tacotron2.0 and directly train Hifi- Gan model to generate speech from HuBERT units. This latter model shows close to human-level per- formances. We leave the use of Hifi-Gan instead of Tacotron2.0 for future works tGSLM.
Finally, tGSLM continuations will not preserve any regional accentuation from the prompt, as our model only generates speech in the voice of the single speaker of the LJ dataset.
A Acknowledgements
This work was funded in part, to the authors in their academic capacities, by the Agence Na- tionale pour la Recherche ANR-I7-EURE-0011 Frontcog, ANR-10-IDEX 10-IDDEX0-000--02 PSL*, ANR- 19-P3IA-0001 PRAIRIE 31A Institute), CIFAR (Learning in Machines and Brains) and Meta AI Research (Research Grant). This work was per- formed using HPC resources from GENCI-IDRIS (Grant 2021-[AD01 1011217]]
References
Robin Algayres, Adel Nabli, Benoit Sagot, and Em- manuel Dupoux. 2022a. Speech sequence embed- dings using nearest neighbors contrastive learning.
Robin Algayres. Tristan Ricoul, Julien Karadayi, Hugo Laurençon, Salah Zaiem, Abdelrahman Mohamed, Benoît Sagot, and Emmanuel Dupoux. 2022b. Dp- parse: Finding word boundaries from raw speech with an instance lexicon.
Alexei Baevski, Steffen Schneider, and Michael Auli. 2019. wq-wav2eec: sulf-super learning of dis- crete speech representations. CoRR, abs/1910.05453.
8 Ethical statement
(GSLM is LM that learns to generate speech sen- tences by predicting its training data. Therefore, (GSLM inherits from ethical concerns associated with text-based LM, speech encoders and speech
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. CoRR, abs/2006. 1477.

---

Emily M. Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language mod- els be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountabiliyy and Trans- parency, FAccT "21, page 610-623, New York, NY, USA. Association for Computing Machinery.
Ewan Dunbar, Robin Algayres, Julien Karadayi, Mathieu Bernard, Juan Benjumea, Xuan-Nga Cao. Lucie Miskic, Charlotte Dugrain, Lucas Ondel, Alan W. Black, Laurent Besacier, Sakriani Sakti, and Emmanuel Dupoux. 2019. The zero resource speech challenge 2019: TTS without T. CoRR, abs/1904, 1469.
Chris Biemann. 2006. Chinese whispers an efficient graph clustering algorithm and its application to nat- ural language processing problems. In Proceedings of Tex:Gruph:: the First Workshop on Graph Based Methods for Natural Language Processing pages 73- 80. New York City. Association for Computational Linguistics.
Ewan Dunbar, Nicolas Hamilakis, and Emmanuel Dupoux. 2022a. Self-supervised language learning from raw audio: Lessons from the zero resource speech challenge. IEEE Journal of Selected Topics in Signal Processing.
Ewan Dunbar, Nicolas Hamilakis, and Emmanuel Dupoux. 2022b. Self-supervised language learning from raw audio: Lessons from the zero resource speech challenge. IEEE Journal of Selected Topics in Signal Processing, 1616):1211-1288
Piotr Bojanowski, Armand Joulin, and Tomás Mikolov. 2015. Alternative structures for character-leve mns. CoRR, abs/1511 1.06003.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eu- gene Kharitonov, Olivier Pietquin, Matt Shariti, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. 2022. Audiolm: a language mod- eling approach to audio generation. arXiv preprint arXiv:2209.031433
Ali Fllkakky, Wei-Ning Hsu, Paden Tomasello, Tu-Anh Nguyen, Robin Algayres, Yossi Adi, Jade Copet. Emmanuel Dupoux, and Abdelrahman Mohamed. 2023. Do coarser units benefit cluster prediction- based speech pre-training? In ICASSP 2023- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5.
Peter F. Brown, Vincent J. Della Pictra, Peter V. deS- ouza, Jenifer C. Lai, and Robert L. Mercer. 1992. Class-based n-gram models of natural language. Computational Linguisties, 18(4):467-480.
Siyuan Feng, Tan Lee, and Zhiyuan Peng. 2019. Com- biming Adversarial Training and Disentangled Speech Representation for Robust Zero Reso-Res Subword Modeling. In Proc. Interspeech 2019, pages 1093- 1097.
Mingjie Chen and Thomas Hain. 2020. Unsupervised Acoustic Unit Representation Learning for Voice Conversion Using WaveNet Auto-Enn Auto-Encoders. In Proc. Interspeech 2020, pages 4866-4870.
Philip Gage. 1994, new algorithm for data compres- sion. The C Users Journal archive, 12:23-38.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko. Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope. James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Helltern,,,,,rn, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways.
Sharon Goldwater, Thomas Griffiths, and Mark Johnson. 2009. A bayesian framework for word segmentation: Exploring the effects of context. Cognition, 112:21- 54.
Alex Graves. 2013. Generating sequences with recur- rent neural networks. CoRR, abs/ 308.0850.
Michael Gutmann and Aapo Hyvärinen. 2010. Noise- contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artifi- cial Intelligence and Statistics, volume 9 of Proceed- ings of Machine Learning Research, pages 297-304, Chia Laguna Resort, Sardinia, Italy. PMLR.
Wei-Ning Hsu, Benjamin Bolte. Yao Hon- Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdel- rahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. CORR,
Rongjie Huang, Max W. Y. Lam, Jun Wang. Dan Su, Dong Yu, Yi Ren, and Zhou 7hao. 2022. Fastdiff: A fast conditional diffusion model for high-quality speech synthesis.
Yu-An Chung and James R. Glass. 2018. Speech2vec: A sequence-to-sequeeee ing word embeddings from speech. CoRR, obs/1803..897..
framework for learn-
Keith Ito and Linda Johnson. 2017. The Ij speech dataset.

---

Christiaan Jacobs, Yevgen Matusevych, and Herman Kamper. 2021. Acoustic word embeddings for zero- resource languages using self-supervised contrastive learning and multilingual adaptation. In 202/ IEEE Spoken Language Technology Workshop (SLT), pages 919-926.
Dupoux, and Yossi Adi 2021. Textless speech emo- tion conversion using decomposed and discrete rep- resentations. arXiv preprint arXiv:2111 1..0402.
Taku Kudo. 2018. Subword regularization: Improv- ing neural network translation models with multiple subword candidates. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 66-75. Melbourne, Australia. Association for Computational Linguistics.
Jeff Johnson, Matthij Douze, and Hervé Jégou. 2017. Billion-scale similarity search with gpus. CoRR, abs/1702.08734.
Jacob Kahn, Morgane Rivière Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, FimaaneE Mazaré, Julien Karadayi, Vitaliy Liptchinsky. Ronan Col- lobert, Christian Fuegen, Tatiana Likhomanenko, Gabriel Synnaeve, Armand Joulin, Abdelrahman Mo- hamed, and Emmanuel Dupoux. 2019. Libri-light: benchmark for ASR with limited or no supervision. CoRR, abss/192.00788
Brian Kulis and Michael I. Jordan. 2011. Revisiting k- means: New algorithms via bayesian nonparametrics. CoRR, abs/ 11 11..025..
Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brebisson, Yoshua Bengio, and Aaron Courville. 2019. Melgan: Generative adversarial networks for conditional waveform synthesis.
Herman Kamper: 2018. Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder models. CoRR, abs/181 0bb//811100005/ 1.00403.
Kushal Lakhotia, Evgeny Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu Anh Nguyen, Jade Copet, Alexei Baevski, Adelrahman Mohamed, and Emmanuel Dupoux. 2021. Gener- ative spoken language modeling from raw audio. CoRR, abs/2102.01 192.
Herman Kamper. 2022. Word segmentation on discov- ered phone units with dynamic programming and self-supervised scoring.
Eugene Kharitonov, Jade Copet. Kushal Lakhotia, Tu Anh Nguyen. Paden Tomasello, Ann Lee, Ali Elkahky, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, and Yossi Adi. 2022. textless- lib: a library for textless spoken language processing.
Andy T. Liu, Po chun Hsu, and Hung- Yi Lee. 2019. Un- supervised End-to-End Learning of Discrete Linguis- tic Units for Voice Conversion. In Proc. Interspeech 2019, pages 1108-1112.
Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Mor- gane Rivière, Abdelrahman Mohamed, Emmanuel Dupoux, and Wei-Ning Hsu. 2021a. Text-free prosody-award generative spoken language modeling. CoRR, abs/2109.0322
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013, Efficient estimation of word representa- tions in vector space.
Tomas Mikolov, Ilya Sutskever, Anoop Deoras, Hai Son Le, Stefan Kombrink, and Jan Honzaernocky. 201 I. Subword language modeling with neural networks.
Eugene Kharitonov, Ann Lee. Adam Polyak, Yossi Adi Jade Copet, Kushal Lakhotia, Tu-Anh Nguyen, Mor- gane Rivière, AbdeIrahman Mohamed, Emmanuel Dupoux, et al. 2021b. Text-free prosody-aware gen- erative spoken language modeling. arXiv preprint arXiv:2109.03264.
Tu Anh Nguyen, Maureen de Seyssel, Robin Algayres. Patricia Roze, Ewan Dunbar, and Emmanuel Dupoux. 2022a. Are word boundaries useful for unsupervised language learning?
Jungil Kong, Jachyeon Kim, and Jackyoung Bae. 2020. Hifi-gan: Generative adversarial networks for ef- ficient and high fidelity speech synthesis. CoRR, abs/2010.05646.
Tu Anh Nguyen, Maureen de Seyssel, Patricia Rozé, Morgane Rivière, Evgeny Kharitonov, Alexei Baevski, Ewan Dunbar, and Emmanuel Dupoux. 2020. The zero resource speech benchmark 2021: Metrics and baselines for unsupervised spoken lan- guage modeling. CoRR, abs/2011.11553
Vijay Korthikanti, Jared Casper, Sangkug Lym. Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. 2022. Reducing acti- vation recomputation in large transformer models.
Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello. Robin Algayres. Benoit Sagot, Abdelrahman Mo- hamed, et al. 2022b. Generative spoken dialogue lan- guage modeling. arXiv preprint 22t3:22003316302
Felix Kreuk, Joseph Keshet, and Yossi Adi 2020. Self-supervised contrastive learning for unsuper- vised phoneme segmentation. arXiv preprint 0tti:2007.134666
Tu Anh Nguyen, Benoit Sagot, and Emmanuel Dupoux. 2022c. Are discrete units necessary for spoken lan- guage modeling? IEEE Journal of Selected Topics in Signal Processing, pages 1-9.
Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu-Anh Nguyen, Morgane Rivière, Wei- Ning Hsu, Abdelrahman Mohamed, Emmanuel

---

Zizheng Pan, Bohan Zhuang, Jing Liu, Haoyu He, and Jianfei Cai. 2021. Scalable vision transformers with hierarchical pooling.
Shane Settle and Karen Livescu. 2016. Discrimina- tive acoustic word embeddings: Recurrent neural network-based approaches. CoRR, abs/161 1.02550.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and San- jeev Khudanpur. 2015. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages -200-
Jonathan Shen, Ruoming Pang. Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang. Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan. Rif A. Saurous, Yannis Agionvrgiannakis, and Yonghui Wu. 2018. Natural tts synthesis by con- ditioning wavenet on mel spectrogram predictions. In 2018 IEEF International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pages 4779 4783.
Puyuan Peng and David Harwath. 2023. Word dis- covery in visually grounded, self-supervised speech
models.
Puyuan Peng, Herman Kamper, and Karen Livescu. 2020. A correspondence variational autoencoder for unsupervised acoustic word embeddings.
Jonathan Shen, Ruoming Pang. Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang. Zhifeng Chen, Yu Zhang, Yuxuan Wang, RJ Skerry-Ryan, Rif A. Saurous, Yannis Agiomyrgiannakis, and Yonghui Wu. 2017. Natural tts synthesis by con- ditioning wavenet on mel spectrogram predictions.
Wei Ping, Kainan Peng, Andrew Gibiansky. Sercan Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. 2017. Deep voice 3: 2000-speaker neu- ral text-to-speech.
Alexis Thual, Corentin Dancette. Julien Karadayi, Juan Benjumea, and Emmanuel Dupoux. 2018. K- nearest neighbours approach to unsupervised spoken term discovery. In IEEE Spoken Language Technol- ogy S1.7-2018, Proceedings of SLT 2018, Athènes, Greece.
Mark A. Pitt, Keith Johnson, Elizabeth Hume, Scott Kiesling, and William Raymond. 2005, The buckeye corpus of conversational speech: labeling conven- tions and a test of transcriber reliability. Speech Communication, 45-19.89-95.
Andros Tjandra, Berrak Sisman, Mingyang Zhang, Sakriani Sakti, Haizhou Li. and Satoshi Nakamura. 2019. VQVAE Unsupervised Unit Discovery and Multi-Scale Code2Spec Inverter for Zerospeech Chal- lenge 2019, In Proc. Interspeech 2019, pages 1118- 1122.
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab- delrahman Mohamed, and Emmanuel Dupoux. 2021. Speech resynthesis from discrete disentangled self- supervised representations. CoRR, abs/2 10.00535.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. 2018. Waveglow: A flow-based generative network for speech synthesis. CoRR, abs//811.00000 1.00002.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals. Alex Graves. Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. 2016. Wavenet: A generative model for raw audio. CoRR, (tss/1699.0399...
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak su- pervision. Technical report, Tech. Rep., OpenAl.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. CoRR, abs//807.00774.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Tlya Sutskever. 2019. Language models are unsupervised multitask learners.
Lisa Van Staden and Herman Kamper. 2020. A compari- son of self self-supervised speech representations as input features for unsupervised acoustic word embeddings.
Okko Johannes Räsänen, Gabriel Doyle, and Michael C. Frank. 2018. Pre-linguistic segmentation of speech into syllable-like units. Cognition, 171:130-1500 150.
A. Vasuki and P.T. Vanathi. 2006. A review of vector quantization techniques. IEEE Potentials, 25(4):39- 47.
Flávio Ribeiro, Dinei Florêncio, Cha Zhang, and Michael Seltzer. 2011. Crowdmos: An approach for crowdsourcing mean opinion score studies. In 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 2416-2419.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.0376..
Morgane Riviere, Jade Copet, and Gabriel Synnaeve. 2021. Asr4real: An extended benchmark for speech models.
Maarten Versteegh, Xavier Anguera, Aren Jansen, and Emmanuel Dupoux. 2016. The zero resource speech challenge 2015: Proposed approaches and results. Procedia Computer Science, 81:67-72, SLTU-2016 5th Workshop on Spoken Language Technologies for Under-resourced languages 09-12 May 2016 Yo- gyakarta, Indonesia.
Morgane Rivière and Emmanuel Dupoux. 2021. To- wards unsupervised learning of speech features in the wild. 2021 IEEE Spoken Language Technology Workshop (SLT), pages 156-163.

---

Junjie Wu, 2012. The Uniform Effect of K-means Clus- tering, pages 17-35. Springer Berlin Heidelberg, Berlin, Heidelberg.
to speech. Table 3 shows generation and zero- shot scores. Overall, regarding speech genera- tion, 200ms-tGSLM outperforms SyLseg-IGSLMM dpparse-IGSLM and also GSLM. For zero-shot tasks, once again, all models score similarly. ABX scores are again obtained for GSLM with embed- dings extracted from the 9th layer of the trans- former and for (GSLM from the lexical tokens.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao. Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah. Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang. Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine trans- lation. CoRR, abs/ 0bss/669.08/// 609.08144.
Even though true word boundaries strongly ben- efit tGSLM, using unsupervised speech segmenta- tion methods did not prove beneficial. We think this is due to the low performances of state-of-the- speech segmentation systems. These latter are only marginally better than random segmentations and lag largely behind text segmentation performances (Dunbar et al., 2022b; Algayres et al., 2022b). This result suggests that progress is needed in unsuper- vised speech segmentation to be able to combine segmented units into intelligible speech. After all, the best segmentation method that we works for is the 200ms method. We have also experimented with other durations as 220ms and 360ms. We chose to go on with 200ms based on a com- promise between maximal duration and maximal zero-shot task performances. These scores can be found in Appendix Table 4.
Shuwen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng- I Jeff' Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang. Wei- Wei-Cheng Tseng, Ko tik Lee, Da-Rong Liu, Zili Huang. Shuyan Dong. Shang-War Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung yi Lee. 2021. SUPERB: Speech Processing Universal PERformance Benchmark, In Proc. Inter- speech 2021, pages 199--118.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Mareo Tagliasacchi. 2021. Sound- stream: An end-to-end neural audio codec. CoRR, abs/2107.03312.
Yaoming Zhu, Sidi Lu, Lei Zheng. Jiaxian Guo, Weinan Zhang. Jun Wang, and Yong Yu. 2018 Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 18, page 1097-1 100, New York, NY, USA. Association for Computing Machinery.
A.2 Discussion on
A.2.1 Mathematical details on
George Kingsley Zipf. 1949. Human behavior and the principle of least effort. Addison-Westt Press.
Let us now derive g computation. Given a train- ing corpus, that is segmented and encoded into a collection of acoustic tokens (ai)sen- A PCA is trained on (a:)ieN and the d first dimensions are kept, let us write (a()isN the resulting vec- tors and (vo..., ..., vt), the explained variance of each PCA dimensions. Then, train d sepa- rate k-means on each dimension of the PCA. The number of cluster per k-means is computed as K Va'
A Supplementary materials
A.1 Speech segmentation
To study the impact of speech segmentation on IGSLM, we trained this model on LibriSpeech with two extra segmentation methods: SylSeg (Räsä- al., 2018), and DP-Parse (Algayres et al., 2022b) Sylseg segments speech into syllable-like units, using damped oscillators that exploit rhyth- mie cues of syllabic structure in speech. DP-Parse (Algayres et al., 2022b) segments speech into word- like units with state-of-the-ar performances. This model adapts non parametric Bayesian model for text segmentation (Goldwater et al., 2009)
The values of d and K were set to maximize the scores at the zero-shot tasks. Once the k-means are trained, the centroids are stored in d dictionaries (ko, ki)) For any S N, we compute q(a,) by assigning Vij < d, [(a) to its closest centroids in ky. Finally, clus- ter ids are turned into one-hot vectors and concate- nated into a single vector. The following operations
K m
).
To
"Wo did not train these models on 1.L6k-clean because DP-Parse is hard to scale to large datascts.

---

WUGGYT
SBLIMPT
ABX sem
ABXPos
PPX@LS-VERT,
PPXØLJ-VERT,
GSLM
65.85
54.35
55.18
61.61
664.23
497.65
sybseg-ISSM
64.39
53.21
54.64
60.01
634.34
505.87
Upparse-IGSLM
65.54
53.82
55.6
58.65
634.34
505.87
200ms-1GSLM 63.15
53.34
55.08
60.24
610.32
490.32
Table Results on zero-shot generation tasks for GSLM for (GSLM on three different speech segmentation methods. Models are all trained LibriSpeech. ABX is computed tGSLM lexical tokens and GSLM 9th layer
models
sWUGGY
sBLIMP1
ABX scm 1
ABXpos
average
12LML-IGSLM
61.55
51,86
54.74
60.12
57.32
200ms-tGSI MM
63.15
53.34
55.08
60.24
57.95
280ms-1GSL.M
61.89
51.64
52.8
56.28
55.65
360ms-tGSLM
60.18
51.29
52.18
55.45
54.75
Table 4: Zero-shot tasks computed (GSLM trained LibriSpeech for different unit durations
sum up the process.
acoustic tokens that interfere with the semantic and syntactic modelling. In our opinion, this sig- nal gives away both local information, direct right and left context due to coarticulation, and global sentence-level information (relative token position and speaker identity).
argmmax (a:[0) koli)) isK
argmnax (a:[1] - kıli))
Vi S n(a()
argmax (a:[d] kaD
A.2.3 On using MFCCs instead of Wav2vec2.0
1
One may say that q used to mitigate the down- sides of the attention mechanism of Wav2vec2.0 why not use more local features like MFCC or Mel- filterbanks? We argue that even though these latter features are still good for supervised tasks as ASR Radford et al. (2022), they are substantially out- performed by recent self-supervised speech mod- els (Wav2vec2.0, CPC, HuBERT,....) at the tasks of zero-shot word discrimination (Algayres et al., 2022a; Van Staden and Kamper, 2020) and key- word spotting (Yang et al., 2021). To prove our point, we compare the performances of MFCCs compare to Wav2vec2.0 at the task of discrimi- nating acoustic tokens. As a reminder, the acous- tic tokens that used in model, are the output of an SSE model from Algayres al. (2022a), pre-trained to embed variable-length se- quences Wav2vec2.0 frames into fixed-size vec- tors. The same kind of SSE model but pre-trained on MFCC frames is also provided by Algayres et al. (2022a). Let segment the LibriSpeech corpus every 200ms and embed the speech segments with both SSE models so that get two collections of acoustic tokens: (a, (af and (a.:2v2) ) )ss. Let also apply the function Wav2vec2.0 acoustic tokens so that we get: (q(a((202) ) To measure performances, we use our N ED metric on the three collections of embeddings. From the
oneeot(q (q(; ()))
onehol(q(a, 1))
((a.)
onchot(q(a:(d)) ((as) < concatenate(g(a.00) ((: (as(d))
A.2.2 Ablation on
The function q introduced in Section 3.1.2, com- posed of a PCA and our d-k-mcans method, is ablated Table In all configurations, the embed- dings right after the Ema function are used to compute the ABX and NED scores. On the one hand, degrades the phonetic information in the lexical tokens (NED increases) and makes train- ing harder (validation loss increases). On the other hand, q maximizes semantic and syntactic informa- tion (ABX increases) as well as generation quality (PPX decreases). A null value Table 6 means that the model is not able to produce intelligible sentences with this setup. First, these experiments show the necessity of g for the 200ms-1GSLM to generate spoken sentences. Second, the combi- nation of these results reveals that y prevents the model from converging quickly bad local min- imum that hinders generalization.
follows our intuition from Section 3.1.2: there seems to be a variance signal encoded in the

---

london lend lemon
land landland
land
land land
eleven london leaden
land
land
in
blood
let london imited limpid
latin
england
lemon limb clump
loncon n
atlanta
englland
ongland @ngglandnggandd
engiandraly
leaven lemon london
london
-0.0
unch london
london england
englandnglond
ongland
ongland"ggland
london
paris
london lemon blended
ondon
africa
and
london
-0.00
london london
london
london leash
-15
-20
25
1.2
134
1330
15e
1.3
feeble
betial
pretty
preposternus
beautiful
beautiful
-0.20
beautiful
pleasant precious
bad
besutiful
beautiful
mulless
beautiful
pure
powertul
bad bad
beautiful
bellef
-0.90
powerful
bad
useful
beautiful
mule's s
powerful
beautiful
0-
beautiful
beautiful
feeble boautiful
beautiful
bewuhiful
beautiful
fruitful
Intused
beautiful
feeble pupils
beautiful
50
beautiful
beautiful
beautiful
beautiful
beautiful
beautiful
useful
useful
beautiful
beautiful
beautiful
big
big
useful
-0.30
beautiful
beautiful
big
dutiful
big big
:00
50
:30
2.2
2.3
214 215 2i6 2.1 2.2
210
2.00
chair
trunk
chair
chair
table
60-
100
share
chair share
dish
chair shared church
cage
desk
chair
chest
chair
50
chair church
chain
chair
church church
chair shared
chair
church
chain
desk
chair
table
chair
chair
chair
chair
church
chair chair share
their
chest
chair
-100
share chair
chair sure share
chair
chair
chair
chest
dish
chair
chair
table
church
-60
chest
chair
blown below below
bright bright
blackens
black
balow below
black
brigh:
bright
below
below
black black
bright
slow
below
blow
below
black
black
bright
led
blow
black
pale
9.29
purple
822
-0.29
Jase
Ioose looms
lose
ute lute
blew
blue
blue
0.2
black
black
bluc
blue
loose
blue
220
black
-0.50
loop
glued
blue
bluebells
618
blew
blue
blue
blue
brown
brown
abe
Figure 4: 1-SNE representations of acoustic (left side) lexical (right side) tokens. After training gold-IGSIMM speech segments corresponding word tokens in the LibriSpeech dev-clean subset indexed into their acoustic or lexical form. By probing an acoustic or lexical token (appearing in red), we can have look at their acoustic and lexical nearest neighbours. The neighbours that appear in green those deemed as semantically related to the probe.

---

models
sWUGGY
<BLIMP
ABX
ABX
average
next word
61.57
$2.08
51.48
53.84
54.75
next Two words
63.02
53.48
54.79
58.01
57.35
next three words
63,15
53.34
55,08
60.24
57.95
next four words
62.25
53.1
54.43
58.81
57.14
Table Zero-shot tasks for 200ms-IGSIM trained on LibriSpeech predict the next one, two, three, or words
PCA d-k-means
Valid loss)
NED.
ARX
ABXpos
PPX@IS-VERT,
PPX@LJ-VERT.
200Ms-GGSM
2.51
35.21
53.87
58.40
null
null
200ms-GSLM
4.33
41.50
54.16
57.99
840.65
null
200ms-GGSLM
6.21
44.32
55.08
60.24
610.32
490.32
gold-IGSLM
3,99
17.21
55.13
63.54
608.24
475.65
gold-IGSLM
6.20
21.87
58.59
67.71
432.78
384.57
gold-IGSLM
7.15
22.70
65.60
75.59
361.84
255.32
Table 6: Results zero-shot and generation tasks for ablations of the PCA and d-k-means components of the Lex Embb function, Models are trained on LibriSpeech, ABX and NED are computed on (GSLM lexical tokens. null means that no intelligible speech can be generated in this setting.
results Table 7, we see that Wav2vec2.0 leads to much better acoustic tokens than MFCCs. More- over, even when is applied on Wav2vec2.0 acous- tie tokens, the NED score of (q(a:1202) )))) is still much lower than on (a. ).g.. This latter has a NED score of 65%, which means that two neighbouring MFCC acoustic tokens have on av- erage less than half of their phonemes in common. For that reason, why excluded MFCC from experiments on speech generation.
norm and ReLU activation. q is of a PCA and collection of k-means that are trained on L.Lkk- clean. The PCA has d = 21 dimensions and the number of centroids for the first k-means is K 10.
Transformer is identical to the one used the original GSLM paper (Lakhotia et al., 2021). It contains 12 transformer layers with 16 heads, 1024-neuron attention matrices, and 4096-neurons FFN. On top of the transformer, the three paral- lel h1, h1,h2,h3 functions are 1024-neurons fully nected layers. L,h1,h2,h3 and the transformer are trained on 32 GPUs, for 200k iterations on either the LibriSpeech or LL6k-clean. Each batch is com- posed of 64 audio sentences that are composed of 64 tokens. The learning rate is set at 5-4 with a warm-up of 5000 updates and polynomial decay. We use Adam optimizer with weight decay of 0.1. A dropout of 0.1 is applied during training. The loss function is the NCE loss with temperature of 0.1 and 500 negative samples.
acoustic tokens (aju)))gg
N WE1)
65.56
(a (aptar)) JeC
31.81
(g(a
36.71
Table 7: NED scores of 200ms-long acoustic tokens built on MFCC: (a (ar"/"")e on Wav2vec2.0 frames: (n)2=2)) Ns and finally on Wav2vec2.0 frames when ? is applied: (q(a)222))
A.3 Hyperparameters
Sampling is performed in FAISS k-NN (John- son et al., 2017) that contains all the lexical tokens segmented in the dev-clean and test-clean from the LibriSpeech (roughly 10 hours of speech). The number of nearest neighbours from which the next token is sampled is 1000.
Wav2vec2.0 and SSE are trained on the Lib- riSpeech corpus respectively by Baevski et al. (2020) and Algayres et al. (2022a). Wav2vec2.0 Base is a stack of convolution layers and trans- former layers. The SSE is composed of a one GLU convolution layer (kernel size: 4, number of chan- nels: 512, stride: 1), transformer layer (attention heads: 4, size of attention matrices: 512 neurons, and FFN: 2048 neurons) and final max-pooling layer along the time axis.
Speech generation model is an encoder and a decoder that shares the same architecture: 4 trans- former layers with 8 heads, 512-nee neurons attention matrices, and 3072-neurons FFN. It is trained on 32 GPUs, for 30k iterations on the LibriSpeech. Each batch composed of four audio sentences that are at maximum 20 seconds long. The learning rate is set at 5 with a warm-up of 103 updates and
LexEmb is composed of two functions o is a stack of five three-layers blocks each formed by a n024-neurons fully connected layer, a layer

---

polynomial decay, We use dropout probability of 0.1 and Adam optimizer with a weight decay of 0.1. The Tacotron2.0 from Lakhotia al. (2021); Kharitonov et al. (2022) was trained on LJ.
A.6 Inference time complexity
Taking the calculation forward cost from Pan et al. (2021), for sequence length of size n, and a transformer of L 16 layers and dimension = 1024, forward pass costs (12nd + 2nd) L. This would be the cost of forward pass in the GSLM model, but our tGSLM costs little bit more with its LexEmb function and its sampling procedure. A forward through our LexEmb function costs 5nd? (5 línear layers) and the sampling procedure costs * 100.000 (we usually take 100k items for the k-NN search). Therefore, 1GSLM cost (12nd + 2nd) + 5nd + d 100000. For sequence of second (therefore n 5 for tGSLM and 25 for GSLM), by replacing those values in the former calculation, we find that forward in tGSLM cost 1.1e6 which is 5 times less costly than forward in GSLM that would cost 5.2e6. Therefore, even at inference time, our tGSLM should be much faster than GSLM to run.
A.4 Probing acoustic and lexical spaces Figure is a visualization of the acoustic and lexi- cal representation learned by gold-IGSLM which echo work on speech word embeddings from Chung and Glass (2018). All speech segments corresponding to real words in the LibriSpeech dev-clean set are indexed in k-NN graphs their acoustic lexical form. Each embedding is la- belled with its true transcription. By scarching for the nearest neighbors of a centre word (in red in the figure), we highlight green the neighbours that we judged semantically related to the centre word. Figure 4 shows that an acoustic token has usually no semantically related neighbour other than ones with the same transcription. By contrast, lexical tokens have semantic and syntactic proper- ties: 'London' is close other cities and countries, 'blue' is close to colour names, beautiful close other positive adjectives, and 'chair' is close to "desk" and 'table'. Nonetheless, it appears acoustic information has leaked from the acoustic tokens into the lexical tokens. For instance, the lexical neighbours of "buue" are colours or shades that start with a 'b' and 'chest' appears in the neighbourhood of "chair".
A.5 Estimation of memory consumption
To estimate the memory consumption of trans- former LM with L 16 layers, a single atten- tion head, a batch size of and an embedding size 1024 let us write C RRadd sen- tence of n tokens represented with embeddings size Using the formula expressed in Kor- thikanti et al. (2022), the number of activations to store in memory during backpropagation is ap- proximately (buffers and negligible values being omitted) @L.., n, d) Lnd(34 + 5)). In the LL6k- clean corpus, sentences with make n 1500 for GSLM and for 200ms-1GSLM. 200ms-IGSLM should expect a memory reduction by factor of 6(1611000.1044) z
60s-long on average 300
0(16,300,0024)
5.83 compared to GSLM. In practice, we observe lower memory reduction (<s 1.76) which can be explained by the additional parameters that are present in 200ms-1GSI.M and not in GSLM, namely the LezEmb function and three prediction heads.

---

200ms-tGSLM examples
Generation at LJ-VERT
What is it ask her mother i want to see you said mrs tumbled want to tell you what you you said mr cockry you are no more chance than you know.
We have no desire to prevent to the astonishment of that person from the government who is not so far for receiving any property or relation to the world.
Her father in her son were under growth her father was just like a treasure man who was a devil and hazards beyond his words she was a very clearly.
We also see that it will be obliged to invite us to applyge them to observe such thing is basc we must not set down that the And although he was not equally successful to him he sought the pririate regularly observed his friends invent to him and presented him their own secret he had did Because they were rested and although they could not expect to be obliged regarded as men of a gold and power they were not really unbusy
You see he is if i miss thing think he is dead it said mrs carpenter rather smallly for anything if he is total let she said He remembered that great city which he tried to entertain in its pointedof view but he was very pleasant to him and could not bring whom away besides this
Having required measure for month before their distance of sixty years he appeared to be affected by any conditions of one state and have in no battle
Now the king's ' brothers came to him and brought him up and said i'll poor woman woke it of you anything but am brother and borrows my
Generation at LS-VERT He turned his hands on the sale exposition and gave him to acy of old meal which wealth had never bore be foreseenly large.
While waiting to him he wished that he would wait for himself into his mother's house and held light he was that that she might be able to look.
And perhaps i have nothing to say about what would you want to know i did don't 't know i supposc you want to know what could call the That's all can't 't do insaid woman looking out of the croad toward him while don't 't know such any end enging his hand you seem to see your And having been described as the great activity of that which he was attempting all that if he now remar it for his purpose was intended with principles as was the time had been prepared for for that such was the place that when he was saing to his land she'd d made up all though blood that he It was just as willing to oppose the person who had been told of his chion he was now about to go to bank in the family to Having been in a moment' officially desirable to acquaint him with his reference with the glorious presence of his master's cabinet he did not return to a subject of Yes was said he but was a general service she began could not file forhard seek want to take my as andt understand the chance of But afterwards they had gone top what waking into the stone doors the weathering tight their habitation and the north were histor carryance and mr carb's face and
Table 8: Example generations of 200ms-IGSLM trained 1.66--llean. These generations are selected from batches of sentences that have a VERT equal 0.113 (S-VERT) or 0.189 (LJ- (L-VERT).

---

gold-IGSLM examples
Generation at LJ-VERT
But you have been wanting to teach mc all her life in the world of her own healthy health and she has her fathers abilities an your pride.
The old woman or that he would have learned all her life amongst the gods and teaching them in their lather's studies and having been up the days.
goes on until know what i am doing while i am going away from my camp the neighbourhood morrow we you from the whole on my.
An clegant geographical character would you think it deed or an excellent thing to do with the hold in the future won't 't you pick up a bit of an
The evening of the twenty fifth of november eighteen united eight he returned to his royal house an the hold of the hospital the next evening he you Guiding them in some ordinary way buying them into cold buying them with copal spoon which should be thrown out of the souls of the bulkhead Secies and germany each of them had undergone more than three thousand roubles and hour saved bit of jewelry from odin share and the hardness and Of the kavin and when to the door where she stood few minutes later to reach the bottom of the harboured near the labyrinth where she reached her It says the king listening the light of his bushy fingers an holding his pipe in his arms do not bother me any more about it you know more than The investigation and on his returned to her fathers room he set down his gun at hundred yards and the middle of the hall to learn the hut Generation at LS-VERT
I can tell her that only one of my friends and loves do you think would read her about this uttered it all the wicked said missus williams
She reached her big house and stood by the dora in turning to the king he said to her you will not marion me any more have you hear
To his voices and his broken heart screen with delight to henry smokeless who had entered into his dining room to limp him a mystic playful of his faintest
I shall not go without thee said heat pausing to her part a of good direction and fixing her ices upon her eyes with a distant cheeks to her.
Than time of missus esplanade visit her own house and china herself alone of theirlocal service and the frere settlements where built for thousand of the happiest teachers
Father and mother were seated at boston waiting for the empty school at ostrog at nine o' d'clock a the knight of july evening a ninth jeanne annic eighteen Minutes later he heard a bill calling against the young man who had denounce him his face became a melancholy shake in his astonishment what said george
poor boy in has good power for somebody's harm be at heaven what could you to givewhat this seemed to him hard proposition
Paper it was needless to be summoned said about recovering the bicycle with her finger to
by the princess and the girl became very much surprise and
To touch he a gentle young ma' am and does not see any other foreign of mortality unconnected with her father who is afraid of his flesh
Table 9: Example generations of gold-GGSL.M, trained LibriSpeech. These generations selected from
batches of sentences that have a VERT equal 0.113 (LS-VERY) or 0.189 (LJ-VERT).

---

GSLM examples
Generation at LJ-VERT
They did time in the desert two or three hundred ycars afterward among them the castle was not my lather and they were found in palestine by fire and they Another excellent is descended a breath let the corp of prisoner and blow was begun the bell rang the gunsprang from the captain's 's paul and dropped into the And then the passing future would have been too much but to waittill the end of the week and after a little time she had gone down to the palace
But he passed along entirely untouched and was still together so frightened in the morning he went look out for some place with barian laine and then he The brast of the bravest of the entire youth and of many of the slaves of the counillllors or of every fine breed and of the princess of france has
He had not in the least delicate way of helping her but had helld her into pretty soft and a passionate graceful manner he told her every day because
But that man did not fight in the second place no ne did pay the attention poverty it is not tpossible to suspect that the man of the previous But all the people had come to see me and had not seen me again and they felt as if were again coming to see me and so little
And people stared at him for moment if they were dead but he had not told them of his destiny that he would do so and they had
And a cow calling up his pipe said that no sign of the procession was ever heard and that no punishment was made or judgment was made nor any other
Generation at LS-VERT
His proposals that being so poing doubful i should very much regard and alia in boa's addition to cloak the great morning had given me plague off waivering she Someone to found brown line and dance spent moment over the vessel all saw the fair young chinese yard and dry he would waved dances in bubbles from
All cathics that are not duc to f co notion or naturalist that is intentionble but if there is a personality of faith in them who was intentupon for seeing
The reef made the partets at the corner of a platform with them rose and ground on the floor of the lobby and of chapter fourteen two thousand se of As if sudden impulse were convinced of their usual impulses and a strong exercise upon them or rather in their progress to bring their education to the reduction of manly And rushing off from the cold winds in the west in the silence of the rock the cherry wavering soft quictness of people makes breath so cheap in a course
He had been burden with visitor and had petched his old preserance for death and mary the intamminable enterprising scenes caused by constantiis this trumpetts und drrawn courage and he Great worked done an artificial lines of bounding is below the had an arm as it were lookedfted itself and everything was so exquisite that the site was hard To jew knew that they had been driven a doctor adreadful mattering to you the young girl whom eyed by a relatives ever since daily matters while week before
Evenings in a ball volume whose close ways were rolted in whther cuts that fiddler devilalonsome his wife soldiers were harassing a women with deafferenren american last grading under fair
Table 10: Example generations of GSLM, trained L.6kk. These generations are selected from two batches of sentences that have a VERT equal 0.113 (LS-VERY) or 0.189 (LJ-VERT)

---

WUGGY1
SBLIMP
B.2
Attempt clustering SSEs
GSLM
70.36
56.31
Here are some attempts and results at the task of clustering speech fragments. First, we retrieved the gold-standard word segmentation of the LibriSpeech corpus and embedded all word tokens with the SSE model from Algayres et al. (2022a). Then, we clustered with k-means and hierarchical- k-means all the SSEs in the LibriSpeech into K classes. To simplify, we set K the true number of word types. Finally, we trained transformer LM to predict the next discrete clusters. We observed that the training loss decreased but not the validation loss. The reason for this failure was simple: most of the clusters found by k-means or hierarchical k-means that were in the validation set were not present in the training set. These results show that those baseline clustering algorithms are not suited to the task of spoken language modelling.
200ms-IGSLM
68.53
55.31
BC-30k
71.32
55.08
BC-2k
69.00
54.44
Table II: sWUGGY and sBLiMP GSLM and 200ms- LGSLM and the large units
B Clustering SSEs
B.1 The problem of clustering large units
In the introduction, we argued that the clustering of a large collection of word-size speech fragments is a daumting challenge. The first difficulty is the very large number of word types in corpus. For instance, there are #44.5k word types in the 10- hour-long Buckeye corpus (Pitt et al., 2005), and 209k the 960-hour-long LibriSpeech (Panayotov et al., 2015). Most clustering algorithms (k-means, siturrhical-k-meaan Spectral, Chinese Whisper (Biemann, 2006). Dirichlet Process-means (Kulis and Jordan, 2011), Brown (Brown al., 1992)), require in input either this unknown number of clusters threshold parameter that controls the number of clusters². By misestimating the num- ber of clusters, we introduce errors that can mis- lead the downstream LM. The second problem is the highly skewed distribution of the frequency of word types. This phenomenon, known as Zipf's Law (Zipf, 1949), states that, in text, there is linear relation between the log-rank of word types and their log-frequencies. In practice, means that, in text, most word tokens are hapaxes (i.e. they have a word type that appears only once), and a small proportion of word types account for most tokens. Therefore, even if the number of word types could be correctly estimated, a clustering algorithm would have to produce many singleton clusters, which is a hard task for clustering models, especially for k-means that tend to create equal-size clusters (Wu, 2012). For those reasons, cluster- ing algorithm is likely to produce ronovecovee errors that will negatively impact the downstream spoken LM.
Regarding more elaborate clustering methods, Elkahky et al. (2023) has used a combination of HuBERT units, smoothing technics, Byte- Pair-Encoding (BPE) and Brown clustering (BC) (Brown al., 1992) with either 30k clusters 2k clusters. The result is discretisation of speech with word-size units. We have proved in Elkahky et (2023) that these units can be used train a HuBERT model and improves its downstream ASR performances. Here, we discretised the Libri- Light 6k clean (Rivière and Dupoux, 2021) with these large units using either 30k and 2k clusters and gave them in input to transformer LM trained to predict the next discrete units. We report in Ta- ble 11 sWUGGY and sBLIMP scores of GSLM, 100ms--GGSLM and the large units (called BC-30k and BC-2k). The scores show that these units per- form equally well to GSLM and 200ms-tGSL.M These results show that even the most elaborate methods of clustering do not bring better results than our method to adapt spoken LM to continuous inputs.
"Some unsupervised methods to find the number of clusters exist, like the 'elbow methods' in k-means, but these methods are quite noisy and hard to apply when the number of classes is that large.


Reviews:
[{"review_id": "QvPuua0GZC", "reviewer": "Reviewer_Uy5K", "summary": "", "questions": "**A.** I think the \"lexical tokens\" and \"acoustic tokens\" namings are confusing at first read, and not really accurate. In the literature, a token is a discrete element, from a finite ensemble. I would recommend to rename them to something that might indicate that these elements are actually continuous / vectors from a continuous space. What is your opinion on this?\n\n**B.** Did you made some benchmarks on the generation speed? I think it could be express in tokens/sec and sec_of_audio/sec. As your model now has additional modules (namely LexEmb, multiple output heads, using FAISS search), I assume that the generation speed gain is not linear to the token sequence reduction;\n\n**C.** Maybe a suggestion for future research: what do you think about conditioning the output heads on the results of the previous one? By that I mean that $\\mathbf{y}\\_i = h(\\cdot)$ could be conditioned on $\\mathbf{y}\\_{i-1}$, e.g. $\\mathbf{y}\\_i = h(\\mathbf{x}, \\mathbf{y}\\_{i-1})$. Some models applied the same method of yours for symbolic music modeling ([Compound Word](https://ojs.aaai.org/index.php/AAAI/article/view/16091), [MusicBERT](https://aclanthology.org/2021.findings-acl.70/)). In practice, generating several distinct elements at the same time, unconditionally although they are meant to be complementary / tied, does not work very well for causal generation (as GANS / Diffusion / any non-autoregressive model). Thus conditioning each output on the very last predicted is almost always guaranteed to give more coherent results. [Compound Word](https://ojs.aaai.org/index.php/AAAI/article/view/16091) does it partially.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer very much for their time and effort into reviewing our paper. This review is impressively complete and thoughtful and will definitely help us a lot in improving the paper. Here are the responses to your questions and comments.\n\n\"The authors could add tokenizations from AudioLM or BEATs.\"\n\nIndeed, our paper is missing a comparison with AudioLM  (it did not exist yet when we started to work on tgslm) which has a similar (or slightly higher) semantic consistency than the original GSLM.  Yet, adding audioLM to our paper would not change the main takeaway which is that we get similar performances with continuous 200ms GSLM compared to discrete 40ms GSLM.\n\n\"There is no information about generation speed at inference [and] it would be great to also consider the FAISS memory consumption for inference, and give concrete examples with numbers, e.g. how pairs of SEE / lexical token translate into memory consumption. This could be a limitation depending on the hardware being used. Also as is correlated to the lengths of the speech sequences, it could raise questions when using such model in real conditions. [...] Did you made some benchmarks on the generation speed? I think it could be express in tokens/sec and sec_of_audio/sec. As your model now has additional modules (namely LexEmb, multiple output heads, using FAISS search), I assume that the generation speed gain is not linear to the token sequence reduction\"\n\nIndeed the analysis on inference time is absent from the paper and could be a nice addition. Here is a little paragraph that we may add to the camera-ready paper.\n\nTaking our calculation from A.5, for a sequence length of size n, and a transformer of L layers and dimension d, a forward pass costs \n\n(12nd²+2n²d)*L FLOP\n\n This would be the cost of a forward pass in the GSLM model, but our tGSLM costs a little bit more with its LexEmb function and its sampling procedure. \nA forward through our LexEmb function cost 5*n*d² (5 linear layers) and the sampling procedure cost d*100.000 (we usually take 100k item for the k-NN search). Therefore, tGSLM cost \n\n(12nd²+2n²d)L + 5nd²+d*100000 FLOP\n\nFor a transformer of d=1024 and L=16, and a sequence of 1 second (therefore n=5 for tGSLM and n=25 for GSLM), by replacing those values in the former calculation, we find that a forward in tGSLM cost 1.1e6 FLOP which is 5 times less costly than a forward in GSLM that would cost 5.2e6 FLOP. \nTherefore, even at inference time, our tGSLM should be much faster than GSLM to run.\n\n\"I think the \"lexical tokens\" and \"acoustic tokens\" namings are confusing at first read, and not really accurate. In the literature, a token is a discrete element, from a finite ensemble. I would recommend renaming them to something that might indicate that these elements are actually continuous / vectors from a continuous space. What is your opinion on this?\"\n\nI agree with you, we could maybe call them acoustic embeddings and lexical embeddings. \n\n\"Maybe a suggestion for future research: what do you think about conditioning the output heads on the results of the previous one?\"\n\nWe did think about this and I might be wrong here but I think it would actually impair performances. Let me try to explain why. The main reason we use several heads is to prevent the causal LM to make prediction by relying on coarticulation (local acoustic information that leaks from one token to the next). This problem emerges because we have continuous tokens (and not discrete ones) that can carry low-level information. Consequently, the token at time t, is a little bit coarticulated with the token at t+1 (both at the acoustic and lexical level), but not coarticulated with tokens at t+2 or t+3 (because coarticulation is typically shorter than a phoneme or 80ms). By doing the three head predictions, the LM cannot rely on coarticulation too much (it actually can for head 1, but not for head 2 nor 3).\nTo come back to your proposition, if we use conditionning on the last result, we basically enable the head to use coarticulation by giving it the last predicted token. But I might be wrong and may it would improve performances as you propose.  \n\nThank you very much for all your comments, corrections and suggestions, that will make this paper much clearer and much easier to read.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The overall goal is very well motivated. Tackling audio modeling as a sequential problem is a very good direction offering a lot of flexibility in real-condition usages. I believe there is a lot of progress to be made, with significant impact, on the representation / tokenization of audio;\n* The implementation choices of tGSLM are well made and well justified (e.g. 200ms segmentation). Tackling this problem required a fair amount of model adaptations (LexEmb, using contrastive loss, FAISS for inference...) that the authors cleverly overcame;\n* The tasks and set of metrics are well chosen;\n* The interpretation of the \"lexical token\" embeddings is valuable and empirically shows that the method works;\n* The results show that LMs can generate speech using word-size frames directly fed as continuous embeddings;\n* The paper is globally well written and structured;\n* The references are very well documented and completed. Only one or two has missing URLs.", "reasons_to_reject": "* Although the experimental setup is well made (tasks + metrics), I think the paper can really benefits from one or two more baselines, even though this kind of audio representation is fairly new and that you answer well to the main goal of the paper that you set (i.e. generate speech using word-size audio embeddings). The authors could add tokenizations from [AudioLM](https://arxiv.org/abs/2209.03143) or [BEATs](https://arxiv.org/abs/2212.09058). Eventually a baseline without $LexEmb$, that could highlight its necessity. I think the reader would benefit to see how tGSLM compares to these baselines.\n* There is no information about generation speed at inference. This information is IMO as well important as the memory usage. It as a decisive point when designing a system / product. Here, considering the additional stack (FAISS) compared to models using finite vocabularies, the reader could want to know id it is worth to spend the time needed to implement it.\n* **(no big deal)** The memory gain estimation in A.5 is valuable. But it would be great to also consider the FAISS memory consumption for inference, and give concrete examples with numbers, e.g. how $N$ pairs of SEE / lexical token translate into memory consumption. This $N$ could be a limitation depending on the hardware being used. Also as $N$ is correlated to the lengths of the speech sequences, it could raise questions when using such model in real conditions.", "questions_to_authors": "**A.** I think the \"lexical tokens\" and \"acoustic tokens\" namings are confusing at first read, and not really accurate. In the literature, a token is a discrete element, from a finite ensemble. I would recommend to rename them to something that might indicate that these elements are actually continuous / vectors from a continuous space. What is your opinion on this?\n\n**B.** Did you made some benchmarks on the generation speed? I think it could be express in tokens/sec and sec_of_audio/sec. As your model now has additional modules (namely LexEmb, multiple output heads, using FAISS search), I assume that the generation speed gain is not linear to the token sequence reduction;\n\n**C.** Maybe a suggestion for future research: what do you think about conditioning the output heads on the results of the previous one? By that I mean that $\\mathbf{y}\\_i = h(\\cdot)$ could be conditioned on $\\mathbf{y}\\_{i-1}$, e.g. $\\mathbf{y}\\_i = h(\\mathbf{x}, \\mathbf{y}\\_{i-1})$. Some models applied the same method of yours for symbolic music modeling ([Compound Word](https://ojs.aaai.org/index.php/AAAI/article/view/16091), [MusicBERT](https://aclanthology.org/2021.findings-acl.70/)). In practice, generating several distinct elements at the same time, unconditionally although they are meant to be complementary / tied, does not work very well for causal generation (as GANS / Diffusion / any non-autoregressive model). Thus conditioning each output on the very last predicted is almost always guaranteed to give more coherent results. [Compound Word](https://ojs.aaai.org/index.php/AAAI/article/view/16091) does it partially.", "paper_topic_and_main_contributions": "The paper treats of \"audio\" language modeling, that is language represented as audio spoken modality. This task is tackled by training a language model (LM) on speech data represented as discrete tokens. Such model work similarly to text LMs, with a fixed-size vocabulary, cross-entropy loss during training and multinomial sampling during inference, and can be used jointly with vocoder models to autoregressively generate speech and audio.\nIn previous works, the vocabulary is built from sets of learned speech sequence embeddings (SSE) of sequences of speech. These sequences of speech are often short, between 20 to 40ms, and may however not be representative of any relevant information (i.e. when shorter than a phoneme or syllable). An other drawback is the extensive token sequence length created by that such method, which is a known bottleneck of Transformers / attention mechanism.\nIn this paper, the authors propose to use tokens of longer speech sequences, which would reduce the token sequence length and in turn reduce the overall complexity / memory usage, while having more semantically rich audio embeddings. Doing so will however create a very large number of possible SSEs / tokens, making having a finite and well balanced vocabulary intractable (called clustering problem). The authors propose then to directly work in a continuous space, that is not using a finite vocabulary but directly feed the model with continuous vectors. The model cannot directly be fed the SEEs, as \"as two speech segments may sound the same, i.e. be close in the acoustic space\", so an additional function, learned with the model (\"lexical embedder\") mapping the SEEs to input embeddings (called here \"lexical tokens\"), is designed. And as there is no finite vocabulary, the model cannot be trained with the cross-entropy loss commonly used. The authors resort to use a contrastive learning loss. Another modification is made on the output of the model: the authors used three separate output heads (fully connecter layers) to predict the next three tokens. This modification allows to \"extend\" the scope of prediction of the model, which would be fairly limited by autoregressively sampling one token representing a piece of a word after another.\nCompared to the [GSLM](https://aclanthology.org/2021.tacl-1.79/) baseline, the method proposed by the authors allows to significantly reduce the memory usage (by reducing the input sequence lengths) while having results of better quality (measured by perplexity and zero-shot metrics, along with human preferences) and interpretable embeddings.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "In your introduction, you mention \"tokenizer representing word or subword units\" (l60) without mentioning examples of such method. If think the comparison with your method is very relevant. It could then be nice (if you have enough space) to mention and cite the most popular methods currently used to build vocabularies for text, that the reader can relate to:\n\n* [BPE](https://www.derczynski.com/papers/archive/BPE_Gage.pdf) and [its application to natural language](https://aclanthology.org/P16-1162/)\n* [WordPiece](https://arxiv.org/abs/1609.08144v2)\n* [Unigram](https://aclanthology.org/P18-1007/)\n\nNCE loss introduction: [Noise-contrastive estimation: A new estimation principle for unnormalized statistical models](http://proceedings.mlr.press/v9/gutmann10a.html)", "Typos_Grammar_Style_and_Presentation_Improvements": "You should consider converting your figures to vector format, this will provide the reader a better quality of visualization. Assuming Fig. 3 and 4 are generated with pyplot, you just have to save them as pdf.\n\nRename \"lexical tokens\" and \"acoustic tokens\" (see reasons to reject);\n\nFig. 1: you could change the contour of the \"Lex Emb\" contours in order to better distinguish methods/functions from models, change \"Transformer\" to \"LM\" (more general), add the signification of \"FC\" somewhere in the main text;\n\nl244: \"there is often a linear FC layer before the transformer\" --> this is not really accurate, it isn't a fully-connected layer. You should rephrase with something like \"a learned embedding matrix acting as a lookup table\";\n\nl255: \"measured\" --> did you mean studied ?\n\nl267: precise acoustic tokens;\n\nl322: add signification of NCE + cite [Gutmann et al](http://proceedings.mlr.press/v9/gutmann10a.html);\n\nl322 footnote (3): It would be good to include in the main text that L2 reconstruction is applicable, but did not get you good results in practice and include them in appendix;\n\nl356 / subsection 3.2.1: You could mention that you are using FAISS (I assumed it was when looking at fig 2 but had to look for appendix to find the answer);\n\nl1080 footnote: \"those\" --> \"these\" + missing dot;\n\nl1355: add the signification of \"BPE\""}}, {"review_id": "KaRXerKTQV", "reviewer": "Reviewer_esLy", "summary": "", "questions": "A. Section 3.2.2: For my understanding, the generated acoustic token vectors are converted into Hubert before applying to Tacotron2.0 vocoder. I understand that the authors try to reuse the available vocoder for comparison. However, directly generating time domain signals from the 200ms acoustic tokens with techniques such as GAN should be feasible, and may achieve better results. Have the authors tried the experiments to generate speech directly from the 200ms tokens?\n\nB.Section 3.1.2: The quantisation parameter $q$ is pre-computed before training. How to determine the PCA size and the K-mean cluster size? Is it possible to replace this pre-computed parameter with other quantisation such as VQ such that this bottle net layer can be trained end-to-end?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer very much for their time and effort into reviewing our paper. Overall we agree that our implementation choices are debatable, and that a tGSLM version 2 should follow your two propositions. Here are the responses to your questions and comments.\n\n“For my understanding, the generated acoustic token vectors are converted into Hubert before applying to Tacotron2.0 vocoder. I understand that the authors try to reuse the available vocoder for comparison. However, directly generating time domain signals from the 200ms acoustic tokens with techniques such as GAN should be feasible, and may achieve better results. Have the authors tried the experiments to generate speech directly from the 200ms tokens?”\n\nThis is indeed the main weakness of our paper. The model would be much stronger with a specific decoder that would map acoustic tokens to speech. We are thinking to do it with a Hifi-GAN model that we would adapt to take acoustic tokens in input instead of the hubert units. \n\n“The quantisation parameter is pre-computed before training. How to determine the PCA size and the K-mean cluster size? Is it possible to replace this pre-computed parameter with other quantisation such as VQ such that this bottle net layer can be trained end-to-end?”\n\nIndeed the quantization is precomputed before training. In appendix section A.2.1 we give a little bit of details on the PCA and k-means parameters. Basically, we have gridsearched the PCA dimension and k-means clusters to maximize the zero-shot ABX metrics on a smaller dataset (the librispeech) than the one used for training (the LL6k clean). We agree that it would be a nicer solution to use learned VQ with a gumble softmax as the authors of Wav2vec2.0 did. We decided to go for PCA and kmeans to make the model easier and lighter to train (in wav2vec2.0 they had to incorporate a diversity loss with a weighting parameter to make the VQ quantisation work)\n\nThank you very much for all your comments that will make this paper much clearer and much easier to read.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The significant longer sequence length is a challenge of applying NLP sequence-to-sequence technique to speech. There are a lot of researches trying to shorten speech sequences for effective computation and better performance. This paper provides another perspective by proposing a method to generate longer fixed size acoustic tokens of 200ms which is much close to duration of sub-word units. The length the speech sequence is now comparable to the sequence with sub-word tokens, which has the potential for improving speech LLM training. While the choice of 200ms is somehow heuristic, the duration aligns well with average syllable length of most languages. The work includes experiments of different tasks and the results are encouraging.", "reasons_to_reject": "The performance of 200ms-tGSLM is not necessary better than conventional 40ms based GSLM, except the MMOS for generated speech. However, I do not think that this is  a strong reason to reject this paper.", "questions_to_authors": "A. Section 3.2.2: For my understanding, the generated acoustic token vectors are converted into Hubert before applying to Tacotron2.0 vocoder. I understand that the authors try to reuse the available vocoder for comparison. However, directly generating time domain signals from the 200ms acoustic tokens with techniques such as GAN should be feasible, and may achieve better results. Have the authors tried the experiments to generate speech directly from the 200ms tokens?\n\nB.Section 3.1.2: The quantisation parameter $q$ is pre-computed before training. How to determine the PCA size and the K-mean cluster size? Is it possible to replace this pre-computed parameter with other quantisation such as VQ such that this bottle net layer can be trained end-to-end?", "paper_topic_and_main_contributions": "The paper proposes a method to transform and concatenate frames extracted by wav2vec 2.0 into 200ms vector-based acoustic tokens. The vector-based acoustic tokens can be applied for training a 200ms-tGSLM speech based language model. Performance of this  200ms-tGSLM is comparable to conventional GSLM based on 240ms frames. The proposed method has the potential to improve speech LLM modelling, in terms of computation efficiency and representation ability.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gzQwTaaDAq", "reviewer": "Reviewer_BKbZ", "summary": "", "questions": "1. How do you get the corresponding Lex tokens for a corresponding 200ms window exactly? Do the sequence of lexical tokens and the sequence of acoustic tokens have the same lengths? \n2. Have you tried any other lengths rather than 200ms?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer very much for their time and effort into reviewing our paper. We understand that this paper is quite long and complex with many new components. Here are the answer to your questions.\n\n\"I was unclear about how the upsampling to HuBERT units was done\"\n\nIndeed our section 3.2.2 that explains the decoding to HuBERT units is not very clearly formulated. In this section, we wished to explain that the HuBERT units are obtained with a transformer encoder-decoder that takes acoustic tokens in input and outputs HuBERT units. In appendix A.3 we explain how we train this model on the librispeech corpus.\n\n\"How do you get the corresponding Lex tokens for a corresponding 200ms window exactly? Do the sequence of lexical tokens and the sequence of acoustic tokens have the same lengths?\"\n\nRegarding this, we will try to make section 3.1.1 and 3.1.2 clearer. These two sections explain how acoustic tokens are built and how each one of them is mapped to a single lexical token with the LexEmb function. This one to one mapping is maybe more visible on figure 1 and 2.\n\n\"Have you tried any other lengths rather than 200ms?\"    \n\nIndeed we have tried, an ablation on the duration of units is provided in appendix at table 4.\n\nAs show in this rebuttal, for each of your questions, there is a dedicated section that answer it. We wish that these answers are satisfactory to you and that you may consider giving this paper a higher grade. Thank you very much for all your comments that will make this paper much clearer and much easier to read.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The alignment between acoustic encoding and lexical encoding to help GSLM is innovative.\n2. Experiments are thorough and in detail.", "reasons_to_reject": "1. As the method involves a lot of specific knowledge and dedicated components, it is a bit hard to follow the description of the system. In particular, I was unclear about how the upsampling to HuBERT units was done. Also, see questions for the authors below.", "questions_to_authors": "1. How do you get the corresponding Lex tokens for a corresponding 200ms window exactly? Do the sequence of lexical tokens and the sequence of acoustic tokens have the same lengths? \n2. Have you tried any other lengths rather than 200ms?", "paper_topic_and_main_contributions": "This paper proposes tGSLM, which is a generative spoken language model with features aligned with lexicon encodings. The advantage of this method is that it operates at a scale similar to phonetic/lexical units so that it achieves high efficiency while maintaining the same performance as the fine-grained GSLM.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to replace 20ms or 40ms-long discrete units used for generative spoken language modelling with 200ms-long continuous word-sized audio tokens. The paper explains the architectural choices needed to achieve this goal and reports that the model's performance is on par with the standard discrete-unit-based language model while being more memory efficient.

The reviewers praised clear writing, good motivation, explanation of the architectural choices, and sound experimental design. The reviewers raised several questions, which the authors answered during the rebuttal. The only comment that needs to be addressed in the camera-ready version is empirical inference speed measures, such as sec of generated audio per sec of inference.
Decision: Accept-Main