wav2tok: Deep Sequence Tokenizer for Audio Retrieval
Authors: Adhiraj Banerjee
Vipul Arora
Venue: ICLR 2023 poster (2023)
Keywords: sequence representation learning
audio search
music retrieval

Abstract: Search over audio sequences is a fundamental problem. In this paper, we propose a method to extract concise discrete representations for audio that can be used for efficient retrieval. Our motivation comes from orthography which represents speech of a given language in a concise and distinct discrete form. The proposed method, wav2tok, learns such representations for any kind of audio, speech or non-speech, from pairs of similar audio. wav2tok compresses the query and target sequences into shorter sequences of tokens that are faster to match. The learning method makes use of CTC loss and expectation-maximization algorithm, which are generally used for supervised automatic speech recognition and for learning discrete latent variables, respectively. Experiments show the consistent performance of wav2tok across two audio retrieval tasks: music search (query by humming) and speech search via audio query, outperforming state-of-the-art baselines.

Published as a conference paper at ICLR 2023
WAV2TOK: DEEP SEQUENCE TOKENIZER FOR AUDIO RETRIEVAL
Adhiraj Banerjee, Vipul Arora
Department of Electrical Engineering Indian Institute of Technology Kanpur, India {achiraj, vipular) witk.ac.........................
ABSTRACT
Search over audio sequences is fundamental problem. In this paper, we propose a method to extract concise discrete representations for audio that can be used for ef- ficient retrieval. Our motivation comes from orthography which represents speech of a given language in concise and distinct discrete form. The proposed method, wav2tok, learns such representations for any kind of audio, speech or non-speech, from pairs of similar audio. wav2tok compresses the query and target sequences into shorter sequences of tokens that are faster to match. The learning method mamittiionmmmimizattiin algorithm, which are gen-
makes use of CTC loss and erally used for supervised automatic speech recognition and for learning discrete latent variables, respectively. Experiments show the consistent performance of wav2tok across two audio retrieval tasks: music search (query by humming) and speech search via audio query, outperforming state-of-the-ar baselines.
INTRODUCTION
Sequence Retrieval aims at retrieving sequences similar to a query sequence, with the constraint that an ordered aligmment exists between the query and the target sequence. In this paper, we address the following problem: Can we extract discrete tokens from any continuous signal for the purpose of retrieval of similar signals? This problem has deep connections with tasks such as child language acquisition, music cognition and learning languages without written forms. Some direct applications of the proposed task include speech search, where the order constituent units, such as phonemes, syllables words, remains same; and music search query by humming or query by example where the order constituent units, such as relative notes or phrases, remains same. Apart from audio, the problem extends tasks such handwritten word search and gesture search.
One can define similarity metrics over sequences using methods based on Dynamic Time Warp- ing (DTW) (Mäller (007). These methods inefficient if the sequences are continuous valued and have high sampling rates. Moreover, they depend on matching hand-made features, which are ineffective in the face of high variability of query sequences.
Problems such as spoken term detection involve detection of a query utterance a long speech audio. The search space is huge, and performing DTW based search of query takes long time |Rodriguez-Fuentess et al. 2014). A more efficient way of sequence retrieval is by mapping them to sequences discrete tokens. Automatic speech recognition (ASR) be employed for this purpose (Mamou et al., 2013). However, ASR training requires knowledge of basic units of transcription. The popularly used units are phonemes and graphemes. This method thus becomes language depen- dent. Non- longlinguistic sounds, such cough and sneeze, could be mapped to certain tokens defined for them. This approach could not used when precise tokens are not defined, e.g., music search. ]
In query by humming based music search, audio is mapped to discrete melody-related tokens, such as notes, and these token sequence are matched for search (Unal et al., 2008). However, several music traditions do not have precise transcription systems. There, one can tell if two pieces, or motifs, are similar but cannot precisely transcribe them to tokens. The embellishments used in music could be too dynamic to be transcribed precisely. Moreover, when musically untrained user sings query, s/he cannot hit the right notes matching the target song. So the matching could rely on several factors other than notes, such phonemes of lyries (Mesaros & Virtanen] 2010), onset times

---

Published as a conference paper at ICLR 2023
(thythmm) (Kosugi et al. 2000), and note transitions (Ranjan & Arora 2020). Hence, the tokens to be used may not be derived from notes alone.
In this way, each tokenizer for speech, music other signals, in general uses domain-specite hand-made tokens defined by a domain expert. In this paper, we propose a tokenizer to map audio sequences to sequences of discrete tokens with an aim of retrieval. The mapping is learned only from pairs of similar audio sequences. The tokens are not defined manually but correspond to sequette distinct semantic units learned from pairs of similar audio sequences. The method is general and
can be applied to signals other than audio. In this paper, we apply the proposed method to speech and music audio search, for the problems of spoken term detection and query by humming, respectively.
The proposed method, named wav2tok, encodes audio via BiLSTM (Schuster & Paliwal 1997) network. The encoder- -ennerreeeee representations are then mapped to discrete tokens via K K-means vector quantizer network. Each discrete token corresponds to a discrete representation in the vector quantizer's ss codebook which initialized and updated via offline K K-means clustering only.
wav2tok is trained with pairs of similar audio sequences in self-super fashion without any transcription using a novel training algorithm. For each pair, we average the srooder-generated rep- resentations, which map to the same token, by the K-means vector quantizer network to generate a prototype for that token. We then perform a contrastive learning task to increase the similarity between the generated prototype for particular token and the quantizer codebook discrete repre- sentation corresponding to the same token. We simultancously minimize the edit distance between the token sequences generated from each sequence in the pair via Connectionist temporal classifica- tion (CTC) (Graves et al. 2006) framework to constrain both sequences to get mapped to the same token sequence.
We compare wav2tok to state-of-the-art (SOTA) methods for discrete representation learning, such as wav2vee 2.0, and SOTA ASR models fine-tuned perform phonetic tokenization. We evaluate the generalization capability of the tokens generated by the models on search experiments, namely, guery-by-humming humming and spoken term detection. wav2tok outperforms the baselines in performance and uses much lesser trainable parameters, ensuring faster inference and deployment.
RELATED WORK
Sequence Labelling. With
tokens, various methods are popularly used for mapping sequences to tokens. In conventional methods, Hidden Markov Models (Rabiner & Juang 1986)
and Conditional Random Fields (Lafferty et al.) 200T) have been popularly used for sequence label- ing. These methods involve a significant amount of domain knowledge and many assumptions to make tractable models, which are avoided by End-to-End learning models such as Recurrent Neu- ral Networks (RNNs) using Connectionist Temporal Classification framework (Graves et al. 2006). Sequence labeling can be used for sequence retrieval by converting the sequences to tokens, which easy to search over. But this approach inevitably depends upon expert defined tokens.
Unsupervised Speech Representation Learning, Automatic Speech Recognition systems are pre- trained on large amounts of untranseribed speech data generate SOTA continuous representations which encode the slowly varying phoneme features in raw speech. The representations are then mapped to phoneme tokens via Connectionist Temporal Classification (CTC) (Graves et al. 2006) fine-tuning on a small amount of transcribed audio. Works like Contrastive Predictive Coding (CPC) (an den Oord al. 2018), Autoregressive Predictive Coding (APC) (Chung & Glass 2020), and wav2vec (Schneider et al. (2019) generate continuous representations with powerful autoregressive models araitrained predict future time-step representations. Further works started discretizing the continuous representations with VQ-VAE (van den Oord et al., 2017) to generate discrete representa- tions for speech.
Works like vq- wav2vec (Baevski et al. 2019) and vQ-APC (Chung et al. 2020) discretize the rep-
resentations and perform the same prediction tasks as in wav2vec (Schneider et al. 2019) and APC (Chung & Glass 2020) respectively but over discrete representations. In wa-wev,vee, the discrete representations are generated with either a K-Means Vector Quantizer (Baevski et al. 2019) or Gumbel-Son Softmmxx based Vector Quantizer (Baevski et al. 2019). The learned discrete representa- tions are used to pre-train a BERT (Devlin et al. 2018) to generate stronger continuous representa- tions much like BERT pre-training in Natural Language Processing. wav2vec 2.0 (Baevski et al.)
2

---

Published as a conference paper at ICLR 2023
2020) uses Gumble Softmax based Vector Quantizer (Baevski et al. 2019) to generate discrete representations. The training involves masking of spans of time steps and then predicting the correct discrete representations at each masked time step with transformer representation at that time step. In these methods, raw audio is discretized in a latent space to model all possible acoustic units than phonetic sub-phonetic units. The tokens generated by the vector quantizers aren't constrained to be interpretable and are initialized in large numbers (~ 102.4K codes). After pre-training, a subset of these codes or tokens are chosen more often by the vector quantizers and are considered to repre- sent acoustic units. CTC-based fine-tuning with transcription groups these discrete acoustic units to K distinct phonemes or linguistic units as present the transcriptions.
Works like HuBERT and wav2vee- Unnupee-ssee learn phonemic units directly. HuBERT (Hsu et al. 202T) pre-trains transformer network via BERT-like masked prediction task over noisy targets generated with clustering model trained offline. The targets may be generated with an ensemble of K-means clusterers with K = {100, 500) clusters on MFCC features or transformer representations. wav2vee- Sew2zee-Unsspervisee (Baevski et al.) 2021 learns phonetic tokens ad adversarially from phonemized unlabelled text data. discriminator identifies if the phoneme sequence generated by model is real or fake based phonemized unlabelled text.
All aforementioned approaches use powerful auto-regressive models pre- tre-trained on large amounts of unlabeled audio and fine-tuned on transcribed audio. Our learning approach can learn semantic tokens with small models while training pairwise on small amount of unlabelled audio data.
Audio Representations for Retrieval. Now Playing (Areas et al. 2017 and (Chang et al., 2020) use a Neural Network Fingerprinter (NNFP) module outputting representations which are efficient for search in query-by-example tasks where the difference between query and the actual song is pretty minute in comparison to humming where only the melody is sung. Now Playing (Arcas et al.
2017 trains representations by optimizing the triplet loss (Schroff al. 2015) and (Chang et al. 2020 trains representations by simulating the Maximum Inner Product Search (MIPS) on mini- batches of representations. For Query by Humming task, (Mostafa al.) 2016) and (Mostafa & Fung, (017) use deep learning models like DNNs and CNNs to generate representations which they map to MIDI-numbers or note tokens. Such works require note- transcrrddddd data to train the models. For Spoken Term Detection task, approaches like (Zhang & Glass) 2009), Grodriguez-Fuente
al.) 2014). (Lee et al. 2015), (Ram et al., 2018) convert audio to sequences of feature vectors and apply different variations of DTW based template matching detect query in long utterances of speech which is consumnn.....g
Cross Domain Alignment. Given pair of semantically similar inputs for training, tasks such as visual question answering (text and image) and machine translation (text) involve learning an alignment. The alignment here is not ordered and the inputs may be from different modalities. Attention models have been used to find aligmment between output entities and input regions (Yao) al. 2018). (Chen et 2020) use to match them. However, there is no notion of tokens there, rather the salient entities in the input are represented as vectors in graph.
distance between output and input entities
Graph Matching. Graph Neural Networks (Gori et al., 2005) are used to generate embeddings for graphs. These embeddings are used to perform graph matching to find similarity of structured graphs (Li et al.) 2019). However, they perform the matching jointly on the pair of inputs, rather than representing each input independently. This makes them unsuitable for the search problem at hand due to large run-time complexity. The distance metrics used for graph matching are based on edit distance (Li et al. 2019) and Wasserstein distance (Chen et al. 2020).
PROBLEM STATEMENT
We aim to map X' a sequence of vectors, to Ý, a sequence of discrete tokens from a finite al- phabet A, such that the similarity of sequences is preserved in the sense of edit distance. The length of sequence 7 may be less than or equal that of the sequence X. In other words, given a pair of similar sequences (x, and sequence X₁ which is not similar to either sequences in the pair, we want to map them to token sequences such that ED T5) should be less than (T, TR), (T, Te)), where :.)( is the edit distance between sequences.
3

---

Published as a conference paper at ICLR 2023
4 MODEL ARCHITECTURE
wav2tok is comprised of an encoder f >> Z which takes as input a temporal sequence of audio features X' xx E t [T]] of length T, where x, is the feature vector at time step f, and outputs a sequence of L-2 normalised representations Z = [zz f(x,) E R E [T]] The encoder is
implemented 2-layer BiLSTM followed by an L-2 normalization layer. BiLSTMs summarise information in both directions and encode surrounding context.
A K-means vector quantizer network g Z + T then labels sequence 2 each time-step with tokens belonging to a finite K-element alphabet A = [K] and generates sequence of tokens T T== g E A; E [T]]. Network g vector quantizes input z; with codebook = {(ex E 2; k e K discrete representations which are cluster centroids in representation space and outputs token Ti = arg max; 24 4.ee, ej. Note, here the dot product gives cosine similarity score since both the vectors are L-2 normalized,
[K]) comprised of |
E closest to Z, is chosen as its discrete representation and index as it's token Tr. The K discrete representations network g are trainable parameters.
result,
A compressor C compresses sequence of tokens T to sequence T of length T < T by deleting all consecutive repetitions of tokens. C also generates the corresponding compressed sequence 2 of length 7 by averaging representations É 2 over the consecutive tokens and I.-2 normalising the averaged representation. Figure presents an illustration demonstrating our model architecture.
TRAINING
wav2tok is trained on pairs of sequences of audio features (X' X') where the raw audio correspond- ing to X' is an augmented replica of that corresponding to X. We apply either pitch shift or time stretch or both augmentations to raw audio to generate its augmented replica. X' and X" may differ in sources well, i.e. different person may sing the recording corresponding to X".
The discrete representations in quantizer g codebook E are initialized K centroids obtained via of- fline K-means clustering over freshly initialized encoder-genen generated representations. Given (X, X"), encoder generates sequence of representations erates sequence of tokens T from input 2 and T from 2' via cosine similarity-based comparison with codebook vectors e E E initialized via offline clustering over freshly initialized representation space Z. Compressor C compresses sequence of tokens T to sequence T and T to T'.
from input X' and 2' from X'. Quantizer 9 gen-
We average all encoder-generated representations in pair (2, 2') which map to the same token, say т, to generate a prototype for т. We then perform a contrastive task where we compare the prototype with each of the K discrete representations in codebook E and increase its similarity with the discrete representation corresponding to т. We also increase the likelihood that wav2tok maps pair X') to the same token sequence via CTC framework to minimize ED T').
Our loss function is defined as,
L C= Lm (X, aLote(X (X', T)
(1)
where C., is loss defined for contrastive task, Lase is the loss maximising aforementioned likelihood, and a; 3 are positive constants. We optimize this loss function in manner similar the Expectation Maximization algorithm. The clustering is used as the E-step to update the discrete representations in quantizer y codebook, while gradient descent over acts the M-step.
Contrastive Loss. Let the set of unique tokens occurring in pair (T,T* be 4 c [ K' K. We generate a list of token prototypes P = {p,: E U)) where p, is L-2 normalised mean of representations in (z [Z; Z') g(() = 7). Figure Ib presents an illustration demonstrating how we generate list of token prototypes P.
Given p. E P. we perform a contrastive task to increase its similarity with discrete representation e, E F. To compare P, with the codebook, metrics such as cosine similarity and Fuclidean distance could be used. However, we find that using the following parameterized score for this purpose gives better performance,
$7,4 [W (p. sg(ex))) [0.1]
(2)
4

---

Published as a conference paper at ICLR 2023
24- (ra Pawna
Dezreerttabiea
X (ec...
Piu (K()
Compenaeer
Telurn Classlier
Soken Canstaar
Token Clamitter
Iiace Classhet (Q)
Theen Claeehee inl
21
8i
Hj
argenx
(a) Model Architecture
(b) Generation of Prototype list P
(c) Likelihood Loss Calculation
Figure 1: X' is an augmented replica of X. la illustrates our model architecture. lb demonstrates
the generation of P required for calculation of Lm- Ic demonstrates our likelihood loss calculation.
where sg(x) = x, dar (z) = 0 the stop-gradient operator, (.) is sigmoid function generating a score in the range [0, 1] and W E R xd is parameter matrix. Sr,k acts as parameterized similarity score between p and discrete representation 0ş E E. We define our contrastive loss C.m as,
log exp(sr.r)
(M, = Σ TEM
(3)
Σ k= exp(s,,,)
Likelihood Loss. We maximize the likelihood that sequence X' maps to token sequence T', which corresponds X', via the CTC framework (see Figure Ic]. puts a constraint to generate the same token sequence for X' and X". We calculate the probability of x, mapping to token T. = k as exp(f(s))=sg(e)) 4, Σ (xp(((x))-eg(e.)) The likelihood P(T is then calculated
of probabilities of
all T-length paths x over tokens such that ((T)) -T' The loss defined as,
Lac((X, =-log REC-1(T"
Σ
P(
(4)
where the path probabilities calculated over token probability scores in sequence 1 {4e framework (Graves et al. (2006) without the use of blanks.
RK ;6 E [T]} via CTC We present the CTC forward and backward variables for our case in Appendix B
Clustering. We perform offline K-means clustering on a subset of encoder representations during initialization of our network and regular intervals during training in codebook E of network g. Initializing the clusters in this way prevents wav2tok from converging local optimum during the matching task, as the case we found with random initialization of centroids. The intermittent clustering during training iteratively refines the discrete representations and prevents codebook collapse. We use the sklearn library to perform K K-means clustering.
the discrete representations
We train wav2tok using the ADAM (Kingma & Ba 2017) optimizer and a linear learning schedule with learning rate of 0.001 and 8% of the training steps as warm-up steps.
EXPERIMENTS
We test the performance of tokens and encoder-generated continuous representations wav2tok in audio retrieval. We perform Query by Humming (QbH) and Spoken Term Detection experiments to evaluate the performance of way2tok in comparison to the baselines.
6.1 MUSIC MELODY SEARCH: QUERY BY HUMMING
Task. Given test query audio, audio database.
find the audio with the most similar melody in the search
Experiment Details. We use the HIR-QbSH dataset which is composed of 4431 humming audio recordings 30s duration corresponding to 48 songs. Each song is sung by several individuals. All individuals sing the same part of the song. The recordings have variations in the environments
5

---

Published as a conference paper at ICLR 2023
they were recorded in, tonal qualities, voices, pitch, and time stretch. We train our models on hums of 40 songs in MIR-QbSH dataset and evaluate search performance on hums of the remaining 8 songs. The training dataset has 1970 hums for training and 676 for validation. The test dataset has 225 hums as a search database and 659 query hums. We evaluate the performance of our models in identifying which song a given query corresponds to via comparison with all sequences in the search database. Each model converts all the audio in our test dataset to sequences of tokens or representations. Each query sequence compared all sequences in the search database via Edit Distance (ED) (if tokens) or DTW (if representations). The song id of the most similar sequence in the search database then selected as query song id. We calculate Mean Reciprocal Ranking (MRR) score with ground-truth song id of the queries for evaluation. The Reciprocal Ranking (RR) score is given as 1/r if the ydt most similar sequence in search database has same song id as query.
All the audio recordings are converted to Short Term Fourier Transform (STFT) matrices before being passed as inputs to our models. The STFT matrices are computed with 513 frequency bins, a window length of 1024 samples (summarising 128 ms of audio), and hop length of 512 samples.
6.2 SPOKEN TERM DETECTION
Task. Given a test query audio, we are to detect its occurrence in a long utterance.
Experiment Details. We the TIMIT dataset which is composed of 6300 utterances of English speech with time-aligned word transcriptions. We choose 59 most-occurring words with more than 2 characters as keywords and all others as non-keywords. We use utterances of random sentences formed with 6 words sampled from a subset of 25 keywords for training and evaluation on STD experiments for the detection of the remaining 34 keywords. The test dataset is composed of 337 utterances corresponding to the 34 queries and 100 long utterances per query, with half containing single occurrence of query amongst non-keywords and the other half containing only non-keywords. Given query and a long utterance, we convert both to sequences of tokens using each audio tok- enizer. We perform approximate string matching (Hall & Dowling 1980) for detection of query in the utterance. The STFT matrix inputs to the models are computed with 185 frequency bins, window length of 368 samples (summarising 23 ms of audio), and a hop length of 92 samples.
6.3 BASELINES
Triplet. We train encoder f X + to generate L-2 normalized continuous representations for retrieval. Encoder ʃ is trained via optimizing the triplet Loss (Schroff et al. 2015) as done in training an NNFP in Now Playing (Areas et al. 2017). Given pair of similar sequences (X, X"), encoder f generates sequences Z and Z'. We form mini-batch of size N of triplets {z, zt where representation z is sampled from sequence Z x and z are positive and negative samples respectively for z sampled from sequence z'. The loss is defined as, L > max [| 1 1|2 - z, + m, 0}, where m is margin for similarity.
MIPS. We train encoder f : X + to generate L-2 normalized continuous representations for retrieval. Encoder f is trained via simulation of MIPS (Mussmann & Ermon] (0066) on mini-batches of representations as proposed by (Chang et al. 2020). Given pair of similar sequences (X', encoder f generates sequences 2 and 2'. We form mini- of size N of pairs of (z, z } where
encoder generated representation z sampled from sequence and z a positive for z sampled
from Z'. The loss defined as, LNIPS ΣN log Σ, (exp(ex z.) ++exp(z, 3)
exp(2,
2av2vec2. We train our audio tokenizer via wav2vec 2.0 Baevski al. 2020) learning framework. Quantizer g in our audio tokenizer is chosen to be a Gumbel Softmax based Vector Quantizer (See Appendix [ for details) as used in (Baevski et al., 2020 but with a single codebook with K mem- bers. Given sequence X, encoder f outputs sequence of L-2 normalised representations 2 of length T. Quantizer g outputs sequence of discrete representations Q = {g: = g(zz E Z); = I,.., T). We mask spans of 10 time steps with random starting indices in sequence Z and then pass the new sequence to a transformer network
Z + O which generates a sequence of contextualized rep- resentations O for h(2 Z):t 1,..., T). For transformer output o, over masked time step t, we identify the true discrete representation q, from a set D, composed of 4, and D dis- tractors which are discrete representations sampled from other time steps. The loss is defined as,

---

Published as a conference paper at ICLR 2023
(o, D,) log exp(sim(o
gem, exp(arm((r.0) La, where sim(a, b)
>²hh is cosine similarity and TollToo
La is codebook diversity loss.
wav2vec2P. We train wav2vec2 audio tokenizer with our variation of wav2vec 2.0 (Baevski et al,) 2020) learning framework which learns discrete representations from pairs of similar sequences. Given pair (X,X'), encoder f outputs sequences Z of length T and Z' of length T" respectively. Assuming T S T', we generate sequence 2 of length T whose time step element z," is a positive for 2, E 2 sampled from sequence Z'. Gumbel Softmax-based Vector Quantizer g quantizes each representation in sequence 2 to generate sequence 2+. We mask sequence Z and g+ at the same time steps. Transformer h inputs masked sequences and generate sequences O and O+, For masked time step t. we use transformer output or to identify 4 E Q+ from set D with distractors sampled from sequence Q+ and transformer output of to identify 9, E - from set DD with distractors sampled from sequence Q. The loss is defined as, LwP L (o,, Dt) (o," D.).
wav2vec2-O. The original wav2vec 2.0 base model with 12 Transformer blocks and 95M param- eters as proposed by (Baevski et al. 2020). It is pre-trained on 960 hours of LibriSpeech data and fine-tuned on TIMIT' dataset. It uses K 32 tokens for tokenization.
wav2vec2-Multi. A wav2vee 2.0 large model with 24 Transformer blocks and 317M parameters pre-trained on 53 languages as proposed by (Conneau et al.) 2020). It is fine-tuned on Common Voice to detect all possible phonemes in training languages with K = 392 tokens.
Triplet and MIPS use 2-layer BiLSTM as encoder with 3.6M parameters. We use the LAMB optimizer (You et al.) 2020) and a Cosine Annealing Learning Schedule (Loshehilov & Hutter, 2017) with learning rate restart of 0.0001 to train them. wav2vec2 and wav2vec2P use 2-layer BiLSTM encoder with 3.6M parameters to generate latent representations and 3 Transformer blocks with 8.5. parameters. Both linear learning schedule with a learning rate of 0.001 and 8% of the training steps as warm-up steps. Proposed wav2tok uses only a 2-layer BiLSTM encoder with 3.6M parameters.
trained using the ADAM (Kingma & Ba 2017) optimizer and
7 RESULTS
7.1 MUSIC MELODY SEARCH: QUERY BY HUMMING
We present search performances for 3 settings of query namely- Query with no augmentation or Vanilla Query (V), Time Stretched Query (TS), and Pitch Shifted Query (PS). Time stretch and pitch shift are the most common augmentations that may be faced in queries by humming data. No augmentations were applied to audio in search database. Evaluations are performed on sequences corresponding to songs not seen during training. The results present the generalizability of the tokens or representations generated by the models. We set the number of tokens as = 25 for wav2tok, wav2vec2, and way2vec2P (See Appendix A.2 for experiments to support our choices).
Quality of Tokenization. Table presents the performance of the sequence of tokens 7 generated by the audio tokenizers on ED-based similarity search. Tokens generated by wav2tok present good generalization capabilities in terms of MRR and outperform all the baselines. It generates time and pitch invariant tokens as we see no drop in performance when either augmentation is applied to query. wav2vec2-O is trained on English speech only. The tokens generated by it do not contain much melodic information but are robust to augmentations. The multilingual training of wav2vec2- Multi infuses both melodic and phonetic information to its 392 tokens, thereby giving good per- formance. way2tok outperforms both way2vec2-O and way2vec2-Multt given its pairwise training which allows it to infuse more melodic information to the tokens while also being trained on a small amount of unlabelled data. The Gumbel deftmax-based quantizer in wav2vec2 and way way2veC2P isn't ideal for infusing melodic information to tokens but it does infuse phonetic information as will be seen in Section 7.2 We compare the tokens with representations learned by MIPS and Triplet evaluated on based similarity search. The continuous representations present sub-par general- izations to unseen songs, We compare wav2tok with SOTA melody extraction algorithm proposed in (Salamon& & Gómez 2012) which converts hums to MIDI sequences. way2tok generates token sequences much smaller than the respective MIDI sequences and outperforms the MIDI tokens in search performance, search time, and robustness. In addition, way2tok outperforms the algorithm in inference time. We further compare wav2tok with SOTA QbH system proposed in (Mostata &
7

---

Published as a conference paper at ICLR 2023
Fung 2017). In implementation, map audio to MIDI sequences using the aforementioned SOTA melody extraction algorithm instead of CNN. Given MIDI sequence 53, 53, 58, 50 with du- rations Os, 0.5s, 1s, 2s, Relative Note sequence is generated as (0, 0), (0, 0.5), (5, 1), (-8, 2) over which DTW is performed for retrieval. way2tok tokens outperform the SOTA QbH system both performance and robustness; the performance of the latter drops drastically with time stretch.
We present the performances of the uncompressed sequences T and 2 and compressed sequence Z generated by the audio tokenizers in Appendix A We observe a drop in performance for all audio tokenizers when apply sequence compression sequences T and 2. wav2tok outperforms all the baselines and generates superior-quality of continuous representations and discrete tokens.
Search Time. Table [ presents the search time taken for similarity search over the tokens repre- sentations generated by the models. The search time taken per query is 2 order of magnitude lesser for ED-based Search over compressed sequence of tokens T than standard DTW-based Search over continuous representations 2. The pre-trained models being fine-tuned on transcribed audio give the best tokens in terms of compression and search time. wav2tok gives comparable tokens but outperforms the pre-trained models in inference time.
Table 1: Quality of Tokenization
Table Ablation Studies and Some Variations
V
PS
Search Time
Infer
Model
TS
PS
(MRR)
(MRR)
(MRR)
(8)
(6)
Model
(MRR)
(MRR)
(MRR)
MIDI
0.75
0.64
0.72
3.84
0.62
log-mel DTW
0.72
0.7
0.67
Relative Note DTW
0.84
0.74
0.8
0.02
0.62
vq-log-mel ED
0.71
0.6
0.62
MIPS DTW
Triplet DTW
0.6
0.48
0.55
0.5
0.58
3,5
0.1
way Nooii ED
0.73
0.73
0.72
wav2vec2 FD
0.66
0.63
Wav2tok+Cos ED
0.79
0.76
0.77
wav2vec22 ED
0.69
0.65
0.67
0.06
0.17
wav2tok=CTCC ED
0.64
0.62
0.63
wav2vec2-O
0.72
0.72
wav2tok=Newlnnd ED
0.77
0.76
0.78
wav2ver2 Multi ED
0.82
0.82
0.82
0.01
1.2
way2tok+MIRIIX ED
0.72
0.64
0.67
wav2tok ED
0.84
wav2tok ED
0.84
0.84
0.84
Ablation Studies. Query by humming involves similarity based on melody information, which is carried by the semantic pairing of the audio in training data. We constrain this pairing include quences not semantically similar and call this model Noootok+NoSim. We optimize the contrastive loss L.m. to train the model. The results are shown in Table (full table in Appendix A.3). There is significant drop in token robustness and performance but the representations suffer small drop (see Appendix A. A..3.. Hence, although the representation space may be well clustered, wav2tok is able to add more semantics to the tokens as it is being trained with pairs of similar sequences in comparison to wav2tok=NoSim. We train wav2tok with cosine similarity scores instead of parameterized score (way2tok+Cos). The drop in performance validates the enhancement brought about by using a parameterized score. We also train wav2tok with Lete only (way2tok+CTCC, The CTC loss con- siders all possible paths which compress to the target label sequence. As result, the learnt tokens aren't 't much semantic. The use of both losses gives the best tokens.
associate the discrete representations with K centroids in the input space X. Such association does not initialize our tokenizer with optimal centroids which cluster the space Z perfectly. This results in significant drop in performance and robustness as shown in Table 2 We train way2tok on MIR-1K dataset (way2tok+MIR11K) which is composed of polyphonic music recordings of 1000 distinct songs. The tokens generalize well to monophonic hums in MIR- HIR-QbSH dataset giving comparable performance to MIDI tokens. This validates that wav2tok tokens do learn melodic information and robust to variations incurred in hums. We fur- ther compare wav2tok with log-mel features and token sequences (with no compression) obtained via quantization of log-mel features. wav2tok tokens outperform both.
Some Variations. In way2tok+Newlnit
7.2 SPOKEN TERM DETECTION
Quality of Tokenization. Table 3 presents the quality of tokenization of the query keywords by the models evaluated in the Spoken Term Detection experiments. We present the performances of wav2vec2, wav2vec2P, way2vec2-O, wav2vec2-Multi and proposed wav2tok. We conduct search experiments on a test dataset composed of a search database of 337 utterances of the 34 keywords used as queries in the STD experiments and 1289 query utterances. We identify the keyword to
8

---

Published as a conference paper at ICLR 2023
which each query corresponds to via comparison all the 337 utterances in the search database via ED-based similarity score. The word of the most similar utterance is selected as the word to which the query corresponds to. We set K 40 equivalent to the number of phonemes in English.
wav2tok gives the best performance in terms of MRR scores. outperforms huge models like wav2vec2-O and mav2vec2-Mult which are fine-tuned for the task of phonetic tokenization of speech audio while using a small number of parameters. wav2vec2 and wav2vec2P also outperform war2vec2-Multi and wav2vec2-O while using smaller number of parameters. wav2vec2-O and wav2vec2-Mult use a blank token to handle consecutive occurrences of the same tokens and to label background noise. The utterances of each keyword in the test dataset are very small in time duration. This causes way2vec2-O) to confuse word utterances as background noise. It generates a sequence of blank tokens and performs poorly in search. wav2vec2-Mull using a larger number of phonetic tokens does not suffer this issue. wav2tok, wav2vec2, and wav2vec2P have no such blank token. This brings drop in search performance with sequence compression. We further present the performance of wav2tok trained on a much larger LibriSpeech 100 hours dataset (war2iok+iibri). is able to outperform wav2vec2-O and give comparable performance to wav2vec2-Multi
Table 3: Quality of Tokenization for speech
Table 4: Spoken Term Detection
Model
Normal (MRR)
Compressed (MRR)
Model
ED
Search Time
DTW
Time
Search
log-mel DDTW
0.7
(FI)
(8)
(FI)
wav2vee2 ED
0.68
0.63
log-mel DTW
0.41
0.003
wav2vec2
0.64
0.46
wav2vec2P ED
0.7
0.65
0,066
0.1
wav2vec2P
0.64
0.47
wav2vec2-O ED
0.4
0.4
wav2vec2-0 O
0.61
0.29
0.43
0.23
wav2vec2-Multi ED
0.67
0.67
way2vec2-Mult
0.63
0.72
0.48
0.66
wav2tok ED
0.74
0.66
wav2tok
0.65
0.52
0.09
0.064
wav2tok-Libri ED
0.64
0.6
way2tok=Libri
0.63
0.44
0.1
Spoken Term Detection. We convert the query word utterance and the long utterance in to se- quences of tokens by all our models and detect the occurrence of the query via approximate string matching. We use fuzzysearch library to perform approximate string matching. It automati- cally chooses the fastest algorithm for matching. Table 4 presents the performance of wav2vec2 wav2vec2P, wav2vec2-O, way2vec2- comparable performance in terms of F1- score with wav2tok performing slightly better. We also im- plement the STD system proposed in (Anguera & Ferrarons) 2013) which performs highly competi- tive STD via subsequence DTW (S-DTW) over gaussian posterior features. In our implementation, we extract the posterior features with SOTA ASR models like way2vec2-O and way2vec2-Mults The results are presented the DTW column in Table 4 Note, the results for other models in same column are for STD via S-DTW over representations. We observe STD over tokens to give better Fl-score.
and proposed wav2tok in STD. All the models give
CONCLUSION AND FUTURE WORK
In this paper, we present an audio sequence tokenizer wav2tok that generates semantically mean- ingful ordered representations (or tokens) that can be used for efficient retrieval by query sequences. The model learns only from pairs of semantically similar sequences and outperforms state-of-the- art approaches for spoken term detection and query by humming. One may apply more efficient search algorithms such as locality-sensitive hashing and longest common subsequence search on the generated tokens to further speed up the search. The proposed framework can also be extended to image and video retrieval, as they also have spatial ordering. We would like to investigate the Nomain-specific, i.e., linguistic musicological, aspects of the extracted tokens. For instance, dur- ing retrieval, the matching algorithm assumes all the tokens to be equidistant from each other. One may study or use the metric space of these tokens.

---

Published as a conference paper at ICLR 2023
9 REPRODUCIBILITY
The codes are available in httpa///aaa////////////////////a performed using standard datasets.
The experiments
REFERENCES
Xavier Anguera and Miquel Ferrarons. Memory efficient subsequence dtw for query-by-exampe spoken term detection. In 2013 IEEE International Conference on Multimedia and Expo (ICME), pp. 1-6, 2013. doi: 10.1 1000//CCEE.21336077555
Blaise Agüera Arcas, Beat Gfeller, Ruiqi Guo, Kevin Kilgour, Sanjiv Kumar, James Lyon, Julian Odell, Marvin Ritter, Dominik Roblek, Matthew Sharifi, and Mihajlo Velimirovic. Now play- ing: Continuous low-power music recognition. CoRR, abs/1711. 2017. URL http:
//axxiv.org/abs//100 10958
Alexei Baevski, Steffen Schneider, and Michael Auli. vq-wav2vec: Self-supervised learning of discrete speech representations. CoRR, abs/1 Abs/1990,00543,, 2019. URL http://arxiv.org/
abs/1910.
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. CoRR, abs/2006. 1177, 2020. URL
abs https://arxiv.. 2000 2000. 11477
Alexei Baevski, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. Unsupervised speech recogni- tion. CoRR, abs/2105. 1084, 2021. URL http::: httpp::://rrixvvv wwt.aaawwww.aiaaaaeeeaaae wttp://///aa....o///oooo
Sungkyun Chang, Donmoon Lee, Jeongsoo Park, Hyungui Lim, Kyogu Lee, Karam Ko, and Yoon- chang Han. Neural audio fingerprint for high-specific audio retrieval based on contrastive learn- ing. CoRR, abs/2010.11910 2020. URL
org/ 2ttp:///1...119991
Liqun Chen, Zhe Gan, Yu Cheng, Linjie Li. Lawrence Carin, and Jingjing Liu. Graph optimal transport for cross-domain alignment. 37th International Conference on Machine Learning, ICML 2020, PartF16811 15:111200110 1530-1531, 2020.
Yu-An Chung and James Glass. Generative pre-training for speech with autoregressive predictive coding. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3497-3501, 2020. doi: 10.1109/CCASSP40776 A00.AASSPP477762200999554333 2020.9054488..
Yu-An Chung. Hao Tang, and James Glass. Vector fector-quantized autoregressive predictive preprint arXiv:2005.088: 2020.
prediict coding, arXiv
Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. Un- supervised cross-lingual representation learning for speech recognition. CoRR, 1bs/2906.13399, 2020.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/18 110.0400,, 2018. URL
http://arxiv:: www.ooggg.bb...o.e.... abs/ 1011110000 04805
John S Garofolo, Lori F Lamel, William M Fisher, Jonathan G Fiscus, and David S Pallett. Darpa timit acoustic-phonetic continous speech corpus ed-rom. nist speech disc 1-1.1. NASA STI/Recon technical report n, 93:27403, 1993.
Marco Gori, Gabriele Monfardini, and Franco Scarselli. A new model for learning in graph domains. In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, pp. 729-734. IEEE, 2005.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tempo- ral classification: Labelling unsegmented sequence data with recurrent neural networks. In Pro- ceedings of the 23rd International Conference on Machine Learning, ICML '06, pp. 369-376, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933832. doi: 10.1145/1 143844. 143891. hwRl.ttts::.. /ooi 115.://.....1111444...1111
10

---

Published as a conference paper at ICLR 2023
Patrick AV Hall and Geoff R Dowling. Approximate string matching. ACM computing surveys (CSUR), 12(4):381-400 1980.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked
prediction of hidden units. CoRR, abs/2106.07447, 2021. URL htttt::ttaaaatt org/ abs/ 2106.07447
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Naoko Kosugi, Yuichi Nishihara, Tetsuo Sakata, Masashi Yamamuro, and Kazuhiko Kushima. A practical query-by-hii bommming system for a large music database. In Proceedings of the EighthACM International Conference on Multimedia, MULTIMEDIA '00, pp. 333-342, New York, NY, USA, 2000. Association for Computing Machinery. ISBN 1581 131984. doi: 10. 10.1145/3545843355222 354520.
URL https: /oi org 1tt.:/00.111555551
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings the the Fighteenth International Conference on Machine Learning, ICMI. '01, pp. 282-289, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558607781.
Lin-shan Lee, James Glass, Hung-yi Lee, and Chun-an Chan. Spoken content Srrieval-beyonee cascading speech recognition with text retrieval. IEEE/ACM Transactions on Audio, Speech, and Language Processing,
2015. doi: 10.1 (0.100/AASLLP221552555555
Yujia Li, Chenjie Gu, Thomas Dullien, Oriol Vinyals, and Pushmeet Kohli. Graph matching net- works for learning the similarity of graph structured objects. In International conference on machine learning. pp. 1835-3845, PMLR, 2019.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, 2017.
Jonathan Mamou, Jia Cui, Xiaodong Cui, Mark F. Gales, Brian Kingsbury, Kate Knill, Lidia Mangu, David Nolden, Michael Picheny, Bhuvana Ramabhadran, Ralf Schlüter, Abhinav Sethy, and Philip C. Woodland. System combination and score normalization for spoken term detection. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 8272- 8276, 2013. doi: 10. N0.1109/CCASSP2211..669999
Annamaria Mesaros and Tuomas Virtanen. Recognition of phonemes and words in singing. In 2010 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 2146- 2149, 2010. doi: 10. 10.1099CAASP2200055555555
Naziba Mostafa and Pascale Fung. A note based query by humming system using convolutional neural network. In INTERSPEECH, pp. 3102-3106, 2017.
Naziba Mostafa, Yan Wan, Unnayan Amitabh, and Pascale Fung. A machine learning based music retrieval and recommendation system. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC' 16), pp. 1970-1977, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA). URL
org/
L16 1161-1
Meinard Müller. Dynamic time warping. Information retrieval for music and motion, pp. 69-84, 2007.
Stephen Mussmann and Stefano Ermon. Learning and inference via maximum inner product search. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learn- ing Research, pp. 2587-2596. New York, New York, USA, 20-22 Jun 2016. PMLR. URL ht.tps: htttps::/pproeeedinng m p.rlrsprsss.... v4 8 / mmmassmann 6. html
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 2206-5210. IEFE, 2015.
11

---

Published as a conference paper at ICLR 2023
L. Rabiner and B. Juang. An introduction hidden markov models. IEEE ASSP Magazine, 3(1): 4-16, 1986. doi: 10.1099/AASSP..88.1165333
Dhananjay Ram, Afsaneh Asaei, and Hervé Bourlard. Sparse subspace modeling for query by example spoken term detection. IEEE ACM Trans. Audio Speech Lang. Process., 26(6):1126 1139, 2018.
http://.......111111////a/aaaaaa 2018 2815780
Shivangi Ranjan and Vipul Arora. A bioinformatic method of semi semi-global aligmment for query-by- humming. In 2020 IEEE 4th Conference on Information Communication Technology (CICT), pp- 1-5, 2020. doi: 10. 109///// (ONCIIITTT604202220 604.2020.99 2085.
Luis J. Rodriguez-Fuuntte, Amparo Varona, Mikel Penagarikano, Germán Bordel, and Mireia Diez. high-pperformanc query-by-example spoken term detection the sws 2013 evaluation. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7819- 7823, 2014. doi: 10.1099/CAASP2221446555111
Justin Salamon and Emilia Gómez. Melody extraction from polyphonic music signals using pitch contour characteristics. IEEE transactions on audio, speech, and language processing, 20(6): 1759-1770, 2012.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vee: Unsupervised pre-training for speech recognition. CoRR, abs/1904.05862, 2019. URL http://arxiv.
org/abs/1 9904. 05862
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. CoRR, 0oss/1503.0355 2015. URL http::/tpx/xxii. wwt.p/ss/rg///////
1550300833
M. Schuster and K.K. Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673-2681 1997. doi: 10.1109/78.650993
Erdem Unal, Elaine Chew, Panayiotis G. Georgiou, and Shrikanth S. Narayanan. Challenging un- certainty in query by humming systems: A fingerprinting approach. IEEE Transactions on Audio, Speech, and Language Processing, 16(2):359-377) 2008. doi: 10.1109/TASLL.2077911333
Aäron den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation leamn ing. CoRR, abs/1711.00937, 2017. URL http://awwxx...rr//////11111.100
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. CoRR, abs/1801 0tss/887.000 007.03748, 2018. ht absgg//bs/1100 03748
Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Exploring visual relationship for image captioning. In Proceedings of the European conference on computer vision (ECCV), pp. 684-699 2018.
Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley J. Osher, Yingyong Qi, and Jack Xin. Understanding straight-through estimator training activation quantized neural nets. CoRR, Abs/1903.00662, 2019. URL httt::///aaii wwtp://gggg.aa..a.a//ee// 05662
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training bert in 76 minutes, 2020.
Yaodong Zhang and James R. Glass. Unsupervised spoken keyword spotting via segmental dtw on gaussian posteriorgrams. 2009 IEEE Workshop on Automatic Speech Recognition Under- standing, pp. 398 403, 2009. doi: 10. 10.109/ASRU.2299.537299
12

---

Published as a conference paper at ICLR 2023
A FURTHER STUDIES
A. SEQUENCE COMPRESSION
We present the quality of sequence of tokens T and sequence of representations 2 and their cor- responding compressed versions sequences T and 2 generated by the audio tokenizers in Table 5
wav2tok outperformed the baselines and generated the best quality of sequences T, 2, T and 2. Sequence compression brings an order of magnitude drop search time for all the audio tokenizers with a trade-off in search performance. Compression from T to T increases the robustness of the token sequences generated by wav2tok to various augmentations. wav2vec2P learnt better tokens and representations than wav2vec2 because of it's pairwise training on similar audio.
Table 5: Compression of Sequences: MRR scores for query by humming, K 25
Model
TS
PS
Search Time
Without Compression
wav2vec2 DTW
0.85
0.84
0.84
wav2vec2P DTW
0.87
0.85
0.87
3.5s
wav2tok DTW
0.92
0.89
0.93
wav2vec2 ED
0.72
0.69
0.73
0.68s
Wav2vec2P ED
0.75
0.71
0.73
wav2tok ED
0.9
0.84
0.9
0.32s
With Compression
wav2vee2 DTW
0.76
0.72
0.74
0.8s
Wav2vec2P DTW
0.81
0.77
0.79
wav2tok DTW
0.88
0.88
0.87
0.6s
wav2vec2 ED
0.66
0.63
0.64
0.06s
wav2vec2P ED
0.69
0.65
0.67
wav2tok ED
0.84
0.84
0.84
0.04s
A.2 VARIATION IN NUMBER OF TOKENS K
The effect of varying the size of alphabet and proposed way2tok with alphabets of size K E {15, 25, 40 Out of the three settings for K, K 25 gives the best performance for all models. wav2tok gives best performance for all settings of K.
shown in Table 6 We train wav2vec2, way2vec2P,
A.3 ABLATION STUDIES AND SOME VARIATIONS
We present the full version of Table in table 7 Note wav2tok+NoSim repsentations are well clustered. wav2tok+Trans representations are also comparable with wav2tok but the tokens are of lesser quality. This is due to model overfitting.
A.4 QUALITY OF REPRESENTATIONS
We present the performance of the continuous represennations generated by wav2tok and the base- lines Table wav2tok generates the best representations for music outperforming representations generated by the large wav2vee 2.0 models. wav2tok trained MIRIK generates representations outperforming domain-specin QbH baselines. Note, way2vec2-O outperforms wav2vec2-Mo as the hums in the dataset were all in english. wav2vec2-O is pre-trained and fine-tuned on English speech only while way2vec2 -uuttii is pre-trained multilingually. Hence, , way2vec2-O gave better results.
13

---

Published as a conference paper at ICLR 2023
Table 6: Effect of varying K: MRR scores for query by humming
Without Compression
With Compression
Models
V
TS
PS
V
TS
PS
15
wav2vec2 DTW
0.85
0.83
0.84
0.7
0.66
0.67
wav2vec2P DTW
0.87
0.85
0.85
0.82
0.77
0.8
wav2tok DTW
0.88
0.87
0.88
0.84
0.8
0.83
wav2vec2 ED
0.79
0.77
0.78
0.58
0.56
0.57
wav2vec2P ED
0.8
0.77
0.79
0.71
0.68
0.7
wav2tok ED
0.82
0.8
0.81
0.77
0.75
0.76
25
wav2vec2 DTW
0.85
0.84
0.84
0.76
0.72
0.74
wav2vec2P DTW
0.87
0.85
0.87
0,81
0.77
0.79
wav2tok DTW
0.92
0.89
0.93
0.88
0.88
0.87
wav2vec2 ED
0.72
0.69
0.73
0.66
0.63
0.64
wav2vec2P ED
0.75
0.71
0.73
0.69
0.65
0.67
wav2tok ED
0.9
0.84
0.9
0.84
0.84
0.84
K
40
wav2vec2 DTW
0.84
0.82
0.83
0.72
0.68
0.7
wav2vec2P DTW
0.86
0.85
0.85
0.81
0.77
0.79
wav2tok DTW
0.9
0.88
0.89
0.86
0.83
0.83
wav2vec2 ED
0.71
0.66
0.69
0.6
0.58
0.58
wav2vee2P ED
0.73
0.7
0.73
0.68
0.65
0.67
wav2tok ED
0.83
0.8
0.82
0.77
0.75
0.76
Table 7: Ablation Studies and Some Variations: MRR scores for query by humming
Without Compression
With Compression
Models
TS
PS
V
TS
PS
log-mel DTW
0.72
0.69
0.67
0.54
0.47
0.43
wav wav2tok+NoSim DTW
0.88
0.87
0.87
0.8
0.84
0.83
wav2tok+Cos DTW
0.88
0.87
0.87
0.83
0.81
0.81
way2tok+Newlng Init DTW
0.9
0.84
0.91
0.84
0.85
0.83
wav2tok+Trans DTW
0.84
0.77
0.85
0.8
0.77
0.76
wav2tok+MIR1 1K DTW
0.88
0.84
0.85
0.82
0.74
0.78
wav2tok DTW
0.92
0.89
0.93
0.88
0.88
0.87
vq- log-eel ED
0.71
0.6
0.62
0.52
0.48
0.47
wav wav2tok+NoSim ED
0.85
0.74
0.84
0.73
0.73
0.72
wav2tok+Cos ED
0.86
0.84
0.85
0.79
0.76
0.77
wav wav2tok+NewIng Init ED
0.83
0.72
0.85
0.77
0.76
0.78
Tav2tok+Trans ED
0.84
0.77
0.85
0.7
0.66
0.67
wav2tok+MIR 1K ED
0.76
0.66
0.71
0.72
0.64
0.67
wav2tok ED
0.9
0.84
0.9
0.84
0.84
0.84
A.5 TRAINING ON LARGER SPEECH DATASET
We train wav2tok on 100-hours subset of LibriSpeech (Panayotov et al., 2015) dataset. We evaluate the quality of tokenization of word utterances done by wav2tok TIMIT (Garofolo et al., 1993) dataset. We use 2-layer BiLSTM network with 3.6 million parameters as encoder network which takes MFCC feature sequences input. We perform tokenization with K 40 tokens.
wav2tok outperforms wav2vec2-O by a large margin and gives comparable performance to wav2vec2-Multt in terms of MRR score. wav2tok uses minute number of parameters in compari- son to 95 million parameters in wav2vec2-O and 317 million parameters in wav2vec2-Mulle Note, wav2vec2-O and wav2vec2-Mult were pre-trained large amount of unlabelled speech data and
14

---

Published as a conference paper at ICLR 2023
Table 8: Quality of Representations: MRR scores for query by humming
Model
TS
PS
(Salamon & Gómez 2012) MIDI ED
0.75
0.64
0.72
(Mostata Fung
2017) Note DTW
0.84
0.74
0.8
Triplet DTW
0.5
0.48
0.5
MIPS DTW
0.6
0.55
0.58
wav2vec2-O DTW
0.91
0.83
0.86
wav2vec2-Mun DTW
0.88
0.83
0.85
wav2tok DTW
0.92
0.9
0.93
wav2tok+MIRIK DTW
0.88
0.84
0.85
fine-tuned with transcription to perform tokenization of audio. Moreover, wav2vec2-O was fine- tuned to perform tokenization on TIMIT (Garofolo et al.) 1993) dataset. Proposed way2tok was trained on 100 hours of LibriSpeech dataset only. The tokens learnt by way2tok on LibriSpeech (Panayotov et al. 2013) dataset generalised well to TIMIT (Garofolo et al. 1993).
Table 9: Quality of Tokenization for speech (MRR Scores)
Model
Normal (7)
Compressed (7)
wav wav2vec2-00 ED
0.4
0,4
wav2vec2- Multi ED
0.67
0.67
wav2tok+Libri ED
0.64
0.6
B CTC WITHOUT BLANKS
We present the forward and backward variables used in calculating the gradients of the CTC loss Lote (X, T') with no blank tokens.
The forward variable is defined as
ar(s)=
Σ
II 1ive
(5)
where æ corresponds to all T-lenghh paths over tokens such that C(T) = 5¹. Here, C a compressor which compresses T T-length sequence of tokens via de-duplication.
We initialise as follows,
a1 (1) '1,7 a1(s) = 0, Vo >
(6)
and recursively calculate (s) as,
ar(s) = (a.-((s) + a.-1(s -1))
(7)
We set (s) = 0, Vs 1. The backward variable is defined as,
Be(s) =
Σ
II
(8)
x:C(Fer)
+ Tx
#==
15

---

Published as a conference paper at ICLR 2023
We initialise follows,
3r((T') 1.,/111 Br(s) 0, Vs 1T'
and recursively calculate (s) as,
Be(s) = (Bt+1(s) + Bet1(s + +1))
(10)
We set (e(6) = 0, 1
C GUMBEL SOFTMAX BASED VECTOR QUANTIZER
The Gumbel Softmax based Vector Quantizer (Baevski et al. 2019) quantizes input latent represen- tation 2ạ R with C' codebooks containing K quantizers
R x each. For our experiments, we set C = and K E {15, 25, 40). Given Zz, one of the K quantizers from each of the C' code- books are chosen resulting vectors 61, cc. The codebook vectors are then concatenated and linearly transformed from Rm to Rd to output discrete representation q: E Rd
z: is mapped to RC*K logits to give probability scores for the choice of codeword. The proba- bility Pe,k of choosing hthh quantizer in t codebook is given as,
exp (le, TT)/T
Pe.k
(11)
Σ exp (le, + n2)/T
where T is a non-negative temperature, -log(-log(u)) and samples from the uniform distribution Unif(0, 1).
During forward pass, the codeword is chosen arg max, Pa.j- During backward pass, the loss calculated over the gumble softmax distribution We use the straight-through gradient estimator
Yin et al. 2019) estimate the gradient.
Codebook Diversity Loss L.- This loss promotes equal use of all the entries each of the C codebooks. Minimization of this loss maximizes the entropy of the averaged softmax distribution over the K entries for each codebook P.. across a batch of utterances.
1 CK
L.
Park log Pe,k
(12)
k=
16


Reviews:
[{"review_id": "zCgIblMcTcA", "reviewer": "Reviewer_U7Da", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes a deep neural network method for discrete tokenization of audio signals for retrieval problems. The model is trained from pairs of audio that correspond to each other. The tokenization has two losses, one is similar to a contrastive version of a VQ-VAE loss and the other is a loss that encourages correspondence of representation between two pairs of audio. This second loss uses CTC which correlates with a low edit distance between pair token sequences.\n\nThe method is applied to query by humming and spoken term detection tasks and it is shown to perform better than other alternatives including using large self-supervised representations (wav2vec 2.0) etc.", "summary_of_the_review": "The method seems novel and seems worth publishing.\n\nSome specific comments are given below:\n\n1. Is the contrastive loss in (3) back-propagated to both p and e?  In VQ-VAE, it is not done to avoid collapse. But here, due to the contrastive nature of the loss, we do not have to worry about that? Some explanations would be nice.\n2. In Table 1, what are the columns V, TS and PS? Validation, test and something else?\n3. In Table 2, there is no mention of Wav2Tok+Cos which is mentioned in the text. Is it equivalent to Wav2Tok+NoRel in the table which is not mentioned in the text.\n4. Do the losses apply to both of the inputs in a pair? Maybe this can be stated more clearly.\n5. If the token sequence is longer than the number of frames in the other audio, how does the CTC loss work? Typically, CTC loss assumes the label sequence is shorter than the number of frames and uses blank symbols to fill in the middle.\n", "strength_and_weaknesses": "Strengths:\n1. A novel method to extract token sequences from audio signals, trainable with pairs of audio signals that should correspond to each other.\n2. Comparison of the method with competitive methods on two different audio retrieval tasks.\n\nWeaknesses:\n1. Uses BiLSTM models and shows that it is better than using a transformer architecture. For audio, U-net style models are also used in token generation which could be compared.\n2. Similarity and difference between the first part of the loss and VQ-VAE loss would be a nice addition to the paper. \n3. Details about how the CTC loss is back-propagated to the encoders is not clear. Does the CTC model use blanks? What happens when the label sequence is longer than the number of frames?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iawtKvJF37c", "reviewer": "Reviewer_LKSS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes a system that learns to encode audio files into sequences of discrete tokens for the purposes of performing sequence matching to a database of such sequences. The approach is evaluated in terms of matching query by humming queries to one another on a dataset of 48 songs each hummed about 10 times as well as a word search experiment with 34 keywords spoken in 337 utterances queried by 1289 queries. In comparison with a number of neural baselines, the proposed approach provided more accurate matches.", "summary_of_the_review": "Overall, this approach is not especially well motivated, but does appear to work better than several baselines (although these are perhaps not the best baselines) on two experiments involving rather small datasets. Several important details of the approach are not entirely clear.", "strength_and_weaknesses": "Strengths: \n* The proposed approach seems to work well on the proposed tasks and datasets\n* The idea of using the CTC from one sequence with the labels of a different sequence (and vice versa) is clever and interesting.\n\nWeaknesses:\n* Scalability. Fundamentally, the proposed approach takes time linear in the size of the database being searched to perform a single query. In general, it is difficult to achieve sub-linear matching speed in the size of a database for sequence matching.\n* Need for sequence matching. Given questions about the speed of sequence matching, I wonder about the need for it. A clear comparison with a system that does not require sequence matching demonstrating that the sequence-based approach is superior would resolve this. For example, the proposed system could be compared with a similar system that does matching based on a fixed length representation. \n* Need for discrete tokens. It is also not clear why discrete tokens are necessary. wav2vec 2.0, which uses discrete tokens has evolved into data2vec, which uses continuous hidden representations as targets of prediction. The reason that discrete tokens are preferable to a continuous representation in this case are not clear. \n* Summarization of tokens. The idea of the replacement of a sequence of the same token with a single instance of it was introduced, but it is not clear why it is necessary. It is not shown in figure 1 and page 8 states \"We observe a drop in performance for all audio tokenizers when we apply sequence compression to sequences T and Z.\" So is it used? Is it necessary?\n* Baselines could be better. In addition to the fixed-length encoders above, it would be good to compare with DTW on mel spectrograms, and perhaps one with edit distance-based matching on k-means quantized mel spectrograms.\n* No citations are provided in the introduction. While many citations are provided later in the paper, they would be very helpful to the reader in the introduction to support various claims.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aA2nIYs16RK", "reviewer": "Reviewer_PPxN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors address the problem of tokenizing audio data for improving the audio based content retrieval. They propose yet another approach for audio tokenization based on clustering the representation in Vector-Quantized space and forcing the representations to get closer their nearest centroid. The authors evaluate on smaller music and speech datasets for the query by humming and spoken term detection tasks and show better results compared to some of the existing approaches. ", "summary_of_the_review": "Overall, I feel this is yet another solution proposed for audio tokenization without clear articulation of what problems in the previous approaches are being solved. While this in itself is fine, the paper also does not contain any thorough justification for this design and seems like author's may have a good intuition about this problem which may have gone into the design but as a reviewer some aspects of the design are not at all clear. ", "strength_and_weaknesses": "Strengths:\n1. The paper is easy to understand and the proposed approach seems to be interesting\n\nWeakness\n1. There is no clear explanation of why different aspects of the proposed approach are necessary. \n2. The ablation studies presented are not good enough to understand the proposed approach thoroughly. \n3. The authors test the proposed work on a few smaller datasets and claim their approach is better than popular approaches trained on large datasets. This conclusions seems a bit premature as the authors have not tested their approach on large datasets. \n4. The paper is not really an easy read and the presentation can be improved. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "ZcrRymkjop", "reviewer": "Reviewer_7rFd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a discrete representation learning strategy for audio sequences designed primarily for retrieval. The learning strategy encourages representations to be similar for pairs of similar inputs, where pairs are created by data augmentation of the input audio. The authors compare their approach on two retrieval tasks: musical query-by-humming and spoken term detection.", "summary_of_the_review": "Overall, I think this paper has some interesting ideas for a novel application of discrete representation learning for audio. However, a weak evaluation prevents me from understanding the strength of the proposed approach. I think the authors should continue working on this, but I don’t think it’s ready for ICLR at this time.", "strength_and_weaknesses": "A strength of this paper is that it focuses on learning discrete representations of audio that are useful for edit distance-based text retrieval, in contrast to other work on discrete audio representations which does not consider text retrieval as a primary goal.\n\nThe primary weaknesses of this paper are (1) a lackluster evaluation which makes straw man comparisons against weak baselines and omits comparisons to domain-specific methods, and (2) an unclear description of critical aspects of the proposed methodology.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HI-EPhC9Dl_", "reviewer": "Reviewer_cgKi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an unsupervised method of computing discrete token sequences from audio, for the purpose of audio retrieval.\nPrevious works either:\n- Use supervised token sequence labels during training, which requires domain expertise and access to labelled data.\n- Train an unsupervised autoencoder on only a single sequence at a time, and are therefore not contrastive in nature.\n\nThe proposal in this paper addresses both of these issues, by contrastively discovering a token codebook. A local minimum, where all codebook entries converge to the same value, is prevented by the joint use of a softmax contrastive criterion and a regular K-means re-clustering.\n", "summary_of_the_review": "The proposed approach seems sufficiently novel.  The proposal seems sufficiently useful in addressing limitations of previous methods.  I do not find any major flaws in the paper.  The experiments seem sufficiently comprehensive.\n\nThe Ithenticate similarity is 3%, which is good.\n", "strength_and_weaknesses": "Strengths:\n- The problem that the proposal in this paper is trying to address is well motivated.\n- The proposal is tested on multiple diverse tasks.\n- The proposal is explicitly compared against multiple baselines.\n\nWeaknesses:\n- In the paper, it is emphasised that the CTC loss is used for the purpose of aligning two audio sequences.  The use of the word \"align\" in this context may lead to misinterpretation, as it differs from the standard use of the word.  The standard meaning of \"align\" is to determine which input audio frames correspond to each token in the output.  This is not what CTC is being used for in this paper.  If this is what CTC is being used for in this paper, then this use comes into conflict with the BLSTM topology of the model, because prior works have shown that the temporal flexibility of a BLSTM allows it to learn any arbitrary alignment between the input and output, without needing any realistic correspondence.  The actual use of the CTC loss in this paper is to encourage the model to output similar token sequences for both inputs, which is not what \"align\" typically means.  Perhaps the authors may consider re-phrasing the description of the motivation for the CTC loss, to abate potential confusion.\n- Wav2Tok+Cos is described in the text, and the text seems to imply that the performance results can be found somewhere.  However, these results seem to be missing from table 2.\n- During audio retrieval, the query is compared against the database using token sequence edit distances.  This assumes that all non-similar tokens are equidistant from each other.  This equidistance assumption may potentially limit the system performance.  This assumption is not discussed in the paper.\n- The unit of measurement in many of the experiment tables is not stated in the table, and a reader would need to search through the text to figure out what the numbers in the tables represent.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster