Sampling strategies in Siamese Networks for unsupervised speech representation learning

Abstract: Recent studies have investigated siamese network architectures for learning invariant speech representations using same-different side information at the word level. Here we investigate systematically an often ignored component of siamese networks: the sampling procedure (how pairs of same vs. different tokens are selected). We show that sampling strategies taking into account Zipf's Law, the distribution of speakers and the proportions of same and different pairs of words significantly impact the performance of the network. In particular, we show that word frequency compression improves learning across a large range of variations in number of training pairs. This effect does not apply to the same extent to the fully unsupervised setting, where the pairs of same-different words are obtained by spoken term discovery. We apply these results to pairs of words discovered using an unsupervised algorithm and show an improvement on state-of-the-art in unsupervised representation learning using siamese networks.

Current speech and language technologies based on Deep Neural Networks (DNNs) BIBREF0 require large quantities of transcribed data and additional linguistic resources (phonetic dictionary, transcribed data). Yet, for many languages in the world, such resources are not available and gathering them would be very difficult due to a lack of stable and widespread orthography BIBREF1 .
The goal of Zero-resource technologies is to build speech and language systems in an unknown language by using only raw speech data BIBREF2 . The Zero Resource challenges (2015 and 2017) focused on discovering invariant sub-word representations (Track 1) and audio terms (Track 2) in an unsupervised fashion. Several teams have proposed to use terms discovered in Track 2 to provide DNNs with pairs of same versus different words as a form of weak or self supervision for Track 1: correspondence auto-encoders BIBREF3 , BIBREF4 , siamese networks BIBREF5 , BIBREF6 .
This paper extends and complements the ABnet Siamese network architecture proposed by BIBREF7 , BIBREF5 for the sub-word modelling task. DNN contributions typically focus on novel architectures or objective functions. Here, we study an often overlooked component of Siamese networks: the sampling procedure which chooses the set of pairs of same versus different tokens. To assess how each parameter contributes to the algorithm performance, we conduct a comprehensive set of experiments with a large range of variations in one parameter, holding constant the quantity of available data and the other parameters. We find that frequency compression of the word types has a particularly important effect. This is congruent with other frequency compression techniques used in NLP, for instance in the computation of word embeddings (word2vec BIBREF8 ). Besides, Levy et al. BIBREF9 reveals that the performance differences between word-embedding algorithms are due more to the choice of the hyper-parameters, than to the embedding algorithms themselves.
In this study, we first show that, using gold word-level annotations on the Buckeye corpus, a flattened frequency range gives the best results on phonetic learning in a Siamese network. Then, we show that the hyper-parameters that worked best with gold annotations yield improvements in the zero-resource scenario (unsupervised pairs) as well. Specifically, they improve on the state-of-the-art obtained with siamese and auto-encoder architectures.
We developed a new package abnet3 using the pytorch framework BIBREF10 . The code is open-sourced (BSD 3-clause) and available on github, as is the code for the experiments for this paper.
For the weakly-supervised study, we use 4 subsets of the Buckeye BIBREF11 dataset from the ZeroSpeech 2015 challenge BIBREF2 with, respectively, 1%, 10%, 50%, and 100% of the original data (see Table 1 ). The original dataset is composed of American English casual conversations recorded in the laboratory, with no overlap, no speech noises, separated in two splits: 12 speakers for training and 2 speakers for test. A Voice Activity Detection file indicates the onset and offset of each utterance and enables to discard silence portions of each file. We use the orthographic transcription from word-level annotations to determine same and different pairs to train the siamese networks.
In the fully unsupervised setting, we obtain pairs of same and different words from the Track 2 baseline of the 2015 ZeroSpeech challenge BIBREF2 : the Spoken Term Discovery system from BIBREF12 . We use both the original files from the baseline, and a rerun of the algorithm with systematic variations on its similarity threshold parameter.
For the speech signal pre-processing, frames are taken every 10ms and each one is encoded by a 40 log-energy Mel-scale filterbank representing 25ms of speech (Hamming windowed), without deltas or delta-delta coefficients. The input to the Siamese network is a stack of 7 successive filterbank frames. The features are mean-variance normalized per file, using the VAD information.
A Siamese network is a type of neural network architecture that is used for representation learning, initially introduced for signature verification BIBREF13 . It contains 2 subnetworks sharing the same architecture and weights. In our case, to obtain the training information, we use the lexicon of words to learn an embedding of speech sounds which is more representative of the linguistic properties of the signal at the sub-word level (phoneme structure) and invariant to non-linguistic ones (speaker ID, channel, etc). A token $t$ is from a specific word type $w$ (ex: “the”,“process” etc.) pronounced by a specific speaker $s$ . The input to the network during training is a pair of stacked frames of filterbank features $x_1$ and $x_2$ and we use as label $y = {1}(\lbrace w_1 = w_2\rbrace )$ . For pairs of identical words, we realign them at the frame level using the Dynamic Time Warping (DTW) algorithm BIBREF14 . Based on the alignment paths from the DTW algorithm, the sequences of the stacked frames are then presented as the entries of the siamese network. Dissimilar pairs are aligned along the shortest word, e.g. the longest word is trimmed. With these notions of similarity, we can learn a representation where the distance between the two outputs of the siamese network $e(x_1)$ and $e(x_2)$ try to respect as much as possible the local constraints between $x_1$ and $x_2$ . To do so, ABnet is trained with the margin cosine loss function: $w$0 
For a clear and fair comparison between the sampling procedures we fixed the network architecture and loss function as in BIBREF5 . The subnetwork is composed of 2 hidden layers with 500 units, with the Sigmoid as non-linearity and a final embedding layer of 100 units. For regularization, we use the Batch Normalization technique BIBREF15 , with a loss margin $\gamma =0.5$ . All the experiments are carried using the Adam training procedure BIBREF16 and early-stopping on a held-out validation set of $30\%$ of spoken words. We sample the validation set in the same way as the training set.
The sampling strategy refers to the way pairs of tokens are fed to the Siamese network. Sampling every possible pairs of tokens becomes quickly intractable as the dataset grows (cf. Table 1 ).
There are four different possible configurations for a pair of word tokens $(t_1,t_2) $ : whether, or not, the tokens are from the same word type, $w_1 = w_2$ . and whether, or not, the tokens are pronounced by the same speaker, $s_1 = s_2$ .
Each specific word type $w$ is characterized by the total number of occurrences $n_w$ it has been spoken in the whole corpus. Then, is deduced the frequency of appearances $f_w \propto n_w$ , and $r_w$ its frequency rank in the given corpus. We want to sample a pair of word tokens, in our framework we sample independently these 2 tokens. We define the probability to sample a specific token word type $w$ as a function of $n_w$ . We introduce the function $\phi $ as the sampling compression function: 
$$\mathbb {P}(w) = \frac{\phi (n_w)}{\sum \limits _{\forall w^{\prime }}\phi (n_{w^{\prime }})}$$   (Eq. 7) 
When a specific word type $w$ is selected according to these probabilities, a token $t$ is selected randomly from the specific word type $w$ . The usual strategy to select pairs to train siamese networks is to randomly pick two tokens from the whole list of training tokens examples BIBREF13 , BIBREF17 , BIBREF5 . In this framework, the sampling function corresponds $\phi : n \rightarrow n$ . Yet, there is a puzzling phenomenon in human language, there exists an empirical law for the distribution of words, also known as the Zipf's law BIBREF18 . Words types appear following a power law relationship between the frequency $f_w$ and the corresponding rank $r_w$ : a few very high-frequency types account for almost all tokens in a natural corpus (most of them are function words such as “the”,“a”,“it”, etc.) and there are many word types with a low frequency of appearances (“magret”,“duck”,“hectagon”). The frequency $f_t$ of type $t$ scales with its corresponding $r_t$ following a power law, with a parameter $\alpha $ depending on the language: $t$0 
One main effect on the training is the oversampling of word types with high frequency, and this is accentuated with the sampling of two tokens for the siamese. These frequent, usually monosyllabic, word types do not carry the necessary phonetic diversity to learn an embedding robust to rarer co-articulations, and rarer phones. To study and minimize this empirical linguistic trend, we will examine 4 other possibilities for the $\phi $ function that compress the word frequency type: : n [2]n, : n [3]n
 : n (1+n), : n 1
The first two options minimize the effect of the Zipf's Law on the frequency, but the power law is kept. The $\log $ option removes the power law distribution, yet it keeps a linear weighting as a function of the rank of the types. Finally with the last configuration, the word types are sampled uniformly.
Another important variation factor in speech realizations is the speaker identity. We expect that the learning of speech representations to take advantage of word pairs from different speakers, to generalize better to new ones, and improve the ABX performance. $
P^s_{-} = \frac{\# \text{Sampled pairs pronounced by different speakers}}{\# \text{Sampled pairs}}
$ 
Given the natural statistics of the dataset, the number of possible "different" pairs exceeds by a large margin the number of possible "same" pairs ( $\sim 1\%$ of all token pairs for the Buckeye-100%). The siamese loss is such that "Same" pairs are brought together in embedding space, and "Different" pairs are pulled apart. Should we reflect this statistic during the training, or eliminate it by presenting same and different pairs equally? We manipulate systematically the proportion of pairs from different word types fed to the network: $
P^w_{-} = \frac{\# \text{Sampled pairs with non-matching word types}}{\# \text{Sampled pairs}}
$ 
To test if the learned representations can separate phonetic categories, we use a minimal pair ABX discrimination task BIBREF19 , BIBREF20 . It only requires to define a dissimilarity function $d$ between speech tokens, no external training algorithm is needed. We define the ABX-discriminability of category $x$ from category $y$ as the probability that $A$ and $X$ are further apart than $B$ and $X$ when $A$ and $X$ are from category $x$ and $x$0 is from category $x$1 , according to a dissimilarity function $x$2 . Here, we focus on phone triplet minimal pairs: sequences of 3 phonemes that differ only in the central one (“beg”-“bag”, “api”-“ati”, etc.). For the within-speaker task, all the phones triplets belong to the same speaker (e.g. $x$3 ) Finally the scores for every pair of central phones are averaged and subtracted from 1 to yield the reported within-talker ABX error rate. For the across-speaker task, $x$4 and $x$5 belong to the same speaker, and $x$6 to a different one (e.g. $x$7 ). The scores for a given minimal pair are first averaged across all of the pairs of speakers for which this contrast can be made. As above, the resulting scores are averaged over all contexts over all pairs of central phones and converted to an error rate.
We first analyze the results for the sampling compression function $\phi $ Figure 1 . For all training datasets, we observe a similar pattern for the performances on both tasks: the word frequency compression improves the learning and generalization. The result show that, compared to the raw filterbank features baseline, all the trained ABnet networks improve the scores on the phoneme discrimination tasks, even in the $1\%$ scenario. Yet, the improvement with the usual sampling scenario $\phi : n \rightarrow n$ is small in all 4 training datasets. The optimal function for the within and across speaker task on all training configuration is the uniform function $\phi : n \rightarrow 1$ . It yields substantial improvements over the raw filterbanks for ABX task across-speaker ( $5.6 $ absolute points and $16.8 \%$ relative improvement for the $1\%$ -Buckeye training). The addition of data for these experiments improves the performance of the network, but not in a substantial way: the improvements from $1\%$ -Buckeye to $100\%$ -Buckeye, for $\phi : n \rightarrow 1$ , is $1\%$0 absolute points and $1\%$1 relative. These results show that using frequency compression is clearly beneficial, and surprisingly adding more data is still advantageous but not as much as the choice of $1\%$2 . Renshaw et al. BIBREF4 , found similar results with a correspondence auto-encoder, training with more training data did not yield improvements for their system.
We now look at the effect on the ABX performances of the proportion of pairs of words pronounced by two different speakers Figure 2 . We start from our best sampling function configuration so far $\phi : n \rightarrow 1$ . We report on the graph only the two extreme training settings. The variations for the 4 different training splits are similar, and still witness a positive effect with additional data on the siamese network performances. Counter-intuitively, the performances on the ABX tasks does not take advantage of pairs from different speakers. It even shows a tendency to increase the ABX error rate: for the $100\%$ -Buckeye we witness an augmentation of the ABX error-rate (2.9 points and $11.6\%$ relative) between $P_{-}^s=0$ and $P_{-}^s=1$ . One of our hypothesis on this surprising effect, might be the poor performance of the DTW alignment algorithm directly on raw filterbanks features of tokens from 2 different speakers.
We next study the influence of the proportion of pairs from different word types $P^w_{-}$ Figure 3 . In all training scenarios, to privilege either only the positive or the negative examples is not the solution. For the different training splits, the optimal number for $P_{-}^w$ is either $0.7$ or $0.8$ in the within and across speaker ABX task. We do not observe a symmetric influence of the positive and negative examples, but it is necessary to keep the same and different pairs. The results collapsed, if the siamese network is provided only with positive labels to match: the network will tend to map all speech tokens to the same vector point and the discriminability is at chance level.
Now, we transfer the findings about sampling from the weakly supervised setting, to the fully unsupervised setting. We report in Table 2 our results for the two ZeroSpeech 2015 BIBREF2 corpus: the same subset of the Buckeye Corpus as earlier and a subset of the NCHLT corpus of Xitsonga BIBREF21 . To train our siamese networks, we use as BIBREF5 , the top-down information from the baseline for the Track 2 (Spoken Term Discovery) of the ZeroSpeech 2015 challenge from BIBREF12 . The resulting clusters are not perfect, whereas we had perfect clusters in our previous analysis.
In Thiolière et al. BIBREF5 the sampling is done with : $P^w_{-} = P^s_{-} = 0.5$ , and $\phi = n \rightarrow n$ . This gives us a baseline to compare our sampling method improvements with our own implementation of siamese networks.
First, the “discovered” clusters – obtained from spoken term discovery system – don't follow the Zipf's law like the gold clusters. This difference of distributions diminishes the impact of the sampling compression function $\phi $ .
We matched state-of-the-art for this challenge only on the ABX task within-speaker for the Buckeye, otherwise the modified DPGMM algorithm proposed by Heck et al. stays the best submissions for the 2015 ZeroSpeech challenge.
Finally, we study the influence of the DTW-threshold $\delta $ used in the spoken discovery system on the phonetic discriminability of siamese networks. We start again from our best finding from weakly supervised learning. The clusters found by the Jansen et al. BIBREF12 system are very sensitive to this parameter with a trade-off between the Coverage and the Normalized Edit Distance (NED) introduced by BIBREF24 .
We find that ABnet is getting good results across the various outputs of the STD system shown in Table 3 and improves over the filterbanks results in all cases. Obtaining more data with the STD system involves a loss in words quality. In contrast with the weakly supervised setting, there is an optimal trade-off between the amount and quality of discovered words for the sub-word modelling task with siamese networks.
We presented a systematic study of the sampling component in siamese networks. In the weakly-supervised setting, we established that the word frequency compression had an important impact on the discriminability performances. We also found that optimal proportions of pairs with different types and speakers are not the ones usually used in siamese networks. We transferred the best parameters to the unsupervised setting to compare our results to the 2015 Zero Resource challenge submissions. It lead to improvements over the previous neural networks architectures, yet the Gaussian mixture methods (DPGMM) remain the state-of-the-art in the phonetic discriminability task. In the future, we will study in the same systematic way the influence of sampling in the fully unsupervised setting. We will then try to leverage the better discriminability of our representations obtained with ABnet to improve the spoken term discovery, which relies on frame-level discrimination to find pairs of similar words. Besides, power law distributions are endemic in natural language tasks. It would be interesting to extend this principle to other tasks (for instance, language modeling).
The team's project is funded by the European Research Council (ERC-2011-AdG-295810 BOOTPHON), the Agence Nationale pour la Recherche (ANR-10-LABX-0087 IEC, ANR-10-IDEX-0001-02 PSL* ), Almerys (industrial chair Data Science and Security), Facebook AI Research (Doctoral research contract), Microsoft Research (joint MSR-INRIA center) and a Google Award Grant.

Figure and table captions:
Table 1: Statistics for the 4 Buckeye splits used for the weakly supervised training, the duration in minutes expressed the total amount of speech for training
Figure 1: ABX across-speaker error rates on test set with various sampling compression functions φ for the 4 different Buckeye splits used for weakly supervised training. Here, the proportions of pairs with different speakers P s− and with different word types Pw− are kept fixed: P s − = 0.5, P w − = 0.5
Figure 2: Average ABX error rates across-speaker with various proportion pairs of different speakers P s−, with φ : n → 1 and Pw− = 0.5.
Figure 3: Average ABX error rates across-speaker with various proportion pairs with different word types Pw− , where φ : n→ 1 and P s− = 0.5
Table 3: Number of found clusters, NED, Coverage, ABX discriminability results with our ABnet with Pw− = 0.7, P s − = 0, φ : n → 1, for the ZeroSpeech2015 Buckeye for various DTW-thresholds δ in the Jansen et al. [13] STD system. The best results for each metric are in bold.
Table 2: ABX discriminability results for the ZeroSpeech2015 datasets. The best error rates for each conditions for siamese architectures are in bold. The best error rates for each conditions overall are underlined.

Questions about this paper:
What is the metric that is measures in this paper?

Answers:
error rate in a minimal pair ABX discrimination task