Year: 2020

Abstract:
In this paper, we present a method for learning discrete linguistic units by incorporating vector quantization layers into neural models of visually grounded speech. We show that our method is capable of capturing both word-level and sub-word units, depending on how it is configured. What differentiates this paper from prior work on speech unit learning is the choice of training objective. Rather than using a reconstruction-based loss, we use a discriminative, multimodal grounding objective which forces the learned units to be useful for semantic image retrieval. We evaluate the sub-word units on the ZeroSpeech 2019 challenge, achieving a 27.3% reduction in ABX error rate over the top-performing submission, while keeping the bitrate approximately the same. We also present experiments demonstrating the noise robustness of these units. Finally, we show that a model with multiple quantizers can simultaneously learn phone-like detectors at a lower layer and word-like detectors at a higher layer. We show that these detectors are highly accurate, discovering 279 words with an F1 score of greater than 0.5.

heading: 1 INTRODUCTION
text: By 8 months of age, human infants learn to recognize not only the names of their caregivers and common objects, but also the contrast between the different vowels and consonants which comprise these words (Dupoux, 2018). Nearly all toddlers learn to carry a conversation long before they can read and write. Humans learn to model the discrete, hierarchical, and compositional nature of their native language not from written text, but from speech audio - a continuous, time-varying waveform which is the product not only of the underlying words which were spoken, but also the physical properties of the speaker’s vocal tract, the speaker’s health and emotional state, and the noise and reverberation present in the environment. The question of how such a complex symbolic system is inferred from continuous and noisy sensory input data is of interest not only to the cognitive science community, but also to machine learning researchers who aim to reproduce this ability with computers. A more comprehensive understanding of human language acquisition has practical significance in real-world applications, such as automatic speech recognition (ASR) and natural language understanding (NLU) systems. In the past several decades, enormous progress has been made in speech recognition research, and nowadays ASR systems are able to achieve human-level accuracy in many domains (Chiu et al., 2018). Unfortunately, the techniques that have been developed to achieve these levels of performance are extremely data-hungry, requiring many thousands of hours of speech audio recordings for training. Since supervised machine learning algorithms form the basis of ASR training, the data also needs to be annotated by expert humans. Due to the immense cost of collecting and annotating speech data, ASR technology currently exists for approximately 120 (Google, 2019) out of the nearly 7,000 (Lewis et al., 2016) human languages spoken worldwide. It is highly unlikely that purely supervised machine learning techniques will be able to scale to include all human languages, necessitating the development of alternative methods by researchers which are able to function with far fewer annotations, or even no annotations at all. Because human beings provide an existence proof of language acquisition from speech completely without language supervision, it is plausible that this ability could be replicated by a machine learning algorithm.
∗Equal contribution
In this paper, we present a method for discovering discrete and hierarchical representations of speech units both at the sub-word level and the word level. Previously proposed linguistic unit discovery methods have only leveraged the speech audio modality in isolation, relying on objective functions that attempt to capture statistical regularities within the speech signal. The key innovation in our work is that we discover units by training models with explicit discretization layers to associate speech waveforms with visual images using a cross-modal grounding objective. This forces our models to learn representations which capture semantic information at the highest layers of the network. Because semantics are predominantly carried by words, and words are composed of subword units (such as phones and syllables), the visual grounding objective indirectly forces the model to learn speaker- and noise-invariant representations of speech units. By incorporating trainable quantization layers into our networks, we are able to capture these units in discrete inventories. Whether these units correspond to word-like or sub-word units depends on where the quantization layers are inserted, and how they are trained.

heading: 2 RELATED WORK
text: Prior work on unsupervised modeling of the speech signal has generally focused on learning representations which either disentangle or isolate the latent factors that are of interest for downstream tasks. In most cases the primary latent factor of interest is the phonetic or lexical identity of a given segment of speech, but other factors, such as the identity of the speaker, are sometimes of interest as well. Because the factors of interest are often inherently discrete (e.g. words and phones), many of the proposed approaches attempt to perform segmentation and clustering of the surface features in one way or another. One family of techniques is based upon Segmental Dynamic Time Warping (S-DTW) (Park & Glass, 2005; 2008; Jansen et al., 2010; Jansen & Van Durme, 2011), which uses a self-comparison algorithm to identify relatively long duration (on the order of a second) patterns which frequently reoccur in a speech corpus; these patterns tend to capture words or short phrases. A different line of work employs probabilistic graphical models to jointly segment and cluster the speech signal (Varadarajan et al., 2008; Zhang & Glass, 2009; Gish et al., 2009; Lee & Glass, 2012; Siu et al., 2014; Lee et al., 2015; Ondel et al., 2016; Kamper et al., 2016; 2017a). With an appropriately designed model, it is possible to learn multiple, hierarchical categories of speech units. However, in order to enable efficient inference, the conditional distributions of these models tend to be simple and therefore have limited modeling power.
Deep neural network models have been successfully used to learn powerful speech representations using weakly or unsupervised objectives (Thiolliere et al., 2015; Kamper et al., 2015; Hsu et al., 2017a;b; Hsu & Glass, 2018; Holzenberger et al., 2018; Milde & Biemann, 2018; van den Oord et al., 2018; Chung et al., 2019; Pascual et al., 2019). These representations have predominantly been continuous in nature, as discrete latent variables are not trivially compatible with backpropagation. To obtain discrete representations, a post-hoc clustering step can be applied to the continuous representations (Kamper et al., 2017b; Feng et al., 2019). More recently, several papers have proposed ways of directly incorporating discrete variables into neural network models, including using Gumbel-Softmax (Eloff et al., 2019b) or straight-through estimators (van den Oord et al., 2017; Chorowski et al., 2019; Razavi et al., 2019).
A different method for learning meaningful representations of speech is via a multimodal grounding objective, which encourages the learning of speech representations that are predictive of the contextual information contained in a separate but accompanying modality, such as vision. Visual grounding of speech is a form of self-supervised learning (Virginia de Sa, 1994), which is powerful in part because it offers a way of training models with a discriminative objective that does not depend on traditional transcriptions or annotations. The first work in this direction relied on phone strings to represent the speech (Roy & Pentland, 2002; Roy, 2003), but more recently this learning has been shown to be possible directly on the speech signal (Synnaeve et al., 2014; Harwath & Glass, 2015; Harwath et al., 2016). Subsequent work on visually-grounded models of speech has investigated improvements and alternatives to the modeling or training algorithms (Leidal et al., 2017; Kamper et al., 2017c; Havard et al., 2019a; Merkx et al., 2019; Chrupała et al., 2017; Scharenborg et al., 2018; Kamper et al., 2019b;a; Surı́s et al., 2019; Ilharco et al., 2019; Eloff et al., 2019a), application to multilingual settings (Harwath et al., 2018a; Kamper & Roth, 2017; Azuh et al., 2019; Havard et al., 2019a), analysis of the linguistic abstractions, such as words and phones, which are learned by the models (Harwath & Glass, 2017; Harwath et al., 2018b; Drexler & Glass, 2017; Alishahi et al.,
2017; Harwath et al., 2019; Harwath & Glass, 2019; Havard et al., 2019b), and the impact of jointly training with textual input (Holzenberger et al., 2019; Chrupała, 2019; Pasad et al., 2019). Representations learned by models of visually grounded speech are also well-suited for transfer learning to supervised tasks, being highly robust to noise and domain shift (Hsu et al., 2019).

heading: 3 DATA AND MODELS

heading: 3.1 DATASET
text: For training our models, we utilize the MIT Places 205 dataset (Zhou et al., 2014) and their accompanying spoken audio captions (Harwath et al., 2016; 2018b). The caption dataset contains approximately 400,000 spoken audio captions, each of which describes a different Places image. These captions are free-form spontaneous speech, collected from over 2,500 different speakers and covering a 40,000 word vocabulary. The average caption duration is approximately 10 seconds, and each caption contains on average 20 words. For vetting our models during training, we use a held-out validation set of 1,000 image-caption pairs.

heading: 3.2 NEURAL MODELS OF VISUALLY-GROUNDED SPEECH
text: We base our model upon the Residual Deep Audio-Visual Embedding network (ResDAVEnet) architecture (Harwath et al., 2019), which contains two branches of fully convolutional networks, one for images and the other for audio. Each branch encodes samples of the corresponding modality into a d-dimensional space, regardless of the original dimensionality of the samples. This is achieved by applying global spatial mean pooling and global temporal mean pooling to the image branch output and the audio branch output, respectively. The image branch is adapted from ResNet50 (He et al., 2016), where the final softmax layer and the preceding fully-connected layers are removed, replaced with a 1x1 linear convolutional layer in order to project the feature map to the desired dimension. To model the audio inputs, a 17-layer fully convolutional network with residual connections is used. The input is a log Mel-frequency spectrogram with 40 frequency bins and 25 ms-wide, Hammingwindowed frames with a shift of 10 ms. The first layer of this network is a 1-D convolution that spans the entire frequency axis of the spectrogram, while the remaining 16 convolutional layers are 1-D across the time axis. These 16 layers are divided into four residual blocks of 4 layers each, and downsampling between these blocks is accomplished by applying the first convolution of each block with a stride of 2. For full details of the model, refer to Harwath et al. (2019).

heading: 3.3 LEARNING HIERARCHICAL DISCRETE UNITS WITH VECTOR QUANTIZING LAYERS
text: Previous analyses reveal that ResDAVEnet-like models learn linguistic abstractions at different levels, including words (Harwath & Glass, 2017) and robust phonetic features (Harwath & Glass, 2019; Hsu et al., 2019). To explicitly learn hierarchical discrete linguistic units within this framework, we propose to incorporate multiple vector quantization (VQ) layers (van den Oord et al., 2017) into the ResDAVEnet audio branch; we refer to this new architecture as ResDAVEnet-VQ.
VQ layers can be understood as a type of bottleneck, which constrain the amount of information that can flow through. While these layers have been used to learn discrete sub-word units (van den Oord et al., 2017; Chorowski et al., 2019; Razavi et al., 2019), previous work injects VQ layers into autoencoders that are trained with a reconstruction loss. As a result, the embedding dimension of each code and the number of codes need to be carefully tuned (Liu et al., 2019). When the embedding dimension is too low or the codebook size too small, the model does not have enough expressive power to capture linguistic variability. When it is too large, the model starts to encode non-linguistic information in order to improve reconstruction. In contrast, the learning signal of ResDAVEnet-VQ is provided by the visual-semantic grounding objective. Rather than encoding as much information about input as possible, the learned codes in ResDAVEnet-VQ only need to capture semantic information. Since semantics in speech are predominantly transmitted by words, and words are composed of sub-word units like phones, the grounding objective places pressure on the model to robustly infer both from speech. Since words and phones are inherently discrete symbols, representing them with learned discrete units may not even hurt the grounding performance.
Figure 1 illustrates the proposed ResDAVEnet-VQ model. We add a quantization layer after each of the first two residual blocks of the ResDAVEnet-VQ model, denoted as VQ2 and VQ3, respectively, with the intention that they should capture discrete sub-word-like and word-like units. A VQ layer is defined as E ∈ RK×D, where K represents the codebook size, and D represents the output dimensionality of the input features to the codebook. Denoting the tth temporal frame of the input to the quantization layer as xt, quantization is performed according to qt = Ek,:, where k = arg minj ||xt−Ej,:||2 The quantized output is then fed as input to the subsequent residual block. As in van den Oord et al. (2017), we use the straight-through estimator (Bengio et al., 2013) to compute the gradient passed from qt to xt. We use the exponential moving average (EMA) codebook updates proposed by van den Oord et al. (2017).

heading: 3.4 CODEBOOK LEARNING SCHEDULES
text: We include multiple VQ layers in the ResDAVEnet-VQ model, each of which can be independently enabled or bypassed without changing the rest of the architecture configuration. When all model weights, including the VQ codebooks, are trained jointly in a single training run we call this a “coldstart” model. Alternatively, a model can be “warm-started” by copying the weights from another trained model that has fewer (or no) VQ layers enabled, and randomly initializing the codebook of the newly activated VQ layer(s). This gives rise to the questions of how many quantizers should be used and in what order they should be enabled. It is unclear whether models with the same VQ layers activated would learn the same representation at each layer regardless of the training curriculum. Let Am denote a subset of all VQ layers, and Am−1 ⊂ Am. We use “A1 → ...→ AM” to denote a model that is obtained by sequentially training models “A1 → ... → Am” initialized from “A1 → ... → Am−1”, where the model A1 is initialized from scratch, and the final model would have VQ layers in AM activated. For instance, a model initialized from scratch with no VQ layers enabled is denoted as “∅”, and a model initialized with that and with both layers enabled is denoted as “∅→ {2, 3}”.

heading: 3.5 TRAINING WITH THE TRIPLET LOSS
text: We train our models using the same loss function as Harwath et al. (2019). This loss function blends two triplet loss terms (Weinberger & Saul, 2009), one based on random sampling of negative examples, and the other based on semi-hard negative mining (Jansen et al., 2018), in order to find more challenging negative samples. Specifically, let the sets of output embedding vectors for a minibatch of B audio/image training pairs respectively be A = {a1, . . . ,aB} and I = {i1, . . . , iB}. To compute the randomly-sampled triplet loss term, we select impostor examples for the jth input according to āj ∼ UniformCategorical({a1, . . . ,aB}\aj) and īj ∼ UniformCategorical({i1, . . . , iB}\ij). The randomly-sampled triplet loss is then computed as:
Ls = B∑
j=1
( max(0, iTj āj − iTj aj + 1) + max(0, īTj aj − iTj aj + 1) ) (1)
For the semi-hard negative triplet loss, we first define the sets of impostor candidates for the jth
example as Âj = {a ∈ A|iTj a < iTj aj} and Îj = {i ∈ I|iTaj < iTj aj}. The semi-hard negative
loss is then computed as:
Lh = B∑
j=1
( max(0,max
â∈Âj (iTj â)− iTj aj + 1) + max(0,max î∈Îj (îTaj)− iTj aj + 1)
) (2)
Finally, the overall loss function is computed by combining the two above losses, L = Ls + Lh, which was found by (Harwath et al., 2019) to outperform either loss on its own.

heading: 3.6 IMPLEMENTATION DETAILS
text: All of our models were trained for 180 epochs using the Adam optimizer (Kingma & Ba, 2014) with a batch size of 80. We used an exponentially decaying learning rate schedule, with an initial value of 2e-4 that decayed by a factor of 0.95 every 3 epochs. Following van den Oord et al. (2017), we use an EMA decay factor of γ = .99 for training each VQ codebook. Our core experimental results all use a codebook size of 1024 vectors for all quantizers, but in the supplementary material we include experiments with smaller and larger codebooks. Following Chorowski et al. (2019), the jitter probability hyperparameter for each quantization layer was fixed at 0.12. While we do not apply data augmentation to the input spectrograms, during training we perform standard data augmentation techniques to the images. We resize each raw image so that its smallest dimension is 256 pixels, and then we apply an Inception-style random crop which is resized to 224 pixels square. During training, we also flip each image horizontally with a probability of 0.5. During evaluation, the center 224 pixel square crop is always taken from the image. Finally, the RGB pixel values are mean and variance normalized. We trained each model on the Places audio caption train split, and computed the image and caption recall at 10 (R@10) scores on the validation split of the Places audio captions after each training epoch. The model snapshot that achieved the highest average R@10 score on the validation set from each training is used for all evaluation. To extract embeddings and units from our models, we simply perform a forward pass through the speech branch of the ResDAVEnet-VQ network and retain the outputs from the target layer at a uniform frame-rate. The frame-rate is determined by the downsampling factor at the target layer relative to the input. For non-quantized layers, these outputs will be continuous embeddings. For quantized layers, these will be quantized embedding retrieved from the assigned entry in the codebook.

heading: 4 EXPERIMENTS

heading: 4.1 SUB-WORD UNIT LEARNING ON THE ZEROSPEECH 2019 ABX TASK
text: Evaluation metrics Learning unsupervised speech representations that are indicative of phonetic content is of high interest to the speech community, and recently has been the focus of the ZeroSpeech Challenge (Versteegh et al., 2015; Dunbar et al., 2017; 2019). One of the core evaluations is the minimal-pair ABX task (Schatz et al., 2013), which aims to benchmark representations in terms of their discriminability between different sub-word speech units. In this task, a model is tasked with extracting representations for a triplet of speech waveform segments denoted by A, B, and X . A and B are constrained to be a triphone minimal pair; that is, both segments capture three phones, but differ only in the identity of their center phone. The third segment, X is chosen to contain the same underlying triphone sequence as A. Supposing f(·) denotes the model’s mapping function from a waveform segment to a sequence of embedding vectors, the ABX error rate under a given similarity metric S(·, ·) is defined as the fraction of ABX triples in which S(f(A), f(X)) > S(f(B), f(X)). An ABX error rate of 50% indicates random assignment, while an ABX of 0% reflects perfect phone discriminability. In the ZeroSpeech challenge, S(·, ·) is implemented using Dynamic Time Warping (DTW) with various distance measures (cosine, KL, etc.). In our evaluation, we use the cosine distance.
The ZeroSpeech 2019 challenge in particular emphasizes on discovering an inventory of discrete sub-word units, rather than continuous representations. Therefore, in addition to an ABX error rate, a bitrate is also computed for each model which reflects the amount of information carried by the learned units. A lower bitrate can be achieved by having a more compact inventory of learned units or having a smaller number of codes per second. The full details of the evaluation can be found in Dunbar et al. (2019). To be clear, all of our ResDAVEnet-VQ models were not trained on the
ZeroSpeech training data, but instead on the Places audio captions, thus there is a domain mismatch between training and testing these models.
In addition to the frame-based bitrate and ABX scores computed by the ZeroSpeech 2019 evaluation toolkit, we implement our own extensions to these metrics. Because it is common for successive frames to be assigned to the same codebook entry and phonetic information is not encoded at a fixed frame rate, lossless run length encoding (RLE) can be a more reasonable measure of the bitrate of a frame-based model. RLE does not change the ABX score since it can be trivially inverted, but it does change the bitrate. For computing the RLE bitrate, we modify the bitrate calculation specified in Dunbar et al. (2019) so that a unique symbol is defined as the tuple (unit, length) where length is the number of frames assigned to a given unit with in a segment. We also consider segment-based ABX and bitrate, which is similar to the RLE metrics except in this case we outright discard the frame length information. This typically results in an even greater reduction in bitrate, but also an accompanying deterioration in ABX score.
Baseline models In Table 1, we compare our results to those derived from two of the topperforming submissions to the ZeroSpeech 2019 challenge: a re-implementation of WaveNetVQ (Chorowski et al., 2019) provided by Cho et al. (2019) and FHVAE-DPGMM (Feng et al., 2019). Using the code accompanied with the WaveNet-VQ submission, we were able to train their model on the set of 400,000 Places audio captions to make a fairer comparison with our ResDAVEnet-VQ models in terms of the amount of speech data used. In addition, when trying to reproduce the reported WaveNet-VQ results, we obtain better performance than previously reported by training for more steps. Table 1 shows that WaveNet-VQ achieves similar bitrates regardless of the training data. However, ABX deteriorates from 19.98 to 24.87, implying the model cannot utilize data of a larger scale but out-of-domain relative to the test set. A similar degradation when testing on out-of-domain data with FHVAE models was observed in Hsu et al. (2019). We did not re-train the model submitted by Feng et al. (2019), and instead compare against the scores reported in Dunbar et al. (2019).
ABX discrimination without using quantization Our first experiment investigates exactly which layer in the ResDAVEnet-VQ model is most suited for ABX phone discrimination, and would thus make a good candidate for learning of quantized sub-word units. The leftmost plot in Figure 2 shows that layers 2 and 3 of a ResDAVEnet-VQ model without any quantization enabled perform
the best in terms of ABX error rate on the ZeroSpeech 2019 English test set; the exact numbers for this model are displayed in the caption of Figure 2. Because layers 2 and 3 achieve the lowest ABX error rates without quantization, we focus our attention on the impact of quantization there.
Quantizing one layer When quantizing only one layer, we examine quantization of layer 2 vs. layer 3, and using cold-start training vs. warm-start initialization from model “∅”. The ABX and bitrate results for these models, as well as the R@10 scores on the Places validation set, are shown in Table 1. In all cases, quantization applied at the output of layer 2 achieves a better ABX score than quantization at layer 3, but VQ3 achieves a better bitrate. Quantization barely impacts the performance of layer 2, whose ABX score very slightly rises from 11.35 to 11.79. Warm-start initialization is beneficial to R@10 and ABX score in both cases, but we notice an intriguing anomaly when applying cold-start quantization to layer 3: the ABX score deteriorates significantly, rising from 10.86 in the case of the non-quantized model to 38.21. This indicates that while VQ2 is capable of learning a finite inventory of units that are highly predictive of phonetic identity from either a warm-start or cold-start initialization, cold-start training of VQ3 results in very little phonetic information captured by the quantizer. Interestingly, this model is still learning to infer visual semantics from the speech signal, as evidenced by a high R@10 score; we later show in Section 4.2 that the reason for this anomaly is because cold-start training of VQ3 results in the learning of word detectors. In all cases except for model “{3}”, we note that the ABX scores achieved by our models are significantly better than the baselines. Our best model in terms of ABX (“∅→ {2}”) achieves a 41.0% reduction in ABX over the WaveNet-VQ baseline, at a cost of a 132.3% increase in RLE bitrate; however, model “∅→ {3}” achieves a 24.7% reduction in ABX error rate with only a 2.4% increase in RLE bitrate. These results do not constitute a fair comparison, however, because the WaveNet-VQ and ResDAVEnet-VQ models were trained on different datasets; when training the WaveNet-VQ model on the same set of audio captions used to train ResDAVEnet-VQ (but without the accompanying images, since WaveNet-VQ is not a multimodal model), the ABX error rate increases to 24.87%, tipping the results even more in favor of the ResDAVEnet-VQ models.
Quantizing two layers Quantizing multiple layers at once offers the possibility of learning a hierarchy of units. Thus, we aim to capture phonetic information in a lower layer quantizer and word-level information at a higher layer quantizer. Cold-start training of two quantizers (“{2, 3}”) results in a significant drop in ABX performance for both VQ2 and VQ3, but also a drop in R@10 on the Places validation set. We see much better results in terms of R@10 and ABX for the remaining 3 models which were initialized from the “∅” model or a model with only one quantizer enabled; for example, model “{2} → {2, 3}” achieves an ABX of 14.52 with an RLE bitrate of 136.11, representing a 27.3% ABX improvement over the best baseline while keeping the bitrate approximately the same. We see in model “{3} → {2, 3}” that the same phenomenon observed with model “{3}” persists: VQ3 achieves relatively poor ABX, despite a high overall R@10 and strong ABX with VQ2 at 13.55%. We confirm in Section 4.2 that the VQ3 layer of model “{3} → {2, 3}” does
indeed capture word-level information, indicating that this model has successfully localized phonetic unit identity in the second layer and lexical unit identity in the third layer. Overall, our results suggest that when learning hierarchical quantized representations with a ResDAVEnet-VQ model, the nature of the representations learned is highly dependent on the training curriculum.
Training and testing on noisy data In Hsu et al. (2019), it was shown that representations learned by a ResDAVEnet model were far more robust to train/test domain mismatch in terms of background noise, channel characteristics, and speaker identity than standard spectral features when training a supervised speech recognizer. Here, we examine whether this robustness is also exemplified by the quantized versions of this model. We construct three additional test sets using the ZeroSpeech 2019 English testing data by adding noise sampled from the AudioSet (Jansen et al., 2018) dataset. For each ZeroSpeech testing waveform, we randomly sampled an AudioSet waveform of the same duration and performed linear mixing with a signal-to-noise ratio (SNR) selected randomly within a specified range. We construct low, medium, and high noise testing sets, corresponding to SNRs of 20-30 dB, 10-20 dB, and 0-10 dB. We then perform the ABX discrimination task on these noisy waveforms, displaying the results in Table 2. We find that for all models, a worsening SNR results in a deterioration in ABX performance. However, the ResDAVEnet-VQ models prove to be far more noise robust than the Wavenet-VQ model; even in the high noise testing set, the best ResDAVEnetVQ model achieves an ABX of 23.96%, while the WaveNet-VQ models degrade to nearly-random ABX scores of 42.55% and 42.67%.
Given that a ResDAVEnet-VQ model trained on the “clean” Places Audio captions is highly robust to additive noise on the ABX discrimination task, we investigated whether adding noise to the Places Audio captions themselves would result in an even higher degree of noise robustness. To that end, we followed a similar data augmentation approach to create a noisy version of the Places Audio captions, where the SNR of each caption was randomly chosen to sit within the range of 0-30 dB. The bottom half of Table 2 shows the results of training several ResDAVEnet-VQ models on the noisy Places Audio captions and testing on the clean and noisy ZeroSpeech ABX tasks. In general, we observe a degradation ABX score in the clean conditions, but with a significantly higher degree of noise robustness in the noisier conditions.
Visualization of learned units To better measure the correspondence between the VQ units and English phones, we compute corpus-level co-occurrence statistics (at the frame-level) across the TIMIT training set, excluding the sa dialect sentences. To facilitate visualization, we use the “∅ → {2}” model with a codebook size of 128. We display the conditional probability matrix P (phone|unit) in Figure 3, with the rows and columns ordered via spectral co-clustering with 10
clusters in order to group together phones that share similar sets of VQ codes. Visually, there is a strong mapping between TIMIT phone labels and ResDAVEnet-VQ codes. In some cases, redundant codes are used for the same phone label (this is especially the case for the silence label), and in other cases we see that phones belonging to the same manner class often tend to share codebook units. We can numerically quantify the mapping between the phone and unit labels with the normalized mutual information measure (NMI), which we found to be .378 in this case. We also include several caption spectrograms with their time-aligned unit sequences in Figures 5, 6, and 7 in the supplementary material.

heading: 4.2 FROM PHONES TO WORDS: LEARNING A HIERARCHY OF UNITS
text: As shown in Table 1, all of the ResDAVEnet-VQ models which underwent cold-start training of VQ3 exhibited a similar phenomenon in which the ABX error rate of that layer was particularly high, despite the model performing well at the image-caption retrieval task. We hypothesized that this could be due to VQ3 learning to recognize higher level linguistic units, such as words. To examine this empirically, we inferred the VQ3 unit sequence for every audio caption in the Places Audio training set according to several different models. Using the estimated word-level transcriptions of the utterances (provided by the Google SpeechRecognition API), we computed precision, recall, and F1 scores for every unique (word, VQ3 code) pair for a given model and quantization layer. We then ranked the VQ codes in descending order according to their maximum F1 score for any word in the vocabulary. Table 3 shows a sampling of these statistics for model “{3} → {2, 3}”. In the supplementary material, we include many more examples for this model in Table 7, as well as
examples for the “{2} → {2, 3}” model (which did not learn VQ3 word detectors) in Table 8. It should be emphasized that these models are exactly the same in all respects, except for the order in which their quantizers were trained.
We examine the overall performance of VQ3 as a word detector for these models in Figure 4. The right hand side of Figure 4 displays the number of VQ3 codes whose maximum F1 score is above a given threshold, while the left hand side shows the distribution of precision and recall scores for the top 250 words ranked by F1. This gives an approximate indication of how many VQ3 codes have learned to specialize as detectors for a specific word. We see that the VQ3 layer of model “{3} → {2, 3}” learns 279 codebook entries with an F1 score above 0.5. In contrast, the VQ3 layer of model “{2} → {2, 3}” learns only a handful of word-detecting codebook entries with an F1 of greater than 0.5. This experiment supports the notion that the reason for the poor ABX performances of the VQ3 layer in models “{3}” and “{3} → {2, 3}” is in fact due to its specialization for detecting specific words, and that this specialization only emerges when the VQ3 layer is learned before the VQ2 layer. Section A.2 in the supplementary material examines this phenomenon in greater experimental detail.

heading: 5 CONCLUSIONS
text: In this paper, we demonstrated that the neural vector quantization layers proposed by van den Oord et al. (2017) can be integrated into the visually-grounded speech models proposed by Harwath et al. (2019). This resulted in the ability of the speech model to directly represent speech units, such as phones and words, as discrete latent variables. We presented extensive experiments and analysis of these learned representations, demonstrating significant improvements in phone discrimination ability over the current state-of-the-art models for sub-word speech unit discovery. We demonstrated that these units are also far more robust to noise and domain shift than units derived from previously proposed models. These results supported the notion that semantic supervision via a discriminative, multimodal grounding objective has the potential to be more powerful than reconstruction-based objectives typically used in unsupervised speech models.
We also showed how multiple vector quantizers could be employed simultaneously within a single ResDAVEnet-VQ model, and that these quantizers could be made to specialize in learning a hierarchy of speech units: specifically, phones in the lower quantizer and words in the upper quantizer. Our analysis showed that hundreds of codebooks in the upper quantizer learned to perform as word detectors, and that these detectors were highly accurate. Our experiments also revealed that this behavior only emerged when VQ3 was trained before VQ2. These results suggest the importance of the learning curriculum, which should be more deeply investigated in future work. Future work should attempt to make explicit what kind of compositional rules are implicitly encoded by these models when mapping sequences of codes from the lower quantizer to word-level units in the upper quantizer; the automatic derivation of a sub-word unit inventory, vocabulary, and pronunciation lexicon could serve as the starting point for a fully unsupervised speech recognition system. Future work should also investigate whether layers above VQ3 could be made to learn even higher-level linguistic abstractions, such as grammar, syntax, and compositional reasoning.

heading: A APPENDIX
text: A.1 VARYING THE CODEBOOK SIZE.
In Table 4, we examine the impact of varying the codebook size of model “∅ → {2}” from 128 through 2048. We find that the ABX score is best for 1024 codebook vectors, although the performance is quite good for all models. Unsurprisingly, models with smaller codebooks also achieve lower bitrates.

heading: 128 .772 14.25 295.65 212.27 15.42 179.38

heading: 256 .756 12.95 341.18 260.10 14.21 228.07

heading: 512 .761 12.59 363.95 288.64 13.10 259.94

heading: 1024 .760 11.79 390.61 317.66 12.66 289.11

heading: 2048 .768 12.41 360.04 283.68 13.15 254.23
text: A.2 THE IMPACT OF THE VQ TRAINING CURRICULUM ON THE LOCALIZATION OF WORD DETECTORS
In Section 4.2, we showed that cold-start training of the VQ3 layer caused its codebook vectors to specialize as word detectors, whereas warm-start training did not. Our subsequent experiments (Table 6) revealed that when adding a third quantization layer at the Res4 position to a model that did not learn word detectors at VQ3, the VQ4 layer did in fact learn many word detectors. This suggests implicit word recognition ability can be localized at different layers in the ResDAVEnet audio model, and exactly where it emerges depends upon the VQ training curriculum. We hypothesize that this is due in part to two factors:
1. In a warm-start model, whatever type of information (subword-like, word-like) the continuous model learned to encode at a particular convolutional layer (or residual block) does not change after a quantizer is appended to that layer.
2. In a cold-start model, each active quantization layer forms a potential bottleneck, restricting the amount of information that is able to pass through to subsequent layers.
According to this hypothesis, if word-level recognition tends to emerge at a particular layer in an unconstrained network with no quantization bottleneck, it will stay there when quantization is introduced for fine-tuning. However, when a quantization bottleneck is introduced from the very beginning of training, the gradient flowing down into the lower network layers is more constrained during the initial training epochs (when the gradient tends to be the largest). This may have the effect of steering the optimizer into a different part of the parameter space, in which word recognition occurs at a different layer than it otherwise would.
We present results from two experiments that support this view. In Table 5, we show the ABX scores of the Res3 layer prior to quantization (if present) during the course of three different training curricula resulting in a “{2, 3}” final model. We observe that the ABX error rate changes very little within each individual curriculum. This indicates that the initial model sets the stage for which layer learns to capture phonetic information with the highest salience, and that subsequent training steps do not tend to move this information elsewhere.
Table 6 displays the number of word detectors learned by various VQ layers across different training curricula. Here, we claim that a codebook vector belonging to a particular VQ layer has learned to be a word detector if its F1 score for any word appearing in the test set exceeds 0.5 (as measured on the test set). There are several interesting things to note here. First, we only observe a significant number of word detectors at the VQ3 layer when that layer is trained from a cold-start. Even when adding a second VQ layer as in the “{3} → {2, 3}” model, these detectors remain at VQ3. Jointly training VQ2 and VQ3 together from a cold-start results in the word detectors being divided between those layers. While this experiment demonstrates that it is possible to jointly train two quantizers at once, the {2, 3} model learned the smallest total number of word detectors of any model. Additionally, we were unable to successfully train a cold-start {2, 3, 4} model; these experiments suggest that training quantizers one by one may be easier in general.
For all models beginning from a “∅” or “{2}” initial model, we do not observe any word detectors at either VQ2 or VQ3. However, a third quantizer at the VQ4 position in the “{2} → {2, 3} → {2, 3, 4}” model was able to capture words. We hypothesize that in all models not trained from a “{3}” initialization, word recognition is implicitly learned by the Res4 layer, and adding a quantizer to the output of this layer serves to make the categorical nature of those representations explicit.
A.3 WORD DETECTOR TABLES FOR VARIOUS MODELS
In Table 7, we show a sampling of 50 word-detecting codebook entries from the VQ layer of the “{3} → {2, 3}” model (many word detectors learned). Analagous results for the “{2} → {2, 3}” model (few word detectors learned) are shown in Table 8.
A.4 UNIT VISUALIZATION FOR INDIVIDUAL CAPTION SPECTROGRAMS
To provide a better intuitive understanding of what the units learned by our models look like, in Figures 5, 6, and 7, we display speech spectrograms for several Places caption fragments. Along with each spectrogram we display the time-aligned, ground-truth, word-level text (top transcription), the inferred unit sequence for the VQ4 layer (middle transcription), and the unit sequence for the VQ3 (bottom transcription) layer. All VQ unit alignments in these figures are derived from the “{2} → {2, 3} → {2, 3, 4}” model.

Review summaries:
The paper proposes a method to learn discrete linguistic units in a low-resource setting using speech paired with images (no labels). The visual grounding signal is different from other recent work, where a reconstruction objective was used to learn discrete representations in unsupervised neural networks. In contrast to other work, a hierarchy of discretization layers are also considered, and the paper shows that, with appropriate initialization, higher discrete layers capture word-like units while lower layers capture phoneme-like units.