Title: LEARNING HIERARCHICAL DISCRETE LINGUISTIC UNITS FROM VISUALLY-GROUNDED SPEECH

Abstract: In this paper, we present a method for learning discrete linguistic units by incorporating vector quantization layers into neural models of visually grounded speech. We show that our method is capable of capturing both word-level and sub-word units, depending on how it is configured. What differentiates this paper from prior work on speech unit learning is the choice of training objective. Rather than using a reconstruction-based loss, we use a discriminative, multimodal grounding objective which forces the learned units to be useful for semantic image retrieval. We evaluate the sub-word units on the ZeroSpeech 2019 challenge, achieving a 27.3% reduction in ABX error rate over the top-performing submission, while keeping the bitrate approximately the same. We also present experiments demonstrating the noise robustness of these units. Finally, we show that a model with multiple quantizers can simultaneously learn phone-like detectors at a lower layer and word-like detectors at a higher layer. We show that these detectors are highly accurate, discovering 279 words with an F1 score of greater than 0.5.

1. INTRODUCTION:

By 8 months of age, human infants learn to recognize not only the names of their caregivers and common objects, but also the contrast between the different vowels and consonants which comprise these words (Dupoux, 2018) . Nearly all toddlers learn to carry a conversation long before they can read and write. Humans learn to model the discrete, hierarchical, and compositional nature of their native language not from written text, but from speech audio -a continuous, time-varying waveform which is the product not only of the underlying words which were spoken, but also the physical properties of the speaker's vocal tract, the speaker's health and emotional state, and the noise and reverberation present in the environment. The question of how such a complex symbolic system is inferred from continuous and noisy sensory input data is of interest not only to the cognitive science community, but also to machine learning researchers who aim to reproduce this ability with computers. A more comprehensive understanding of human language acquisition has practical significance in real-world applications, such as automatic speech recognition (ASR) and natural language understanding (NLU) systems. In the past several decades, enormous progress has been made in speech recognition research, and nowadays ASR systems are able to achieve human-level accuracy in many domains (Chiu et al., 2018) . Unfortunately, the techniques that have been developed to achieve these levels of performance are extremely data-hungry, requiring many thousands of hours of speech audio recordings for training. Since supervised machine learning algorithms form the basis of ASR training, the data also needs to be annotated by expert humans. Due to the immense cost of collecting and annotating speech data, ASR technology currently exists for approximately 120 (Google, 2019) out of the nearly 7,000 (Lewis et al., 2016) human languages spoken worldwide. It is highly unlikely that purely supervised machine learning techniques will be able to scale to include all human languages, necessitating the development of alternative methods by researchers which are able to function with far fewer annotations, or even no annotations at all. Because human beings provide an existence proof of language acquisition from speech completely without language supervision, it is plausible that this ability could be replicated by a machine learning algorithm.

In this paper, we present a method for discovering discrete and hierarchical representations of speech units both at the sub-word level and the word level. Previously proposed linguistic unit discovery methods have only leveraged the speech audio modality in isolation, relying on objective functions that attempt to capture statistical regularities within the speech signal. The key innovation in our work is that we discover units by training models with explicit discretization layers to associate speech waveforms with visual images using a cross-modal grounding objective. This forces our models to learn representations which capture semantic information at the highest layers of the network. Because semantics are predominantly carried by words, and words are composed of subword units (such as phones and syllables), the visual grounding objective indirectly forces the model to learn speaker-and noise-invariant representations of speech units. By incorporating trainable quantization layers into our networks, we are able to capture these units in discrete inventories. Whether these units correspond to word-like or sub-word units depends on where the quantization layers are inserted, and how they are trained.

2. RELATED WORK:

Prior work on unsupervised modeling of the speech signal has generally focused on learning representations which either disentangle or isolate the latent factors that are of interest for downstream tasks. In most cases the primary latent factor of interest is the phonetic or lexical identity of a given segment of speech, but other factors, such as the identity of the speaker, are sometimes of interest as well. Because the factors of interest are often inherently discrete (e.g. words and phones), many of the proposed approaches attempt to perform segmentation and clustering of the surface features in one way or another. One family of techniques is based upon Segmental Dynamic Time Warping (S-DTW) (Park & Glass, 2005; 2008; Jansen et al., 2010; Jansen & Van Durme, 2011) , which uses a self-comparison algorithm to identify relatively long duration (on the order of a second) patterns which frequently reoccur in a speech corpus; these patterns tend to capture words or short phrases. A different line of work employs probabilistic graphical models to jointly segment and cluster the speech signal (Varadarajan et al., 2008; Zhang & Glass, 2009; Gish et al., 2009; Lee & Glass, 2012; Siu et al., 2014; Lee et al., 2015; Ondel et al., 2016; Kamper et al., 2016; 2017a) . With an appropriately designed model, it is possible to learn multiple, hierarchical categories of speech units. However, in order to enable efficient inference, the conditional distributions of these models tend to be simple and therefore have limited modeling power.

Deep neural network models have been successfully used to learn powerful speech representations using weakly or unsupervised objectives (Thiolliere et al., 2015; Kamper et al., 2015; Hsu et al., 2017a; b; Hsu & Glass, 2018; Holzenberger et al., 2018; Milde & Biemann, 2018; van den Oord et al., 2018; Chung et al., 2019; Pascual et al., 2019) . These representations have predominantly been continuous in nature, as discrete latent variables are not trivially compatible with backpropagation. To obtain discrete representations, a post-hoc clustering step can be applied to the continuous representations (Kamper et al., 2017b; Feng et al., 2019) . More recently, several papers have proposed ways of directly incorporating discrete variables into neural network models, including using Gumbel-Softmax (Eloff et al., 2019b) or straight-through estimators (van den Oord et al., 2017; Chorowski et al., 2019; Razavi et al., 2019) .

A different method for learning meaningful representations of speech is via a multimodal grounding objective, which encourages the learning of speech representations that are predictive of the contextual information contained in a separate but accompanying modality, such as vision. Visual grounding of speech is a form of self-supervised learning (Virginia de Sa, 1994), which is powerful in part because it offers a way of training models with a discriminative objective that does not depend on traditional transcriptions or annotations. The first work in this direction relied on phone strings to represent the speech (Roy & Pentland, 2002; Roy, 2003) , but more recently this learning has been shown to be possible directly on the speech signal (Synnaeve et al., 2014; Harwath & Glass, 2015; Harwath et al., 2016) . Subsequent work on visually-grounded models of speech has investigated improvements and alternatives to the modeling or training algorithms (Leidal et al., 2017; Kamper et al., 2017c; Havard et al., 2019a; Merkx et al., 2019; Chrupała et al., 2017; Scharenborg et al., 2018; Kamper et al., 2019b; a; Surís et al., 2019; Ilharco et al., 2019; Eloff et al., 2019a) , application to multilingual settings (Harwath et al., 2018a; Kamper & Roth, 2017; Azuh et al., 2019; Havard et al., 2019a) , analysis of the linguistic abstractions, such as words and phones, which are learned by the models (Harwath & Glass, 2017; Harwath et al., 2018b; Drexler & Glass, 2017; Alishahi et al., 2017; Harwath et al., 2019; Harwath & Glass, 2019; Havard et al., 2019b) , and the impact of jointly training with textual input (Holzenberger et al., 2019; Chrupała, 2019; Pasad et al., 2019) . Representations learned by models of visually grounded speech are also well-suited for transfer learning to supervised tasks, being highly robust to noise and domain shift (Hsu et al., 2019) .

3. DATA AND MODELS:



3.1. DATASET:

For training our models, we utilize the MIT Places 205 dataset (Zhou et al., 2014) and their accompanying spoken audio captions (Harwath et al., 2016; 2018b) . The caption dataset contains approximately 400,000 spoken audio captions, each of which describes a different Places image. These captions are free-form spontaneous speech, collected from over 2,500 different speakers and covering a 40,000 word vocabulary. The average caption duration is approximately 10 seconds, and each caption contains on average 20 words. For vetting our models during training, we use a held-out validation set of 1,000 image-caption pairs.

3.2. NEURAL MODELS OF VISUALLY-GROUNDED SPEECH:

We base our model upon the Residual Deep Audio-Visual Embedding network (ResDAVEnet) architecture (Harwath et al., 2019) , which contains two branches of fully convolutional networks, one for images and the other for audio. Each branch encodes samples of the corresponding modality into a d-dimensional space, regardless of the original dimensionality of the samples. This is achieved by applying global spatial mean pooling and global temporal mean pooling to the image branch output and the audio branch output, respectively. The image branch is adapted from ResNet50 (He et al., 2016) , where the final softmax layer and the preceding fully-connected layers are removed, replaced with a 1x1 linear convolutional layer in order to project the feature map to the desired dimension. To model the audio inputs, a 17-layer fully convolutional network with residual connections is used. The input is a log Mel-frequency spectrogram with 40 frequency bins and 25 ms-wide, Hammingwindowed frames with a shift of 10 ms. The first layer of this network is a 1-D convolution that spans the entire frequency axis of the spectrogram, while the remaining 16 convolutional layers are 1-D across the time axis. These 16 layers are divided into four residual blocks of 4 layers each, and downsampling between these blocks is accomplished by applying the first convolution of each block with a stride of 2. For full details of the model, refer to Harwath et al. (2019) .

3.3. LEARNING HIERARCHICAL DISCRETE UNITS WITH VECTOR QUANTIZING LAYERS:

Previous analyses reveal that ResDAVEnet-like models learn linguistic abstractions at different levels, including words (Harwath & Glass, 2017) and robust phonetic features (Harwath & Glass, 2019; Hsu et al., 2019) . To explicitly learn hierarchical discrete linguistic units within this framework, we propose to incorporate multiple vector quantization (VQ) layers (van den Oord et al., 2017) into the ResDAVEnet audio branch; we refer to this new architecture as ResDAVEnet-VQ.

VQ layers can be understood as a type of bottleneck, which constrain the amount of information that can flow through. While these layers have been used to learn discrete sub-word units (van den Oord et al., 2017; Chorowski et al., 2019; Razavi et al., 2019) , previous work injects VQ layers into autoencoders that are trained with a reconstruction loss. As a result, the embedding dimension of each code and the number of codes need to be carefully tuned (Liu et al., 2019) . When the embedding dimension is too low or the codebook size too small, the model does not have enough expressive power to capture linguistic variability. When it is too large, the model starts to encode non-linguistic information in order to improve reconstruction. In contrast, the learning signal of ResDAVEnet-VQ is provided by the visual-semantic grounding objective. Rather than encoding as much information about input as possible, the learned codes in ResDAVEnet-VQ only need to capture semantic information. Since semantics in speech are predominantly transmitted by words, and words are composed of sub-word units like phones, the grounding objective places pressure on the model to robustly infer both from speech. Since words and phones are inherently discrete symbols, representing them with learned discrete units may not even hurt the grounding performance.

Figure 1 illustrates the proposed ResDAVEnet-VQ model. We add a quantization layer after each of the first two residual blocks of the ResDAVEnet-VQ model, denoted as VQ2 and VQ3, respectively, with the intention that they should capture discrete sub-word-like and word-like units. A VQ layer is defined as E ∈ R K×D , where K represents the codebook size, and D represents the output dimensionality of the input features to the codebook. Denoting the t th temporal frame of the input to the quantization layer as x t , quantization is performed according to q t = E k,: , where k = arg min j ||x t -E j,: || 2 The quantized output is then fed as input to the subsequent residual block. As in van den Oord et al. (2017) , we use the straight-through estimator (Bengio et al., 2013) to compute the gradient passed from q t to x t . We use the exponential moving average (EMA) codebook updates proposed by van den Oord et al. (2017) . For instance, a model initialized from scratch with no VQ layers enabled is denoted as "∅", and a model initialized with that and with both layers enabled is denoted as "∅ → {2, 3}".

3.5. TRAINING WITH THE TRIPLET LOSS:

We train our models using the same loss function as Harwath et al. (2019) . This loss function blends two triplet loss terms (Weinberger & Saul, 2009) , one based on random sampling of negative examples, and the other based on semi-hard negative mining (Jansen et al., 2018) , in order to find more challenging negative samples. Specifically, let the sets of output embedding vectors for a minibatch of B audio/image training pairs respectively be A = {a 1 , . . . , a B } and I = {i 1 , . . . , i B }. To compute the randomly-sampled triplet loss term, we select impostor examples for the j th input according to āj ∼ UniformCategorical({a 1 , . . . , a B }\a j ) and īj ∼ UniformCategorical({i 1 , . . . , i B }\i j ).

The randomly-sampled triplet loss is then computed as:

EQUATION

For the semi-hard negative triplet loss, we first define the sets of impostor candidates for the j th example as Âj = {a ∈ A|i T j a < i T j a j } and Îj = {i ∈ I|i T a j < i T j a j }. The semi-hard negative loss is then computed as:

EQUATION

Finally, the overall loss function is computed by combining the two above losses, L = L s + L h , which was found by (Harwath et al., 2019) to outperform either loss on its own.

3.6. IMPLEMENTATION DETAILS:

All of our models were trained for 180 epochs using the Adam optimizer (Kingma & Ba, 2014) with a batch size of 80. We used an exponentially decaying learning rate schedule, with an initial value of 2e-4 that decayed by a factor of 0.95 every 3 epochs. Following van den Oord et al. ( 2017), we use an EMA decay factor of γ = .99 for training each VQ codebook. Our core experimental results all use a codebook size of 1024 vectors for all quantizers, but in the supplementary material we include experiments with smaller and larger codebooks. Following Chorowski et al. (2019) , the jitter probability hyperparameter for each quantization layer was fixed at 0.12. While we do not apply data augmentation to the input spectrograms, during training we perform standard data augmentation techniques to the images. We resize each raw image so that its smallest dimension is 256 pixels, and then we apply an Inception-style random crop which is resized to 224 pixels square. During training, we also flip each image horizontally with a probability of 0.5. During evaluation, the center 224 pixel square crop is always taken from the image. Finally, the RGB pixel values are mean and variance normalized. We trained each model on the Places audio caption train split, and computed the image and caption recall at 10 (R@10) scores on the validation split of the Places audio captions after each training epoch. The model snapshot that achieved the highest average R@10 score on the validation set from each training is used for all evaluation. To extract embeddings and units from our models, we simply perform a forward pass through the speech branch of the ResDAVEnet-VQ network and retain the outputs from the target layer at a uniform frame-rate.

The frame-rate is determined by the downsampling factor at the target layer relative to the input. For non-quantized layers, these outputs will be continuous embeddings. For quantized layers, these will be quantized embedding retrieved from the assigned entry in the codebook.

4. EXPERIMENTS:



4.1. SUB-WORD UNIT LEARNING ON THE ZEROSPEECH 2019 ABX TASK:

Evaluation metrics Learning unsupervised speech representations that are indicative of phonetic content is of high interest to the speech community, and recently has been the focus of the Ze-roSpeech Challenge (Versteegh et al., 2015; Dunbar et al., 2017; 2019) . One of the core evaluations is the minimal-pair ABX task (Schatz et al., 2013) , which aims to benchmark representations in terms of their discriminability between different sub-word speech units. In this task, a model is tasked with extracting representations for a triplet of speech waveform segments denoted by A, B, and X. A and B are constrained to be a triphone minimal pair; that is, both segments capture three phones, but differ only in the identity of their center phone. The third segment, X is chosen to contain the same underlying triphone sequence as A. Supposing f (•) denotes the model's mapping function from a waveform segment to a sequence of embedding vectors, the ABX error rate under a given similarity metric S(•, •) is defined as the fraction of ABX triples in which

S(f (A), f (X)) > S(f (B), f (X)

). An ABX error rate of 50% indicates random assignment, while an ABX of 0% reflects perfect phone discriminability. In the ZeroSpeech challenge, S(•, •) is implemented using Dynamic Time Warping (DTW) with various distance measures (cosine, KL, etc.). In our evaluation, we use the cosine distance.

The ZeroSpeech 2019 challenge in particular emphasizes on discovering an inventory of discrete sub-word units, rather than continuous representations. Therefore, in addition to an ABX error rate, a bitrate is also computed for each model which reflects the amount of information carried by the learned units. A lower bitrate can be achieved by having a more compact inventory of learned units or having a smaller number of codes per second. The full details of the evaluation can be found in Dunbar et al. (2019) . To be clear, all of our ResDAVEnet-VQ models were not trained on the ZeroSpeech training data, but instead on the Places audio captions, thus there is a domain mismatch between training and testing these models.

In addition to the frame-based bitrate and ABX scores computed by the ZeroSpeech 2019 evaluation toolkit, we implement our own extensions to these metrics. Because it is common for successive frames to be assigned to the same codebook entry and phonetic information is not encoded at a fixed frame rate, lossless run length encoding (RLE) can be a more reasonable measure of the bitrate of a frame-based model. RLE does not change the ABX score since it can be trivially inverted, but it does change the bitrate. For computing the RLE bitrate, we modify the bitrate calculation specified in Dunbar et al. (2019) so that a unique symbol is defined as the tuple (unit, length) where length is the number of frames assigned to a given unit with in a segment. We also consider segment-based ABX and bitrate, which is similar to the RLE metrics except in this case we outright discard the frame length information. This typically results in an even greater reduction in bitrate, but also an accompanying deterioration in ABX score.

Baseline models:

In Table 1 , we compare our results to those derived from two of the topperforming submissions to the ZeroSpeech 2019 challenge: a re-implementation of WaveNet-VQ (Chorowski et al., 2019) provided by Cho et al. (2019) and FHVAE-DPGMM (Feng et al., 2019) .

Using the code accompanied with the WaveNet-VQ submission, we were able to train their model on the set of 400,000 Places audio captions to make a fairer comparison with our ResDAVEnet-VQ models in terms of the amount of speech data used. In addition, when trying to reproduce the reported WaveNet-VQ results, we obtain better performance than previously reported by training for more steps. Table 1 shows that WaveNet-VQ achieves similar bitrates regardless of the training data. However, ABX deteriorates from 19.98 to 24.87, implying the model cannot utilize data of a larger scale but out-of-domain relative to the test set. A similar degradation when testing on out-of-domain data with FHVAE models was observed in Hsu et al. (2019) . We did not re-train the model submitted by Feng et al. (2019) , and instead compare against the scores reported in Dunbar et al. (2019) .

ABX discrimination without using quantization Our first experiment investigates exactly which layer in the ResDAVEnet-VQ model is most suited for ABX phone discrimination, and would thus make a good candidate for learning of quantized sub-word units. The leftmost plot in Figure 2 shows that layers 2 and 3 of a ResDAVEnet-VQ model without any quantization enabled perform 77, 11.35, 10.86, and 14 .05 for the conv1, res2, res3, and res4 layers.

the best in terms of ABX error rate on the ZeroSpeech 2019 English test set; the exact numbers for this model are displayed in the caption of Figure 2 . Because layers 2 and 3 achieve the lowest ABX error rates without quantization, we focus our attention on the impact of quantization there.

Quantizing one layer When quantizing only one layer, we examine quantization of layer 2 vs. layer 3, and using cold-start training vs. warm-start initialization from model "∅". The ABX and bitrate results for these models, as well as the R@10 scores on the Places validation set, are shown in Table 1 . In all cases, quantization applied at the output of layer 2 achieves a better ABX score than quantization at layer 3, but VQ3 achieves a better bitrate. Quantization barely impacts the performance of layer 2, whose ABX score very slightly rises from 11.35 to 11.79. Warm-start initialization is beneficial to R@10 and ABX score in both cases, but we notice an intriguing anomaly when applying cold-start quantization to layer 3: the ABX score deteriorates significantly, rising from 10.86 in the case of the non-quantized model to 38.21. This indicates that while VQ2 is capable of learning a finite inventory of units that are highly predictive of phonetic identity from either a warm-start or cold-start initialization, cold-start training of VQ3 results in very little phonetic information captured by the quantizer. Interestingly, this model is still learning to infer visual semantics from the speech signal, as evidenced by a high R@10 score; we later show in Section 4.2 that the reason for this anomaly is because cold-start training of VQ3 results in the learning of word detectors. In all cases except for model "{3}", we note that the ABX scores achieved by our models are significantly better than the baselines. Our best model in terms of ABX ("∅ → {2}") achieves a 41.0% reduction in ABX over the WaveNet-VQ baseline, at a cost of a 132.3% increase in RLE bitrate; however, model "∅ → {3}" achieves a 24.7% reduction in ABX error rate with only a 2.4% increase in RLE bitrate. These results do not constitute a fair comparison, however, because the WaveNet-VQ and ResDAVEnet-VQ models were trained on different datasets; when training the WaveNet-VQ model on the same set of audio captions used to train ResDAVEnet-VQ (but without the accompanying images, since WaveNet-VQ is not a multimodal model), the ABX error rate increases to 24.87%, tipping the results even more in favor of the ResDAVEnet-VQ models.

Quantizing two layers Quantizing multiple layers at once offers the possibility of learning a hierarchy of units. Thus, we aim to capture phonetic information in a lower layer quantizer and word-level information at a higher layer quantizer. Cold-start training of two quantizers ("{2, 3}") results in a significant drop in ABX performance for both VQ2 and VQ3, but also a drop in R@10 on the Places validation set. We see much better results in terms of R@10 and ABX for the remaining 3 models which were initialized from the "∅" model or a model with only one quantizer enabled; for example, model "{2} → {2, 3}" achieves an ABX of 14.52 with an RLE bitrate of 136.11, representing a 27.3% ABX improvement over the best baseline while keeping the bitrate approximately the same. We see in model "{3} → {2, 3}" that the same phenomenon observed with model "{3}" persists: VQ3 achieves relatively poor ABX, despite a high overall R@10 and strong ABX with VQ2 at 13.55%. We confirm in Section 4.2 that the VQ3 layer of model "{3} → {2, 3}" does indeed capture word-level information, indicating that this model has successfully localized phonetic unit identity in the second layer and lexical unit identity in the third layer. Overall, our results suggest that when learning hierarchical quantized representations with a ResDAVEnet-VQ model, the nature of the representations learned is highly dependent on the training curriculum. Training and testing on noisy data In Hsu et al. (2019) , it was shown that representations learned by a ResDAVEnet model were far more robust to train/test domain mismatch in terms of background noise, channel characteristics, and speaker identity than standard spectral features when training a supervised speech recognizer. Here, we examine whether this robustness is also exemplified by the quantized versions of this model. We construct three additional test sets using the ZeroSpeech 2019 English testing data by adding noise sampled from the AudioSet (Jansen et al., 2018) dataset.

For each ZeroSpeech testing waveform, we randomly sampled an AudioSet waveform of the same duration and performed linear mixing with a signal-to-noise ratio (SNR) selected randomly within a specified range. We construct low, medium, and high noise testing sets, corresponding to SNRs of 20-30 dB, 10-20 dB, and 0-10 dB. We then perform the ABX discrimination task on these noisy waveforms, displaying the results in Table 2 . We find that for all models, a worsening SNR results in a deterioration in ABX performance. However, the ResDAVEnet-VQ models prove to be far more noise robust than the Wavenet-VQ model; even in the high noise testing set, the best ResDAVEnet-VQ model achieves an ABX of 23.96%, while the WaveNet-VQ models degrade to nearly-random ABX scores of 42.55% and 42.67%.

Given that a ResDAVEnet-VQ model trained on the "clean" Places Audio captions is highly robust to additive noise on the ABX discrimination task, we investigated whether adding noise to the Places Audio captions themselves would result in an even higher degree of noise robustness. To that end, we followed a similar data augmentation approach to create a noisy version of the Places Audio captions, where the SNR of each caption was randomly chosen to sit within the range of 0-30 dB.

The bottom half of Table 2 shows the results of training several ResDAVEnet-VQ models on the noisy Places Audio captions and testing on the clean and noisy ZeroSpeech ABX tasks. In general, we observe a degradation ABX score in the clean conditions, but with a significantly higher degree of noise robustness in the noisier conditions.

Visualization of learned units To better measure the correspondence between the VQ units and English phones, we compute corpus-level co-occurrence statistics (at the frame-level) across the TIMIT training set, excluding the sa dialect sentences. To facilitate visualization, we use the "∅ → {2}" model with a codebook size of 128. We display the conditional probability matrix P (phone|unit) in Figure 3 , with the rows and columns ordered via spectral co-clustering with 10

ResDAVEnet-VQ Code Indices clusters in order to group together phones that share similar sets of VQ codes. Visually, there is a strong mapping between TIMIT phone labels and ResDAVEnet-VQ codes. In some cases, redundant codes are used for the same phone label (this is especially the case for the silence label), and in other cases we see that phones belonging to the same manner class often tend to share codebook units. We can numerically quantify the mapping between the phone and unit labels with the normalized mutual information measure (NMI), which we found to be .378 in this case. We also include several caption spectrograms with their time-aligned unit sequences in Figures 5, 6 , and 7 in the supplementary material.

Table 3 : Performance of the VQ3 layer from the "{3} → {2, 3}" model when codes are treated as word detectors. Codes are ranked by the highest F1 score among the retrieved words for a given code. Word hypotheses for a given code are ranked by the F1 score. P denotes precision, R recall, and occ the number of co-occurrences of the code and word in the data. 1 , all of the ResDAVEnet-VQ models which underwent cold-start training of VQ3 exhibited a similar phenomenon in which the ABX error rate of that layer was particularly high, despite the model performing well at the image-caption retrieval task. We hypothesized that this could be due to VQ3 learning to recognize higher level linguistic units, such as words. To examine this empirically, we inferred the VQ3 unit sequence for every audio caption in the Places Audio training set according to several different models. Using the estimated word-level transcriptions of the utterances (provided by the Google SpeechRecognition API), we computed precision, recall, and F1 scores for every unique (word, VQ3 code) pair for a given model and quantization layer.

We then ranked the VQ codes in descending order according to their maximum F1 score for any word in the vocabulary. Table 3 shows a sampling of these statistics for model "{3} → {2, 3}". In the supplementary material, we include many more examples for this model in examples for the "{2} → {2, 3}" model (which did not learn VQ3 word detectors) in Table 8 . It should be emphasized that these models are exactly the same in all respects, except for the order in which their quantizers were trained.

We examine the overall performance of VQ3 as a word detector for these models in Figure 4 . The right hand side of Figure 4 displays the number of VQ3 codes whose maximum F1 score is above a given threshold, while the left hand side shows the distribution of precision and recall scores for the top 250 words ranked by F1. This gives an approximate indication of how many VQ3 codes have learned to specialize as detectors for a specific word. We see that the VQ3 layer of model "{3} → {2, 3}" learns 279 codebook entries with an F1 score above 0.5. In contrast, the VQ3 layer of model "{2} → {2, 3}" learns only a handful of word-detecting codebook entries with an F1 of greater than 0.5. This experiment supports the notion that the reason for the poor ABX performances of the VQ3 layer in models "{3}" and "{3} → {2, 3}" is in fact due to its specialization for detecting specific words, and that this specialization only emerges when the VQ3 layer is learned before the VQ2 layer. Section A.2 in the supplementary material examines this phenomenon in greater experimental detail.

5. CONCLUSIONS:

In this paper, we demonstrated that the neural vector quantization layers proposed by van den Oord et al. ( 2017) can be integrated into the visually-grounded speech models proposed by Harwath et al. (2019) . This resulted in the ability of the speech model to directly represent speech units, such as phones and words, as discrete latent variables. We presented extensive experiments and analysis of these learned representations, demonstrating significant improvements in phone discrimination ability over the current state-of-the-art models for sub-word speech unit discovery. We demonstrated that these units are also far more robust to noise and domain shift than units derived from previously proposed models. These results supported the notion that semantic supervision via a discriminative, multimodal grounding objective has the potential to be more powerful than reconstruction-based objectives typically used in unsupervised speech models.

We also showed how multiple vector quantizers could be employed simultaneously within a single ResDAVEnet-VQ model, and that these quantizers could be made to specialize in learning a hierarchy of speech units: specifically, phones in the lower quantizer and words in the upper quantizer. Our analysis showed that hundreds of codebooks in the upper quantizer learned to perform as word detectors, and that these detectors were highly accurate. Our experiments also revealed that this behavior only emerged when VQ3 was trained before VQ2. These results suggest the importance of the learning curriculum, which should be more deeply investigated in future work. Future work should attempt to make explicit what kind of compositional rules are implicitly encoded by these models when mapping sequences of codes from the lower quantizer to word-level units in the upper quantizer; the automatic derivation of a sub-word unit inventory, vocabulary, and pronunciation lexicon could serve as the starting point for a fully unsupervised speech recognition system. Future work should also investigate whether layers above VQ3 could be made to learn even higher-level linguistic abstractions, such as grammar, syntax, and compositional reasoning.

A APPENDIX:

A.1 VARYING THE CODEBOOK SIZE.

In Table 4 , we examine the impact of varying the codebook size of model "∅ → {2}" from 128 through 2048. We find that the ABX score is best for 1024 codebook vectors, although the performance is quite good for all models. Unsurprisingly, models with smaller codebooks also achieve lower bitrates. In Section 4.2, we showed that cold-start training of the VQ3 layer caused its codebook vectors to specialize as word detectors, whereas warm-start training did not. Our subsequent experiments (Table 6 ) revealed that when adding a third quantization layer at the Res4 position to a model that did not learn word detectors at VQ3, the VQ4 layer did in fact learn many word detectors. This suggests implicit word recognition ability can be localized at different layers in the ResDAVEnet audio model, and exactly where it emerges depends upon the VQ training curriculum. We hypothesize that this is due in part to two factors:

1. In a warm-start model, whatever type of information (subword-like, word-like) the continuous model learned to encode at a particular convolutional layer (or residual block) does not change after a quantizer is appended to that layer. 2. In a cold-start model, each active quantization layer forms a potential bottleneck, restricting the amount of information that is able to pass through to subsequent layers.

According to this hypothesis, if word-level recognition tends to emerge at a particular layer in an unconstrained network with no quantization bottleneck, it will stay there when quantization is introduced for fine-tuning. However, when a quantization bottleneck is introduced from the very beginning of training, the gradient flowing down into the lower network layers is more constrained during the initial training epochs (when the gradient tends to be the largest). This may have the effect of steering the optimizer into a different part of the parameter space, in which word recognition occurs at a different layer than it otherwise would.

We present results from two experiments that support this view. In Table 5 , we show the ABX scores of the Res3 layer prior to quantization (if present) during the course of three different training curricula resulting in a "{2, 3}" final model. We observe that the ABX error rate changes very little within each individual curriculum. This indicates that the initial model sets the stage for which layer learns to capture phonetic information with the highest salience, and that subsequent training steps do not tend to move this information elsewhere. Here, we claim that a codebook vector belonging to a particular VQ layer has learned to be a word detector if its F1 score for any word appearing in the test set exceeds 0.5 (as measured on the test set). There are several interesting things to note here. First, we only observe a significant number of word detectors at the VQ3 layer when that layer is trained from a cold-start. Even when adding a second VQ layer as in the "{3} → {2, 3}" model, these detectors remain at VQ3.

Jointly training VQ2 and VQ3 together from a cold-start results in the word detectors being divided between those layers. While this experiment demonstrates that it is possible to jointly train two quantizers at once, the {2, 3} model learned the smallest total number of word detectors of any model. Additionally, we were unable to successfully train a cold-start {2, 3, 4} model; these experiments suggest that training quantizers one by one may be easier in general.

For all models beginning from a "∅" or "{2}" initial model, we do not observe any word detectors at either VQ2 or VQ3. However, a third quantizer at the VQ4 position in the "{2} → {2, 3} → {2, 3, 4}" model was able to capture words. We hypothesize that in all models not trained from a "{3}" initialization, word recognition is implicitly learned by the Res4 layer, and adding a quantizer to the output of this layer serves to make the categorical nature of those representations explicit.

A.3 WORD DETECTOR TABLES FOR VARIOUS MODELS:

In Table 7 , we show a sampling of 50 word-detecting codebook entries from the VQ layer of the "{3} → {2, 3}" model (many word detectors learned). Analagous results for the "{2} → {2, 3}" model (few word detectors learned) are shown in Table 8 .

A.4 UNIT VISUALIZATION FOR INDIVIDUAL CAPTION SPECTROGRAMS:

To provide a better intuitive understanding of what the units learned by our models look like, in Figures 5, 6 , and 7, we display speech spectrograms for several Places caption fragments. Along with each spectrogram we display the time-aligned, ground-truth, word-level text (top transcription), the inferred unit sequence for the VQ4 layer (middle transcription), and the unit sequence for the VQ3 (bottom transcription) layer. All VQ unit alignments in these figures are derived from the "{2} → {2, 3} → {2, 3, 4}" model. 956, 265, 80, 401, 262, 762, 246, 774, 828, 386, bottom transcription) .

Figure 6 : Two different captions containing word "train". In both cases, the VQ4 layer infers the same unit sequence (680, 248, top transcription) surrounding the word "train". The VQ3 alignments contain the same common subsequence (358, 306, 908, 564, 950, 770, bottom transcription) . Notice that the same (358, 306) VQ3 unit sequence is aligned to the /tr/ phone cluster at the beginning of both instances of the word "train," as well as the /tr/ at the beginning of both instances of "tree" in Figure 7 . Unit 358 is also found covering the /tr/ at the beginning of the word "tracks" in the topmost spectrogram (although unit 306 is absent in this case).

Figure 7 : Two different captions containing word "trees". In both cases, the VQ4 layer infers the same unit sequence (8, 412, 50, top transcription) surrounding the word "trees". The VQ3 alignments contain the same common subsequence (358, 306, 648, 677, 730, bottom transcription) .

Notice that the (358, 306) VQ3 unit sequence is aligned to the /tr/ phone cluster at the beginning of both instances of "trees," and this same unit sequence is inferred for the /tr/ phone sequence in both instances of "train" in Figure 6 .

:



Figures and Tables:

Figure 1: Diagram of the ResDAVEnet-VQ model. On the left, we show the placement of the vector quantization blocks in the audio branch. Note that each "Res" block is comprised of a stack of multiple sub-layers (see Harwath et al. (2019) for details). The right half of the figure depicts the quantization mechanism of each VQ block, as well as the bypass path when the block is disabled.

Figure2: R@10 and ABX tracked at various training epochs. The "∅" model achieves a final R@10 of .735, with ABX scores of19.77, 11.35, 10.86, and 14.05 for the conv1, res2, res3, and res4 layers.

Figure 3: Conditional probability matrix displaying P (phone|unit) using the "∅ → {2}" model with a VQ2 codebook size of 128. For visualization, we saturate the color scaling at probability 0.5.

Figure 4: Visualization of the precision, recall, and F1 scores of individual VQ3 codes when treated as word detectors on the Places Audio captions.

Figure5: Two different captions containing the phrase "many people." In both cases, the VQ4 layer infers the same unit sequence (872, 360, 712, middle transcription) beneath the phrase. The VQ3 units are somewhat noisier, but contain the common subsequence(956, 265, 80, 401, 262, 762, 246,  774, 828, 386, bottom transcription).

Comparison of R@10, ABX scores, and bit-rates between different configurations and baseline models trained on ZeroSpeech 2019 data or Places Audio Caption. All quantizers reflected in this table used a codebook size of 1,024 vectors. We do not compute RLE or segment scores for the FHVAE-DPGMM model, since we did not re-implement that model.

Table content: <table><tr><td>Model ID</td><td colspan="2">Layer R@10</td><td colspan="5">Frame-Based ABX Bitrate RLE Bitrate ABX Bitrate Segment-Based</td></tr><tr><td>FHVAE-DPGMM (ZS)</td><td>N/A</td><td>N/A</td><td colspan="2">21.67 413.23</td><td>-</td><td>-</td><td>-</td></tr><tr><td>WaveNet-VQ (ZS)</td><td>N/A</td><td>N/A</td><td colspan="2">19.98 151.55</td><td>136.74</td><td colspan="2">20.48 126.17</td></tr><tr><td>WaveNet-VQ (PA)</td><td>N/A</td><td>N/A</td><td colspan="2">24.87 149.00</td><td>136.27</td><td colspan="2">25.23 126.22</td></tr><tr><td>"∅"</td><td>Res2 Res3</td><td>.735</td><td>11.35 10.86</td><td>N/A N/A</td><td>N/A N/A</td><td>N/A N/A</td><td>N/A N/A</td></tr><tr><td>"{2}"</td><td>VQ2</td><td>.753</td><td colspan="2">12.33 433.30</td><td>361.09</td><td colspan="2">12.78 332.86</td></tr><tr><td>"∅ → {2}"</td><td>VQ2</td><td>.760</td><td colspan="2">11.79 390.61</td><td>317.66</td><td colspan="2">12.66 289.11</td></tr><tr><td>"{3}"</td><td>VQ3</td><td>.734</td><td colspan="2">38.21 213.92</td><td>129.65</td><td colspan="2">38.68 108.84</td></tr><tr><td>"∅ → {3}"</td><td>VQ3</td><td>.794</td><td colspan="2">15.04 182.93</td><td>140.04</td><td colspan="2">16.53 121.26</td></tr><tr><td>"{2, 3}"</td><td>VQ2 VQ3</td><td>.667</td><td colspan="2">25.62 408.75 32.23 218.76</td><td>258.37 156.69</td><td colspan="2">26.32 217.58 32.49 136.90</td></tr><tr><td>"∅ → {2, 3}"</td><td>VQ2 VQ3</td><td>.787</td><td colspan="2">13.15 405.43 14.95 199.91</td><td>334.39 172.05</td><td colspan="2">13.30 303.03 15.60 159.07</td></tr><tr><td>"{2} → {2, 3}"</td><td>VQ2 VQ3</td><td>.764</td><td colspan="2">12.51 415.13 14.52 167.84</td><td>341.85 136.11</td><td colspan="2">13.06 311.82 15.68 121.17</td></tr><tr><td>"{3} → {2, 3}"</td><td>VQ2 VQ3</td><td>.760</td><td colspan="2">13.55 421.23 33.70 208.63</td><td>271.91 117.37</td><td colspan="2">14.38 232.87 33.58 98.29</td></tr></table>

ABX scores and RLE bitrates for various SNRs on the noisy ZeroSpeech19 English test set. "R-B" stands for "RLE-Bitrate," and (n) denotes a model trained on the noisy Places Audio dataset. For the WaveNet-VQ models, (ZS) and (PA) respectively denote training on the ZeroSpeech 19 English training set, and the clean Places Audio dataset.

Table content: <table><tr><td>Model</td><td>Layer</td><td colspan="2">Clean</td><td colspan="2">20-30 dB</td><td colspan="2">10-20 dB</td><td colspan="2">0-10 dB</td></tr><tr><td/><td/><td>ABX</td><td>R-B</td><td>ABX</td><td>R-B</td><td>ABX</td><td>R-B</td><td>ABX</td><td>R-B</td></tr><tr><td>WaveNet-VQ (ZS)</td><td>N/A</td><td colspan="8">19.98 136.74 21.22 141.07 27.51 144.28 42.55 126.96</td></tr><tr><td>WaveNet-VQ (PA)</td><td>N/A</td><td colspan="8">24.87 136.27 27.18 137.70 33.29 132.34 42.67 110.50</td></tr><tr><td>"∅"</td><td colspan="2">Res2 11.35</td><td>N/A</td><td>11.63</td><td>N/A</td><td>13.17</td><td>N/A</td><td>19.44</td><td>N/A</td></tr><tr><td>"∅"</td><td colspan="2">Res3 10.86</td><td>N/A</td><td>11.16</td><td>N/A</td><td>12.96</td><td>N/A</td><td>19.43</td><td>N/A</td></tr><tr><td>"∅ → {2}"</td><td>VQ2</td><td colspan="8">11.79 317.66 12.15 325.40 14.62 332.21 23.96 327.15</td></tr><tr><td>"{2} → {2, 3}"</td><td>VQ2</td><td colspan="8">12.51 341.85 12.56 350.28 14.82 362.73 25.02 330.54</td></tr><tr><td>"{2} → {2, 3}"</td><td>VQ3</td><td colspan="8">14.52 136.11 14.73 137.68 17.44 143.14 27.68 133.13</td></tr><tr><td>"{3} → {2, 3}"</td><td>VQ2</td><td colspan="8">13.55 271.91 13.65 272.46 15.69 267.70 24.06 244.52</td></tr><tr><td>"{3} → {2, 3}"</td><td>VQ3</td><td colspan="8">33.70 117.37 32.56 118.22 34.65 115.40 39.82 102.48</td></tr><tr><td>"∅" (n)</td><td colspan="2">Res2 13.32</td><td>N/A</td><td>12.30</td><td>N/A</td><td>12.97</td><td>N/A</td><td>16.91</td><td>N/A</td></tr><tr><td>"∅" (n)</td><td colspan="2">Res3 11.85</td><td>N/A</td><td>11.90</td><td>N/A</td><td>12.44</td><td>N/A</td><td>16.09</td><td>N/A</td></tr><tr><td>"∅ → {2}" (n)</td><td>VQ2</td><td colspan="8">12.64 342.53 12.20 348.57 13.34 359.43 18.82 373.60</td></tr><tr><td>"{2} → {2, 3}" (n)</td><td>VQ2</td><td colspan="8">13.42 365.89 13.71 359.14 14.57 370.67 18.78 392.10</td></tr><tr><td>"{2} → {2, 3}" (n)</td><td>VQ3</td><td colspan="8">14.39 179.19 14.92 180.36 15.38 182.27 19.58 188.32</td></tr><tr><td>"{3} → {2, 3}" (n)</td><td>VQ2</td><td colspan="8">16.52 223.28 16.47 223.61 17.75 225.72 22.68 230.01</td></tr><tr><td>"{3} → {2, 3}" (n)</td><td>VQ3</td><td colspan="8">26.21 187.31 25.88 187.92 26.34 188.49 31.26 191.28</td></tr></table>



Table content: <table><tr><td>, as well as</td></tr></table>

ABX scores and bitrates for various codebook sizes on the clean ZeroSpeech19 English test set, using the "∅ → {2}" model.

Table content: <table><tr><td colspan="6">Codebook size R@10 ABX Bitrate RLE-Bitrate Segment-ABX Segment-Bitrate</td></tr><tr><td>128</td><td>.772</td><td>14.25 295.65</td><td>212.27</td><td>15.42</td><td>179.38</td></tr><tr><td>256</td><td>.756</td><td>12.95 341.18</td><td>260.10</td><td>14.21</td><td>228.07</td></tr><tr><td>512</td><td>.761</td><td>12.59 363.95</td><td>288.64</td><td>13.10</td><td>259.94</td></tr><tr><td>1024</td><td>.760</td><td>11.79 390.61</td><td>317.66</td><td>12.66</td><td>289.11</td></tr><tr><td>2048</td><td>.768</td><td>12.41 360.04</td><td>283.68</td><td>13.15</td><td>254.23</td></tr><tr><td colspan="6">A.2 THE IMPACT OF THE VQ TRAINING CURRICULUM ON THE LOCALIZATION OF WORD</td></tr><tr><td>DETECTORS</td><td/><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td colspan="2">Table 5: ABX scores on the Ze-</td></tr><tr><td/><td/><td/><td/><td colspan="2">roSpeech 19 English test set using</td></tr><tr><td/><td/><td/><td/><td colspan="2">features derived from the output of</td></tr><tr><td/><td/><td/><td/><td colspan="2">the Res3 block of the ResDAVEnet</td></tr><tr><td/><td/><td/><td/><td colspan="2">audio branch (pre-quantization).</td></tr><tr><td/><td/><td/><td/><td>Model ID</td><td>Res3 ABX</td></tr><tr><td/><td/><td/><td/><td>"∅"</td><td>10.86</td></tr><tr><td/><td/><td/><td/><td>"∅ → {2}"</td><td>11.61</td></tr><tr><td/><td/><td/><td/><td>"∅ → {3}"</td><td>10.91</td></tr><tr><td/><td/><td/><td/><td>"∅ → {2, 3}"</td><td>12.68</td></tr><tr><td/><td/><td/><td/><td>"{2}"</td><td>11.45</td></tr><tr><td/><td/><td/><td/><td>"{2} → {2, 3}"</td><td>11.37</td></tr><tr><td/><td/><td/><td/><td>"{3}"</td><td>32.24</td></tr><tr><td/><td/><td/><td/><td>"{3} → {2, 3}"</td><td>28.33</td></tr></table>

The number of codebook vectors at a particular VQ layer that learned to be a detector for any word with an F1 score greater than 0.5.

Table content: <table><tr><td># Quantized Layers</td><td>Model ID</td><td colspan="2">VQ Layer # Word Detectors (F1 &gt; 0.5)</td></tr><tr><td>1</td><td>"{3}" "∅ → {3}"</td><td>3 3</td><td>210 20</td></tr><tr><td/><td>"{2, 3}"</td><td>2 3</td><td>93 100</td></tr><tr><td>2</td><td>"∅ → {2, 3}"</td><td>2 3</td><td>1 18</td></tr><tr><td/><td>"{2} → {2, 3}"</td><td>2 3</td><td>0 5</td></tr><tr><td/><td>"{3} → {2, 3}"</td><td>2 3</td><td>32 279</td></tr><tr><td/><td/><td>2</td><td>0</td></tr><tr><td>3</td><td>"{2} → {2, 3} → {2, 3, 4}"</td><td>3</td><td>8</td></tr><tr><td/><td/><td>4</td><td>253</td></tr></table>

displays the number of word detectors learned by various VQ layers across different training curricula.

Table content: <table/>

Performance of the VQ3 layer from the "{3} → {2, 3}" model when codes are treated as word detectors. Codes are ranked by the highest F1 score among the retrieved words for a given code. Word hypotheses for a given code are ranked by the F1 score.

Table content: <table><tr><td colspan="2">rank code</td><td>word</td><td>Top Hypotheses F1 P</td><td>R</td><td>occ</td><td>word</td><td colspan="2">Second Hypotheses F1 P</td><td colspan="2">R occ</td></tr><tr><td>1</td><td>918</td><td>pantry</td><td colspan="2">90.67 88.29 93.18</td><td>41</td><td>spice</td><td>3.96</td><td>2.20</td><td>20.00</td><td>1</td></tr><tr><td>2</td><td>596</td><td>kitchen</td><td colspan="2">90.08 91.59 88.63</td><td>304</td><td>countertop</td><td>1.64</td><td>0.84</td><td>29.63</td><td>8</td></tr><tr><td>3</td><td>88</td><td>classroom</td><td colspan="2">88.97 89.05 88.89</td><td colspan="2">72 classrooms</td><td>5.01</td><td colspan="2">2.57 100.00</td><td>2</td></tr><tr><td>4</td><td>58</td><td>baseball</td><td colspan="2">88.71 88.63 88.78</td><td>182</td><td>player</td><td>3.01</td><td>1.65</td><td>17.11</td><td>13</td></tr><tr><td>5</td><td>706</td><td colspan="3">background 87.86 91.93 84.14</td><td>838</td><td>ground</td><td>0.58</td><td>0.39</td><td>1.18</td><td>4</td></tr><tr><td>6</td><td>736</td><td>museum</td><td colspan="2">87.35 93.44 82.00</td><td>41</td><td>museums</td><td>5.47</td><td colspan="2">2.81 100.00</td><td>1</td></tr><tr><td>7</td><td>274</td><td>subway</td><td colspan="2">87.26 88.34 86.21</td><td>75</td><td>assembly</td><td>5.32</td><td>2.85</td><td>40.00</td><td>4</td></tr><tr><td>8</td><td colspan="4">116 construction 87.07 89.78 84.52</td><td colspan="2">131 constructed</td><td>2.43</td><td>1.25</td><td>38.46</td><td>5</td></tr><tr><td>9</td><td>892</td><td>walking</td><td colspan="2">87.06 87.57 86.55</td><td>412</td><td>walk</td><td>7.07</td><td>3.97</td><td>31.94</td><td>23</td></tr><tr><td>10</td><td>557</td><td>concrete</td><td colspan="2">86.53 90.98 82.50</td><td>99</td><td>concur</td><td>1.21</td><td colspan="2">0.61 100.00</td><td>1</td></tr><tr><td>11</td><td>48</td><td>desert</td><td colspan="2">86.50 90.30 83.01</td><td>171</td><td>dozen</td><td>2.76</td><td>1.49</td><td>18.18</td><td>2</td></tr><tr><td>12</td><td>534</td><td colspan="3">background 86.18 81.95 90.86</td><td>905</td><td>back</td><td>8.95</td><td>5.83</td><td>19.34</td><td>64</td></tr><tr><td>13</td><td>44</td><td>patio</td><td colspan="2">85.82 90.87 81.29</td><td>113</td><td>patios</td><td>1.56</td><td colspan="2">0.79 100.00</td><td>1</td></tr><tr><td>14</td><td>625</td><td colspan="3">background 85.17 92.92 78.61</td><td>783</td><td>back</td><td>1.63</td><td>1.01</td><td>4.23</td><td>14</td></tr><tr><td>15</td><td>732</td><td>closet</td><td colspan="2">84.92 94.64 77.01</td><td>67</td><td>closets</td><td>4.96</td><td>2.68</td><td>33.33</td><td>2</td></tr><tr><td>16</td><td>30</td><td>waterfall</td><td colspan="2">84.90 75.73 96.61</td><td>57</td><td>waterfalls</td><td>14.26</td><td colspan="2">7.68 100.00</td><td>7</td></tr><tr><td>17</td><td>388</td><td>courtyard</td><td colspan="2">84.89 92.16 78.69</td><td>48</td><td>graveyard</td><td>5.50</td><td>3.24</td><td>18.18</td><td>4</td></tr><tr><td>18</td><td>560</td><td>hospital</td><td colspan="2">84.70 91.48 78.85</td><td>41</td><td>horses</td><td>1.55</td><td>1.92</td><td>1.30</td><td>1</td></tr><tr><td>19</td><td>18</td><td>driveway</td><td colspan="2">84.56 90.10 79.66</td><td>47</td><td>driveways</td><td>3.52</td><td>1.82</td><td>50.00</td><td>1</td></tr><tr><td>20</td><td>598</td><td>palm</td><td colspan="2">84.39 82.08 86.84</td><td>99</td><td>plum</td><td>1.57</td><td colspan="2">0.79 100.00</td><td>1</td></tr><tr><td>21</td><td>85</td><td>yellow</td><td colspan="2">84.30 83.93 84.66</td><td>574</td><td>yellowish</td><td>1.98</td><td colspan="2">1.00 100.00</td><td>7</td></tr><tr><td>22</td><td>584</td><td colspan="3">playground 84.18 77.37 92.31</td><td>36</td><td>play</td><td>6.32</td><td>4.75</td><td>9.43</td><td>5</td></tr><tr><td>23</td><td>162</td><td>stadium</td><td colspan="2">83.82 84.50 83.15</td><td>74</td><td>boardwalk</td><td>9.12</td><td>4.91</td><td>63.16</td><td>12</td></tr><tr><td>24</td><td>769</td><td>bamboo</td><td colspan="2">83.79 93.68 75.79</td><td>72</td><td>baboons</td><td>2.03</td><td colspan="2">1.03 100.00</td><td>1</td></tr><tr><td>25</td><td>193</td><td>small</td><td colspan="2">83.55 90.46 77.63</td><td>791</td><td>smaller</td><td>2.15</td><td>1.10</td><td>50.00</td><td>15</td></tr><tr><td>26</td><td>412</td><td>podium</td><td colspan="2">83.53 76.73 91.67</td><td colspan="2">22 auditorium</td><td>8.69</td><td>5.82</td><td>17.14</td><td>6</td></tr><tr><td>27</td><td>108</td><td>highway</td><td colspan="2">83.52 79.58 87.88</td><td>58</td><td>highlights</td><td>5.44</td><td>2.87</td><td>50.00</td><td>3</td></tr><tr><td>28</td><td>394</td><td>church</td><td colspan="2">83.34 75.98 92.28</td><td>227</td><td>religious</td><td>6.34</td><td>3.45</td><td>39.39</td><td>13</td></tr><tr><td>29</td><td>661</td><td>distance</td><td colspan="2">83.32 78.96 88.19</td><td>351</td><td>lounge</td><td>1.63</td><td>0.86</td><td>14.71</td><td>5</td></tr><tr><td>30</td><td>708</td><td>distance</td><td colspan="2">82.97 96.32 72.86</td><td>290</td><td>farmland</td><td>1.35</td><td>0.68</td><td>55.56</td><td>5</td></tr><tr><td>31</td><td>14</td><td>gallery</td><td colspan="2">82.97 85.08 80.95</td><td>17</td><td>art</td><td colspan="2">11.39 12.15</td><td>10.71</td><td>9</td></tr><tr><td>32</td><td>996</td><td>large</td><td colspan="3">82.95 87.05 79.21 1753</td><td>very</td><td>2.83</td><td>1.72</td><td>8.01</td><td>94</td></tr><tr><td>33</td><td>944</td><td>cathedral</td><td colspan="2">82.78 79.22 86.67</td><td>52</td><td>feed</td><td>2.74</td><td>1.41</td><td>50.00</td><td>1</td></tr><tr><td>34</td><td>122</td><td>purple</td><td colspan="2">82.63 91.98 75.00</td><td>138</td><td>proportion</td><td>1.66</td><td>0.84</td><td>50.00</td><td>1</td></tr><tr><td>35</td><td>630</td><td>trees</td><td colspan="3">82.52 80.39 84.77 1258</td><td>tree</td><td>15.48</td><td>9.47</td><td colspan="2">42.33 171</td></tr><tr><td>186</td><td>375</td><td>boy</td><td colspan="2">69.90 65.45 75.00</td><td>93</td><td>boys</td><td colspan="2">20.87 13.09</td><td>51.43</td><td>18</td></tr><tr><td>187</td><td>634</td><td>ground</td><td colspan="2">69.78 73.92 66.08</td><td colspan="2">224 playground</td><td>7.45</td><td>3.94</td><td>69.23</td><td>27</td></tr><tr><td>188</td><td>69</td><td>courtyard</td><td colspan="2">69.55 57.98 86.89</td><td>53</td><td>plaza</td><td colspan="2">28.89 19.87</td><td>52.94</td><td>9</td></tr><tr><td>189</td><td>281</td><td>wooden</td><td colspan="2">69.50 57.89 86.92</td><td>525</td><td>wood</td><td colspan="2">23.55 14.52</td><td colspan="2">62.20 153</td></tr><tr><td>190</td><td>812</td><td>lighthouse</td><td colspan="2">69.41 59.74 82.81</td><td colspan="3">53 lighthouses 11.69</td><td colspan="2">6.21 100.00</td><td>5</td></tr><tr><td>191</td><td>225</td><td>house</td><td colspan="2">69.13 61.59 78.78</td><td>516</td><td>houses</td><td colspan="2">18.29 10.31</td><td>80.73</td><td>88</td></tr><tr><td>192</td><td>705</td><td>dark</td><td colspan="2">69.11 68.57 69.66</td><td>186</td><td>darker</td><td>3.05</td><td>1.56</td><td>75.00</td><td>6</td></tr><tr><td>193</td><td>980</td><td>building</td><td colspan="3">69.10 77.48 62.35 1161</td><td>buildings</td><td colspan="2">25.72 15.71</td><td colspan="2">70.78 281</td></tr><tr><td>194</td><td>844</td><td>grass</td><td colspan="2">69.01 61.85 78.04</td><td>455</td><td>grassy</td><td colspan="2">21.70 12.42</td><td colspan="2">85.83 109</td></tr><tr><td>195</td><td>446</td><td>lake</td><td colspan="2">68.69 78.63 60.98</td><td>125</td><td>late</td><td>5.02</td><td>2.90</td><td>18.52</td><td>5</td></tr><tr><td>196</td><td>182</td><td>trash</td><td colspan="2">68.64 66.79 70.59</td><td>48</td><td>boulders</td><td>10.16</td><td>6.27</td><td>26.67</td><td>4</td></tr><tr><td>197</td><td>437</td><td colspan="3">photograph 68.63 59.06 81.89</td><td colspan="4">588 photographs 30.56 18.46</td><td colspan="2">88.73 181</td></tr><tr><td>198</td><td>237</td><td>lobby</td><td colspan="2">68.43 56.77 86.11</td><td>31</td><td>waiting</td><td>9.93</td><td>7.86</td><td>13.46</td><td>14</td></tr><tr><td>199</td><td>829</td><td>shirt</td><td colspan="2">68.41 71.49 65.58</td><td>322</td><td>shirts</td><td colspan="2">18.28 10.37</td><td>76.79</td><td>43</td></tr><tr><td>200</td><td>59</td><td>grass</td><td colspan="2">68.31 56.53 86.28</td><td>503</td><td>grassy</td><td>15.30</td><td>8.67</td><td>65.35</td><td>83</td></tr></table>

Performance of the VQ3 layer from the "{2} → {2, 3}" model when codes are treated as word detectors. Codes are ranked by the highest F1 score among the retrieved words for a given code. Word hypotheses for a given code are ranked by the F1 score.

Table content: <table><tr><td colspan="2">rank code</td><td>word</td><td colspan="2">Top Hypotheses F1 P</td><td>R</td><td>occ</td><td>word</td><td colspan="2">Second Hypotheses F1 P</td><td colspan="2">R occ</td></tr><tr><td>1</td><td>924</td><td>people</td><td colspan="4">76.71 67.49 88.85 1665</td><td>computer</td><td>2.17</td><td colspan="2">1.12 40.40</td><td>40</td></tr><tr><td>2</td><td>749</td><td>white</td><td colspan="4">76.47 66.92 89.21 2265</td><td>one</td><td>4.15</td><td colspan="3">2.50 12.14 134</td></tr><tr><td>3</td><td>530</td><td colspan="5">building 75.47 64.84 90.28 1681</td><td>buildings</td><td colspan="4">23.93 13.81 89.67 356</td></tr><tr><td>4</td><td>505</td><td>blue</td><td colspan="4">59.12 46.90 79.96 1093</td><td>pool</td><td>10.74</td><td colspan="3">5.89 60.80 152</td></tr><tr><td>5</td><td>581</td><td>snow</td><td colspan="3">57.61 41.77 92.83</td><td>466</td><td>snowy</td><td>16.63</td><td colspan="3">9.12 94.50 103</td></tr><tr><td>6</td><td>778</td><td colspan="5">building 52.10 36.71 89.69 1670</td><td>buildings</td><td>14.30</td><td colspan="3">7.78 88.16 350</td></tr><tr><td>7</td><td>144</td><td>with</td><td colspan="4">49.12 41.58 59.99 3386</td><td>wooden</td><td>6.32</td><td colspan="3">3.34 60.60 366</td></tr><tr><td>8</td><td>299</td><td>small</td><td colspan="3">47.83 32.78 88.42</td><td>901</td><td>snow</td><td colspan="4">30.55 18.35 91.24 458</td></tr><tr><td>9</td><td>550</td><td>large</td><td colspan="4">45.13 30.50 86.76 1920</td><td>car</td><td>8.57</td><td colspan="3">4.52 82.13 216</td></tr><tr><td>10</td><td>76</td><td>trees</td><td colspan="4">44.82 29.76 90.77 1347</td><td>tree</td><td>15.21</td><td colspan="3">8.31 89.36 361</td></tr><tr><td>11</td><td>831</td><td>water</td><td colspan="4">41.84 27.50 87.43 1210</td><td>wall</td><td>17.57</td><td colspan="3">9.87 79.97 491</td></tr><tr><td>12</td><td>1015</td><td>large</td><td colspan="4">39.59 26.06 82.29 1821</td><td>cars</td><td>6.58</td><td colspan="3">3.42 84.21 224</td></tr><tr><td>13</td><td>80</td><td>red</td><td colspan="3">39.15 26.13 78.05</td><td>992</td><td>bed</td><td>9.14</td><td colspan="3">4.91 66.67 168</td></tr><tr><td>14</td><td>719</td><td colspan="4">woman 38.71 24.95 86.31</td><td>687</td><td>women</td><td>7.75</td><td colspan="3">4.09 72.41 126</td></tr><tr><td>15</td><td>614</td><td>people</td><td colspan="4">37.71 25.10 75.83 1421</td><td>table</td><td colspan="4">22.10 12.75 82.93 656</td></tr><tr><td>16</td><td>816</td><td>water</td><td colspan="4">37.71 24.25 84.75 1173</td><td>river</td><td>5.72</td><td colspan="3">2.97 80.22 215</td></tr><tr><td>17</td><td>457</td><td>sky</td><td colspan="3">35.71 23.20 77.47</td><td>540</td><td>skies</td><td>11.81</td><td colspan="3">6.33 88.12 141</td></tr><tr><td>18</td><td>480</td><td>has</td><td colspan="4">34.88 25.01 57.64 1204</td><td>house</td><td>9.88</td><td colspan="3">5.48 50.38 330</td></tr><tr><td>19</td><td>245</td><td>yellow</td><td colspan="3">34.14 21.17 88.05</td><td>597</td><td>flowers</td><td>13.44</td><td colspan="3">7.27 89.43 237</td></tr><tr><td>20</td><td>968</td><td>picture</td><td colspan="4">34.11 22.32 72.33 1686</td><td>pictures</td><td>16.79</td><td colspan="3">9.38 79.77 698</td></tr><tr><td>21</td><td>985</td><td>trees</td><td colspan="4">33.88 22.04 73.25 1087</td><td>tree</td><td>10.96</td><td colspan="3">5.93 72.52 293</td></tr><tr><td>22</td><td>536</td><td>man</td><td colspan="4">33.53 20.71 88.06 1128</td><td>standing</td><td>9.06</td><td colspan="3">4.86 66.17 532</td></tr><tr><td>23</td><td>0</td><td>black</td><td colspan="9">33.49 20.81 85.77 1163 background 21.15 12.28 76.20 759</td></tr><tr><td>24</td><td>815</td><td>with</td><td colspan="4">33.21 22.76 61.36 3463</td><td>white</td><td>8.60</td><td colspan="3">4.85 38.05 966</td></tr><tr><td>25</td><td>293</td><td>large</td><td colspan="4">33.13 23.80 54.50 1206</td><td>bridge</td><td colspan="4">19.09 10.78 83.18 371</td></tr><tr><td>26</td><td>870</td><td>trees</td><td colspan="4">32.87 20.79 78.44 1164</td><td>train</td><td>12.97</td><td colspan="3">7.00 88.03 353</td></tr><tr><td>27</td><td>153</td><td>yellow</td><td colspan="3">32.42 19.94 86.73</td><td>588</td><td>area</td><td>15.39</td><td colspan="3">9.47 40.92 354</td></tr><tr><td>28</td><td>243</td><td>front</td><td colspan="3">32.13 20.97 68.69</td><td>895</td><td>from</td><td>14.34</td><td colspan="3">8.51 45.49 358</td></tr><tr><td>29</td><td>538</td><td>black</td><td colspan="4">31.40 19.64 78.24 1061</td><td>glass</td><td>8.28</td><td colspan="3">4.36 82.90 223</td></tr><tr><td>30</td><td>526</td><td>small</td><td colspan="3">31.34 19.08 87.83</td><td>895</td><td>large</td><td>5.91</td><td colspan="3">4.04 11.03 244</td></tr><tr><td>31</td><td>395</td><td>picture</td><td colspan="4">31.32 20.90 62.46 1456</td><td>pictures</td><td>13.98</td><td colspan="3">7.80 67.54 591</td></tr><tr><td>32</td><td>133</td><td>white</td><td colspan="4">29.82 18.98 69.55 1766</td><td>black</td><td>16.34</td><td colspan="3">9.32 66.37 900</td></tr><tr><td>33</td><td>715</td><td>white</td><td colspan="4">29.45 19.73 58.05 1474</td><td>like</td><td>9.92</td><td colspan="3">5.89 31.29 388</td></tr><tr><td>34</td><td>39</td><td>picture</td><td colspan="3">29.37 22.51 42.26</td><td>985</td><td>like</td><td>9.82</td><td colspan="3">5.79 32.26 400</td></tr><tr><td>35</td><td>740</td><td>trees</td><td colspan="5">29.31 18.52 70.35 1044 showcasing</td><td>3.88</td><td colspan="3">2.00 61.51 155</td></tr><tr><td>186</td><td>129</td><td>area</td><td>6.62</td><td colspan="2">3.69 32.37</td><td>280</td><td>snow</td><td>6.08</td><td colspan="3">3.23 51.20 257</td></tr><tr><td>187</td><td>288</td><td>picture</td><td>6.61</td><td colspan="2">3.76 27.63</td><td>644</td><td>pictures</td><td>4.91</td><td colspan="3">2.58 52.91 463</td></tr><tr><td>188</td><td>374</td><td>there's</td><td>6.61</td><td colspan="2">3.73 28.71</td><td>775</td><td>that</td><td>5.14</td><td colspan="3">2.91 21.81 614</td></tr><tr><td colspan="2">189 1003</td><td>front</td><td>6.59</td><td colspan="2">3.65 34.00</td><td>443</td><td>some</td><td>4.56</td><td colspan="3">2.64 16.84 345</td></tr><tr><td>190</td><td>396</td><td>multi</td><td>6.45</td><td>6.25</td><td>6.67</td><td>1</td><td>towers</td><td>5.41</td><td>6.25</td><td>4.76</td><td>1</td></tr><tr><td>191</td><td>790</td><td>photo</td><td>6.23</td><td colspan="2">3.35 44.27</td><td>247</td><td>by</td><td>5.66</td><td colspan="3">3.06 37.60 229</td></tr><tr><td>192</td><td>522</td><td>field</td><td>6.14</td><td colspan="2">3.22 63.22</td><td>385</td><td>photo</td><td>6.01</td><td colspan="3">3.19 51.79 289</td></tr><tr><td>193</td><td>42</td><td>man</td><td>5.96</td><td colspan="2">3.19 45.12</td><td>578</td><td>middle</td><td>3.85</td><td colspan="3">1.98 68.11 314</td></tr><tr><td>194</td><td>801</td><td>sitting</td><td>5.91</td><td colspan="2">3.15 48.56</td><td>405</td><td>with</td><td>5.00</td><td>3.65</td><td colspan="2">7.92 447</td></tr><tr><td>195</td><td>181</td><td>trees</td><td>5.89</td><td colspan="2">3.16 43.19</td><td>641</td><td>with</td><td>5.13</td><td colspan="3">3.22 12.72 718</td></tr><tr><td>196</td><td>791</td><td>with</td><td>5.84</td><td colspan="2">3.58 15.86</td><td>895</td><td>that</td><td>3.21</td><td colspan="3">1.79 15.88 447</td></tr><tr><td>197</td><td>975</td><td>purple</td><td>5.72</td><td colspan="2">2.99 64.13</td><td>118</td><td>parked</td><td>4.67</td><td colspan="3">2.42 65.93 120</td></tr><tr><td>198</td><td>38</td><td>structure</td><td>5.62</td><td>6.20</td><td>5.14</td><td>13</td><td>graph</td><td>4.03</td><td colspan="2">2.19 25.00</td><td>2</td></tr><tr><td>199</td><td>432</td><td>parking</td><td>5.61</td><td colspan="2">2.90 83.12</td><td>133</td><td>park</td><td>5.56</td><td colspan="3">2.87 82.89 126</td></tr><tr><td>200</td><td>329</td><td>standing</td><td>5.60</td><td colspan="2">2.97 49.63</td><td>399</td><td>woman</td><td>3.09</td><td colspan="3">1.62 34.67 276</td></tr></table>

Peer review (iclr2020):
Overview: The paper proposes a method to learn discrete linguistic units in a low-resource setting using speech paired with images (no labels). The visual grounding signal is different from other recent work, where a reconstruction objective was used to learn discrete representations in unsupervised neural networks. In contrast to other work, a hierarchy of discretization layers are also considered, and the paper shows that, with appropriate initialization, higher discrete layers capture word-like units while lower layers capture phoneme-like units. Strengths: The paper is extremely well-written with a clear motivation (Section 1). The approach is novel. But I think the paper*s biggest strength is in its very thorough experimental investigation. Their approach is compared to other very recent speech discretization methods on the same data using the same (ABX) evaluation metric. But the work goes further in that it systematically attempts to actually understand what types of structures are captured in the intermediate discrete layers, and it is able to answer this question convincingly. Finally, very good results on standard benchmarks are achieved. Weaknesses: Although I think the paper is very well-motivated, my first criticism is that discretization itself is not motivated: why is it necessary to have a model with discrete intermediate layers? Does this give us something other than interpretability (which we obtain due to the sparse bottleneck)? In the detailed questions below, I also specifically ask whether, for instance, the downstream speech-image task actually benefits from including discrete layers. My second point is that it is unclear why word-like units only appear when the higher-level discrete layers are trained from scratch; as soon as warm-starting is used, the higher level layers capture phoneme-like units (Table 1). Is it possible to answer/speculate why this is the case? Overall assessment: The paper presents a new approach with a thorough experimental investigation. I therefore assign an *accept*. The weaknesses above asks for additional motivation and some speculation. Questions, suggestions, typos, grammar and style: - Section 3.3: It maybe makes less sense for the end-task, but did the authors consider discretization on the image side of the network? This could maybe lead to parts of objects being composed to form larger objects (in analogy to the speech network). - Section 3.3, par. 3: *with the intention that they should capture discrete word-like and sub-word-like units* -> *with the intention that they should capture discrete *sub-word-like and word-like units** (easier to read with first part of sentence) - Section 3.3: The more standard VQ-VAE adds a commitment loss and a loss for updating the embeddings; was this used or considered at all, or is this all captured through the exponential moving average method? - Section 3.4: *with same VQ layers* -> *with *the* same VQ layers* - Section 3.5: Can you briefly outline the motivation for adding the two losses (so that it is not required to read the previous work). - Section 4.1: Following from the first weakness listed above, the caption under Figure 2 states that the non-discrete model achieves a speech-image retrieval R@10 of 0.735. This is lower than some of the best scores achieved in Table 1. Can this be taken as evidence that discretization actually improves the downstream task? If so, it would be worth highlighting the point more; if there is some other reason, that would also be worth knowing. - Figure 1: Did the authors ever consider putting discrete layers right at the top of the speech component, just before the pooling layer? Would this more consistently lead to word-like units?