Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech
Authors: David Harwath*
Wei-Ning Hsu*
James Glass
Venue:  (2020)
Keywords: visually-grounded speech
self-supervised learning
discrete representation learning
vision and language
vision and speech
hierarchical representation learning

Abstract: In this paper, we present a method for learning discrete linguistic units by incorporating vector quantization layers into neural models of visually grounded speech. We show that our method is capable of capturing both word-level and sub-word units, depending on how it is configured. What differentiates this paper from prior work on speech unit learning is the choice of training objective. Rather than using a reconstruction-based loss, we use a discriminative, multimodal grounding objective which forces the learned units to be useful for semantic image retrieval. We evaluate the sub-word units on the ZeroSpeech 2019 challenge, achieving a 27.3% reduction in ABX error rate over the top-performing submission, while keeping the bitrate approximately the same. We also present experiments demonstrating the noise robustness of these units. Finally, we show that a model with multiple quantizers can simultaneously learn phone-like detectors at a lower layer and word-like detectors at a higher layer. We show that these detectors are highly accurate, discovering 279 words with an F1 score of greater than 0.5.

Published as a conference paper at ICLR 2020
LEARNING HIERARCHICAL DISCRETE LINGUISTIC UNITS FROM VISUALLY-GROUNDED SPEECH
David Harwath; Wei-Ning Hsu", and James Glass Computer Science and Artificial Intelligence Lab Massachusetts Institute of Technology Cambridge, MA 02139, USA (dharwath, wnhsu, g glass @csaall 1. mi edu
ABSTRACT
In this paper, we present method for learning discrete linguistic units by incorpo- rating vector quantization layers into neural models of visually grounded speech. We show that our method is capable of capturing both word-level and sub-word units, depending on how it is configured. What differentiates this paper from prior work on speech unit learning is the choice of training objective. Rather than us- ing a reconstruction- objective which forces the learned units to be useful for semantic image retrieval. We evaluate the sub-word units on the ZeroSpeech 2019 challenge, achieving a 27.3% reduction in ABX error rate over the top-performing submission, while keeping the bitrate approximately the same. We also present experiments demon- strating the noise robustness of these units. Finally, we show that a model with multiple quantizers can simultancously learn phone-like detectors at a lower layer and word-like detectors at a higher layer. We show that these detectors are highly accurate, discovering 279 words with an F1 score of greater than 0.5.
loss, we use a discriminative, multimodal grounding
INTRODUCTION
By 8 months of age, human infants learn to recognize not only the names of their caregivers and common objects, but also the contrast between the different vowels and consonants which comprise these words (Dupoux 2018). Nearly all toddlers learn to carry a conversation long before they can read and write. Humans learn to model the discrete, hierarchical, and compositional nature of their native language not from written text, but from speech audio a continuous, vimevngrying waveform which is the product not only of the underlying words which were spoken, but also the physical properties of the speaker's vocal tract, the speaker's health and emotional state, and the noise and reverberation present in the environment. The question of how such a complex symbolic system is inferred from continuous and noisy sensory input data is of interest not only to the cognitive science community, but also to machine learning researchers who aim to reproduce this ability with comput- ers. A more comprehensive understanding of human language acquisition has practical significance in real-world applications, such as automatic speech recognition (ASR) and natural language under- standing (NLU) systems. In the past several decades, enormous progress has been made in speech recognition research, and nowadays ASR systems are able to achieve human- level accuracy in many domains (Chiu et al. 2018). Unfortunately, the techniques that have been developed achieve these levels of performance are extremely data-hungry, requiring many thousands of hours of speech au- dio recordings for training. Since supervised machine learning algorithms form the basis of ASR training, the data also needs to be annotated by expert humans. Due to the immense cost of collect- ing and annotating speech data, ASR technology currently exists for approximately 120 (Google) 2019) out of the nearly 7,000 (Lewis et al. 2016) human languages spoken worldwide. It is highly unlikely that purely supervised machine learning techniques will be able to scale to include all hu- man languages, necessitating the development of alternative methods by researchers which are able to function with far fewer annotations, or even no annotations at all. Because human beings provide an existence proof of language acquisition from speech completely without language supervision, is plausible that this ability could be replicated by a machine learning algorithm.
Equul contribution

---

Published as a conference paper at ICLR 2020
In this paper, we present method for discovering discrete and hierarchical representations of speech units both at the sub-word level and the word level. Previously proposed linguistic unit discovery methods have only leveraged the speech audio modality in isolation, relying objective functions that attempt to capture statistical regularities within the speech signal. The key innovation in our work is that discover units by training models with explicit discretization layers associate speech waveforms with visual images using a cross-modal grounding objective. This forces our models to learn representations which capture semantic information the highest layers of the network. Because semantics are predominantly carried by words, and words are composed of sub- word units (such as phones and syllables), the visual grounding objective indirectly forces the model to learn speaker- and noise-invarian representations of speech units. By incorporating trainable quantization layers into our networks, we are able to capture these units in discrete inventories. Whether these units correspond to word-like or sub-word units depends on where the quantization layers are inserted, and how they are trained.
RELATED WORK
Prior work on unsupervised modeling of the speech signal has generally focused on learning repre- sentations which either disentangle isolate the latent factors that are of interest for downstream tasks. In most cases the primary latent factor of interest is the phonetic or lexical identity of a given segment of speech, but other factors, such as the identity of the speaker, are sometimes of interest as well. Because the factors of interest are often inherently discrete (e.g. words and phones), many of the proposed approaches attempt to perform segmentation and clustering of the surface features in one way or another. One family of techniques is based upon Segmental Dynamic Time Warping (S-DTW) (Park& & Glass] 2005; 2008 Jansen et al., 2010; Jansen & Van Durme (011); which uses self-comparison algorithm to identify relatively long duration (on the order of second) pattems which frequently reoccur in a speech corpus; these patterns tend to capture words or short phrases. A different line of work employs probabilistic graphical models to jointly segment and cluster the speech signal /Varadarajan et al. 2008: Zhang & Glass| 2009 Gish et al. 2009 Lee & Glass) 2012] SSuu et al. 2014 Lee et al. 2015 Ondel et [2166] Kamper et al.] [2016] 2017a). With ap- propriately designed model, it is possible to learn multiple, hierarchical categories of speech units. However, in order to enable efffccint inference, the conditional distributions of these models tend to be simple and therefore have limited modeling power.
Deep neural network models have been successfully used to learn powerful speech representations using weakly or unsupervised objectives (Thiolliere et al., 2015) Kamper et al. 2015 Hsu et al.)
2017a b. Hsu & Glass| 2018, Holzenberger et al. 2018:] Milde & Biemann, 2018 van den Oord
et al.) 2018: Chung et al. 2019] Pascual et al.) 2019). These representations have predominantly been continuous in nature, as discrete latent variables are not trivially compatible with backpropaga- tion. To obtain discrete representations, post-hoc clustering step can be applied to the continuous representations (Kamper et al. 20176; Feng et al. 2019). More recently, several papers have pro- posed ways of directly incorporating discrete variables into neural network models, including using Gumbel-Softtax (Eloff et al. 2019b) or straight- estimators (van den Oord et al., 2017
Chorowski al. 2019 Razavi et al. 2019).
A different method for learning meaningful representations of speech is via a multimodal ground- ing objective, which encourages the learning of speech representations that predictive of the contextual information contained in a separate but accompanying modality, such as vision. Visual grounding of speech is form of self-supervised learning (Virginia de Sa 1994), which is powerful in part because it offers a way of training models with a discriminative objective that does not depend on traditional transcriptions or annotations. The first work in this direction relied on phone strings to represent the speech (Roy & Pentland 2002, Roy] 2003). but more recently this learning has been shown to be possible directly on the speech signal (Synnaeve et al. 2014 Harwath & Glass 2015] Harwath et al.) (016). Subsequent work on
models of speech has investigated improvements and alternatives to the modeling or training algorithms (Leidal et al. 2017 Kamper
et al.
2017c| Havard et al. 2019a| Merkx et al. 20199 @01rppaaaaaaa et al. [017]] Scharenborg et al.
2018 Kamper et al. [201bb a| Suris et al. [ 1019:
Ilharco et al. 2019 Eloff et al.) (019a)), application 20199 Havard
to m multilingual settings (Harwath et al. 2018a Kamper & Roth 2017| Azuh
jet al. 2019a), analysis of the linguistic abstractions, such as words and phones, which are learned by the models (Harwath & Glass| 2017 Harwath et al. 20T8b Drexler & Glass 2017 Alishahi al.
2

---

Published as a conference paper at ICLR 2020
2017 Harwath al. 2019 Harwath & Glass 2019 Havard al. (2019b). and the impact of jointly
training with textual imput (olzenberger et al. 2019 Chrupata 2019 Pasad et al. (219)) Repre- sentations learned by models of visually grounded speech are also well-suited for transfer learning to supervised tasks, being highly robust to noise and domain shift (Hsu et al. 2019).
3 DATA AND MODELS
3.1 DATASET
For training our models, we utilize the MIT Places 205 dataset Zhou et al. 2014) and their ac- companying spoken audio captions (Harwath al., 2016 2018b). The caption dataset contains approximately 400,000 spoken audio captions, each of which describes different Places image. These captions are free-form spontaneous speech, collected from over 2,500 different speakers and covering 40,000 word vocabulary. The average caption duration is approximately 10 seconds, and each caption contains on average 20 words. For vetting our models during training, we use a held-out validation set of 1,000 image-caption pairs.
3.2 NEURAL MODELS OF AISUALLY-GROUNDEE SPEECH
We base our model upon the Residual Deep Audio- Visuul Embedding network (ResDAVEnet) ar- chitecture (Harwath et al. 2019), which contains two branches of fully convolutional networks, one for images and the other for audio. Each branch encodes samples the corresponding modality into d-dimensional space, regardless of the original dimensionality of the samples. This is achieved by applying global spatial mean pooling and global temporal mean pooling to the image branch output and the audio branch output, respectively. The image branch is adapted from ResNet50 (He al., 2016). where the final softmax layer and the preceding fully-connected layers are removed, replaced with a Ixl linear convolutional layer in order to project the feature map to the desired dimension. To model the audio inputs, a 17-layer fully convolutional network with residual connections is used. The imput is a log Mel-frequency spectrogram with 40 frequency bins and 25 ms- ms-wide, Hamming- windowed frames with a shift of 10 ms. The first layer of this network is 1-D convolution that spans the entire frequency axis of the spectrogram, while the remaining 16 convolutional layers are 1-D across the time axis. These 16 layers are divided into four residual blocks of 4 layers each, and downsampling between these blocks is accomplished by applying the first convolution of each block with stride of 2. For full details of the model, refer to Harwath et al. (2019).
3.3 LEARNING HIERARCHICAL DISCRETE UNITS WITH VECTOR QUANTIZING LAYERS
Previous analyses reveal that ResDAVEnnt-like models learn linguistic abstractions at different lev- els, including words (Harwath & Glass 2017) and robust phonetic features (Harwath & Glass 2019 Hsu et al.) 2019). To explicitly learn hicrarchical discrete linguistic units within this framework, we propose incorporate multiple vector quantization (VQ) layers (van den Oord et al. 2017) into the ResDAVEnet audio branch; we refer to this new architecture as ResDAVEnet- VQ.
type of bottleneck, which constrain the amount of information that can flow through. While these layers have been used to learn discrete sub-word units (van den Oord al. 2017 Chorowski al.) 2019; Razavi al. 2019), previous work injects VQ layers into autoencoders that are trained with a reconstruction loss. As a result, the embedding dimen- sion of each code and the number of codes need be carefully tuned (Liu al., 2019). When the embedding dimension is too low or the codebook size too small, the model does not have enough expressive power to capture linguistic variability. When it too large, the model starts to encode non-linguistic information in order to improve reconstruction. In contrast, the learning signal of ResDAVEnet-VO is provided by the visual-semantic grounding objective. Rather than encoding as much information about input as possible, the learned codes in ResDAVEnet- En only need to cap- ture semantic information. Since semantics in speech are predominantly transmitted by words, and words are composed of sub-word units like phones, the grounding objective places pressure on the model to robustly infer both from speech. Since words and phones are inherently discrete symbols, representing them with learned discrete units may not even hurt the grounding performance.
VQ layers can be understood
3

---

Published as a conference paper at ICLR 2020
Figure fillustrates the proposed ResDAVEnet- VQ model. We add quantization layer after each of the first two residual blocks of the ResDAVEnet-Ve model, denoted as VQ2 and VQ3, respectively, with the intention that they should capture discrete sub-word-like and word-like units. A VQ layer is defined as E E IR Kxxx where K represents the codebook size, and / represents the output dimensionality of the input features to the codebook. Denoting the t temporal frame of the input to the quantization layer as Zt, quantization is performed according to qr ER,;, where k = arg min, ||= E The quantized output is then fed as input the subsequent residual block. As in van den Oord et al. (2017), we use the straight-through estimator (Bengio et al. 2013) to compute the gradient passed from qi to zt. We use the exponential moving average (EMA) codebook updates proposed by van den Oord et al. (2017).
Impostor
Bypass
image
Image CNN
Pool
Push
6,
Pool
Res 3
2
2
3
VQ Block
Figure 1: Diagram of the ResDAVEnnt--e model. On the left, we show the placement of the vector quantization blocks in the audio branch. Note that each "Res" block is comprised of a stack of multiple sub-layers (see Harwwthh et al. (2019) for details). The right half of the figure depicts the quantization mechanism of each VQ block, as well as the bypass path when the block disabled.
3.4 CODEBOOK LEARNING SCHEDULES
We include multiple VQ layers in the ResDAVEnct. VQ model, each which can be independently enabled or bypassed without changing the rest of the architecture configuration. When all model weights, including the VQ codebooks, are trained jointly in single training run we call this a "cold- start" model. Alternatively, model can be "warm-started" by copying the weights from another trained model that has fewer (or no) VQ layers enabled, and randomly initializing the codebook of the newly activated VQ layer(s). This gives rise the questions of how many quantizers should be used and in what order they should be enabled. It is unclear whether models with the same VQ layers activated would learn the same representation at each layer regardless of the training curriculum. Let Am denote a subset of all VQ layers, and Am-1 C Am- We use "A1 -> A to denote model that is obtained by sequentially training models "A1 > ... - Am initialized from "A1 - ... + Am-1", where the model A is initialized from scratch, and the final model would have VQ layers AM activated. For instance, model initialized from scratch with no VQ layers enabled is denoted "Ø", " and model initialized with that and with both layers enabled is denoted as "Ø + (2.3)".
3.5 TRAINING WITH THE TRIPLET LOSS
We train models using the same loss function Harwath al. (2019). This loss function blends two triplet loss terms (Weinberger & Saul 2009). one based on random sampling of negative exam- ples, and the other based on semi-hard negative mining (Jansen et al. 2018). in order find more challenging negative samples. Specifically, let the sets of output embedding vectors for a minibatch of B audio/image training pairs respectively be
{aI..... a and I (f......ig) To com- pute the randomly-sample triplet loss term, we select impostor examples for the jth input according to aj UniformCatteorrica (a1,..., ag) } (a,) and i, UniformCategorical ((i,...., (4y)- The aandomly-sample triplet loss is then computed as:
C. (0. ifa 1T +1 + max(0, - 13 a, + +1)
For the semi-hard negative triplet loss, first define the sets of impostor candidates for the jth example Å, (a Ali3 < a and i, {i EliTTa i a.). The semi-hard negative
4

---

Published as a conference paper at ICLR 2020
loss is then computed as:
B
Lh = max(0, max a5 + 1) + max(0, max(i"aa) -1 deAs
- + +1)
(2)
iel,
Finally, the overall loss function is computed by combining the two above losses, L C. Lrn which was found by (Harwath et al., 2019) to outperform either loss on its own.
3.6 IMPLEMENTATION DETAILS
All of models were trained for 180 epochs using the Adam optimizer (Kingma & Ba 2014)
with a batch size of 80. We used an exponentially decaying learning rate schedule, with an initial value of 2e-4 that decayed by factor of 0.95 every 3 epochs. Following van den Oord et al. (2017).
we use an EMA decay factor of = .99 for training each VQ codebook. Our core experimental results all use codebook size of 1024 vectors for all quantizers, but in the supplementary material we include experiments with smaller and larger codebooks, Following Chorowski et al. (2019), the jitter probability hyperparameter for each quantization layer was fixed 0.12. While we do not apply data augmentation to the input spectrograms, during training we perform standard data augmentation techniques to the images. We resize each raw image so that its smallest dimension is 256 pixels, and then we apply an Inception-style random crop which is resized to 224 pixels square. During training, we also flip each image horizontally with a probability of 0.5. During evaluation, the center 224 pixel square crop is always taken from the image. Finally, the RGB pixel values are mean and variance normalized. We trained each model on the Places audio caption train split, and computed the image and caption recall at 10 (R @ 10) scores on the validation split of the Places audio captions after each training epoch. The model snapshot that achieved the highest average R @ 10 score on the validation set from each training is used for all evaluation. To extract embeddings and units from our models, we simply perform a forward pass through the speech branch of the ResDAVEnet VQ network and retain the outputs from the target layer at a uniform frame-rate. The rate-rate is determined by the downsampling factor at the target layer relative to the input. For non non-quantized layers, these outputs will be continuous embeddings. For quantized layers, these will be quantized embedding retrieved from the assigned entry in the codebook.
EXPERIMENTS
4.1 WUR- UNIT LEARNING ON THE ZEROSPEECH 2019 ABX TASK
Evaluation metrics Learning unsupervised speech representations that are indicative of phonetic content is of high interest to the speech community, and recently has been the focus of the Ze- roSpeech Challenge (Versteegh et al., 2015 Dunbar et al.) 2017 2019p. One of the core evaluations is the minimal-pair ABX task (Schatz et al. 2013)), which aims to benchmark representations in terms of their discriminability between different sub-word speech units. In this task, a model is tasked with extracting representations for triplet of speech waveform segments denoted by A, B, and X. and B are constrained to be a triphone minimal pair; that is, both segments capture three phones, but differ only in the identity of their center phone. The third segment, X cho- sen to contain the same underlying triphone sequence as A. Supposing f() denotes the model's mapping function from a waveform segment a sequence of embedding vectors, the ABX er- ror rate under given similarity metric S(...) is defined as the fraction of ABX triples in which (((()), fX)) > (Bf f(X). An ABX error rate of 50% indicates random assignment, while an ABX of 0% reflects perfect phone discriminability. In the ZeroSpeech challenge, S(, .) is imple- mented using Dynamic Time Warping (DTW) with various distance measures (cosine, KI., etc.). In evaluation, use the cosine distance.
The ZeroSpeech 2019 challenge in particular emphasizes on discovering an inventory of discrete sub-word units, rather than continuous representations. Therefore, in addition to an ABX error rate, a bitrate is also computed for each model which reflects the amount of information carried by the leamned units. A lower bitrate can be achieved by having more compact inventory of learned units or having a smaller number of codes per second. The full details the evaluation can be found in Dunbar et al. (2019). To be clear, all of our ResDAVEnet- VQ models were not trained on the
5

---

Published as a conference paper at ICLR 2020
Table 1: Comparison of R@ 10, ABX scores, and bit-rates between different configurations and baseline models trained on ZeroSpeech 2019 data or Places Audio Caption. All quantizers reflected in this table used a codebook size of 1,024 vectors. We do not compute RLE or segment scores for the FHVAE-DPGMM model, since we did not re-implement that model.
Frame-Base
Segment- Based ABX Bitrate
Model ID
Layer
R@ 10
ABX
Bitrate
RLE Bitrate
FHVAE-DPGMM (ZS)
N/A
N/A
21.67
413.23
WaveNet- VQ (ZS)
N/A
N/A
19.98
151.55
136.74
20.48
126. 17
WaveNet- VQ (PA)
N/A
N/A
24.87
149.00
136.27
25.23
126.22
Res2
11.35
N/A
N/A
N/A
N/A
"Ø"
.735
Res3
10.86
N/A
N/A
N/A
N/A
"(2)"
VQ2
.753
12.33
433.30
361.09
12.78
332.86
"Ø + (2)"
VQ2
.760
11.79
390.61
317.66
12.66
289.11
"(3)"
VQ3
.734
38.21
213.92
129.65
38.68
108.84
"Ø - (3)"
VQ3
.794
15.04
182.93
140.04
16.53
121.26
VQ2
25.62
408.75
258.37
26.32
217.58
"[2,3)"
.667
VQ3
32.23
218.76
156.69
32.49
136.90
VQ2
13.15
405.43
334.39
13.30
303.03
"Ø + (2
.787
VQ3
14.95
199.91
172.05
15.60
159.07
VQ2
12.51
415.13
341.85
13.06
311.82
((2) - (2,3)"
.764
VQ3
14.52
167.84
136.11
15.68
121.17
VQ2
13.55
421.23
271.91
14.38
232.87
((3) - (2,3)"
.760
VQ3
33.70
208.63
117.37
33.58
98.29
ZeroSpeech training data, but instead on the Places audio captions, thus there is a domain mismatch between training and testing these models.
In addition the frame-based bitrate and ABX scores computed by the ZeroSpeech 2019 evaluation toolkit, we implement our own extensions to these metrics. Because it is common for successive frames be assigned to the same codebook entry and phonetic information frame rate, lossless run length encoding (RLE) can be more reasonable measure of the bitrate of frame-based model. RLE does not change the ABX score since can be trivially inverted, but does change the bitrate. For computing the RLE bitrate, we modify the bitrate calculation specified in Dunbar et al. (019)) so that a unique symbol is defined as the tuple (unit, length) where length is the number of frames assigned ABX and bitrate, which is similar to the RLE metrics except in this case we outright discard the frame length information. This typically results in even greater reduction in bitrate. but also accompanying deterioration in ABX score.
encoded at fixed
given unit with in segment. We also consider segment-based
Baseline models In Table 1 we compare results those derived from two of the top- performing submissions to the ZeroSpeech 2019 challenge: a re-implementation of WaveNet- VQ (Chorowski al. 2019) provided by Cho et al. (2019) and FHVAE-DPGMM (Feng et al. 2019). Using the code accompanied with the WaveNet-VQ submission, we were able to train their model on the set of 400,000 Places audio captions make fairer comparison with ResDAVEnet-VO models in terms of the amount of speech data used. In addition, when trying to reproduce the re- ported WaveNet- VQ results, we obtain better performance than previously reported by training for more steps. T that WaveNet-V VQ achieves similar bitrates regardless of the training data. However, ABX deteriorates from 19.98 24.87, implying the model cannot utilize data of larger scale but out-of-domain relative to the test set. A similar degradation when testing out-of-domain data with FHVAE models was observed in Hsu et al. (2019). We did not re-train the model submitted by Feng et al. (2019), and instead compare against the scores reported in Dunbar et al. (2019).
ABX discrimination without using quantization Our first experiment investigates exactly which layer in the ResDAVEnet- VQ model is most suited for ABX phone discrimination, and would thus make a good candidate for learning quantized sub-word units. The leftmost plot in Figure P shows that layers 2 and 3 of ResDAVEnet-V VQ model without any quantization enabled perform

---

Published as a conference paper at ICLR 2020
Model "ø"
Model "(2)"
Model "[3]"
40
layerl
40
40
ABX Error Rate
layer2
30
layer3
30
30
layer4
20
20
20
10
10
10
R@ 10
60
60
60
30
30
30
0
0
25
50
0
25
50
25
50
Epoch
Epoch
Epoch
Figure 2: R@ 10 and ABX tracked at various training epochs. The "Ø" model achieves final R@ 10 of .735, with ABX scores of 19.77, 11.35, 10.86, and 14.05 for the convl, I, res2, res3, and res4 layers.
the best in terms of ABX error rate on the ZeroSpeech 2019 English test set; the exact numbers for this model are displayed in the caption of Figure Because layers 2 and achieve the lowest ABX error rates without quantization, we focus our attention on the impact of quantization there.
Quantizing one layer When quantizing only one layer, we examine quantization of layer 2 vs. layer 3, and using cold-start training vs. warm-start initialization from model "Ø" The ABX and bitrate results for these models, as well as the R@ scores on the Places validation set, are shown in Table In all cases, quantization applied at the output of layer 2 achieves a better ABX score than quantization at layer 3, but VQ3 achieves a better bitrate. Quantization barely impacts the perfor- mance of layer 2, whose ABX score very slightly rises from 11.35 to 11.79. Warm-start initialization beneficial to @@ 10 and ABX score both cases, but we notice an intriguing anomaly when ap- plying cold-start quantization layer 3: the ABX score deteriorates significantly, rising from 10.86 in the case of the non-quantized model to 38.21. This indicates that while VQ2 is capable of learn- ing a finite inventory of units that are highly predictive of phonetic identity from either a warm-start or cold-start initialization, cold-start training of VQ3 results in very little phonetic information cap- tured by the quantizer. Interestingly, this model is still learning to infer visual semantics from the speech signal, as evidenced by high R @ 10 score: we later show in Section 4.2 that the reason for this anomaly is because cold-start training of VQ3 results in the learning of word detectors. In all cases except for model "(3)", " we note that the ABX scores achieved by our models are significantly better than the baselines. Our best model terms of ABX ("2 + {2}") achieves a 41.0% .0% reduction in ABX over the WaveNet-VQ VQ baseline, at cost of 132.3% increase in RLE bitrate; however, model "Ø > {3}" achieves a 24.7% reduction in ABX eттor rate with only a 2.4% increase in RLE bitrate. These results do not constitute fair comparison, however, because the WaveNet-VQ and ResDAVEnet- VQ models were trained on different datasets; when training the WaveNet- VQ model on the same set of audio captions used to train ResDAVEnet-VO (but without the accompanying images, since WaveNet-VQ is not a multimodal model), the ABX error rate increases to 24.87%, tipping the results even more in favor of the EesDAVEnet-VQ models.
Quantizing two layers Quantizing multiple layers at once offers the possibility of learning a hi- erarchy of units. Thus, we aim to capture phonetic information in lower layer quantizer and word-level information at a higher layer quantizer. Cold-start training of two quantizers (" 3)") results in significant drop in ABX performance for both VQ2 and VQ3, but also a drop in R@ 10 on the Places validation set. We see much better results in terms of R @ 10 and ABX for the remaining 3 models which were initialized from the "ø" model or a model with only one quantizer enabled; for example, model "(2] + (2,3}" achieves an ABX of 14.52 with RLE bitrate of 136. representing a 27.3% ABX improvement over the best baseline while keeping the bitrate approxi- mately the same. We see in model "(3) + [2,3}" that the same phenomenon observed with model "(3)" persists: VQ3 achieves relatively poor ABX, despite a high overall R@ 10 and strong ABX with VQ2 13.55% We confirm in Section 4.2 that the VQ3 layer of model "(3) [2,3)" does
7

---

Published as a conference paper at ICLR 2020
indeed capture word-level information, indicating that this model has successfully localized pho- netic unit identity in the second layer and lexical unit identity in the third layer. Overall, our results suggest that when learning hierarchical quantized representations with a ResDAVEnet-V VQ model, the nature of the representations learned is highly dependent on the training curriculum.
Table ABX scores and RLE bitrates for various SNRs on the noisy ZeroSpeech1 English test set. "R-B" stands for "RIFEBittraee" and (n) denotes model trained on the noisy Places Audio dataset. For the WQ models, (ZS) and (PA) respectively denote training on the ZeroSpeech 19
English training set, and the clean Places Audio dataset.
Model
Layer
Clean
20-30
10-20 dB
0-10 dB
ABX
R-B
ABX
R-B
ABX
R-B
ABX
R-B
WaveNet- VQ (ZS)
N/A
19.98
136.74
21.22
141.07
27.51
144.28
42.55
126.96
WaveNet- VQ (PA)
N/A
24.87
136.27
27.18
137.70
33.29
132.34
42.67
110.50
"Ø"
Rcs2
11.35
N/A
11.63
N/A
13.17
N/A
19.44
N/A
"Ø"
Res3
10.86
N/A
11.16
N/A
12.96
N/A
19.43
N/A
"2 + (2)
VQ2
11.79
317.66
12.15
325.40
14.62
332.21
23.96
327.15
"[2] (2,3)"
VQ2
12.51
341.85
12.56
350.28
14.82
362.73
25.02
330.54
[[2] (2.3)"
VQ3
14.52
136.11
14.73
137.68
17.44
143.14
27.68
133.13
-13] + [2.3)"
VQ2
13.55
271.91
13.65
272.46
15.69
267.70
24.06
244.52
'[3) + (2,3)"
VQ3
33.70
117.37
32.56
118.22
34.65
115.40
39.82
102.48
"Ø"
Res2
13.32
N/A
12.30
N/A
12.97
N/A
16.91
N/A
"Ø" (n)
Res3
11.85
N/A
11.90
N/A
12.44
N/A
16.09
N/A
"Ø + (2+"(2""m
VQ2
12.64
342.53
12.20
348.57
13.34
359.43
18.82
373.60
'(2) - (2)
VQ2
13.42
365.89
13.71
359.14
14.57
370.67
18.78
392.10
'[2) + (2,3)"
VQ3
14.39
179.19
14.92
180.36
15.38
182.27
19.58
188.32
-[3) + (2,3)"(m)
VQ2
16.52
223.28
16.47
223.61
17.75
225.72
22.68
230.01
([3) (2, (n)
VQ3
26.21
187.31
25.88
187.92
26.34
188.49
31.26
191.28
Training and testing noisy data In Hs/Hsu et al.) (011)) it was shown that representations learned
by a ResDAVEnet model were far more robust to train/test domain mismatch in terms of background noise, channel characteristics, and speaker identity than standard spectral features when training supervised speech recognizer. Here, we examine whether this robustness is also exemplified by the quantized versions of this model. We construct three additional test sets using the ZeroSpeech 2019 English testing data by adding noise sampled from the AudioSet (Jansen et al., 2018) dataset. For each ZeroSpeech testing waveform, we randomly sampled an AudioSet waveform of the same duration and performed linear mixing with signal-to-noise ratio (SNR) selected randomly within a specified range. We construct low, medium, and high noise testing sets, corresponding to SNRs of 20-30 dB, 10-20 dB, and 0-10 dB. We then perform the ABX discrimination task these noisy waveforms, displaying the results in Table We find that for all models, a worsening SNR results in deterioration in ABX performance. However, the ResDAVEnet VQ models prove to be far more noise robust than the Wavenet-VQ model; even in the high noise testing set, the best ResDAVEnet. VQ model achieves ABX of 23.96%, while the WaveNet-VQ models degrade nearly-random ABX scores of 42.55% and 42.67%.
Given that ResDAVEnet-Ve VQ model trained on the "clean" Places Audio captions is highly robust to additive noise on the ABX discrimination task, we investigated whether adding noise to the Places Audio captions themselves would result in an even higher degree of noise robustness. To that end, we followed similar data augmentation approach to create a noisy version of the Places Audio captions, where the SNR of each caption was randomly chosen to sit within the range of 0-30 dB. The bottom half of Table 2 shows the results of training several ResDAVEnet--O models on the noisy Places Audio captions and testing on the clean and noisy ZeroSpeech ABX tasks. In general, observe degradation ABX score in the clean conditions, but with significantly higher degree of noise robustness in the noisier conditions.
Visualization of learned units To better measure the correspondence between the VQ units and English phones, we compute corpus-level co-occurrence statistics (at the frame-level) across the TIMIT training set, excluding the sa dialect sentences. To facilitate visualization, use the "Ø + (2)" model with a codebook size of 128. We display the conditional probability matrix P(phonelumiz (miit) in Figure 3 with the rows and columns ordered via spectral co-clustering with 10
8

---

Published as a conference paper at ICLR 2020
ResDAVEnet-VO Code Indices
$. $1 (osif c888865
TIMIT Phone Labels
Figure 3: Conditional probability matrix displaying P(phonel umit) using the "Ø (2)" model with VQ2 codebook size of 128. For visualization, we saturate the color scaling at probability 0.5.
clusters in order group together phones that share similar sets of VQ codes. Visually, there is strong mapping between TIMIT phone labels and ResDAVEnet-VO codes. In some cases, redun- dant codes are used for the same phone label (this is especially the case for the silence label), and in other cases we see that phones belonging to the same manner class often tend to share codebook units. We can numerically quantify the mapping between the phone and unit labels with the normal-
ized mutual information measure (NMI), which we found to be .378 in this case. We also include
several caption spectrograms with their time-aligned unit sequences in Figures 5 and in the supplementary material.
Table Performance of the VQ3 layer from the "{3) 32, model when codes are treated as word detectors. Codes are ranked by the highest Fl score among the retrieved words for given code. Word hypotheses for a given code are ranked by the FI score. denotes precision, recall, and occ the number of co-occurrences of the code and word in the data.
Top Hypotheses F1
Second Hypotheses
rank
code
word
occ
word
occ
918
pantry
90.67
88.29
93.18 18
41
spice
3.96
2.20
20.00
596
kitchen
90.08
91.59
88.63
304
countertop
1.64
0.84
29.63
8
3
88
classroom
88.97
89.05
88.89
72
classrooms
5.01
2.57
100.00
2
4
58
baseball
88.71
88.63
88.78
182
player
3.01
1.65
17.11
13
706
background
87.86
91.93
84.14
838
ground
0.58
0.39
1.18
4
198
237
lobby
68.43
56.77
86.11
31
waiting
9.93
7.86
13.46
14
199
829
shirt
68.41
71.49
65.58
322
shirts
18.28
10.37
76.79
43
59
grass
68.31
56.53
86.28
503
grassy
15.30
8.67
65.35
4.2
FROM PHONES TO WORDS: LEARNING A HIERARCHY OF UNITS
As shown in Table T all of the ResDAVEnet- VQ models which underwent cold-start training of VQ3 exhibited similar phenomenon in which the ABX error rate of that layer was particularly high, despite the model performing well at the image-caption retrieval task. We hypothesized that this could be due to VQ3 learning to recognize higher level linguistic units, such as words. To examine this empirically, we inferred the VQ3 unit sequence for every audio caption in the Places Audio training set according to several different models. Using the estimated word-level transcriptions of the utterances (provided by the Google SpeechReooonition API), we computed precision, recall, and Fl scores for every unique (word, VQ3 code) pair for given model and quantization layer. We then ranked the VQ codes in descending order according to their maximum Fl score for any word in the vocabulary. Table shows a sampling of these statistics for model "[3} + {2, 3}". In the supplementary material, include many more examples for this model in Table 7 as well as

---

Published as a conference paper at ICLR 2020
Words of Top 250 F1-Scores
Number of Words ahove Fl Threshold
122) (2.3)
201
":3""
Numbar of Words
"22,33'
227
*0-x02 33
Recall
[2)'
227
0.4
112) -02 113) (2 (2.3))
((3)
0.4
0.6
1.0 5.0
5.4
0.0
Precision
F1 Threshold
Figure 4: Visualization of the precision, recall, and Fl scores of individual VQ3 codes when treated as word detectors on the Places Audio captions.
examples for the [[2} + 32. model (which did not learn VQ3 word detectors) in Table g It should be emphasized that these models are exactly the same all respects, except for the order in which their quantizers were trained.
We examine the overall performance of VQ3 as a word detector for these models in Figure f The right hand side of Figure 4 displays the number of VQ3 codes whose maximum Fl score is above a given threshold, while the left hand side shows the distribution of precision and recall scores for the top 250 words ranked by FI. This gives approximate indication of how many VQ3 codes have learned to specialize as detectors for a specific word. We see that the VQ3 layer of model "(3] - {2, 3)" learns 279 codebook entries with F1 score above 0.5. In contrast, the VQ3 layer of model "{2) + [2,3}" learns only a handful of word-detecting codebook entries with an F1 of greater than 0.5. This experiment supports the notion that the reason for the poor ABX performances of the VQ3 layer in models "[3]" and "(3) + {2, 3)" " is in fact due to its specialization for detecting specific words, and that this specialization only emerges when the VQ3 layer is learned before the VQ2 layer. Section A.2 in the supplementary material examines this phenomenon in greater experimental detail.
CONCLUSIONS
In this paper, we demonstrated that the neural vector quantization layers proposed by van den Oord et al.] (2017) can be integrated into the visually- (2019). This resulted in the ability of the speech model to directly represent speech units, such as phones and words, discrete latent variables. We presented extensive experiments and analysis of these learned representations, demonstrating significant improvements ability over the current state-of-the- art models for sub- word speech unit discovery. We demonstrated that these units are also far more robust to noise and domain shift than units derived from previously proposed models. These results supported the notion that semantic supervision via discriminative, multimodal grounding objective has the potential to be more powerful than reconstruction-basee objectives typically used in unsupervised speech models.
speech models proposed by Harwath et al.
phone discrimination
We also showed how multiple vector quantizers could be employed simultaneously within a single ResDAVEnet-vO model, and that these quantizers could be made to specialize in learning hier- archy of speech units: specifically, phones in the lower quantizer and words in the upper quantizer. Our analysis showed that hundreds of codebooks in the upper quantizer learned to perform as word detectors, and that these detectors were highly accurate. Our experiments also revealed that this behavior only emerged when VQ3 was trained before VQ2. These results suggest the importance of the learning curriculum, which should be more deeply investigated in future work. Future work should attempt to make explicit what kind of compositional rules are implicitly encoded by these models when mapping sequences of codes from the lower quantizer word-level units in the up- per quantizer; the automatic derivation sub-word unit inventory, vocabulary, and pronunciation lexicon could serve the starting point for fully unsupervised speech recognition system. Future work should also investigate whether layers above VQ3 could made to learn even higher-level linguistic abstractions, such grammar, syntax, and compositional reasoning.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Afra Alishahi, Marie Barking, and Grzegorz Chrupata. Encoding of phonology in a recurrent neural model of grounded speech. In Proc. ACL Conference on Natural Language Learning (CoNLL), 2017.
Emmanuel Azuh, David Harwath, and James Glass. Towards bilingual lexicon discovery from visu- ally grounded speech audio. In Proc. Annual Conference of International Speech Communication Association (INTERSPEECH), 2019.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 330.3432,
Chung-Cheng Chiu, Tara N Sainath, Yonghui Wu, Rohit Prabhavalkar, Patrick Nguyen, Zhifeng Chen, Anjuli Kannan, Ron Weiss, Kanishka Rao, Ekaterina Gonina, al. State-of-the-an speech recognition with Acoustics, Speech and Signal Processing (ICASSP), 2018.
models. In Proc. International Conference on
Suhce Cho, Yeonjung Hong. Yookyunk Shin, and Youngsun Cho. VQVAE with speaker adversarial training, 2019, URL htttps://ggthhbb com/ /tt.eee3///errsseechhh2/
Jan Chorowski, Ron J. Weiss, Samy Bengio, and Aäron van den Oord. Unsupervised speech rep- resentation learning using wavenet autoencoders. IEEE Transactions on Audio, Speech and Lan- guage Processing, 2019.
Grzegorz Chrupata.
Symbolic inductive bias for visually grounded learning of spoken language. In Proc. Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
Grzegorz Chrupata, Lieke Gelderloos, and Afra Alishahi. Representations of language in model of visually grounded speech signal. In Proc. Annual Meeting of the Association for Computational Linguistics (ACL), 2017.
Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James R. Glass. An unsupervised autoregressive model for speech representation learning. In Proc. Annual Conference of International Speech Communication Association (INTERSPÉECH), 2019.
Jennifer Drexler and James Glass. Analysis of audio-visual features for unsupervised speech recog- nition. Proc. Grounded Language Understanding Workshop. 2017.
Ewan Dunbar, Xuan Nga Cao, Juan Benjumea, Julien Karadayi, Mathieu Bernard, Laurent Besacier, Xavier Anguera, and Emmanuel Dupoux. The zero resource speech challenge 2017. In Proc. IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), 2017.
Ewan Dunbar, Robin Algayres, Julien Karadayi, Mathieu Bernard, Juan Benjumea, Xuan-Nga Cao, Lucie Miskic, Charlotte Dugrain, Lucas Ondel, Alan W. Black, Laurent Besacier, Sakriani Sakti, and Emmanuel Dupoux. The zero resource speech challenge 2019: TTS without T. In Proc. Annual Conference of International Speech Communication Association (INTERSPEECH), 2019.
Emmanuel Dupoux. Cognitive science in the of artificial intelligence: A roadmap for reverse- engineering the infant language-learner. In Cognition, 2018.
Ryan Eloff, Herman Engelbrecht, and Herman Kamper. Multimodal one-shot learning of speech and images. In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019a.
Ryan Eloff, André Nortje, Benjamin van Niekerk, Avashna Govender, Leanne Nortje, Arnu Preto- rius, Elan van Biljon, Ewald van der Westhuizen, Lisa van Staden, and Herman Kamper. Un- supervised acoustic unit discovery for speech synthesis using discrete latent-variable neural net- works. In Proc. Annual Conference of International Speech Communication Association (INTER- SPEECH), 2019b.
Siyuan Feng, Tan Lee, and Zhiyuan Peng. Combining adversarial training and disentangled speech representation for robust zero-resource subword modeling. In Proc. Annual Conference of Inter- national Speech Communication Association (INTERSPEECH), 2019.

---

Published as a conference paper at ICLR 2020
Herb Gish, Man-Hung Siu, Arthur Chan, and William Belfield. Unsupervised training of an HMM- based speech recognizer for topic classification. In Proc. Annual Conference of International Speech Communication Association (INTERSPEECH), 2009.
Google. Google cloud speech-to-text API.
com/
speech-to-texte 2019. Accessed: 2019-09-16.
David Harwath and James Glass. Deep multimodal semantic embeddings for speech and images. In Proc. IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), 2015.
David Harwath and James Glass, Learning word-like units from joint audio-visur analysis. Proc. Annual Meeting of the Association for Computational Linguistics (ACL), 2017.
David Harwath and James Glass. Towards visually grounded sub-word speech unit discovery. In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2019.
David Harwath, Antonio Torralba, and James R. Glass. Unsupervised learning of spoken language with visual context. In Proc. Neural Information Processing Systems (NewrIPS), 2016.
David Harwath, Galen Chuang, and James Glass. Vision as an interlingua: Learning multilingual semantic embeddings of untranscribed speech. In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018a.
David Harwath, Adrià Recasens, Dídac Surís, Galen Chuang, Antonio Torralba, and James Glass. Jointly discovering visual objects and spoken words from raw sensory input. In Proc. IEEE European Conference on Computer Vision (ECCV), 2018b.
David Harwath, Adrià Recasens, Dídac Surís, Galen Chuang, Antonio Torralba, and James Glass. Jointly discovering visual objects and spoken words from raw sensory input. International Jour- nal of Computer Vision, 2019.
William Havard, Jean-Pierre Chevrot, and Laurent Besacier. Models of visually grounded speech signal pay attention to nouns: bilingual experiment on English and Japanese. In Proc. Interna- tional Conference on Acousties, Speech and Signal Processing (ICASSP). 2019a.
William N. Havard, Jean-Piert Chevrot, and Laurent Besacier. Word recognition, competition, and activation in a model of visually grounded speech. Proc. ACL Conference on Natural Language Learning (CoNLL), 2019b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Nils Holzenberger, Mingxing Du, Julien Karadayi, Rachid Riad, and Emmanuel Dupoux. Learning word embeddings: Unsupervised methods for fixed-size representations of variable-length speech segments. In Proc, Annual Conference of International Speech Communication Association (IN- TERSPEECH)) 2018.
Nils Holzenberger, Shruti Palaskar, Pranava Madhyastha, Florian Metze, and Raman Arora. Learn- ing from multiview correlations in open-domain videos. In Proc. International Conference Acoustics, Speech and Signal Processing (ICASSP), 2019.
Wei-Ning Hsu and James Glass. Scalable factorized hierarchical variational autoencoder training. In Proc. Annual Conference of International Speech Communication Association (INTERSPEECH)) 2018.
Wei-Ning Hsu, Yu Zhang, and James Glass. Learning latent representations for speech generation and transformation. In Proc. Annual Conference of International Speech Communication Associ- ation (INTERSPEECH), 2017a
Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable representations from sequential data. In Proc. Neural Information Processing Systems (NeurIPS), 2b.
12

---

Published as a conference paper at ICLR 2020
Wei-Ning Hsu, David Harwath, and James Glass. Transfer learning from audio grounding to speech recognition. In Proc. Annual Conference of International Speech Communication Associ- ation (INTERSPEECH), 2019.
Gabriel Ilharco, Yuan Zhang, and Jason Baldridge. Large-scale representation learning from visu- ally grounded untranscribed speech. In Proc. ACL Conference on Natural Language Learning (CoNLL.), 2019.
Aren Jansen and Benjamin Van Durme. Efficient spoken term discovery using randomized algo- rithms. Proc. IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), 2011.
Aren Jansen, Kenneth Church, and Hynek Hermansky. Toward spoken term discovery at scale with zero resources. In Proc. Annual Conference of International Speech Communication Association (INTERSPEECH), 2010.
Aren Jansen, Manoj Plakal, Ratheet Pandya, Daniel P.W. Ellis, Shawn Hershey, Jiayang Liu, R. Channing Moore, and Rif A. Saurous. Unsupervised learning of semantic audio represen- tations. In Proc. International Conference Acoustics, Speech and Signal Processing (ICASSP), 2018.
Herman Kamper and Michael Roth. Visually grounded cross-lingual keyword spotting in speech. In Proc. of the Workshop on Spoken Language Technologies for Under-Resourced Languages (SLTU), 2017.
Herman Kamper, Aren Jansen, and Sharon Goldwater. Fully unsupervised small-vocabulary speech recognition using segmental Bayesian model. In Proc. Annual Conference of International Speech Communication Association (INTERSPEECI), 2015.
Herman Kamper, Aren Jansen, and Sharon Goldwater. Unsupervised word segmentation and lexicon discovery using acoustic word embeddings. IEEE Transactions on Audio, Speech and Language Processing, 2016.
Herman Kamper, Aren Jansen, and Sharon Goldwater. A segmental framework for fully- unsupervised crrge-vocabllryy speech recognition. Computer Speech and Language, 46(3):154- 174, 2017a
Henman Kamper, Karen Livescu, and Sharon Goldwater. An embedded segmental k-means model for unsupervised segmentation and clustering of speech. In Proc. IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), 2017b.
Herman Kamper, Shane Settle, Gregory Shakhnarovich, and Karen Livescu. Visually grounded learning of keyword prediction from untranscribed speech. In Proc. Annual Conference of Inter- national Speech Communication Association (INTERSPEECH), 2017c
Henman Kamper, Aristotelis Anastassiou, and Karen Liveseu. Semantic query-by-exammee speech search using visual grounding. In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019a.
Herman Kamper, Gregory Shakhnarovich, and Karen Livescu. Semantic speech retrieval with a visually grounded model of untranscribed speech. IEFE Transactions Audio, Speech and Language Processing, 2019b.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. Interna- tional Conference on Learning Representations (ICLR), 2014.
Chia- Ying Lee and James Glass. A nonparametric Bayesian approach to acoustic model discovery. In Proc. Annual Meeting of the Association for Computational Linguistics (ACL), 2012.
Chia-Ying
Chia-Ying Ying Lee, Timothy J. O' O'Donnell, and James Glass. Unsupervised lexicon discovery from acoustic input. In Proc. Annual Meeting of the Association for Computational Linguistics (ACL), 2015.
13

---

Published as a conference paper at ICLR 2020
Kenneth Leidal, David Harwath, and James Glass. Learning modality-ivvariant invariaat representations for speech and images. In Proc. IEEE Workshop on Automatic Speech Recognition and Understand- ing (ASRU), 2017.
M. Paul Lewis, Gary F. Simon, and Charles D. Fennig. Ethnologue: Languages of the World, Nineteenth edition. SIL International. Online version: http://www.cchnollguuccco 2016.
Andy T Liu, Po-chun Hsu, and Hung-yi Lee. Unsupervised end-to-end learning of discrete linguistic units for voice conversion. In Proc. Annual Conference of International Speech Communication Association (INTERSPEECT)), 2019.
Danny Merkx, Stefan L. Frank, and Mirjam Ernestus. Language learning using speech to image retrieval. In Proc. Annual Conference of International Speech Communication Association (IN- TERSPEECH), 2019.
Benjamin Milde and Chris Biemann. Unspeech: Unsupervised speech context embeddings. In Proc. Annual Conference of International Speech Communication Association (INTERSPEECH), 2018.
Lucas Ondel, Lukás Burget, and Jan Černocky. Variational inference for acoustic unit discovery. In Proc. of the Workshop on Spoken Language Technologies for Resour-e Languages (SLTU), 2016.
Alex Park and James Glass. Towards unsupervised pattern discovery in speech. In Proc. IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), 2005.
Alex Park and James Glass. Unsupervised pattern discovery in speech. IEEE Transactions on Audio, Speech and Language Processing, 2008.
Ankita Pasad, Bowen Shi, Herman Kamper, and Karen Livescu. On the contributions of visual and textual supervision low-resource semantic speech retrieval. Proc. Annual Conference of International Speech Communication Association (INTERSPEECHI, 2019.
Santiago Pascual, Mirco Ravanelli, Joan Serrà, Antonio Bonafonte, and Yoshua Bengio. Learning problem-agnostic speech representations from multiple self-supervised tasks. Proc. Annual Conference International Speech Communication Association (INTERSPEECH), 2019.
Ali Razavi, Aaron den Oord, and Oriol Vinyals. Generating diverse high-fidelit images with vq-vae-2. arXiv preprint arXiv: Niki:I190 004460 2019.
Deb Roy. Grounded spoken language acquisition: Experiments in word learning. IEEE Transactions Multimedia, 5(2):197-200) 2003.
Deb Roy and Alex Pentland. Learning words from sights and sounds: computational model. Cognitive Science, 26:1 13-116-44,, 2002.
Odette Scharenborg, Laurent Besacier, Alan W. Black, Mark Hasegawa-Johnson, Florian Metze, Graham Neubig. Sebastian Stüker,, Pierre Godard, Markus Müller, Lucas Ondel, Shruti Palaskar, Philip Arthur, Francesco Ciannella, Mingxing Du, Elin Larsen, Danny Merkx, Rachid Riad, Lim- ing Wang, and Emmanuel Dupoux. Linguistic unit discovery from multi-modal inputs in unwrit- ten languages: Summary of the "Speaking Rosetta" ISALT 2017 workshop. In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2018.
Thomas Schatz, Vijayaditya Peddinti, Francis Bach, Aren Jansen, Hynek Hermansky, and Em- manuel Dupoux. Evaluating speech features with the minimal-pair ABX task: Analysis of the classical MFC/PLP pipeline. In Proc. Annual Conference of International Speech Communica- tion Association (INTERSPEECH)) 2013.
Man-Hume Hung. Siu, Herb Gish, Arthur Chan, William Belfield, and Steve Lowe. Unsupervised training of an HMM-based self-organizing unit recognizer with applications to topic classification and keyword discovery. Computer Speech and Language, 28(1):210 2221100222,000 2014.
Didac Surís, Adrià Recasens, David Bau, David Harwath, James Glass, and Antonio Torralba. Learning words by drawing images. In Proc. IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR), 2019.
14

---

Published as a conference paper at ICLR 2020
Gabriel Synnaeve, Maarten Versteegh, and Emmanuel Dupoux. Learning words from images and speech. In Proc. Neural Information Processing Systems (NeurIPS), 2014.
R. Thiolliere, E. Dunbar, G. Synnaeve, M. Versteegh, and E. Dupoux. A hybrid dynamic time warping-deep neural network architecture for unsupervised acoustic modeling. In Proc. Annual Conference of International Speech Communication Association (INTERSPEECI), 2015.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In Proc. Neural Information Processing Systems (NeurIPS), 2017.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. CoRR, abs/1 807 00774, 2018.
Balakrishnan Varadarajan, Sanjeev Khudanpur, and Emmanuel Dupoux. Unsupervised learning of acoustic sub-word units. In Proceedings of ACL-08: HLT. Short Papers, 2008.
Martin Versteegh, Roland Thiolliere, Thomas Schatz, Xuan Nga Cao, Xavier Anguera, Aren Jansen, and Emmanuel Dupoux. The zero resource speech challenge 2015. Proc. Annual Conference of International Speech Communication Association (INTERSPEECH), 2015.
Virginia de Sa. Learning classification with unlabeled data. In Proc. Neural Information Processing Systems (NeurIPS), 1994.
Kilian Q Weinberger and Lawrence K. Saul. Distance metric learning for large margin nearest neighbor classification. Journal of Machine Learning Research (JMLR), 2009.
Yaodong Zhang and James Glass. Unsupervised spoken keyword spotting via segmental dtw on gaussian posteriorgrams. In Proc. IEEE Workshop on Automatic Speech Recognition and Under- standing (ASRU), 2009.
Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for recognition using places database. In Proc. Neural Information Processing Systems (NeurIPS), 2014.
15

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 VARYING THE CODEBOOK SIZE.
In Table 4 we examine the impact of varying the codebook size of model "Ø + {2)" from 128 through 2048. We find that the ABX score is best for 1024 codebook vectors, although the perfor- mance is quite good for all models. Unsurprisingly, models with smaller codebooks also achieve lower bitrates.
Table 4: ABX scores and bitrates for various codebook sizes the clean ZeroSpeech19 English test set, using the "Ø1 + {2}" model.
Codebnok size
R@10
ABX
Bitrate
RIE-Bitrate
Segment-ABX
Segment- Bitrate
128
.772
14.25
295.65
212.27
15.42
179.38
256
.756
12.95
341.18
260.10
14.21
228.07
512
.761
12.59
363.95
288.64
13.10
259.94
1024
.760
11.79
390.61
317,66
12.66
289.11
2048
.768
12.41
360.04
283.68
13.15
254.23
A.2 THE IMPACT OF THE VQ TRAINING CURRICULUM ON THE LOCALIZATION OF WORD DETECTORS
In Section 4.2 we showed that cold-start training of the VQ3 layer caused its codebook vectors to specialize word de- tectors, whereas warm-start training did not. Our subsequent experiments (Table 6) revealed that when adding third quan- tization layer at the Res4 position to a model that did not learn word detectors at VQ3, the VQ4 layer did in fact learn many word detectors. This suggests implicit word recognition ability can be localized at different layers in the ResDAVEnet audio model, and exactly where it emerges depends upon the VQ training curriculum. We hypothesize that this is due in part to two factors:
Table 5: ABX scores on the Ze- roSpeech 19 English test set using features derived from the output of the Res3 block of the ResDAVEnee Enet audio branch (pre-quantization)
Model
Res3 ABX
"Ø"
10.86
"Ø + {2)"
11.61
"Ø + {3)"
10.91
"Ø + (2,3)"
12.68
1. In a warm-star model, whatever type of informa- tion (subword-like, word-like) the continuous model learned to encode at a particular convolutional layer (or residual block) does not change after quantizer is appended to that layer.
"(2]"
11.45
"(2) + (2,3)"
11.37
"[3]"
32.24
"(3) + (2,3)"
28.33
2. In cold-start model, each active quantization layer forms a potential bottleneck, restricting the amount of information that able to pass through to subsequent layers.
According to this hypothesis, if word-level recognition tends to emerge at a particular layer in an unconstrained network with quantization bottleneck, it will stay there when quantization is in- troduced for fine-tuning. However, when a quantization bottleneck is introduced from the very beginning of training, the gradient flowing down into the lower network layers is more constrained during the initial training epochs (when the gradient tends to be the largest). This may have the ef- fect steering the optimizer into a different part of the parameter space, in which word recognition occurs at different layer than it otherwise would.
We present results from two experiments that support this view. In Table 5 we show the ABX scores of the Res3 layer prior to quantization (if present) during the course of three different training curricula resulting in a "{2,3)" 3)" final model. We observe that the ABX error rate changes very little within each individual curriculum. This indicates that the initial model sets the stage for which layer learns to capture phonetic information with the highest salience, and that subsequent training steps do not tend to move this information elsewhere.
16

---

Published as a conference paper at ICLR 2020
Table 6: The number of codehook vectors particular VQ layer that learned to be detector for any word with an Fl score greater than 0.5.
Quantized Layers
Model ID
VQ Layer
Word Detectors (F1 > 0.5)
"(3)" "Ø + [3]"
210
20
93
3)"
100
"Ø -+ (2,3)"
"[2) + (2
32
"(3} + 32
279
[[2] + (2.3) - (2,3.43"
253
displyys the number of word detectors learned by various VQ layers across different training curricula. Here, we claim that a codebook vector belonging to a particular VQ layer has learned to be word detector if its FI score for any word appearing in the test set exceeds 0.5 (as measured on the test set). There are several interesting things to note here. First, we only observe a significant number of word detectors the VQ3 layer when that layer is trained from cold-start. Even when adding a second VQ layer as in the "(3} > (), model, these detectors remain at VQ3.
Jointly training VQ2 and VQ3 together from cold-start results in the word detectors being di- vided between those layers. While this experiment demonstrates that it is possible to jointly train two quantizers at once, the {2,3} model learned the smallest total number of word detectors of any model. Additionally, we were unable to successfully train cold-start {2, 3, 4} model; these experiments suggest that training quantizers one by one may be easier general.
For all models beginning from "Ø" or "[2]" initial model, we do not observe any word detectors either VQ2 or VQ3. However, a third quantizer at the VQ4 position the "(2) + {2,3) + (2.3.4) 4)" model was able to capture words. We hypothesize that in all models not trained from "[3]" initialization, word recognition implicitly learned by the Res4 layer, and adding a quantizer to the output of this layer serves to make the categorical nature of those representations explicit.
A.3 WORD DETECTOR TABLES FOR VARIOUS MODELS
In Table 7 we show a sampling of 50 word-detecting codebook entries from the VQ layer of the "[3} 32,3)" model (many word detectors learned). Analagous results for the "(2) {2,3]"
model (few word detectors learned) are shown in Table
A.4 UNIT VISUALIZATION FOR INDIVIDUAL CAPTION SPECTROGRAMS
To provide better intuitive understanding of what the units learned by our models look like, in Figures 5) 6 and 7 display speech spectrograms for several Places caption fragments. Along with each spectrogram we display the time-aligned, ground-truth, word-level text (top transcription), the inferred unit sequence for the VQ4 layer (middle transcription), and the unit sequence for the VQ3 (bottom transcription) layer. All VQ unit alignments in these figures are derived from the "[2] + (2,3) + (2,3, 4)" model.
17

---

Published as a conference paper at ICLR 2020
Table 7: Performance of the VQ3 layer from the "[3) - {2.3)" model when codes are treated as word detectors. Codes are ranked by the highest Fl score among the retrieved words for given code. Word hypotheses for given code are ranked by the F1 score.
Top Hypotheses
Second Hypotheses
rank
code
word
P
word
F1
918
pantry
90.67
88.29
93.18
41
spice
3.96
2.20
20.00
596
kitchen
90.08
91.59
88.63
304
countertop
1.64
0.84
29.63
88
classroom
88.97
89.05
88.89
72
classrooms
5.01
2.57
100.00
baschall
88.71
88.63
88.78
182
player
3.01
1.65
17.11
706
background
87.86
91.93
84.14
838
ground
0.58
0.39
1.18
736
museum
87.35
82.00
museums
5.47
2.81
100.00
274
subway
87.26
88.34
86.21
assembly
2.85
40.00
116
construction
87.07
89.78
84.52
131
constructed
2.43
1.25
38.46
892
walking
87.16
87.57
86.55
412
walk
7.07
3.97
31.94
557
concrete
86.53
90.98
82.50
99
concur
1.21
0.61
100.00
11
48
desert
86.50
90.30
83.01
171
dozen
2.76
1.49
18.18
2
12
534
background
86.18
81.95
90.86
905
back
8.95
5.83
19.34
64
13
44
patio
85,82
90.87
81.29
113
patios
1.56
0.79
100.00
14
625
background
85.17
92.92
78.61
783
back
1.63
1.01
4.23
14
15
732
closet
84.92
94.64
77.01
67
closets
4.96
2.68
33.33
2
16
30
waterfall
84.90
75.73
96.61
57
waterfalls
14.26
7.68
100.00
17
388
courtyard
84.89
92.16
78.69
48
graveyard
5.50
3.24
18.18
18
560
hospital
84.70
91.48
78.85
41
horses
1.55
1.92
1.30
18
driveway
84.56
90.10
79.66
47
driveways
3.52
1.82
50.00
20
598
palm
84.39
82.08
86.84
plum
1.57
0.79
100.00
21
85
yellow
84.30
84.66
574
yellowish
1.00
100.00
584
playground
84.18
77.37
92.31
play
4.75
9.43
162
stadium
83.82
83.15
74
boardwalk
4.91
769
83.79
93.68
75.79
72
babuons
1.03
100.00
25
193
small
83.55
90.46
77.63
791
smaller
2.15
1.10
50.00
15
26
412
podium
83.53
76.73
91.67
22
auditorium
8.69
5.82
17.14
27
108
highway
83.52
79.58
87,88
58
highlights
5.44
2.87
50.00
3
28
394
church
83.34
75.98
92.28
227
religious
6.34
3.45
39.39
13
29
661
distance
83.32
78.96
88.19
351
lounge
1.63
0.86
14.71
5
30
708
distance
82.97
96.32
72.86
290
farmland
1.35
0.68
55.56
5
31
14
gallery
82.97
85.08
80.95
17
art
11.39
12.15
10.71
32
996
large
82.95
87.05
79.21
1753
2.83
1.72
8.01
94
33
944
cathedral
82.78
79.22
86.67
52
feed
2.74
1.41
50.00
34
122
purple
82,63
91.98
75.00
138
proportion
1.66
0.84
50.00
35
630
trees
82.52
80.39
84.77
1258
15.48
9.47
42.33
171
186
375
boy 69.90
65.45
75.00
93
boys
20.87
13.09
51.43
18
634
ground
69.78
66.08
224
playground
7.45
3.94
69.23
courtyard
69.55
57.98
86.89
plaza
28.89
wooden
69.50
86.92
525
wood
23.55
62.20
153
190
lighthouse
69.41
59.74
82.81
53
lighthouses
11.69
6.21
100.00
5
191
225
house
69.13
61.59
78.78
516
houses
18.29
10.31
80.73
192
705
dark
69.11
68.57
69.66
186
darker
3.05
1.56
75.00
193
980
building
69.10
77.48
62.35
1161
buildings
25.72
15.71
70.78
281
194
844
grass
69.01
61.85
78.04
455
grassy
21.70
12.42
85.83
109
195
446
lake
68.69
78.63
60.98
125
late
5.02
2.90
18.52
5
196
182
trash
68.64
66.79
70.59
48
boulders
10.16
6.27
26.67
4
197
437
photograph
68.63
59.06
81.89
588
photographs
30.56
18.46
88.73
181
198
237
lobby
68.43
56.77
86.11
31
wailing
9.93
7.86
13.46
14
199
829
shirt
68.41
71.49
65.58
322
shirts
18.28
10.37
76.79
200
59
grass
68.31
56.53
86.28
503
grassy
15.30
8.67
65.35
83
18

---

Published as a conference paper at ICLR 2020
Table 8: Performance of the VQ3 layer from the "[2] + {2,3}" model when codes are treated as word detectors. Codes are ranked by the highest Fl score among the retrieved words for given code. Word hypotheses for given code ranked by the F1 score.
Top Hypotheses Fi
Second Hypotheses
rank
code
word
R
word
F1
P
R
924
people
76.71
67.49
88.85
1665
computer
2.17
1.12
40.40
40
749
white
76.47
66.92
89.21
2265
one
4.15
2.50
12.14
134
530
building
75.47
64.84
90.28
1681
buildings
23.93
13.81
89.67
356
4
505
blue
59.12
46.90
79.96
1093
pool
10.74
5.89
60.80
152
581
snow
57.61
41.77
92.83
466
snowy
16.63
9.12
94.50
103
778
building
52.10
36.71
89.69
1670
buildings
14.30
7.78
88.16
350
144
with
49.12
41.58
59.99
3386
wooden
6.32
3.34
60.60
366
299
small
47.83
32.78
88.42
901
snow
30.55
18.35
91.24
458
550
large
45.13
30.50
86.76
1920
car
8.57
4.52
82.13
216
10
76
trees
44.82
29.76
90.77
1347
tree
15.21
8.31
89.36
361
water
41.84
27.50
87.43
1210
wall
17.57
79.97
12
1015
large
39.59
26.06
82.29
1821
cars
6,58
3.42
84.21
224
13
80
red
39.15
26.13
78.05
992
bed
9.14
4.91
66.67
168
14
719
woman
38.71
24.95
86.31
687
women
7.75
4.09
72.41
126
15
614
people
37.71
25.10
75.83
1421
table
22.10
12.75
82.93
656
water
37.71
24.25
84.75
1173
river
5.72
2.97
80.22
215
457
sky
35.71
23.20
77.47
540
skies
11.81
6.33
88.12
141
18
480
has
34.88
25.01
57.64
1204
house
9.88
5,48
50.38
330
19
245
yellow
34.14
21.17
88.05
597
flowers
13.44
7.27
89.43
237
20
968
picture
34.11
22.32
72.33
1686
pictures
16.79
9.38
79.77
698
trees
33.88
22.04
73.25
1087
tree
10.96
5.93
72.52
293
22
536
man
33.53
20.71
88.06
1128
standing
9.06
4.86
66.17
532
23
black
33.49
20.81
85.77
1163
background
21.15
12.28
76.20
759
24
815
with
33.21
22.76
61.36
3463
white
8.60
4.85
38.05
966
293
large
33.13
23.80
54.50
1206
bridge
19.09
10.78
83.18
371
trees
32.87
20.79
78.44
1164
train
12.97
7.00
88.03
353
27
153
yellow
32.42
19.94
86.73
588
area
15.39
9.47
40.92
354
28
243
front
32.13
20.97
68.69
895
from
14.34
8.51
45.49
358
29
538
black
31.40
19.64
78.24
1061
glass
8.28
4.36
82.90
223
526
small
31.34
19.08
87.83
895
large
5.91
4.04
11.03
244
picture
31.32
20.90
62.46
1456
pictures
13.98
7.80
67.54
32
133
white
29.82
18.98
69.55
1766
black
16.34
9.32
66.37
900
33
715
white
29.45
19.73
58.05
1474
like
9.92
5.89
31.29
388
39
picture
29.37
22.51
42.26
985
like
9.82
5.79
32.26
400
trees
29.31
18.52
70.35
1044
showcasing
3.88
61.51
155
186
129
area
6.62
3.69
32.37
snow
6.08
3.23
51.20
257
187
288
picture
6.61
3.76
27.63
644
pictures
4.91
2.58
52.91
463
188
374
there's s
6.61
3.73
28.71
775
that
5.14
2.91
21.81
614
189
1003
front
6.59
3.65
34.00
443
4.56
2.64
16.84
34S
190
396
multi
6.45
6.25
6.67
towers
5.41
6.25
4.76
191
790
photo
6.23
3.35
44.27
247
by
5.66
3.06
37.60
229
192
522
field
6.14
3.22
63.22
385
photo
6.01
3.19
51.79
289
193
42
man
5.96
3.19
45.12
578
middle
3.85
1.98
68.11
314
194
801
sitting
5.91
3.15
48.56
405
with
5.00
3.65
7.92
447
trees
3.16
43.19
with
5.13
12.72
718
196
791
with
5.84
3.58
15.86
895
that
3.21
1.79
15.88
447
197
975
purple
5.72
2.99
64.13
118
parked
4.67
2.42
65.93
120
198
38
structure
5.62
6.20
5.14
13
graph
4.03
2.19
25.00
2
199
432
parking
5.61
2.90
83.12
133
park
5.56
2.87
82.89
126
standing
5.60
49.63
399
woman
3.09
34.67
276
19

---

Published as a conference paper at ICLR 2020
250
SPREET
302
010
101
THE
351
007
600
196
804
507
309
702
301
300
600
10 Time Secoonds
Time feeecooea
114
ACPPL
386
RCOREE
200
20
110
1400
WAYY
205
872
950
132
680
542
250
285
8000
7000
8000
5000
2000
1000
8000
7000
4000
3000
1000
62H Aouanbsay
960
020 Anuanbayy
Figure 5: Two different captions containing the phrase "many people." " In both cases, the VQ4 layer infers the same unit sequence (872, 360, 712, middle transcription) beneath the phrase. The VQ3 units are somewhat noisier, but contain the common subsequence (956, 265, 80, 401, 262, 762, 246, 774, 828, 386, bottom transcription).
20

---

Published as a conference paper at ICLR 2020
57
25
724
100
706
724
8
500
144
301
77
600
10
150
400
733
100
Time Secords)
154
000
994
Timee
100
200
144
101
300
633
200
40
800
400
102
500
101
280
7B
000
500
600
04
208
47
360
1944
1000
600
9000
1000
2000
1000
8000
2000
6000
5000
1000
2000
2000
Asuanbary
(260 Arcombou
Figure 6: Two different captions containing word "train". In both cases, the VQ4 layer infers the same unit sequence (680, 248, top transcription) surrounding the word "train". The VQ3 alignments contain the same common subsequence (358, 306, 908, 564, 950, 770, bottom transcription). Notice that the same (358, 306) VQ3 unit sequence is aligned to the /tr/ phone cluster at the beginning of both instances of the word "train," well the /tr/ at the beginning of both instances of "tree" in Figure Unit 358 is also found covering the /tr/ at the beginning of the word "tracks" in the topmost spectrogram (although unit 306 is absent in this case).
21

---

Published as a conference paper at ICLR 2020
20
11
OREPLOOSS
220
10
500
200
522
600
200
951
000
10
122
100
700
200
671
Time Evecomet
677
30
1823
600
23
34
200
200
22
22
200
232
122
176
100
123
225
500
50
82
63
30
254
2001
635
0000
2000
2000
5000
1000
5000
I000
723 m00
10:0
7010
1000
9000
2000
3000
3000
Eail Kbuanbar
Figure 7: Two different captions containing word "trees". In both cases, the VQ4 layer infers the same unit sequence (8. 412. 50, top transcription) surrounding the word "trees". The VQ3 alignments contain the same common subsequence (358, 306, 648, 677, 730, bottom transcription). Notice that the (358, 306) VQ3 unit sequence is aligned to the /tr/ phone cluster the beginning of both instances of "trees." and this same unit sequence is inferred for the /tr/ phone sequence in both instances "train" Figure
22


Reviews:
[{"review_id": "SyxuhAn2FS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper attempts to learn discrete speech units in a hierarchical (phone and word) fashion by incorporating multiple vector quantization layers into the audio encoder branch of a model that visually grounds speech segments with accompanying images.\n\nThe model has been tested and compared against two algorithms and implementations that set the SOTA on the Zero Speech 2019 challenge (further improving one of them in the process, it seems), and outperforms these significantly using the ABX metric, so the proposed method seems to perform well (the model is using additional supervision, though). In addition, this is an interesting and timely research problem with implications far beyond the core machine learning setup. The hierarchical setup, and the finding that successful learning here depends on the curriculum, is intriguing indeed. The paper is a pleasure to read and provides a rich set of results and analyses. \n\nA few remarks:\n- It is probably worth explaining how the ABX test is performed, i.e. that features are extracted from some layer of a model, and then a time alignment is performed to get the score - this is in the text somehow, but i had to read it multiple times.\n- Did you try other architectures like 5 layers (rather than 4) in Figure 2\n- Figure 2 is a bit hard to interpret. Maybe plot log of ABX error rate or something, to pull apart the different layers?\n- Could you explain the difference between cold-start and warm-start? One is adding the discretization to a pre-trained model, the other is training from the start?\n- When you measure ABX at layer 2 and 3, in a model trained with quantization, do you measure ABX on the features before or after quantization? does it make a difference?\n- Table 7: some of the top word hypothesis pairs make sense acoustically (building-buildings, red-bed, ...), some could be neighboring words (large-car, ...), but some are just weird (people-computer) - any intuition as to what is going on?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJxsX9aL9B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pretty interesting paper attempting to learn discrete linguistic units via vector quantization of visually grounded, speech related features. I think this is a worthwhile contribution. My main complaint is that the exposition is a bit diffuse and fails to crystallize the essence of the work. In particular, the claim is that the novelty is from the use of a \"discriminative, multi-modal grounding objective\". Reading the work, this seems to be the triplet loss described in Section 3.5. Is that the novel objective? In my my the really interesting aspect that should be stressed is the visual grounding -- I encourage the authors to highlight that aspect more directly. I fear that essential and interesting point is somewhat diluted in the detailed exposition of results.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlEmq1jFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview:\n\nThe paper proposes a method to learn discrete linguistic units in a low-resource setting using speech paired with images (no labels). The visual grounding signal is different from other recent work, where a reconstruction objective was used to learn discrete representations in unsupervised neural networks. In contrast to other work, a hierarchy of discretization layers are also considered, and the paper shows that, with appropriate initialization, higher discrete layers capture word-like units while lower layers capture phoneme-like units.\n\nStrengths:\n\nThe paper is extremely well-written with a clear motivation (Section 1). The approach is novel. But I think the paper's biggest strength is in its very thorough experimental investigation. Their approach is compared to other very recent speech discretization methods on the same data using the same (ABX) evaluation metric. But the work goes further in that it systematically attempts to actually understand what types of structures are captured in the intermediate discrete layers, and it is able to answer this question convincingly. Finally, very good results on standard benchmarks are achieved.\n\nWeaknesses:\n\nAlthough I think the paper is very well-motivated, my first criticism is that discretization itself is not motivated: why is it necessary to have a model with discrete intermediate layers? Does this give us something other than interpretability (which we obtain due to the sparse bottleneck)? In the detailed questions below, I also specifically ask whether, for instance, the downstream speech-image task actually benefits from including discrete layers.\n\nMy second point is that it is unclear why word-like units only appear when the higher-level discrete layers are trained from scratch; as soon as warm-starting is used, the higher level layers capture phoneme-like units (Table 1). Is it possible to answer/speculate why this is the case?\n\nOverall assessment:\n\nThe paper presents a new approach with a thorough experimental investigation. I therefore assign an \"accept\". The weaknesses above asks for additional motivation and some speculation.\n\nQuestions, suggestions, typos, grammar and style:\n\n- Section 3.3: It maybe makes less sense for the end-task, but did the authors consider discretization on the image side of the network? This could maybe lead to parts of objects being composed to form larger objects (in analogy to the speech network).\n- Section 3.3, par. 3: \"with the intention that they should capture discrete word-like and sub-word-like units\" -> \"with the intention that they should capture discrete *sub-word-like and word-like units*\" (easier to read with first part of sentence)\n- Section 3.3: The more standard VQ-VAE adds a commitment loss and a loss for updating the embeddings; was this used or considered at all, or is this all captured through the exponential moving average method?\n- Section 3.4: \"with same VQ layers\" -> \"with *the* same VQ layers\"\n- Section 3.5: Can you briefly outline the motivation for adding the two losses (so that it is not required to read the previous work).\n- Section 4.1: Following from the first weakness listed above, the caption under Figure 2 states that the non-discrete model achieves a speech-image retrieval R@10 of 0.735. This is lower than some of the best scores achieved in Table 1. Can this be taken as evidence that discretization actually improves the downstream task? If so, it would be worth highlighting the point more; if there is some other reason, that would also be worth knowing.\n- Figure 1: Did the authors ever consider putting discrete layers right at the top of the speech component, just before the pooling layer? Would this more consistently lead to word-like units?\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is extremely well-written with a clear motivation (Section 1). The approach is novel. But I think the paper's biggest strength is in its very thorough experimental investigation. Their approach is compared to other very recent speech discretization methods on the same data using the same (ABX) evaluation metric. But the work goes further in that it systematically attempts to actually understand what types of structures are captured in the intermediate discrete layers, and it is able to answer this question convincingly. Finally, very good results on standard benchmarks are achieved.

To authors: Please do include the additional discussions and results in the final paper. 
Decision: Accept (Talk)