Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech

1 INTRODUCTION . By 8 months of age , human infants learn to recognize not only the names of their caregivers and common objects , but also the contrast between the different vowels and consonants which comprise these words ( Dupoux , 2018 ) . Nearly all toddlers learn to carry a conversation long before they can read and write . Humans learn to model the discrete , hierarchical , and compositional nature of their native language not from written text , but from speech audio - a continuous , time-varying waveform which is the product not only of the underlying words which were spoken , but also the physical properties of the speaker ’ s vocal tract , the speaker ’ s health and emotional state , and the noise and reverberation present in the environment . The question of how such a complex symbolic system is inferred from continuous and noisy sensory input data is of interest not only to the cognitive science community , but also to machine learning researchers who aim to reproduce this ability with computers . A more comprehensive understanding of human language acquisition has practical significance in real-world applications , such as automatic speech recognition ( ASR ) and natural language understanding ( NLU ) systems . In the past several decades , enormous progress has been made in speech recognition research , and nowadays ASR systems are able to achieve human-level accuracy in many domains ( Chiu et al. , 2018 ) . Unfortunately , the techniques that have been developed to achieve these levels of performance are extremely data-hungry , requiring many thousands of hours of speech audio recordings for training . Since supervised machine learning algorithms form the basis of ASR training , the data also needs to be annotated by expert humans . Due to the immense cost of collecting and annotating speech data , ASR technology currently exists for approximately 120 ( Google , 2019 ) out of the nearly 7,000 ( Lewis et al. , 2016 ) human languages spoken worldwide . It is highly unlikely that purely supervised machine learning techniques will be able to scale to include all human languages , necessitating the development of alternative methods by researchers which are able to function with far fewer annotations , or even no annotations at all . Because human beings provide an existence proof of language acquisition from speech completely without language supervision , it is plausible that this ability could be replicated by a machine learning algorithm . ∗Equal contribution In this paper , we present a method for discovering discrete and hierarchical representations of speech units both at the sub-word level and the word level . Previously proposed linguistic unit discovery methods have only leveraged the speech audio modality in isolation , relying on objective functions that attempt to capture statistical regularities within the speech signal . The key innovation in our work is that we discover units by training models with explicit discretization layers to associate speech waveforms with visual images using a cross-modal grounding objective . This forces our models to learn representations which capture semantic information at the highest layers of the network . Because semantics are predominantly carried by words , and words are composed of subword units ( such as phones and syllables ) , the visual grounding objective indirectly forces the model to learn speaker- and noise-invariant representations of speech units . By incorporating trainable quantization layers into our networks , we are able to capture these units in discrete inventories . Whether these units correspond to word-like or sub-word units depends on where the quantization layers are inserted , and how they are trained . 2 RELATED WORK . Prior work on unsupervised modeling of the speech signal has generally focused on learning representations which either disentangle or isolate the latent factors that are of interest for downstream tasks . In most cases the primary latent factor of interest is the phonetic or lexical identity of a given segment of speech , but other factors , such as the identity of the speaker , are sometimes of interest as well . Because the factors of interest are often inherently discrete ( e.g . words and phones ) , many of the proposed approaches attempt to perform segmentation and clustering of the surface features in one way or another . One family of techniques is based upon Segmental Dynamic Time Warping ( S-DTW ) ( Park & Glass , 2005 ; 2008 ; Jansen et al. , 2010 ; Jansen & Van Durme , 2011 ) , which uses a self-comparison algorithm to identify relatively long duration ( on the order of a second ) patterns which frequently reoccur in a speech corpus ; these patterns tend to capture words or short phrases . A different line of work employs probabilistic graphical models to jointly segment and cluster the speech signal ( Varadarajan et al. , 2008 ; Zhang & Glass , 2009 ; Gish et al. , 2009 ; Lee & Glass , 2012 ; Siu et al. , 2014 ; Lee et al. , 2015 ; Ondel et al. , 2016 ; Kamper et al. , 2016 ; 2017a ) . With an appropriately designed model , it is possible to learn multiple , hierarchical categories of speech units . However , in order to enable efficient inference , the conditional distributions of these models tend to be simple and therefore have limited modeling power . Deep neural network models have been successfully used to learn powerful speech representations using weakly or unsupervised objectives ( Thiolliere et al. , 2015 ; Kamper et al. , 2015 ; Hsu et al. , 2017a ; b ; Hsu & Glass , 2018 ; Holzenberger et al. , 2018 ; Milde & Biemann , 2018 ; van den Oord et al. , 2018 ; Chung et al. , 2019 ; Pascual et al. , 2019 ) . These representations have predominantly been continuous in nature , as discrete latent variables are not trivially compatible with backpropagation . To obtain discrete representations , a post-hoc clustering step can be applied to the continuous representations ( Kamper et al. , 2017b ; Feng et al. , 2019 ) . More recently , several papers have proposed ways of directly incorporating discrete variables into neural network models , including using Gumbel-Softmax ( Eloff et al. , 2019b ) or straight-through estimators ( van den Oord et al. , 2017 ; Chorowski et al. , 2019 ; Razavi et al. , 2019 ) . A different method for learning meaningful representations of speech is via a multimodal grounding objective , which encourages the learning of speech representations that are predictive of the contextual information contained in a separate but accompanying modality , such as vision . Visual grounding of speech is a form of self-supervised learning ( Virginia de Sa , 1994 ) , which is powerful in part because it offers a way of training models with a discriminative objective that does not depend on traditional transcriptions or annotations . The first work in this direction relied on phone strings to represent the speech ( Roy & Pentland , 2002 ; Roy , 2003 ) , but more recently this learning has been shown to be possible directly on the speech signal ( Synnaeve et al. , 2014 ; Harwath & Glass , 2015 ; Harwath et al. , 2016 ) . Subsequent work on visually-grounded models of speech has investigated improvements and alternatives to the modeling or training algorithms ( Leidal et al. , 2017 ; Kamper et al. , 2017c ; Havard et al. , 2019a ; Merkx et al. , 2019 ; Chrupała et al. , 2017 ; Scharenborg et al. , 2018 ; Kamper et al. , 2019b ; a ; Surı́s et al. , 2019 ; Ilharco et al. , 2019 ; Eloff et al. , 2019a ) , application to multilingual settings ( Harwath et al. , 2018a ; Kamper & Roth , 2017 ; Azuh et al. , 2019 ; Havard et al. , 2019a ) , analysis of the linguistic abstractions , such as words and phones , which are learned by the models ( Harwath & Glass , 2017 ; Harwath et al. , 2018b ; Drexler & Glass , 2017 ; Alishahi et al. , 2017 ; Harwath et al. , 2019 ; Harwath & Glass , 2019 ; Havard et al. , 2019b ) , and the impact of jointly training with textual input ( Holzenberger et al. , 2019 ; Chrupała , 2019 ; Pasad et al. , 2019 ) . Representations learned by models of visually grounded speech are also well-suited for transfer learning to supervised tasks , being highly robust to noise and domain shift ( Hsu et al. , 2019 ) . 3 DATA AND MODELS . 3.1 DATASET . For training our models , we utilize the MIT Places 205 dataset ( Zhou et al. , 2014 ) and their accompanying spoken audio captions ( Harwath et al. , 2016 ; 2018b ) . The caption dataset contains approximately 400,000 spoken audio captions , each of which describes a different Places image . These captions are free-form spontaneous speech , collected from over 2,500 different speakers and covering a 40,000 word vocabulary . The average caption duration is approximately 10 seconds , and each caption contains on average 20 words . For vetting our models during training , we use a held-out validation set of 1,000 image-caption pairs . 3.2 NEURAL MODELS OF VISUALLY-GROUNDED SPEECH . We base our model upon the Residual Deep Audio-Visual Embedding network ( ResDAVEnet ) architecture ( Harwath et al. , 2019 ) , which contains two branches of fully convolutional networks , one for images and the other for audio . Each branch encodes samples of the corresponding modality into a d-dimensional space , regardless of the original dimensionality of the samples . This is achieved by applying global spatial mean pooling and global temporal mean pooling to the image branch output and the audio branch output , respectively . The image branch is adapted from ResNet50 ( He et al. , 2016 ) , where the final softmax layer and the preceding fully-connected layers are removed , replaced with a 1x1 linear convolutional layer in order to project the feature map to the desired dimension . To model the audio inputs , a 17-layer fully convolutional network with residual connections is used . The input is a log Mel-frequency spectrogram with 40 frequency bins and 25 ms-wide , Hammingwindowed frames with a shift of 10 ms . The first layer of this network is a 1-D convolution that spans the entire frequency axis of the spectrogram , while the remaining 16 convolutional layers are 1-D across the time axis . These 16 layers are divided into four residual blocks of 4 layers each , and downsampling between these blocks is accomplished by applying the first convolution of each block with a stride of 2 . For full details of the model , refer to Harwath et al . ( 2019 ) . 3.3 LEARNING HIERARCHICAL DISCRETE UNITS WITH VECTOR QUANTIZING LAYERS . Previous analyses reveal that ResDAVEnet-like models learn linguistic abstractions at different levels , including words ( Harwath & Glass , 2017 ) and robust phonetic features ( Harwath & Glass , 2019 ; Hsu et al. , 2019 ) . To explicitly learn hierarchical discrete linguistic units within this framework , we propose to incorporate multiple vector quantization ( VQ ) layers ( van den Oord et al. , 2017 ) into the ResDAVEnet audio branch ; we refer to this new architecture as ResDAVEnet-VQ . VQ layers can be understood as a type of bottleneck , which constrain the amount of information that can flow through . While these layers have been used to learn discrete sub-word units ( van den Oord et al. , 2017 ; Chorowski et al. , 2019 ; Razavi et al. , 2019 ) , previous work injects VQ layers into autoencoders that are trained with a reconstruction loss . As a result , the embedding dimension of each code and the number of codes need to be carefully tuned ( Liu et al. , 2019 ) . When the embedding dimension is too low or the codebook size too small , the model does not have enough expressive power to capture linguistic variability . When it is too large , the model starts to encode non-linguistic information in order to improve reconstruction . In contrast , the learning signal of ResDAVEnet-VQ is provided by the visual-semantic grounding objective . Rather than encoding as much information about input as possible , the learned codes in ResDAVEnet-VQ only need to capture semantic information . Since semantics in speech are predominantly transmitted by words , and words are composed of sub-word units like phones , the grounding objective places pressure on the model to robustly infer both from speech . Since words and phones are inherently discrete symbols , representing them with learned discrete units may not even hurt the grounding performance . Figure 1 illustrates the proposed ResDAVEnet-VQ model . We add a quantization layer after each of the first two residual blocks of the ResDAVEnet-VQ model , denoted as VQ2 and VQ3 , respectively , with the intention that they should capture discrete sub-word-like and word-like units . A VQ layer is defined as E ∈ RK×D , where K represents the codebook size , and D represents the output dimensionality of the input features to the codebook . Denoting the tth temporal frame of the input to the quantization layer as xt , quantization is performed according to qt = Ek , : , where k = arg minj ||xt−Ej , :||2 The quantized output is then fed as input to the subsequent residual block . As in van den Oord et al . ( 2017 ) , we use the straight-through estimator ( Bengio et al. , 2013 ) to compute the gradient passed from qt to xt . We use the exponential moving average ( EMA ) codebook updates proposed by van den Oord et al . ( 2017 ) .

Summary: The paper proposes a method to learn discrete linguistic units in a low-resource setting using speech paired with images (no labels). The visual grounding signal is different from other recent work, where a reconstruction objective was used to learn discrete representations in unsupervised neural networks. In contrast to other work, a hierarchy of discretization layers are also considered, and the paper shows that, with appropriate initialization, higher discrete layers capture word-like units while lower layers capture phoneme-like units.