Zero-shot Learning for Speech Recognition with Universal Phonetic Model

1 INTRODUCTION . Over the last decade , Automatic Speech Recognition ( ASR ) has achieved great successes in many well-resourced languages such as English , French and Mandarin ( Amodei et al. , 2016 ; Xiong et al. , 2016 ; Collobert et al. , 2016 ) . On the other hand , speech resources are still sparse for the majority of languages ( Lewis , 2009 ) . They can not thus benefit directly from recent technologies . As a result , there is an increasing interest in building speech recognition systems for low-resource languages . As collecting annotations for low resource languages is expensive and time-consuming , researchers have exploited data augmentation ( Ragni et al. , 2014 ; Kanda et al. , 2013 ) , articulatory features ( Stüker et al. , 2003a ) and multilingual techniques ( Schultz & Waibel , 2001 ; Vu & Schultz , 2013 ; Tüske et al. , 2013 ; Dalmia et al. , 2018b ) . An even more challenging problem is to recognize utterances in a language with zero training data . This task has significant implications in documenting endangered languages and preserving the associated cultures ( Gippert et al. , 2006 ) . This data setup has mainly been studied for unsupervised speech processing field ( Glass , 2012 ; Jansen et al. , 2013 ; Versteegh et al. , 2015 ; Dunbar et al. , 2017 ; Heck et al. , 2017 ; Hermann & Goldwater , 2018 ) , which typically uses an unsupervised technique to learn representations which can be used towards speech processing tasks , detailed in Section 4 . Some of them also deal with the Out-Of-Vocabulary ( OOV ) problem in the language modeling side of ASR ( Maas et al. , 2015 ) , or visual-only speech recognition ( Stafylakis & Tzimiropoulos , 2018 ) . In terms of zero-shot learning in speech recognition , which consist of learning an acoustic model without any audio data for a given target language , there has not been much work . Previous works typically consider some amount of audio corpus to be present , because the phonetics of a target language usually differ from existing training languages , especially when there are unseen phonemes which are unavailable in existing language resources . In this work , we aim to solve this problem without even considering any target audio data . The prediction of unseen objects has also been a long-time critical problem in the computer vision field . For specific object classes such as faces , vehicles and cats , a significant number manually labeled data is usually available , but collecting sufficient data for every object human could recognize is impossible . Zero-shot learning attempts to solve this problem to classify even unseen objects using mid-level side information . For example , zebra can be recognized by detecting attributes such as stripped , black and white . Inspired by approaches in computer vision research , we propose the Universal Phonetic Model ( UPM ) to apply zero-shot learning to acoustic modeling . In this model we decompose the phoneme into its attributes and learn to predict a distribution over various phonetic attributes . This can then be used to infer the unseen phonemes for the test language . For example , the phoneme /a/ can be decomposed into its attributes : vowel , open , front and unrounded . Our approach is summarized in Figure 1 . First , frames are extracted and a standard acoustic model is applied to map each frame into the acoustic space H. Next we transform it into the phonetic space P which reflects the phonetic properties of each frame ( such as whether it indicates a vowel or a consonant ) . Then , we compute the distribution of phonemes for that frame using a predefined signature matrix S which describes relationships between phonetic attributes and phonemes in each language . Finally we adjust the phoneme distribution with a prior distribution estimated from a small set of text corpus ( usually 1000 sentences ) . To the best of our knowledge , this is the first paper applying zero-shot learning for acoustic model in speech recognition without any audio data . To evaluate our UPM model , we trained our model on English , and evaluated the model on 20 languages . We also trained a standard English acoustic model as a baseline for comparison . The result indicates that we consistently outperform the baseline CTC model , and we achieved 9.9 % improvements in phone error rate on average . As we mainly focus on acoustic model in this paper , we adopt phone error rate as our main metric in this work . The main contributions of this paper are as follows : 1 . We propose the Universal Phonetic Model ( UPM ) that can recognize phonemes that are unseen during training by incorporating knowledge from the phonetics domain.1 2 . We introduce a sequence prediction model to integrate a zero-shot learning framework for sequence prediction problem . 3 . We show that our model is effective for 20 languages , and our model gets 9.9 % better phone error rate over the baseline on average . 2 APPROACH . This section explains details of our Universal Phonetic Model ( UPM ) . In the first section , we describe how we constructed a proper set of phonetic attributes for acoustic modeling . Next , we 1Our code would be released upon acceptance demonstrate how to assign attributes to each phoneme by giving an algorithm to parse X-SAMPA format . Finally we show how we integrate the phonetic information into the sequence model with a CTC loss ( Graves et al. , 2006 ) . 2.1 PHONETIC ATTRIBUTES . Unlike attributes in the computer vision field , attributes of phones are independent of the corpus and dataset , they are well investigated and defined in the domain of articulatory phonetics ( Ladefoged & Johnson , 2014 ) . Articulatory phonetics describes the mechanism of speech production such as the manner of articulation and place of articulation , and it tends to describe phones using discrete features such as voiced , bilabial ( made with the two lips ) and fricative . These articulatory features have been shown to be useful in speech recognition ( Kirchhoff , 1998 ; Stüker et al. , 2003b ; Müller et al. , 2017 ) , and are a good choice for phonetic attributes for our purpose . We provide some categories of phonetic attributes below . The full list of phonetic attributes we used is attached in the Appendix . Consonants . Consonants are formed by obstructing the airstream through the vocal tract . They can be categorized in terms of the place and the manner of this obstruction . The places can be largely divided into three classes : labial , coronal , dorsal . Each of the class have more fine-grained classes . The manners of articulation can be grouped into : stop , fricative , approximant etc . Vowel . In the production of vowels , the airstream is relatively unobstructed . Each vowel sound can be specified by the positions of lips and tongue ( Ladefoged & Johnson , 2014 ) . For instance , the tongue is at its highest point in the front of the mouth for front vowels . Additionally , vowels can be characterized by properties such as whether the lips are rounding or not ( rounded , unrounded ) . Diacritics . Diacritics are small marks to modify vowels and consonants by attaching to them . For instance , nasalization marks a sound for which the velopharyngeal port is open and air can pass through the nose . To make the phonetic attribute set manageable , we assign attributes of diacritics to some existing consonants attributes if they share similar phonetic property . For example , nasalization is treated as the nasal attribute in consonants . In addition to phonetic attributes mentioned above , we note that we also need to allocate an special attribute for blank in order to predict blank labels in CTC model , and backpropagate their gradients into the acoustic model . Thus , our phonetic attribute set Aphone is defined as the union of these three domain attributes as well as the blank label . Aphone = Aconsonants ∪Avowels ∪Adiacritics ∪ { blank } ( 1 ) 2.1.1 ATTRIBUTE ASSIGNMENT . Algorithm 1 A simple algorithm to assign attributes to phonemes input : X-SAMPA representation of phoneme p output : Phonetic attribute set A ⊆ Aphone for p A← empty set while p 6∈ Pbase do find the longest suffix ps ∈ Pbase Add f ∣∣ Pbase ( ps ) to A Remove suffix ps from p end Add f ∣∣ Pbase ( p ) to A Next , we need to assign each phoneme with appropriate attributes . There are multiple approaches to retrieve phonetic attributes . The most simple one is to use tools to collect articulatory features for each phoneme ( Mortensen et al. , 2016 ; Moran et al. , 2014 ) . However , those tools only provide coarse-grained phonological features and we expect more fine-grained and customized phonetic features . In this section , we propose a naive but useful approach for attribute assignment . We note that we use X-SAMPA format to denote each IPA ( Decker et al. , 1999 ) in this work . X-SAMPA was devised to produce a computer-readable representation for IPA ( Wells , 1995 ) . Each IPA segment can be mapped to X-SAMPA with appropriate rule-based tools ( Mortensen et al. , 2018 ) . For example , IPA / @ / can be represented as @ in X-SAMPA . The assignment can be formulated as a problem to construct an assignment function f : Pxsampa → 2Aphone where Pxsampa is the set of all valid X-SAMPA phonemes . The assignment function should map each phone into its corresponding subset of Aphone . To construct the function in the entire domain , we first manually mapped a small subset Pbase ⊂ Pxsampa and constructed a restricted assignment function f ∣∣ Pbase . The mapping is customizable and has been verified with the IPA handbook ( Decker et al. , 1999 ) . Then for every p ∈ Pxsampa , we continue to remove diacritics suffix from it until it could be find in Pbase . For example , to recognize ts > , we can first match the suffix , > as an ejective , and then recognize ts as a consonant defined in Pbase . The Algorithm 1 summarizes our approach , and the mapping of f ∣∣ Pbase is provided in Appendix . 2.2 SEQUENCE MODEL FOR ZERO-SHOT LEARNING . Zero-shot learning has rarely been applied to sequence prediction problems so far , in this section we describe a novel sequence model architecture for zero-shot learning . We adapt a modified ESZSL architecture from Romera-Paredes & Torr ( 2015 ) . While the original architecture is devised to solve the classification problem with CNN ( DECAF ) features ( Donahue et al. , 2014 ) , our model aims to optimize a CTC loss over a sequence model as shown in Figure 2 . We note our architecture is a general model , and it can also be used for other sequence prediction problems in zero-shot learning . Given the training set S = { ( xn , yn ) , n = 1 ... N } where each input xn ∈ X is an utterance , and yn ∈ Y is the corresponding phonetic transcription . Suppose that xn = ( x1n , ... , xTn ) is the input sequence where xtn is the frame of time step t , and T is the length of xn . Each frame x t n is first projected into a feature vector htn ∈ Rd in the acoustic spaceH with a Bidirectional LSTM model . htn = θ ( x t n ; WLSTM ) ( 2 ) where WLSTM is the parameter of Bidirectional LSTM model . We assume that our phoneme inventory consists of z phonemes in the training set , each of them having a signature of a attributes constructed as mentioned above . Then we can represent our attributes in a constant signature matrix S ∈ [ 0 , 1 ] z×a . Then , we transform htn into points in the phonetic space P with V ∈ Ra×d , and further into the phoneme logits ltn with S. ltn = SV h t n ( 3 ) The logits ln = ( l1n , ... , l T n ) is then combined with yn to compute the CTC loss ( Graves et al. , 2006 ) . Additionally , regularizing V has been proved to be useful in the original ESZSL architecture ( Romera-Paredes & Torr , 2015 ) . So our target is to minimize following loss function . minimize V , WLSTM CTC ( xn , yn ; V , WLSTM ) + Ω ( V ) ( 4 ) where Ω ( V ) is an simple L2 regularization . This objective can be easily optimized using standard gradient descent methods . At inference stage , we usually consider a new language Ltest with a new phoneme inventory . Suppose that the new inventory is composed of z′ classes , then we can create a new signature matrix S′ ∈ [ 0 , 1 ] z′×a , and estimate probability distribution of each phoneme Pacoustic ( p|xtn ) from logits using S′ instead of S. Additionally , we expect the phoneme should distribute similarly to the phoneme distribution in Ltest . We adjusted the posterior as follows . Pupm ( p|xtn ) = Pacoustic ( p|xtn ) P ( p|Ltest ) α ( 5 ) where α is a hyperparameter , and P ( p|Ltest ) is the prior distribution of Ltest which can be easily estimated from a small set of the text corpus in Ltest ( we used 1000 sentences here ) . Finally we exploit greedy decoding to extract phoneme from the adjusted posterior Pupm ( Graves et al. , 2006 ) .

Summary: This paper proposes to train a Universal Phonetic Model for building speech recognition for new languages without any training data. It suggests to use X-SAMPA to map phones from all the languages into a single phonetic space. The prediction models are designed to first predict the phonetic features and then the phones depending on the target language.