Zero-shot Learning for Speech Recognition with Universal Phonetic Model

1 INTRODUCTION . Over the last decade , Automatic Speech Recognition ( ASR ) has achieved great successes in many well-resourced languages such as English , French and Mandarin ( Amodei et al. , 2016 ; Xiong et al. , 2016 ; Collobert et al. , 2016 ) . On the other hand , speech resources are still sparse for the majority of languages ( Lewis , 2009 ) . They can not thus benefit directly from recent technologies . As a result , there is an increasing interest in building speech recognition systems for low-resource languages . As collecting annotations for low resource languages is expensive and time-consuming , researchers have exploited data augmentation ( Ragni et al. , 2014 ; Kanda et al. , 2013 ) , articulatory features ( Stüker et al. , 2003a ) and multilingual techniques ( Schultz & Waibel , 2001 ; Vu & Schultz , 2013 ; Tüske et al. , 2013 ; Dalmia et al. , 2018b ) . An even more challenging problem is to recognize utterances in a language with zero training data . This task has significant implications in documenting endangered languages and preserving the associated cultures ( Gippert et al. , 2006 ) . This data setup has mainly been studied for unsupervised speech processing field ( Glass , 2012 ; Jansen et al. , 2013 ; Versteegh et al. , 2015 ; Dunbar et al. , 2017 ; Heck et al. , 2017 ; Hermann & Goldwater , 2018 ) , which typically uses an unsupervised technique to learn representations which can be used towards speech processing tasks , detailed in Section 4 . Some of them also deal with the Out-Of-Vocabulary ( OOV ) problem in the language modeling side of ASR ( Maas et al. , 2015 ) , or visual-only speech recognition ( Stafylakis & Tzimiropoulos , 2018 ) . In terms of zero-shot learning in speech recognition , which consist of learning an acoustic model without any audio data for a given target language , there has not been much work . Previous works typically consider some amount of audio corpus to be present , because the phonetics of a target language usually differ from existing training languages , especially when there are unseen phonemes which are unavailable in existing language resources . In this work , we aim to solve this problem without even considering any target audio data . The prediction of unseen objects has also been a long-time critical problem in the computer vision field . For specific object classes such as faces , vehicles and cats , a significant number manually labeled data is usually available , but collecting sufficient data for every object human could recognize is impossible . Zero-shot learning attempts to solve this problem to classify even unseen objects using mid-level side information . For example , zebra can be recognized by detecting attributes such as stripped , black and white . Inspired by approaches in computer vision research , we propose the Universal Phonetic Model ( UPM ) to apply zero-shot learning to acoustic modeling . In this model we decompose the phoneme into its attributes and learn to predict a distribution over various phonetic attributes . This can then be used to infer the unseen phonemes for the test language . For example , the phoneme /a/ can be decomposed into its attributes : vowel , open , front and unrounded . Our approach is summarized in Figure 1 . First , frames are extracted and a standard acoustic model is applied to map each frame into the acoustic space H. Next we transform it into the phonetic space P which reflects the phonetic properties of each frame ( such as whether it indicates a vowel or a consonant ) . Then , we compute the distribution of phonemes for that frame using a predefined signature matrix S which describes relationships between phonetic attributes and phonemes in each language . Finally we adjust the phoneme distribution with a prior distribution estimated from a small set of text corpus ( usually 1000 sentences ) . To the best of our knowledge , this is the first paper applying zero-shot learning for acoustic model in speech recognition without any audio data . To evaluate our UPM model , we trained our model on English , and evaluated the model on 20 languages . We also trained a standard English acoustic model as a baseline for comparison . The result indicates that we consistently outperform the baseline CTC model , and we achieved 9.9 % improvements in phone error rate on average . As we mainly focus on acoustic model in this paper , we adopt phone error rate as our main metric in this work . The main contributions of this paper are as follows : 1 . We propose the Universal Phonetic Model ( UPM ) that can recognize phonemes that are unseen during training by incorporating knowledge from the phonetics domain.1 2 . We introduce a sequence prediction model to integrate a zero-shot learning framework for sequence prediction problem . 3 . We show that our model is effective for 20 languages , and our model gets 9.9 % better phone error rate over the baseline on average . 2 APPROACH . This section explains details of our Universal Phonetic Model ( UPM ) . In the first section , we describe how we constructed a proper set of phonetic attributes for acoustic modeling . Next , we 1Our code would be released upon acceptance demonstrate how to assign attributes to each phoneme by giving an algorithm to parse X-SAMPA format . Finally we show how we integrate the phonetic information into the sequence model with a CTC loss ( Graves et al. , 2006 ) . 2.1 PHONETIC ATTRIBUTES . Unlike attributes in the computer vision field , attributes of phones are independent of the corpus and dataset , they are well investigated and defined in the domain of articulatory phonetics ( Ladefoged & Johnson , 2014 ) . Articulatory phonetics describes the mechanism of speech production such as the manner of articulation and place of articulation , and it tends to describe phones using discrete features such as voiced , bilabial ( made with the two lips ) and fricative . These articulatory features have been shown to be useful in speech recognition ( Kirchhoff , 1998 ; Stüker et al. , 2003b ; Müller et al. , 2017 ) , and are a good choice for phonetic attributes for our purpose . We provide some categories of phonetic attributes below . The full list of phonetic attributes we used is attached in the Appendix . Consonants . Consonants are formed by obstructing the airstream through the vocal tract . They can be categorized in terms of the place and the manner of this obstruction . The places can be largely divided into three classes : labial , coronal , dorsal . Each of the class have more fine-grained classes . The manners of articulation can be grouped into : stop , fricative , approximant etc . Vowel . In the production of vowels , the airstream is relatively unobstructed . Each vowel sound can be specified by the positions of lips and tongue ( Ladefoged & Johnson , 2014 ) . For instance , the tongue is at its highest point in the front of the mouth for front vowels . Additionally , vowels can be characterized by properties such as whether the lips are rounding or not ( rounded , unrounded ) . Diacritics . Diacritics are small marks to modify vowels and consonants by attaching to them . For instance , nasalization marks a sound for which the velopharyngeal port is open and air can pass through the nose . To make the phonetic attribute set manageable , we assign attributes of diacritics to some existing consonants attributes if they share similar phonetic property . For example , nasalization is treated as the nasal attribute in consonants . In addition to phonetic attributes mentioned above , we note that we also need to allocate an special attribute for blank in order to predict blank labels in CTC model , and backpropagate their gradients into the acoustic model . Thus , our phonetic attribute set Aphone is defined as the union of these three domain attributes as well as the blank label . Aphone = Aconsonants ∪Avowels ∪Adiacritics ∪ { blank } ( 1 ) 2.1.1 ATTRIBUTE ASSIGNMENT . Algorithm 1 A simple algorithm to assign attributes to phonemes input : X-SAMPA representation of phoneme p output : Phonetic attribute set A ⊆ Aphone for p A← empty set while p 6∈ Pbase do find the longest suffix ps ∈ Pbase Add f ∣∣ Pbase ( ps ) to A Remove suffix ps from p end Add f ∣∣ Pbase ( p ) to A Next , we need to assign each phoneme with appropriate attributes . There are multiple approaches to retrieve phonetic attributes . The most simple one is to use tools to collect articulatory features for each phoneme ( Mortensen et al. , 2016 ; Moran et al. , 2014 ) . However , those tools only provide coarse-grained phonological features and we expect more fine-grained and customized phonetic features . In this section , we propose a naive but useful approach for attribute assignment . We note that we use X-SAMPA format to denote each IPA ( Decker et al. , 1999 ) in this work . X-SAMPA was devised to produce a computer-readable representation for IPA ( Wells , 1995 ) . Each IPA segment can be mapped to X-SAMPA with appropriate rule-based tools ( Mortensen et al. , 2018 ) . For example , IPA / @ / can be represented as @ in X-SAMPA . The assignment can be formulated as a problem to construct an assignment function f : Pxsampa → 2Aphone where Pxsampa is the set of all valid X-SAMPA phonemes . The assignment function should map each phone into its corresponding subset of Aphone . To construct the function in the entire domain , we first manually mapped a small subset Pbase ⊂ Pxsampa and constructed a restricted assignment function f ∣∣ Pbase . The mapping is customizable and has been verified with the IPA handbook ( Decker et al. , 1999 ) . Then for every p ∈ Pxsampa , we continue to remove diacritics suffix from it until it could be find in Pbase . For example , to recognize ts > , we can first match the suffix , > as an ejective , and then recognize ts as a consonant defined in Pbase . The Algorithm 1 summarizes our approach , and the mapping of f ∣∣ Pbase is provided in Appendix . 2.2 SEQUENCE MODEL FOR ZERO-SHOT LEARNING . Zero-shot learning has rarely been applied to sequence prediction problems so far , in this section we describe a novel sequence model architecture for zero-shot learning . We adapt a modified ESZSL architecture from Romera-Paredes & Torr ( 2015 ) . While the original architecture is devised to solve the classification problem with CNN ( DECAF ) features ( Donahue et al. , 2014 ) , our model aims to optimize a CTC loss over a sequence model as shown in Figure 2 . We note our architecture is a general model , and it can also be used for other sequence prediction problems in zero-shot learning . Given the training set S = { ( xn , yn ) , n = 1 ... N } where each input xn ∈ X is an utterance , and yn ∈ Y is the corresponding phonetic transcription . Suppose that xn = ( x1n , ... , xTn ) is the input sequence where xtn is the frame of time step t , and T is the length of xn . Each frame x t n is first projected into a feature vector htn ∈ Rd in the acoustic spaceH with a Bidirectional LSTM model . htn = θ ( x t n ; WLSTM ) ( 2 ) where WLSTM is the parameter of Bidirectional LSTM model . We assume that our phoneme inventory consists of z phonemes in the training set , each of them having a signature of a attributes constructed as mentioned above . Then we can represent our attributes in a constant signature matrix S ∈ [ 0 , 1 ] z×a . Then , we transform htn into points in the phonetic space P with V ∈ Ra×d , and further into the phoneme logits ltn with S. ltn = SV h t n ( 3 ) The logits ln = ( l1n , ... , l T n ) is then combined with yn to compute the CTC loss ( Graves et al. , 2006 ) . Additionally , regularizing V has been proved to be useful in the original ESZSL architecture ( Romera-Paredes & Torr , 2015 ) . So our target is to minimize following loss function . minimize V , WLSTM CTC ( xn , yn ; V , WLSTM ) + Ω ( V ) ( 4 ) where Ω ( V ) is an simple L2 regularization . This objective can be easily optimized using standard gradient descent methods . At inference stage , we usually consider a new language Ltest with a new phoneme inventory . Suppose that the new inventory is composed of z′ classes , then we can create a new signature matrix S′ ∈ [ 0 , 1 ] z′×a , and estimate probability distribution of each phoneme Pacoustic ( p|xtn ) from logits using S′ instead of S. Additionally , we expect the phoneme should distribute similarly to the phoneme distribution in Ltest . We adjusted the posterior as follows . Pupm ( p|xtn ) = Pacoustic ( p|xtn ) P ( p|Ltest ) α ( 5 ) where α is a hyperparameter , and P ( p|Ltest ) is the prior distribution of Ltest which can be easily estimated from a small set of the text corpus in Ltest ( we used 1000 sentences here ) . Finally we exploit greedy decoding to extract phoneme from the adjusted posterior Pupm ( Graves et al. , 2006 ) .

Summary: This paper presents an approach to address the task on zero-shot learning for speech recognition, which consist of learning an acoustic model without any resources for a given language. The universal phonetic model is proposed, which learns phone attributes (instead of phone label), which allows to do prediction on any phone set, i.e. on any language. The model is evaluated on 20 languages and is shown to improve over a baseline trained only on English.