Title: ZERO-SHOT LEARNING FOR SPEECH RECOGNITION WITH UNIVERSAL PHONETIC MODEL

Abstract: There are more than 7,000 languages in the world, but due to the lack of training sets, only a small number of them have speech recognition systems. Multilingual speech recognition provides a solution if at least some audio training data is available. Often, however, phoneme inventories differ between the training languages and the target language, making this approach infeasible. In this work, we address the problem of building an acoustic model for languages with zero audio resources. Our model is able to recognize unseen phonemes in the target language, if only a small text corpus is available. We adopt the idea of zero-shot learning, and decompose phonemes into corresponding phonetic attributes such as vowel and consonant. Instead of predicting phonemes directly, we first predict distributions over phonetic attributes, and then compute phoneme distributions with a customized acoustic model. We extensively evaluate our English-trained model on 20 unseen languages, and find that on average, it achieves 9.9% better phone error rate over a traditional CTC based acoustic model trained on English.

1. INTRODUCTION:

Over the last decade, Automatic Speech Recognition (ASR) has achieved great successes in many well-resourced languages such as English, French and Mandarin (Amodei et al., 2016; Xiong et al., 2016; Collobert et al., 2016) . On the other hand, speech resources are still sparse for the majority of languages (Lewis, 2009) . They cannot thus benefit directly from recent technologies. As a result, there is an increasing interest in building speech recognition systems for low-resource languages. As collecting annotations for low resource languages is expensive and time-consuming, researchers have exploited data augmentation (Ragni et al., 2014; Kanda et al., 2013) , articulatory features (Stüker et al., 2003a) and multilingual techniques (Schultz & Waibel, 2001; Vu & Schultz, 2013; Tüske et al., 2013; Dalmia et al., 2018b ).

An even more challenging problem is to recognize utterances in a language with zero training data. This task has significant implications in documenting endangered languages and preserving the associated cultures (Gippert et al., 2006) . This data setup has mainly been studied for unsupervised speech processing field (Glass, 2012; Jansen et al., 2013; Versteegh et al., 2015; Dunbar et al., 2017; Heck et al., 2017; Hermann & Goldwater, 2018) , which typically uses an unsupervised technique to learn representations which can be used towards speech processing tasks, detailed in Section 4. Some of them also deal with the Out-Of-Vocabulary (OOV) problem in the language modeling side of ASR (Maas et al., 2015) , or visual-only speech recognition (Stafylakis & Tzimiropoulos, 2018) .

In terms of zero-shot learning in speech recognition, which consist of learning an acoustic model without any audio data for a given target language, there has not been much work. Previous works typically consider some amount of audio corpus to be present, because the phonetics of a target language usually differ from existing training languages, especially when there are unseen phonemes which are unavailable in existing language resources. In this work, we aim to solve this problem without even considering any target audio data.

The prediction of unseen objects has also been a long-time critical problem in the computer vision field. For specific object classes such as faces, vehicles and cats, a significant number manually labeled data is usually available, but collecting sufficient data for every object human could recognize is impossible. Zero-shot learning attempts to solve this problem to classify even unseen objects using mid-level side information. For example, zebra can be recognized by detecting attributes such Figure 1 : Illustration of the proposed zero-shot learning framework. Each utterance is first mapped into acoustic space H. Then we transform each point in the acoustic space into phonetic space P with a linear transformation V . Finally phoneme distributions can be obtained by applying a signature matrix S as stripped, black and white. Inspired by approaches in computer vision research, we propose the Universal Phonetic Model (UPM) to apply zero-shot learning to acoustic modeling. In this model we decompose the phoneme into its attributes and learn to predict a distribution over various phonetic attributes. This can then be used to infer the unseen phonemes for the test language. For example, the phoneme /a/ can be decomposed into its attributes: vowel, open, front and unrounded.

Our approach is summarized in Figure 1 . First, frames are extracted and a standard acoustic model is applied to map each frame into the acoustic space H. Next we transform it into the phonetic space P which reflects the phonetic properties of each frame (such as whether it indicates a vowel or a consonant). Then, we compute the distribution of phonemes for that frame using a predefined signature matrix S which describes relationships between phonetic attributes and phonemes in each language. Finally we adjust the phoneme distribution with a prior distribution estimated from a small set of text corpus (usually 1000 sentences). To the best of our knowledge, this is the first paper applying zero-shot learning for acoustic model in speech recognition without any audio data.

To evaluate our UPM model, we trained our model on English, and evaluated the model on 20 languages. We also trained a standard English acoustic model as a baseline for comparison. The result indicates that we consistently outperform the baseline CTC model, and we achieved 9.9% improvements in phone error rate on average. As we mainly focus on acoustic model in this paper, we adopt phone error rate as our main metric in this work.

The main contributions of this paper are as follows:

1. We propose the Universal Phonetic Model (UPM) that can recognize phonemes that are unseen during training by incorporating knowledge from the phonetics domain.1 2. We introduce a sequence prediction model to integrate a zero-shot learning framework for sequence prediction problem.

3. We show that our model is effective for 20 languages, and our model gets 9.9% better phone error rate over the baseline on average.

2. APPROACH:

This section explains details of our Universal Phonetic Model (UPM). In the first section, we describe how we constructed a proper set of phonetic attributes for acoustic modeling. Next, we demonstrate how to assign attributes to each phoneme by giving an algorithm to parse X-SAMPA format. Finally we show how we integrate the phonetic information into the sequence model with a CTC loss (Graves et al., 2006) .

2.1. PHONETIC ATTRIBUTES:

Unlike attributes in the computer vision field, attributes of phones are independent of the corpus and dataset, they are well investigated and defined in the domain of articulatory phonetics (Ladefoged & Johnson, 2014) . Articulatory phonetics describes the mechanism of speech production such as the manner of articulation and place of articulation, and it tends to describe phones using discrete features such as voiced, bilabial (made with the two lips) and fricative. These articulatory features have been shown to be useful in speech recognition (Kirchhoff, 1998; Stüker et al., 2003b; Müller et al., 2017) , and are a good choice for phonetic attributes for our purpose. We provide some categories of phonetic attributes below. The full list of phonetic attributes we used is attached in the Appendix.

Consonants. Consonants are formed by obstructing the airstream through the vocal tract. They can be categorized in terms of the place and the manner of this obstruction. The places can be largely divided into three classes: labial, coronal, dorsal. Each of the class have more fine-grained classes.

The manners of articulation can be grouped into: stop, fricative, approximant etc.

Vowel. In the production of vowels, the airstream is relatively unobstructed. Each vowel sound can be specified by the positions of lips and tongue (Ladefoged & Johnson, 2014) . For instance, the tongue is at its highest point in the front of the mouth for front vowels. Additionally, vowels can be characterized by properties such as whether the lips are rounding or not (rounded, unrounded).

Diacritics. Diacritics are small marks to modify vowels and consonants by attaching to them. For instance, nasalization marks a sound for which the velopharyngeal port is open and air can pass through the nose. To make the phonetic attribute set manageable, we assign attributes of diacritics to some existing consonants attributes if they share similar phonetic property. For example, nasalization is treated as the nasal attribute in consonants.

In addition to phonetic attributes mentioned above, we note that we also need to allocate an special attribute for blank in order to predict blank labels in CTC model, and backpropagate their gradients into the acoustic model. Thus, our phonetic attribute set A phone is defined as the union of these three domain attributes as well as the blank label. Next, we need to assign each phoneme with appropriate attributes. There are multiple approaches to retrieve phonetic attributes. The most simple one is to use tools to collect articulatory features for each phoneme (Mortensen et al., 2016; Moran et al., 2014) . However, those tools only provide coarse-grained phonological features and we expect more fine-grained and customized phonetic features. In this section, we propose a naive but useful approach for attribute assignment. We note that we use X-SAMPA format to denote each IPA (Decker et al., 1999) in this work. X-SAMPA was devised to produce a computer-readable representation for IPA (Wells, 1995) . Each IPA segment can Then it is transformed into a phoneme distribution by a language dependent signature matrix S be mapped to X-SAMPA with appropriate rule-based tools (Mortensen et al., 2018) . For example, IPA /@/ can be represented as @ in X-SAMPA.

EQUATION

The assignment can be formulated as a problem to construct an assignment function f : P xsampa → 2 A phone where P xsampa is the set of all valid X-SAMPA phonemes. The assignment function should map each phone into its corresponding subset of A phone . To construct the function in the entire domain, we first manually mapped a small subset P base ⊂ P xsampa and constructed a restricted assignment function f P base . The mapping is customizable and has been verified with the IPA handbook (Decker et al., 1999) . Then for every p ∈ P xsampa , we continue to remove diacritics suffix from it until it could be find in P base . For example, to recognize ts >, we can first match the suffix, >as an ejective, and then recognize ts as a consonant defined in P base . The Algorithm 1 summarizes our approach, and the mapping of f P base is provided in Appendix.

2.2. SEQUENCE MODEL FOR ZERO-SHOT LEARNING:

Zero-shot learning has rarely been applied to sequence prediction problems so far, in this section we describe a novel sequence model architecture for zero-shot learning. We adapt a modified ESZSL architecture from Romera-Paredes & Torr (2015) . While the original architecture is devised to solve the classification problem with CNN(DECAF) features (Donahue et al., 2014) , our model aims to optimize a CTC loss over a sequence model as shown in Figure 2 . We note our architecture is a general model, and it can also be used for other sequence prediction problems in zero-shot learning.

Given the training set S = {(x n , y n ), n = 1...N } where each input x n ∈ X is an utterance, and y n ∈ Y is the corresponding phonetic transcription. Suppose that x n = (x 1 n , ..., x T n ) is the input sequence where x t n is the frame of time step t, and T is the length of x n . Each frame x t n is first projected into a feature vector h t n ∈ R d in the acoustic space H with a Bidirectional LSTM model.

EQUATION

) where W LSTM is the parameter of Bidirectional LSTM model. We assume that our phoneme inventory consists of z phonemes in the training set, each of them having a signature of a attributes constructed as mentioned above. Then we can represent our attributes in a constant signature matrix S ∈ [0, 1] z×a . Then, we transform h t n into points in the phonetic space P with V ∈ R a×d , and further into the phoneme logits l t n with S.

l t n = SV h t n (3)

The logits l n = (l 1 n , ..., l T n ) is then combined with y n to compute the CTC loss (Graves et al., 2006) . Additionally, regularizing V has been proved to be useful in the original ESZSL architecture (Romera-Paredes & Torr, 2015) . So our target is to minimize following loss function. minimize

EQUATION

where Ω(V ) is an simple L2 regularization. This objective can be easily optimized using standard gradient descent methods.

At inference stage, we usually consider a new language L test with a new phoneme inventory. Suppose that the new inventory is composed of z classes, then we can create a new signature matrix S ∈ [0, 1] z ×a , and estimate probability distribution of each phoneme P acoustic (p|x t n ) from logits using S instead of S. Additionally, we expect the phoneme should distribute similarly to the phoneme distribution in L test . We adjusted the posterior as follows.

P upm (p|x t n ) = P acoustic (p|x t n )P (p|L test ) α (5) where α is a hyperparameter, and P (p|L test ) is the prior distribution of L test which can be easily estimated from a small set of the text corpus in L test (we used 1000 sentences here). Finally we exploit greedy decoding to extract phoneme from the adjusted posterior P upm (Graves et al., 2006) .

3. EXPERIMENTS 3.1 DATASET:

We prepared two datasets for this experiment. The training set consists of three English corpora, and the test set is composed of corpora from 20 languages. They are used by both our model and the baseline model described later. Details regarding each corpus and phonetic information about each language are provided at Table 1 . Our model is trained with three English corpora because we tried to make the acoustic model robust to various channels and speech styles: TED (Rousseau et al., 2012) is the conference style, Switchboard (Godfrey et al., 1992) is the spontaneous conversation style and Librispeech is reading style (Panayotov et al., 2015) . We note that 5 percent of the entire corpus was used as the validation set. The evaluation set were selected from a variety of languages: not only from well-resourced languages, but also low-resourced languages from Asia and Africa.

We list phoneme details for each language on the right sides in Table 1 . All the phonemes were obtained using Epitran (Mortensen et al., 2018) . The phonemes column shows the number of distinct phonemes in each language, the shared column is the number of shared phonemes with English, and the unseen column indicates the number of phonemes unseen in English, which means those unseen phonemes are not available in the training set.

3.2. EXPERIMENTAL SETTINGS:

We used the EESEN framework for the acoustic modeling (Miao et al., 2015) . Each corpus is first re-sampled to 8000Hz, and we extracted 40 dimension Mel-frequency cepstral coefficients (MFCCs) (Davis & Mermelstein, 1990) from each utterance, the length of each frame is 25ms, and the shift between two continuous frames is 10ms. All the transcripts were transcribed into phonemes with Epitran (Mortensen et al., 2018) . We used a 5 layer Bidirectional LSTM model, each layer having 320 cells. The signature matrix is designed as we discussed above, and we used different signature matrices for different languages. We train the acoustic model with stochastic gradient descent, using a learning rate of 0.005. The prior distribution P (p|L test ) over phonemes is estimated by taking 1000 randomly selected sentences in the test language. Those 1000 sentences were removed from the test corpus. We fixed prior hyperparameter α = 1 for all evaluations.

A standard English acoustic model is used as our baseline model (Miao et al., 2015) . Phoneme distribution P (p|x t n ) is estimated with only Bidirectional LSTM θ(x t n ; W LSTM ). Then we decode phonemes with greedy decoding as we used in our approach. We use the same configuration of LSTM architecture as well as the training criterion. We also tried applying prior of target language to our baseline model as well. However, it was not helpful to improve the results because the phoneme inventory of English and target language are different, thus unique phonemes in target language cannot benefit from the prior distribution. As we focus on acoustic modeling in this work, we use phone error rate as the metric for evaluation. when evaluated with Amharic, and our approach obtained 68.7% in the same test set. For each language in our evaluation, we observed that we improved the phone error rate from 0.8% (French) to 21.7% (Italian) respectively. On average, the baseline model has 82.9% and our model get 9.9 % better phone error rate. The table also indicates one interesting relationship across different languages.

3.3. RESULTS:



Our results are summarized in:

The language which has close linguistic relationship with English tends to obtain better phone error rate. Dutch and German are classified in the West Germanic branch in the Indo-European language family like English. For Dutch, we obtained a 64.5% phone error rate and for German, a 61.6% phone error rate, which are the fourth and third best phone error rates among 20 languages. Additionally Italian and Russian belong to other branches of the Indo-European language family, but they also have very good phone error rate: 50.8% and 59.6% which are the first and second best phone error rates.

On the other hand, languages which are not a member of the Indo-European family tend to have worse phone error rate. The languages whose UPM phone error rate is worse than 80% are Lao, French, Zulu, Vietnamese. All languages except French are members of a different language family. Lao is in the KraDai language family, Zulu belongs to NigerCongo language family, and Vietnamese is a member of Austroasiatic language family. While French is a Indo-European language, its performance is much worse than other members of the language family. The reason is that French is a language known for its ambiguous orthographies (Ziegler et al., 1996) , which makes it harder to make rule based grapheme-to-phoneme mappings. As our phonetic tool is designed as a rule-based grapheme-to-phoneme mappings, the prediction of French phonemes is difficult.

To further investigate the reason of improvements for our model, we computed the (phone) substitution error rate, shown in the two right columns of To highlight the ability of our model to predict unseen phonemes, we show an example of Bengali in Table 3 . According to Table 1 , Bengali has the largest number of unseen phonemes among our test languages. Table 3 marks the unseen phonemes in English using bold font. The row of gold labels shows that target labels contain 7 unseen phonemes out of 22 phonemes. The baseline model was able to recognize part of seen phonemes such as the first phoneme /b/, however it fails to recognize all unseen phonemes due to its limitation. On the contrary, our model could correctly recognize two unseen phonemes /l d/ which aligned with two /l d/ phonemes in the gold labels. Random examples from other languages are available in Appendix.

4. RELATED WORK:

We briefly outline several areas of related works, and describe their connections and differences with this paper. Firstly zero-shot learning has been applied to recognize unseen objects during training in the computer vision field. One line of works has focused on two-stage approach (Lampert et al., 2009; Palatucci et al., 2009; Lampert et al., 2014; Jayaraman & Grauman, 2014; Al-Halah et al., 2016; Liu et al., 2011) . Another line of approaches adapts multimodal models for zero-shot learning (Socher et al., 2013; Frome et al., 2013; Huang et al., 2012; Lei Ba et al., 2015; Rohrbach et al., 2011) . Socher et al. (2013) . However those works rarely mention speech recognition.

There has been growing interests in zero-resource speech processing (Glass, 2012; Jansen et al., 2013) , most of the work focusing on tasks like acoustic unit discovery (Heck et al., 2017) , unsupervised segmentation (Kamper et al., 2017) , representation learning (Lake et al., 2014; Hermann & Goldwater, 2018) , spoken term discovery (Dunbar et al., 2017; Jansen et al., 2013) . These models are useful for various extrinsic speech processing tasks like topic identification (Kesiraju et al., 2017; Liu et al., 2017) . However, those unsupervised concepts cannot be grounded to actual words or phonemes, hence making it impracticable to do speech recognition or acoustic modeling. The usual intrinsic evaluations that these zero resource tasks are tested on is ABX discriminability task (Dunbar et al., 2017; Heck et al., 2017) or the unsupervised word error rate (Ludusan et al., 2014; Kamper et al., 2017) which are good for quality estimates but not practical as they use an oracle or ground truth labels to assign cluster labels. Secondly these approaches demands a modest size of audio corpus of targeting language (e.g: 2.5h to 40h). In contrast, our approach assumes no audio corpus but a small set of text corpus to estimate phone prior. This is a reasonable assumption as text corpus is usually easier to obtain than audio corpus. The idea of decomposing speech into concepts was also discussed by Lake et al. (2014) , where the authors propose a generative model to learn representations for spoken words which they then use to classify words with only one training sample available per word. Though this is also in the same line as the zero-resource speech processing papers, we feel the motivation behind the decomposition is very similar to this work.

The authors in Scharenborg et al. (2017) focus on a pipeline towards adaptation to a low resource language with little training data, they present an interesting method to map Dutch/English phonemes in the same space using an extrapolation approach to predict phones in English that are unseen in Dutch. Our work proposes a generic algorithm to recognize any unknown phones by decomposing them into its phone attributes. We have also shown that our approach is effective over 20 languages from different language families.

Another group of researchers explore adaptation techniques for multilingual speech recognition, especially for low resource languages. In these multilingual settings, the hidden layers are either HMM or DNN models which are shared by multiple languages, and the output layer is either language specific phone set or a universal IPA-based phone set (Tong et al., 2017; Vu & Schultz, 2013; Tüske et al., 2013; Thomas et al., 2010; Vu et al., 2014; Lin et al., 2009; Chen & Mak, 2015; Dalmia et al., 2018b) . However predictable phonemes are restricted to the phonemes in the training set, thus they fail to predict unseen phonemes in the test set. In contrast, our model can predict unseen phonemes by taking advantage of their phonetic attributes.

Articulatory features have been shown to be useful in speech recognition under several situation. For example, articulatory features has been used to improve robustness under noisy and reverberant environment (Kirchhoff, 1998) , compensate for crosslingual variability (Stüker et al., 2003b) , improve word error rate in multilingual models (Stüker et al., 2003a) , be beneficial for low resource languages (Müller et al., 2016) , clustering phoneme-like units for unwritten languages (Müller et al., 2017) . Those approaches generally treat articulatory features as additional feature for classifications, and did not provide a model to predict unseen phones.

5. CONCLUSION AND FUTURE WORK:

In this work, we propose the Universal Phonetic Model to apply zero-shot learning to acoustic models in speech recognition. Our experiment shows that it outperforms the baseline by 9.9 % phone error rate on average for 20 languages. While the performance of our approach is still not practical in the actual applications, it paves the way to tackle zero-shot learning of speech recognition with a new framework. We note that several approaches can be investigated on top of this framework.

For instance, multilingual acoustic features can be used instead of standard ones (Hermann & Goldwater, 2018; Dalmia et al., 2018a) . Label smoothing can be applied to regularize distributions of phonemes or attributes (Pereyra et al., 2017) . The training objective can be replaced with discriminative training objectives such as MMI and sMBR (Veselỳ et al., 2013; Povey) . Additionally, various encoder and decoder models can be explored with this framework (Chan et al., 2016; Zhou et al., 2018) . 

:

Our code would be released upon acceptance

Figures and Tables:

Figure 2: Illustration of the sequence model for zero-shot learning. The input layer is first processed with a Bidirectional LSTM acoustic model, and produces a distribution over phonetic attributes.Then it is transformed into a phoneme distribution by a language dependent signature matrix S

s' e b t k a r i r a h\ a m l d a ts\ a l d a e baseline Prediction b I k O r\e j t h { m l E k E l i UPM Prediction b i k o e a n d l d a k a l d eTable 3: A comparison of predictions in Bengali test set. The phonemes in bold font are unseen ones in English set, indicating our model's capability at predicting unseen phonemes.

As it shows, our approach has consistently outperformed the baseline in terms of phone error rate. For example, the baseline achieves 77.3% phone error rate

Table content: <table><tr><td>Language</td><td>Corpus Name</td><td colspan="3"># Phonemes # Shared #Unseen</td></tr><tr><td>English</td><td>TED (Rousseau et al., 2012)</td><td>38</td><td>38</td><td>0</td></tr><tr><td>English</td><td>Switchboard (Godfrey et al., 1992)</td><td>38</td><td>38</td><td>0</td></tr><tr><td>English</td><td>Librispeech (Panayotov et al., 2015)</td><td>38</td><td>38</td><td>0</td></tr><tr><td>Amharic</td><td>ALFFA Amharic (Tachbelie et al., 2014)</td><td>57</td><td>26</td><td>31</td></tr><tr><td>Bengali</td><td>OpenSLR37 (Gutkin et al.)</td><td>70</td><td>38</td><td>32</td></tr><tr><td>Cebuano</td><td>IARPA-babel301b-v2.0b</td><td>39</td><td>38</td><td>1</td></tr><tr><td>Dutch</td><td>Voxforge</td><td>49</td><td>34</td><td>15</td></tr><tr><td>French</td><td>Voxforge</td><td>42</td><td>33</td><td>9</td></tr><tr><td>German</td><td>Voxforge</td><td>37</td><td>25</td><td>12</td></tr><tr><td>Haitian</td><td>IARPA-babel201b-v0.2b</td><td>40</td><td>37</td><td>3</td></tr><tr><td>Italian</td><td>Voxforge</td><td>43</td><td>26</td><td>17</td></tr><tr><td>Javanese</td><td>OpenSLR35</td><td>45</td><td>36</td><td>9</td></tr><tr><td>Kazakh</td><td>IARPA-babel302b-v1.0a</td><td>45</td><td>36</td><td>9</td></tr><tr><td>Kurmanji</td><td>IARPA-babel205b-v1.0a</td><td>27</td><td>23</td><td>4</td></tr><tr><td>Lao</td><td>IARPA-babel203b-v3.1a</td><td>39</td><td>21</td><td>18</td></tr><tr><td>Mongolian</td><td>IARPA-babel401b-v2.0b</td><td>50</td><td>18</td><td>32</td></tr><tr><td>Russian</td><td>Voxforge</td><td>48</td><td>22</td><td>26</td></tr><tr><td>Sinhala</td><td>openSLR52</td><td>50</td><td>38</td><td>12</td></tr><tr><td>Tagalog</td><td>IARPA-babel106b-v0.2g</td><td>25</td><td>24</td><td>1</td></tr><tr><td>Turkish</td><td>IARPA-babel105b-v0.4</td><td>40</td><td>38</td><td>2</td></tr><tr><td>Swahili</td><td>ALFFA Swahili (Gelas et al., 2012)</td><td>47</td><td>38</td><td>9</td></tr><tr><td>Vietnamese</td><td>IARPA-babel107b-v0.7</td><td>47</td><td>37</td><td>10</td></tr><tr><td>Zulu</td><td>IARPA-babel206b-v0.1e</td><td>40</td><td>25</td><td>15</td></tr></table>

Corpus of training set and test set used in the experiment. The acoustic model is trained with three English corpus, and tested on the 20 other languages below.

Table content: <table/>



Table content: <table><tr><td colspan="5">Language Baseline PER% UPM PER% Baseline Sub% UPM Sub%</td></tr><tr><td>Amharic</td><td>77.3</td><td>68.7</td><td>53.1</td><td>37.6</td></tr><tr><td>Bengali</td><td>82.6</td><td>67.3</td><td>62.9</td><td>38.2</td></tr><tr><td>Cebuano</td><td>88.7</td><td>72.1</td><td>48.9</td><td>27.2</td></tr><tr><td>Dutch</td><td>70.8</td><td>64.5</td><td>53.7</td><td>43.7</td></tr><tr><td>French</td><td>84.1</td><td>83.3</td><td>57.6</td><td>57.4</td></tr><tr><td>German</td><td>71.0</td><td>61.6</td><td>48.7</td><td>31.0</td></tr><tr><td>Haitian</td><td>84.5</td><td>74.4</td><td>54.5</td><td>41.5</td></tr><tr><td>Italian</td><td>72.5</td><td>50.8</td><td>56.4</td><td>27.5</td></tr><tr><td>Javanese</td><td>86.4</td><td>79.4</td><td>45.8</td><td>37.9</td></tr><tr><td>Kazakh</td><td>89.0</td><td>79.3</td><td>50.9</td><td>39.8</td></tr><tr><td>Kurmanji</td><td>91.3</td><td>79.1</td><td>46.6</td><td>30.5</td></tr><tr><td>Lao</td><td>87.9</td><td>85.6</td><td>55.2</td><td>54.8</td></tr><tr><td>Mongolian</td><td>87.6</td><td>79.8</td><td>43.7</td><td>28.6</td></tr><tr><td>Russian</td><td>73.4</td><td>59.6</td><td>55.2</td><td>33.9</td></tr><tr><td>Sinhala</td><td>76.3</td><td>71.2</td><td>55.3</td><td>45.5</td></tr><tr><td>Tagalog</td><td>86.2</td><td>70.9</td><td>46.6</td><td>25.7</td></tr><tr><td>Turkish</td><td>84.4</td><td>75.1</td><td>49.5</td><td>43.2</td></tr><tr><td>Swahili</td><td>83.8</td><td>70.8</td><td>45.3</td><td>27.0</td></tr><tr><td>Vietnamese</td><td>85.2</td><td>82.1</td><td>54.3</td><td>51.6</td></tr><tr><td>Zulu</td><td>95.8</td><td>85.1</td><td>56.5</td><td>41.9</td></tr><tr><td>Average</td><td>82.9</td><td>73.0</td><td>52.0</td><td>38.2</td></tr></table>



Table content: <table/>

phonemes and attributes

Table content: <table><tr><td>A</td></tr><tr><td>PHONETIC ATTRIBUTE FULL LIST</td></tr><tr><td>vowel consonant alveolar affricate alveolo-palatal approximant aspirated back bilabial breathy-voice</td></tr><tr><td>central click close close-mid dental ejective epiglottal flap fricative front glottal implosive labial-</td></tr><tr><td>palatal labial-velar labial labiodental lateral long nasal near-close near-open non-syllabic open open-</td></tr><tr><td>mid palatal palatal-velar pharyngeal plosive postalveolar retroflex rounded schwa stop syllabic trill</td></tr><tr><td>unrounded uvular velar voiced voiceless vowel coronal dorsal blank</td></tr></table>

Peer review (iclr2019):
 Overview: This paper proposed an approach for zero-shot phoneme recognition, where it is possible to recognise phonemes in a target language which has never been seen before. Rather than just training a phoneme recogniser directly on background data and then applying it to unseen data, phonetic features are first predicted, allowing phonemes not in the source language set to be predicted. Main strengths: The paper*s main strength lies in that this is a very unexplored area that could assist in the development of speech technology where it is currently not possible. The proposed model (Section 2) has also not been considered in prior work. Main weaknesses: The paper*s main weakness is in some of its claims and that it misses some very relevant literature. Detailed comments together with a minimal list of references are given below (but I would encourage the authors to also read a bit more broadly). But in short I do not think it is that easy to claim that this is the first paper to do zero-shot learning on speech; many of the zero-resource studies where unlabelled audio is used could be seen as doing some for of zero-shot matching. Specifically [5] is able to predict unseen phoneme targets. Multilingual bottleneck features can be applied to languages that have never been seen before [2], and the output of phoneme recognisers trained on one language have long been applied to get output on another unseen language. The first one-shot learning speech paper [4] (to my knowledge) is also not mentioned at all. The approach in the paper also still relies on some text data from the target language; if this then can be described as "zero-shot" learning, then I think many of these previous studies c also make this claim. Overall feedback: There is definitely value in this work, but it should be much better situated within the broader literature. Below I give some editorial suggestions and also outline some suggestions for further experiments. Detailed comments, suggestions and questions: - Abstract: It would be useful to have some details of the "baseline model" here already, especially since it is such a new task. - Introduction: "... but they can hardly predict phones or words directly due to their unsupervised nature." This is a strong statement that maybe requires more justification. On the one hand, the statement is true, and the high word error rates in e.g. [3] can be cited. On the other hand, it has been shown that at the phone-distinction level, these models perform quite well and sometimes outperform supervised models [1]. Since this paper also considers phone error rate as a metric, I think care should be taken with such statements. - Introduction: "While zero-shot learning has attracted a lot of attention in *the* computer vision community, this setup has hardly been studied in speech recognition research especially in acoustic modeling." Definitely look at some of the studies mentioned below, and also [4] specifically. - "However, we note that our model can be combined with a well-resourced language model to recognize words." How would this be done, since I think this is actually quite a challenging task. - Section 2: "... useful the original ESZSL architecture ..." -> "... useful in the original ESZSL architecture ..." - Section 2.2: I assume the small text corpus is at the phone level (and not characters directly)? This should be clarified, and it could raise the question of whether this approach is truly "zero-shot". - Section 3.2: "We used EESEN framework ..." -> "We used the EESEN framework ..." - Section 4: You could look at the recent work in [2], which uses multilingual bottleneck features trained on 10 languages and applied to multiple unseen languages. It would be interesting to also train your approach on multiple languages instead of only English. Missing references: 1. M. Heck, S. Sakti, and S. Nakamura, "Feature Optimized DPGMM Clustering for Unsupervised Subword Modeling: A Contribution to Zerospeech 2017," in Proc. ASRU, 2017. 2. E. Hermann and S. J. Goldwater, "Multilingual bottleneck features for subword modeling in zero-resource languages," in Proc. Interspeech, 2018. 3. H. Kamper, K. Livescu, and S. Goldwater, An embedded segmental k-means model for unsupervised segmentation and clustering of speech," in Proc. ASRU, 2017. 4. B. M. Lake, C.-Y. Lee, J. R. Glass, and J. B. Tenenbaum, "One-shot learning of generative speech concepts," in Proc. CogSci, 2014. 5. O. Scharenborg, F. Ciannella, S. Palaskar, A. Black, F. Metze, L. Ondel, and M. Hasegawa-Johnson, "Building an ASR system for a low-resource language through the adaptation of a high-resource language asr system: Preliminary results,"in Proc. ICNLSSP, 2017. Edit: Based on the rebuttal I*ve changed my rating from 4 to 5.