Abstract: Audio Word2Vec offers vector representations of fixed dimensionality for variable-length audio segments using Sequence-to-sequence Autoencoder (SA). These vector representations are shown to describe the sequential phonetic structures of the audio segments to a good degree, with real world applications such as query-by-example Spoken Term Detection (STD). This paper examines the capability of language transfer of Audio Word2Vec. We train SA from one language (source language) and use it to extract the vector representation of the audio segments of another language (target language). We found that SA can still catch phonetic structure from the audio segments of the target language if the source and target languages are similar. In query-by-example STD, we obtain the vector representations from the SA learned from a large amount of source language data, and found them surpass the representations from naive encoder and SA directly learned from a small amount of target language data. The result shows that it is possible to learn Audio Word2Vec model from high-resource languages and use it on low-resource languages. This further expands the usability of Audio Word2Vec.

Introduction: Embedding audio word segments into fixed-length vectors has many useful applications in natural language processing such as speaker identification BIBREF0 , audio emotion classification BIBREF1 , and spoken term detection (STD) BIBREF2 , BIBREF3 , BIBREF4 . In these applications, audio segments are usually represented as feature vectors to be applied to a standard classifiers which determines the speaker's identification, emotion or whether the input queries are included. By representing the audio segments in fixed-length vectors instead of using the original segments in variable lengths, we can reduce the effort for indexing, accelerate the speed of calculation, and improve the efficiency for the retrieval task BIBREF5 , BIBREF6 , BIBREF7 . Recently, deep learning has been used for encoding acoustic information into vectors BIBREF8 , BIBREF9 , BIBREF10 . Existing works have shown that it is possible to transform audio word segments into fixed dimensional vectors. The transformation successfully produces vector space where word audio segments with similar phonetic structures are closely located. In BIBREF10 , the authors train a Siamese convolutional neural network with side information to obtain embeddings that separate same-word pairs and different-word pairs. Human annotated data is required under this supervised learning scenario. Besides supervised approaches BIBREF11 , BIBREF10 , BIBREF12 , BIBREF13 , unsupervised approaches are also proposed to reduce the annotation effort BIBREF14 . As for the unsupervised learning for the audio embedding, LSTM-based sequence-to-sequence autoencoder demonstrates a promising result BIBREF14 . The model is trained to minimize the reconstruction error of the input audio sequence and then provides the embedding, namely Audio Word2Vec, from its bottleneck layer. This is done without any annotation effort. Although deep learning approaches have produced satisfactory result, the data-hungry nature of the deep model makes it hard to produce the same performance with low-resource data. Both supervised and unsupervised approaches assume that a large amount of audio data of the target language is available. A question arises whether it is possible to transfer the Audio Word2Vec model learned from a high-resource language into a model targeted at a low-resource language. While this problem is not yet to be fully examined in Audio Word2Vec, works in neural machine translation (NMT) successfully transfer the model learned on high-resource languages to low-resource languages. In BIBREF15 , BIBREF16 , the authors first train a source model with high-resource language pair. The source model is used to initialize the target model which is then trained by low-resource language pairs. For audio, all languages are uttered by human beings with a similar vocal tract structure, and therefore share some common acoustic patterns. This fact implies that knowledge obtained from one spoken language can be transferred onto other languages. This paper verifies that sequence-to-sequence autoencoder is not only able to transform audio word segments into fixed-length vectors, the model is also transferable to the languages it has never heard before. We also demonstrate its promising applications with a query-by-example spoken term detection (STD) experiment. In the query-by-example STD experiment, even without tunning with partial low-resource language segments, the autoencoder can still produce high-quality vectors.

Question: How do they compare representations performance obtained from a naive encoder versus ones learned from large amount of source language data?
Answer: They compare MAP performance of query-by-example STD using representations obtained from naive encoder and their method