Abstract: Embedding acoustic information into fixed length representations is of interest for a whole range of applications in speech and audio technology. We propose two novel unsupervised approaches to generate acoustic embeddings by modelling of acoustic context. The first approach is a contextual joint factor synthesis encoder, where the encoder in an encoder/decoder framework is trained to extract joint factors from surrounding audio frames to best generate the target output. The second approach is a contextual joint factor analysis encoder, where the encoder is trained to analyse joint factors from the source signal that correlates best with the neighbouring audio. To evaluate the effectiveness of our approaches compared to prior work, we chose two tasks - phone classification and speaker recognition - and test on different TIMIT data sets. Experimental results show that one of our proposed approaches outperforms phone classification baselines, yielding a classification accuracy of 74.1%. When using additional out-of-domain data for training, an additional 2-3% improvements can be obtained, for both for phone classification and speaker recognition tasks.

Introduction: In recent years, word embeddings have been successfully used in natural language processing (NLP), the most commonly known models are Word2Vec BIBREF0 and GloveBIBREF1. The reasons for such success are manifold. One key attribute of embedding methods is that word embedding models take into account context information of words, thereby allowing a more compact and manageable representation for wordsBIBREF2, BIBREF3. The embeddings are widely applied in many downstream NLP tasks such as neural machine translation, dialogue system or text summarisation BIBREF4, BIBREF5, BIBREF6, as well as in language modelling for speech recognitionBIBREF7. Embeddings of acoustic (and speech) signals are of more recent interest. The objective is to represent audio sequence information in compact form, replacing the raw audio data with one that contains latent factors BIBREF8, BIBREF9. The projection into such (latent) spaces should take into account different attributes, such as phonemes, speaker properties, speaking styles, the acoustic background or the recording environment. Acoustic embeddings have been explored for a variety of speech tasks such as speech recognitionBIBREF10, speaker verificationBIBREF11 or voice conversionBIBREF12. However, learning acoustic embeddings is challenging: attributes mentioned above, e.g. speaker properties and phonemes, operate at different levels of abstraction and are often strongly interdependent, and therefore are difficult to extract and represent in a meaningful formBIBREF8. For speech processing BIBREF13, BIBREF14, BIBREF15 also use context information to derive acoustic embeddings. However, BIBREF13, BIBREF14 focus on learning word semantic representations from raw audio instead of signal properties such as phonemes and speaker properties. BIBREF15 focus on learning speaker representations by modelling of context information with a Siamese networks that discriminate whether a speech segment is the neighbourhood of a target segment or not. In this paper, two unsupervised approaches to generate acoustic embeddings using context modelling are proposed. Both methods make use of the variational auto-encoder framework as proposed inBIBREF16 and both approaches aim to find joint latent variables between the target acoustic segments and its surrounding frames. In the first instance a representation is derived from surrounding audio frames that allows to predict current frame, thereby generating target audio from common factors. The encoder element of the associated auto-encoder is further referred to as Contextual Joint Factor Synthesis (CJFS) encoder. In the second instance an audio frame is used to predict surrounding audio, which is further referred to as Contextual Joint Factor Analysis (CJFA) encoding. As shown in previous work variational auto-encoders can be used to derive latent variables such as speaker information and phonemesBIBREF8 more robustly. In this work it is shown that including temporal information can further improve performance and robustness, for both phoneme classification and speaker identification tasks. Furthermore the use of additional unlabelled out-of-domain data can improved modelling for the proposed approaches. As outlined above, prior work has made use of surrounding audio in different forms. To the best of our knowledge this work is the first to show that predicting surrounding audio allows for efficient extraction of latent factors in speech signals. The rest of paper is organised as follows: In §SECREF2 related work is described, Methods for deriving acoustic embeddings, and context modelling methods in NLP, computer vision and speech are discussed. This is followed by the description of the two approaches for modelling context as used in this work, in §SECREF3. The experimental framework is described in §SECREF4, including the data organisation, baseline design and task definitions; in §SECREF5 and §SECREF6 experiments results are shown and discussed. This is followed by the conclusions and future work in §SECREF7. 

Question: What TIMIT datasets are used for testing?
Answer: Once split into 8 subsets (A-H), the test set used are blocks D+H and blocks F+H