Contextual Joint Factor Acoustic Embeddings

Abstract: Embedding acoustic information into fixed length representations is of interest for a whole range of applications in speech and audio technology. We propose two novel unsupervised approaches to generate acoustic embeddings by modelling of acoustic context. The first approach is a contextual joint factor synthesis encoder, where the encoder in an encoder/decoder framework is trained to extract joint factors from surrounding audio frames to best generate the target output. The second approach is a contextual joint factor analysis encoder, where the encoder is trained to analyse joint factors from the source signal that correlates best with the neighbouring audio. To evaluate the effectiveness of our approaches compared to prior work, we chose two tasks - phone classification and speaker recognition - and test on different TIMIT data sets. Experimental results show that one of our proposed approaches outperforms phone classification baselines, yielding a classification accuracy of 74.1%. When using additional out-of-domain data for training, an additional 2-3% improvements can be obtained, for both for phone classification and speaker recognition tasks.

In recent years, word embeddings have been successfully used in natural language processing (NLP), the most commonly known models are Word2Vec BIBREF0 and GloveBIBREF1. The reasons for such success are manifold. One key attribute of embedding methods is that word embedding models take into account context information of words, thereby allowing a more compact and manageable representation for wordsBIBREF2, BIBREF3. The embeddings are widely applied in many downstream NLP tasks such as neural machine translation, dialogue system or text summarisation BIBREF4, BIBREF5, BIBREF6, as well as in language modelling for speech recognitionBIBREF7.
Embeddings of acoustic (and speech) signals are of more recent interest. The objective is to represent audio sequence information in compact form, replacing the raw audio data with one that contains latent factors BIBREF8, BIBREF9. The projection into such (latent) spaces should take into account different attributes, such as phonemes, speaker properties, speaking styles, the acoustic background or the recording environment. Acoustic embeddings have been explored for a variety of speech tasks such as speech recognitionBIBREF10, speaker verificationBIBREF11 or voice conversionBIBREF12. However, learning acoustic embeddings is challenging: attributes mentioned above, e.g. speaker properties and phonemes, operate at different levels of abstraction and are often strongly interdependent, and therefore are difficult to extract and represent in a meaningful formBIBREF8.
For speech processing BIBREF13, BIBREF14, BIBREF15 also use context information to derive acoustic embeddings. However, BIBREF13, BIBREF14 focus on learning word semantic representations from raw audio instead of signal properties such as phonemes and speaker properties. BIBREF15 focus on learning speaker representations by modelling of context information with a Siamese networks that discriminate whether a speech segment is the neighbourhood of a target segment or not.
In this paper, two unsupervised approaches to generate acoustic embeddings using context modelling are proposed. Both methods make use of the variational auto-encoder framework as proposed inBIBREF16 and both approaches aim to find joint latent variables between the target acoustic segments and its surrounding frames. In the first instance a representation is derived from surrounding audio frames that allows to predict current frame, thereby generating target audio from common factors. The encoder element of the associated auto-encoder is further referred to as Contextual Joint Factor Synthesis (CJFS) encoder. In the second instance an audio frame is used to predict surrounding audio, which is further referred to as Contextual Joint Factor Analysis (CJFA) encoding. As shown in previous work variational auto-encoders can be used to derive latent variables such as speaker information and phonemesBIBREF8 more robustly. In this work it is shown that including temporal information can further improve performance and robustness, for both phoneme classification and speaker identification tasks. Furthermore the use of additional unlabelled out-of-domain data can improved modelling for the proposed approaches. As outlined above, prior work has made use of surrounding audio in different forms. To the best of our knowledge this work is the first to show that predicting surrounding audio allows for efficient extraction of latent factors in speech signals.
The rest of paper is organised as follows: In §SECREF2 related work is described, Methods for deriving acoustic embeddings, and context modelling methods in NLP, computer vision and speech are discussed. This is followed by the description of the two approaches for modelling context as used in this work, in §SECREF3. The experimental framework is described in §SECREF4, including the data organisation, baseline design and task definitions; in §SECREF5 and §SECREF6 experiments results are shown and discussed. This is followed by the conclusions and future work in §SECREF7.
Most interest in acoustic embeddings can be observed on acoustic word embeddings, i.e. projections that map word acoustics into a fixed size vector space. Objective functions are chosen to project different word realisations to close proximity in the embedding space. Different approaches were used in the literature - for both supervised and unsupervised learning. For the supervised case, BIBREF9 introduced a convolutional neural network (CNN) based acoustic word embedding system for speech recognition, where words that sound alike are nearby in Euclidean distance. In their work, a CNN is used to predict a word from the corresponding acoustic signal, the output of the bottleneck layer before the final softmax layer is taken to be the embedding for the corresponding word. Further work used different network architectures to obtain acoustic word embeddings: BIBREF10 introduces a recurrent neural network (RNN) based approach instead.
For the case that the word boundary information is available but the word label itself is unknown, BIBREF12 proposed word similarity Siamese CNNs. These are used to minimise a distance function between representations of two instances of the same word type whilst at the same time maximising the distance between two instances of different words.
Unsupervised approaches also exist. BIBREF17 proposed a convolutional variational auto-encoder based approach to derive acoustic embedding, in unsupervised fashion. The authors chose phoneme and speaker classification tasks on TIMIT data to assess the quality of their embeddings - an approach replicated in the work presented in this paper. BIBREF8, BIBREF18 proposed an approach called factorised hierarchical variational auto-encoder. The work introduces the concepts of global and local latent factors, i.e. latent variables that are shared on the complete utterance, or latent variables that change within the sequence, respectively. Results are again obtained using the same data and tasks as above.
Context information plays a fundamental role in speech processing. Phonemes could be influenced by surrounding frames through coarticulationBIBREF19 - an effect caused by speed limitations and transitions in the movement of articulators. Normally directly neighbouring phonemes have important impact on the sound realisation. Inversely, the surrounding phonemes also provide strong constraints on the phoneme that can be chosen at any given point, subject to to lexical and language constraints. This effect is for example exploited in phoneme recognition, by use of phoneme $n$-gram modelsBIBREF20. Equivalently inter word dependency - derived from linguistic constraints - can be exploited, as is the case in computing word embeddings with the aforementioned word2vecBIBREF0 method. The situation differs for the global latent variables, such as speaker properties or acoustic environment information. Speaker properties remains constant - and environments can also be assumed stationary over longer periods of time. Hence these variables are common between among neighbouring frames and windows. Modelling context information is helpful for identifying such information BIBREF21.
There is significant prior work that takes surrounding information into account to learn vector representations. For text processing the Word2VecBIBREF0 model directly predicts the neighbouring words from target words or inversely. This helps to capture the meanings of wordsBIBREF2. In computer vision, BIBREF22 introduced an visual feature learning approach called context encoder ,which is based on context based pixel prediction. Their model is trained to generate the contents of an image region from its surroundings. In speech processing BIBREF13, BIBREF14 proposed a sequence to sequence approach to predict surrounding segments of a target segment. However, the approach again aims at capturing word semantics from raw speech audio, words has similar semantic meanings are nearby in Euclidean distance. BIBREF15 proposed an unsupervised acoustic embedding approach. In their approach, instead of directly estimating the neighbourhood frames of a target segment, a Siamese architecture is used to discriminate whether a speech segment is in the neighbourhood of a target segment or not. Furthermore, their approach only aims at embedding of speaker properties. To the best of our knowledge, work presented here is the first derive phoneme and speaker representations by temporal context prediction using acoustic data.
As shown in BIBREF17, variational auto-encoders (VAE)BIBREF8 can yield good representations in the latent space. One of the benefits is that the models allow to work with the latent distributionsBIBREF23, BIBREF8, BIBREF24. In this work, VAE is used to model the joint latent factors between the target segments and its surroundings.
Different from normal auto-encoders, where the input data is compressed into latent code which is a point estimation of latent variablesBIBREF16, the variational auto-encoder model defines a probabilistic generative process between the observation $x$ and the latent variable $z$. At the encoder step, the encoder provides an estimation of the latent variable $z$ given observation $x$ as $p(z|x)$. The decoder finds the most likely reconstruction $\hat{x}$ subject to $p(\hat{x}|z)$. The latent variable estimation $p(z|x)$, or the probability density function thereof, has many interpretations, simply as encoding, or as latent state space governing the construction of the original signal.
Computing $p(z|x)$ requires an estimate of the marginal likelihood $p(x)$ which is difficult to obtain in practice. A recognition model $q(z|x)$ is used to approximate $p(z|x)$ KL divergence between $p(z|x)$ and $q(z|x)$, as shown in Eq DISPLAY_FORM4, is minimisedBIBREF16.
From Eq DISPLAY_FORM4, the objective function for VAE training is derived shown in Eq DISPLAY_FORM5: BIBREF16, BIBREF17
where $E_{q(z|x)}log[p(x|z)]$ is also called the reconstruction likelihood and $ D_{KL}(q(z|x)||p(z))$ ensures the learned distribution $q(z|x)$ is close to prior distribution $p(z)$.
An audio signal is represented sequence of feature vectors $S=\lbrace S_1,S_2,...S_T\rbrace $, where $T$ is the length of the utterance. In the proposed method the concept of a target window is used, to which and embedding is related. A target window $X_t$ is a segment of speech representing features from $S_t$ to $S_{t+C-1}$, where $t \in \lbrace 1,2,...T-C+1\rbrace $ and $C$ denotes the target window size. The left neighbour window of the target window is defined as the segment between $S_{t-N}$ and $S_{t-1}$, and the segment between $S_{t+C}$ and $S_{t+C+N-1}$ represents the right neighbour window of the target window, with $N$ being the single sided neighbour window size. The concatenation of left and right neighbour segments is further referred to $Y_t$. The proposed approach aims to find joint latent factors between target window segment $X_t$ and the concatenation of left and right neighbour window segments $Y_t$, for all segments. For convenience the subscript $t$ is dropped in following derivations where appropriate. Two different context use configurations can be used.
Figure FIGREF7 illustrate these two approaches. The audio signal is split into a sequence of left neighbour segment, target segment and right neighbour segment. In the first approach (left side on figure FIGREF7), the concatenation of the left neighbour segment and right neighbour segment ($Y$) is input to a VAE modelBIBREF16, and target window ($X$) is predicted. In the second approach (right side on figure FIGREF7) the target window ($X$) is the input to a VAE model, and neighbour window ($Y$) is predicted.
The first approach is referred to as the contextual joint factor synthesis encoder as it aims to synthesise the target frame $X$. Only factors common between input and output can form the basis for such prediction, and the encoded embedding can be considered a representation of these joint factors. Similar to the standard VAE formulations, the objective function of CJFS is given in Eq. DISPLAY_FORM8:
The first term represents the reconstruction likelihood between predicted target window segments and the neighbour window segments, and the second term denotes how similar the learned distribution $q(z|Y)$ is to the prior distribution of $z$, $p(z)$
In practice, for the reconstruction term can be based on the mean squared error (MSE) between the true target segment and the predicted target segment. For the second term in Eq DISPLAY_FORM8, samples for $p(z)$ are obtained from Gaussian distribution with zero mean and a variance of one ($p(z) \sim \mathcal {N} (0,1)$).
The second approach is the contextual joint factor analysis encoder. The objective is to predict the temporal context $Y$ based on input from a single centre segment $X$. Again joint factors between the three windows are obtained, and encoded in an embedding. However this time an analysis of one segment is enough. Naturally the training objective function of CJFA is represented by change of variables, as given in Eq DISPLAY_FORM9.
Taking the VAE experiments as baseline, the TIMIT data is used for this workBIBREF25. TIMIT contains studio recordings from a large number of speakers with detailed phoneme segment information. Work in this paper makes use of the official training and test sets, covering in total 630 speakers with 8 utterances each. There is no speaker overlap between training and test set, which comprise of 462 and 168 speakers, respectively. All work presented here use of 80 dimensional Mel-scale filter bank coefficients.
Work on VAE in BIBREF17 to learn acoustic embeddings conducted experiments using the TIMIT data set. In particular the tasks of phone classification and speaker recognition where chosen. As work here is an extension of such work we we follow the experimentation, however with significant extensions (see Section SECREF13). With guidance from the authors of the original workBIBREF17 our own implementation of VAE was created and compared with the published performance - yielding near identical results. This implementation then was also used as the basis for CJFS and CJFA, as introduced in § SECREF6.
For the assessment of embedded vector quality our work also follows the same task types, namely phone classification and speaker recognition (details in §SECREF13), with identical task implementations as in the reference paper. It is important to note that phone classification differs from the widely reported phone recognition experiments on TIMIT. Classification uses phone boundaries which are assumed to be known. However, no contextual information is available, which is typically used in the recognition setups, by means of triphone models, or bigram language models. Therefore the task is often more difficult than recognition. The baseline performance for VAE based phone classification experiments in BIBREF17 report an accuracy of 72.2%. The re-implementation forming the basis for our work gave an accuracy of 72.0%, a result that was considered to provide a credible basis for further work.
For the purpose of speaker recognition it is important to take into account the overlap between training and testing. Thus three different task configurations are considered, different to the setting in BIBREF17. Their baseline will be further referred as VAE baseline.
The phone classification implementation operates on segment level, using a convolutional network to obtain frame by frame posteriors which are then accumulated for segment decision (assuming frame independence). The phone class with the highest segment posterior is chosen as output. An identical approach is used for speaker recognition. In this setting 3 different data sets are required: a training set for learning the encoder models, a training set for learning the classification model, and an evaluation test set. For the phone classification task, both embedding and classification models are trained on the official TIMIT training set, and makes use of the provided phone boundary information. A fixed size window with a frame step size of one frame is used for all model training. As noted, phone classification makes no use of phone context, and no language model is applied.
For speaker recognition overlap speaker between any of the datasets (training embeddings, training classifier and test) will cause a bias. Three different configurations (Tasks a,b,c) are used to assess this bias. Task a reflects the situation where both classifier and embedding are trained on the same data. As the task is to detect a speaker the speakers present in the test set need to be present in training. Task b represents a situation where classifier and embedding are trained on independent data sets, but with speaker overlap. Finally Task c represents complete independence in training data sets and no speaker overlap. Table TABREF15 summarises the relationships.
In order to achieve these configuration the TIMIT data was split. Fig. FIGREF12 illustrates the split of the data into 8 subsets (A–H). The TIMIT dataset contains speech from 462 speakers in training and 168 speakers in the test set, with 8 utterances for each speaker. The TIMIT training and test set are split into 8 blocks, where each block contains 2 utterances per speaker, randomly chosen. Thus each block A,B,C,D contains data from 462 speakers with 924 utterances taken from the training sets, and each block E,F,G,H contains speech from 168 test set speakers with 336 utterances.
For Task a training of embeddings and the classifier is identical, namely consisting of data from blocks (A+B+C+E+F+G). The test data is the remainder, namely blocks (D+H). For Task b the training of embeddings and classifiers uses (A+B+E+F) and (C+G) respectively, while again using (D+H) for test. Task c keeps both separate: embeddings are trained on (A+B+C+D), classifiers on (E+G) and tests are conducted on (F+H). Note that H is part of all tasks, and that Task c is considerably easier as the number of speakers to separate is only 168, although training conditions are more difficult.
For comparison the implementation, follows the convolutional model structure as deployed in BIBREF17. Both VAE encoder and decoder contain three convolutional layers and one fully-connected layer with 512 nodes. In the first layer of encoder, 1-by-80 filters are applied, and 3-by-1 filters are applied on the following two convolutional layer (strides was set to 1 in the first layer and 2 in the rest two layers). The decoder has the symmetric architecture to the encoder. Each layer is followed by a batch normalisation layerBIBREF26 except for the embedding layer, which is linear. Leaky ReLU activationBIBREF27 is used for each layer except for the embedding layer. The Adam optimiserBIBREF28 is used in training, with $\beta _1$ set to 0.95, $\beta _2$ to 0.999, and $\epsilon $ is $10^{-8}$. The initial learning rate is $10^{-3}$
Table TABREF17 shows phone classification and speaker recognition results for the three model configurations: the VAE baseline, the CJFS encoder and the CJFA encoder. In our experiments the window size was set to 30 frames, namely 10 frames for the target and 10 frames for left and right neighbours, and an embedding dimension of 150. This was used for both CJFS and CJFA models alike. Results show that the CJFA encoder obtains significantly better phone classification accuracy than the VAE baseline and also than the CJFS encoder. These results are replicated for speaker recognition tasks. The CJFA encoder performs better on all tasks than the VAE baseline by a significant margin. It is noteworthy that performance on Task b is generally significantly lower than for Task a, for reasons of training overlap but also smaller training set sizes.
To further explore properties of the embedding systems a change of window size ($N$) and embedding dimension ($K$) is explored. One might argue that modelling context effectively widens the input data access. Hence these experiments should explore if there is benefit in the structure beyond data size. Graphs in Fig. FIGREF14 illustrate phone classification accuracy and speaker recognition performance for all three models under variation of latent size and window sizes. It is important to note that the target window size remains the same (10 frames) with an increase of $N$. Therefore e.g. $N=70$ describes the target window size is 10 frames, and the other two neighbour windows have 30 frames at either side (30,10,30 left to right). Better speaker recognition results are consistently obtained with the CJFA encoder for any configuration with competitive performance, compared with the VAE baseline and also CJFS settings - and CJFS settings mostly outperform the baseline. However the situation for phone classification is different. It is not surprising to see CJFS perform poorly on phone classification as the target frame in not present in the input, therefore the embedding just does not have the phone segment information. However, as per speaker recognition results, speaker information is retained.
A variation of the window sizes to larger windows seems detrimental in almost all cases, aside from the more difficult Task b. This may be in part the effect of the amount of training data available, however it confirms that contextual models outperform the baseline VAE model configuration, generally, and in particular also with the same amount of input data for speaker recognition. It is also noticeable that the decline or variation as a function of window size is less pronounced for the CJFA case, implying increased stability. For phone classification the trade-off benefit for window size is less clear.
For phone classification, increasing the embedding $K$ is helpful, but performance remains stable at $K=150$. Hence in all of the rest of our experiments, the embedding dimension is set to 150 for all of the rest configurations. For speaker recognition the observed variations are small.
A further set of experiments investigated the use of out of domain data for improving classification in a completely unsupervised setting. The RM corpus BIBREF29 was used in this case to augment the TIMIT data for training the embeddings only. All other configurations an training settings are unchanged. Table TABREF18 shows improvement after using additional out-of-domain data for training, except for in the case of CJFS and for phone classification. The improvement on all tasks with the simple addition of unlabelled audio data is remarkable. This is also true for the baseline, but the benefit of the proposed methods seems unaffected. The CJFA encoder performs better in comparison of the other two approaches and a absolute accuracy improvement of 7.9% for speaker recognition Task b is observed. The classification tasks benefits from the additional data even though the labelled data remains the same.
To further evaluate the embeddings produced by the 3 models, visualisation using the t-SNE algorithmBIBREF30 is a common approach, although interpretation is sometimes difficult. Fig. FIGREF19 visualises the embeddings of phonemes in two-dimensional space, each phoneme symbol represents the mean vector of all of the embeddings belonging to the same phone classBIBREF31. One can observe that the CJFA encoder appears to generate more meaningful embeddings than the other two approaches - as phonemes belonging to the same sound classesBIBREF32 are grouped together in closer regions. The VAE baseline also has this behaviour but for example plosives are split and nasal separation seems less clear. Instead CJFS shows more confusion - as expected and explained above.
In this paper, two unsupervised acoustic embedding approaches to model the joint latent factors between the target window and neighbouring audio segments were proposed. Models are based on variational auto-encoders, which also constitute the baseline. In order to compare against the baseline models are assessed using phone classification and speaker recognition tasks, on TIMIT, and with additional RM data. Results show CJFA (contextual joint factor analysis) encoder performs significantly better in both phone classification and speaker recognition tasks compared with other two approaches. The CJFS (contextual joint factor synthesis) encoder performs close to CJFA in speaker recognition task, but poorer for phone classification. Overall a gain of up to 3% relative on phone classification accuracy is observed, relative improvements on speaker recognition show 3–6% gain. The proposed unsupervised approaches obtain embeddings and can be improved with unlabelled out-of-domain data, the classification tasks benefits even though the labelled data remains the same. Further work needs to expand experiments on larger data sets, phone recognition and more complex neural network architectures.

Figure and table captions:
Fig. 1. The architecture of CJFS (Left) and CJFA (Right). Both CJFS and CJFA were built based on variational auto encoder, embeddings were extracted on the bottleneck layer.
Fig. 2. Data split of the TIMIT corpus for definition of data sets for speaker recognition. Training and test sets are split int 4 parts of 2 utterances each. Different combination of sets for training and test are used of different tasks.
Table 1. Definition of training configurations a, b, and c.
Fig. 3. Phone classification accuracy, and speaker recognition accuracy for Tasks a,b,and c (as defined at 4.3), when varying the embedding dimension (top row), and window sizes (bottom row).
Table 2. % Phone classification and speaker recognition accuracy with three different model types. Embedding dimension is 150 and target window size is 10 frames, neighbour window sizes are 10 frames each.
Fig. 4. The t-SNE visualisation of phones in the test set for three models: (a): VAE baseline (b): CJFS encoder and (c): CJFA encoder
Table 3. % Phone classification and speaker recognition accuracy with TIMIT data and RM data.

Questions about this paper:
What TIMIT datasets are used for testing?

Answers:
Once split into 8 subsets (A-H), the test set used are blocks D+H and blocks F+H