Dataset	We used LibriSpeech  as the audio corpus in the experiments, which is a corpus of read speech in English derived from audiobooks. This corpus contains 1000 hours of speech sampled at 16 kHz uttered by 2484 speakers. We used the “clean" and “others" sets with a total of 960 hours, and extracted 39-dim MFCCs as the acoustic features.

Results of Spoken Document Retrieval	The goal here is to retrieve not only those spoken documents including the spoken query (e.g. “President Donald Trump") based on the phonetic structures, but those including words semantically related to the query word (e.g. “White House"). Below we show the effectiveness of the phonetic-and-semantc embedding proposed here in this application. We used the 960 hours of “clean" and “other" parts of LibriSpeech dataset as the target archive for retrieval, which consisted of 1478 audio books with 5466 chapters. Each chapter included 1 to 204 utterances or 5 to 6529 spoken words. In our experiments, the queries were the keywords in the book titles, and the spoken documents were the chapters. We chose 100 queries out of 100 randomly selected book titles, and our goal was to retrieve query-relevant documents. For each query INLINEFORM0 , we defined two sets of query-relevant documents: The first set INLINEFORM1 consisted of chapters which included the query INLINEFORM2 .

The speaker discriminator INLINEFORM7 learns to maximize INLINEFORM8 in ( EQREF9 ), while the phonetic encoder INLINEFORM9 learns to minimize INLINEFORM10 , DISPLAYFORM0  where INLINEFORM0 is a real number. The optimization procedure of Stage 1 consists of four parts: (1) training INLINEFORM0 , INLINEFORM1 and INLINEFORM2 by minimizing INLINEFORM3 , (2) training INLINEFORM4 by minimizing INLINEFORM5 , (3) training INLINEFORM6 by minimizing INLINEFORM7 , and (4) training INLINEFORM8 by maximizing INLINEFORM9 . Parts (1)(2)(3) are jointly trained together, while iteratively trained with part (4)  .

In our evaluation as mentioned below, labeled pairs of the audio and text embeddings of each word is available, that is, we know INLINEFORM0 and INLINEFORM1 for each word INLINEFORM2 . So we can train the transformation matrices INLINEFORM3 and INLINEFORM4 using the gradient descent method to minimize the following objective function: DISPLAYFORM0  where the last two terms in ( EQREF15 ) are cycle-constraints to ensure that both INLINEFORM0 and INLINEFORM1 are almost unchanged after transformed to the other space and back. In this way we say the two sets of embeddings are parallelized.

On the other hand, we also obtained three different types of text embedding (TXT) on the same set of top 1000, 3000 and 5000 words. Type (a) Phonetic Text embedding (TXT-ph) considered precise phonetic structure but not context or semantics at all. This was achieved by a well-trained sequence-to-sequence autoencoder encoding the precise phoneme sequence of a word into a latent embedding. Type (b) Semantic Text embedding considered only context or semantics but not phonetic structure at all, and was obtained by a standard skip-gram model using one-hot representations as the input (TXT-(se,1h)). Type (c) Semantic and Phonetic Text embedding (TXT-(se,ph)) considered context or semantics as well as the precise phonetic structure, obtained by a standard skip-gram model but using the Type (a) Phonetic Text embedding (TXT-ph) as the input. So these three types of text embeddings provided the reference embeddings obtained from text and/or phoneme sequences, not disturbed by audio signals at all.

The goal of Stage 1 is to disentangle the phonetic structure and speaker characteristics in acoustic features, and extract a vector representation for the phonetic structure only. As shown in the middle of Figure FIGREF3 , a sequence of acoustic features INLINEFORM0 is entered to a phonetic encoder INLINEFORM1 and a speaker encoder INLINEFORM2 to obtain a phonetic vector INLINEFORM3 in orange and a speaker vector INLINEFORM4 in green. Then the phonetic and speaker vectors INLINEFORM5 , INLINEFORM6 are used by the decoder INLINEFORM7 to reconstruct the acoustic features INLINEFORM8 . This phonetic vector INLINEFORM9 will be used in the next stage as the phonetic embedding. The two encoders INLINEFORM10 , INLINEFORM11 and the decoder INLINEFORM12 are jointly learned by minimizing the reconstruction loss below: DISPLAYFORM0  It will be clear below how to make INLINEFORM0 and INLINEFORM1 separately encode the phonetic structure and speaker characteristics. The speaker encoder training requires speaker information for the spoken words. Assume the spoken word INLINEFORM0 is uttered by speaker INLINEFORM1 .

Introduction	Word embedding or Word2Vec  ,  ,  ,  has been widely used in the area of natural language processing  ,  ,  ,  ,  ,  ,  , in which text words are transformed into vector representations of fixed dimensionality  ,  ,  . This is because these vector representations carry plenty of semantic information learned from the context of the considered words in the text training corpus. Similarly, audio Word2Vec has also been proposed in the area of speech signal processing, in which spoken words (signal segments for words without knowing the underlying word it represents) are transformed into vector representations of fixed dimensionality  ,  ,  ,  ,  ,  ,  ,  ,  ,  ,  . These vector representations carry the phonetic structures of the spoken words learned from the signals within the spoken words, and have been shown to be useful in spoken term detection, in which the spoken terms are detected simply based on the phonetic structures. Such Audio Word2Vec representations do not carry semantics, because they are learned from individual spoken words only without considering the context.

Given a pair of phonetic vectors INLINEFORM0 , the training criteria for INLINEFORM1 and INLINEFORM2 is to maximize the similarity between INLINEFORM3 and INLINEFORM4 if INLINEFORM5 and INLINEFORM6 are contextual, while minimizing the similarity otherwise. The basic idea is parallel to that of text Word2Vec. Two different spoken words having similar context should have similar semantics. Thus if two different phonetic embeddings corresponding to two different spoken words have very similar context, they should be close to each other after projected by the semantic encoder INLINEFORM7 . The semantic and context encoders INLINEFORM8 and INLINEFORM9 learn to minimize the semantic loss INLINEFORM10 as follows: DISPLAYFORM0  The sigmoid of dot product of INLINEFORM0 and INLINEFORM1 is used to evaluate the similarity. With ( EQREF13 ), if INLINEFORM2 and INLINEFORM3 are in the same context window, we want INLINEFORM4 and INLINEFORM5 to be as similar as possible. We also use the negative sampling technique, in which only some pairs INLINEFORM6 are randomly sampled as negative examples instead of enumerating all possible negative pairs.

Stage 1 - Phonetic Embedding with Speaker Characteristics Disentangled	A text word with a given phonetic structure corresponds to infinite number of audio signals with varying acoustic factors such as speaker characteristics, microphone characteristics, background noise, etc. All the latter acoustic factors are jointly referred to as speaker characteristics here for simplicity, which obviously disturbs the goal of phonetic-and-semantic embedding. So Stage 1 is to obtain phonetic embeddings only with speaker characteristics disentangled. Also, because the training of phonetic-and-semantic embedding is challenging, in the initial effort we slightly simplify the task by assuming all training utterances have been properly segmented into spoken words. Because there exist many approaches for segmenting utterances automatically  , and automatic segmentation plus phonetic embedding of spoken words has been successfully trained and reported before  , such an assumption is reasonable here. We denote the audio corpus as INLINEFORM0 , which consists of INLINEFORM1 spoken words, each represented as INLINEFORM2 , where INLINEFORM3 is the acoustic feature vector for the tth frame and INLINEFORM4 is the total number of frames in the spoken word.

Since column (a) (TXT-ph) considered precise phonetic structures but not semantics at all, the relatively high accuracies in column (a) for all three versions of audio embedding (i)(ii)(iii) implied the three versions of audio embedding were all rich of phonetic information. But when the semantics were embedded in (ii)(iii) (AUD-(ph-+se), AUD-(ph+se)), the phonetic structures were inevitably disturbed (0.519, 0.598 vs 0.637). On the other hand, column (b) (TXT-(se,1h)) considered only semantics but not phonetic structure at all, the relatively lower accuracies implied the three versions of audio embedding did bring some good extent of semantics, except (i) AUD-ph, but obviously weaker than the phonetic information in column (a). Also, the Stage 2 training in rows (ii)(iii) (AUD-(ph-+se), AUD-(ph+se)) gave higher accuracies than row (i) (AUD-ph) (0.339, 0.332 vs 0.124 in column (b)), which implied the Stage 2 training was successful.

Text words can be trained and embedded as vectors carrying plenty of semantics because the phonetic structures are not considered at all. On the other hand, because spoken words are just a different version of representations for text words, it is also natural to believe they do carry some semantic information, except disturbed by phonetic structures plus some other acoustic factors such as speaker characteristics and background noise  ,  ,  ,  ,  ,  . So the goal of embedding spoken words to carry both phonetic structures and semantics is possible, although definitely hard. But a nice feature of such embeddings is that they may include both phonetic structures and semantics  ,  . A direct application for such phonetic-and-semantic embedding of spoken words is spoken document retrieval  ,  ,  ,  ,  . This task is slightly different from spoken term detection, in the latter case spoken terms are simply detected based on the phonetic structures. Here the goal of the task is to retrieve all spoken documents (sets of consecutive utterances) relevant to the spoken query, which may or may not include the query.

The results are listed in Table TABREF21 for two definitions of groundtruth for the query-relevant documents: the union of INLINEFORM0 and INLINEFORM1 and INLINEFORM2 alone. As can be found from this table, AUD-(ph+se) offered better retrieval performance than AUD-ph in both rows. Note that those chapters in INLINEFORM3 in the second row of the table did not include the query INLINEFORM4 , so couldn't be well retrieved using phonetic embedding alone. That is why the phonetic-and-semantic embedding proposed here can help. In Table TABREF22 , we list some chapters in INLINEFORM0 retrieved using AUD-(ph+se) embeddings to illustrate the advantage of the phonetic-and-semantic embeddings.

Proposed Approach	The proposed framework of phonetic-and-semantic embedding of spoken words consists of two stages: Stage 1 - Phonetic embedding with speaker characteristics disentangled. Stage 2 - Semantic embedding over phonetic embeddings obtained in Stage 1. In addition, we propose an approach for parallelizing the audio and text embeddings to be used for evaluating the phonetic and semantic information carried by the audio embeddings. These are described in Subsections SECREF2 , SECREF11 and SECREF14 respectively.

Audio Word2Vec was recently extended to Segmental Audio Word2Vec  , in which an utterance can be automatically segmented into a sequence of spoken words  ,  ,  ,  and then transformed into a sequence of vectors of fixed dimensionality by Audio Word2Vec, and the spoken word segmentation and Audio Word2Vec can be jointly trained from an audio corpus. In this way the Audio Word2Vec was upgraded from word-level to utterance-level. This offers the opportunity for Audio Word2Vec to include semantic information in addition to phonetic structures, since the context among spoken words in utterances bring semantic information. This is the goal of this work, and this paper reports the first set of results towards such a goal. In principle, the semantics and phonetic structures in words inevitably disturb each other. For example, the words “brother" and “sister" are close in semantics but very different in phonetic structure, while the words “brother" and “bother" are close in phonetic structure but very different in semantics. This implies the goal of embedding both phonetic structures and semantics for spoken words is naturally very challenging.

So we try to learn a mapping relation between the two spaces. It will be clear below such a mapping relation can be used to evaluate the phonetic and semantic information carried by the audio embeddings. Mini-Batch Cycle Iterative Closest Point (MBC-ICP)  previously proposed as described below is used here. Given two sets of embeddings as mentioned above, INLINEFORM0 and INLINEFORM1 , they are first projected to their respective top INLINEFORM2 principal components by PCA. Let the projected sets of vectors of INLINEFORM3 and INLINEFORM4 be INLINEFORM5 and INLINEFORM6 respectively. If INLINEFORM7 can be mapped to the space of INLINEFORM8 by an affine transformation, the distributions of INLINEFORM9 and INLINEFORM10 would be similar after PCA  . Then a pair of transformation matrices, INLINEFORM0 and INLINEFORM1 , is learned, where INLINEFORM2 transforms a vector INLINEFORM3 in INLINEFORM4 to the space of INLINEFORM5 , that is, INLINEFORM6 , while INLINEFORM7 maps a vector INLINEFORM8 in INLINEFORM9 to the space of INLINEFORM10 . INLINEFORM11 and INLINEFORM12 are learned iteratively by the algorithm proposed previously  .

For example, for the spoken query of “President Donald Trump", not only those documents including the spoken query should be retrieved based on the phonetic structures, but those documents including semantically related words such as “White House" and “trade policy", but not necessarily “President Donald Trump", should also be retrieved. This is usually referred to as “semantic retrieval", which can be achieved by the phonetic-and-semantic embedding discussed here. This paper proposes a two-stage framework of phonetic-and-semantic embedding for spoken words. Stage 1 performs phonetic embedding but with speaker characteristics disentangled using separate phonetic and speaker encoders and a speaker discriminator. Stage 2 then performs semantic embedding in addition. We further propose to evaluate the phonetic-and-semantic nature of the audio embeddings obtained in Stage 2 by parallelizing with text embeddings  ,  . Very encouraging results including those for an application task of spoken document retrieval were obtained in the initial experiments.

Stage 2 - Semantic Embedding over Phonetic Embeddings Obtained in Stage 1	As shown in Figure FIGREF12 , similar to the Word2Vec skip-gram model  , we use two encoders: semantic encoder INLINEFORM0 and context encoder INLINEFORM1 to embed the semantics over phonetic embeddings INLINEFORM2 obtained in Stage 1. On the one hand, given a spoken word INLINEFORM3 , we feed its phonetic vector INLINEFORM4 obtained from Stage 1 into INLINEFORM5 as in the middle of Figure FIGREF12 , producing the semantic embedding (in yellow) of the spoken word INLINEFORM6 . On the other hand, given the context window size INLINEFORM7 , which is a hyperparameter, if a spoken word INLINEFORM8 is in the context window of INLINEFORM9 , then its phonetic vector INLINEFORM10 is a context vector of INLINEFORM11 . For each context vector INLINEFORM12 of INLINEFORM13 , we feed it into the context encoder INLINEFORM14 in the upper part of Figure FIGREF12 , and the output is the context embedding INLINEFORM15 .

Model Implementation	In Stage 1, The phonetic encoder INLINEFORM0 , speaker encoder INLINEFORM1 and decoder INLINEFORM2 were all 2-layer GRUs with hidden layer size 128, 128 and 256, respectively. The speaker discriminator INLINEFORM3 is a fully-connected feedforward network with 2 hidden layers with size 128. The value of INLINEFORM4 we used in INLINEFORM5 in ( EQREF7 ) was set to 0.01. In Stage 2, the two encoders INLINEFORM0 and INLINEFORM1 were both 2-hidden-layer fully-connected feedforward networks with size 256. The size of embedding vectors was set to be 128. The context window size was 5, and the negative sampling number was 5. For parallelizing the text and audio embeddings in Subsection SECREF14 , we projected the embeddings to the top 100 principle components, so the affine transformation matrices were INLINEFORM0 . The mini-batch size was 200, and INLINEFORM1 in ( EQREF15 ) was set to 0.5.

Now we can perform the transformation from the above three versions of audio embeddings (AUD-ph, AUD-(ph-+se), AUD-(ph+se)) to the above three types of text embeddings (TXT-ph, TXT-(se,1h), TXT-(se,ph)) by parallelizing the embeddings as described in Subsection SECREF14 . The evaluation metric used for this parallelizing test is the top-k nearest accuracy. If the audio embedding representation INLINEFORM0 of a word INLINEFORM1 is transformed to the text embedding INLINEFORM2 by INLINEFORM3 , and INLINEFORM4 is among the top-k nearest neighbors of the text embedding representation INLINEFORM5 of the same word, this transformation for word INLINEFORM6 is top-k-accurate. The top-k nearest accuracy is then the percentage of the words considered which are top-k-accurate. The results of top-k nearest accuracies for k=1 and 10 are respectively listed in Tables TABREF18 and TABREF19 , each for 1000, 3000 and 5000 pairs of spoken and text words. First look at the top part of Table TABREF18 for top-1 nearest accuracies for 1000 pairs of audio and text embeddings.

Parallelizing Audio and Text Embeddings for Evaluation Purposes	In this paper we further propose an approach of parallelizing a set of audio embeddings (for spoken words) with a set of text embeddings (for text words) which will be useful in evaluating the phonetic and semantic information carried by these embeddings. Assume we have the audio embeddings for a set of spoken words INLINEFORM0 INLINEFORM1 , where INLINEFORM2 is the embedding obtained for a spoken word INLINEFORM3 and INLINEFORM4 is the total number of distinct spoken words in the audio corpus. On the other hand, assume we have the text embeddings INLINEFORM5 INLINEFORM6 , where INLINEFORM7 is the embedding of the INLINEFORM8 -th text word for the INLINEFORM9 distinct text words. Although the distributions of INLINEFORM10 and INLINEFORM11 in their respective spaces are not parallel, that is, a specific dimension in the space for INLINEFORM12 does not necessarily correspond to a specific dimension in the space for INLINEFORM13 , there should exist some consistent relationship between the two distributions. For example, the relationships among the words {France, Paris, Germany} learned from context should be consistent in some way, regardless of whether they are in text or spoken form.

The second set INLINEFORM3 consisted of chapters whose content didn't contain INLINEFORM4 , but these chapters belonged to books whose titles contain INLINEFORM5 (so we assume these chapters are semantically related to INLINEFORM6 ). Obviously INLINEFORM7 and INLINEFORM8 were mutually exclusive, and INLINEFORM9 were the target for semantic retrieval, but couldn't be retrieved based on the phonetic structures only. For each query INLINEFORM0 and each document INLINEFORM1 , the relevance score of INLINEFORM2 with respect to INLINEFORM3 , INLINEFORM4 , is defined as follows: DISPLAYFORM0  where INLINEFORM0 is the audio embedding of a word INLINEFORM1 in INLINEFORM2 . So ( EQREF25 ) indicates the documents INLINEFORM3 were ranked by the minimum distance between a word INLINEFORM4 in INLINEFORM5 and the query INLINEFORM6 . We used mean average precision (MAP) as the evaluation metric for the spoken document retrieval test. We compared the retrieval results with two versions of audio embedding: AUD-(ph+se) and AUD-ph.

Phonetic-and-Semantic Embedding of Spoken Words with Applications in Spoken Content Retrieval	Word embedding or Word2Vec has been successful in offering semantics for text words learned from the context of words. Audio Word2Vec was shown to offer phonetic structures for spoken words (signal segments for words) learned from signals within spoken words. This paper proposes a two-stage framework to perform phonetic-and-semantic embedding on spoken words considering the context of the spoken words. Stage 1 performs phonetic embedding with speaker characteristics disentangled. Stage 2 then performs semantic embedding in addition. We further propose to evaluate the phonetic-and-semantic nature of the audio embeddings obtained in Stage 2 by parallelizing with text embeddings. In general, phonetic structure and semantics inevitably disturb each other. For example the words"brother"and"sister"are close in semantics but very different in phonetic structure, while the words"brother"and"bother"are in the other way around. But phonetic-and-semantic embedding is attractive, as shown in the initial experiments on spoken document retrieval.

Evaluation by Parallelizing Audio and Text Embeddings	Each text word corresponds to many audio realizations in spoken form. So we first took the average of the audio embeddings for all those realizations to be the audio embedding for the spoken word considered. In this way, each word has a unique representation in either audio or text form. We applied three different versions of audio embedding (AUD) on the top 1000, 3000 and 5000 words with the highest frequencies in LibriSpeech: (i) phonetic embedding only obtained in Stage 1 in Subsection SECREF2 (AUD-ph); (ii) phonetic-and-semantic embedding obtained by Stages 1 and 2 in Subsections SECREF2 , SECREF11 , except the speaker characteristics not disentangled (AUD-(ph-+se)), or INLINEFORM0 , INLINEFORM1 in ( EQREF7 ), ( EQREF9 ) not considered; (iii) complete phonetic-and-semantic embedding as proposed in this paper including Stages 1 and 2 (AUD-(ph+se)). So this is for ablation study.

Conclusions and Future Work	In this paper we propose a framework to embed spoken words into vector representations carrying both the phonetic structure and semantics of the word. This is intrinsically challenging because the phonetic structure and the semantics of spoken words inevitably disturbs each other. But this phonetic-and-semantic embedding nature is desired and attractive, for example in the application task of spoken document retrieval. A parallelizing transformation between the audio and text embeddings is also proposed to evaluate whether such a goal is achieved.

However, column (c) (TXT-(se,ph)) is for the text embedding considering both the semantic and phonetic information, so the two versions of phonetic-and-semantic audio embedding for rows (ii)(iii) had very close distributions (0.750, 0.800 in column (c)), or carried good extent of both semantics and phonetic structure. The above are made clearer by the numbers in bold which are the highest for each row, and the numbers in red which are the highest for each column. It is also clear that the speaker characteristics disentanglement is helpful, since row (iii) for AUD-(ph+se) was always better than row (ii) for AUD-(ph-+se). Similar trends can be observed in the other parts of Table TABREF18 for 3000 and 5000 pairs, except the accuracies were lower, probably because for more pairs the parallelizing transformation became more difficult and less accurate.

In this table, column (a) is the query INLINEFORM1 , column (b) is the title of a book INLINEFORM2 which had chapters in INLINEFORM3 , column (c) is a certain chapter INLINEFORM4 in INLINEFORM5 , column (d) is the rank of INLINEFORM6 out of all chapters whose content didn't contain INLINEFORM7 , and column (e) is a part of the content in INLINEFORM8 where the word in red is the word in INLINEFORM9 with the highest similarity to INLINEFORM10 . For example, in the first row for the query “nations", the chapter “Prometheus the Friend of Man" of the book titled “Myths and Legends of All Nations" is in INLINEFORM11 . The word “nations" is not in the content of this chapter. However, because the word “king" semantically related to “nations" is in the content, this chapter was ranked the 13th among all chapters whose content didn't contain the word “nations". This clearly verified why the semantics in the phonetic-and-semantic embeddings can remarkably improve the performance of spoken content retrieval.

When the speaker information is not available, we can simply assume that the spoken words in the same utterance are produced by the same speaker. As shown in the lower part of Figure FIGREF3 , INLINEFORM2 is learned to minimize the following loss: DISPLAYFORM0  In other words, if INLINEFORM0 and INLINEFORM1 are uttered by the same speaker ( INLINEFORM2 ), we want their speaker embeddings INLINEFORM3 and INLINEFORM4 to be as close as possible. But if INLINEFORM5 , we want the distance between INLINEFORM6 and INLINEFORM7 larger than a threshold INLINEFORM8 . As shown in the upper right corner of Figure FIGREF3 , a speaker discriminator INLINEFORM0 takes two phonetic vectors INLINEFORM1 and INLINEFORM2 as input and tries to tell if the two vectors come from the same speaker. The learning target of the phonetic encoder INLINEFORM3 is to "fool" this speaker discriminator INLINEFORM4 , keeping it from discriminating the speaker identity correctly. In this way, only the phonetic structure information is learned in the phonetic vector INLINEFORM5 , while only the speaker characteristics is encoded in the speaker vector INLINEFORM6 .

The only difference is that in these parts column (a) for TXT-ph had the highest accuracies, probably because the goal of semantic embedding for rows (ii)(iii) (AUD-(ph-+se), AUD-(ph+se)) was really difficult, and disturbed or even dominated by phonetic structures. Similar trends can be observed in Table TABREF19 for top-10 accuracies, obviously with higher numbers for top-10 as compared to those for top-1 in Table TABREF18 . In Table TABREF20 , we list some examples of top-10 nearest neighbors in AUD-(ph+se) (proposed), AUD-ph (with phonetic structure) and TXT-(se,1h) (with semantics). The words in red are the common words for AUD-(ph+se) and AUD-ph, and the words in bold are the common words of AUD-(ph+se) and TXT-(se,1h). For example, the word “owned" has two common semantically related words “learned" and “known" in the top-10 nearest neighbors of AUD-(ph+se) and TXT-(se,1h).

The word “owned" also has three common phonetically similar words “armed", “own" and “only" in the top-10 nearest neighbors of AUD-(ph+se) and AUD-ph. This is even clearer for the function word “didn't". These clearly illustrate the phonetic-and-semantic nature of AUD-(ph+se).

Not only spoken documents including the spoken query can be retrieved based on the phonetic structures, but spoken documents semantically related to the query but not including the query can also be retrieved based on the semantics.

Question: What is the dataset that is used to train the embeddings?
Answer: LibriSpeech