Abstract: Word embedding or Word2Vec has been successful in offering semantics for text words learned from the context of words. Audio Word2Vec was shown to offer phonetic structures for spoken words (signal segments for words) learned from signals within spoken words. This paper proposes a two-stage framework to perform phonetic-and-semantic embedding on spoken words considering the context of the spoken words. Stage 1 performs phonetic embedding with speaker characteristics disentangled. Stage 2 then performs semantic embedding in addition. We further propose to evaluate the phonetic-and-semantic nature of the audio embeddings obtained in Stage 2 by parallelizing with text embeddings. In general, phonetic structure and semantics inevitably disturb each other. For example the words"brother"and"sister"are close in semantics but very different in phonetic structure, while the words"brother"and"bother"are in the other way around. But phonetic-and-semantic embedding is attractive, as shown in the initial experiments on spoken document retrieval. Not only spoken documents including the spoken query can be retrieved based on the phonetic structures, but spoken documents semantically related to the query but not including the query can also be retrieved based on the semantics.

Introduction: Word embedding or Word2Vec BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 has been widely used in the area of natural language processing BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , in which text words are transformed into vector representations of fixed dimensionality BIBREF11 , BIBREF12 , BIBREF13 . This is because these vector representations carry plenty of semantic information learned from the context of the considered words in the text training corpus. Similarly, audio Word2Vec has also been proposed in the area of speech signal processing, in which spoken words (signal segments for words without knowing the underlying word it represents) are transformed into vector representations of fixed dimensionality BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18 , BIBREF19 , BIBREF20 , BIBREF21 , BIBREF22 , BIBREF23 , BIBREF24 . These vector representations carry the phonetic structures of the spoken words learned from the signals within the spoken words, and have been shown to be useful in spoken term detection, in which the spoken terms are detected simply based on the phonetic structures. Such Audio Word2Vec representations do not carry semantics, because they are learned from individual spoken words only without considering the context. Audio Word2Vec was recently extended to Segmental Audio Word2Vec BIBREF25 , in which an utterance can be automatically segmented into a sequence of spoken words BIBREF26 , BIBREF27 , BIBREF28 , BIBREF29 and then transformed into a sequence of vectors of fixed dimensionality by Audio Word2Vec, and the spoken word segmentation and Audio Word2Vec can be jointly trained from an audio corpus. In this way the Audio Word2Vec was upgraded from word-level to utterance-level. This offers the opportunity for Audio Word2Vec to include semantic information in addition to phonetic structures, since the context among spoken words in utterances bring semantic information. This is the goal of this work, and this paper reports the first set of results towards such a goal. In principle, the semantics and phonetic structures in words inevitably disturb each other. For example, the words “brother" and “sister" are close in semantics but very different in phonetic structure, while the words “brother" and “bother" are close in phonetic structure but very different in semantics. This implies the goal of embedding both phonetic structures and semantics for spoken words is naturally very challenging. Text words can be trained and embedded as vectors carrying plenty of semantics because the phonetic structures are not considered at all. On the other hand, because spoken words are just a different version of representations for text words, it is also natural to believe they do carry some semantic information, except disturbed by phonetic structures plus some other acoustic factors such as speaker characteristics and background noise BIBREF30 , BIBREF31 , BIBREF32 , BIBREF33 , BIBREF34 , BIBREF35 . So the goal of embedding spoken words to carry both phonetic structures and semantics is possible, although definitely hard. But a nice feature of such embeddings is that they may include both phonetic structures and semantics BIBREF36 , BIBREF37 . A direct application for such phonetic-and-semantic embedding of spoken words is spoken document retrieval BIBREF38 , BIBREF39 , BIBREF40 , BIBREF41 , BIBREF42 . This task is slightly different from spoken term detection, in the latter case spoken terms are simply detected based on the phonetic structures. Here the goal of the task is to retrieve all spoken documents (sets of consecutive utterances) relevant to the spoken query, which may or may not include the query. For example, for the spoken query of “President Donald Trump", not only those documents including the spoken query should be retrieved based on the phonetic structures, but those documents including semantically related words such as “White House" and “trade policy", but not necessarily “President Donald Trump", should also be retrieved. This is usually referred to as “semantic retrieval", which can be achieved by the phonetic-and-semantic embedding discussed here. This paper proposes a two-stage framework of phonetic-and-semantic embedding for spoken words. Stage 1 performs phonetic embedding but with speaker characteristics disentangled using separate phonetic and speaker encoders and a speaker discriminator. Stage 2 then performs semantic embedding in addition. We further propose to evaluate the phonetic-and-semantic nature of the audio embeddings obtained in Stage 2 by parallelizing with text embeddings BIBREF43 , BIBREF44 . Very encouraging results including those for an application task of spoken document retrieval were obtained in the initial experiments.

Question: What speaker characteristics are used?
Answer: Acoustic factors such as speaker characteristics, microphone characteristics, background noise.