# Voice Keyword Retrieval Method Using Attention Mechanism and Multimodal Information Fusion

Authors: Hongli Zhang
Journal: Scientific Programming
 (2021)

---

## Abstract

A cross-modal speech-text retrieval method using interactive learning convolution automatic encoder (CAE) is proposed. First, an interactive learning autoencoder structure is proposed, including two inputs of speech and text, as well as processing links such as encoding, hidden layer interaction, and decoding, to complete the modeling of cross-modal speech-text retrieval. Then, the original audio signal is preprocessed and the Mel frequency cepstrum coefficient (MFCC) feature is extracted. In addition, the word bag model is used to extract the text features, and then the attention mechanism is used to combine the text and speech features. Through interactive learning CAE, the shared features of speech and text modes are obtained and then sent to modal classifier to identify modal information, so as to realize cross-modal voice text retrieval. Finally, experiments show that the performance of the proposed algorithm is better than that of the contrast algorithm in terms of recall rate, accuracy rate, and false recognition rate.


---

## Body

## 1. Introduction

With the development of communication and Internet technologies, a large amount of multimedia data has been produced. In the face of massive multimedia data, how to quickly detect, retrieve, and manage information is an important research topic facing information science [1, 2]. Among a large amount of multimedia data, voice and text are the two most important main carriers in information exchange. The development of text information retrieval technology is relatively mature and is currently widely used in a large number of commercial search engines. As for the information retrieval of voice, the information retrieval based on the content of voice itself is still in the research stage [3, 4]. As the most natural way of human communication, voice is not only a bridge in the direct communication between human beings but also one of the main carriers in the transmission of audio and video information. Therefore, in the field of multimedia information processing, voice information retrieval is one of the important research hotspots [5].Voice keyword retrieval is an important means to realize content-based voice information retrieval. When it is necessary to analyze the voice content, it has a wide range of application scenarios [6–8]. (1) Human-computer interaction application. Human-computer interaction based on voice can liberate hands and is the most natural way of interaction. At present, human-computer interaction mainly uses keyboards, touch screens, buttons, and so on, which face many inconveniences in many scenarios. For example, in the vehicle environment, it is very dangerous to query the road navigation through touch screen or keyboard while driving. If the navigation destination query can be realized automatically by voice command, it will be much safer. In family life, controlling TV, air conditioner, and other household appliances by voice command will also bring great convenience to people's life. Keyword retrieval based on voice is a method to realize human-computer command interaction. If there are specific keywords or combinations of user commands in the retrieval, the user's simple intention can be judged. (2) Multimedia information retrieval application. Nowadays, there is a lot of multimedia information on the Internet. Whether it is a traditional radio and television station, film and television drama studio, or engaged in education, advertising, live broadcast website enterprises, or personal microvideo enthusiasts, the production of a large number of multimedia audio and video data. How to search for the needed information from massive data is an important requirement for users. For example, in the microclasses of education and teaching, you can quickly locate the knowledge points of interest and search for the moments of a certain star's game in the sports video. Voice keyword retrieval, through the construction of index, can help users get the information they need faster and more targeted. (3) Internet audio and video monitoring applications. The openness of the Internet determines that users can easily and quickly publish and obtain all kinds of information. If there are no appropriate regulatory measures, the audio and video information such as counterfeiting, gambling, terror, and violence spread on the Internet will seriously damage the Internet Ecology and have a negative impact on social development. Due to the unstructured nature of audio and video, the traditional information supervision technology based on text content detection is powerless. Voice keyword retrieval technology can help the relevant institutions to supervise the Internet audio and video through information detection of audio and video content and provide a good Internet environment for Internet users.Voice is developed from the continuous communication and cooperation of human beings and contains rich information [9, 10]. In addition to content information, it also includes language information, speaker information, emotional information, and surrounding environment information. For the research of speech information retrieval technology, there are also different technical routes, such as speech recognition, language determination, speaker retrieval, emotion recognition, and other tasks [11, 12]. Considering that the voice content is the most direct information transmitted in the voice signal, this paper will focus on the content-based voice keyword retrieval method.


### 1.1. Related Research

Multimodal data retrieval can improve the user retrieval and interactive experience, comprehensively enhance the attractiveness and competitiveness of multimedia information retrieval system, and effectively supplement the traditional single-mode retrieval mode [13–15]. Multimodal information retrieval usually uses a set of semantic vectors (weight of feature vectors) to represent the data of each mode. An effective mapping mechanism is used to map the data of different modes to a common potential feature space. An effective mapping mechanism not only needs to maintain the semantic similarity between the internal data of each mode but also maintains the semantic similarity between different modal data [16–18]. At present, cross-modal retrieval mainly focuses on media data feature learning, cross-modal retrieval model design, and so on. Among them, the advantages and disadvantages of cross-modal retrieval model are one of the key factors restricting the retrieval performance [19]. Reference [20] proposed a text image collaborative attention network model (CoAN), which enhanced the measurement of content similarity by selectively focusing on the key information parts of multimodal data. CoAN uses pretrained VGGNet model and recurrent neural network to extract fine-grained features of images and texts. The text visual attention mechanism is used to capture the subtle interaction between language and vision. At the same time, the model learns the hash representation of text and image, respectively, and improves the retrieval speed by using the low storage characteristics and computational efficiency of hash method.Multimodal retrieval technology promotes the development and application of machine learning, provides theoretical support for the development of new retrieval technology, and has extensive application value and theoretical significance for the research of cross-modal retrieval method [21–24]. With the extensive application of deep learning method in machine learning field, researchers have proposed a series of multimodal information retrieval methods based on deep neural network. For example, [25] proposed a novel word discovery framework using dynamic shape coding to retrieve text in natural scene images and video frames. A novel framework based on end-to-end deep semantic-preserving ordinal hashing (DSPOH) is proposed in [26], which uses neural networks to learn hash functions. In DSPOH, the ordinal representation of relative rank order representing feature size is studied to generate hash codes. Such sequential embedding benefits from the numerical stability of rank correlation measures. In [27], in the construction process of mapping mechanism of multimodal information retrieval, the deep learning method avoids the feature extraction of single modal data and greatly improves the construction speed of the model. However, most of the existing research based on deep neural network is to construct a common subspace for different modes to mine the global correspondence of heterogeneous data. However, the local fine-grained features with semantic identification are ignored [28]. In [29], the author proposed an architecture called DenseNet-BiLSTM for keyword retrieval. However, in their tasks, the keywords allowed to be input are selected from a group of command words, while the application scenarios of our model allow the input of any word or phrase of the language. Most of these methods assume that the semantic data of different modes have the same amount of information. However, different modes often have unbalanced and complementary relations, and the amount of information contained in the description of the same semantics is not the same.In view of the increasingly strong demand for cross-modal retrieval, the existing methods have some problems, such as low retrieval accuracy and unsuitable for high dimensions. A cross-modal speech-text retrieval method based on interactive learning classical convolution automatic encoder (CAE) is proposed. The main contents of this paper are as follows:(1)
Based on an interactive learning convolutional autoencoder structure, including two input of speech and text, as well as encoding, hidden layer interaction, decoding, and other processing links, the modeling of cross-modal speech-text retrieval is completed(2)
Through interactive learning of CAE, the shared characteristics of speech and text modalities are obtained, and they are sent to the modal classifier to identify modal information and realize cross-modal speech-text retrieval


### 1.2. System Architecture

The overall architecture of the proposed method is shown in Figure1. For the input speech samples, the proposed method preprocesses the original audio signal and extracts Mel frequency cepstral coefficients (MFCC) features. For text samples, the bag-of-words model (BoW) is used to extract text features. Then, it combines the features of text and speech through the attention mechanism and then learns the shared features of speech and text modalities through interactive learning CAE. Then, the discriminator will be used to obtain the confidence score of the final keyword search and conduct adversarial training between the generative model and the discriminant model. The generative model embeds semantic label information and promotes the difference between modal-specific features and shared modal features to generate discriminative modal sharing and modal-specific representations. The discriminant model inputs the learned modal shared representations corresponding to the speech and text modalities into the modal classifier to identify modal information and realize cross-modal speech-text retrieval.Figure 1
Overall framework of the proposed method.


### 1.3. Feature Extraction

#### 1.3.1. Feature Extraction of Speech Signal

The original voice signal is often difficult to process directly, so for the voice matching model, the feature extraction of the voice signal is required. The feature extraction of the speech signal includes the preprocessing of the original speech signal, and its steps include preemphasis, framing, and windowing operations. The preprocessing of the speech signal is also called front end processing. It processes the original speech before extracting features, so that the processed signal can better meet actual needs. The preemphasis, framing, and windowing operations are used in the preprocessing of speech keyword matching:(1) Pre-emphasis. Pre-emphasis is usually realized with 6 db/octave digital filter frequency characteristics. When the voice signal is affected by the nose or mouth, the high-frequency components will have a greater drop. Therefore, it is absolutely necessary to pass a high-pass filter before preemphasis. The filter can be expressed as shown in the following equation:(1)Hz=1−uz−1,where u is the preemphasis coefficient and the value is 0.9415 according to actual needs.(2) Framing. Framing is to divide the speech signal into several segments and take the signal within a short period of time at the center of each segment as a whole speech unit, and its unit is a frame. In this way, the short-term characteristics of the speech signal at a certain moment can be obtained, and the frequency spectrum can be processed on it, and the continuous changes between frames can be analyzed. The framing of the speech signal can be continuous; that is, there is no overlap between adjacent frames. But more often there is overlap between adjacent frames during framing, and the overlapping part between adjacent frames is called frameshift. Generally, the length of the frameshift is set to be half of the frame length. The time length of each frame is generally between 15 ms and 20 ms as shown in Figure 2.Figure 2
Schematic diagram of voice framing.(3) Windowing. In terms of windowing operation, it is to add a window function to the signal. The value of the window function in a given interval is a real number, and the values outside the interval are all 0. Therefore, the effect of multiplying a signal by a window function is like looking at other functions through this window. Typical window functions include rectangular windows, Gaussian windows, Hamming windows, and so on.Hamming window is used in this paper. The time-domain expression of Hamming window is(2)wn=0.54−0.46cos2πnN−1,where n is time and wn is the window function of the Hamming window.Mel cepstral coefficient is a commonly used feature in the current speech recognition field, and it is also one of the speech recognition features used in this paper. The Mel cepstrum coefficient is the Mel frequency characteristic parameter based on the cepstrum, and the functional relationship between the Mel frequency and the linear frequency is nonlinear. Their functional relationship is as follows:(3)Melf=2595×lg1+f700.The main steps to extract Mel cepstrum coefficients are as follows: (1) The input speech signal is transformed into frequency domain signal by Fourier transform. (2) Perform the filtering operation and set the bandpass filter based on the Mel scale from low to high according to different frequencies. (3) The signals processed by bandpass filter are further processed, such as logarithm operation. (4) Carrying out discrete cosine transform, these processed signal characteristics can be regarded as the final characteristic parameters of the speech signal. Based on the voice features extracted by MFCC, good recognition results will be achieved when applied to a voice matching system.


### 1.4. Text Feature Extraction

The BoW model ignores the grammar, word order, and other elements of the text and treats the text as a collection of several words. The words in the text appear independently, using a set of disordered words to express a text. Complete the extraction of text features according to the statistical information of words in the text.First, two simple text documents are given as follows:Tom likes to go sports. Tim likes tooTom also likes to play video gamesAccording to the words that appear in the document, build the following dictionary:(4)“Tom”:1,“likes”:2,“to”:3,“go”:4,“sports”:5,“also”:6,“vesio”:7,“games”:8,“Tim”:9,“too”:10.The above dictionary contains 10 words; each word has a unique index (label). Then, each text can be represented by a 10-dimensional vector as follows:(5)1,2,1,1,1,0,0,0,1,1,1,1,1,1,0,1,1,1,0,0.The dimension of the vector is the number of words, and the value is the frequency of each word in the text. The above vector can also be represented by a histogram of words. The vocabulary is equivalent to the base of the histogram, and the text to be expressed is mapped to this base. The BoW model is used for text feature extraction, including three steps: vocabulary establishment, sample training, and feature extraction of new samples.


### 1.5. Voice Keyword Search Implementation

#### 1.5.1. CAE Based on Interactive Learning

Convolutional autoencoder is proposed on the basis of autoencoder (AE) and denoising autoencoder (DAE). All three are based on unsupervised training to extract features. However, AE and DAE need to pull the input data into a one-dimensional vector in the raster scan order for processing, which will inevitably cause the loss of the two-dimensional structure information of the speech and may introduce redundancy. On the basis of AE and DAE, the convolution and downsampling methods of CNN are introduced, and CAE is proposed. It is a neural network model with a hierarchical structure for two-dimensional data, which coincides with the topological structure of speech. It can directly use voice data as the input of CAE, avoiding the loss of two-dimensional information caused by voice reconstruction.The purpose of CAE is to perform convolutional encoding and decoding on input data. Through convolution and downsampling coding, find out the structure and law of local features in the data and restore the data through reverse decoding. The structure of CAE includes four layers: convolutional layer, downsampling layer, upsampling layer, and deconvolution layer.Suppose the input data isX=X1,X2,…,Xn, the filter map is defined as F1=F1,F2,…,Fm, and each depth is D. The convolution operation obtains m activation maps with depth m, as follows:(6)fX=α∑d=1D∑u=−2k−12k+1∑v=−2k−12k+1FPd1u,vXdi−u,j−v,where α is nonlinear and p∈1,…,m.These activation maps are low-dimensional coded representations of the input dataX, which are downsampled using the maximum pooling operation. The proposed method transmits this activation feature map through two hidden neural layers and then projects it into a d-dimensional common subspace.In order to project the data back into the input space, deconvolution is used as a decoding step. This is achieved by padding the zero sequences to the input before the convolution step. The number of filtersFm2 used is N and the size is 2k+1,2k+1,n. The reconstruction data are given by the following formula:(7)X˜=αfX∗Fm2.Finally, CAE performs input data reconstruction by minimizing the loss between input and reconstruction data.(8)LX,X˜=12X−X˜22,where LX,X˜ is mean square error (MSE). Because MSE captures potential comparisons by performing point or pixel comparisons, it is optimal in defining the loss term.In order to obtain a common representation of the two views of speech and text, CAE based on interactive learning is used, which can be extended to multiple hidden layers. Its structure is shown in Figure3. Multiple hidden layers pass data through more nonlinearities, ensuring better interaction between the two views of data.Figure 3
CAE structure of interactive learning.In order to train the proposed model, the objective function set is as follows: minimize the self-reconstruction error; minimize the cross-reconstruction error of each intermediate step; maximize the correlation between the hidden representations of the two views in each coding step.Given input aszi=xi;yi, where zi is the associated representation of input views xi and yi, self-reconstruction loss and cross-reconstruction loss are defined as(9)Q1=∑i=0NLzi,ghzi,Q2=∑i=0NLzi,ghxi,Q3=∑i=0NLzi,ghyi,Q4=∑k=0K∑i=0NLhxik,hyikQ5=∑i=0NLghxi,ghyi,where g,h are the nonlinearity generally regarded as ReLU, ghxik and ghyik are the hidden representation of the kth intermediate hidden layer, S is the mean square error function, and Q1 is the step length calculation. In losses Q2 and Q3 (for cross reconstruction), the 0 vectors are used instead of another view to calculate xi and yi. The reason for introducing step calculation is the similarity of hidden representations. Therefore, reconstructing one hidden representation from another hidden representation helps the model finally reconstruct the view and improve the interaction between layers. In order to maximize the interaction between the two views, add related losses as follows:(10)Q6=λcorrhX,hY,Q7=∑k=0KλkcorrhXk,hYk,where hX and hY are the projection of the combined model. X and Y are the representation of the input view obtained after convolution and pooling layers. λk is the relative regularization hyperparameter used for each kth intermediate coding step. The correlation between the two views hX and hY is given by(11)ρhX,hY=∑i=1nhxi−hX¯hyi−hY¯∑i=1nhxi−hX¯2∑i=1nhyi−hY¯2,where hX¯ and hY¯ are, respectively, the mean vector of the hidden representation of the two views. hxi and hyi are represented based on a single view and can also be represented as hxi,0 and h0,yi, respectively.Interactive learning CAE usesQ6 to increase the correlation between two views. And use the combined correlation loss calculated in each intermediate hidden layer (the K value is 2 during encoding and 1 during decoding). With the help of the Rmsprop optimizer, the interactive learning CAE is optimized using the following objective function:(12)Qθ=∑i=15Qi−∑j=67Qj,where θ is the interactive learning CAE parameter.


### 1.6. Attention Mechanism

The attention mechanism combines audio features and text features to perform keyword retrieval. The network structure of the attention mechanism is shown in Table1.Table 1
Network structure of attention mechanism.
TypeValueOutputText feature128Repeating vector (1) audio characteristicsRepeats 5050 × 12850 × 256Connection (2)Audio characteristics and (1) output50 × 384LSTM-RNNNumber of neurons 25650 × 256Fully connected layer (3)Number of neurons 25650 × 50Attention mechanism weighted sum(2) and (3) output50 × 384Batch standardization layer50 × 384First, connect the repeated text feature and audio feature to get the output of (2) in Table1. It is represented by F in formula(13)Ct:=∑τ=150Wt,τ⋅Fτ:.First, we repeat the text feature to change it to an appropriate shape so that it can be connected with the audio feature. Then, we pass it through Long Short-Term Memory Recurrent Neural Network (LSTM-RNN) and pass the output of LSTM-RNN through a fully connected layer with softmax as the activation function to obtain a weight matrix ((3) in Table1). It is represented by W in formula (12). Finally, F and W are used as input and input to the weighted sum operation of the attention mechanism. This part is the core part of the attention mechanism. The result of this part is represented by C in formula (13), where a·b represents the scalar a multiplied by each element in the vector b. Specifically, for formula (13), we can make the following explanation. Since the activation function of softmax has passed, the sum of all elements of any row vector of the weight matrix W is equal to 1; we can regard it as a set of weights. Applying this set of weights to the weighted summation of all row vectors of the total feature F can get a vector, which we call the context vector. The same operation is performed for each row vector of W. That is, by performing a weighted summation on the row vectors in F according to this row vector, many context vectors (50 in the model parameters in this article) can be obtained. Finally, all context vectors are put together to form a context matrix C or context feature (Context). The attention mechanism allows us to pay attention to certain special areas when searching for keywords.


### 1.7. Hash Index Strategy

As a compression map, the hash function can be well applied to encryption algorithms. This paper uses hash function to hash the reachable nodes of a single data set of graph-like data as the first level of data index. Let the data setK=η,M1,M2,ξ,β, M1, M2 and β=β1,β2,…,βt be n+1×n+1 dimensional invertible matrix and random number, respectively.The query path label is vectorized, random numbers are introduced in the vector conversion process, and then the query indication vector is calculated. Its expression is(14)D=β+∑k=1nξkγi,j,k,where ξk is a k-dimensional random vector; γi,j,k is the path label value of the graph data vector γi,j.If you want to preserve the similarity of the data, the converted hash code must also have the similarity between the original data. The hash code corresponding to two data is represented byhi and hj. Let sij denote the similarity between two data: sij=1 means the data is similar; sij=0 means the data is not similar. Set two sets S=sij and H=h1,…,hn to be the set of similarity and the set of hash codes respectively.Then, there is a weighted maximum likelihood estimate:(15)logPS|H=∑sij∈SwijlogPsij|hi,hj,where PS|H is the weighted likelihood function and wij is the weight of each training on xi,xj,sij. However, since the similarity can only be 1 or 0, the weight is set. The calculation method is(16)wij=cij⋅SS1,sij=1,SS0,sij=0,where S1=sij∈S:sij=1 and S0=sij∈S:sij=0. In this way, the problem of data imbalance can be solved by weighting the training data pairs. Psij|hi,hj is the conditional probability of the similarity label sij given a pair of hash codes hi and hj. It can be defined as a logical function, as shown in equation(17)Psij|hi,hj=σhi,hjsij=11−σhi,hjsij=0=σhi,hjsij1−σhi,hj1−sij,where σx=1/1+e−αx to control the bandwidth. The smaller the Hamming distance distHhi,hj, the larger the inner product hi,hj and the larger P1|hi,hj. hi and hj should be considered similar. Otherwise, the conditional probability P0|hi,hj will be greater, and hi and hj should be considered dissimilar. By transforming equation (17) into equation (15), the optimization function can be obtained as(18)minΘ∑sij∈Swijlog1+expαhi,hj−αsijhi,hj.where Θ represents the set of all parameters in the deep network. HashNet extends the weighted maximum likelihood estimation from points to pairs for the first time and can accurately generate binary hash codes while storing paired similarity information.


### 1.8. Discriminator

So far, after processing by the attention mechanism, we have obtained the contextual features, and then the discriminator will be used to obtain the confidence score of the final keyword retrieval. Its network structure is shown in Table2. First, global maximum pooling is used to remove the time dimension, the purpose of which is to obtain the most robust output from the context. Then, pass it through a fully connected layer with a sigmoid to obtain a confidence score from 0 to 1.Table 2
The network structure of the discriminator.
TypeValueOutputContext feature50 × 384Global maximum pooling layer384Fully connected layerNumber of neurons 11Activation function = sigmoid


### 1.9. Voice Keyword Retrieval Realization Experiment Scheme

#### 1.9.1. Experimental Setup

The experimental environment of the keyword matching model for speech keyword retrieval with attention mechanism and multimodal information fusion is a GPU server using Ubuntu 14.04 LTS. The specific configuration of its implementation and operation is shown in Table3.Table 3
Experimental hardware environment configuration table.
Device nameItemsConfigurationGPU serverCPU frequency2.8 GHz, 150CoreRAM48 GBVideo RAM48 GB GDDR6Video cardM467Number of video cards4Implementation languagePython3.5Learning libraryTensorFlow1.2.0Since the combination of several loss terms sometimes leads to overfitting and convergence problems, the CAE architecture of interactive learning is used. In the encoding process, convolutional layers and 2 dense neural layers withk = 2 are used. For decoding, the deconvolution layer is used for reconstruction together with a dense layer (k = 1), and the L7 loss term is included in the architecture.At the same time, the proposed method tries to use L1 and L2 regularization to prevent parameter overfitting. Regularization can be independently applied to convolution, deconvolution, and feedforward neural layers, where dropout is used as a regularization technique to prevent overfitting. Randomly selectk neurons from a layer (convolution, deconvolution, or feedforward) and ignore their role in the gradient propagation process.In practice, combining different regularizations proves to be effective. However, a common disadvantage of this method is the increase in the number of hyperparameters. The hyperparameter settings of the convolution model are shown in Table4.Table 4
Hyperparameter settings.
HyperparameterValueConvolution filter300, 150, 50PoolingMaxCommon subspace dimension80OptimizerRmspropDropout0.3When building a Chinese phonetic keyword detection system, the main tools used in this paper are Kaldi and Eesen. Kaldi is an open source speech recognition system toolkit developed by Daniel Povey and others of Microsoft Research. It is developed using C++ under the Apache license. This toolbox can be compiled on windows and unix platforms. Compared with other open source speech recognition tools, Kaldi's technical innovation is mainly manifested in the following aspects: Supports acoustic modeling primitives of any length, training, and decoding based on WFST, including standard machine learning training models based on subspace Gaussian mixture model acoustic model, Gaussian mixture model acoustic model, and DNN. Feature extraction integrates more advanced feature processing and speaker adaptive processing technologies such as VTLN, CMVN, and LDA. The Kaldi toolbox architecture is shown in Figure4.Figure 4
Kaldi open source voice toolbox architecture.The training data used in this experiment is Tsinghua Mandarin 30-hour continuous speech data set THCHS-30. THCHS-30 is an open source Chinese phonetic database provided free of charge in Speech and Language Technology of Tsinghua University. The voice was recorded from 50 speakers with a total of 35 hours of data. The recorded text has a total of 1,000 sentences, which evenly covers common voices in Mandarin. On standard Mandarin pronunciation, recorded by a single carbon microphone in a quiet indoor environment, the sampling rate is 16 kHz, the number of sampling bits is 16 bit, and the storage format is standard wav format. The training set contains 11,296 voices from 35 speakers, and the test set contains 2,135 voices from 10 speakers.


#### 1.9.2. Evaluation Index

The evaluation indicators of the keyword detection system mainly include recall rate, false alarm rate, and sound error rate. The three can be represented by formulas (19)–(21), respectively.(19)α=M2N1,(20)β=M1N2,(21)γ=M2N2,where α is the sound error rate; β is the recall rate; γ is the false alarm rate; M1 represents the number of keywords correctly identified; M2 represents the number of misidentified keywords; N1 represents a summary of detected keywords; N2 represents the total number of referenced keywords.


### 1.10. Simulation Experiment

The accuracy of the network training process varies with the number of iterations as shown in Figure5. After 8 iterations of the training process, the prediction accuracy of the training set stabilized at about 60%.·  ·Figure 5
The accuracy of the network training process varies with the number of iterations.Different amount of training data was used for comparative experiments. For the performance of the algorithm proposed in this paper, the algorithm in [26] and [29] is compared, as well as the performance impact on the Chinese speech keyword detection system.It can be seen from Figure6 that the performance of the three speech recognition systems has been greatly improved with the increase in training data. In addition, the performance improvement trend of these three speech recognition systems is the same. When the training data is insufficient (e.g., 40 hours), the speech recognition performance of the acoustic modeling proposed in [26] is better than the performance of the algorithm proposed in this paper. When relatively sufficient data is provided (e.g., 100 hours), the speech recognition performance based on the algorithm proposed in this paper is about 15% higher than that of [26] and about 17% higher than that of [29].Figure 6
Comparison of sound error rate.It can be seen from Figures7 and 8 that, with the increase of training data, the performance change trends of the two speech keyword detection systems are basically the same. When the training data is insufficient (e.g., 20 hours), the performance of the speech keyword detection system based on the acoustic modeling of literature [26] is better than the performance of the proposed algorithm. When sufficient data is provided (e.g., 200 hours), the recall rate of the proposed algorithm acoustic modeling based voice keyword detection system increases by about 9% compared with [26] and increases by about 13% compared with [29].Figure 7
Comparison of recall rate.Figure 8
Comparison of false alarm rate.


## 1.1. Related Research

Multimodal data retrieval can improve the user retrieval and interactive experience, comprehensively enhance the attractiveness and competitiveness of multimedia information retrieval system, and effectively supplement the traditional single-mode retrieval mode [13–15]. Multimodal information retrieval usually uses a set of semantic vectors (weight of feature vectors) to represent the data of each mode. An effective mapping mechanism is used to map the data of different modes to a common potential feature space. An effective mapping mechanism not only needs to maintain the semantic similarity between the internal data of each mode but also maintains the semantic similarity between different modal data [16–18]. At present, cross-modal retrieval mainly focuses on media data feature learning, cross-modal retrieval model design, and so on. Among them, the advantages and disadvantages of cross-modal retrieval model are one of the key factors restricting the retrieval performance [19]. Reference [20] proposed a text image collaborative attention network model (CoAN), which enhanced the measurement of content similarity by selectively focusing on the key information parts of multimodal data. CoAN uses pretrained VGGNet model and recurrent neural network to extract fine-grained features of images and texts. The text visual attention mechanism is used to capture the subtle interaction between language and vision. At the same time, the model learns the hash representation of text and image, respectively, and improves the retrieval speed by using the low storage characteristics and computational efficiency of hash method.Multimodal retrieval technology promotes the development and application of machine learning, provides theoretical support for the development of new retrieval technology, and has extensive application value and theoretical significance for the research of cross-modal retrieval method [21–24]. With the extensive application of deep learning method in machine learning field, researchers have proposed a series of multimodal information retrieval methods based on deep neural network. For example, [25] proposed a novel word discovery framework using dynamic shape coding to retrieve text in natural scene images and video frames. A novel framework based on end-to-end deep semantic-preserving ordinal hashing (DSPOH) is proposed in [26], which uses neural networks to learn hash functions. In DSPOH, the ordinal representation of relative rank order representing feature size is studied to generate hash codes. Such sequential embedding benefits from the numerical stability of rank correlation measures. In [27], in the construction process of mapping mechanism of multimodal information retrieval, the deep learning method avoids the feature extraction of single modal data and greatly improves the construction speed of the model. However, most of the existing research based on deep neural network is to construct a common subspace for different modes to mine the global correspondence of heterogeneous data. However, the local fine-grained features with semantic identification are ignored [28]. In [29], the author proposed an architecture called DenseNet-BiLSTM for keyword retrieval. However, in their tasks, the keywords allowed to be input are selected from a group of command words, while the application scenarios of our model allow the input of any word or phrase of the language. Most of these methods assume that the semantic data of different modes have the same amount of information. However, different modes often have unbalanced and complementary relations, and the amount of information contained in the description of the same semantics is not the same.In view of the increasingly strong demand for cross-modal retrieval, the existing methods have some problems, such as low retrieval accuracy and unsuitable for high dimensions. A cross-modal speech-text retrieval method based on interactive learning classical convolution automatic encoder (CAE) is proposed. The main contents of this paper are as follows:(1)
Based on an interactive learning convolutional autoencoder structure, including two input of speech and text, as well as encoding, hidden layer interaction, decoding, and other processing links, the modeling of cross-modal speech-text retrieval is completed(2)
Through interactive learning of CAE, the shared characteristics of speech and text modalities are obtained, and they are sent to the modal classifier to identify modal information and realize cross-modal speech-text retrieval


## 1.2. System Architecture

The overall architecture of the proposed method is shown in Figure1. For the input speech samples, the proposed method preprocesses the original audio signal and extracts Mel frequency cepstral coefficients (MFCC) features. For text samples, the bag-of-words model (BoW) is used to extract text features. Then, it combines the features of text and speech through the attention mechanism and then learns the shared features of speech and text modalities through interactive learning CAE. Then, the discriminator will be used to obtain the confidence score of the final keyword search and conduct adversarial training between the generative model and the discriminant model. The generative model embeds semantic label information and promotes the difference between modal-specific features and shared modal features to generate discriminative modal sharing and modal-specific representations. The discriminant model inputs the learned modal shared representations corresponding to the speech and text modalities into the modal classifier to identify modal information and realize cross-modal speech-text retrieval.Figure 1
Overall framework of the proposed method.


## 1.3. Feature Extraction

### 1.3.1. Feature Extraction of Speech Signal

The original voice signal is often difficult to process directly, so for the voice matching model, the feature extraction of the voice signal is required. The feature extraction of the speech signal includes the preprocessing of the original speech signal, and its steps include preemphasis, framing, and windowing operations. The preprocessing of the speech signal is also called front end processing. It processes the original speech before extracting features, so that the processed signal can better meet actual needs. The preemphasis, framing, and windowing operations are used in the preprocessing of speech keyword matching:(1) Pre-emphasis. Pre-emphasis is usually realized with 6 db/octave digital filter frequency characteristics. When the voice signal is affected by the nose or mouth, the high-frequency components will have a greater drop. Therefore, it is absolutely necessary to pass a high-pass filter before preemphasis. The filter can be expressed as shown in the following equation:(1)Hz=1−uz−1,where u is the preemphasis coefficient and the value is 0.9415 according to actual needs.(2) Framing. Framing is to divide the speech signal into several segments and take the signal within a short period of time at the center of each segment as a whole speech unit, and its unit is a frame. In this way, the short-term characteristics of the speech signal at a certain moment can be obtained, and the frequency spectrum can be processed on it, and the continuous changes between frames can be analyzed. The framing of the speech signal can be continuous; that is, there is no overlap between adjacent frames. But more often there is overlap between adjacent frames during framing, and the overlapping part between adjacent frames is called frameshift. Generally, the length of the frameshift is set to be half of the frame length. The time length of each frame is generally between 15 ms and 20 ms as shown in Figure 2.Figure 2
Schematic diagram of voice framing.(3) Windowing. In terms of windowing operation, it is to add a window function to the signal. The value of the window function in a given interval is a real number, and the values outside the interval are all 0. Therefore, the effect of multiplying a signal by a window function is like looking at other functions through this window. Typical window functions include rectangular windows, Gaussian windows, Hamming windows, and so on.Hamming window is used in this paper. The time-domain expression of Hamming window is(2)wn=0.54−0.46cos2πnN−1,where n is time and wn is the window function of the Hamming window.Mel cepstral coefficient is a commonly used feature in the current speech recognition field, and it is also one of the speech recognition features used in this paper. The Mel cepstrum coefficient is the Mel frequency characteristic parameter based on the cepstrum, and the functional relationship between the Mel frequency and the linear frequency is nonlinear. Their functional relationship is as follows:(3)Melf=2595×lg1+f700.The main steps to extract Mel cepstrum coefficients are as follows: (1) The input speech signal is transformed into frequency domain signal by Fourier transform. (2) Perform the filtering operation and set the bandpass filter based on the Mel scale from low to high according to different frequencies. (3) The signals processed by bandpass filter are further processed, such as logarithm operation. (4) Carrying out discrete cosine transform, these processed signal characteristics can be regarded as the final characteristic parameters of the speech signal. Based on the voice features extracted by MFCC, good recognition results will be achieved when applied to a voice matching system.


## 1.3.1. Feature Extraction of Speech Signal

The original voice signal is often difficult to process directly, so for the voice matching model, the feature extraction of the voice signal is required. The feature extraction of the speech signal includes the preprocessing of the original speech signal, and its steps include preemphasis, framing, and windowing operations. The preprocessing of the speech signal is also called front end processing. It processes the original speech before extracting features, so that the processed signal can better meet actual needs. The preemphasis, framing, and windowing operations are used in the preprocessing of speech keyword matching:(1) Pre-emphasis. Pre-emphasis is usually realized with 6 db/octave digital filter frequency characteristics. When the voice signal is affected by the nose or mouth, the high-frequency components will have a greater drop. Therefore, it is absolutely necessary to pass a high-pass filter before preemphasis. The filter can be expressed as shown in the following equation:(1)Hz=1−uz−1,where u is the preemphasis coefficient and the value is 0.9415 according to actual needs.(2) Framing. Framing is to divide the speech signal into several segments and take the signal within a short period of time at the center of each segment as a whole speech unit, and its unit is a frame. In this way, the short-term characteristics of the speech signal at a certain moment can be obtained, and the frequency spectrum can be processed on it, and the continuous changes between frames can be analyzed. The framing of the speech signal can be continuous; that is, there is no overlap between adjacent frames. But more often there is overlap between adjacent frames during framing, and the overlapping part between adjacent frames is called frameshift. Generally, the length of the frameshift is set to be half of the frame length. The time length of each frame is generally between 15 ms and 20 ms as shown in Figure 2.Figure 2
Schematic diagram of voice framing.(3) Windowing. In terms of windowing operation, it is to add a window function to the signal. The value of the window function in a given interval is a real number, and the values outside the interval are all 0. Therefore, the effect of multiplying a signal by a window function is like looking at other functions through this window. Typical window functions include rectangular windows, Gaussian windows, Hamming windows, and so on.Hamming window is used in this paper. The time-domain expression of Hamming window is(2)wn=0.54−0.46cos2πnN−1,where n is time and wn is the window function of the Hamming window.Mel cepstral coefficient is a commonly used feature in the current speech recognition field, and it is also one of the speech recognition features used in this paper. The Mel cepstrum coefficient is the Mel frequency characteristic parameter based on the cepstrum, and the functional relationship between the Mel frequency and the linear frequency is nonlinear. Their functional relationship is as follows:(3)Melf=2595×lg1+f700.The main steps to extract Mel cepstrum coefficients are as follows: (1) The input speech signal is transformed into frequency domain signal by Fourier transform. (2) Perform the filtering operation and set the bandpass filter based on the Mel scale from low to high according to different frequencies. (3) The signals processed by bandpass filter are further processed, such as logarithm operation. (4) Carrying out discrete cosine transform, these processed signal characteristics can be regarded as the final characteristic parameters of the speech signal. Based on the voice features extracted by MFCC, good recognition results will be achieved when applied to a voice matching system.


## 1.4. Text Feature Extraction

The BoW model ignores the grammar, word order, and other elements of the text and treats the text as a collection of several words. The words in the text appear independently, using a set of disordered words to express a text. Complete the extraction of text features according to the statistical information of words in the text.First, two simple text documents are given as follows:Tom likes to go sports. Tim likes tooTom also likes to play video gamesAccording to the words that appear in the document, build the following dictionary:(4)“Tom”:1,“likes”:2,“to”:3,“go”:4,“sports”:5,“also”:6,“vesio”:7,“games”:8,“Tim”:9,“too”:10.The above dictionary contains 10 words; each word has a unique index (label). Then, each text can be represented by a 10-dimensional vector as follows:(5)1,2,1,1,1,0,0,0,1,1,1,1,1,1,0,1,1,1,0,0.The dimension of the vector is the number of words, and the value is the frequency of each word in the text. The above vector can also be represented by a histogram of words. The vocabulary is equivalent to the base of the histogram, and the text to be expressed is mapped to this base. The BoW model is used for text feature extraction, including three steps: vocabulary establishment, sample training, and feature extraction of new samples.


## 1.5. Voice Keyword Search Implementation

### 1.5.1. CAE Based on Interactive Learning

Convolutional autoencoder is proposed on the basis of autoencoder (AE) and denoising autoencoder (DAE). All three are based on unsupervised training to extract features. However, AE and DAE need to pull the input data into a one-dimensional vector in the raster scan order for processing, which will inevitably cause the loss of the two-dimensional structure information of the speech and may introduce redundancy. On the basis of AE and DAE, the convolution and downsampling methods of CNN are introduced, and CAE is proposed. It is a neural network model with a hierarchical structure for two-dimensional data, which coincides with the topological structure of speech. It can directly use voice data as the input of CAE, avoiding the loss of two-dimensional information caused by voice reconstruction.The purpose of CAE is to perform convolutional encoding and decoding on input data. Through convolution and downsampling coding, find out the structure and law of local features in the data and restore the data through reverse decoding. The structure of CAE includes four layers: convolutional layer, downsampling layer, upsampling layer, and deconvolution layer.Suppose the input data isX=X1,X2,…,Xn, the filter map is defined as F1=F1,F2,…,Fm, and each depth is D. The convolution operation obtains m activation maps with depth m, as follows:(6)fX=α∑d=1D∑u=−2k−12k+1∑v=−2k−12k+1FPd1u,vXdi−u,j−v,where α is nonlinear and p∈1,…,m.These activation maps are low-dimensional coded representations of the input dataX, which are downsampled using the maximum pooling operation. The proposed method transmits this activation feature map through two hidden neural layers and then projects it into a d-dimensional common subspace.In order to project the data back into the input space, deconvolution is used as a decoding step. This is achieved by padding the zero sequences to the input before the convolution step. The number of filtersFm2 used is N and the size is 2k+1,2k+1,n. The reconstruction data are given by the following formula:(7)X˜=αfX∗Fm2.Finally, CAE performs input data reconstruction by minimizing the loss between input and reconstruction data.(8)LX,X˜=12X−X˜22,where LX,X˜ is mean square error (MSE). Because MSE captures potential comparisons by performing point or pixel comparisons, it is optimal in defining the loss term.In order to obtain a common representation of the two views of speech and text, CAE based on interactive learning is used, which can be extended to multiple hidden layers. Its structure is shown in Figure3. Multiple hidden layers pass data through more nonlinearities, ensuring better interaction between the two views of data.Figure 3
CAE structure of interactive learning.In order to train the proposed model, the objective function set is as follows: minimize the self-reconstruction error; minimize the cross-reconstruction error of each intermediate step; maximize the correlation between the hidden representations of the two views in each coding step.Given input aszi=xi;yi, where zi is the associated representation of input views xi and yi, self-reconstruction loss and cross-reconstruction loss are defined as(9)Q1=∑i=0NLzi,ghzi,Q2=∑i=0NLzi,ghxi,Q3=∑i=0NLzi,ghyi,Q4=∑k=0K∑i=0NLhxik,hyikQ5=∑i=0NLghxi,ghyi,where g,h are the nonlinearity generally regarded as ReLU, ghxik and ghyik are the hidden representation of the kth intermediate hidden layer, S is the mean square error function, and Q1 is the step length calculation. In losses Q2 and Q3 (for cross reconstruction), the 0 vectors are used instead of another view to calculate xi and yi. The reason for introducing step calculation is the similarity of hidden representations. Therefore, reconstructing one hidden representation from another hidden representation helps the model finally reconstruct the view and improve the interaction between layers. In order to maximize the interaction between the two views, add related losses as follows:(10)Q6=λcorrhX,hY,Q7=∑k=0KλkcorrhXk,hYk,where hX and hY are the projection of the combined model. X and Y are the representation of the input view obtained after convolution and pooling layers. λk is the relative regularization hyperparameter used for each kth intermediate coding step. The correlation between the two views hX and hY is given by(11)ρhX,hY=∑i=1nhxi−hX¯hyi−hY¯∑i=1nhxi−hX¯2∑i=1nhyi−hY¯2,where hX¯ and hY¯ are, respectively, the mean vector of the hidden representation of the two views. hxi and hyi are represented based on a single view and can also be represented as hxi,0 and h0,yi, respectively.Interactive learning CAE usesQ6 to increase the correlation between two views. And use the combined correlation loss calculated in each intermediate hidden layer (the K value is 2 during encoding and 1 during decoding). With the help of the Rmsprop optimizer, the interactive learning CAE is optimized using the following objective function:(12)Qθ=∑i=15Qi−∑j=67Qj,where θ is the interactive learning CAE parameter.


## 1.5.1. CAE Based on Interactive Learning

Convolutional autoencoder is proposed on the basis of autoencoder (AE) and denoising autoencoder (DAE). All three are based on unsupervised training to extract features. However, AE and DAE need to pull the input data into a one-dimensional vector in the raster scan order for processing, which will inevitably cause the loss of the two-dimensional structure information of the speech and may introduce redundancy. On the basis of AE and DAE, the convolution and downsampling methods of CNN are introduced, and CAE is proposed. It is a neural network model with a hierarchical structure for two-dimensional data, which coincides with the topological structure of speech. It can directly use voice data as the input of CAE, avoiding the loss of two-dimensional information caused by voice reconstruction.The purpose of CAE is to perform convolutional encoding and decoding on input data. Through convolution and downsampling coding, find out the structure and law of local features in the data and restore the data through reverse decoding. The structure of CAE includes four layers: convolutional layer, downsampling layer, upsampling layer, and deconvolution layer.Suppose the input data isX=X1,X2,…,Xn, the filter map is defined as F1=F1,F2,…,Fm, and each depth is D. The convolution operation obtains m activation maps with depth m, as follows:(6)fX=α∑d=1D∑u=−2k−12k+1∑v=−2k−12k+1FPd1u,vXdi−u,j−v,where α is nonlinear and p∈1,…,m.These activation maps are low-dimensional coded representations of the input dataX, which are downsampled using the maximum pooling operation. The proposed method transmits this activation feature map through two hidden neural layers and then projects it into a d-dimensional common subspace.In order to project the data back into the input space, deconvolution is used as a decoding step. This is achieved by padding the zero sequences to the input before the convolution step. The number of filtersFm2 used is N and the size is 2k+1,2k+1,n. The reconstruction data are given by the following formula:(7)X˜=αfX∗Fm2.Finally, CAE performs input data reconstruction by minimizing the loss between input and reconstruction data.(8)LX,X˜=12X−X˜22,where LX,X˜ is mean square error (MSE). Because MSE captures potential comparisons by performing point or pixel comparisons, it is optimal in defining the loss term.In order to obtain a common representation of the two views of speech and text, CAE based on interactive learning is used, which can be extended to multiple hidden layers. Its structure is shown in Figure3. Multiple hidden layers pass data through more nonlinearities, ensuring better interaction between the two views of data.Figure 3
CAE structure of interactive learning.In order to train the proposed model, the objective function set is as follows: minimize the self-reconstruction error; minimize the cross-reconstruction error of each intermediate step; maximize the correlation between the hidden representations of the two views in each coding step.Given input aszi=xi;yi, where zi is the associated representation of input views xi and yi, self-reconstruction loss and cross-reconstruction loss are defined as(9)Q1=∑i=0NLzi,ghzi,Q2=∑i=0NLzi,ghxi,Q3=∑i=0NLzi,ghyi,Q4=∑k=0K∑i=0NLhxik,hyikQ5=∑i=0NLghxi,ghyi,where g,h are the nonlinearity generally regarded as ReLU, ghxik and ghyik are the hidden representation of the kth intermediate hidden layer, S is the mean square error function, and Q1 is the step length calculation. In losses Q2 and Q3 (for cross reconstruction), the 0 vectors are used instead of another view to calculate xi and yi. The reason for introducing step calculation is the similarity of hidden representations. Therefore, reconstructing one hidden representation from another hidden representation helps the model finally reconstruct the view and improve the interaction between layers. In order to maximize the interaction between the two views, add related losses as follows:(10)Q6=λcorrhX,hY,Q7=∑k=0KλkcorrhXk,hYk,where hX and hY are the projection of the combined model. X and Y are the representation of the input view obtained after convolution and pooling layers. λk is the relative regularization hyperparameter used for each kth intermediate coding step. The correlation between the two views hX and hY is given by(11)ρhX,hY=∑i=1nhxi−hX¯hyi−hY¯∑i=1nhxi−hX¯2∑i=1nhyi−hY¯2,where hX¯ and hY¯ are, respectively, the mean vector of the hidden representation of the two views. hxi and hyi are represented based on a single view and can also be represented as hxi,0 and h0,yi, respectively.Interactive learning CAE usesQ6 to increase the correlation between two views. And use the combined correlation loss calculated in each intermediate hidden layer (the K value is 2 during encoding and 1 during decoding). With the help of the Rmsprop optimizer, the interactive learning CAE is optimized using the following objective function:(12)Qθ=∑i=15Qi−∑j=67Qj,where θ is the interactive learning CAE parameter.


## 1.6. Attention Mechanism

The attention mechanism combines audio features and text features to perform keyword retrieval. The network structure of the attention mechanism is shown in Table1.Table 1
Network structure of attention mechanism.
TypeValueOutputText feature128Repeating vector (1) audio characteristicsRepeats 5050 × 12850 × 256Connection (2)Audio characteristics and (1) output50 × 384LSTM-RNNNumber of neurons 25650 × 256Fully connected layer (3)Number of neurons 25650 × 50Attention mechanism weighted sum(2) and (3) output50 × 384Batch standardization layer50 × 384First, connect the repeated text feature and audio feature to get the output of (2) in Table1. It is represented by F in formula(13)Ct:=∑τ=150Wt,τ⋅Fτ:.First, we repeat the text feature to change it to an appropriate shape so that it can be connected with the audio feature. Then, we pass it through Long Short-Term Memory Recurrent Neural Network (LSTM-RNN) and pass the output of LSTM-RNN through a fully connected layer with softmax as the activation function to obtain a weight matrix ((3) in Table1). It is represented by W in formula (12). Finally, F and W are used as input and input to the weighted sum operation of the attention mechanism. This part is the core part of the attention mechanism. The result of this part is represented by C in formula (13), where a·b represents the scalar a multiplied by each element in the vector b. Specifically, for formula (13), we can make the following explanation. Since the activation function of softmax has passed, the sum of all elements of any row vector of the weight matrix W is equal to 1; we can regard it as a set of weights. Applying this set of weights to the weighted summation of all row vectors of the total feature F can get a vector, which we call the context vector. The same operation is performed for each row vector of W. That is, by performing a weighted summation on the row vectors in F according to this row vector, many context vectors (50 in the model parameters in this article) can be obtained. Finally, all context vectors are put together to form a context matrix C or context feature (Context). The attention mechanism allows us to pay attention to certain special areas when searching for keywords.


## 1.7. Hash Index Strategy

As a compression map, the hash function can be well applied to encryption algorithms. This paper uses hash function to hash the reachable nodes of a single data set of graph-like data as the first level of data index. Let the data setK=η,M1,M2,ξ,β, M1, M2 and β=β1,β2,…,βt be n+1×n+1 dimensional invertible matrix and random number, respectively.The query path label is vectorized, random numbers are introduced in the vector conversion process, and then the query indication vector is calculated. Its expression is(14)D=β+∑k=1nξkγi,j,k,where ξk is a k-dimensional random vector; γi,j,k is the path label value of the graph data vector γi,j.If you want to preserve the similarity of the data, the converted hash code must also have the similarity between the original data. The hash code corresponding to two data is represented byhi and hj. Let sij denote the similarity between two data: sij=1 means the data is similar; sij=0 means the data is not similar. Set two sets S=sij and H=h1,…,hn to be the set of similarity and the set of hash codes respectively.Then, there is a weighted maximum likelihood estimate:(15)logPS|H=∑sij∈SwijlogPsij|hi,hj,where PS|H is the weighted likelihood function and wij is the weight of each training on xi,xj,sij. However, since the similarity can only be 1 or 0, the weight is set. The calculation method is(16)wij=cij⋅SS1,sij=1,SS0,sij=0,where S1=sij∈S:sij=1 and S0=sij∈S:sij=0. In this way, the problem of data imbalance can be solved by weighting the training data pairs. Psij|hi,hj is the conditional probability of the similarity label sij given a pair of hash codes hi and hj. It can be defined as a logical function, as shown in equation(17)Psij|hi,hj=σhi,hjsij=11−σhi,hjsij=0=σhi,hjsij1−σhi,hj1−sij,where σx=1/1+e−αx to control the bandwidth. The smaller the Hamming distance distHhi,hj, the larger the inner product hi,hj and the larger P1|hi,hj. hi and hj should be considered similar. Otherwise, the conditional probability P0|hi,hj will be greater, and hi and hj should be considered dissimilar. By transforming equation (17) into equation (15), the optimization function can be obtained as(18)minΘ∑sij∈Swijlog1+expαhi,hj−αsijhi,hj.where Θ represents the set of all parameters in the deep network. HashNet extends the weighted maximum likelihood estimation from points to pairs for the first time and can accurately generate binary hash codes while storing paired similarity information.


## 1.8. Discriminator

So far, after processing by the attention mechanism, we have obtained the contextual features, and then the discriminator will be used to obtain the confidence score of the final keyword retrieval. Its network structure is shown in Table2. First, global maximum pooling is used to remove the time dimension, the purpose of which is to obtain the most robust output from the context. Then, pass it through a fully connected layer with a sigmoid to obtain a confidence score from 0 to 1.Table 2
The network structure of the discriminator.
TypeValueOutputContext feature50 × 384Global maximum pooling layer384Fully connected layerNumber of neurons 11Activation function = sigmoid


## 1.9. Voice Keyword Retrieval Realization Experiment Scheme

### 1.9.1. Experimental Setup

The experimental environment of the keyword matching model for speech keyword retrieval with attention mechanism and multimodal information fusion is a GPU server using Ubuntu 14.04 LTS. The specific configuration of its implementation and operation is shown in Table3.Table 3
Experimental hardware environment configuration table.
Device nameItemsConfigurationGPU serverCPU frequency2.8 GHz, 150CoreRAM48 GBVideo RAM48 GB GDDR6Video cardM467Number of video cards4Implementation languagePython3.5Learning libraryTensorFlow1.2.0Since the combination of several loss terms sometimes leads to overfitting and convergence problems, the CAE architecture of interactive learning is used. In the encoding process, convolutional layers and 2 dense neural layers withk = 2 are used. For decoding, the deconvolution layer is used for reconstruction together with a dense layer (k = 1), and the L7 loss term is included in the architecture.At the same time, the proposed method tries to use L1 and L2 regularization to prevent parameter overfitting. Regularization can be independently applied to convolution, deconvolution, and feedforward neural layers, where dropout is used as a regularization technique to prevent overfitting. Randomly selectk neurons from a layer (convolution, deconvolution, or feedforward) and ignore their role in the gradient propagation process.In practice, combining different regularizations proves to be effective. However, a common disadvantage of this method is the increase in the number of hyperparameters. The hyperparameter settings of the convolution model are shown in Table4.Table 4
Hyperparameter settings.
HyperparameterValueConvolution filter300, 150, 50PoolingMaxCommon subspace dimension80OptimizerRmspropDropout0.3When building a Chinese phonetic keyword detection system, the main tools used in this paper are Kaldi and Eesen. Kaldi is an open source speech recognition system toolkit developed by Daniel Povey and others of Microsoft Research. It is developed using C++ under the Apache license. This toolbox can be compiled on windows and unix platforms. Compared with other open source speech recognition tools, Kaldi's technical innovation is mainly manifested in the following aspects: Supports acoustic modeling primitives of any length, training, and decoding based on WFST, including standard machine learning training models based on subspace Gaussian mixture model acoustic model, Gaussian mixture model acoustic model, and DNN. Feature extraction integrates more advanced feature processing and speaker adaptive processing technologies such as VTLN, CMVN, and LDA. The Kaldi toolbox architecture is shown in Figure4.Figure 4
Kaldi open source voice toolbox architecture.The training data used in this experiment is Tsinghua Mandarin 30-hour continuous speech data set THCHS-30. THCHS-30 is an open source Chinese phonetic database provided free of charge in Speech and Language Technology of Tsinghua University. The voice was recorded from 50 speakers with a total of 35 hours of data. The recorded text has a total of 1,000 sentences, which evenly covers common voices in Mandarin. On standard Mandarin pronunciation, recorded by a single carbon microphone in a quiet indoor environment, the sampling rate is 16 kHz, the number of sampling bits is 16 bit, and the storage format is standard wav format. The training set contains 11,296 voices from 35 speakers, and the test set contains 2,135 voices from 10 speakers.


### 1.9.2. Evaluation Index

The evaluation indicators of the keyword detection system mainly include recall rate, false alarm rate, and sound error rate. The three can be represented by formulas (19)–(21), respectively.(19)α=M2N1,(20)β=M1N2,(21)γ=M2N2,where α is the sound error rate; β is the recall rate; γ is the false alarm rate; M1 represents the number of keywords correctly identified; M2 represents the number of misidentified keywords; N1 represents a summary of detected keywords; N2 represents the total number of referenced keywords.


## 1.9.1. Experimental Setup

The experimental environment of the keyword matching model for speech keyword retrieval with attention mechanism and multimodal information fusion is a GPU server using Ubuntu 14.04 LTS. The specific configuration of its implementation and operation is shown in Table3.Table 3
Experimental hardware environment configuration table.
Device nameItemsConfigurationGPU serverCPU frequency2.8 GHz, 150CoreRAM48 GBVideo RAM48 GB GDDR6Video cardM467Number of video cards4Implementation languagePython3.5Learning libraryTensorFlow1.2.0Since the combination of several loss terms sometimes leads to overfitting and convergence problems, the CAE architecture of interactive learning is used. In the encoding process, convolutional layers and 2 dense neural layers withk = 2 are used. For decoding, the deconvolution layer is used for reconstruction together with a dense layer (k = 1), and the L7 loss term is included in the architecture.At the same time, the proposed method tries to use L1 and L2 regularization to prevent parameter overfitting. Regularization can be independently applied to convolution, deconvolution, and feedforward neural layers, where dropout is used as a regularization technique to prevent overfitting. Randomly selectk neurons from a layer (convolution, deconvolution, or feedforward) and ignore their role in the gradient propagation process.In practice, combining different regularizations proves to be effective. However, a common disadvantage of this method is the increase in the number of hyperparameters. The hyperparameter settings of the convolution model are shown in Table4.Table 4
Hyperparameter settings.
HyperparameterValueConvolution filter300, 150, 50PoolingMaxCommon subspace dimension80OptimizerRmspropDropout0.3When building a Chinese phonetic keyword detection system, the main tools used in this paper are Kaldi and Eesen. Kaldi is an open source speech recognition system toolkit developed by Daniel Povey and others of Microsoft Research. It is developed using C++ under the Apache license. This toolbox can be compiled on windows and unix platforms. Compared with other open source speech recognition tools, Kaldi's technical innovation is mainly manifested in the following aspects: Supports acoustic modeling primitives of any length, training, and decoding based on WFST, including standard machine learning training models based on subspace Gaussian mixture model acoustic model, Gaussian mixture model acoustic model, and DNN. Feature extraction integrates more advanced feature processing and speaker adaptive processing technologies such as VTLN, CMVN, and LDA. The Kaldi toolbox architecture is shown in Figure4.Figure 4
Kaldi open source voice toolbox architecture.The training data used in this experiment is Tsinghua Mandarin 30-hour continuous speech data set THCHS-30. THCHS-30 is an open source Chinese phonetic database provided free of charge in Speech and Language Technology of Tsinghua University. The voice was recorded from 50 speakers with a total of 35 hours of data. The recorded text has a total of 1,000 sentences, which evenly covers common voices in Mandarin. On standard Mandarin pronunciation, recorded by a single carbon microphone in a quiet indoor environment, the sampling rate is 16 kHz, the number of sampling bits is 16 bit, and the storage format is standard wav format. The training set contains 11,296 voices from 35 speakers, and the test set contains 2,135 voices from 10 speakers.


## 1.9.2. Evaluation Index

The evaluation indicators of the keyword detection system mainly include recall rate, false alarm rate, and sound error rate. The three can be represented by formulas (19)–(21), respectively.(19)α=M2N1,(20)β=M1N2,(21)γ=M2N2,where α is the sound error rate; β is the recall rate; γ is the false alarm rate; M1 represents the number of keywords correctly identified; M2 represents the number of misidentified keywords; N1 represents a summary of detected keywords; N2 represents the total number of referenced keywords.


## 1.10. Simulation Experiment

The accuracy of the network training process varies with the number of iterations as shown in Figure5. After 8 iterations of the training process, the prediction accuracy of the training set stabilized at about 60%.·  ·Figure 5
The accuracy of the network training process varies with the number of iterations.Different amount of training data was used for comparative experiments. For the performance of the algorithm proposed in this paper, the algorithm in [26] and [29] is compared, as well as the performance impact on the Chinese speech keyword detection system.It can be seen from Figure6 that the performance of the three speech recognition systems has been greatly improved with the increase in training data. In addition, the performance improvement trend of these three speech recognition systems is the same. When the training data is insufficient (e.g., 40 hours), the speech recognition performance of the acoustic modeling proposed in [26] is better than the performance of the algorithm proposed in this paper. When relatively sufficient data is provided (e.g., 100 hours), the speech recognition performance based on the algorithm proposed in this paper is about 15% higher than that of [26] and about 17% higher than that of [29].Figure 6
Comparison of sound error rate.It can be seen from Figures7 and 8 that, with the increase of training data, the performance change trends of the two speech keyword detection systems are basically the same. When the training data is insufficient (e.g., 20 hours), the performance of the speech keyword detection system based on the acoustic modeling of literature [26] is better than the performance of the proposed algorithm. When sufficient data is provided (e.g., 200 hours), the recall rate of the proposed algorithm acoustic modeling based voice keyword detection system increases by about 9% compared with [26] and increases by about 13% compared with [29].Figure 7
Comparison of recall rate.Figure 8
Comparison of false alarm rate.


## 2. Conclusion

This paper proposes a voice keyword retrieval method using attention mechanism and multimodal information fusion. This method extracts speech MFCC features and uses bag-of-words model to extract text features. The attention mechanism combines text and audio features and outputs context features. It will give higher weight to time positions that may contain keywords. Finally, the discriminator processes the context features output by the attention mechanism and gives a confidence score. The modal classifier is used to identify modal information and realize cross-modal speech-text retrieval. The final experiment shows that the algorithm proposed in this paper has better performance. At present, the relevant technical theory of Chinese phonetic keyword detection has been relatively mature and is gradually developing in the direction of commercial use.Next, we will mainly study the robustness of voice keyword detection in a complex background environment. It can be considered to extract features with better noise resistance, such as the average Hilbert envelope coefficient, features that are better against echo, such as frequency domain linear prediction. Acoustic features such as the characteristic fundamental frequency change feature of MFCC and prosody are integrated.



---

*Source: 6662841-2021-01-25.xml*