## 3 Systems
### Query-by-Example Spoken Term Detection
#### 3.2.1 End-to-End with Language Adaptation System (E2E + LA)  
This system is based on attentive pooling networks [136], which were proposed in the context of question answering in NLP. These networks use a two-way attention mechanism able to compare a query (question) and a document (answer) of different lengths by focusing on the most relevant parts of both the query and the document. This approach was recently proposed for the QbE STD task [137]. The system developed is based on this work and has the structure depicted in Figure 2.  
First, the WebRTC VAD is employed to remove silence and random noise effects that may appear at the beginning and end of the audio queries. An LSTM network is used to obtain embedding representations of both the acoustic queries and the audio segments in which the utterances are divided. To do so, for a spoken query with acoustic features given by \(Q=\{q_{1},q_{2},\ldots,q_{M}\}\), where \(M\) is the number of frames of the query, the shared recurrent neural networks (RNNs) consisting of LSTM units project the query into a hidden state sequence \(H_{Q}=\{h_{1}^{Q},h_{2}^{Q},\ldots,h_{M}^{Q}\}\), where \(h_{M}^{Q}\) contains information of the whole query. In the same way, the audio segment with acoustic features given by \(S=\{s_{1},s_{2},\ldots,s_{N}\}\), where \(N\) is the number of frames of the audio segment, is encoded into the corresponding hidden state sequence \(H_{S}=\{h_{1}^{S},h_{2}^{S},\ldots,h_{N}^{S}\}\), hence the query and utterance representations convey the same feature space. Then, the attention matrix, \(G\), is computed as shown in Equation (5) so that the query and audio segment hidden state vector sequences can then be compared:  
\[G=\tanh(H_{Q}^{T}UH_{S}). \tag{5}\]  
Matrix \(U\) in Equation (5) is a measure of the \(H_{Q}\) and \(H_{S}\) representations and is learned during training. To build a more symmetric system, the measure matrix, \(U\), is defined to be symmetric (i.e., \(U=U^{T}\)), which allows it to exchange query and audio segment representations, as given in Equation (6):  
\[H_{Q}^{T}UH_{S}=H_{S}^{T}U^{T}H_{Q}=H_{S}^{T}UH_{Q}. \tag{6}\]  
Figure 2: Attentive pooling network structure for the E2E + LA QE STD system. Matrix \(U\) is initialised with random normal samples with zero mean and \(10^{-4}\) variance. Then, matrix \(G\) can be seen as a soft alignment score between each query and audio segment frame. The next step is to apply column-wise and row-wise max-pooling to matrix \(G\) to generate the corresponding weight vectors \(g^{Q}\in\mathbb{R}^{M}\) and \(g^{S}\in\mathbb{R}^{N}\), which are computed as in Equations (7) and (8), respectively:  
\[[g^{Q}]_{j}=\max_{1\leq i\leq N}[G_{j,i}] \tag{7}\]  
\[[g^{S}]_{i}=\max_{1\leq j\leq M}[G_{j,i}]. \tag{8}\]  
The \(j\)-th element in \(g^{Q}\) represents the weight applied to the \(j\)-th frame in the query \(Q\). This results in an estimation of which frames of the queries are actually relevant for query/audio segment matching. Since \(g^{Q}\) and \(g^{S}\) are weight vectors, these are then normalized with a softmax function, whose output represents the attention weight vectors denoted as \(\sigma^{Q}\) and \(\sigma^{S}\) in Figure 2.  
Finally, the dot product between the hidden states representing both the query and the audio segments in the utterance, and the attention vectors \(\sigma^{Q}\) and \(\sigma^{S}\), is employed to enhance the meaningful information. To do so, \(V_{Q}\) and \(V_{S}\) are computed as in Equation (9):  
\[V_{Q}=H_{Q}\sigma^{Q},V_{S}=H_{S}\sigma^{S}. \tag{9}\]  
The whole system is trained using a large margin cost function (i.e., hinge loss), whose aim is maximising the inter-class distance and minimising the intra-class distance. To do so, the training set comprises 3-element groups: a spoken query \(Q=\{q_{1},q_{2},\ldots,q_{M}\}\), a positive audio segment \(S^{(P)}=\{s_{1}^{(P)},s_{2}^{(P)},\ldots,s_{N}^{(P)}\}\) and a negative audio segment \(S^{(N)}=\{s_{1}^{(N)},s_{2}^{(N)},\ldots,s_{N}^{(N)}\}\).