# Explore Long-Range Context Features for Speaker Verification
## 4 Depthwise Separable Self-Attention Module  
The attention mechanism is generally used to model the pairwise relations from the global perspective. In this section, we introduce our proposed method, the DSSA module. Scaled Dot-Product Attention, a core module of the famous Transformer architecture, is  
Figure 3: Detail structure of HS-ResNet50 and Depthwise Separable Self-Attention module for the combined system.  
also the core of our module. Scaled Dot-Product Attention consists of three inputs: queries, keys, and values. The output of each query will be calculated by the dot products with outputs of all keys; the results are divided by \(\sqrt{d_{k}}\) and applied a softmax function to obtain the weights of the values' output. Here, \(d_{k}\) means the dimension of keys, and \(d_{q}\) and \(d_{v}\) are similar. In practice, the above calculations are implemented in the form of matrices. The outputs are computed as  
\[Attention(Q,K,V)=softmax(QK^{T}/\sqrt{d_{k}})V \tag{5}\]  
In the equation, \(d_{q}=d_{k}=d_{v}=d_{in}=d\), the projection matrices \(W^{Q},W^{K},W^{V}\in\mathbb{R}^{d\times d}\). \(d_{in}\) means the dimension of the frequency axes of the input feature map.  
Here are a few challenges when using Multi-Head Scaled Dot-Product Attention in CNN structures, especially the parameters. The depthwise separable convolution strategy, which decomposes ordinary convolutions into depthwise convolution and pointwise convolution, is introduced in the module to avoid the massive parameters.  
In the DSSA module, the depthwise separable convolution applies a Scaled Dot-Product Attention to each input channel. As decipted in Figure 3c, the input feature maps \(F\) are split into several groups, and the number of groups is equal to the number of channels \(C\) in the feature maps, \(F_{i},i=1,...,C,F\in\mathbb{R}^{C\times T\times W}\). That is, each group only contains one channel of the feature maps. Then, each group is fed into three independent 1D-convolution to generate the queries, keys, and values, \(Q_{i},K_{i},V_{i}\). The dimension of these is the same as the input. Specifically, the DSSA module inputs a \(C\times T\times W\) feature map and produces three independent \(C\times T\times W\) feature maps after the convolution layers.  
\[Q_{i},K_{i},V_{i}=Conv1d_{Q,K,V}(F_{i}) \tag{6}\]  
After that, the dot products between each query and each key are calculated and divided by \(\sqrt{d_{k}}\). Before applying a softmax function to obtain the weights on the values', another square operation acts on the weights, which is to maintain weights in the normal range and avoid it being too big or too small.  
\[\begin{split} D_{i}&=\sqrt{Q_{i}K_{i}^{T}/\sqrt{d_ {k}}}\\ Y_{i}&=Softmax(D_{i})V_{i}\end{split} \tag{7}\]  
We perform the layer-norm operation on output of each group \(Y_{i}\) in parallel. Finally, the outputs of groups are concatenated to get \(y\).  
\[y=LayerNorm(x+Y) \tag{8}\]  
In addition, to facilitate the processing of speech with a long duration, the explicit sparse attention mechanism is designed in this module. The explicit sparse attention only pays attention to the nearest k frames. The attention weights degenerate to the sparse attention through k-nearest selection. The weights of k nearest frames of each row in the attention weight matrix are selected, and the others are replaced with \(-\infty\).  
\[\mathcal{M}(D_{i},k)_{mn}=\left\{\begin{array}{ll}D_{i,mn},&if\ m\geq n-k/2 \ \ \&\ m\leq n+k/2\\ -\infty,&else\end{array}\right. \tag{9}\]  
where \(D_{i,mn}\) means the value of row \(m\) and column \(n\) in \(i\) channel.