# Global-Local Self-Attention Based Transformer for Speaker Verification
## 3 Proposed Architecture
### Transformer Block
#### 3.2.1 Global-Local Self-Attention  
Despite its strong ability to model global dependencies, the original full self-attention mechanism struggles to capture the local information for utterances longer than text. In a recent study [12], local self-attention with a sliding window was applied to speaker recognition and achieved competitive performance. Inspired by [23], we proposed a novel global-local self-attention mechanism that improves the capability to capture local features while retaining the capability to model long-distance dependencies. As shown in Figure 2b, for half of the channels of the feature maps, the self-attention mechanism is implemented as local self-attention, and each attention head has a sliding window of the same size, while for the other half of the channel, it is implemented as global self-attention without a sliding window. Similar to the original full self-attention mechanism, the input features \(X\in R^{C\times T}\) are linearly transformed to \(K\) attention heads, and then each attention head will perform local or global self-attention. For global-local self-attention, we a used a non-overlapping sliding window to partition \(X\) into \([X_{1},\cdots,X_{N}]\) of an equal window size \(w\). \(w\) is the size of the window, which gives the model better learning ability. Assuming that the matrix queries, keys, and values of the \(k\)th attention head all have the dimension \(d_{k}\), then the proposed local self-attention mechanism for the \(k\)th head is defined as:  
\[X=[X_{1},X_{2},\cdots,X_{N}],N=\frac{T}{w}, \tag{3}\]  
Figure 2: (**a**) The illustration of our proposed Transformer block; (**b**) the illustration of the global–local self-attention.  
\[\gamma_{i}^{k}=Attention\left(X_{i}W_{k}^{Q},X_{i}W_{k}^{K},X_{i}W_{k}^{V}\right), \tag{4}\]  
\[Local-Attention_{k}(X)=\left[\gamma_{1}^{k},\gamma_{2}^{k},\cdots,\gamma_{i}^{k}\right] \tag{5}\]  
where \(W_{k}^{Q},W_{k}^{K},W_{k}^{V}\in R^{C\times d_{k}}\), are the linear projection parameter matrices of the queries, keys, and values for the \(k\)th attention head, respectively, and \(d_{k}\) is set as \(C/K\). We divided the \(K\) attention heads equally into two distinct groups. \(K\) is usually an even number so that the attention heads are evenly divided into two groups. The first group of attention heads performs local self-attention, while the second group of attention heads performs global self-attention. The difference between the calculation method of global self-attention and multi-head self-attention is that the former needs to add locally enhanced positional encoding, and the output of the \(k\)th head is recorded as \(Global-Attention_{k}(X)\).  
\[head_{k}=\left\{\begin{array}{l}Local-Attention_{k}(X),k=1,\cdots K/2\\ Global-Attention_{k}(X),k=K/2+1,\cdots,K\end{array}\right. \tag{6}\]  
Finally, the results of these two kinds of attention are concatenated together as the input of MLP and denoted as \(GL-Attention(X)\):  
\[GL-Attention(X)=cat(head_{1},\cdots,head_{k})W \tag{7}\]  
where \(W\in R^{C\times C}\) is the projection matrix that projects the self-attention results into the target output dimension. The key design is to split the attention heads into two different groups and perform local and global self-attention operations in parallel. This enables local attention to function under the guidance of global attention so that global information can better interact with local information. We adopted an optimal window size that outperformed other window sizes to achieve the best performance for our proposed method.