# Self-Attention Network for Human Pose Estimation
## 3 Model Architecture
### Self-Attention Network Design  
When observing batches of input images including humans, we find that the relationships and constraints between joints will produce more useful information. Many human pose estimation methods use convolution neural networks (CNNs), the performance of which is limited by valid receptive field such that they are only capable of adjacent content in feature maps and cannot process long-range relations and grasp high-level semantic information. To compensate for this drawback, we propose a nonlocal approach called a Self-Attention Network (SAN). SANs not only receive efficient features in a local region, but also perceive contextual information over a wide range. The details are shown in Figure 2.  
Feature maps from the previous hidden layer \(\mathbf{X}\in\mathbb{R}^{C\times B\times H\times W}\) (\(C\) is channel number, \(B\) is batch size, \(H\times W\) is the pixel number) are first transformed to three feature spaces, where:  
\[W_{q}^{t}=\beta_{q}\odot x_{t},W_{k}^{t}=\beta_{k}\odot x_{t},W_{v}^{t}=\beta _{v}\odot x_{t} \tag{1}\]  
\(t\) indicates the target feature maps index. All three space vectors come from the same input. \(W_{q}^{t}\) is a query space vector. \(W_{k}^{t}\) is a key space vector. \(W_{q}^{t}\) and \(W_{k}^{t}\) are used to calculate weights which represent the similarity features between feature maps. \(W_{v}^{t}\) is a value space vector and is an output from original feature maps. Reweighting the long-term information on \(W_{v}^{t}\) enables the network to capture joint relationships easily. \(\beta_{q}\) is a weight matrix of the query space vector, which maps the input matrix of \(B\times C\times W\times H\) dimensions to \(B\times\frac{C}{8}\times W\times H\) dimensions. \(\beta_{q}\) needs to be transposed for the following operations. \(\beta_{k}\) is a weight matrix of the key space vector, which maps the input matrix of \(B\times C\times W\times H\) dimensions to \(B\times\frac{C}{8}\times W\times H\) dimensions. \(\beta_{v}\) is a weight matrix of the value space vector, which maps the input matrix of \(B\times C\times W\times H\) dimensions to \(B\times C\times W\times H\) dimensions. \(\beta_{q}\), \(\beta_{k}\) and \(\beta_{v}\) are all trainable weight matrixes that transform feature maps to corresponding vector spaces and were implemented as \(1\times 1\) convolutions in our experiment. Figure 1: Framework: Illustration of the proposed approach. The basic structure contains three parts. (**a**) ResNet backbone which is used to extract features from images. (**b**) Self-Attention Network (SAN), which learns long-range dependency to compensate the lost features from original images and reweight obtained feature maps. With increasing \(\delta\), the model will depend more on nonlocal information than local content. (**c**) An upsampling block to regress the feature maps to higher resolutions to refine joint locations. \(W_{q}^{t}\) as a search vector of feature maps for one image \(\mathrm{x}\in\mathbb{R}^{\mathbb{C}\times B\times H\times W}\) matches to the key vector \(W_{k}^{t}\) of all feature maps in this batch to calculate the positional encoding result, which is used to represent the similarity and relevance of features in the image. \[\mathrm{A_{t}}=\left[\sum_{m=1}^{t}W_{q}^{m}\odot P_{m}\right]^{T}W_{k}^{t} \tag{2}\]  
\(\mathrm{A_{t}}\) is the self-attention distribution and is one element of self-attention matrix \(\mathrm{A}\in\mathbb{R}^{\mathbb{C}}\). \(m\) and \(k\) indicate the feature map index. \(\mathrm{A_{t}}\) represents the degrees of influence of the \(m\) feature map to \(k\) feature map, whereby the model obtains any two elements dependencies of the global context. \(P_{m}\) is an element of a corresponding feature map, and the dimension of \(P_{m}\) is the same as \(W_{q}^{m}\). In the early stage of training, the feature extraction module was not fully trained due to weight matrix and bias, and it picks up limited helpful features which will lead to a small number of \(\mathrm{A_{t}}\).