## 3 Methods
### Pyramid Attention  
The presence of pattern repeatability in images has been established as a crucial factor in image restoration. Self-similarity, as a form of repeatability, refers to the occurrence of small but similar patterns at different locations and scales within an image. It serves as valuable prior information for image restoration algorithms [26]. However, most existing deep neural network-based face super-resolution methods employ attention mechanisms that solely focus on the same scale, neglecting the full potential of self-similarity in face images. Given the symmetric nature of faces, facial images exhibit various repeatable and similar structures. Therefore, capturing rich self-similarity information can significantly enhance the performance of super-resolution reconstruction and reduce the model's reliance on external datasets. Based on this, we introduce a pyramid attention that captures feature correspondences at a distance from a multi-scale feature pyramid. This mechanism enhances the interaction between nearby and distant features, resulting in the recovery of finer facial details.  
The pyramid attention structure, as depicted in Figure 4, incorporates a scale-independent attention module called Scale Agnostic Attention (SAA) to capture the intrinsic correlation of the multi-scale feature maps. To input the feature \(F_{in}^{PA}\) into the pyramid attention, a five-layer feature pyramid with varying scales is constructed using Bicubic interpolation. The pixel feature at layer \(i\) is represented as \(x_{i}\), and its corresponding mapping  
Figure 3: (**a**) The architecture of efficient structure extraction module. (**b**) Illustration of efficient channel attention.  
across different scales is denoted as \(K\). Additionally, the pixel feature at the subsequent level is downsampled to \(x_{i}\) and referred to as \(z^{j}\). This downsampling operation is beneficial for reducing image noise and effectively preserving the original structural information even after scaling down. Hence, this operation contributes to noise reduction in the image while maintaining the integrity of the original structural details.  
For each scale feature, two image blocks are extracted: block \(f\) for reconstruction and block \(g\) for matching, corresponding to the block feature \(W_{f}\) and block feature \(W_{g}\), respectively. Regarding the block features \(W_{g}\), channel concatenation is applied as weights, and convolutional matching is performed with the input feature \(x_{i}\). This process yields the self-similarity feature map, which is obtained through a Softmax operation. On the other hand, the block feature \(W_{f}\), channel concatenation is directly applied as transposed convolution kernel weights. The correlation \(f_{out}^{i}\) between the two scales is obtained after performing the deconvolution operation with the self-similarity feature map. Finally, the pyramid attention is obtained by summing up the contributions from all positions of each scale, as expressed by the following Equation:  
\[f_{out}^{i}=\frac{1}{\sigma(x,K)}\underset{z\in K}{\sum}\underset{j\in Z}{ \sum}\varnothing\left(x_{\delta(r)^{\prime}}^{i}z_{\delta(r)}^{j}\right)\theta \left(z^{j}\right) \tag{9}\]  
where \(\varnothing\) represents the Gaussian embedding function for calculating the similarity between two-pixel features, \(\theta\) denotes the linear feature transformation function, \(\sigma\) represents a scalar function for normalizing the features, and \(\delta_{(r)}\) denotes nearest neighbor similarity constraint, where two-pixel features are considered highly correlated when and only when they are also highly similar to their corresponding neighbors, which helps to make the network focus more on feature-related regions while suppressing irrelevant regions and is used to improve the robustness of the feature matching process.