# RCA-PixelCNN: Residual Causal Attention PixelCNN for Pulsar Candidate Image Lossless Compression
## 3 The Proposed Methods
### Residual-Causal-Attention Block  
To address the issues present in PixelCNN, we consider two approaches: residual neural networks and self-attention networks. To tackle the problem of gradient vanishing and gradient explosion caused by deep network layers, ([PERSON] et al., 2016) [25] introduced residual neural networks. These networks incorporate residual learning branches into the main network using skip connections. The main network approximates the target, while the residual branch learns the difference between the main network and the target. This ensures stable performance even with increased network depth. Residual connections expand the receptive field, but convolutional networks still tend to emphasize nearby information over distant information. On the other hand, self-attention models [26] utilize a square matrix of the same length as the input data sequence to store the importance of correlations between input elements. This breaks away from the convolutional bias, allowing access to long-range information indiscriminately. However, self-attention modules access all elements of the input, rather than just the pixels preceding the current pixel in the spatial position. This limitation prevents direct utilization of self-attention modules for autoregressive modeling.  
([PERSON], 2018) [27] analyzed the implementation process of self-attention and introduced causal attention modules by setting specific mask matrices. This allows networks containing self-attention modules to satisfy the autoregressive property. According to a  
Figure 3: The network architecture of RCA-PixelCNN model.  
certain autoregressive order, a series of 2D feature map vectors are named as \(y_{1},y_{2},\cdots,y_{N}\). The autoregressive mapping relationship is as follows:  
\[z_{i}=\sum_{i<j}p_{ij}f_{value}(y_{j}) \tag{2}\]  
\[p_{i}=softmax([f_{key}(y_{1})^{T}f_{query}(y_{i}),\cdots,f_{key}(y_{i-1})^{T}f_{ query}(y_{i})]) \tag{3}\]  
The attention distribution \(p_{i}\) corresponds to the dependency level of all features of feature \(\mathrm{y_{i}}\). Each conditional probability is established based on accessing all pixels within the attention-constrained context \(\sum_{i<i}p_{ij}=1\). As evident from Equation (2), to achieve autoregressive conditions, it is sufficient to constrain the summation terms during the summation process.  
As shown in Figure 4, this is our proposed residual-causal-attention block. The dashed box represents the causal attention module [12], which carries essential information in the main path of the network. Below it, there are three stacked masked convolutional layers, with ReLU layers and BatchNorm layers in between. The first convolutional layer has a kernel size of \(1\times 1\), and the number of channels decreases by a factor as indicated by the downward-pointing arrow in Figure 4. The second convolutional layer has a kernel size of \(3\times 3\) and maintains the same number of channels. The third convolutional layer has a kernel size of \(1\times 1\) and restores the input's original number of channels. During the process of information transmission within the network, the feature map dimensions remain unchanged. The attention module captures the importance of positional information, while the residual connection preserves detailed information in the features. Both the causal attention module and the residual branch impose connectivity constraints, ensuring the extracted information maintains autoregressive properties. The attention module can also employ a multi-head mechanism to enhance the weighting of importance. The residual causal attention module is an independent network module that can be used as a plugin within any part of an existing autoregressive network.