# An Attention-Based Residual Neural Network for Efficient Noise Suppression in Signal Processing
## 2 Methods and Evaluation Index
### ResATnet Denoising Principle
The resultant features are then subjected to the activation function ReLU, and the two features are subsequently combined through summation and subsequently passed through the sigmoid activation function to generate the final channel attention coefficients. Upon derivation of the said coefficients, they are applied to the original feature map, culminating in the creation of a channel-weighted feature map. The formula can be written as:  
\[ \begin{split}\text{M}_{\text{c}}(\text{F})&=\sigma( \text{MLP}(\text{AvgPool}(\text{F}))+\text{MLP}(\text{MaxPool}(\text{F})))\\ &=\sigma(\text{W}_{1}(\text{W}_{0}(\text{F}_{\text{avg}}^{\text {c}}))+\text{W}_{1}(\text{W}_{0}(\text{F}_{\text{max}}^{\text{c}})))\end{split}  \tag{4}\]  
where F is the feature map, and W\({}_{0}\), W\({}_{1}\) are the corresponding parameters. Then, the spatial attention module (SAM) involves two distinct processing steps executed on the feature maps that have been initially subjected to channel attention module. These steps encompass MaxPool and AvgPool processing to obtain two feature maps, each of which has a channel number of 1. These two feature maps are then concatenated by channel, followed by a convolutional layer that compresses the channel number to 1 once again. The resulting feature is subsequently input to a sigmoid function, yielding the final spatial attention feature map. Finally, the spatial attention feature map is element-wise multiplied with the original feature map of the input spatial attention module to generate the fully optimized feature map for downstream analysis. The formula can be written as:  
\[ \begin{split}\text{M}_{s}(\widetilde{\text{F}})&= \sigma(\tilde{t}^{\prime}([\text{AvgPool}(\widetilde{\text{F}});\text{MaxPool}( \widetilde{\text{F}})]))\\ &=\sigma(\tilde{t}^{\prime}([\text{F}_{\text{avg}}^{\text{s}}; \text{F}_{\text{max}}^{\text{s}}]))\end{split}  \tag{5}\]  
where \(\widetilde{\text{F}}\) is the feature map after channel processing, \(\tilde{t}^{\prime}\) is the convolution layer with reduced number of channels, and its convolution kernel size is 7. To avoid the introduction of any superfluous channel attention mechanisms, the current study strategically employs the channel attention module that is embedded in CBAM for the residual block's shortcut connections. This circumvents the need for an  
Figure 3: ResATnet’s residual block. additional channel attention model, thereby ensuring streamlined architecture and efficient utilization of computational resources. Moreover, it is noteworthy to mention that a salient point of differentiation between our proposed model and the Resnet18 network lies in the modification of the activation function. Specifically, we have made a deliberate departure from the conventional activation function employed in Resnet18 architecture, and we instead opted for a novel activation function that better caters to the specific requirements of our model. The ELU [29] activation function amalgamates the desirable properties of both sigmoid and ReLU functions, thereby rendering it particularly adept at handling noise and converging swiftly. This can be attributed to the fact that the output mean of ELU closely approximates zero, thereby promoting greater stability and faster convergence during training. As such, the present study strategically employs ELU activation function to supplant the ReLU function in both the Resnet18 and CBAM architectures. This deliberate substitution is aimed at further augmenting the denoising capabilities of the residual network, which is an important facet of the overall performance of the model. The mathematical expression of the ELU activation function is:  
\[\text{ELU}(x)=\left\{ \begin{array}{ll}x&\text{if }\ x>0\\ \beta(\text{e}^{x}-1)&\text{if }\ x\leq 0\end{array} \right. \tag{6}\]  
where \(0<\beta\) is the hyperparameter and \(x\) is the neural network eigenvalue of the input activation function.