# Vibration Control with Reinforcement Learning Based on Multi-Reward Lightweight Networks
## 3 Method
### Actor-Critic (AC) Network
#### 3.3.2 The Basic Architecture Design of DRSNs  
In this section, the motivation for developing DRSNs is introduced, and the architecture of two developed DRSNs is described in detail, including a deep residual shrinkage network with channel-shared thresholds (DRSN-CS) and a deep residual shrinkage network with channel-wise thresholds (DRSN-CW). 1. Theoretical background  
In the past 20 years, soft threshold segmentation has often been used as a key step in many signal denoising methods [37,38]. The advantage of this method is that it transforms the original signal into a domain where the number of near-zeros is not important, and then applies the soft threshold to convert the near-zero feature to zero. In order to ensure the good performance of signal denoising, a key task of the wavelet threshold method is to design a filter to transform useful information into very positive or negative features, and transform noise information into near-zero features. However, designing such filters requires a lot of signal processing expertise and has always been a challenging problem. Deep learning provides a new way to solve this problem. Deep learning is not designed by experts, but uses the gradient descent algorithm to automatically learn the filter. The formula for the soft threshold can be expressed as follows:  
\[y= \begin{cases}x+\xi&x>\xi\\ 0&-\xi\leq x\leq\xi\\ x+\xi&x<-\xi\end{cases}  \tag{27}\]  
Within this formula is the input feature, the output feature, and the threshold, that is, the positive parameter. The soft threshold method does not set the negative features in the ReLU activation function to zero, but sets the features close to zero to zero, thereby retaining useful negative features. It can be seen that the derivative of the output to the input is either 1 or 0, which can effectively prevent gradient disappearance and explosion problems. The derivative can be expressed as follows:  
\[\frac{\partial_{y}}{\partial_{x}}= \begin{cases}1&x>\xi\\ 0&-\xi\leq x\leq\xi\\ 1&x<-\xi\end{cases}  \tag{28}\]  
In a classical signal denoising algorithm, it is usually difficult to set an appropriate value for the threshold. In addition, the optimal value varies from case to case. To solve this problem, the threshold used in the developed DRSNs is automatically determined in the deep architecture, avoiding the trouble of manual operation. In the following chapters, the method of determining the threshold in the developed DRSNs will be introduced. 2. Developed DRSN-CS framework  
The developed DRSN-CS is a variant of the ResNet, which uses soft thresholds to remove noise-related features. The soft threshold is inserted into the building unit as a nonlinear transformation layer. In addition, the value of the threshold can be learned in the building unit, as described below. As shown in Figure 7a, unlike the RBU in Figure 6a, RSBU-CS has a dedicated module for soft threshold estimation. In this special module, Global Average Pooling (GAP) is performed on the absolute values of the feature map to obtain a one-dimensional vector. Then, the one-dimensional vector is propagated into a two-layer FC network to obtain a scaling parameter, similar to [39]. The sigmoid function is applied at the end of the two-layer FC network so that the scaling parameters are scaled to the range of (0, 1), which can be expressed as follows:  
\[\alpha=\frac{1}{1+e^{-z}} \tag{29}\]  
From this equation, we can find the output of the two-layer FC network in the RSBU-CS, and the delay parameter is the corresponding scaling parameter. After that, the scaling parameter delay is multiplied by the average value of the \(|x|\) score to obtain the threshold. The reason for this arrangement is that the threshold of the soft threshold should not only be positive, but also not too large. If the threshold is greater than the maximum absolute value of the feature map, the output of the soft threshold is zero.