# Vibration Control with Reinforcement Learning Based on Multi-Reward Lightweight Networks
## 3 Method
### Actor-Critic (AC) Network
#### 3.3.2 The Basic Architecture Design of DRSNs
In summary, the thresholds used in the RSBU-CS are expressed as follows:  
\[\xi=\alpha\cdot\textit{average}|x_{i,j,c}| \tag{30}\]  
Here, \(\alpha\) is the threshold, and i, j, and c are the width, height, and channel index of the feature map, respectively. The threshold can be kept within a reasonable range so that the output of the soft threshold will not be all zeroes. The two-step RSBU-CS with a doubled number of channels can be constructed similar to the RBUs in Figure 6b,c. The brief architecture of the developed DRSN-CS is shown in Figure 7b, which is similar to the classic ResNet in Figure 6d. The only difference is that the RSBU-CS is used as a building block instead of RBUs. Multiple RSBU-CSs are stacked in the DRSN-CS to gradually reduce noise-related features. Another advantage of the DRSN-CS is that the thresholds are automatically learned in the deep architecture, rather than being manually set by experts. Therefore, professional signal processing knowledge is not required to implement the developed DRSN-CS. Figure 7: **(a)** A residual shrinkage building unit with channel-shared thresholds (RSBU-CS). **(b)** DRSN-CS overall architecture. **(c)** A residual shrinkage building unit with channel-shared thresholds (RSBU-CW). **(d)** DRSN-CW overall architecture, where K is the number of convolution kernels in the convolutional layer; m is the number of neurons in the FC network; and the C, W, and 1 in C \(\times\) W\(\times\) 1 represent the number of channels, width, and height of the feature map, respectively. When determining the threshold, the index of the feature map to be used is \(\alpha\). 3. _Developed DRSN-CW framework_  
The developed DRSN-CW is another variant of a ResNet. Unlike the DRSN-CS, it applies a separate threshold to each channel of the feature map, which will be introduced below. Figure 7c shows the RSBU-CW. Through absolute operation and the GAP layer, the upper and lower lines of the feature map \(x\) are reduced to one-dimensional vectors, and they are propagated to the two-layer FC network. \[\alpha_{c}=\frac{1}{1+e^{-z_{c}}} \tag{31}\]  
Here, \(z_{c}\) is the feature at the cth neuron, and \(\alpha_{c}\) is the cth scaling parameter. After that, the threshold can be calculated:  
\[\xi_{c}=\alpha_{c}\text{-}\underset{i,j}{\text{average}}|x_{i,j,c}| \tag{32}\]  
where i, j, and \(c\) denote the width, height, and channel index of the feature map x, respectively. Similar to the DRSN-CS, the threshold can be positive and maintained within a reasonable range to prevent the output feature from being all zeros. The overall architecture of the developed DRSN-CW is shown in Figure 7d. By superimposing multiple RSBU-CWs, through various nonlinear transformations, the soft threshold is used as the shrinkage function to learn the discriminant features and eliminate the noise-related information. (4) Detailed Network Architecture of the Actor  
In Figure 8, there are dashed- and solid-line skips. When the input and output dimensions are the same (solid-line skips), residual learning is directly applied, i.e., \(x+f(x)\), where \(f(x)\) represents the output result of the construction unit. In cases where the dimensions increase (dashed-line skips), matching dimensions occur (achieved by \(1\times 1\) convolution) with a stride of 2. Finally, through Avgpooling and flattening, we obtain [B, 32], which, via FC, leads to [B, 4] dimensions, representing the action signals from four sensors. The structure of the Critic's four networks resembles that of the Actor network, differing only in the input and output. The input is \(\{s_{t},a_{t}\}\) = {[B, 1, 4, 100], [B, 4]}, and the output is [B, 1]. Here, the input data are preprocessed as [B, 1, 4, 101], signifying 100 dimensions for each sensor's data and the action's value.