# Vibration Control with Reinforcement Learning Based on Multi-Reward Lightweight Networks
## 3 Method
### Algorithm Framework  
We consider a standard reinforcement learning setting consisting of an agent interacting with a discrete-time step-size environment \(E\). At each time step \(t\), the agent can receive the observed value \(s_{t}\), take an action \(a_{t}\), and receive a scalar reward \(r_{t}\). The behavior of the agent is defined according to the strategy \(\pi\), which maps the state to the probability distribution of the action \(\pi\): \(S\to P(A)\). The environment \(E\) may also be random. We model it as a state-space \(S\), action-space \(A=R^{N}\), initial-state distribution \(P(s_{t})\), transition dynamics \(P(s_{t+1}|s_{t},a_{t})\) reward function \(r(s_{t},a_{t})\). The reward is defined as the sum of future reward discounts: \(R_{t}=\sum_{i=t}^{r}\gamma^{i-t}r(s_{i},a_{i})\). The discount factor is \(\gamma\in\) [0,1]. The payoff depends on the chosen behavior, and therefore depends on the strategy \(\pi\), and may be random. The goal of reinforcement learning is to learn a strategy that maximizes the expected revenue of the initial distribution: \(J=E_{r,s_{t}\in E,a_{t}\in\pi}[R_{1}]\). When \(t=\) i, we use the minimum value of the output of the two critic networks, i.e., the output of \(Q_{1t}(s_{t},a_{t})\) and \(Q_{2t}(s_{t},a_{t})\) critic networks. \[Q_{t}(s_{t},a_{t})=min(Q_{1t}(s_{t},a_{t}),Q_{2t}(s_{t},a_{t})) \tag{17}\]  
Many of the methods in reinforcement learning utilize the recursive relationship of the Bellman equation:  
\[Q^{\pi}(s_{t},a_{t})=E_{r_{t},s_{t+1}eE}\big{[}r(s_{t},a_{t})+\gamma\cdot E_{a _{t+1}eE}[Q^{\pi}(s_{t+1},a_{t+1})]\big{]} \tag{18}\]  
If the target policy is deterministic, we can describe it as a function \(\mu:S\gets A\), thus avoiding internal expectations. \[Q^{\pi}(s_{t},a_{t})=E_{r_{t},s_{t+1}eE}[r(s_{t},a_{t})+\gamma\cdot Q^{\mu}(s _{t+1},\mu(s_{t+1}))] \tag{19}\]  
Expectations depend only on the environment. This means that the transfer generated by different random behavior strategies \(\beta\) can be used to learn the \(Q^{\mu}\) offline strategy. We consider a function approximator with \(\theta^{Q}\) as a parameter and optimize it by minimizing the loss:  
\[L(\theta^{Q})=E_{s_{t}\in E,a_{t}\in\pi,r_{t}eE}[Q^{\mu}(s_{t+1},a_{t}|\theta^ {Q})-y_{t}]^{2} \tag{20}\]  
\[y_{t}=r_{t}+\gamma\cdot min\big{(}Q_{1t+1}(s_{t+1},a_{t+1}),Q_{2t+1}(s_{t+1},a _{t+1})) \tag{21}\]Here, \(r_{t}\) is the environmental reward, \(\gamma\) is the discount factor, and \(t+1\) is the index of the next state. As shown in Figure 5, the algorithm used in this paper is mainly divided into two parts: the yellow part represents the environment (simulator) and experience playback, and the virtual part is the Actor-Critic (AC) network. Specifically, first, the environment interacts with the AC network and stores multiple data \((s_{t},a_{t},s_{t+1})\) into the experience pool. Here, \(s_{t}\) denotes the state at time \(t\), \(a_{t}\) denotes the action at time \(t\), and \(s_{t+1}\) denotes the state at time \(t+1\). Each data point in the experience pool is given an initial priority. When the data reach a certain amount, priority data sampling \(B\) is performed for the data in the experience pool. \(s_{t}\) is passed to the Actor network, and the Actor network is passed to the critic a network and the Critic b network together with \(s_{t}\), based on \(s_{t}\)'s \(a_{t}\). At the same time, \(s_{t+1}\) is transmitted to the Actor Target network, and the \(a_{t+1}\) obtained by the Actor Target network based on \(s_{t+1}\) is transmitted to the Critic Target c network and the Critic Target d network together with \(s_{t+1}\). The task of the Critic networks is to score the actions obtained from the Actor network. Here, we use four Critic networks, where the Critic a network and the Critic b network take the minimum value, and the Critic Target c network and the Critic Target d network also take the minimum value. Finally, the error is calculated using the Formulas (20) and (21).