# Aortic Pressure Control Based on Deep Reinforcement Learning for _Ex Vivo_ Heart Perfusion
## 2 Materials and Methods
### Reinforcement Learning for Pulsatile Pump Control
The aortic pressure value is varied by changing the flow rate of the pulsatile blood pump to achieve control under different perfusion conditions. The action parameters are as follows:  
\[a_{t}=\left\{v^{*}(t),v^{*}_{0}(t)\right\} \tag{8}\]  
where \(v^{*}(t)\) and \(v^{*}_{0}(t)\) denote the normalized \(v(t)\) and \(v_{0}(t)\), which is the output value of the agent. The pump action affects the aortic blood flow rate, and thus, if the flow rate is too high, myocardial edema can easily occur. Conversely, if the flow rate is too low, insufficient oxygenation can easily occur. To avoid either too high or too low a flow rate, a saturation function \(\sigma_{s}\) is introduced to limit the pump output and ensure the safety of the control system. \[\sigma_{s}(t)=[sat(v_{1}),sat(v_{2}),\cdots,sat(v_{n})] \tag{9}\]  
\[sat(v_{i})=\left\{ \begin{array}{ll}s_{max}&\quad\quad\text{if $v_{i}\geq s_{ max}$}\\ v_{i}&\quad\quad\text{if $s_{min}<v_{i}<s_{max}$}\\ s_{min}&\quad\quad\text{if $v_{i}\leq s_{min}$}\end{array} \right. \tag{10}\]  
where \(n\) is the total number of changes in perfusion conditions, \(i\) is the number of changes in perfusion conditions, and \(s_{max}\) and \(s_{min}\) denote the saturation limit. 3. Reward  
In order to evaluate the efficiency of the MAP value control, it is included in the reward as a reference for a comprehensive assessment of the perfusion impact. The immediate reward is shown in Equation (11), while the value of \(\beta\) is shown in Equation (12). \[r(s_{t},a_{t})=\beta(MAP[k]) \tag{11}\]  
\[\beta(MAP[k])=\left\{ \begin{array}{ll}5&\quad\text{if $|\Delta MAP[k]|\leq 2$} \\ -1&\quad\text{else}\end{array} \right. \tag{12}\]  
The cumulative discount bonus per episode is illustrated in Equation (13), in which \(\gamma\) represents the discount factor, taking values between 0 and 1. \[R=\sum_{t=0}^{T}\gamma^{t}r(s_{t},a_{t}) \tag{13}\]  
The Deep Deterministic Policy Gradient (DDPG) algorithm is a model-free, non-strategic actor-critic network with a continuous action space. It is a combination of the Deep Q Network (DQN) and the Deterministic Policy Gradient (DPG) algorithm [26]. The optimal action in a given state is selected by combining a policy function with a value function, with the aim of obtaining the maximum reward. The critic network comprises two input layers, the first of which is determined by the number of state values, followed by a fully connected layer with 256 nodes and a fully connected layer with 128 nodes; the second input layer is determined by the number of action values, followed by a fully connected layer with 128 hidden nodes. The two input layers are connected to the final fully connected layer, which serves as the output. The output action of the critic network is the evaluated value of the final network result. In order to ensure that the output result of the network is an action value for evaluating the output action of the actor network, the final action value \(Q\) is output using the tanh activation function. The loss function is shown in Equation (14). \[L\left(\theta^{Q}\right)=\frac{1}{N}\sum_{t}\left(Q\left(s_{t},a_{t}\middle| \theta^{Q}\right)-y_{t}\right)^{2} \tag{14}\]where \(N\) denotes the number of experiences in the memory cache, \(Q\) denotes the critic network, and \(\theta^{Q}\) represents the weight associated with the critic network's representation of the \(Q\) function. The critic network \(Q\big{(}s_{t},a_{t}|\theta^{Q}\big{)}\) is utilized to illustrate the extent to which the gains from the actor network concerning a specific input \((s_{t},a_{t})\) are approximated by a function of the target value. \(y_{t}\) represents the target value, and the formula is shown in Equation (15), where \(Q^{\prime}\) denotes the target critic network. And Equation (16) is the updated equation for the parameter \(\theta^{Q}\) and \(\tau\ll 1\).