# Aortic Pressure Control Based on Deep Reinforcement Learning for _Ex Vivo_ Heart Perfusion
## 2 Materials and Methods
### Reinforcement Learning for Pulsatile Pump Control  
The control of aortic pressure during Langendorff perfusion is highly species-dependent, as different species require different coronary flow rates. In the heart, coronary blood flow is closely related to myocardial energy delivery, and an increase in coronary resistance limits coronary blood flow and oxygen delivery, whereas a decrease in resistance increases blood flow and oxygen delivery to the heart [23]. Coronary vascular resistance tends to change with perfusion time and perfusate temperature; [PERSON] et al. [24] found that there is a large difference in coronary artery resistance of isolated hearts perfused at three different temperatures (5 \({}^{\circ}\)C, 25 \({}^{\circ}\)C, and 35 \({}^{\circ}\)C) and that resistance decreases rapidly with increasing temperature. And there is variability in coronary artery resistance in different hearts [25]. Therefore, different perfusion strategies are being developed based on different coronary resistances to develop better perfusion decision protocols. Deep reinforcement learning combines the advantages of deep neural networks and reinforcement learning and is being explored in the medical field to learn the inherent laws of medical control problems and improve control effectiveness. The aim of this paper is to achieve control of AoP during perfusion based on the adaptive capability of deep reinforcement learning. Deep reinforcement learning can learn the coronary resistance of different hearts online, thus reducing the damage to the heart during the perfusion process. During heart perfusion, the demand for coronary blood flow must be met without providing excessive blood flow. We add a saturation module to detect and limit peaks in pump output to avoid overperfusion of the heart, so the algorithm is abbreviated as PD-DDPG in this paper. In this paper, we use the output pump speed of the DDPG agent to control the output pressure and the flow rate of the heart perfusion model, simulate the sensors to detect its AoP and flow variations, and preprocess the signals to feed the computed state and reward  
Figure 2: The blood flow curve of the pulsatile pump. values back to the agent. In clinical applications, noise may be present in the aortic pressure measurement part, leading to errors in the control process, which requires noise reduction in the measured pressure values in the pre-processing stage of MAP estimation in order to improve the adaptability and robustness of the control method to the measurement noise. To simulate the different states of the heart, we stochastically initialize the parameters of the heart model. Figure 3 shows the schematic diagram of the control algorithm used in this paper. The state, action, and reward are as follows. 1. State  
During the perfusion process, it is important to maintain an adequate MAP to minimize the delivery of maintenance solutions. In addition to the MAP value, we choose the error value (AMAP), and the sum of the error values of MAP is selected as the state parameters and kept within a reasonable range. The state parameters are as follows:  
\[s_{t}=\left\{MAP[k],\Delta MAP[k],\sum\Delta MAP[k]\right\} \tag{5}\]  
where \(t\) is the sampling time and \(k\) is the number of sampling cycles. The parameters can be obtained from the following equations:  
\[MAP[k]=\frac{1}{T[k]}\int_{t}^{t+\Delta t}P_{A}(t)dt \tag{6}\]  
\[\Delta MAP[k]=MAP[k]-MAP_{ref}[k] \tag{7}\]  
where \(\Delta t\) is the duration of the entire cardiac cycle (\(\Delta t=60/HR\)), \(T[k]\) denotes the number of AoP sampling values throughout the cardiac cycle, and \(MAP_{ref}\) is the reference value of the MAP, which is usually set according to the metabolic state of the heart during the perfusion period. Figure 3: Schematic diagram of the novel EVHP coupled reinforcement learning control system. Action  
Blood pump flow is related to aortic pressure and coronary flow.