# Aortic Pressure Control Based on Deep Reinforcement Learning for _Ex Vivo_ Heart Perfusion
## 2 Materials and Methods  
### Lumped Parameter Model of the EVHP System in Langendorff Mode  
In Langendorff perfusion, oxygenated perfusate is pumped into the aorta, the aortic valve closes, and the perfusate enters the coronary arteries retrogradely, providing adjustable coronary blood flow [18]. Langendorff perfusion is widely used because it is easy to use and works with a wide range of species [19]. A lumped parameter model can characterize hemodynamics over time, and hemodynamic characteristics can provide an understanding of the state of the cardiovascular system, which is very important for clinical diagnosis [20]. The EVHP model is simulated using MATLAB/SIMULINK version R2022b (The MathWorks, Natick, MA, USA) in order to replicate real physiological conditions. The EVHP model used in this study, which neglects the effect of the oxygenator, consists of a pulsatile pump model and a coronary perfusion model, as shown in Figure 1. The coronary perfusion model, which is based on an article regarding our previous research [21], consists of systemic impedance, the coronary artery, and the myocardial vessels. The blood flow in the coronary part mainly originates from the aortic root and eventually converges in the vein. The intramyocardial pressure and the internal pressure of the pulsatile pump are indicated by external pressure elements. In Langendorff perfusion, the heart is empty-beating, the left ventricular volume is essentially constant, and the aortic valve is closed [22], so for simplicity, we set the \(P_{IMC}\) to 0. The parameters of the coronary perfusion model are shown in Table 1.  
During the perfusion process, the pulsatile pump delivers oxygenated blood into a cannula that is connected to the aorta, thereby achieving coronary perfusion. As illustrated in Figure 2, the pump flow \(Q_{in}(t)\) is based on a sine wave with a pulsating flow rate, which is expressed by Equation (1). \[Q_{in}(t)=v(t)\cdot sin\Big{(}2\pi f\cdot\Big{(}t-t_{delay}\Big{)}\Big{)}+v_{0}(t) \tag{1}\]  
where \(f=HR/60\), \(HR\) denotes the heart rate, \(v(t)\) denotes the fluctuating gain of the pulsatile blood pump flow control, \(v_{0}(t)\) denotes the constant part of the pulsatile blood pump flow that determines the average flow rate of the output, and \(t_{delay}\) denotes the time difference between the heart and the pump. In this paper, the lumped parameter model of a pulsatile blood pump is employed to simulate the dynamics of the blood flow rate, \(Q_{in}(t)\) can also be represented as follows:  
\[Q_{in}(t)=C_{sac}\frac{dP_{O}(t)}{dt} \tag{2}\]  
where \(C_{sac}\) is the blood lumen compliance inside the blood pump, and \(P_{O}(t)\) is the blood pressure at the blood pump outlet valve. Since heart valves work in a similar way to diodes, it is expected that a diode with forward conduction and infinite reverse resistance will be used to represent the valve in the lumped parameter model, as shown in Figure 1, where \(D_{I}\) and \(D_{O}\) denote the inlet and \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Parameter** & **Value** & **Parameter** & **Value** \\ \hline \(R_{S}\) & 0.6 mmHg\(\cdot\)s/mL & \(L_{CA}\) & 0.007 mmHg\(\cdot\)s\({}^{2}\)/mL \\ \(C_{S}\) & 0.5 mL/mmHg & \(R_{MP}\) & 37.5 mmHg\(\cdot\)s/mL \\ \(L_{S}\) & 0.12 mmHg\(\cdot\)s\({}^{2}\)/mL & \(R_{\text{M1}}\) & 9.75 mmHg\(\cdot\)s/mL \\ \(R_{CA}\) & 1.2 mmHg\(\cdot\)s/mL & \(R_{\text{M2}}\) & 24.75 mmHg\(\cdot\)s/mL \\ \(C_{CA}\) & 0.6 mL/mmHg & \(C_{IMC}\) & 0.001 mL/mmHg \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The parameters of the coronary perfusion model. Figure 1: A lumped parameter model for an EVHP system in Langendorff mode.