# Evaluation of Different Control Algorithms for Carbon Dioxide Removal with Membrane Oxygenators
## 3 Results and Discussion
In the context of cardiopulmonary bypass and blood oxygenation, the gas flow rate was the main manipulated variable because the system usually takes over the whole blood circulation and oxygen requirement. There were strict constraints on the values that the blood flow rate can take, so that physiological conditions were maintained, and there was minimal clotting in the oxygenator [4]. However, for carbon dioxide removal systems, the blood flow rate seemed to be more appropriate. The plasma carbon dioxide partial pressure was much more sensitive to the blood flow rate, compared to the sweep flow rate--602.7 mmHg L\({}^{-1}\) s\({}^{-1}\) for blood and 363.2 mmHg L\({}^{-1}\) s\({}^{-1}\) for gas flow rate in the range of 0 to 1.2 L/min. Since the systemic circulation was usually intact, in the carbon dioxide removal context, the blood flow rate through the catheter could be freely regulated to achieve the desired conditions. Under this assumption, the RL controller optimized to a state in which it omitted the gas flow rate for the control of the CO\({}_{2}\) partial pressure, setting it to an almost constant value. When the control action was not constrained with a lower bound, all controllers simply set the desired CO\({}_{2}\) concentration and then switched off all action. This was not the desired behavior because the CO\({}_{2}\) flux, in this case, became 0. The literature refers to this behavior as reward hacking, which commonly arises in reinforcement learning tasks [33]. Interestingly, not only the RL but also the PI and the NMPC controllers found this to be an optimal solution to the task. The goal of PI and NMPC controllers is to set \(p_{\text{CO}_{2},pl}\) with as little deviation as possible from the reference. Blood coming into the oxygenator from the body circulation has almost always different \(p_{\text{CO}_{2}}\) then the one in the oxygenator, because the gas gradients in the oxygenator are opposite to the ones in the body. Thus, the incoming blood counteracts the efforts of the controller to set the \(p_{\text{CO}_{2},pl}\) at the reference value. It is, therefore, beneficial for the controller to turn off this 'disturbance', which leads to this undesired behavior. We solved this issue by implementing a low limit of 60 mL/min for the flow rates that counteract this behavior. The upper limit of 1.2 L/min was chosen, with regard to physiology, equipment limits, and the sensitivity curve of the \(p_{\text{CO}_{2},pl}\). Applying these saturation limits after the controller output was the only way to solve this problem for the PI. NMPC allows the constraints to be implemented in the optimization, which makes its solution optimal. However, this issue could also be solved by including a CO\({}_{2}\) flux term in the cost function of the NMPC. On the other hand, the RL controller does not always converge to this sub-optimal solution. Our data indicate that the stochasticity in the exploration strategy of the RL agent could be the cause for this behavior. While compared to the NMPC, the RL and PI controllers showed inferior accuracy in the reference tracking and disturbance rejection task, and they need much fewer computation resources to work, which might be a decisive factor if the optimization of the NMPC does not allow for online use. The tuned NMPC needed, on average, 0.13 s to optimize for an action, which was less than its sample time of 0.4 s, and can, thus, be implemented for online use. However, we found NMPC controllers with different weights that needed, on average, up to 0.63 s to optimize for a control step. This problem arose from the time complexity of the interior point method, which was hard to determine and was dependent, among others, on the weights of the controller and on the quality of the starting point [34]. In comparison, the time complexity of the RL agent actions was \(\sim\)O(\(\sum_{k-1}n\times m\)), with \(m\)--the number of neurons for the current layer of the actor network, \(n\)-the number of neurons of the next layer, and \(k\) was the number of layers.