# Evaluation of Different Control Algorithms for Carbon Dioxide Removal with Membrane Oxygenators
## 3 Results and Discussion
On average, the RL agent needed 7.24 \(\times\) 10\({}^{-4}\) s to compute an action. The time complexity of the PI algorithm was \(\sim\)O(\(p\)), where \(p\) was the number of outputs. The average time needed to compute an action of the PI controller was 6.69 \(\times\) 10\({}^{-7}\) s. There was a performance advantage of \(\sim\)3 orders of magnitudes between PI and RL and RL and NMPC, when it came to computation time. Regarding memory requirements, PI needs 6 coefficients and takes about 1 kB of memory, NMPC needs around 30 coefficients and takes about 10 kB of memory, while the RL algorithm uses around 1.2 \(\times\) 10\({}^{-4}\) coefficients and takes around 300 kB of memory. Additionally, the PI controller is a very well-understood off-the-shelf algorithm that is very easy to implement, while the NMPC is a much more complicated control scheme that also needs an accurate model of the system, which might also imply the need for domain knowledge. On the other hand, RL is a model-free algorithm and, thus, does not need any domain knowledge. Training of the neural networks takes a long time, but after the agent is deployed, the selection of actions is fast and computationally inexpensive. The biggest disadvantage of the PI is the noisy nature of its actions. During operation, the high variability in \(Q_{b}\) and \(Q_{g}\) could lead to patient discomfort, or even potential harm [8, 35]. This is not the case with NMPC, because it penalizes varying inputs in its cost function. In order to investigate the effect on the policy divergence of \(n\)--the steps to look ahead for the \(n\)-step temporal difference algorithm, three agents with \(n=1\), 5, and 10 were trained for 15,000 (6 million steps) episodes on the membrane oxygenator model. The trained controllers were tested on a reference tracking task consisting of 50 random step-like setpoint changes (with the same random seed for comparison), and the RMSE between the setpoint and the actual \(p_{\text{CO}_{2},pl}\) was calculated. Figure 4 shows the results. The agents with \(n=1\) and 10 showed diverging behavior with increased training. After the full 15,000 episodes, both controllers kept constant blood and gas flow rates, without attempting to track the reference signal. Divergence often occurs when the critic learns unfeasibly large values for state-action pairs and then, based on those critiques, the actions of the actor, thus changing its policy [36]. On the other hand, the agent with \(n=5\) remained stable after 15,000 episodes, even though its RMSE slightly increased from its minimum at 10,000 episodes--from 1.1 mmHg to 1.18 mmHg. There was a huge increase in the RMSE for \(n=5\) at 5000 episodes. This agent _gave up_ and set the blood flow rate at the minimum value for the rest of the experiment after trying for several steps to track the highest setpoint in the validation set of 42.4 mmHg. This, again, can be explained as a mistake by the critic, who probably estimated more value in switching off the control action (which is penalized slightly by the reward function), instead of trying to track this _untrackable_ reference. This, however, is unacceptable behavior, which illustrates the issues of black box control methods for medical devices.