# Evaluation of Different Control Algorithms for Carbon Dioxide Removal with Membrane Oxygenators
## 2 Modeling and Control
### Control
#### 2.2.5 Controller Implementation  
The controllers were implemented in MATLAB/Simulink and were interacting with a copy of the same model of the oxygenator system. The PI takes, as an input, only the tracking error and outputs blood flow rate and gas flow rate, which are min-max saturated to meet design criteria.  
The NMPC has access to the reference, all states of the model, and the measured disturbances. It outputs the optimal blood flow rate and gas flow rate. The constraints on the control action are natively implemented in the optimization algorithm, which ensures an optimal solution. The value of the equal concern for the relaxation of constraints (ECR) parameter was set to 20, meaning that violating the constraints is allowed if no solution to the optimization can be found. After tuning the controller, the weights on the manipulated variables, the manipulated variable's rates, and the weight for the output variable were set to (0.1, 0.1), (1, 1), and 10, respectively. The NMPC had a control horizon of 2 samples and a prediction horizon of 10 samples. It is a good idea to _'warm start'_ the controller by passing the resulting sequences of the last optimization procedure as initialization of the current one because it facilitates finding a solution faster. The actual numerical minimization of the cost function was achieved with an algorithm based on the interior point method [30; 31].  
The reinforcement learning controller also had access to all states and measured disturbances, but also to the tracking error and the integrated tracking error, which helped with disturbance rejection [32]. The constraints on the control action were implemented in the definition of the actor. For each controller, both the blood flow rate and the gas flow rate were constrained between 60 mL/min and 1200 mL/min. The RL agent was trained for 15,000 episodes, where each episode consisted of 40 s of simulated time with a constant \(p_{CO2,pl}\) setpoint (different for each episode) and changing saturation.  
The actor networks were represented by an input and normalization layer, followed by three fully connected hidden layers with 64, 32, and 16 nodes, respectively. The fully connected layers had a rectified linear unit (ReLU) non-linear activation function. The output blocks of the network comprised an action layer with two nodes for each action, combined with a hyperbolic tangent non-linear activation function and, finally, a scaling output layer.  
The critic networks had state and action input paths and a common path. The state input path consisted of a normalization input layer, followed by three fully connected hidden layers with 64, 32, and 32 nodes, respectively. The fully connected layers had a ReLU non-linear activation function. The action path consisted, again, of an input normalization layer and two fully connected hidden layers, with 64 and 32 nodes with a ReLU function. Both the state path and action path were fed into the common path, which had three fully connected hidden layers with 16 nodes with ReLU. The output layers of the network comprise a fully connected layer with eight nodes and the final layer which output the estimated value--a scalar.  
The learning rate for the actor network was set to 0.001 and for the critic at 0.0001. The standard deviation of the Ornstein-Uhlenbeck process used for exploration started at 5% of the action range and decayed to 0.5% of the action range. The critic learned with the five-step temporal difference algorithm.