# Evaluation of Different Control Algorithms for Carbon Dioxide Removal with Membrane Oxygenators
## 2 Modeling and Control
### Control
#### 2.2.3 Reinforcement Learning Controller  
Reinforcement learning (RL) is a machine learning paradigm where an agent acts in an environment (Figure 1e)), which is often modeled as a Markov decision process, that gives rewards as feedback. There are many types of RL algorithms, depending on how the actions are selected or how the learning happens [24]. In this work, a modification of the deep deterministic policy gradient (DDPG) [25] algorithm was used--the twin delayed deep deterministic policy gradient algorithm (TD3) [26]. DDPG is a model-free, off-policy, online reinforcement learning algorithm that uses an actor-critic architecture aiming to learn a policy \(\pi\) that maximizes the expected future reward \(r\). The actor and critic are represented by deep neural networks to approximate the policy and \(Q\)-value functions. The actor follows the policy to select and apply actions \(a_{t}\sim\pi\) that transition the environment to new states \(s_{t+1}\). The critic estimates the long-term value of an action \(Q^{\pi}\) and critiques the actor on its actions by updating its estimation of the optimal policy [27]. The value function essentially estimates, in the long-term, what states would be visited, what action will be taken, and what reward will be accumulated by the agent, following the current policy. The rewards are given by a reward function, which defines the goal of the agent and has to be tailored for the specific application. The agent actively interacts with the environment, in order to maximize its reward. The \((s_{t},a_{t},r_{t},s_{t+1})\)(state-action-reward-next state) tuples are the basis for changing the policy. An action followed by a high reward should be preferred over an action followed by a low reward. The \(Q\)-value of a particular state is equal to the reward collected after choosing an action according to the policy, plus the \(Q\)-value of the state reached, i.e.,  
\[Q^{\mu}(s_{t},a_{t})=\mathbb{E}[r_{t}+\gamma Q^{\mu}(s_{t+1},\mu(s_{t+1}))] \tag{8}\]  
Equation (8) is called the Bellman equation. Here, the policy \(\pi\) is deterministic and parametrized by the function \(\mu\)--a deep neural network. \(\gamma\) is a discount factor, giving less weight to future rewards. The learning is achieved through the n-step temporal difference algorithm, which looks \(n\) steps ahead to estimate the total value of a state and is based on the Bellman equation [28], but instead of the \(Q\)-value function, a target \(Q\)-value function is used, which is updated slowly throughout the training process. The update step of the parameters of the target value function is:  
\[\theta^{Q^{\prime}}\leftarrow\tau\theta^{Q}+(1-\tau)\theta^{Q^{\prime}} \tag{9}\]  
with \(\tau\ll 1\). The objective of the policy function is to optimize the expected future rewards, i.e.,  
\[J(\theta)=\mathbb{E}\Big{[}\left.Q(s,a)\right|_{s=s_{t},a=\mu(s_{t})}\Big{]} \tag{10}\]  
The policy update (Equation (7)) is achieved in the direction of the gradient of the \(Q\)-value function [27]. \[\
abla\theta^{\mu}=\frac{1}{N}\sum_{i}\
abla_{a}Q(s,a|\theta^{Q})\
abla_{ \theta^{\mu}}\mu(s|\theta^{\mu}) \tag{11}\]  
The sum in Equation (11) is over a random mini-batch of previous experiences from an experience buffer. Again, a target policy function is used that slowly updates throughout the training process. The target functions are approximated by a deep neural network and with the same architecture as the actor and critic. The target functions increase the stability of the optimization process because the networks are not referring to themselves in the weight update step (Equations (8) and (11)). The additional modifications that TD3 introduces are that there are two critics, instead of one, used to update the target value function. Additionally, the policy is updated less frequently than the value function, and the target policy is regularized, such that similar actions bring similar rewards.