## 2 Materials and Methods
### Reinforcement Learning
#### 2.3.2 Reinforcement Learning Agent  
A RL agent acts in an environment to change certain conditions. For each action in each time step \(t\), the agent obtains a reward. The goal of the RL agent is to optimize the expected rewards, taking into account the conditions defined in the environment.  
This environment is usually defined as a Markov decision process. Nevertheless, this model is not mandatory. RL also supports model-free algorithms [34]. In these methods, the agent has to obtain the model of the environment as well as the optimal policy. This is the case for the RL agent defined in the BSM1 model, where the main objective of the agent is to lower the operation cost (OC) of the plant as much as possible while keeping the N-ammonia under 4 mg/L.  
The RL agent receives the value of \(S_{NH}\) and \(S_{O}\) as inputs and, as outputs, the DO set-point for tank 5 for each time step \(t\). The control loop that sets the DO set-point with the RL agent is shown in Figure 7.  
Figure 6: Control of the aerobic volume.  
Figure 7: Control loop using the RL agent. Reprinted/adapted with permission from Ref. [8]. Copyright 2016, [PERSON] et al.  
The agent should optimize the value of OC selection between three possible DO set-points for tank 5: 1.2 mg/L, 1.5 mg/L 1.85 mg/L. The operation cost OC for each time step \(t\) is defined in [8] (see Equation (1)):  
\[OC(t)=\gamma_{1}(AE(t)+ME(t)+PE(t))+\gamma_{2}SP(t)+EF(t) \tag{1}\]  
where \(AE\) is the aeration energy (kWh), \(ME\) is the mixing energy (kWh), \(PE\) is the pumping energy (kWh), \(SP\) is the sludge production for disposal (kg) and \(EF\) stands for the effluent fines (\(\epsilon\)). Weights \(\gamma_{1}\) and \(\gamma_{2}\) are set in proportion to the weights in the operating cost index defined in the benchmark BSM1. In the same way as [PERSON] et al. [35], we consider 0.1 \(\in\)/kWh. Hence, \(\gamma_{1}\) = 0.1 \(\in\)/kWh, and \(\gamma_{2}\)= 0.5 \(\in\)/kg following closely [32] that considers \(\gamma_{2}\) as five times \(\gamma_{1}\). OC considered by RL agent and how the values of \(AE\), \(ME\), \(PE\) and \(SP\) are obtained are defined in [8]. The algorithm of the RL agent is shown in Algorithm 1.  
```
Setup: \(\gamma=\) Time horizon \(max\_action=2\)// actions: {0,1,2} \(DO_{max}=\) Set-point max \(DO_{min}=\) Set-point min \(DO_{step}=(DO_{max}-DO_{min})/(max\_action+1)\) Input: \(s(t)=[NH_{4}(t),O_{2}(t)]\)// state of the environment \(r(t)=-OC(t)\)// reward Output: \(DO\) : Real Internal: \(Q(s,a)\) : random initialization \(a\) : action (0..max_action) Algorithm Main:  Initialize Q(s,a) whiletrue do // execute every 15 minutes \(s(t)=[NH_{4}(t),O_{2}(t)]\) \(a=next\_action(Q,s)\) \(DO=DO_{min}+a\times DO_{step}\) execute(DO)// now the plant has the control \(r(t)=-OC(t)\)// the plant returns its reward \(Q=update\_Q(Q,s,a,r,\gamma)\) end
```  
**Algorithm 1**RL agent pseudocode.  
Even though there are several approaches with which to achieve an adaptive behavior using ML (see Section 1), most of them belong to a _non-interactive_ way of learning. In fact, the use of past data is always necessary to build an accurate ML model of the plant and the influent. However, in our work, we focused on the RL approach because we wanted to show an ML system that updates its model in run-time while it interacts with the plant [31]. In fact, the most distinct characteristic is that the RL agent keeps _its own interaction with the plant_ into the model. This approach has a lot of advantages, but the main one is that the agent adapts to drifting characteristics of the influent and the plant in an autonomous way.  
This agent acts as a new block in the BSM1 model, as shown in Figure 8.  
The weather considered by the RL agent in the simulation is defined in the BSM1, where it rains 20% of the time, there is a storm 10% of the time and the remaining 70% the time is dry. The distribution is random. The climate is defined with a value of \"0\" for dry, \"1\" for rain and \"2\" for storm.