## 4 Conclusions  
The results of this paper show that it is possible to use an RL agent without an initial model in the control of WWTPs. In fact, a better performance is shown compared with the use of traditional PID configurations. This improvement is achieved because the RL agent can use adaptive control strategies to follow the changes in the influent. In this way, the behavior of the plant can be optimized to achieve its objective: minimizing the WWTP operation cost. The cost function considers two factors: the fines due to high concentrations of pollutants (mostly nitrogen from N-ammonia) and the energy consumption to supply oxygen to the plant.  
Figure 14: Operation Cost \(OC\) for the **final** phase.  
In fact, the RL agent finds a balance between these opposing factors. In this way, it often allows the pollutants to exceed their limit to save the large expense of reducing their concentration. In addition, it can reduce the energy consumption of the blowers whenever it observes a low concentration of \(S_{\text{NH}}\).  
Although the so-called cold start problem is a challenge in the reinforcement learning area, in this paper, we did not focus on it because we just wanted to compare the evolution of the RL agent and compare different stages of the RL agent with the control methods. However, we are conscious of this problem and we have already worked on it [31]. As a first step, in order to observe the learning process of the agent, this work shows a simulation of the functioning of the RL agent in comparison with traditional PID approaches, in different weather conditions and how the agent can adapt to varying conditions. In general, the RL agent behaves differently on tops and bottoms of influent pollution concentration. In fact, on tops, the RL agent is more influenced by fines (because of the high concentration of pollutants to the effluent) while, on bottoms, the agent is more guided by the energy consumption. Finally, it is noteworthy that the RL agent performs especially well on weekends and in rainy and stormy weather conditions, where it greatly reduces consumption compared to the PIDs configurations. That is, due to the adaptative behavior of the RL agent, it can follow the changing environment better than the PIDs.  
The results of this work show that RL technique can be applied with improvements as an alternative to classical WWTP control solutions. However, we consider that there is still a lack of application of this strategy in other areas, not only under simulation but also in the real plant.  
Finally, we would like to note the BSM1 limitations. In future, we want to work on a more detailed model that, among other things, takes into account the inertia in turbine/motors of the blowers. In this case, we will measure aspects such as load rejection performance and see how this affects the OC, the different control methods and the RL agent's learning.  
Conceptualization, F.H.-d.-O.; Methodology, E.G. and R.D.; Software, F.H.-d.-O. and M.G.; Validation, F.H.-d.-O. and M.G.; Formal analysis, F.H.-d.-O. and E.G.; Investigation, F.H.-d.-O., N.D. and R.D.; Writing--original draft, F.H.-d.-O., E.G., N.D. and R.D.; Writing--review & editing, F.H.-d.-O., E.G. and N.D. All authors have read and agreed to the published version of the manuscript.  
This work was supported, in part, by the Spanish Ministry of Economy and Competitiveness under Project CICYT PID2019-108377 RB-C32, by the Spanish State Research Agency (AEI) under Project PID2020-112658 RB-I00.  
Not applicable.  
Influent data and more about BSM1 can be obtained at [[http://iwamia.org/benchmarking#BSM1](http://iwamia.org/benchmarking#BSM1)]([http://iwamia.org/benchmarking#BSM1](http://iwamia.org/benchmarking#BSM1)) (accessed on 4 April 2023).  
The authors declare no conflict of interest.