## 1 Introduction
For example, recurrent neural networks are used to detect anomalies in influent conditions [25]. Additionally, deep learning algorithms, namely, recurrent neural network, long-short term memory and gated recurrent unit [26], are used to analyze and predict water quality. Besides the control of variables in the plant, machine learning algorithms are also used to develop soft sensors that predict values that are difficult to obtain using physical sensors [27; 28; 29; 30]. In addition, we also proposed a Reinforcement Learning (RL) agent to control DO concentration in the active sludge process of a WWTP [8; 31] (in Table 1 some of current control approaches in WWTPs are summarized). However, a recurrent finding in our search was a lack of comparison of novel techniques with current approaches already working. In fact, as a first approach in a previous work [8], we compared our RL agent against a PI cascade control, but we also proposed in that same work that a more exhaustive search would be interesting. In [8] we have improved the economic and environmental performance of the WWTPs using a RL approach. Using the BSM1 improvements in the operating costs of the N-ammonia removal process have been obtained. The proposed control scheme shows better performance than a manual plant operator when disturbances affect the plant and savings in a year of a working BSM1 plant are shown. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Reference** & **Techniques** & **Goal** \\ \hline
[10; 11] & PIDs & Control of DO concentration \\ \hline
[12] & \
\begin{tabular}{c} PIDs and \\ fuzzy logic \\ techniques \\ \end{tabular} &
\begin{tabular}{c} Control of DO and \\ ammonium/nitrate \\ concentration \\ \end{tabular} \\ \hline
[15] & \
\begin{tabular}{c} Genetic algorithm \\ optimization \\ \end{tabular} &
\begin{tabular}{c} Control of nitrogen and \\ ammonia concentration \\ \end{tabular} \\ \hline
[16] & Fuzzy control &
\begin{tabular}{c} Control of nitrogen and \\ ammonia concentration \\ \end{tabular} \\ \hline
[17; 21] & Artificial Neural Networks &
\begin{tabular}{c} Control of nitrogen \\ and/or DO concentration \\ \end{tabular} \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Control approaches in WWTPs. In [9] a review that covers automatic control of continuous aeration systems is presented. The paper focuses on recent published research that describes different control structures to control the DO concentration and the aerobic volume, with a special focus on plants with nitrogen removal. The main goal of this paper is to compare the RL agent with more complex PI-based techniques; more specifically, those described in [9]. The advantages of using the RL agent approach against the PID structures are shown: (i) the RL agent operates without knowing the initial model, (ii) the RL agent outperforms all configurations of traditional PIDs, even when the changing environment is more significant (for example, on weekends or during storms) and (iii) a comparative study of the savings obtained is presented and it is shown how the RL agent minimizes the costs. The rest of the paper is organized as follows. In Section 2, we briefly describe the BSM1 simulation model. The PI-based techniques considered for comparison and an overview of the proposed RL agent [8; 31] are also presented in this section. In Section 3 we describe the results and in Section 4, the conclusions are presented.