# Deep Echo State Networks in Industrial Applications

Stefano Dettori, Ismael Matino, Valentina Colla, Ramon Speets

► To cite this version:

Stefano Dettori, Ismael Matino, Valentina Colla, Ramon Speets. Deep Echo State Networks in Industrial Applications. 16th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2020, Neos Marmaras, Greece. pp.53-63, 10.1007/978-3-030-49186-4_5 hal-04060680

HAL Id: hal-04060680

https://inria.hal.science/hal-04060680

Submitted on 6 Apr 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Deep Echo State Networks in Industrial Applications

Stefano Dettori¹, Ismael Matino¹, Valentina Colla¹ and Ramon Speets²

¹ Scuola Superiore Sant'Anna - TeCIP Institute – ICT-COISP
{s.dettori, i.matino, v.colla}@santannapisa.it

² Tata Steel, CA IJmuiden, The Netherlands
ramon.speets@tatasteeleurope.com

**Abstract.** This paper analyzes the impact of reservoir computing, and, in particular, of Deep Echo State Networks, to the modeling of highly non-linear dynamical systems that can be commonly found in the industry. Several applications are presented focusing on forecasting models related to energy content of steelwork byproduct gasses. Deep Echo State Network models are trained, validated and tested by exploiting datasets coming from a real industrial context, with good results in terms of accuracy of the predictions.

**Keywords:** Reservoir Computing, Deep Echo State Networks, Industrial Application.

## 1 Introduction

These last few years have shown a dizzying growth trend in the application of Artificial Intelligence (AI) techniques, not only in the academic world, in which AI is considered a standard technique, with many fields that have reached a satisfactory level of development, but also outside the academia and, in particular, in industrial context. The new consciousness of some industrial sectors, and in particular the steel industry, towards the concepts of circular economy and sustainable development requires to tackle challenging technological and scientific problems, such as the optimization of the use and reuse of energy sources. At the same time, the need arises to improve the product quality in order to face the increasing competition in the markets of goods and services. Nowadays stakeholders are converging towards the idea that AI is a clear turning point for optimally addressing current and future challenges [1] and have begun, in the last decade, a race to cancel the knowledge gaps that exist in the industrial world between the engineering experience, which is currently based on standard modeling, optimization and control techniques and the so-called *data-driven* tools and techniques. Among
these latter ones, surely AI plays a fundamental role in facing the challenge of digitali-
zation [2] and nowadays the level of understanding and, above all, of acceptance of AI
as an effective and reliable technique [3] is surely increased.

In this context, reservoir computing is inserted, a tool capable of making the philos-
ophy and technique underlying the AI, aimed at the study and modeling of timeseries
and dynamic processes, more understandable and affordable in terms of computation
skills, efficiency and quality of obtained results. Reservoir computing has been intro-
duced by the work of Maass et al., in which has been described a particular Recurrent
Neural Network (RNN) architecture, called *Liquid-state machine* [4].

The literature in the field of reservoir computing applied to industrial processes is
quite extensive, and in particular among the various techniques emerges that based on
the Echo State Network, thanks to characteristics that make this approach more attrac-
tive and effective. Interesting examples of their application are shown in the work of
Wang [5] in which ESN and sparse adaboost method are exploited to forecast the elec-
tricity consumption in industrial areas; Matino presented a work related to the predic-
tion of blast furnace gas through ESN techniques [6] and Dettori applied several AI
methodologies aiming at modelling energy transformation equipment [7]. Colla et al
presented a work related to the use of outlier detection and advanced variable selection
to reservoir application in industry [8]. Another interesting work related to the applica-
tion of ESN to Model predictive control methodologies is presented by Pan [9].

This work presents some models developed within the European project entitled "Optimization of the management of the process gas network within the integrated steelworks" (GASNET), which aims at supporting the optimal use and distribution of valuable energy resources and byproduct gasses while minimizing the environmental impact. This problem is of utmost importance for integrated steelworks, i.e. the industrial plants which produce steel from virgin raw material, as considerable savings in CO₂ emissions as well as in natural gas consumptions can be achieved by means of an optimal distribution of the off-gases, such as discussed in [10-12]. In particular, in the present work, the models are developed through a recent reservoir computing methodology called Deep Echo State Network (DESN). The novelty of the work consists in the application of this novel Neural Network (NN) architecture, which allows modeling complex nonlinear dynamics that can be typically found in industrial processes. The effectiveness of the proposed methodology has been compared respect to other state of the art neural network architectures.

The paper is organized as follows: Section 2 describes the DESN architecture, sec-
tion 3 presents processes, models and datasets used, Section 4 describes methods and
results, while Section 5 provides some concluding remarks.

2 Deep Echo-State Network Architecture

ESN is an efficient tool and a universal uniform approximator [13], well known for
its intrinsic capability of reconstructing complex dynamical input/output relationships.
The concept behind the ESN approach is to generate within a reservoir a rich set of
dynamics starting from the exciting input. The frequency information content of the input is somehow distorted and enriched through the non-linear reservoir filter and then used to compute a regression on a target. In the last decade, the research in the field of reservoir computing has deepened the study of the characteristics of ESNs up to a further evolutionary step in its architecture, which borrows the concepts introduced by Deep Learning and amplifies its effectiveness through the use of algorithms that do not use heavy backpropagation routines. The resulted DESN approach introduced by Gallicchio [14, 15] consists of the exploitation of N reservoirs r connected in series in a deep learning fashion and a readout that collects all the reservoirs dynamics to compute the output of the network, as shown in Figure 1.

At time k, the state of neurons in each reservoir layers $x_i(k)$ is updated due to the action of its exciting input. The first reservoir layer receives in input the vector $u(k)$, the following layers receive in input the updated state of the previous reservoir layer $x_{i-1}(k)$, whose dynamic is updated according to the following equations:

$$x_1(k) = f(c_{in}W_{in1}u(k) + W_{r1}x_1(k-1) + v_1(k)) \quad (1)$$

$$x_i(k) = f(c_{is}W_{in_i}x_{i-1}(k) + W_{r_i}x_i(k-1) + v_i(k)) \quad (2)$$

where i is the i-th reservoir layer, f is typically a tanh function, $c_{in}$ and $c_{is}$ are respectively the input scaling and inter-scaling factors, $W_{in1}$ and $W_{in_i}$ are, respectively, the input matrix of the first and i-th reservoir layer with dimensions $n_1 \times n_{in}$ and $n_i \times n_{i-1}$, $n_i$ is the number of neurons of the i-th reservoir, $W_{r_i}$ is the reservoir matrix of i-th reservoir, $y$ is the output of the DESN and $v_i$ is a small amplitude white noise. The output of the readout is updated as:

$$y(k) = f_{out}(W_{out}x(k)) \quad (3)$$

where $x(k)$ is the complete vector of states, $f_{out}$ is the readout neuron function, typically the identical function for time series regression tasks, $W_{out}$ a $n_{out} \times n_T$ matrix and $n_T$ is the total number of reservoir neurons of the DESN.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Architecture of a DESN.</figcaption></figure>
Some of the aspects that make the approach particularly effective compared to those based on other recurrent networks are the architectural characteristics of the DESN and the training procedures that focus only in the weights of the readout.

More in detail, as said before, the only scope of the reservoir is to generate a rich set of dynamics, while the readout has to combine them in effective way, in order to minimize the regression error through non-iterative algorithms. The training phase is carried out in two sequential routines: The reservoir and input initializations and the readout training.

In the first phase, input and reservoir matrixes are initialized with the only constraint of obtaining a reservoir characterized by stable dynamics. In other words, the state of the reservoir has to be contractive and each neuron has to gradually forget their previous activation. These properties are summarized in the so-called Echo State Property (ESP), widely studied and deepen in several works of Yildiz and Jaeger [16] and then extended also to the case of DESN in the work of Gallicchio [15]. These works define necessary, sufficient conditions and empirical guidelines for the design of a contractive reservoir.

More in detail, each reservoir matrix $W_{ri}$ is initialized as a sparse randomized $\tilde{W}_{ri}$, which entries are defined in the range [-1, 1]. The sparsity, defined as the percentual of non-zero element over the total, is the percentual number of connections within the reservoir. This parameter is typically set below 5 %.

The first step to design a single contractive reservoir is to scale its matrix $\tilde{W}_{ri}$ in order to obtain the desired spectral radius $\tilde{\rho}_i$:

$$W_{ri} = \tilde{\rho}_i \frac{\tilde{W}_{ri}}{\rho(\tilde{W}_{ri})} \qquad (4)$$

In the case of DESN, the necessary condition to guarantee the ESP is that the greater of the spectral radius of all the reservoirs is less than one. This condition is a guideline to design a contractive global reservoir, but empirically is also sufficient. The spectral radius of each layer is an important hyperparameter, whose choice is subject to optimization. While the reservoir system is initialized, also each reservoir input matrix $W_{in_i}$ is initialized randomly with weights in the range [-1, 1]. In the equation (1) and (2) these matrixes are actually scaled by further coefficients $c_{in}$ and $c_{is}$, namely input scaling and inter-scaling factors, that are important hyperparameters and subject to optimization. Once all these matrixes are initialized and left untrained, it is possible to train the readout by exploiting linear regression routines. A straightforward solution is the Tikhonov regularization least square algorithm, calculated on the training dataset:

$$W_{out} = \bar{Y}X^T(XX^T + \lambda I)^{-1} \qquad (5)$$

where $\bar{Y}$ and $X$ are respectively the sequences of targets and states of all the reservoirs, the latter calculated by using equations (1) and (2). $\lambda$ is the regularization coefficient that allows to solve also ill-posed inversions and, in general, could be object of optimization during the hyperparameter selection phase.
## 3 Models and datasets

Between the several processes involved during the development of GASNET project, in this paper we present two case studies related to the prediction of energetic content and chemical characteristics of Blast Furnace (BF) off-gas, a particular byproduct gas (BFG) generated during the production of pig iron that can be reused as a valuable energy carrier to produce electrical energy or process steam. In particular, for the aims of GASNET project, it is useful to predict the future behavior and characteristics of the off-gas by using a restrict number of process measurements and future knowledge of the process scheduling for 2 hours ahead. Together with the prediction of the BFG production, this paper presents an application in which is consumed, the hot blast stoves, also called Cowpers.

In particular, in this process the preheating of the air takes place, subsequently blown inside the BF for the production of pig iron, through the combustion of BFG and other byproduct gasses typically available in some integrated steelworks, such as Coke Oven Gas (COG).

The targets of the first two models are the BFG volume flow and its Net Calorific Value (NCV). For each target will be developed a specific model capable of predicting the entire sequence of 2 hours ahead dynamic behavior in a one-shot multistep manner.

The targets of the third and fourth model are respectively the consumption of BFG and COG burned in the cowpers. Also in this case a specific model capable of predicting the entire sequence of 2 hours ahead dynamic behavior in a one-shot multistep manner.

The models will be trained exploiting real data of industrial partners, through datasets related to a period of 30 days with sampling time of 1 minute, sufficient to describe the main dynamics of the process.

Data pre-processing has represented a fundamental step for models design. Unreliable data have been identified through suitable outliers detection techniques [17, 18]. Moreover, the inputs of each model have been selected by exploiting a variable selection approach based on Genetic Algorithms [19, 20].

All the models have in input the scheduling of the respective process for 2 hours ahead, a Boolean information that describes if the respective process is active or not. The model that predicts BFG Flow has in input also the current measurements of the O₂ content in the cold wind, the cold wind volume flow, the pressure of the hot wind and the BFG volume flow. The model that predicts the BFG NCV has in input also the same measurements and in addition the current CO and H₂ contents in the BFG. The third model, which predicts the BFG consumption in the cowpers, takes in input its abovementioned process scheduling and the current measurements of the cold wind flow and the BFG and COG consumed in the cowpers. The fourth model, which predicts the COG consumption in the cowpers takes the same inputs of the previous model and in addition the future 2 hours predictions of BFG NCV. The input/output architecture of each model is depicted in figure 2, while the inputs and target dataset descriptions and Units of Measurement (UoM) are shown in the table 1.
<table><caption>**Table 1.** Input/output descriptions of each model</caption><thead><tr><th>Variables</th><th>Description</th><th>UoM</th></tr></thead><tbody><tr><td>Scheduling of BFG process</td><td>Boolean variable aimed at describing the status of the BF process. On (1), Off (0).</td><td>-</td></tr><tr><td>Scheduling of cowpers</td><td>Boolean variable aimed at describing the status of the cowpers. On (1), Off (0).</td><td>-</td></tr><tr><td>O<sub>2</sub> content in cold wind</td><td>Volume percentage of oxygen in the cold wind in input to the BF after heating</td><td>%</td></tr><tr><td>Cold wind flow</td><td>Volume flowrate of cold wind in input to the BF after heating</td><td>m<sup>3</sup>/h</td></tr><tr><td>Hot wind pressure</td><td>Pressure of hot wind in input to the BF</td><td>bar</td></tr><tr><td>BFG production</td><td>Volume of produced BF gas</td><td>m<sup>3</sup>/h</td></tr><tr><td>CO content</td><td>Volume percentage of carbon monoxide in BF gas</td><td>%</td></tr><tr><td>H<sub>2</sub> content</td><td>Volume percentage of hydrogen in BF gas</td><td>%</td></tr><tr><td>BFG consumption in the cowpers</td><td>Consumption of BF gas in cowpers</td><td>m<sup>3</sup>/h</td></tr><tr><td>COG consumption in the cowpers</td><td>Consumption of COG gas in cowpers</td><td>m<sup>3</sup>/h</td></tr></tbody></table>

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Input/Output architecture of the models</figcaption></figure>
# 4 Methods and numerical results

In order to evaluate the effectiveness of the DESN architectures for predicting the future behavior of the considered processes, a comparison is proposed between the results achievable through DESN and another rather efficient architecture for modeling timeseries characterized by non-linear dynamics, i.e. the Long Short-Term Memory (LSTM) [21]. In this work, the LSTM architecture is configured a series of input layer, a LSTM layer, $L_{LSTM}$ fully connected layers, and a linear readout.

Each model has been developed through a systematic procedure. In first place, it is necessary to define an optimal architecture for each model, by choosing a good set of hyperparameters. For the DESN the considered hyperparameters are the of number of layers and neurons of each reservoir, the spectral radius, the input scaling factor and the inter-layer scaling factor. For the LSTM the considered hyperparameters are the number of fully connected layers in series $L_{LSTM}$, and the number of neurons of each fully connected layer, with the i-th layer characterized by $N_{LSTM_i}$ neurons.

For the sake of simplicity, each layer of the DESN has the same number of neurons $N_{DESN}$ and the same spectral radius $\rho_L$. Moreover, each layer after the first has the same inter-layer scaling factor. The selection of the hyperparameters is an open topic for scientific research, and several works in literature give guidelines or recommend some particular algorithms for their choice, such as [22]. In our work their optimization is carried out through a random search technique [23] during which 1000 training trials have been carried out using a uniform distribution in the intervals: number of reservoir layers $L_{DESN} = [2 \ 12]$; Total Number of Reservoir Neurons = [200 2000], $\rho_L = [0.1 \ 1]$; Input scaling $c_{in} = [0.01 \ 10]$; Inter Scaling $c_{is} = [0.1 \ 1]$.

In the case of LSTM, the random search is performed over 1000 training trials with a uniform distribution of the hyperparameters in the intervals: $L_{LSTM} = [1 \ 10]$, $N_{LSTM_i} = [30 \ 300]$. The LSTMs have been trained through the ADAM training method [24]. The random search algorithm does not allow finding the best set of hyperparameters, but in general a good approximation of the optimal solution.

The optimal set of DESN and LSTM hyperparameters minimizes the mean value of the Normalized Root Mean Square Error (NRMSE) of all the outputs of the model mNRMSE, evaluated on the validation set. The comparison between the two different architectures is assessed by evaluating the mNRMSE, which is a particularly robust index with respect to the Mean Absolute Percent Error (MAPE) or other common metrics, due to the formulation that takes into account the overall range of the targets. Furthermore, the MAPE is not an adequate measure when intermittent target values (too many values equal or near to 0) are treated.

$$mNRMSE = 100 \frac{1}{n_y} \sum_{j=1}^{n_y} \left( \frac{\sqrt{\frac{1}{N_s} \sum_{k=1}^{N_s} (\bar{y}_j(k) - y_j(k))^2}}{\max(y_j) - \min(y_j)} \right) \quad (6)$$

The dataset is composed of about 45000 samples, divided in two parts: the first 50% is used for the choice of hyperparameters and subsequently for training the optimal
networks, the remaining 50% is used for the test of the trained models. During the phase of the hyperparameters selection, the first fraction of the overall dataset is in turn divided into 60% training and 40% validation.

The DESN results related to the trained optimal network are summarized in Table 2, which shows the results on the test dataset and the optimal architecture of each model. The comparison between the DESN- and LSTM-based architectures is reported in Table 3, in terms of *mNRMSE* on the training and test dataset.

<table><caption>**Table 2.** Optimal DESN architectures and test results of each model.</caption><thead><tr><th>Model</th><th>L<sub>DESN</sub></th><th>N<sub>DESN</sub></th><th>ρ<sub>L</sub></th><th>c<sub>in</sub></th><th>c<sub>is</sub></th><th>Test mNRMSE</th></tr></thead><tbody><tr><td>BFG Flow</td><td>7</td><td>90</td><td>0.549</td><td>0.154</td><td>0.952</td><td>6.7</td></tr><tr><td>BFG NCV</td><td>5</td><td>139</td><td>0.991</td><td>0.214</td><td>0.031</td><td>7.3</td></tr><tr><td>Cowpers BFG cons</td><td>5</td><td>293</td><td>0.447</td><td>5.10</td><td>0.022</td><td>6.09</td></tr><tr><td>Cowpers COG cons</td><td>5</td><td>180</td><td>0.722</td><td>0.073</td><td>0.691</td><td>9.87</td></tr></tbody></table>

<table><caption>**Table 3.** Comparison between DESN and LSTM architectures</caption><thead><tr><th>Model</th><th>Architecture</th><th>Training mNRMSE</th><th>Test mNRMSE</th></tr></thead><tbody><tr><td rowspan="2">BFG Flow</td><td>DESN</td><td>5.02</td><td>6.70</td></tr><tr><td>LSTM</td><td>7.95</td><td>10.92</td></tr><tr><td rowspan="2">BFG NCV</td><td>DESN</td><td>5.93</td><td>7.31</td></tr><tr><td>LSTM</td><td>8.36</td><td>16.6</td></tr><tr><td rowspan="2">Cowpers BFG consumption</td><td>DESN</td><td>4.60</td><td>6.09</td></tr><tr><td>LSTM</td><td>4.95</td><td>6.38</td></tr><tr><td rowspan="2">Cowpers COG consumption</td><td>DESN</td><td>8.08</td><td>9.87</td></tr><tr><td>LSTM</td><td>11.6</td><td>13.4</td></tr></tbody></table>

In the one-shot multistep ahead forecast approach, the *k*-th output is referred to *k* sample ahead prediction. For all the models, the error is low in the first 10-20 samples of prediction and, as it can normally be expected, it tends to increase as we want to predict the phenomenon in the distant future. An example of prediction 2 hours ahead of the BFG and COG consumption in the cowpers through DESN and LSTM architectures are shown in Figure 3 and Figure 4, respectively, which allow a comparison between the measured-target values (in blue) and the LSTM and DESN forecasted values (respectively in orange and yellow). In deeper detail, the figures show an example of one-shot multistep prediction of the abovementioned process for a specific instant of prediction, during which the target is characterized by a rich dynamic content. For confidentiality constraints, in the figures the absolute ranges of the measured and predicted values are normalized.
<figure><img src="image_5.png" /><figcaption>Fig. 3. Prediction example of BFG consumption in the cowpers.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Fig. 4. Prediction example of COG consumption in the cowpers.</figcaption></figure>

The results achieved during the test of the modelling approach are very encouraging. In particular, the models related to the BFG production and its energy contents are characterized of errors around 7% that, considering the heavy nonlinearity of the multi input multi output BF process, for control application are very low. This allows predicting the BFG production with good accuracy, sufficient to optimize its use in an energy
control and optimization strategy, and to provide a support to process operators for the
following 2 hours.

In the case of BFG consumption in the cowpers, the prediction errors are satisfactory.
The model for the prediction of COG consumption in the cowpers show a greater error,
but also in this case the model can be considered useful in an energy control and opti-
mization strategy.

The comparison between DESN and LSTM shows a clear difference between the
results and the quality of the prediction obtainable with the two different architectures.
In each proposed modelled process, DESNs outperform LSTMs on both training and
test dataset. In particular, with respect to LSTM architecture, DESNs allow obtaining
an improvement of the performances of the 4 models, equal to 4.22%, 9.29%, 0.29%
and 3.53%, respectively.

5 Conclusions

This paper describes the application of a particular reservoir computing technique called Deep Echo State Network to the modelling of nonlinear dynamics typical of complex industrial processes. The presented case studies are the forecast of energetic content in blast furnace gasses, produced during the production of pig iron in steelworks and one application of its consumption in the process called Hot Blast Stoves (Cowpers). The models have been trained, validated and tested by using real data. In particular, the hyperparameters of the DESN-based models are optimized through a random search approach that aims to minimize the validation error. The proposed DESN-based methodology has been compared with an LSTM-based architecture in order to assess the accuracy with respect to the state of art. The results show a great advantage in using DESNs to model the dynamic behavior of the considered processes, with respect to the LSTM architecture. The achieved results are satisfactory and the trained models are effectively used inside a control strategy for the optimal distribution of byproduct gases aiming at minimize the environmental impact of steelworks.

Acknowledgments

The work described in the present paper was developed within the project entitled "Op-
timization of the management of the process gases network within the integrated steel-
works - GASNET" (Contract No. RFSR-CT-2015-00029) and received funding from
the Research Fund for Coal and Steel of the European Union, which is gratefully
acknowledged. The sole responsibility of the issues treated in the present paper lies
with the authors; the Union is not responsible for any use that may be made of the in-
formation contained therein.
References

1. Brynjolfsson, E., Rock, D., and Syverson, C.: Artificial intelligence and the modern productivity paradox: A clash of expectations and statistics. National Bureau of Economic Research (2017). doi:10.3386/w24001.

2. Branca, T. A., Fornai, B., Colla, V., Murri, M. M., Streppa, E., and Schröder, A. J.: The Challenge of Digitalization in the Steel Sector. Metals 10(2), p. 288, (2020). doi:10.3390/met10020288.

3. Ransbotham, S., Kiron, D., Gerbert, P., and Reeves, M.: Reshaping business with artificial intelligence: Closing the gap between ambition and action. MIT Sloan Management Review 59(1) (2017).

4. Maass, Wolfgang, Thomas Natschläger, and Henry Markram. "Real-time computing without stable states: A new framework for neural computation based on perturbations." Neural computation 14.11 (2002): 2531-2560.

5. Wang, L., Lv, S.-X. and Zeng, Y.-R.: Effective sparse adaboost method with ESN and FOA for industrial electricity consumption forecasting in China. Energy 155, pp.1013–1031 (2018). doi:10.1016/j.energy.2018.04.175.

6. Matino, I., Dettori, S., Colla, V., Weber, V., and Salame, S.: Forecasting blast furnace gas production and demand through echo state neural network-based models: Pave the way to off-gas optimized management. Applied Energy 253, p.113578, (2019). doi:10.1016/j.apenergy.2019.113578.

7. Dettori, S., Matino, I., Colla, V., Weber, V., and Salame, S.: Neural network-based modeling methodologies for energy transformation equipment in integrated steelworks processes. Energy Procedia 158, pp.4061-4066, (2019). doi:10.1016/j.egypro.2019.01.831.

8. Colla, V., Matino, I., Dettori, S., Cateni, S., and Matino, R.: Reservoir Computing Approaches Applied to Energy Management in Industry. International Conference on Engineering Applications of Neural Networks, pp.66–79, Springer, Cham, (2019). doi:10.1007/978-3-030-20257-6_6.

9. Pan, Y., Wang, J. Model predictive control of unknown nonlinear dynamical systems based on recurrent neural networks. IEEE Transactions on Industrial Electronics 59(8), pp. 3089-3101, (2011). doi:10.1109/tie.2011.2169636.

10. Maddaloni, A., Porzio, G. F., Nastasi, G., Colla, V., and Branca, T. A.: Multi-objective optimization applied to retrofit analysis: A case study for the iron and steel industry. Applied Thermal Engineering, 91, pp.638-646, (2015). doi:10.1016/j.aplthermaleng.2015.08.051.

11. Colla, V. et al.: Assessing the efficiency of the off-gas network management in integrated steelworks. Materiaux et Techniques 107(1), p. 104, (2019). doi:10.1051/mattech/2018068.

12. Porzio, G. F. et al.: Process integration in energy and carbon intensive industries: An example of exploitation of optimization techniques and decision support. Applied Thermal Engineering 70(2), pp. 1148–1155, (2014). doi:10.1016/j.aplthermaleng.2014.05.058.

13. Grigoryeva, L., Ortega, J. P.: Echo state network are universal, Neural Networks 108, pp. 495-508, (2018). doi:10.1016/j.neunet.2018.08.025.

14. Gallicchio, C., Micheli, A., and Pedrelli, L.: Deep reservoir computing: a critical experimental analysis. Neurocomputing 268, pp. 87-99, (2017). doi:10.1016/j.neucom.2016.12.089.

15. Gallicchio, C., Micheli, A.: Echo state property of deep reservoir computing networks. Cognitive Computation 9(3), pp. 337-350, (2017). doi:10.1007/s12559-017-9461-9.

16. Yildiz, I. B., Jaeger, H., Kiebel, S. J.: Re-visiting the echo state property. Neural networks 35, pp. 1-9, (2012). doi:10.1016/j.neunet.2012.07.005.
17. Cateni, S., Colla, V., Nastasi, G.: A multivariate fuzzy system applied for outliers detection. Journal of Intelligent and Fuzzy Systems 24(4), pp. 889–903, (2013). doi:10.3233/ifs-2012-0607.

18. Cateni, S., Colla, V., Vannucci, M.: A fuzzy logic-based method for outliers detection. Proceedings of the IASTED International Conference on Artificial Intelligence and Applications, AIA 2007, pp. 561–566, (2007)

19. Cateni, S., Colla, V., Vannucci, M.: General purpose input variables extraction: A genetic algorithm based procedure GIVE a GAP. 9th International Conference on Intelligent Systems Design and Applications ISDA 2009, pp. 1278-1283, (2009). doi:10.1109/isda.2009.190.

20. Cateni, S., Colla, V., Vannucci, M. A genetic algorithm-based approach for selecting input variables and setting relevant network parameters of a SOM-based classifier. International Journal of Simulation Systems Science and Technology 12(2), pp. 30–37, (2011). doi:10.1109/ems.2010.23.

21. Hochreiter, Sepp, and Jürgen Schmidhuber. "Long short-term memory." Neural computation 9.8 (1997): 1735-1780.

22. Gallicchio, C., Micheli, A., Pedrelli, L.: Design of deep echo state networks. Neural Networks, 108, pp. 33-47, (2018). doi:10.1016/j.neunet.2018.08.002.

23. Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. Journal of Machine Learning Research 13, pp. 281-305 (2012).

24. Kingma, Diederik P., and Jimmy Ba. "Adam: A method for stochastic optimization." arXiv preprint arXiv:1412.6980 (2014).