## 2 Materials and Methods
### Neural Networks Architectures for TEC Forecasting
#### 2.2.5 The Proposed Deep Neural Networks for TEC Forecasting  
The architectures considered in the work can be divided into two main categories--recurrent neural networks and temporal convolutional neural networks. Each of these categories, in turn, is further subdivided into conventional and bidirectional. Hereinafter, the following notion will be employed: the prefix \"Bi\" denotes bidirectional architecture; for recurrent neural networks, the postfix \"RFF\" or \"FRF\" is used to indicate the recurrent layer position in the layer stack, defining the line of architectures. The postfix of the abbreviation \"RFF\" denotes a recurrent layer preceding two fully connected layers, and the ending \"FRF\" corresponds to a recurrent layer enclosed between two fully connected layers. Activation functions and normalization layers are not noted in the abbreviations. The \"LSTM\" or \"GRU\" sub-prefix specifies whether the long short-term memory or gated recurrent unit layer is used as a recurrent layer; temporal convolutional networks are marked using the \"TCN\" postfix. During the first stage of this research, the development of RFF and FRF multilayer architectures was undertaken. The architectures of the RFF line include the recurrent layer preceding two fully connected layers, while the FRF-line architectures consist of the recurrent layer enclosed between the two fully connected layers. The schematic diagrams of the proposed recurrent architectures are shown in Figure 8. The value of \(N\) determines the number of cells in each respective layer. The recurrent layer included in the architecture differs in each individual case and is defined either as LSTM, or as GRU, or as BiLSTM, or as BiGRU. These layers were followed by a parametric rectified linear unit (PReLU) activation function with the slope parameter specified using parentheses. PreLU multiplies the negative input values by the given parameter. For RFF architectures, the batch normalization layer was included to perform interlayer input normalization across mini-batches and to stabilize the learning process. The same approach applied to the FRF architecture provided no positive feedback, thus the batch normalization layer is not included. Figure 7: Schematic diagram of the bidirectional recurrent neural network. Weight initialization in fully connected layers is performed using the Glorot initializer [25]. Input and recurrent weights for recurrent layers were initialized using an orthogonal matrix decomposition initializer [26]. For recurrent layers, the tanh activation function is used for state activation and sigmoid is used for gate activation, and each recurrent layer is followed by the PreLU activation function. The schematic diagram of the proposed temporal convolutional network is presented in Figure 9.  
Figure 8: Schematic diagrams of the proposed multilayer recurrent neural network architectures: (**a**) RFF, (**b**) FRF. The selected recurrent layer and indices vary in each specific case. For temporal convolutional networks, the number of filters was equal to 256 with a filter kernel size of 2. Two dropout layers with a dropout factor of 0.005 were used in the main branch. The outputs are connected using addition. Per-channel symmetric rescaling was used for input data normalization so that each feature was in the range of \([-1;1]\). For conventional temporal convolutional networks, the optional convolution skip connection was used. For the bidirectional temporal convolutional network, the convolution skip layer was connected in parallel to the layers forming the bidirectional part. The schematic diagram of the bidirectional temporal convolutional network is presented in Figure 10.  
Figure 9: Schematic diagram of the proposed temporal convolutional network. The training was conducted using MATLAB environment and Deep Learning and Parallel Computing Toolboxes.