# Deep-Learning Model Selection and Parameter Estimation from a Wind Power Farm in Taiwan +
## 2 Relate Work
### Convolutional Neural Networks for Sequence-to-Sequence Data  
Recently, researchers developed a DLN for time-series data processing that contains RNN, LSTM, GRU, and TCN used for handling sequence-to-sequence inputs, performing feature extraction of long-term sequence climate data, and estimating the expected electrical outputs for wind turbines [7, 8, 9, 10, 11, 12, 13]. Typically, an RNN is a class of artificial neural networks where connections between nodes form a directed graph along a sequence [7]. RNNs focus on sequential data forecasting, speech recognition, text generation, stock market prediction, and language translation. This specific design allows the network to process sequences of inputs using their internal state. This makes them suitable for tasks such as unsegmented connected handwriting recognition and speech recognition. However, RNNs suffer from the problem of vanishing gradients and have limited learning in long memory applications. Due to the problem of exploding and vanishing gradients with long sequences, LSTMs were designed with three specific gates operating by controlling the cell states. Basically, LSTM units are units of an RNN developed to deal with the exploding and vanishing gradient problems that can be encountered when training traditional RNNs. Their relative insensitivity to gap length gives LSTM networks a key advantage over RNNs, hidden Markov models, and other sequence learning methods in numerous applications. The compact forms of the equations for an LSTM unit are as follows [8, 9]:  
Updating rules for the forget gate, input gate and output gate in an LSTM block  
\[f_{t}=\sigma\Big{(}W_{f}[h_{t-1},\ x_{t}]+b_{t}\Big{)}, \tag{1}\]  
\[i_{t}=\sigma(W_{i}[h_{t-1},\ x_{t}]+b_{i}), \tag{2}\]\[\sigma_{t}=\sigma(W_{o}[h_{t-1},\ x_{t}]+b_{o}), \tag{3}\]  
Candidate (memory) cell state  
\[\widetilde{c}_{t}=tanh(W_{c}[h_{t-1},\ x_{t}]+b_{c}), \tag{4}\]  
Memory cell and hidden state  
\[c_{t}=f_{t}*c_{t-1}+\mathrm{i}_{t}*\widetilde{c}_{t}, \tag{5}\]  
\[h_{t}=o_{t}*tanh(c_{t}), \tag{6}\]  
Cell output  
\[y_{t}=h_{t}. \tag{7}\]  
where symbol \(x_{t}\) denotes the input vector to the neural network, \(f\) is the forget gate (a neural network (NN) with sigmoid), \(\widetilde{c}\) the candidate (memory) cell state (an NN with _tanh_ function), \(i\) the input gate (an NN with sigmoid), \(W\) the weight matrices and b the bias vector parameters that need to be learned during training, \(o\) the output gate (an NN with sigmoid), \(h\) the hidden state (a vector), \(c\) the memory state (a vector) and \(\sigma\) is the activation function in an LSTM unit. Notably, \(b_{t},\ b_{i},\ b_{o},\) and \(b_{c}\) represent bias vector parameters to be learned during training in forget gate, input gate, output gate, and memory cell, respectively. Later, GRU was proposed to handle simple problems for shorter sequential data by reducing the number of gates. Unlike LSTM, GRU replaces the forget gate and input gate in the LSTM with an update gate. Engineers usually choose GRU, considering the effects of computing power and time overhead of the hardware [10]. In 2017, [PERSON] et al. [11] proposed a new neural framework for sequence modelling tasks, namely temporal convolutional networks (TCNs) to address the problem for long sequential data of uncertain length. TCN consists of three parts: dilated causal convolution, nonlinear activation function, and residual connection. The TCN was designed using two basic principles: (i) Causal convolutions--the convolutions are causal, indicating that there is no information leakage from the future to past; (ii) One-dimensional fully convolutional network architecture--the architecture can take a sequence of any length and map it to an output sequence of the same length [12]. In other words, TCNs use one-dimensional (1D) separable convolutions that are a form of factorised convolutions that factorise a standard convolution into a depth-wise and pointwise convolution to preserve the massive memory requirements of deep learning models [13].