# Enhanced Wind Energy Forecasting Using an Extended Long Short-Term Memory Model
## 2 Extended Long Short-Term Memory Networks
### sLSTM Model  
The sLSTM model is developed with exponential gating and normalized memory updates to enhance conventional LSTM networks. The memory cell update in sLSTM at time \(t\) can be obtained by the cell state equation, which can be represented as  
\[c_{t}=f_{t}c_{t-1}+i_{t}z_{t}, \tag{1}\]  
where \(c_{t}\in\mathbb{R}^{d}\) represents the cell state at time \(t\), \(f_{t}\) denotes the forget gate output, \(i_{t}\) is the input gate output, and \(z_{t}\) represents the cell input. To maintain proper scaling of cell states, sLSTM implements a normalizer state:  
\[n_{t}=f_{t}n_{t-1}+i_{t}, \tag{2}\]  
where \(n_{t}\in\mathbb{R}^{d}\) represents the normalizer state that accumulates gating operation effects over time. The sLSTM model processes input information through a cell input, which is  
\[z_{t}=\phi(\omega_{z}^{\top}\mathbf{x}_{t}+r_{2}h_{t-1}+b_{2}), \tag{3}\]where \(\mathbf{x}_{t}\in\mathbb{R}^{m}\) is the input vector, \(\mathbf{\omega}_{z}\) represents the input weight matrix between inputs \(\mathbf{x}_{t}\) and the cell input, \(r_{z}\) is the recurrent weight between the hidden state value \(h_{t-1}\) and the cell input, \(b_{z}\) is the bias term, and \(\phi(\cdot)\) is the cell input state activation function.  
The normalized hidden state value in sLSTM can be represented as  
\[h_{t}=o_{t}\frac{c_{t}}{n_{t}}, \tag{4}\]  
where \(h_{t}\) is the hidden state value and \(o_{t}\) represents the output gate value in sLSTM. The gating mechanism of the sLSTM model comprises three components, i.e., input, output, and forget gates. The input gate in sLSTM employs exponential activation, which can be represented as  
\[i_{t}=\exp\left(\mathbf{\omega}_{i}^{\top}\mathbf{x}_{t}+r_{i}h_{t-1}+b_{i}\right), \tag{5}\]  
where \(\mathbf{\omega}_{i}\) represents the input weight matrix between inputs \(\mathbf{x}_{t}\) and the input gate, \(r_{i}\) is the recurrent weight between the hidden state value \(h_{t-1}\) and the input gate, and \(b_{i}\) is the scalar bias of the input gate. The forget gate in sLSTM allows flexibility in activation choice, which can be represented as  
\[f_{t}=\gamma\left(\mathbf{\omega}_{f}^{\top}\mathbf{x}_{t}+r_{f}h_{t-1}+b_{f} \right), \tag{6}\]  
where \(\mathbf{\omega}_{f}\) represents the forget weight matrix between inputs \(\mathbf{x}_{t}\) and the forget gate, \(r_{f}\) is the recurrent weight between the hidden state value \(h_{t-1}\) and the forget gate, \(b_{f}\) is the scalar bias of the forget gate, and \(\gamma(\cdot)\) represents either sigmoid \(\sigma\) or exponential activation. The output gate in sLSTM uses sigmoid activation, which is  
\[o_{t}=\sigma\left(\mathbf{\omega}_{o}^{\top}\mathbf{x}_{t}+r_{o}h_{t-1}+b_{o} \right), \tag{7}\]  
where \(\mathbf{\omega}_{o}\) represents the output weight matrix between inputs \(\mathbf{x}_{t}\) and the output gate, \(r_{o}\) is the recurrent weight between the hidden state value \(h_{t-1}\) and the output gate, and \(b_{o}\) is the scalar bias of the output gate.  
To ensure numerical stability with exponential activations, sLSTM incorporates a stabilization mechanism, which is  
\[m_{t}=\max(\log(f_{t})+m_{t-1},\log(i_{t})), \tag{8}\]  
where \(m_{t}\in\mathbb{R}\) is the scalar stabilizer state in sLSTM. Equation (8) enables the computation of stabilized input and forget gate values, which are  
\[i_{t}^{\prime} =\exp(\log(i_{t})-m_{t}) \tag{9}\] \[f_{t}^{\prime} =\exp(\log(f_{t})+m_{t-1}-m_{t}), \tag{10}\]  
where \(i_{t}^{\prime}\) and \(f_{t}^{\prime}\) represent stabilized input and forget gates in sLSTM, respectively.