# Enhanced Wind Energy Forecasting Using an Extended Long Short-Term Memory Model
## 2 Extended Long Short-Term Memory Networks
### mLSTM Model  
The mLSTM model extends xLSTM by upgrading scalar cell states \(c_{t}\) in Equation (1) to matrix memory cell states \(\mathbf{C}_{t}\in\mathbb{R}^{d\times d}\), which enables enhanced storage capacity and parallel computation. The core innovation of mLSTM lies in its matrix memory cell state update using a covariance-based update rule, which can be represented as  
\[\mathbf{C}_{t}=\mathbf{f}_{t}\mathbf{C}_{t-1}+\mathbf{i}_{t}\mathbf{v}_{t} \mathbf{k}_{t}^{\top}, \tag{11}\]where \(\mathbf{C}_{t}\) represents the matrix memory cell state at time \(t\), \(\mathbf{f}_{t}\) is the forget gate vector in mLSTM, and \(\mathbf{v}_{t}\) and \(\mathbf{k}_{t}\) represent value and key vectors, which are  
\[\mathbf{v}_{t} =\mathbf{W}_{v}^{\top}\mathbf{x}_{t}+\mathbf{b}_{v} \tag{12}\] \[\mathbf{k}_{t} =\frac{1}{\sqrt{d}}(\mathbf{W}_{k}^{\top}\mathbf{x}_{t}+\mathbf{ b}_{k}) \tag{13}\]  
respectively, in which \(\mathbf{W}_{v}\) is the weight matrix between the value vector and the input vector, \(\mathbf{W}_{k}\) are the weight matrix between the key vector and the input vector, \(\mathbf{b}_{v}\) and \(\mathbf{b}_{k}\) represent bias vectors for the value and key vectors, respectively, and \(d\) is the dimension of the key/value space in the matrix memory. The normalizer state tracks the cumulative effect of key vectors \(\mathbf{k}_{t}\) weighted by gates. The normalizer state vector in the mLSTM model can be represented as  
\[\mathbf{n}_{t}=\mathbf{f}_{t}\mathbf{n}_{t-1}+\mathbf{i}_{t}\mathbf{k}_{t}, \tag{14}\]  
where \(\mathbf{n}_{t-1}\) represents the normalizer state vector at time \(t-1\). The matrix hidden state computation in mLSTM involves query-based memory retrieval, which is  
\[\mathbf{h}_{t}=\mathbf{o}_{t}\odot\tilde{\mathbf{h}}_{t}, \tag{15}\]  
where \(\mathbf{o}_{t}\) is the output gate vector in mLSTM, \(\odot\) denotes element-wise multiplication, and \(\tilde{\mathbf{h}}_{t}\) can be represented as  
\[\tilde{\mathbf{h}}_{t}=\frac{\mathbf{C}_{t}\mathbf{q}_{t}}{\max\bigl{(}| \mathbf{n}_{t}^{\top}\mathbf{q}_{t}|,1\bigr{)}}, \tag{16}\]  
in which \(\mathbf{q}_{t}\) is the query vector. The query vector is computed from the input:  
\[\mathbf{q}_{t}=\mathbf{W}_{q}^{\top}\mathbf{x}_{t}+\mathbf{b}_{q}, \tag{17}\]  
where \(\mathbf{W}_{q}\) is the weight matrix between the input vector and the query vector and \(\mathbf{b}_{q}\) is the bias vector. The gating mechanism employs exponential activation for the input gate:  
\[\mathbf{i}_{t}=\exp\Bigl{(}\mathbf{W}_{i}^{\top}\mathbf{x}_{t}+\mathbf{b}_{i} \Bigr{)}. \tag{18}\]  
The forget gate allows for either sigmoid or exponential activation:  
\[\mathbf{f}_{t}=\gamma\Bigl{(}\mathbf{W}_{f}^{\top}\mathbf{x}_{t}+\mathbf{b}_{ f}\Bigr{)}, \tag{19}\]  
where \(\mathbf{\omega}_{f}\in\mathbb{R}^{m}\) represents the forget gate weight matrix, \(\mathbf{b}_{f}\) is the bias vector for the forget gate, and \(\gamma\) denotes either sigmoid \(\sigma\) or exponential activation. The output gate vector of mLSTM can be represented as  
\[\mathbf{o}_{t}=\sigma\Bigl{(}\mathbf{W}_{o}^{\top}\mathbf{x}_{t}+\mathbf{b}_{ o}\Bigr{)}, \tag{20}\]  
where \(\mathbf{W}_{o}\in\mathbb{R}^{d\times m}\) represents the output gate weight matrix and \(\mathbf{b}_{o}\in\mathbb{R}^{d}\) is the bias vector for the output gate vector. Algorithm 1 captures the essential computational flow of the xLSTM model, incorporating both the sLSTM and mLSTM variants.