# Walking Back the Data Quantity Assumption to Improve Time Series Prediction in Deep Learning
## 3 Methodology
### Long Short-Term Memory (LSTM) Networks  
Long short-term memories (LSTMs) are a sophisticated neural network architecture designed to handle time-dependent data such as text, speech, or translations [20]. These networks are equipped with internal memory and feedback loops within neurons in the same layer, allowing them to process sequences effectively. Due to their complexity, the processing units in LSTMs are referred to as \"cells\" instead of neurons. As depicted in Figure 1, the temporal dynamics of inputs (\(\mathbf{x}_{t}\)), outputs (\(\mathbf{y}_{t}\)), feedback (\(\mathbf{y}_{t-1}\)), and memory states (\(c_{t-1}\) and \(c_{t}\)) are clearly illustrated. Each cell contains three gates, the input gate, forget gate, and output gate, which regulate the flow of data within the cell over time. These gates determine how much of the information is used to update the internal state. The forget gate is responsible for deciding whether to retain or discard the stored information, \(c_{t-1}\). This decision is made by computing the weighted sum of the new inputs, \(\mathbf{x}_{t}\), and the outputs from all neurons in the preceding time step, \(\mathbf{y}_{t-1}\). This computation is processed through a sigmoid function \(\sigma\), which produces a response ranging from 0 (indicating complete erasure of the \"memory\") to 1 (indicating full retention of the \"memory\"):  
\[f_{t}=\sigma\Big{(}\mathbf{W}^{f}\big{[}\mathbf{y}_{t-1},\mathbf{x}_{t}\big{]}+b^{f} \Big{)}, \tag{2}\]  
In this expression, \(\big{[}\mathbf{y}_{t-1},\mathbf{x}_{t}\big{]}\) represents the input vector (column vector) to the cell, comprising the new inputs at time \(t\), \(\mathbf{x}_{t}\), and the outputs of all neurons in the same layer at the previous time step, \(\mathbf{y}_{t-1}\) (feedback). The symbol \(\mathbf{W}^{f}\) denotes the weight vector associated with the forget gate, which includes weights for both the new inputs and the feedback, while \(b_{f}\) signifies the corresponding bias. It is important to note that in neural network literature, weights are typically represented as matrices when referring to an unspecified neuron within a layer, with each row corresponding to a specific neuron. However, since the formula reflects the response of a single neuron, the weight matrix is represented as a row vector, which is then multiplied (through the dot or scalar product) by the column vector comprising the new inputs and feedback. The input gate determines whether to incorporate new information into the internal state of the cell (its \"memory\"). Similarly, the weighted sum of new inputs and feedback is processed through a sigmoid function to ascertain the extent to which new information should be integrated into the cell's \"memory\":  
\[i_{t}=\sigma\Big{(}\mathbf{W}^{i}\big{[}\mathbf{y}_{t-1},\mathbf{x}_{t}\big{]}+b^{i}\Big{)}, \tag{3}\]  
In this context, \(\mathbf{W}^{i}\) and \(b^{i}\) represent the associated weight matrix and bias. The information intended for inclusion in the cell's \"memory\" is calculated using the hyperbolic tangent function, which yields values ranging from \(-1\) to +1. This is applied to the weighted sum (with \(\mathbf{W}^{c}\) as the weight matrix) of the new information supplied to the cell, in conjunction with contributions from other neurons within the same layer. A bias term, \(b^{c}\), is also incorporated in this computation:  
\[c_{t}^{\prime}=\tanh\big{(}\mathbf{W}^{c}\big{[}\mathbf{y}_{t-1},\mathbf{x}_{t}\big{]}+b^{ c}\big{)}. \tag{4}\]  
The updated internal state is derived from the combination of the portion of its prior value preserved by the forget gate and the input of the new information supplied by the input gate. Therefore, it can be expressed in the following manner:  
\[c_{t}=f_{t}c_{t-1}+i_{t}c_{t}^{\prime}. \tag{5}\]  
The output gate determines the proportion of the new internal state that will be emitted as the cell's output.