# Time Series Visualization and Forecasting from Australian Building and Construction Statistics
## 3 Methodology
### Time Series Forecasting
#### 3.2.3 Prediction  
LSTM-based forecasting is composed of four parts: (1) merging all features and target data and dropping empty data rows with empty cells; (2) splitting target data and features and reducing the feature dimension; (3) training the model; and (4) making forecasting. The first step is to merge all the dataframe objects. A dataframe object is created when loading a data sheet. Therefore, the program first sets the timestamp as the index for each of the dataframe object, then the dataframe objects are joined, one-by-one, using the timestamp index. The reason for dropping rows with empty cells is that the LSTM model does not support missing data or none-type data. Besides, it is not possible to do data imputation because all the missing data are at the head of the data sheet. The missing data are simply not recorded at that time, some types of data only start being recorded after 2003. The current data imputation techniques, such as setting a constant number or using previous or later value to replace missing data, cannot be applied to our case. The reason is that these data are not recorded, any types of misleading information or make up data will dramatically affect the trend of correlation between the features and target data. Therefore, we drop all the rows/timestamps containing missing values. Since the PCA is only applied to the features, after merging all the features and target dataset, they need to be split up into target dataframe object and the features dataframe object. Then PCA is applied to the feature dataframe object, and the wanted number of dimensions are extracted from the features to be prepared for training. Then, the extracted features are merged with the target data to drop out empty cells, as in the first step. After dropping out the data entries, around 70 data entries remain in the end. The next step is to train the model using the data. Before putting data into the model, normalization is used to bring both the feature and target data in the same footing, without any upfront importance, so that it can reduce the effect that a feature with a higher value range may dominate, when calculating distances between data points. For example, the target data may have a big difference with the reduced dimension feature on numeric value. If one of them is greatly larger than the other one, then the whole model will be dominated by the one with the larger value. Therefore, normalization is a way that can reduce such effects, but keep the completeness of the data. We scale and translate each feature individually, such that it is in the given range on the training set, and the range is between zero and one. After scaling the data, it can be put into the LSTM model. A cell, input gate, output gate, and forget gate make up a typical LSTM unit. The three gates control the flow of information, into and out of the cell, and the cell remembers valuesacross arbitrary time intervals. The following equations represent the components in the LSTM model:  
\[i_{t} =\sigma_{i}(W_{i}x_{t}+U_{i}c_{t-1}+b_{i})\] \[f_{t} =\sigma_{f}(W_{f}x_{t}+U_{f}c_{t-1}+b_{f})\] \[o_{t} =\sigma_{o}(W_{o}x_{t}+U_{o}c_{t-1}+b_{o})\] \[c_{t} =f_{t}*C_{t-1}+i_{t}*c_{c}(W_{c}x_{t}+b_{c})\] \[h_{t} =O_{t}*o_{t}(C_{t})\]  
where \(t\) refers to the time, and \(x_{t}\) is the input vector at time \(t\), while \(h_{t}\) is the output vector at time \(t\), and \(c_{t}\) stores the state of union. \(i_{t}\) is the vector of input gate, \(f_{t}\) is the vector of forgotten gate, and \(o_{t}\) is the vector of output gate. \(W_{i},W_{f},W_{o},W_{c},U_{i},U_{f},U_{o}\) are weights for \(x_{t}\) and \(c_{t-1}\), respectively, and \(b_{i},b_{f},b_{o},b_{c}\) are the shift vectors. All of these weights and biases are learned through the training process. \(\sigma_{i},\sigma_{f},\sigma_{o},c_{c},o_{h}\) are activation functions. The last step is to make forecasting.