# Short Term Electric Load Forecasting Based on Data Transformation and Statistical Machine Learning
## 2 Proposed Methodology  
### Convolutional Neural Networks (CNN)  
CNN is a class of deep neural networks, most commonly applied to analyzing visual imagery. They are also known as shift invariant or space invariant artificial neural networks (SIANN), based on their shared-weights architecture and translation invariance characteristics [51, 52]. CNNs are regularized versions of multi-layer perceptron but they take a different approach towards regularization, since they take advantage of the hierarchical form in data and assemble more complex patterns using smaller and simpler patterns. Therefore, on the scale of connectivity and complexity, CNNs are functioning on the lower extreme. CNNs use relatively little pre-processing compared to other image classification algorithms. This means that the network trains the filters that in traditional algorithms were hand-engineered. This independence from prior knowledge and human effort in feature design is a major advantage. As presented in Figure 1, a CNN consists of an input and an output layer, as well as multiple hidden layers. These layers perform operations that alter data with the intent of extracting features specific to the data. Three of the most common layers are: convolution, activation or ReLU, and pooling. The hidden layers of a CNN typically consist of a series of convolutional layers that convolve through multiplication or other dot product each of which activates certain image features. The input layer reads an input image into the CNN. It consists of various low-level image-processing functions to pre-process the input image as an appropriate data type for the minimal CNN. Practically, the size of the input image is preferably on the order of power of 2 so that to ensure computation efficiency for a CNN. However, input images with uneven row-column dimensions can also be used. The convolution layer extracts pixel-wise visual features from an input image [54, 55, 56]. The trainable convolution kernels in this layer adjust their (kernel) weights automatically through back-propagation to learn the input image features [56, 57]. Image features learned by the convolution layer allow the successive algorithmic layers to process the extracted image features for other computational operations. The convolution layer is the dot product of the input image \(I\) and the kernel used, \(K\). The result is a convolved feature map, \(f_{c}\) which is derived by the Equation (1). \[f_{c}=conv(i,j)=(I\otimes K)(i,j)=\sum_{m}\sum_{n}I(m,n)K(i-m,j-n) \tag{1}\]  
Figure 1: CNN architecture [53]. Here \(\otimes\) denotes a two-dimensional discrete convolution operator. Equation (1) shows that the convolution kernel \(K\) spatially slides over the input image \(I\) to compute the element-wise multiplication and sum to produce an output, a convolved feature map \(f_{c}\). Convolution provides weights sharing, sparse interaction (and local connectivity), and equivariant representation for unsupervised feature learning. More in depth technical details on these convolution properties can be found in [54; 55; 56; 57; 58; 59; 60]. The activation function is commonly a Rectified Linear Unit (_ReLU_) layer. _ReLU_ allows for faster and more effective training by mapping negative values to zero and maintaining positive values. This is sometimes referred to as activation, because only the activated features are carried forward to the next layer. The objective of the _ReLU_ layers is to introduce a point-wise nonlinearity to a CNN, which allows it to learn through a nonlinear input function. _ReLU_ has also proven to be an effective solution to resolve vanishing gradient problems when training a CNN using the backpropagation algorithm [55; 56]. The mathematical structure of the ReLU function is a piecewise nonlinear operator with a max output indicative function [54; 58]. The output of a _ReLU_ is a rectified feature map, \(f_{r}\) that can be obtained by Equation (2).