# Temporal Convolutional Networks Applied to Energy-Related Time Series Forecasting
## 2 Materials and Methods
### Methodology
#### 2.2.2 Temporal Convolutional Neural Network  
TCNs are a type of convolutional neural network with a specific design that makes them suitable for handling time series. TCNs satisfy two main principles: the network's output has the same length as the input sequence (similarly to LSTM networks); and they prevent leakage of information from future to the past by using causal convolutions [24]. Causal convolution differs from standard convolution in the fact that the convolutional operation performed to obtain the output at time \(t\) does not take future values as inputs. This implies that, using a kernel size \(k\), the output \(O_{t}\) is obtained using the values of \(X_{t-(k-1)},X_{t-(k-2)},\ldots,X_{t-1},X_{t}\) (Figure 4). Zero-padding of length \(k-1\) is used at every layer to maintain the same length as the input sequence.  
Furthermore, with the aim of capturing longer-term patterns, TCNs use one-dimensional dilated convolutions. This convolution increases the receptive field of the network without using pooling operations, hence there is no loss of resolution [35]. Dilation consists of skipping \(d\) values between the inputs of the convolutional operation, as can be seen in Figure 4c. The complete dilated causal convolution operation over consecutive layers can be formulated as follows [36]:  
\[x_{l}^{t}=g\left(\sum_{k=0}^{K-1}w_{l}^{k}\,x_{(l-1)}^{(t-(k\times d))}+b_{l} \right), \tag{1}\]  
where \(x_{l}^{t}\) is the output of the neuron at position \((t)\) in the \(l\)-th layer; \(K\) is the width of the convolutional kernel; \(w_{l}^{k}\) stands for the weight of position \((k)\); \(d\) is the dilation factor of the convolution; and \(b_{l}\) is the bias term. Rectified Linear Units (ReLU) layers are used as activation function (\(g(x)=max(0,x)\)) [37]. Another common approach to further increase the network's receptive field is to concatenate several TCN blocks, as can be seen in Figure 5[38]. However, this leads to deeper architectures with many more parameters which complicates the learning procedure. For this reason, a residual connection is added to the output of each TCN block. Residual connections were proposed by [39] in order to improve performance in very deep architectures, and consist of adding the input of a TCN block to its output (\(o=g(x+F(x))\)).  
All these characteristics make TCNs a very suitable deep learning architecture for complex time series problems. The main advantage of TCNs is that, similarly to RNNs, they can handle variable-length inputs by sliding the one-dimensional causal convolutional kernel. Furthermore, TCNs are more memory efficient than recurrent networks due to the shared convolution architecture which allows them to process long sequences in parallel. In RNNs, the input sequences are processed sequentially, which results in higher computation time. Moreover, TCNs are trained with the standard backpropagation algorithm, hence avoiding the gradient problems of the backpropagation-through-time algorithm (BPTT) used in RNN [40].  
Figure 4: Differences between (**a**) standard convolutional network, (**b**) causal convolutional network, and (**c**) dilated causal convolutional network.