## 5 Methodologies
### Models
#### 5.1.1 Temporal Convolutional Networks  
Temporal convolutional networks (TCNs) are deep convolutional networks that use 'dilated' kernels (i.e., kernels that skip nodes in input or hidden layer tensors) in their convolution operations. A dilated kernel of dilation \(D\) and kernel size \(K\) inserts 'gaps' [63] of size \(D-1\) between each of the \(K\) kernel nodes for convolution. Figure 3 below shows two examples of dilated convolutional operations. Both are dilated kernels of kernel size \(K=2\) and convolved with stride \(s=1\). The left shows a kernel of dilation \(D=1\) over an input time series of size \(T\in\mathbb{N}\) reducing the layer output to size \(T-1\). The right shows a kernel of dilation \(D=2\), reducing the layer output to size \(T-2\). Dilated convolution operations are autoregressive.  
A TCN \(f(X\in\mathbb{R}^{N_{0}\times T_{0}};L,K,D,\theta)\), parameterised by \(\theta\), consists of \(L\) hidden layers referred to as _temporal blocks_, which are composites of element-wise activation functions \(\phi\) and dilated convolution operations of dilation \(D\) and kernel size \(K\). The input datum \(X\) is an \(N_{0}\)-variate tensor of time sequence length \(T_{0}\). At layer \(l\in L\) in the TCN, the input and output dimensions will be denoted \(X^{l-1}\in\mathbb{R}^{N_{l-1}\times T_{l-1}}\) and \(X^{l}\in\mathbb{R}^{N_{l}\times T_{l}}\), respectively.  
Each temporal block for layer \(l\), \(\psi_{l}\) contains an element-wise activation function \(\phi\) applied after a dilated convolution operation. The basic temporal block contains an affine operation that convolves some kernel \(W_{sD}^{(l)}\) of size \(K\) and dilation factor \(D\) striding over the input nodes \(X^{l-1}\). This is said to be a _dilated convolution of factor_\(D\). Kernel strides are convolved along the discrete-time dimension \(t\in T,T\in\mathbb{N}\).  
Figure 4 below demonstrates the application of the dilated convolution operation on input matrix \(X\in\mathbb{R}^{N_{l-1}\times T_{l-1}}\) for \(K=2,D=2\), and unit stride. This produces an activation tensor \(A\in\mathbb{R}^{N_{l}\times T_{l}}\) where the time dimension \(T_{l}=T_{l-1}-D(K-1)\) before element-wise activation function \(\phi\) is applied.  
Figure 5 shows that temporal blocks may also contain stacks of 'inner blocks', each consisting of dilated convolution layers and activations.  
Indeed, time series index \(t\in\mathbb{N}\) of the output from a dilated convolution operation only incorporates the input from the receptive field of past time series indices \((t-s),s\geq 0\), \(s\leq t,s\in\mathbb{N}\). This yields a receptive field size (RFS) of \(s+1\), inclusive of endpoints. Thus, only past information is propagated through the TCN network to learn time-lagged feature representations at different scale resolutions.  
In the special case where the kernel size is constant for all TCN layers, \(l\), and the dilation factor \(D\geq 2\) at layer \(l\) is given by \(D_{l}=D^{l}\), then the TCN is said to be a _vanilla TCN_.  
Figure 4: A dilated convolutional layer applied to a matrix \(X\) of data dimension 1. A dilated convolution operation is applied followed by an activation \(\phi\).  
Finally, the TCN network can include skip layers such as those used in the ResNet [47] architecture to mitigate vanishing gradient problems in learning through deep neural architectures. Figure 6 below explains the topology.  
The skip temporal block layer \(\eta_{l}\) transforms the input \(X^{l-1}\) to an output \(H_{l}\) that is accumulated to the final layer \(L\) such that:  
\[Y=f(X;N_{0},N_{L+1},K,B,D,\theta)=\sum_{l=1}^{L}H_{l}=\sum_{l=1}^{L}\eta_{l}(X^ {l-1}) \tag{8}\]  
where parameters for the skip layers \(\theta_{\eta_{l}},l=\{L,L-1,\ldots 1\}\) are found through recursion.