# Temporal Convolutional Neural Network for the Classification of Satellite Image Time Series
## 2 Temporal Convolutional Neural Networks
### General Principles  
Deep neural networks are based on the concatenation of different layers where each layer takes the outputs of the previous layer as inputs. Figure 2 shows an example of a fully-connected network where the neurons in green represent the input, the neurons in blue belong to the hidden layers and the neurons in red are the outputs. As depicted, each layer is composed of a certain number of units, also known as neurons. The input layer size depends on the dimension of the instances, whereas the output layer is composed of \(C\) units for a classification task with \(C\) classes. The number of hidden layers and their number of units need to be selected by the practitioner.  
Formally, the outputs of a layer \(l\), the activation maps denoted by \(A^{[l]}\), are obtained with a two-step calculation: it first takes a linear combination of the inputs--which are the output of layer \(l-1\), i.e., \(A^{[l-1]}\), and then it applies a non-linear activation function \(g^{[l]}\) to this linear combination. It can be written as follow:  
\[A^{[l]}=g^{[l]}(W^{[l]}A^{[l-1]}+b^{[l]}), \tag{1}\]  
where \(W^{[l]}\) and \(b^{[l]}\) are the weights and the biases of the layer \(l\), respectively, that need to be learned.  
The activation function, denoted by \(g^{[l]}\) in Equation (1), is crucial as it allows to introduce non-linear combinations of the features. If only linear functions were used, the depth of the network would have little effect since the final output would simply be a linear combination of the input, which could be achieved with a single layer. In this work, we use the well-kown Rectified Linear Units (ReLU), calculated as \(\text{ReLU}(z)=\max(0,z)\)[23].  
Stacking several layers allows to increase the capacity of the network to represent complex functions, while keeping the layers simple, i.e., composed of a small number of units. Section 4.4 will provide some experimental results for different network depths.  
Let \((X,Y)\) be a set of \(n\) training instances such as \((X,Y)=\{(\mathbf{x_{i}},y_{1}),(\mathbf{x_{2}},y_{2}),\cdots,(\mathbf{x_{n}}, y_{n})\  
The loss function \(\mathcal{L}(\hat{y}_{i},y_{i})\), for a multi-class classification problem, is usually the cross-entropy loss:  
\[\mathcal{L}(\hat{y}_{i},y_{i}) =-\sum_{y\in\{1,\cdots,\mathcal{C}\}}\mathbb{1}\left\{y=y_{i}\right\} log(p(y|x_{i})) \tag{3}\] \[=-log(p(y_{i}|x_{i})), \tag{4}\]  
where \(p(y_{i}|x_{i})\) represents the probability of predicting the true class \(y_{i}\) of instance \(i\) computed by the last layer of the network (the Softmax layer) and denoted \(A^{[L]}\) for a network with \(L\) layers..  
Training deep neural networks presents two main challenges which are offset by a substantial benefit. First, it requires significant expertise to engineer the architecture of the network, choose its hyper-parameters, and decide how to optimize it. In return, such models require less feature engineering than more traditional classification algorithms and have shown to provide superior accuracy across a wide range of tasks. It is in some sense shifting the difficulty linked to feature engineering to the one of architecture engineering. Second, deep neural networks are usually prone to overfitting because of their very low bias: they have so many parameters that they can fit a very large family of functions, which in turn creates an overfitting issue [54]. Section 4.4 will provide an analysis of TempCNN accuracy as a function of the number of learned parameters.