# Temporal Convolutional Neural Network for the Classification of Satellite Image Time Series
## 4 Experimental Results
### How Wide and Deep a Model for Our Data?  
#### 4.4.1 Influence of the Width or Bias-Variance Trade-Off  
This section has two goals: 1) justify the complexity of the learned models, and 2) be able to give an idea about how wide should the model be. Both objectives relate to the bias-variance trade-off of the model for our quantity of data. The more complex the model (i.e., more parameters), the lower its bias, i.e., the fewer incorrect assumptions the model makes about the distribution from which the data is sampled. Conversely, given a fixed quantity of training data, the more complex the model, the higher the variance. Many classifiers vary their bias-variance trade-offs automatically by changing their bias, such as decision trees tend to grow deeper as the quantity of data increases. For neural networks however, the bias is mostly fixed by the architecture, and the variance by the architecture and the quantity of training data. We use here the number of trainable parameters as a proxy for model complexity, which provides a reasonable measure when dealing with a specific classification problem where the quantity of data and the number of classes are fixed ([53] (Chapter 7, Introduction)).  
We study seven CNN architectures with increasing number of parameters. Each architecture has three convolutional layers, one dense layer (256 neurons), and the Softmax layer as depicted in Figure 4. We then vary the number of neurons--or _width_--of the convolutional layers (16 to 1024 neurons). The depth of the model is specifically studied right after. The total number of trainable parameters then ranges from about 320,000 to 50 million. All models are trained following Section 2.3, with the exception of not using a validation set in order to observe more accuracy variations by letting the models being more prone to overfitting. Figure 9(a) shows the OA values as a function of number of parameters in logarithmic-scale.  
Figure 10: Overall Accuracy (\(\pm\)one standard deviation in orange) as a function of the number of parameters (**a**) or the number of successive convolutional layers (**b**) for seven temporal Convolutional Neural Network models. We use the dataset with three spectral bands and a regular temporal sampling at two days.  
Figure 10a shows that the architecture is very robust to a drastic change in number of parameters, as exhibited by OA varying only between 93.28% for 50M parameters and 93.69% for 2.5M parameters. The standard deviation increases with the number of parameters, but even at 50M parameters, most results are between 91% and 95% accuracy. Note that 50M parameters is an extremely large number of parameters for our dataset of about 620,000 training instances. From this result, one can see that models having about 2.5M of parameters with three convolutional layers and one dense layers have a good bias-variance trade-off.  
If more training data is available, using a similar architecture is likely to conservatively work well, because more data is likely to drive the variance down while bias is fixed. If less data is available, one could decide to use a smaller architecture but again, overall, the results are very stable.