# Tiny Machine Learning Battery State-of-Charge Estimation Hardware Accelerated
## 6 Tiny Architectures Description
### Temporal Convolutional Network--Parallel Ensemble Approach  
The architecture was based on three parallel branches, each consisting of a specific TCN, producing three outputs merged into one. Combining the output of different neural networks predicting the same target variable is a common ensembling technique to improve  
Figure 8: Stateful GRU-based architecture. Figure 6: Stateful LSTM-based architecture. Figure 7: Stateless GRU-based architecture. predictive performance over the task [77]. However, the model developed by this work was not based on three independently trained ANN, and consisted of three TCN branches with different structures trained together as they were part of the same model. TCN met the constraint of at least 10 times less number of model's parameters then the training data; and also minimised the error achieved on the validation dataset. Experiments with a different number of parallel branches were performed, and the error metric guided this work toward the architecture here reported. The proposed TCN was based on two main concepts: preventing information leakage from the future to the past and residual connections (or shortcut connections, skip connections). As suggested by the authors [65], the technique of causal convolutions was applied; the use of causal padding meant that the output of the 1D CNN layers at time t depended only on contemporaneous or preceding elements of previous layers. The connection in the residual block was implemented by employing a \(1\times 1\) convolution due to the difference among the number of filters used and the size of the input data. By using the optional \(1\times 1\) Convolution layer, it has been ensured that the shortcut connection could be successfully performed. A TCN branch consists of a Residual Block as shown in Figure 9 (RB) which can be used to build a more complex topology as shown in Figures 10-12. When several RBs were used within the branch of TCN, as with the second branch (Figure 11) and third branch (Figure 12), the output of the entire branch was calculated by imposing skip connections between the individual outputs of each RB. Skip connections among RB were used to mitigate the vanishing gradient problem, as in deeper branches, there was a risk that the gradient of the loss function would become small during BP and thus slow down or block the learning process. This hypothesis was confirmed by the results achieved, introducing skip connections resulted in a marked improvement in the performance of the model. A single RB consists of the following layers (Figure 9):  
1. The first layer performs a 1-dimensional convolution with a specified number of filters, kernel size and dilation rate. Causal padding was adopted to keep temporal causality and kernel weights were initialized using He initializer. 2. Batch normalization was applied to the output of the first 1D CNN to stabilize the training of the neural network. 3. The ReLU activation function is then employed on the batch normalization output, convolution is a linear operation so this activation layer was meant to introduce non-linearity into the block. 4. A 0.05 spatial dropout followed the activation layer. 5. A second 1D convolutional layer was put within the residual block, it had same parameters of the first 1D CNN and it was connected to the dropout output. 6. Batch normalization was applied to the output of the second 1D CNN. 7. The ReLU activation function was computed on each output neuron of the previous batch normalization. 8. A 0.05 spatial dropout followed the activation layer. 9. A 1D CNN featured a kernel size of 1 and it was connected directly to the input layer to successfully perform shortcut connection. 10. An Add layer performed addition on the outputs of the \(1\times 1\) 1D CNN and the final ReLU to match initial lenght. 11. After reconstructing the initial size, the ReLU activation was calculated again and this was the last Residual Block layer.