# Ensemble of Networks for Multilabel Classification
## 6 Experimental Results  
The goal of the first experiment was to build and evaluate the performance of the different variants of the base models combined with all the components detailed in Sections 4 and 5. All ensembles were fused by the average rule. In Table 3, we provide a summary of each of these ensembles: the number of classifiers and hidden units, the number of training epochs, and the loss function. Given a base standalone GRU_A with 50 hidden units trained by [PERSON] for 50 epochs (labeled Adam_sa), we generated different ensembles by incrementally increasing complexity. We do this by combining ten Adam_sa  
Figure 3: The plot of Equation (35). Figure 2: Cyclic learning rate. (Adam_10s), increasing the number of epochs to 150 (Adam_10), selecting differnt optimizers (DG_10, Cos_10, Exp_10, Sto_10), and fusing the best results in the following ways:  
* DG_Cos is the fusion of DG_10 + Cos_10;
* DG_Cos_Exp is the fusion of DG_10 + Cos_10 + Exp_10;
* DG_Cos_Exp_Sto is the fusion of DG_10 + Cos_10 + Exp_10 + Sto_10;
* StoGRU is an ensemble composed of 40 GRU_A, combined by average rule, each coupled with the stocastic approach explained in Section 4.10;
* StoGRU_B as StoGRU but based on GRU_B;
* StoTCN is an ensemble of 40 TCN_A, combined by average rule, each coupled with the stochastic approach explained in Section 4.10;
* StoTCN_B as StoTCN but based on TCN_B;
* StoGRU_TCN is an ensemble of 40 GRU_TCN each coupled with the stochastic approach explained in Section 4.10;
* StoLSTM_GRU is an ensemble of 40 LSTM_GRU each coupled with the stochastic approach explained in Section 4.10;
* ENNbase is the fusion by average rule of StoGRU and StoTCN;
* ENN is the fusion by average rule of StoGRU, StoTCN, StoGRU_B, StoTCN_B and StoGRU_TCN;
* ENNlarge is the fusion by average rule of StoGRU, StoTCN, StoGRU_B, StoTCN_B, StoGRU_TCN and StoLSTM_GRU. An ablation study for assessing the performance improvement that each module of our ensemble achieved is reported in Table 4. Only tests on GRU_A is reported here. The other topologies tested in this work produced similar conclusions. In Table 4, we compare approaches using Wilcoxon signed rank test. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Name** & **Hidden Units** & **\#Classifiers** & **\#Epoch** & **Optimizer** \\ \hline Adam\_sa & 50 & 1 & 50 & Adam \\ Adam\_10s & 50 & 10 & 50 & Adam \\ Adam\_10 & 50 & 10 & 150 & Adam \\ DG\_10 & 50 & 10 & 150 & DGrad \\ Cos\_10 & 50 & 10 & 150 & Cos \\ Exp\_10 & 50 & 10 & 150 & Exp \\ Sto\_10 & 50 & 10 & 150 & Sto \\ DG\_Cos = DG\_10 + Cos\_10 & 50 & 20 & 150 & DGrad, Cos \\ DG\_Cos\_Exp = DG\_10 + Cos\_10 & 50 & 30 & 150 & DGrad, Cos, Exp \\ + Exp\_10 & 50 & 40 & 150 & DGrad, Cos, Exp, Sto \\ DG\_Cos\_Exp\_Sto = DG\_10 + Cos\_10 + & 50 & 40 & 150 & DGrad, Cos, Exp, Sto \\ Exp\_10 + Sto\_10 & 50 & 40 & 150 & DGrad, Cos, Exp, Sto \\ StoGRU\_B & 50 & 40 & 150 & DGrad, Cos, Exp, Sto \\ StoTCN & — & 40 & 100 & DGrad, Cos, Exp, Sto \\ StoGRU\_TCN & 50 (GRU) & 40 & 150 & DGrad, Cos, Exp, Sto \\ StoLSTM\_GRU & 125 (LSTM layer) 100 & 40 & 150 & DGrad, Cos, Exp, Sto \\ ENNbase = StoTCN+StoGRU & 50 (GRU) & 80 & 100 (TCN)/150 (GRU) & DGrad, Cos, Exp, Sto \\ ENN = StoTCN + StoGRU + StoTCN\_B & 50 (GRU, GRU\_B \& & 100 (TCN \& \\ + StoGRU\_B + StoGRU\_TCN) & GRU\_TCN) & 200 & TCN\_B)/150 (GRU, \\  & & & & GRU\_B \& GRU\_TCN) \\ ENNlarge = StoTCN + StoGRU + & 50 (GRU, GRU\_B \& & & \\ StoTCN\_B + StoGRU\_B + StoGRU\_TCN & GRU\_TCN) 100/125 & 240 & TCN\_B)/150 (GRU, \\  & & & GRU\_B, LSTM\_GRU \& \\  & & & GRU\_TCN) & \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Summary of tested ensembles. Table 5 shows the performance of the ensembles in Table 3 in terms of average precision. Moreover, we report the performance of IMCC [5] and TB. For both IMCC and TB, the hyperparameters were chosen by a five-fold cross-validation on the training set.