# Sound Event Detection Using Derivative Features in Deep Neural Networks
## 4 Experimental Results
### Experimental Results  
To train the basic CRNN, various combinations of weakly and strongly labeled and unlabeled data were considered. Specifically, four combinations, [weakly + unlabeled], [weakly + unlabeled + strongly], [strongly] and [weakly + strongly] were chosen. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Label Type** & **Weak Label** & **Strong Label** & **Unlabeled** \\ \hline No. of clips & 1578 & 2045 & 14,412 \\ \hline Properties & Clip-level & Frame-level & None \\ \hline Clip length & & 10 s & \\ \hline Classes (10) & Speech, Dog, Cat, Alarm bell ring, Dishes, Frying, Blender, Running water, Vacuum cleaner, Electric shaver toothbrush & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Training data. \begin{table}
\begin{tabular}{c c} \hline \hline
**DCASE 2018 Test Set** & **DCASE 2019 Test Set** \\ \hline No. of clips & 288 & 1168 \\ Properties & Frame-level \\ Clip length & 10 s \\ Classes & Same as training data & \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Test data. Binary cross-entropy was used as the loss function, and the Adam optimizer with a learning rate of 0.001 was used to train the basic CRNN. We applied early stopping with a minimum of five epochs and a patience of 15 epochs. These hyper parameters in this study are based on the baseline systems announced at the DCASE 2018 and 2019 challenges. The classification results on the DCASE 2018 test set are shown in Table 3, where \"Single channel\" implies that only the static log-mel filterbank was used as the input of the basic CRNN, and \"Three channels\" implies that derivative features (first and second) were also used as the input. It can be seen that [weakly + unlabeled + strongly] yields the best performance because it has the largest amount of training data. However, the performance is not satisfactory considering that the unlabeled training data constitute 80% of all the training data. This implies that the basic CRNN cannot efficiently use the unlabeled data to update its parameters. Furthermore, using derivative features in all combinations of the training data results in a consistent performance improvement in terms of the F-score. In the [weakly + unlabeled + strongly] combination, which yields the best results, a relative improvement of 7.2% in the F-score is observed. The average relative improvement of all combinations is 11.6%. However, the performance improvement is not sufficiently large to manifest itself in the ER. We compared the performance of our system with the DCASE 2018 baseline system [24] which uses the same training and test data as the [weakly + unlabeled] combination in Table 3. We could find that, although the baseline system employs similar CRNN architecture as ours, it showed 14.06% in F-score and 1.54 in ER. It has a better F-score and worse ER than the [weakly + unlabeled] combination. Slightly different hyper parameters and differences in the learning process may be the main reasons. However, by using the derivative features, we could obtain better results than the baseline system in both F-score and ER, as shown in the first row of the table. In Table 4, the classification results using the DCACSE 2019 test set are shown. A similar trend as in Table 3 is shown. The performance improvement by using the derivative features is rather diminished in the DCASE 2019 test set. On average, a 5.3% relative improvement in the F-score is attained, and [weakly + unlabeled + strongly] yields a 6% improvement. The F-score learning curve during the training of the basic CRNN when only the weakly labeled data are used is shown in Figure 5. Twenty percent of the weakly labeled data were used as validation data. At epoch 45, the optimal performance is attained by early stopping. Further training only increases the F-score of the training data, possibly resulting in overfitting.