# Synthesis of Non-Linguistic Utterances for Sound Design Support Using a Genetic Algorithm
## 3 GA Generated Sounds Valence and Arousal Evaluation Experiment
### NLU Evaluation Experiment Results  
To assess inter-rater reliability, the inter-class correlation (ICC) [31; 32] was used. The inter-class correlation coefficient is appropriate for this data as it was compiled from multiple raters rating items (sounds) on a 7-point continuous ordinal Likert scale. The reliability of all raters working as a group using the mean of all raters' values (ICC3k) was used since the mean of all raters' ratings was used in subsequent analyses in the rest of the paper. The guidelines are described by [PERSON] and [PERSON] [32].  
Figure 4: Snapshots over time of the GA over the valence arousal plane. (**a**) GA progress after 2700 individuals, (**b**) GA progress after 5400 individuals, (**c**) GA progress after 8100 individuals, (**d**) GA progress after 10,800 individuals.  
Figure 5: Scatterplot of the valence and arousal of the 80 sounds used in the experiment.  
The equation used to determine ICC is the two-way mixed effects, consistency-seeking, multiple raters/measurements version shown below.  
\[\frac{(\text{MS}_{R}-\text{MS}_{E})}{\text{MS}_{R}} \tag{3}\]  
where \(\text{MS}_{R}\) is the mean square for rows, and \(\text{MS}_{E}\) is the mean square for error.  
The ICC results are shown in Table 1 below:  
The values of 0.83 for valence and 0.75 for arousal indicate very good reliability from the group of raters. The result means that there is consistency between the raters as a group and that the average of all the raters' ratings can be reliably used in the further analyses presented in this work.  
The subject ratings were compared with the inferred valence and arousal from the random forest model. The mean absolute error (MAE) for valence was 0.117 and for arousal it was 0.067. The correlation coefficient for inferred vs experiment values was 0.22 for valence and 0.63 for arousal. FOne-way \(t\)-tests were applied to determine the statistical significance of the MAE results. The \(p\)-value for the valence MAE (0.117) was \(2.34\times 10^{-6}\) which was significant based on an alpha of 0.05. The \(p\)-value for the arousal MAE (0.067) was 0.612, which was not significant based on an alpha of 0.05.  
To verify the significance of the correlations, firstly the distributions of the valence and arousal ratings as well as those of the inferences were assessed to check their distributions. The results are shown in Table 2.  
For data that are not normally distributed, the Spearman Rank Correlation may be used instead of the Pearson Correlation. The Spearman Rank Correlation Coefficient for valence was 0.23 with a \(p\)-value of 0.038, which is significant based on an alpha of 0.05. Since the arousal ratings and inferences were both normally distributed, the Pearson Correlation coefficient of 0.63 may be used. However, the \(p\)-value based on an alpha of 0.05 was 0.473, which was not significant.  
These results show that the subjects were generally in agreement with the model in terms of their ratings for valence and arousal. The low correlation for valence suggests that there was some difficulty in interpreting the valence of the sounds while the relatively high correlation for arousal suggests that subjects found it easier to assess the arousal of the sounds.