## 4 Results
### Impression Estimation from Video and Prosody Correlation
For arousal, the results are similar, but since it has lower variance when compared to valence, linear fitting describes the evolution of the emotional state of participants during a session, as one can see in Figure 10. However, as it is possible to see in the bottom right figure, that is not the case for every volunteer. We have calculated the average mean squared error between the predicted and the actual impression for average arousal of every session, obtaining a value of 0.006, while the same metric for valence is of 0.023. Figure 9: Scatter plots of the average valence of each time volunteers M6, F3, F7, and F12 listened to a GS utterance and the results of the linear regression for each session and across multiple sessions. Points with the same color were obtained in the same session, and the line for that session shares the color with the points. In the context of the present work, the error of the estimate of the emotion change caused by gibberish speech \(S(w,P)\) is defined as the norm of the difference between the actual emotional state change \(\widehat{\delta_{E_{S,A}}}\) and the predicted emotional state change \(\widehat{\delta_{E_{S,P}}}\), that is,  
\[EE_{S}=||\widehat{\delta_{E_{S,A}}}-\widehat{\delta_{E_{S,P}}}||=\sqrt{(\delta _{\varphi_{a,S}}-\delta_{\varphi_{p,S}})^{2}+(\delta_{a_{a,S}}-\delta_{a_{p,S} })^{2}}\]  
Two different neural network architectures, VGG-16 and ResNet18, were used for estimating arousal and valence of volunteers from their facial expressions, respectively. However, since the aforementioned neural networks estimate human emotions from still image frames and the collected data consist of video samples, it was necessary to choose a metric that was capable of representing the impact the gibberish speech had on the listener. This way, it is necessary to obtain the initial emotional state \(E_{t}\) of the volunteer and the emotional state \(E_{t+1}\) after listening to the utterance. The chosen metric is then the difference between the emotion estimation from the initial (just before [PERSON] starts speaking) and the last frames of each video sample. This metric is referred to as \(\widehat{\delta_{E}^{\dagger}}=(\delta_{\varphi},\delta_{a})\), where \(\delta_{\varphi}\), \(\delta_{a}\) is the change in the displayed valence and arousal. It is possible to see how a given utterance has affected research subjects in the valence-arousal space shown in Figure 6. If the valence improved, the vector is shown as blue; otherwise, as red. The original research hypothesis during the development of the Talk to Kotaro platform was that prosodic choice is the most important factor in generating emotional responses in listeners; since gibberish has no meaning, it was expected that volunteers would respond according to prosodic features. Furthermore, it was expected that there would be a cross-cultural preference for certain prosodic parameters, similar to the Bouba-Kiki effect [56]. To test this hypothesis, it is necessary to compute the correlation between the prosody parameters and \(\delta_{\varphi}\) and \(\delta_{a}\). This analysis was performed pairwise using [PERSON]'s \(\tau_{\mathbb{C}}\) correlation coefficient for each participant, all male volunteers, all female  
Figure 10: Scatter plots of the average arousal of each time volunteers M2, M8, F3, and F7 listened to a GS utterance and the results of the linear regression for each session and across multiple sessions. Points with the same color were obtained in the same session, and the line for that session shares the color with the points. volunteers, all Japanese nationals, and all Brazilian nationals; their correlation matrices are shown in Figure 11. The correlation coefficient was also calculated for other demographics, but for the sake of brevity, the matrices are not shown. It is very clear from Figure 11 that there is no statistically relevant correlation between the acoustic prosody characteristics and the generated impression for all volunteers, except for a very weak correlation between pitch and valence.