## 4 Results
### Impression Estimation from Video and Prosody Correlation  
In this subsection, the videos of the volunteers' facial expressions are analyzed and the impression \(\overrightarrow{I_{S}}\), the immediate emotional response to the speech act \(S\), is obtained by the vector \(\overrightarrow{\delta_{E}}=(\delta_{v},\delta_{a})\), which is obtained by subtracting the estimated emotional state of the initial and final frames of the video. In this way, a data set is generated that associates the speech acts and the human impression, allowing us to verify if there is a correlation between the prosody parameters and the impression \(\overrightarrow{I_{S}}\), and to use machine learning to obtain an embedding matrix for the IPA phones. This is achieved by using the VGG-16 and ResNet-18 neural networks described in Section 3.2. Plotting all obtained \(\widetilde{\delta_{E,S}}=(\delta_{a,S},\delta_{v,S})\) vectors, the impression caused by each speech \(S\) in the valence-arousal space yields Figure 6. However, since there are 734 vectors, it is difficult to visualize the results in valence-arousal space. Plotting the histograms of the results of the analysis, shown in the left and middle histograms of Figure 7, shows the results of the analysis performed on the video samples of the participants' reactions to each utterance spoken by [PERSON] during the entire experiment. It can be seen that many utterances had little effect on the participants' valence or arousal. However, if we calculate \begin{table}
\begin{tabular}{r r r r r r r r r r r r} \hline \hline
**Country/Region of Origin** & **Male** & **Female** & **All** & **Mother Language** & **Male** & **Female** & **All** & **Total Speakers** & **Male** & **Female** & **All** \\ \hline Japan & 9 & 10 & 19 & Japanese & 9 & 11 & 20 & English & 19 & 15 & 34 \\ Brazil & 6 & 1 & 7 & Portuguese (Brazil) & 6 & 1 & 7 & Japanese & 14 & 11 & 25 \\ Malaysia & 2 & 0 & 2 & Mandarin & 3 & 0 & 3 & Portuguese (Brazil) & 6 & 1 & 7 \\ China & 1 & 0 & 1 & Cantonese & 0 & 1 & 1 & Mandarin & 3 & 0 & 3 \\ Hong Kong (China) & 1 & 0 & 1 & English & 0 & 1 & 1 & Malaysian & 2 & 0 & 2 \\ India & 0 & 1 & 1 & Marathi & 0 & 1 & 1 & Arabic & 1 & 0 & 1 \\ Peru & 1 & 0 & 1 & Spanish & 1 & 0 & 1 & Cantonese & 1 & 0 & 1 \\ USA & 0 & 1 & 1 & Arabic & 1 & 0 & 1 & Spanish & 1 & 0 & 1 \\ Bangladesh & 1 & 0 & 1 & Sinhala & 1 & 0 & 1 & Sanskrit & 0 & 1 & 1 \\ Egypt & 1 & 0 & 1 & Bengali & 1 & 0 & 1 & Korean & 0 & 1 & 1 \\ Sri Lanka & 1 & 0 & 1 & & & & & & & & \\ Undisclosed & 0 & 1 & 1 & & & & & & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Breakdown of the cultural background of the participants. the norm of the \(\overrightarrow{\delta_{E}}\) vector for each speech act, we obtain the right histogram in Figure 7, which shows that many utterances caused little to no change in the emotional state of the listeners, but most still made an impression. The set of all \(||\overrightarrow{\delta_{E}}||\) has a mean of 0.124 and a standard deviation of 0.135. Since [PERSON]'s two-dimensional model of valence and arousal is defined over \(\{v\in\mathbb{R}|-1\leq v\leq 1\}\) and \(\{a\in\mathbb{R}|-1\leq a\leq 1\}\), where \(v\) and \(a\) are valence and arousal, respectively, the impression space is defined over \(\{\delta_{v}\in\mathbb{R}|-2\leq\delta_{v}\leq 2\}\) and \(\{\delta_{a}\in\mathbb{R}|-2\leq\delta_{a}\leq 2\}\). Thus, we can obtain another representation for all the impressions caused by [PERSON]'s utterances, shown in Figure 8, where outlier impressions are highlighted in red. Since the obtained impressions consist of two variables, we used the Mahalanobis distance  
Figure 6: Emotional state changes caused by every utterance in the “Talk to Kotaro” experiment in the valence–arousal space, where a blue arrow denotes a positive change in valence, a red one denotes negative valence change, and a grey triangle denotes no visible emotional change.