###### Abstract
When automatically generating appropriate prosody for synthetic semantic speech, it is possible to learn from prosodic databases extracted from natural speech which patterns are appropriate for a given input text to make it sound natural [9; 10] or to make a certain impression on the listener [11; 12]. However, a similar approach for gibberish speech is not as feasible since, to the best of the authors' knowledge, there is only one emotional speech database for gibberish speech, the EMOGIB data set [13], from which it would be possible to extract prosodic parameters. However, it consists only of words composed by phones present in English and Dutch, which limits the ability to develop prosody attribution systems from it. Furthermore, the emotional label present in the data set is the perceived feeling that the utterances convey, not how they made the listeners feel. To fill such gaps, the web-based crowdsourcing experiment \"Talk to [PERSON]\" [14] was conducted to investigate how IPA-based gibberish speech affect humans with distinct cultural backgrounds and to generate an data set with different acoustic prosody patterns, whose labels are the immediate emotion change the utterances caused on listeners in an open ended conversation setting. To achieve such a goal, we developed a web platform where volunteers talked to a screen-based ECA [15] inspired by the [PERSON] robot [16], which responded with gibberish speech. The developed website recorded both the audio of the volunteers' speech and the video of their facial expressions while listening to the ECA's utterances. A total of 37 volunteers from 8 different countries participated, speaking a total of 14 different languages; they contributed over 734 video samples. After the conversation with [PERSON], the volunteers were asked to fill out a Likert scale questionnaire, which was optional. The questionnaire was filled out by 22 participants. Thus, this paper presents the results of the analysis performed on the audiovisual data and questionnaire responses. It is the first paper to use IPA-based gibberish speech, which allows the results to be extended to many different cultures. We have found out that gibberish speech is not particularly engaging in a conversational setting, since most the average emotion estimates for different GS utterances had slightly negative valence and positive arousal. However, some utterances caused a positive average emotion and some positively impacted the emotional state of volunteers. Using such data, we have developed and trained a recurrent neural network-based architecture to predict the human impression of gibberish Speech. Moreover, the obtained data set suggests that for non-Yulean GS, there is little to no correlation between the acoustic prosody parameters and the emotion change on research subjects. Moreover, similar-sounding phones seem not to be close in the learned embedding space. However, more data are necessary to strengthen both claims. The structure of this paper is as follows: Section 1 is the Introduction; Section 2 is the Background, where the theoretical background necessary for understanding this paper is briefly explained, as well as related works; Section 3 is Materials and Methods, where the Talk to Kotaro experiment and the methods employed for analyzing the data obtained through it are presented; Section 4 is the Results, where the results of the analysis of the audio and video recordings and the Likert scale questionnaire responses of the experiment, as well as the results of the phone embedding, are presented; Section 5 is the Discussion, where the implications of the results of the analysis performed over the experiment data are showcased; and Section 6 is Conclusions and Future Work, where the conclusions from the present work and the future work to be developed with the data obtained are presented.