GSIP: A GRU-based system for human impression prediction
and automatic prosody selection for gibberish speech
Antonio Galiza Cerdeira Gonzalez1 and Ikuo Mizuuchi1
Abstract— This work presents the development andevaluation of GSIP (Gibberish Speech Impression
Predictor), a bi-directional GRU neural network
which serves as a human impression prediction model
for speech inputs, incorporating both phonetic information and prosody matrices. The objective is
to select appropriate acoustic prosody for gibberishand semantic speech. Experimental validation of the
proposed system was conducted through a user study.
Participants ranked the system’s performance against
constant prosody and random prosody patterns andcompleted adapted Godspeed scale questionnaires toassess their perception of the GSIP-based prosody
system and conversational agents. The experimentemployed three embodied conversational agents, two
screen-based avatars and a physical robot. The experiment found support for the understanding thatGibberish Speech is not so engaging for conversation, that higher anthropomorphism degrees createa higher perception of intelligence, and that the proposed system accurately predicts human impression,but failed to generate more engaging reactions thanconstant prosody.

I. Introduction
As societies across the globe move towards the Society 5.0 and Industry 4.0 paradigms, smart devices
with ever increasing computing capabilities and everreducing size are bound to become ubiquitous. Sinceverbal communication is still the most natural means ofcommunication for most humans, the shift from Graphic
user Interfaces to Conversational User interfaces [5] is
expected. Conversational agents – entities with artificial
intelligence capable of understanding human language
and generating appropriate responses – are bound to*This experiment has been approved by Tokyo Univeristy ofAgriculture and Technology’s ethics committee, under the approval
number No.230101-0453.
**Part of this work was supported by the Doctoral Program forWorld-leading Innovative & Smart Education of Tokyo university ofAgriculture and Technology (WISE Program of TUAT): ”Excellent
Leader Development for Super Smart Society by New Industry
Creation and Diversity” granted by the Ministry of Education,
Culture, Sports, Science and Technology (MEXT), Japan. Antonio
Galiza Cerdeira Gonzalez gratefully acknowledges the support fromthe Support for Pioneering Research Initiated by the Next Generation of FLOuRISH Institute,Tokyo University of Agriculture andTechnology (TUAT) granted by the Ministry of Education, Culture,
Sports, Science and Technology (MEXT), Japan.
***The authors would like to thank Professor Yukiko Horikiri
and Patricia McGahan for their invaluable support on recruiting
volunteers.
1 Mechanical
Systems Engineering department, Graduate
School of Engineering, Tokyo University of Agriculture andTechnology, Tokyo, Koganei, Nakacho 2-24-16 {antonio,ikuo}@mizuuchi.lab.tuat.ac.jp

have increased presence in industry and society as awhole. Such agents already take upon multiple roles,
such as personal digital assistants, eHealth advisors,
salesperson; among many others [4]. The aforementioned
agents may have a representation of a physical body(embodied conversational agents, ECAs) or not (software
agents) [3].
Such agents utilize speech synthesis systems to speak
and, thus, need to assign prosody to the resulting speech.
However, off the shelf speech synthesizers often fail togenerate adequate prosody in accordance to the content
of their speech, which leads to the perception from usersthat the speech of the ECAs feels unnatural [9]. Prosody
has many components, but this work is concerned withthe acoustic prosody parameters: speed, volume andpitch. Timbre is also an acoustic prosodic parameter,
but as it represents the quality of the voice, we do notconsider changing it in this research.
However, not all agents need to use semantic speech:
the use of gibberish in conversational agents offers a
key advantage in that it allows the communication ofemotions without the need for understandable language,
allowing to deploy a system across the language barrier.
The problem of assigning adequate prosody has been
approached in two ways, mainly: rule-based approachand a neural synthesis approach. Such approaches are
viable for semantic speech, since linguists can study anddevelop rules for assigning prosody for a given language
and there are audiovisual data sets from which neuralnetworks can be trained to assign prosody to text inputs.
However, such approaches cannot be used for GS, since
there are no speakers of gibberish speech (GS) nor data
sets from which appropriate prosody assigning can belearned. In order to investigate the effects of prosody andphone selection for gibberish, the web-based crowdsourcing experiment “Talk to Kotaro” was executed, fromwhich a data set of IPA-based GS and the impression– the immediate emotional change – they caused onlisteners was created, which is available in Mendeley
Data.
This work focuses in development and the evaluation abi-direction GRU neural network (architecture shown in
Figure 2), called called Gibberish Speech Impression Predictor (hereon referred to as GSIP), a human impression
prediction model for given input Speech S(t, p), where t isa vector containing the phones of the speech and p is theassociated ||t|| × 3 prosody matrix. The aforementioned
system is used for selecting appropriate acoustic prosodySpeech Content

Conversation Agents

a

camer

volunteer

KotaroAvatarconversational
computeragent
microphone

Gibberish
Semantic

Speech Style
table

Plantroid
Avatarconstant

Plantroid
Robot

GSIP

randomchair

Fig. 1. Left to right: experimental setup, ECAs, speech content
and styles for the GSIP validation experiment.

Valence/arousal predictionDense 64
Relu
Dense 1
LinearDense 256
Relu

Average layer

Dense 128
Relu
Dense 128
Relu
Bi-directional
GRU 1024
Dense 512
Relu

Dense 1
LinearDense 80
Relu
EmbeddingLayer 61x64

Dense 128
Relu

Proﬁle+Prosody
1x80
Vectorized
IPA phone input

Proposed GSIP modelfor gibberish and semantic speech and the experimental
validation of the proposed system by an experimentwere research subjects ranked the performance of thesystem against constant prosody and against random
prosody patterns and answered adapted Godspeed scalequestionnaires to rate their perception of the prosody
selection system and ECAs. Three distinct embodied
conversational agents were used in the experiment, which
are shown in Figure 1, along with the experiment setup.
The experiment was designed then, to verify the following hypotheses: H1 – speech generated by using theGSIP system is perceived as more natural than speechwith constant/random prosody; H2 – speech generated
by using the GSIP system generates more positive impression on volunteers than speech with constant or random prosody patterns; H3 – the system could generate
the desired impression in research subjects; H4 – test
subjects are more lenient with a non-humanoid looking
TABLE I
Adapted Godspeed Scale questionnaire for the Prosody
Generation Systems.
Artificial
Unfriendly
Unpleasant
Unintelligent
Apathetic

1
1
1
1
1

2
2
2
2
2

3
3
3
3
3

4
4
4
4
4

5
5
5
5
5

NaturalFriendly
Pleasant
Intelligent
Responsive

TABLE II
Adapted Godspeed Scale questionnaire for the ECA.
S1
S2
S3
S4
S5

II. Methods
A. GSIP prosody selection

Fig. 2.

S1
S2
S3
S4
S5

avatar regarding semantic-free speech and eventual bad
selection of prosody; H5 – the system can be successfully used for physical robots; H6 – the system can be
successfully used for semantic speech.
The structure of this work is as follows: Section I introduces the present work, Section II describes the proposedarchitecture and experiment; Section III analyzes anddiscusses the data obtained through the experiment and,
finally, Section IV lays out future research possibilities
and conclusions.

Unfriendly
Unpleasant
Unintelligent
Apathetic
Unkind

1
1
1
1
1

2
2
2
2
2

3
3
3
3
3

4
4
4
4
4

5
5
5
5
5

Friendly
Pleasant
Intelligent
Responsive
Kind

The proposed Gibberish Speech Impression Prediction was trained using the data obtained in the Talk
to Kotaro Experiment, using the profile information ofvolunteers, generated t and p of GS S(t, p) and theimpression I~S – immediate emotion change caused by
GS S. Since emotions are represented as coordinates in
Russel’s 2D emotion space [8], emotion changes can be
represented as vectors. Two models with the proposedarchitecture shown in Figure 2 were trained, one forpredicting valance changes and the other for predicting
arousal changes. The average error for the predicted
emotion change, measured by the norm of error vector
between the actual Impression vector and the predicted
Impression; was of 0.19.
In order to attribute prosody p to a string of IPA
phones t for a listener with a known profile P , the proposed GSIP system is used. Since the system took around0.2s to evaluate one prosody candidate, in order to not
generate an extremely significant delay in conversation
between humans and ECAs, in a Monte Carlo fashion,
10 prosody candidates were randomly selected inside theprosody space for the eSpeak speech synthesizer, that isvolume between 10% and 200% of the regular volume,
80 and 450 words per minute for the speed and 0 and
99 for the pitch. The 10 candidates were then evaluatedand the best performing one was chosen. For gibberish
speech, each word received a different prosody.
However, for semantic speech, the phrases were split
into “chunks” by commas, semicolons, periods, exclamation and interrogation marks. The chunks were, then,converted into their IPA phone representation and the
GSIP system was used to attribute prosody to all
chunks.The same chunking system was used for the otherprosody selection systems.
B. Experiment
To test hypotheses Hn , n = 1 . . . 6 we designed a twophase experiment (P1 for n = 1 . . . 5and P2 for n = 6)
in which volunteers engage in brief open-ended conversations with all ECAs shown in Figure 1. For each phase,volunteers held conversations consisting of 10 exchanges
3 times per agent; each time the agents used one of the
prosody 3 prosody selection systems: constant, GSIP and
random. One exchange is defined and the segment of thelarger conversation where a participant says something
and the agent replies. In P1 , agents replied with GS, whilein P2 agents replied in English. For the generation of the
responses in English OpenAI’s GPT-3 davinci model wasused.
The first five exchanges of each conversation in P1
were fixed utterances (1. baɪɹˈuːleɪ, 2. mˈahɐθɹˌuːɛθi, 3.
hˌaɹɐtˈɑːkuː, 4. ʌshbaɭˈimlcɡ and 5. qdˈo) and the first
three exchanges of all conversation during P2 were fixedquestions that received fixed answers from the agents (1.
“What is the current temperature?” - “The temperaturein this room is of 26 degrees Celsius.”, 2. “How long until
I need to water the plant? ”- “I estimate that you willneed to water the plant in 2h15.”, and 3. “What is thesalinity of the soil?” - “The current salinity of the soil is of
2 deci-Siemens per meter; within the safe range for yourplant.”), in order to allow us to have fixed utterancesin order to better compare reaction of volunteers. The
remaining 5 utterances for P1 were randomly generated
and the remaining 7 exchanges for P2 were open ended,that is, the GPT-3 model generated appropriate answers(most of the time) to the queries of participants.
After having a conversation with one agent using
one of the prosody selection systems, participants had
to answer a shortened version of the Godspeed scalequestionnaire QS (shown in Table I) about their impression of the speech style of the agent. After talking
with an agent 3 times, covering all prosody generation
systems, a volunteer had to answer the adapted Godspeed scale questionnaire QA (shown in Table II) about
their perception of the agent. After that, volunteers had
to rank which system they preferred. After talking toevery agent, volunteers had to rank which agent they
preferred interacting with. The ranking system consistedof ranking the system or agent by the order of interaction,for example, if a volunteer preferred the first system tosecond, but though that the second and third had similar
performance, would answer the ranking questionnaire as
1 > 2 = 3. The volunteers were not told the names ofthe systems in order to avoid any sort of bias.
Shortening the Godspeed scale questionnaires was necessary, given that for Phases P1 and P2 of the experiment,
volunteers had to answer many questionnaires. Thus,
it is necessary to verify their internal consistency by
calculating the Cronbach’s alpha for the responses of thequestionnaires. We obtained an α of 0.828 for GS and0.825 for QA, good internal consistency values.
All interactions were recorded by a condenser microphone and a 4K camera, allowing us to perform emotion
estimation from the voices and facial expressions ofvolunteers. A VGG-16 DCNN and a ResNet18 NN [7]were used for estimating the emotion in terms of valence
and arousal [8] from the facial expressions shown by
volunteers while listening to the speech of the ECAs,
whose results for each one of the systems are shown in
Table IV

Fig. 3. Responses to the adapted Godspeed scale questionnaires
about the prosody selection systems.

Fig. 4. Responses to the adapted Godspeed scale questionnaires
about the ECAs.

III. Results and discussion
The experiment was conducted at the Tokyo University of Agriculture and Technology (hereafter referred toas TUAT) between February 13th and March 3rd of 2023,
with a total of 28 participants, of which 27 could havetheir data used. Of these 27 participants (14 women and
13 men). All participants were recruited among TUAT
students, with a mean age of 25.56 years (standard
deviation: 4.05 years). The youngest participant was 19
years old and the oldest was 33 years old. The nationalitythat had the most participants was Japanese, with a totalof 5 volunteers, and Malaysian was second with a total of
4 volunteers. This section presents the results, analyzes
them, and discusses their implications.
From the adapted questionnaires, we could obtain theoverall perception of the prosody selection systems forphases P1 and P2 , shown in Figure 3 and the overall
perception about the ECAs, shown in Figure 4.
TABLE III
Total of 1st , 2nd and 3rd place rankings for each system
and ECAs.
Phase
P1
P1
P1
P2
P2
P2

System
Const
GSIP
Rand
Const
GSIP
Rand

1st
54
28
49
52
21
20

2nd
20
28
25
13
36
33

3rd
4
22
4
4
12
16

Agent
KotaroScreenRobot
KotaroScreen
Robot

1st
11
13
13
11
5
9

2nd
8
9
11
6
12
4

3rd
7
4
2
2
2
6

It is possible to notice that GS, independently from theprosody selection system, is perceived as less natural, lessfriendly, less pleasant and less intelligent then semantic
speech, by comparing the results for all systems between

TABLE IV
Average estimated emotion for every system, agent andphase.
Phase
P1
P2
All
All
All
P1
P1
P1
P2
P2
P2

Agent
All
All
Robot
ScreenKotaro
All
All
All
All
All
All

System
All
All
All
All
All
Const
GSIP
Rand
Const
GSIP
Rand

Avg (val, aro)
-0.3171, 0.1385-0.2897, 0.1245-0.2704, 0.1378-0.3230, 0.1264-0.3163, 0.1303-0.3031, 0.1359-0.3226, 0.1406-0.3255, 0.1389-0.2833, 0.1231-0.2828, 0.1231-0.3039, 0.1275

Avg duration
0.33932.5983
1.57571.62791.23710.21900.43290.3654
1.87693.12192.8135TABLE V
Total of first, second and third place rankings for eachuser obtained through the video estimation analysis.
P1
GSIP
RandomConstant

1st
7
7
10

2nd
9
9
6

3rd
8
8
8

P2

1st
8
4
12

2nd
6
1
6

3rd
11
8
5

P1 and P2 . Also, the perception of the agents also
improved between the phases.
Regarding the research hypotheses, no support was
found for H1 since there is no difference in the naturalness scale for P1 and is perceived as as less responsive
and is the least natural in P2 . From the video analysis the
GSIP-based prosody selection system does not generate
a better experience than constant prosody in P1 , but isbetter than Random prosody. For P2 , GSIP has a better
average performance, but in a per-user basis, result isthe same for P1 . One of the possible reasons for such
performance is that the proposed system introduced adelay in the responses of the agents. Another reason isthat since few candidates were evaluated, good prosody
candidates might not have been selected.
For H2 , it is possible to see in Table IV that for
GS, the GSIP-based prosody selection system does not
generate a better experience than constant prosody,despite generating more positive emotions that Random
prosody. However, for semantic speech, GSIP seems tohave a better average performance, but, by breaking theperformance down to a per-user basis (Table V), we get
the same results of P1 .
H3 , GSIP successfully generated the desired impression on volunteers, even outperforming the results obtained for training data (0.17 average error);
We can see that H4 also does not hold, since for P1
volunteers evaluated the pet-like agents worse.
About H5 , the robot agent was perceived to perform
as well as the avatar and even better than it for the
Friendliness scale, showing that the system can be usedin robots.
Finally, for H6 , since the system has not performed
well in P1 , it was very surprising to see that the performance has improved in P2 . It received the lowest ranking
among all systems for P1 , as one can see in Table III, but

for P2 the perceived improved, being ranked better than
the random prosody selection system.
IV. Conclusions and Future works
This paper proposed the GSIP impression prediction
system and an experiment to investigate its efficacy
for gibberish and semantic speech. It has used adapted
Godspeed Scale questionnaires, ranking questionnaires,and emotion estimation from the video of the interaction
with ECAs to measure the perception of volunteers about
gibberish and semantic speech with constant, random
and GSIP-selected prosody. It supports previous findingsthat Gibberish is not very engaging in conversation [10]and that humanoid-like ECA tend elicit a higher intelligence expectation in users [11].
While the proposed method method did not succeed
in generating more engaging prosody that the constantprosody, the GSIP system was proven to provide accurate
prediction and, thus, it is necessary to re-evaluate how
to better use it for prosody generation.
References[1] Gonzalez, A. G. C., Lo, W., & Mizuuchi, I. (2022, August).
Talk to Kotaro: a web crowdsourcing study on the impact ofphone and prosody choice for synthesized speech on humanimpression. In 2022 31st IEEE International Conference on
Robot and Human Interactive Communication (RO-MAN)(pp. 244-251). IEEE.
[2] Hirschberg, J. (2002). Communication and prosody: Functional aspects of prosody. Speech Communication, 36(1-2), 3143.
[3] Lester, James, Karl Branting, and Bradford Mott. ”Conversational agents.” The practical handbook of internet computing(2004): 220-240.
[4] Ter Stal, S., Kramer, L. L., Tabak, M., op den Akker, H.,& Hermens, H. (2020). Design features of embodied conversational agents in eHealth: a literature review. International
Journal of Human-Computer Studies, 138, 102409.
[5] Janarthanam, Srini. Hands-on chatbots and conversational UIdevelopment: build chatbots and voice user interfaces with
Chatfuel, Dialogflow, Microsoft Bot Framework, Twilio, andAlexa Skills. Packt Publishing Ltd, 2017.
[6] Dale, R. (2021). GPT-3: What’s it good for? Natural Language
Engineering, 27(1), 113-118. doi:10.1017/S1351324920000601[7] Dimitrios Kollias and Stefanos Zafeiriou. A multi-component
CNN-RNN approach for dimensional emotion recognition inthe-wild. arXiv preprint arXiv:1805.01452, 2018.
[8] James A. Russell. A circumplex model of affect. Journal of
personality and social psychology, vol. 39, no. 6, pp. 1161, 1980.
American Psychological Association.
[9] Ehret, J., Bönsch, A., Aspöck, L., Röhr, C. T., Baumann, S.,
Grice, M., ... & Kuhlen, T. W. (2021). Do prosody and embodiment influence the perceived naturalness of conversational
agents’ speech?. ACM Transactions on Applied Perception(TAP), 18(4), 1-15.
[10] Kumagai, K., Hayashi, K., & Mizuuchi, I. (2017, December).
Hanamogera speech robot which makes a person feel a talking
is fun. In 2017 IEEE International Conference on Robotics andBiomimetics (ROBIO) (pp. 463-468). IEEE.
[11] Bartneck, C., Croft, E., & Kulic, D. (2008). Measuring theanthropomorphism, animacy, likeability, perceived intelligenceand perceived safety of robots.