Experiment Plan

Development of a Human Impression Predictionbased Prosody Selection System for Synthesized
Speech of Conversation Agents.

Student: Antonio Galiza Cerdeira Gonzalez
Student ID: 20833651
Supervisor: Professor Ikuo Mizuuchi
Submission date: February 10, 2024

RESTRICTED CIRCULATIONThis research experiment plan contains preliminary results of unpublished research and, thus, must not be shared with anyone besides members of Mizuuchi Lab,of the Ethics Committee and related Tokyo University of Agriculture and Technology
staff.

i

Contents
1 Introduction
1.1 Research questions/hypotheses . . . . . . . . . . . . . . . . . . . . . .

1
2

2 Experiment Plan
2.1 General Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3 Experiment explanation phase . . . . . . . . . . . . . . . . . . . . . . .
2.4 Breaks between phases . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.5 Phase 1 (P1 ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.6 Phase 2 (P2 ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.7 Phase 3 (P3 ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3
3
4
6
7
7
8
9

3 Collected Data

10

4 About Research Volunteers
4.1 Target demographic . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 Number of Participants . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3 Duration of the experiment . . . . . . . . . . . . . . . . . . . . . . . .
4.4 Recruiting methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.5 Location of the experiment . . . . . . . . . . . . . . . . . . . . . . . . .
4.6 Burdens to research volunteers and compensation . . . . . . . . . . . .
4.7 Protection of the Physical and Mental Integrity of Research Volunteers
4.8 Protection of the anonymity of Research Volunteers . . . . . . . . . . .

11
11
11
11
11
11
12
12
12

A Explanation Document

14

B Consent Form
20
B.1 Research Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
B.2 Voluntary research cooperation and freedom of withdrawal . . . . . . . 20
B.3 Research duration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
B.4 Benefits and burdens for study subjects . . . . . . . . . . . . . . . . . . 22
B.5 Protection of personal information . . . . . . . . . . . . . . . . . . . . 22
B.6 Method of Disclosure of Research Protocols and Information on Research 23
B.7 Disclosure of the results of the subject’s own data . . . . . . . . . . . . 23
B.8 Publication of Research Results . . . . . . . . . . . . . . . . . . . . . . 23
B.9 Attribution of Intellectual Property Rights arising from Research . . . 23
B.10 Data Handling Policy after research completion . . . . . . . . . . . . . 23
B.11 Research funding and conflict of interest . . . . . . . . . . . . . . . . . 24
B.12 Contact Information: . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
B.13 Consent for cooperation in the experiment . . . . . . . . . . . . . . . . 24
C Volunteer Profile Form

25

D Questionnaires
D.1 Godspeed Questionnaire for Prosody Selection Systems . . . . . . . . .

25
25

ii

D.2 Godspeed Questionnaire for Agents . . . . . . . . . . . . . . . . . . . .
D.3 Communication Systems Ranking . . . . . . . . . . . . . . . . . . . . .
D.4 Agents Ranking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

26
26
27

E Selected Semantic-Free Utterances, and Questions and their respective responses
28
E.1 Previously generated Gibberish Speech . . . . . . . . . . . . . . . . . . 28
E.2 Previously generated questions and their answers . . . . . . . . . . . . 29

iii

Figure 1: Valence-arousal wheel.

1. Introduction
In order to make robotic speech more interesting to human listeners, the research
titled “Image-based emotion detection system to improve human-robot communication” is currently being developed in Mizuuchi Lab. The first step of the aforementioned research consisted of the Talk to Kotaro experiment, where volunteers held
conversation online with a robot avatar of the Kotaro robot. The website recorded
what research subjects told the robot avatar and filmed their facial expressions while
they listed to the semantic free utterances – speech composed from human phones but
without meaning – Kotaro replied with. The experimented yielded enough data to allow the development of a module called GSIP – Gibberish Speech Impression Predictor.
GSIP is capable of predicting the impression – immediate emotional response – of humans after listening to a semantic-free utterance spoken with a given prosody pattern.
Its predictions are in terms of valence and arousal. Valence is the measure of how positive a given emotion is and it is defined over {v ∈ R| − 1 ≤ v ≤ 1}, -1 being the worst
possible emotion and 1 being the most positive possible. Arousal measures how intense
the displayed emotion is and its values are defined over {a ∈ R| − 1 ≤ a ≤ 1}, -1 being
the most apathetic reaction and 1 being the most intense possible. The valence-arousal
space is shown in Figure 1.
With the help of the proposed GSIP system, robots will be able to generate engaging prosody parameters for pre-generated or randomly generated text of the semanticfree utterances (SFU). Batches of candidate prosodic patterns are randomly created
and the [SFU, prosody] pairs are evaluated by the GSIP module – the pair which
1Figure 2: Proposed architecture for the prosody Selection system.
is predicted to generate the most positive impression (although other criteria can bechosen) is selected and spoken by the robot. The structure of the system is shown in
Figure 2.

1.1 Research questions/hypotheses
The proposed system has been already developed but it is necessary to validate
its performance experimentally and, thus, this Research Experiment Plan was devisedin order to allow us to validate the hypotheses listed below:• H1 – speech generated by the proposed system is perceived as more human-likethan speech with constant prosody or with randomly generated prosody;• H2 – speech generated by the proposed system generates more positive impression
on volunteers than speech with constant or random prosody patterns;• H3 – the system could generate the desired impression in research subjects;• H4 – test subjects are more lenient with a non-humanoid looking avatar regarding
semantic-free speech and eventual bad selection of prosody.
• H5 – the system can be successfully used for physical robots;• H6 – the system can be successfully used for semantic speech;• H7 – novelty bias plays a heavy factor when users consider physical robots moreengaging than virtual agents.
It is necessary to clarify that, in H3 , generating the desired impression on volunteers is to obtain a valence-arousal impression with an mean absolute error (henceforthreferred to as MAE) no larger than 0.15 (obtained with current data).
This research experiment plan, thus, explains the workflow of the experiments
to be executed, what data will be recorded, how it will be recorded, stored and how
research subjects will have their mental and physical integrity and anonymity protected.

22. Experiment Plan
2.1 General Overview
Since there are seven hypotheses to be experimentally tested for the developed
system, the desired experiment is divided in three experimental phases P1 , P2 and P3 ,which are expected to last around 35 minutes each, totaling an 1h45 of experiment foreach research subject. However there will be an initial explanation of the experiment
where volunteers are free to ask any question. If they consent participating on theexperiment, they will sign the consent form. As users may ask as many questions asnecessary, it is hard to guess how long this preliminary phase might last but, if there are
no questions, it is expected to last around 15 minutes. Between every experiment phase,participants will be given around 5 minutes to rest (more if they deem it necessary).
Finally, by the end of the experiment, we will confirm that the volunteer still wants
to have the data contributed in the experiment used for research fort a final time; thisis expected to be very brief and to not take more than 1 or 2 minutes. This way, thetotal experiment time goes up to around 2h12, but it might be faster depending how
quickly a subject answers questionnaires.
In order to test hypotheses H1 , H2 , H3 , H4 and H5 , the first experiment phase
P1 was designed. P1 consists of having research subjects holding conversations withthree different conversation agents A1 , A2 and A3 , which will employ prosody selection
systems c1,gs , c2,gs and c3,gs for semantic free utterances. c1,gs always use the sameprosody regardless of what the gibberish speech is; c2,gs is the system where prosody
is selected using GSIP and, finally, c3,gs randomly selects prosody characteristics forsemantic speech. Agent A1 consists of the same avatar of the Kotaro robot used in the
Talk to Kotaro Experiment; A2 consists of a 2D avatar of the Plantroid Robot, which
resembles an animal A3 is the physical Plantroid robot. All conversation agents areshown in Table 1 The order in which volunteers talk to the agents will be randomly
selected in order to avoid order bias; and the order on which the conversation agentswill use the prosody selection systems will also be randomly selected for the same
reason.
Every agent will hold a 10 exchange-long conversation for each one of the prosody
selection systems, totaling 30 exchanges per agent and 90 in total for P1 . An exchange
is defined as a segment of a longer conversation, consisting of an initial saying by the
research volunteer and a response from a conversation agent. Research subjects arefree to say whatever they want to the conversation agent, which will reply using pregenerated semantic free utterances during the first five exchanges. For the remaining
five utterances, the first conversation agent will generate them using GSIP for the firstprosody selection system it uses and they will be used for every other prosody selection
system and conversation agent.
In order to test hypothesis H6 , experiment P2 will perform the very similar tests
performed in P1 , but the same conversation agents will use prosody selection systems
c1,ss , c2,ss and c3,ss , where c1,ss applies the same prosody pattern for any semantic
speech inputs, c2,ss uses GSIP to select the prosody characteristics of the speech andc3,ss randomly selects the prosody patterns. Volunteers will ask 3 given questions
3

Figure 3: Proposed experiment phases and its components.
about the health of a plant and the conversation agents will answer with pre generated
speeches. The remaining 7 exchanges are free; research subjects can talk about any
topic with the conversation agents.
The final experiment phase P3 consists of letting test subjects freely talk withconversation agents A2 , A3 and A4 . A4 is a holographic 3D model of the Plantroid
robot shown using a Looking Glass holographic display. This experiment phase will
test hypothesis H7 , which postulates that the common perception that virtual agentsare always less engaging than a physical embodied robot is because of novelty bias,
which will be tested by introducing the novelty of interacting with a 3D hologram. If
A4 is considered to be as engaging or more engaging than A3 , hypothesis H7 will bevalidated. If the interactions with A3 are found out to be more engaging than thosewith A2 , but less than those with A3 , it will show a strong support for novelty bias,
but will validate the previous conception that physical embodiment of agents are moreengaging than virtual agents. If it is found to be less engaging than A2 , it will showthat novelty plays little to no role into the interactions and a very strong support tothe current conceptions of physical agents being more engaging. The prosody selection
system for the responses of the different actors is c1,ss , that is, same constant prosody.
After talking with every conversation agent, subjects will be asked to answer threequestionnaires (Godspeed anthropomorphism and likeability) and will be asked if theywant to rest, continue or give up the experiment. After all Phases, volunteers will be
asked a final time for if they still consent to have their data used for the research.
The experiment Phases will be described in more detail along with the questionnaires
in the following sections. The flow of the experiment can be seen in Figure 3. Allconversation agents are listed in Table 1 and every prosody selection system is show inTable 2.

2.2 Experiment Setup
The setup of the experiment, shown in Figure 4, consists of having a volunteer
sitting down in front of a table where the conversation agent will be right in front of theeyes of the volunteer. Conversation Agents A1 and A2 will be shown in a LCD monitor.
When Agent A3 or A4 are interacting with volunteers, the monitor will be removed andreplaced by the physical Plantroid robot and by the Looking Glass holographic display.
4

Name

Description

A1

Avatar of the robot Kotaro.

A2

Avatar of the robot Plantroid.

A3

Physical Plantroid robot.

A4

Holographic Plantroid robot.

Image

Table 1: Conversation agents, their descriptions and images.

Name
c1,gs
c2,gs
c3,gs
c1,gs
c2,gs
c3,gs

Description
Constant prosody parameters for gibberish speech.
Prosody parameters of gibberish speech selected using the GSIP system.
Prosody patterns randomly selected for gibberish speech.
Constant prosody parameters for semantic speech.
Prosody parameters of semantic speech selected using the GSIP system.
Prosody patterns randomly semantic for gibberish speech.

Table 2: Prosody parameters selection systems and their descriptions.

5Figure 4: Experiment Setup common for all phases.
A computer, used for generating the prosody parameters and showing agents A1 , A2
and A4 on their respective displays will also be above the table. An USB camera anda microphone will be placed in positions appropriate to capture the facial expressionof users and their speeches while not obstructing their view of the conversation agents.
Volunteers will be provided with a pen for answering questionnaires and signing thenecessary forms. This configuration is common to every phase of the experiment.

2.3 Experiment explanation phase
Research volunteers will receive a thorough explication (both in text and oral
form) on how the experiment will happen, together with a brief demonstration on how
to interact with the conversation agents. In case the subject is still willing to partakein the experiment, two forms will be presented: the consent form and a profile form,
which will be used for the experimental setup and for post-hoc analysis of the research.
Volunteers will be informed both in the explanation and consent form that they can stopthe experiment at any time for any reason they deem appropriate. The explanation
document is fully presented in Appendix A, the consent form is fully presented inAppendix B and the profile form is presented in Appendix C. After finishing filling andsigning the forms, Phase P1 of the experiment will begin.

6

2.4 Breaks between phases
Since every phase of the experiment is expected to be quite long, there will bebrief break periods where research volunteers can stretch, move around, use the toilet
and drink water. The users will be asked if they desire to continue with the experiment.
In case users want to give up, they will be asked if the data recorded until that point
can still be used for the research. If the answer is negative, all recorded files will bedeleted in front of the subject and the questionnaires destroyed in a paper shredderand subsequently discarded. If research subjects want to continue, they will be told
once more that they are free to give up participation or to request a pause according
to their necessity and, whenever they feel ready, next phase will start.

2.5 Phase 1 (P1 )Phase one consists of having the research volunteer holding ten exchange-longconversations with three distinct conversation agents A1 (avatar of the Kotaro robot),A2 (avatar of the Plantroid robot) and A3 (the real Plantroid Robot), all of which will
use three distinct prosody selection system, c1,gs , c1,gs and c3,gs for their semantic freeresponses. In order to avoid any bias caused by having volunteers talk to the agent in
the same order, the first agent with whom a volunteer will be selected among A1 , A2
and A3 . For example, suppose that A2 was selected as the first agent. A2 will, then,randomly select a prosody generation system among c1,gs , c1,gs and c3,gs . Suppose c1,gs
was chosen. The volunteer will have, then, ten exchanges with A2 , which will reply forthe first five times with pre-generated semantic-free utterances (showcased in Table 3),
whose prosody will be always the same, accordingly to c1,gs . The remaining five utterances will be randomly generated in this first interaction and reused for the subsequentinteractions for the other prosody generation systems and for the other conversation
agents. After the volunteer has talked with A2 for ten exchanges, the subject has to filla very brief Godspeed questionnaire, shown in Appendix D: Godspeed Questionnaire.
After that, the next prosody selection system will be randomly selected among theremaining two – the process is repeated for the next ten exchanges. Finally, the volunteer needs to talk to A2 using the remaining prosody selection system. After filling
the last Godspeed questionnaire, the volunteer will be asked to rank which prosody
selection system he/she felt had the most human-like and most engaging speech. The
same procedures will be repeated for the next conversation agent, which will be randomly selected among the remaining two. Finally, it will be repeated once again for
the remaining agent. After interacting with every conversation agent, the volunteer
will be asked to rank the most engaging conversation agent. An example of every stepin of Phase 1 would be:
1. A2 , c1,gs selected, volunteer interacts for 5 exchanges;
2. A2 , c1,gs generates remaining 5 exchanges;
3. Volunteer answers Godspeed questionnaire about c1,gs ;4. Selects A2 , c3,gs , volunteer interacts for 10 exchanges;;
5. Volunteer answers Godspeed questionnaire about c3,gs ;6. Selects A2 , c1,gs , volunteer interacts for 10 exchanges;
77. Volunteer answers Godspeed questionnaire about c1,gs ;8. Volunteer ranks most engaging prosody system;
9. A1 , c3,gs selected, volunteer interacts for 10 exchanges;
10. Answer Godspeed questionnaire about c3,gs
11. Selects A1 , c1,gs , volunteer interacts for 10 exchanges;
12. Answer Godspeed questionnaire about c1,gs
13. A1 , c1,gs selected, volunteer interacts for 10 exchanges;
14. Answer Godspeed questionnaire about c1,gs
15. Volunteer ranks most engaging prosody system
16. A3 , c1,gs selected, volunteer interacts for 10 exchanges;
17. Answer Godspeed questionnaire about c1,gs
18. A3 , c3,gs selected, volunteer interacts for 10 exchanges;
19. Answer Godspeed questionnaire about c3,gs
20. A3 , c1,gs selected, volunteer interacts for 10 exchanges;
21. Answer Godspeed questionnaire about c1,gs
22. Volunteer ranks most engaging prosody system;
23. Volunteer ranks most engaging agent.

2.6 Phase 2 (P2 )Phase 2 is quite similar to P1 in many aspects, however, conversation agents A1 ,A2 and A3 will reply using semantic speech, assigning prosody to it using systems c1,ss ,c2,ss , and c3,ss . Conversations with the (agent-prosody selection system) pairs consistsof three exchanges where users will be request to ask the same predefined questions(show in Table 4), which will receive pre-generated responses and seven exchanges
where they are free to say anything they want to the conversation agents, which will
reply using a GPT-2-based chatbot. The chatbot used for every interaction is always
the same. The remainder of the phase happens exactly like in phase 1 and an exampleof how a phase 2 could happen is given below:
1. A1 , c1,ss selected, volunteer interacts for 3 exchanges;
2. volunteer freely interacts with A1 , c1,ss for 7 additional exchanges;
3. Volunteer answers Godspeed questionnaire about c1,ss ;4. Selects A1 , c3,ss , volunteer interacts for 3 exchanges;
5. volunteer freely interacts with A1 , c3,ss for 7 additional exchanges;
6. Volunteer answers Godspeed questionnaire about c3,ss ;7. Selects A1 , c2,ss , volunteer interacts for 3 exchanges;
88. Volunteer freely interacts with A1 , c2,ss for 7 additional exchanges;
9. Volunteer answers Godspeed questionnaire about c2,ss ;10. Volunteer ranks most engaging prosody system;
11. A2 , c2,ss selected, volunteer interacts for 3 exchanges;
12. Volunteer freely interacts with A2 , c2 , ss for 7 additional exchanges;
13. Volunteer answers Godspeed questionnaire about c2,ss ;14. Selects A2 , c3,ss , volunteer interacts for 3 exchanges;
15. Volunteer freely interacts with A2 , c3,ss for 7 additional exchanges;
16. Volunteer answers Godspeed questionnaire about c3,ss ;17. Selects A2 , c1,ss , volunteer interacts for 3 exchanges;
18. Volunteer freely interacts with A2 , c1,ss for 7 additional exchanges;
19. Volunteer answers Godspeed questionnaire about c1,ss ;20. Volunteer ranks most engaging prosody system;
21. A3 , c3,ss selected, volunteer interacts for 3 exchanges;
22. Volunteer freely interacts with A2 , c3,ss for 7 additional exchanges;
23. Volunteer answers Godspeed questionnaire about c3,ss ;24. Selects A3 , c3,ss , volunteer interacts for 3 exchanges;
25. Volunteer freely interacts with A3 , c1,ss for 7 additional exchanges;
26. Volunteer answers Godspeed questionnaire about c1,ss ;27. Selects A3 , c2,ss , volunteer interacts for 3 exchanges;
28. Volunteer freely interacts with A3 , c2,ss for 7 additional exchanges;
29. Volunteer answers Godspeed questionnaire about c2,ss ;30. Volunteer ranks most engaging prosody system;
31. Volunteer ranks most engaging agent.

2.7 Phase 3 (P3 )Phase P3 is held in order to verify if novelty bias plays a heavy role on humanimpression regarding interacting with virtual agents and physical agents, that is anagent in a display and the physical robot. Current consensus is that embodiment is very
important to make agents more engaging, however, there has been little considerationof the role of novelty bias in that conclusion, that is, since most people have hadvery few interactions with robots, they consider interacting with a physical agent moreinteresting than interacting ones with screen-based virtual agents. To verify that, aholographic display will be used to show virtual agent A4, a 3D model of Plantroid
robot. Since most people have not interacted with holographic agents, this introduces9a novelty factor that might make research subjects feel more engaged while holding
conversations with a virtual agent. In order to increase engagement, only semantic
language exchanges will happen beteween the agents and research subjects. However, inorder to not introduce prosody as an experience-changing factor, the prosody selection
system c1,ss – constant prosody – will be used by all agents. Once again, to reduceorder bias, the sequence on which research subjects will talk to the conversation agentswill be determined at random and hold a five exchange long conversation. An exampleof how Phase 3 could happen is as follows:• Selects A2 , c1,ss , volunteer interacts for five free exchanges;• Selects A4 , c1,ss , volunteer interacts for five free exchanges;• Selects A3 , c1,ss , volunteer interacts for five free exchanges;• Volunteer answers Godspeed questionnaire about agents• Volunteer ranks most engaging agent.

3. Collected Data
For the experiment, user information will be stored in many different forms. At
the beginning of the experiment, a volunteer needs to fill two copies of the consent
form, one which the research subject will take home and another which will be kept
in Mizuuchi Lab, in a locker drawer. After that, it is necessary to fill a form about
his/her personal information, specifically:• Age;• Gender;• Country/Region of Origin;• Mother Language;• Other Languages you Speak;• If you live or have lived abroad, write where;• Years living abroad.
Such form will be filled on paper and, later, digitized. The original paper forms
will be destroyed immediately after digitization.
During the experiment, audio of what volunteers say to the conversation agentsand of the replies of the conversation agents will be recorded by a microphone, while
videos of the research subject’s facial expressions will be recorder by an USB camera.
After every interaction with a conversation agent using a given prosody selection system, volunteers need to fill a Godspeed questionnaire; and after interacting with every
prosody selection system, they need to rank which systems were the most engaging,
human-like etc. After speaking to every conversation agent, research subjects need toanswer an identical questionnaire, this time for ranking the conversation agents. All
aforementioned questionnaires will be initially recorded initially on paper, but they10

will be immediately digitized once the experiment is over. The original questionnaire
responses will be destroyed using a paper shredded machine.
All digital files will be encrypted using a state of the art AES-256 encryption
algorithm and stored in Mizuuchi Lab servers for no longer than 3 years, after which,such files will be permanently deleted using BleachBit, a program which is used tosecurely delete files by overwriting the section of the hard drive were they were stored
many times, making it impossible to recover the files even with advanced forensic tools.

4. About Research Volunteers
This section describes many aspects regarding potential research volunteers, suchas who can participate, necessary number of participants, what burdens are expected
to be incurred to participants and how they will be compensated for their efforts.
Moreover it will be explained how the psychological and physical integrity are going to
be protected, together with their privacy.

4.1 Target demographic
Any person aged 18 or older can participate on the experiment if they are able tospeak and understand a basic level of English language, since Phases P2 and P3 require
talking with English speaking conversation agents.
Research subjects must also be willing to participate without wearing a face maskfor extended periods of time. This way, vaccinated participants are desired.

4.2 Number of Participants
In order for the result to be statistically relevant, the experiment requires between
18 and 32 participants. The goal is to have as many as possible, but if there are lessthan 18 participants, it might be difficult to draw any strong conclusions from theexperiment.

4.3 Duration of the experiment
The current experiment plan will take place in one week, starting when the ethics
committee approves the experiment (preferentially at the beginning of January).

4.4 Recruiting methods
Announcements about the experiment, its contents and the 2,000 yen reward will
be made in social media (Facebook, Instagram, Twitter and LinkedIn), TUAT BBS,TUAT mailing lists and by means of recruitment posters to be posted in Seikyo.

4.5 Location of the experiment
The experiments will be executed in Room 208 of building 14 of TUAT Koganei
Campus, which belong to Mizuuchi Lab. It was chosen because it has a table with11

suitable chairs and it is an environment with good lightning, allowing the camera tocapture the facial expressions of research volunteers very well.
Full address: Tokyo to, Koganei shi, Naka cho 2-24-16 Building 14, Room 208,
Mizuuchi Lab.

4.6 Burdens to research volunteers and compensation
Research volunteers are not expected to contribute with anything besides circa
2h15 of their free time, for which they will be compensated with a 2,000 yen Amazon
Gift Card.

4.7 Protection of the Physical and Mental Integrity of Research Volunteers
The experiment poses no direct threat to the physical health of participants,
since the will not physically interact with any experimental equipment. As the wholeexperiment is expected to be somewhat long, volunteers are able to request pauses
for any reason at any moment or to request ending the experiment earlier. Between
every phase, volunteers will be granted time to move, stretch, use the toilet and to takecare of any other necessities. While it is quite safe to assume that the mental wellbeing of participants will not be threatened by semantic-free utterances, the dataset
used to train the GPT-2-based chatbot was curated in order to remove any possible
offensive phrases, safeguarding the mental well-being of participants. Once again, every
volunteers will be informed that he/she can give up the experiment at any given time,in case participants feel too tired, irritated or for any reason.
4.7.1 Special measures for preventing COVID-19 infection.
Since research subjects will need to talk to the conversation agents without wearing a face mask, special measures for the protection of participants need to be taken.
The researcher supervising the experiment will wear a N95 mask and a face-shield,
to avoid contamination and spreading of COVID-19. The room in which experiments
will be held has windows and good sunlight illumination. The windows will be open
during the experiments for ventilation and no less then 30 minutes will be given for air
circulation between two different research subjects. The LCD display, the holographic
display and the Plantroid robot will be cleaned thoroughly with alcohol (even though
research subjects are not expected to touch them) between research subjects and thepen used for writing the impressions will be discarded after every session. The chair andthe table will also be cleaned with alcohol. The temperature of potential participants
will be measured by a thermographic camera and, if it is over 37.5o C, the volunteer
will have the participation refused and will be advised to seek medical attention if any
other symptoms develop.

4.8 Protection of the anonymity of Research Volunteers
Unfortunately, since videos of facial expressions and audio recordings of the voicesof participants will be stored, it is not fully possible to make captured data completely
12anonymous. However, no real names will be assigned to the stored data, but a randomly
generated UUID (Universally Unique Identifier). The data will only be accessible byresearchers involved in this research (currently only the author of this experiment plan
and Professor Ikuo Mizuuchi) and, if any examples become necessary for reports or
research papers/presentations, the author of this research plan will provide his own
image, voice and responses as an example. The videos and audio recordings will not
be shared with any 3rd parties and the dataset obtained from the experiment will only
contain the valence-arousal and emotion classification of the recorded audio samples,making it impossible to identify the volunteers just from the dataset.
Moreover if any volunteer later regrets participating on the experiment, he or she
will be able to request the deletion of the contributed data for a period of five years,
since the data will be stored for a period no longer than that.

13

A. Explanation Document
Head of Research: Professor Ikuo Mizuuchi
Ethics Committee approval number: No. (Approval number)Thank you very much for your interest in participating in this research experiment! This document explains in detail what the experiment consists of and what a
volunteer needs to do in order to participate, so you can make an informed decision
about your participation.
In this experiment, you will be asked to hold conversation with four different
conversation agents. A conversation agent is an entity that is capable of understandhuman speech and of generating responses. Such entities can be virtual (e.g. an
animated character in a screen) or physical (e.g. a robot). The four conversation
agents are shown in Table 3.

Image

Name

Avatar Kotaro

Avatar Plantroid

Plantroid robot

Holographic Plantroid

Table 3: Conversation agents.
Each one of the four conversation agents is capable of synthesizing speech, thatis, capable of converting text into speech. The speech generated by the agents has
four characteristics: (i) speed, that is, how fast the agent is speaking; (ii) volume,
representing how loud the voice is; (iii) pitch, which defines how deep the agent’s voice
will be and (iv) timbre, which determines how great is the quality of the generated
audio. Those parameters, together, are called prosody. In this research, only the first
three parameters are changed, since lowering the quality of the generated audio is
expected to generate bad results only.
Distinct prosody selection systems were developed in Mizuuchi Lab. Those systems select the most adequate prosody parameters according to their own criteria for agiven text. The pair text and prosody will, then, be used for synthesizing speech thatis expected to be enjoyable and more natural sounding. Each one of the agents will
use all three of the systems during your interactions.
The experiment consists of three phases P1 , P2 and P3 . In the first phase P1 , threeof the conversation agents (Avatar Kotaro, Avatar Plantroid and Plantroid robot) will
generate Gibberish speech only, that is, speech that sound human, but has no actual
meaning. In phases P2 and P3 , agents will answer with semantic speech, that is, speechwith actual meaning.
In every phase of the experiment, volunteers need to have conversations witha conversation agent which is using one of the developed prosody selection systems.
A conversation consists of ten exchanges for the first two phases P1 and P2 , and of
14

five exchanges during P3 . An exchange is defined as the part of a conversation inwhich one of the participants says something and the other answers. Phases P1 and P2
consists of talking three times to every agent; each time the conversation agents will
select a distinct prosody selection system. However, during P3 , only a single prosody
selection system will be used by conversation agents Avatar Plantroid, Plantroid robot
and Holographic Plantroid.
In phases P1 and P3 you are completely free to say whatever you want to theconversation agents, however, in phase P2 , for the first three exchanges, you will needto ask the questions below to the conversation agents for each prosody selection system:
1. What is the current temperature?
2. How long until I need to water the plant?
3. What is the salinity of the soil?
This is done to set up a fixed framework so you can better evaluate the performance of the prosody selection systems.
The order in which you will talk to the agents during each phase will be selected
at random, as will be the order of the systems which they use to synthesized speech.
Once an agent is randomly chosen, it will use all of the three systems in a random
order. After you have finished talking to an agent, the next one will be randomly
picked among the remaining agents. After you have talked to every agent, the current
phase will be over and the next one will start after a break, where you will be able torest, move around, stretch, drink some water etc.
At the end of an interaction with a pair (Agent, prosody selection system), youwill be asked to evaluate your impression on the way the conversation agent speaksby means of a Godspeed scale questionnaire. A Godspeed Scale questionnaire is aquestionnaire where one can rate the impression of some characteristics of the system,
which are described by two words which represents the extremes of said characteristics;
for example unintelligent-intelligent, for rating the impression about the intelligence ofa system. Two distinct Godspeed scale questionnaires are used in the experiment. The
first one, shown in the left side of Figure 5, is used to evaluate what impression you
have about the speech characteristics. The questionnaire on the right is an example of
how to fill the Godspeed scale questionnaire. The first scale, for example, ArtificialNatural has been filled closer to the natural side of the scale, showing that, in thatexample, the research subject thinks that the speech sounds natural, but still a little
artificial. It must be noticed that a participant does not need to fill the ID, agent andsystem fields, which will be filled by the researcher supervising the experiment once
you finish answering the questionnaire.
The second Godspeed scale questionnaire, shown in Figure 6 is used to evaluate
the impression of a given conversation agent. This questionnaire must be filled once
you have interacted three times with a given agent. The card in the right represents
the questionnaire and the one in the left is an example of how to fill it, which is very
similar to the procedure for the first questionnaire. Once again, you should not fill the
ID and Agent field, which will be filled by the supervising researcher.
Additionally, after filling the Godspeed questionnaire for the conversation agent,
15

Figure 5: Godspeed scale questionnaire for evaluating the impression caused by the
speech characteristics of a prosody selection system.

Figure 6: Godspeed scale questionnaire for evaluating the impression caused by aconversation agent.
you will need to rank the prosody generation systems as shown in Figure 7. There are
two types of field: system and symbol in the card. Regarding the system field, you will
fill it in the following manner: for the first system you have interacted with, you shallwrite ”1”, ”2” for the second and ”3” for the third. The symbol field can be filed with
one of the three following symbols: =, < and >. The equal sign is used to declare thatthe enjoyment of the interaction with both system is equal, while ”<” is used to statethat the system to the left of it has performed worse than the system to the right of it.
Finally, ”>” is used to declare that the system to the left of it has performed better
than the system to the right of it. An example of how to fill the card can be seen in theright card of Figure 7. Once again, the ID field is filled by the researcher supervising
the experiment.
After finishing interacting three times with each of the conversation agents, youwill be requested to rank the enjoyment of the interaction with the agents themselvesin a similar process. The card used to rank the conversation agents and an example of16

Figure 7: Left. Card for ranking the enjoyment of talking to each prosody selection
system. Right. Example of how to fill such card.
how to fill it is shown in Figure 8.

Figure 8: Left. Card for ranking the enjoyment of talking to each conversation agent.
Right. Example of how to fill such card.
It must be noticed that during the experiment, videos will be recorded of yourfacial expressions in order to allow the researchers to estimate your immediate impression about the responses of the conversation agents. Additionally, everything you
tell the robot will be recorded in audio, allowing us to better estimate your emotional
state before the answer of the conversation agent. This way, it is important to not say
anything that you do not want to be recorded.
The expected flow of the experiment can be described in Figure 9 . Have in mind
that it is only an example, since the order in which the agents and the prosody selection
systems are chosen is completely random.

17

P1
x10

x10

A

free

Answer questionnaire about
prosody selection systems

free

x10

C

Answer questionnaire about
prosody selection systems

Answer questionnaire about
prosody selection systems
x10

x10

free

C

free

Answer questionnaire about
prosody selection systemsx10

A

Answer questionnaire about
prosody selection systems

Answer questionnaire about
prosody selection systems
x10

B

free

B

C

free

x10

x10

free

A

free

B

free

Answer questionnaire about
prosody selection systems

Answer questionnaire about
prosody selection systems

Answer questionnaire about
prosody selection systems

Rank the prosody selection
systems

Rank the prosody selection
systems

Answer questionnaire about
conversation agent

Answer questionnaire about
conversation agent

Rank the prosody selection
systems
Answer questionnaire about
conversation agent
Rank the conversation
agents

Break
P2
x7

x3

x3

B

free

Question

Answer questionnaire about
prosody selection systems

A

free

Question

Answer questionnaire about
prosody selection systems

x3

Answer questionnaire about
prosody selection systems

free

A

C

x7

x3

free

Question

x7

x3

A

free

Question

Answer questionnaire about
prosody selection systems

Answer questionnaire about
prosody selection systems

free

C

free

Questionx7

Questionx7

x3
QuestionB

Answer questionnaire about
prosody selection systems
x7

x3

x7

x3

x7

free

Question

Answer questionnaire about
prosody selection systems

Answer questionnaire about
prosody selection systems

Rank the prosody selection
systems

Rank the prosody selection
systems

Answer questionnaire about
conversation agent

Answer questionnaire about
conversation agent

C

x7

x3
Questionfree

BAnswer questionnaire about
prosody selection systems
Rank the prosody selection
systems
Answer questionnaire about
conversation agent
Rank the conversation
agents

Break
P3
x5
x5

free

free

x5

A

Answer questionnaire about
conversation agent

free

A

A

Answer questionnaire about
conversation agent

Answer questionnaire about
conversation agent
Rank the conversation
agentsFigure 9: Example of a possible experiment configuration.
18

It must be highlighted that you have the right to give up your participation on
the experiment at any given moment for any reason you deem appropriate and that,even after the participation is finished, you can request for the deletion of any data
you have contributed.
You will be asked in every break if you are willing to proceed with the experiment
and, in case you want to give up, the supervising researcher will ask you if the data
captured so far can still be used for the research. In case of a negative answer, all data
will be promptly deleted.
If there are any questions, please feel free to ask now or during the experiment.
We hope you are still willing to participate and, if that is the case, please read andsign the Consent Form.

19

B. Consent Form
Head of Research: Professor Ikuo Mizuuchi
Ethics Committee approval number: No. (Approval number)
We are pleased to announce that the following human-robot interaction researchfor participants aged 18 or older will be conducted in our laboratory in compliance with
the Japanese ethical guidelines, laws and regulations; with the approval of the Ethics
Review Committee of the Tokyo University of Agriculture and Technology (hereon
referred as TUAT) and the permission of the President of TUAT.
With the approval of the TUAT Ethics Review Committee and the permissionof the President, we will conduct the following research in compliance with the ethicalguidelines and laws and regulations. We would like to kindly ask for your cooperation.
There will be no excessive burden on you by participating in this research. In addition,
we will take the utmost care to protect your personal information.

B.1 Research Purpose
This experiment is being performed in order to validate the performance of thegibberish speech impression prediction (GSIP) module developed in Mizuuchi Lab, toverify if its accuracy is sufficient and that the prosody (characteristics of the speech,such as speed, volume, pitch and timbre) parameters it chooses lead to enjoyable talks.
This will be done by means of having research volunteers talk with three distinct conversation agents, which will use different prosody selection criterion. You and otherresearch volunteers will react to the select speech and, later, will evaluate your impression of the systems and of the agents by answering a few Godspeed Scale questionnaires
and ranking their performance.
Moreover, research participants will be asked about their impressions of the conversation agents themselves and to rank which agents generated the most engaging
interactions. It must be noted that the same artificial intelligence is generating theanswers of the robots, the only aspects that change are the prosody parameter, theappearance and the embodiment of the conversation agents.

B.2 Voluntary research cooperation and freedom of withdrawal
Participation in this research is completely voluntary. In other words, if you wantto contribute to the development of science, you can participate; and you can stop
participating whenever you want without any repercussions. The data you contributeto this study will be securely stored for 5 years, during which time you can request thedeletion of your data through email or in person.

B.3 Research duration
The experiment will take place between 2023/2/11 and 2023/3/31.

20

B.3.1 Research Method
This experiment is being performed in order to validate the performance of thegibberish speech impression prediction module developed in Mizuuchi Lab, to verify
if its accuracy is sufficient and that the chosen prosody (characteristics of the speech,such as speed, volume, pitch and timbre) parameters are enjoyable. This will be doneby means of having research volunteers talk with three distinct conversation agents,
which will use different prosody selection criterion. You and other research volunteers
will react to the select speech and, later, will evaluate your impression of the systems
and of the agents by answering a few Godspeed Scale questionnaires and ranking theirperformance.
This research experiment will also allow the researchers to better understandwhich characteristics of conversation agents make them more or less engaging.
At first, conversation will be held with three conversation agents who are capableof speaking only gibberish speech, that is, speech that sounds human-like, but has noreal meaning. Each agent will use three different methods of changing the way they
speak and, after talking for 10 exchanges per method, participants will evaluate theirimpression of the selection method using the Godspeed scale questionnaires.
Facial expression of participants will be recorded in order to allow the researchers
to estimate the immediate impression on the answers of the conversation agents andto later compare with the impressions in the Godspeed Scale questionnaires.
After that, same process will be repeated, but the conversation agents will answer
the participants with semantic speech. Participants will ask three preselected questions
and the will be able to have a free conversation with the agents for seven exchanges
for each one of the three distinct prosody parameter selection systems. Participants
will evaluate their impression on the prosody selection systems and the conversation
agents using the same scales used before.
Finally, volunteers will hold a five exchange free conversation with three conversation agents and will evaluate the aforementioned interactions using Godspeed scalequestionnaires and will rank the conversation agents in terms of how engaging they
were during the interactions.
B.3.2 Research Cooperation Items
Participation in this research does not have many requirements, except that you
need to understand and to speak a basic level of English and you are willing to spend
around 2h15 of your time. You will receive through explanation of how to participate
in the experiment in written and verbal forms; and you are free (and encourage) tomake questions whenever you feel more explanation is necessary.
The experiment consists of 3 phases:
P1 : you will talk with three different conversation agents, which will use different
speech characteristic generation criteria for gibberish speech. You will talk ten times
for each criterion, and the conversation agents will reply to you ten times, totalling 30
times per agent and 90 times in this phase. You will be asked to evaluate the impressionof each of one the prosody selection systems and of every conversation agent.
21

P2 : you will talk with three different conversation agents, which will use different
speech characteristic generation criteria for semantic speech (speech that has meaning).
You will be able to talk 10 times for each prosody selection system, three of which will
be preselected questions. The remaining exchanges are free – you can say whatever
you wish to the conversation agents. After 10 interactions, you will need to evaluate
the impression on the prosody selecting system and, after interacting with every conversation agent, you will need to rank how enjoyable were the interactions with every
agent. In total, you will talk 30 times per agent and 90 times in this phase.
P3 : this is the shortest phase, you will be asked to talk five times with threedifferent conversation agents. Each one of them will use the same prosody selection
mechanism and, thus, you will only have a single round of talks with each one of them.
After talking to every agent, you will need to evaluate your impression of each one ofthem and rank the interaction in terms of which one was the most enjoyable.

B.4 Benefits and burdens for study subjects
Research volunteers will spend time talking with different conversation agents foraround 2h15. We hope that the volunteers will enjoy the experience. There will bea financial compensation of an Amazon Gift card containing 3,000 (three thousand)Japanese Yen for participating on the experiment.

B.5 Protection of personal information
For this research, personal data of research voluntaries will be stored in Mizuuchi
Lab temporarily in paper and, then, in the servers of the laboratoty. The aforementioned data consists of:
1. Name;2. Email;
3. Signed consent form;
4. Research ID;
5. Age ;6. Gender;
7. Country/Region of Origin;
8. Mother Language;
9. Other Languages you Speak;
10. If you live or have lived abroad; where;
11. Years living abroad.
12. Voice records of what research volunteers say to the conversation agents;
13. Video records of facial expressions of participants after they listen to the speechof conversation agents;
2214. Questionnaire answers.
All data will be encrypted using state of the art AES-256 encryption algorithm,
which is approved by the U.S. National Security Agency (NSA) for top secret information.
After a period of 5 years after the research is finished, all data will be destroyed.
The consent forms will be shredded and discarded and all digital data deleted using
BleachBit, a software which deletes files by writing garbage data over the hard drive
section where they were stored countless times, making recovery impossible even forforensic techniques.

B.6 Method of Disclosure of Research Protocols and Information on Research
The full research protocol and methods will be disclosed after March 31st of 2023
at the following web address: https://mizuuchi.lab.tuat.ac.jp/ antonio/
The protocol cannot currently be shared because full knowledge of the research
protocol may alter results of the research.

B.7 Disclosure of the results of the subject’s own data
Research participants who wish to have access to their own data can request
access by sending an email to: antonio@mizuuchi.lab.tuat.ac.jp using the email given
to the researchers as the contact email.
The subject should be ”Access request to own experiment data”. In the body ofthe email, volunteers need to give their full name, time of the experiment and assigned
ID during the experiment. Data will be sent if the provided information is correct.

B.8 Publication of Research Results
The results of this research will be presented at domestic and international conferences and published in academic papers.

B.9 Attribution of Intellectual Property Rights arising from
Research
Intellectual property cannot be attributed to volunteers. Any intellectual property resulting from this research experiment belongs to TUAT and the researchers.

B.10 Data Handling Policy after research completion
Five years after the start of the experiment, the profile data, audio and video
recordings of all volunteers will be deleted using the latest software (BleachBit) toguarantee that no one, not even the researchers, will be able to recover the data.

23

B.11 Research funding and conflict of interest
Researchers involved declare that they do not have any conflict of interest regarding this research and experiment. This experiment was not funded by any organization
other than TUAT.

B.12 Contact Information:
For inquiries regarding this experiment and the current research, please contact
the following:
Antonio Galiza Cerdeira Gonzalez,Tokyo University of Agriculture and Technology,
Graduate School of Engineering,
Mechanical Systems Engineering Department
Email: antonio@mizuuchi.lab.tuat.ac.jp

B.13 Consent for cooperation in the experiment
I have received an explanation of the research project entitled ”Development of a
Human Impression Prediction-based Prosody Selection System for Synthesized Speechof Conversation Agents.” in writing, and I understand the following items and agree tocooperate in the research of my own volition.
- If you have received and understood the explanation, please tick the appropriate
box in the  column.
I understand and agree to the following items.
1 Purpose of the research�2 Voluntary nature of the research and freedom to withdraw
�3 Content of research method and research cooperation
�4 Benefits and disadvantages to subjects
�5 Privacy of personal information
�6 Method of disclosure of research protocol and information on research�7 Disclosure of results to subjects themselves
�8 Disclosure of research results
�9P roprietary ownership of intellectual property rights arising from the research�10 Handling of samples after the completion of the research�11 Cost sharing and conflict of interest
�12 Contact information
Date of Consent:
Name of Research Subject:24

<Signature or Seal>�

C. Volunteer Profile Form
Dear volunteer, thank you very much for your collaboration in this experiment.
However, before the experiment start, we need to collect some personal information
which will be used to better understand how certain groups of people react to different
speech patterns. If you are still willing, please, fill the fields below.
• ID (to be filled by the experiment supervisor):• Age:• Gender:• Country/Region of Origin:• Mother Language:• Other Languages you Speak:• If you live or have lived abroad, write where:• Years living abroad:• Experience with pets:• Experience with robots
If you are willing to receive updates about how the research is going and invitation
for participating on further experiments, please, write your email below:

D. Questionnaires
This appendix showcases the Godspeed questionnaires used for evaluating thedifferent prosody selection systems and the conversation agents and the phrasing usedin the form for ranking the same aforementioned prosody selection systems and agents.

D.1

Godspeed Questionnaire for Prosody Selection Systems

Research volunteers will receive a small paper card which looks like the one
present in Figure 10 where they can evaluate their perception of the speech generated
by a prosody selection system in the following scales: artificial-natural, unfriendlyfriendly, unpleasant-pleasant, unintelligent-intelligent, apathetic-responsive. Researchsubjects will be handed out an example card on how to fill their card at the beginningof the experiment (also shown in Figure 10), which they will be able to consult at
any moment. The fields ”ID”, ”agent” and ”system” will be filled by the researcher
25

Figure 10: Godspeed scale questionnaire for evaluating the impression caused by the
speech characteristics of a prosody selection system.
supervising the experiment after the volunteer has finished rating the prosody selection
system, since the volunteers cannot know they will be talking with the same prosody
selection system in different conversation agents (even though they might figure thatout by themselves, they should not receive any confirmation in that regard, at least
until the experiment is over).

D.2

Godspeed Questionnaire for Agents

Research volunteers will receive a small paper card which looks like the one
present in Figure 11 where they can evaluate their perception of the conversation
agent itself across distinct prosody selection systems in the following scales: unfriendlyfriendly, unpleasant-pleasant, unintelligent-intelligent, apathetic-responsive, kind-unkind.
Research subjects will be handed out an example card on how to fill their card at the
beginning of the experiment (also shown in Figure 11), which they will be able toconsult at any moment. The fields ”ID” and ”agent” are filled by the researcher supervising the experiment after the volunteer has finished rating the interaction withthe agent, just to be consistent with how the system worked for rating the prosody
selecting system.

D.3

Communication Systems Ranking

Participants will be asked to rank the performance of the prosody generation
systems with which they just finished interacting with using a card shown in Figure 12.
They will refer to the system as 1, 2 or 3, accordingly to the order on which they wereused in the interactions with the current conversation agent. The researcher supervising
the experiment will then attribute the correct names of the prosody selection systems
after the volunteer hands out the card. While ranking the systems regarding how
enjoyable the interactions were, subjects can use the comparison symbols <, > and =,to denote that the system on the left performed worse than the system on the right,that the system on the left performed better than the system on the right and that thesystems on the left and right had comparable performance, respectively. Volunteers
26

Figure 11: Left. Card for ranking the enjoyment of talking to each prosody selection
system. Right. Example of how to fill such card.

Figure 12: Left. Card for ranking the enjoyment of talking to each prosody generation
system. Right. Example of how to fill such card.
will also receive an example card, shown in the right side of Figure 12 and explanation
on how to rank the systems.

D.4

Agents Ranking

Participants will be asked to rank the performance of the conversation agentswith which they just finished interacting with using a card shown in Figure 13. They
will refer to the agents as 1, 2 or 3, accordingly to the order on which they were usedin the interactions. The researcher supervising the experiment will then attribute thecorrect names of the agents after the volunteer hands out the card. While ranking theagents regarding how enjoyable the interactions were, subjects can use the comparison
symbols <, > and =, to denote that the agent on the left performed worse than thesystem on the right, that the agent on the left performed better than the systemon the right and that the agents on the left and right had comparable performance,
respectively. Volunteers will also receive an example card, shown in the right side of27

Figure 13: Left. Card for ranking the enjoyment of talking to each conversation agent.
Right. Example of how to fill such card.
Figure 13 and explanation on how to rank agents.

E. Selected Semantic-Free Utterances,
and Questions and their respective
responses
In the Talk to Kotaro Experiment, every Semantic-Free Utterance (SFU) was generated by a Gibberish Speech (GS) generating algorithm, which caused every researchsubject to listen and react to distinct utterances. This, while helpful for understanding
the role of phone selection on human impression, made analyzing how changing the
prosody parameters impacts the impression of participants difficult. This time, however, five fixed GS were generated beforehand and, thus, every participant will react tothe same SFU, allowing the researchers better analyze the role of personal preferences
and prosody changes in human impression.
The same principle applies for having users ask previously selected questions
which will receive semantic speech responses with the same text, so it is possible tobetters study the effects of different prosody parameters and of personal preferences.
This appendix will, then, list the GS in Subsection E.1 and the questions andtheir respective responses in Subsection E.2

E.1

Previously generated Gibberish Speech

All Gibberish Speeches listed in this subsection were randomly generated and,for prosody selection system c1,gs , were evaluated by GSIP and, thus, have an expected
impression prediction associated, allowing the researchers to compare with the actual
impression displayed during experiments. All gibberish text (in IPA), volume, speed,
28

IPA phones
speedbaɪɹˈuːleɪb150
mˌahɐθɹiːwˈɛθɪ
150
hˈakɪʃˌɪdʒɪn150pˌiːˈɑːtˈakɹaθslmnpək 150
ˌaləbˈambˈeɪniə
150

volume pitch valence arousal
audio file
100
45
0.05
0.2395 utterance1.wav
100
45
0.1079 0.3779 utterance2.wav
100
45
0.1256 0.2048 utterance3.wav
10045-0.4050.206 utterance4.wav
100
45
-0.05
0.3974 utterance5.wav

Table 4: Pregenerated Gibberish speeches and their predicted valence/arousal values.
Table 5: Pregenerated questions and their answers

Question

AnswerWhat is the current temperature?

The temperature in thisroom is of 26 degrees Celsius.
I estimate that you will need to water
the plant in two hours and fifteen minutes.
The current salinity of the soil
is of 2 deci-Siemens per meter; within the safe range for your plant.

How long until I need to water the plant?
What is the salinity of the soil?

pitch, predicted valence, predicted arousal and audio file are shown in Table 4

E.2

Previously generated questions and their answers

The decision of having research volunteers asking selected questions and receiving
selected responses from conversation agents has the same rationale of pre-generating
the first five gibberish speeches of the agents, as explained in Subsection E.1. However,the decision of having three defined question instead of five is in order to not let
volunteers feel that their freedom was restricted, since in the previous experimentphase participants could say anything the wished to the robot. All question, listed inTable 5, are related to the plant which Plantroid takes care of, since it is a neutral
conversation topic that also gives purpose of existing to the conversation agents in theeyes of volunteers. For the same reason, every answer is either neutral or positive, inorder to not negatively influence the participants’ perception of the generated speeches.

29