Research Plan

1. Introduction
Human communication has two main components, verbal and nonverbal. Most human-robot interaction systems so far have focused on the verbalcomponents, since it conveys very useful information that requires only
microphones and speech-to-text technologies, which nowadays are cheap andreadily available. Nonetheless, ignoring the non-verbal components, mainly
bodily and facial expressions, is to lose very important information, which canbe obtained by digital cameras, which have become cheap and present in mostmoderndevices.
Since robots are bound to become increasingly more present in daily andsocial environments, good communication skills are essential to provide
pleasant and efficient interactions with humans. For example, a store that uses
a social robot whose communication is interesting and dynamic is expected to
attract more customers than one that employs a monotonic stereotypicalrobotic voice system. Thus, this research is interested in using facial
expressions to estimate the reaction of users to what a robotic system is saying,in order to change its speech-patterns into a more pleasant one.
This experiment is a first step on this research. By collecting humanfacial expressions elicited by randomly generated semantic free speech
patterns, we intend to create a system that is capable of predicting humanreaction to several different words and ways of saying words.

2. Experiment Structure

1

The experiment consists of three steps. First step is the creation of a
profile. Test subjects need to read the consent form(presented in Section 3),
which will inform the subjects what the experiment consists of, what data
users(presented in Section 4), need to provide for the profile, that their video
and voice recordings will be stored and used for scientific research purposes.
Also, the requirements of having a webcam and having a room with goodlightning will be made very clear. Every information will be available in several
languages (at least in Japanese, English, Portuguese and Spanish). If they agreeand consent, users will have to answer a few questions about themselves andselect both an ID and password. With this information stored, they can login andbegin the experiment. Every necessary action for step 1 is highlighted in orangeinFigure
1.
The second step, highlighted in Blue in Figure 1, is the experiment itself.
Users will chat with a robot avatar for as long as they wish. Voluntaries need toclick a button in order to start recording its speech. After the user finishes
speaking, the same button should be clicked again. The robot-like avatar will,then, speak a randomly generated speech pattern as a response, to which
subjects will react with facial expressions. These expressions will be recorded
and analyzed by Mizuuchi Lab’s servers, yielding estimates of the emotions
behind the presented facial expressions. This step will be repeated for as long
as users wish to keep talking with the avatar; and the experiment itself can betakenasmanytimes
assubjects
desire.
The third step, highlighted in green, is optional, since many subjects may
close the website without clicking the logout button. Nonetheless, it consists ofa Likert scale composed of 10 prompts(presented in Section 5), to which users
must answer how much they agree or disagree with. After this step is ended,
the answers will be recorded and users will be sent back to the login page.

3. Consent Form
Since the experiment is to be held online, in order to recruit as manypeople from as many places with as many different backgrounds as possible, it
is impossible to send physical consent forms. This wait, the best close solution
is to provide users with a screen that describes in detail what the experimentconsists of and how the collected data will be used. If users agree onparticipating, they should click that they agree and fill a form with theirpersonal information. The intended text, in english, is shown below:“ABOUT THE EXPERIMENTThis experiment presents a friendly robot-like avatar capable of talking, withwhom you can have a conversation for as long as you wish. There is no minimum ormaximum experiment duration, but . Video will be captured from the webcam of yourdevice, from which your emotional state will be estimated. This is necessary to create asystem capable of predicting which emotions are elicited by different semantic-freespeech patterns. Said video will be stored in our lab’s server for further studies,
together with the estimated emotional response. What you say to the avatar will bestored for posterior analysis, so be careful with what you say.

INSTRUCTIONS
2

1. You need a webcam to participate;
2. Run the experiment on a place with good lighting;
3. When you are ready to speak, click on the button "Speak";
4. After you finished speaking, click on the button again;
5. The avatar will then speak, listen to it and react naturally;
6. You can repeat this procedure for as long as you wish, but please, do not close thepage before the avatar stops talking;7. When you wish to stop the experiment, if you have the time to answer a brief survey,
click the logout button and answer the 10 prompts.
If you would like to participate in the experiment, click on the button ‘I Agree’.”4. Voluntaries’ Data
This experiment intends on recording the voice and facial expression ofusers; and this data is bound to provide a lot of information. Nonetheless, some
information is either hard or impossible to obtain from video and audio, such as
the country of origin, how many languages users speak, where users have lived
and how many years abroad they have lived. Other information, like gender,age, the language an user is using to talk with the avatar can be obtained fromthe data, but it demands additional processing. Thus, a cheap and easy solution
is to ask users to fill a few questions during profile creation. The information
asked are as follows:

1.

Login ID;

2.

Password;

3.

Age ;

4.

Gender;

5.

Country/Region of Origin;

6.

Mother Language;

7.

Other Languages you Speak;8.

If you live or have lived abroad, write where;

9.

Years living abroad.

Login and Password are useful for security and to allow the researchers
to correlate videos, audios and estimated emotions to a certain user and its
3

profile data without storing real names. The other information will be used toverify if there is any difference between the reactions of distinct groups ofpeople, in order to decide if that information must be used as an input for thefinal
Improved
human-robot
communicationarchitecture.
During the experiment, audio will be recorded to .wav files with a codethat allows us to correlate what the test subject has said to the answer given by
the robot avatar to the subsequent user facial expression and emotional
analysis. File naming convention is: <username>_<video or audio>_<time ofaudio
recording>.<wavoravi>.
This data will be stored and kept safely on MizuuchiLab’s server for nolonger than an year. Everything will be deleted using BleachBit, a Linux toolcapable of ensuring that the relevant files really are deleted beyond any pointof possible recovery.

5. Likert scale
Users that are willing to spend a little more time on the experiment, whenlogging out ofthe platform, will be prompter a Likert scale consisting of the following 10
prompts, in order to measure their feelings and opinion about the experiment:
1. Talking with the robot avatar was interesting;
2. Variation of the speech characteristics made conversation more natural ;3. Some randomly generated words are less pleasant than others;
4. Some speechcharacteristics, such as speed, loudness or pitch
influence more than others;
5. Different random words didn’t have an impact on your enjoyment;
6. You felt that the robot was answering your speech accordingly;
7. Longer phrases were more interesting;
8. The turn-based conversation felt unnatural;
9. Foreign phonemes sounded more interesting;
10. The robot seemed to be intelligent.

6. Web Platform Structure

4

theFigures below show how the website will approximately look by the time
experiments
start.

Login ScreenConsent Form Screen

Profile Creation Screen

Experiment Screen
5Likert Scale Screen

6