<figure><img src="image_1.png" /></figure>

HAL
open science

# Artificial Intelligence Model of an Smartphone-Based Virtual Companion

Elham Saadatian, Thoriq Salafi, Hooman Samani, Yu De Lim, Ryohei Nakatsu

► To cite this version:

Elham Saadatian, Thoriq Salafi, Hooman Samani, Yu De Lim, Ryohei Nakatsu. Artificial Intelligence Model of an Smartphone-Based Virtual Companion. 13th International Conference Entertainment Computing (ICEC), Oct 2014, Sydney, Australia. pp.173-178, 10.1007/978-3-662-45212-7_22. hal-01408519

HAL Id: hal-01408519

https://inria.hal.science/hal-01408519

Submitted on 5 Dec 2016

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Artificial Intelligence Model of Smartphone-based Virtual Companion

Elham Saadatian<fnref n="1" />, Thoriq Salafi<fnref n="1" />, Hooman Samani<fnref n="2" />, Yu De Lim<fnref n="1" />, and Ryohei Nakatsu<fnref n="1" />

**Abstract.** This paper introduces an Artificial Intelligence (AI) model of a virtual companion system on smartphone. The proposed AI model is composed of two modules of Probabilistic Mood Estimation (PME) and Behavior Network. The PME is designed for the purpose of automatic estimation of the mood, under uncertain and dynamic smartphone context. The model combines Support Vector Machine (SVM) and Dynamic Bayesian Networks (DBNs) to estimate the probabilistic mood state of the user. The behavior network contorts the behavior of the interactive and intelligent virtual companion, considering the detected mood and external factors. In order to make the virtual companion more believable, the system consists of an internal mood state structure. The mood of the agent, could also be inferred from another real human such as a remote partner. The fitness of the artificial companion behavior in relation to the users mood state was evaluated by user study and effectiveness of the system was confirmed.

**Keywords:** Artificial Intelligence, Entertaining Virtual Companion, Affective Computing

## 1 Introduction

The aim of this research is to employ various sensors integrated in the modern smartphones to observe the behavior of users and develop smart systems to enrich communications and interactions. Emerging field of mediated affective communication relates to the technologies that aim to mediate personal communication across distance. Contemporary lifestyle changes have led to design and adoption of technologies in support of modern lifestyle. In our earlier work, a comprehensive study of existing prototypical systems and related conceptual studies was investigated [9].

Intelligent virtual agents are human-like embodied characters [5]. These autonomous artificial characters have applications in many fields, such as computer game [11], conversational agents [1], and affective robotics systems [10] and many more. In this study, we have proposed and AI system for intelligent affective virtual companion on smartphone. In this respect, a model is proposed to detect

<footnote n="1">¹ Keio-NUS CUTE Center, Interactive and Digital Media Institute, Singapore
{elham,thoriqsalafi,limyudee,idmnr}@nus.edu.sg</footnote>

<footnote n="2">² Department of Electrical Engineering, NTUP, Taiwan, hooman@mail.ntpu.edu.tw</footnote>
the mood state of a user (based on the smartphones sensors' data), and generate corresponding reactions to the detected mood. The methodology is described in details by exploring mood recognition from smartphone sensory data, controlling the behavior of the virtual companion and visualization of the working process. The proposed model has been implemented and evaluated, which is described in this paper.

## 2 Methodology

The proposed methodology is anchored on the idea of combining uncertainty modeling with behavior networks, proposed in few previous works such as [3, 4]. The improvement on the initial previously proposed ideas is made by considering the mood history of user using DBNs, using more sensors from the smartphone and fusion of inference techniques, as well as customized design of the behavior network for the specific purpose of the artificial affective companion. The system performs two functions of mood inference and behavior generation, which are detailed below.

### 2.1 User mood Recognition from smartphone data

The mood estimation module of the system (PME) is composed of SVM and DBNs. SVM detects the physical activity level using accelerometer data and DBNs is designed to infer and map the valence and arousal result to the two-dimensional valence arousal model as illustrated in [8].

The DBNs is selected due to the uncertainties and dynamic environment of the smartphone [7]. DBNs is well suited to the problem of inferring high-level information such as the mood state by using the low-level data obtained from the smartphone. That is because the mood state of the user depends probabilistically on many different factors. For example, with the accelerometer, we can know whether he is moving or not, and if so, whether he is moving fast or slow, and with the GPS, we can estimate the user location, whether he is at home or at work or somewhere outdoors. This low-level data obtained from the smartphone sensors, can predict the conditional probability of the user status. By setting the logic based, conditional probabilities of the user mood in relation to the smartphone sensor data, the mood state could be estimated without the need of precise knowledge and training the system such as in the method proposed by [6]. Figure 1 shows the designed DBNs model. The model infers the valence and arousal level of the user, which could be mapped to the Valance-Arousal Model (VAM) and define the detected mood state.

### 2.2 Behavior controller of the virtual companion

After detecting the user's mood state, there is a need for an algorithm to control the respective artificial companion behavior. In this respect, a behavior network
<figure><img src="image_3.png" /><figcaption>Fig. 1: Dynamic baysian network for mood estimation</figcaption></figure>

is designed. In the proposed behavior network, The environment and sensing part of the behavior network comes from DBNs output, which consists of user and the agents internal (or a remote user) mood and local time. There are three goals in this behavior network, imitation, feedback and special time. The imitation goal triggers the character to emulate his or her partner’s mood. The feedback goal intends to give feedback for the user’s mood. The special time goal greets the user about the time change. The special time goal is triggered when there is a change in the time period, such as morning afternoon and evening. The feedback goal will be triggered only when there is a prominent user’s mood that reaches a certain threshold, otherwise the imitation goal will be activated. In the feedback mode, the personality of the partner is considered and represented by the animation speed. The behavior network will determine the animation of the character behavior, and the animation speed of the character depend on the personality of the partner. The more extrovert the partner is, the faster the animation of the character will move. The designed behavior network is shown in 2.

## 2.3 Visualization of the working process of the system

In order to demonstrate the generated behaviors of artificial companion, an experiment with two partners is performed. The reason for choosing a second real user (partner) is to attribute the artificial agent's mood to a real user. Artificial agents with anthropomorphic features could be perceived more natural and realistic [2]. The first five sets of the mood data from the user, and its partner with the respective animated agent's behavior is shown in figure 3. The proposed AI is applied in a smartphone Android app and the prototype screen shot is shown in Figure 3f.
<figure><img src="image_4.png" /><figcaption>Fig. 2: Behavior network for agent's behavior control</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 3: Visualization of the auto-generated animated agent behaviors for different scenarios.</figcaption></figure>

(a) Special-time morning- User Mood 22.02% Relax- Partner Mood 20.84% Relax.

(b) Not an special-time- User Mood 44.39% Relax- Partner Mood 16.77% Happy.

(c) Not an special-time - User Mood 64.61% Sad - Partner Mood 22.37% Happy

(d) Not an special-time User Mood 46.71% Relax Partner Mood 46.40% delighted.

(e) Not an special-time- User Mood 54.99% Relax- Partner Mood 52.16% Sleepy.

(f) An screen shot of the app on Android phone
## 3 System Evaluation

In order to evaluate the system, ($n = 10$) active smartphone users, including five females and five males, aged between 21 to 33, (Mean = 25.8, SD = 3.82), were randomly selected from National University of Singapore's staff and students. None of the participants had previous experience of interacting with similar technologies such as empathetic virtual agents or emotional robots. The evaluation was performed in two stages using a simulator. The first stage the PME module was evaluated and in the second stage, the fitness of the virtual companion behaviors was tested.

The PME was tested by comparing 120 samples of mood data inferred from the proposed PME model against the self-reported user moods. Each user has logged the smartphone usage and context data such as time, location, running apps, call logs etc. in relation to their self-perceived mood. The self-perceived user mood state was compared against the result of the proposed model. The Mann-Whitney U test was used to evaluate whether or not there was any significance difference in the valence arousal space between self-perceived and the automatically inferred results. The p value for two tailed is 0.64 for the valence space, and 0.73 for the arousal space. Both p values are higher than 5%. Therefore, could be concluded that there is no significance difference in the proposed model and user self-perceived affective state. This suggests the success of PME model in the correct inference of the mood state of the user.

In the second experiment, the goal was to confirm that the interactive companion could produce corresponding behaviors, which relates to the mood. The fitness of the virtual companion behaviors points to the correspondence of the auto-generated agent's behavior from the point of view of the users. Since, the suitability of the behavior is a subjective issue and cannot be measured quantitatively we adopted the method proposed by [12] to assess the model. In this method, the participants were given 10 different combinations of the user mood (themselves), artificial companion's mood, and time of the day (special time or not). For each scenario of the mentioned combinations, they observed 5 randomly generated behavior by the animated agent followed by 5 behavior generated by the proposed behavior network. Afterward, they were asked to rate the appropriateness of the behavior from 1 (strongly inappropriate) to 5 (strongly appropriate). The mean fitness scores of each participant were calculated as shown in table 1. The result was analyzed by Wilcoxon signed-rank test with the fitness scores. As a result, the p value was obtained as 0.005 < 0.5% which confirms the proposed model succeeded in generating suitable behavior compared to randomly generated behaviors.

## 4 Conclusion

The aim of this study was designing an (AI) model for smartphone to detect the user's mood state and generate adaptive behaviors via an animated artificial
<table><caption>Table 1: Mean fitness ranks</caption><thead><tr><th>Participant</th><th>pa1</th><th>pa2</th><th>pa3</th><th>pa4</th><th>pa5</th><th>pa6</th><th>pa7</th><th>pa8</th><th>pa9</th><th>pa10</th></tr></thead><tbody><tr><td>Random</td><td>2.75</td><td>2.25</td><td>1.87</td><td>2.5</td><td>3.125</td><td>2.25</td><td>2.75</td><td>1.62</td><td>2.5</td><td>2.12</td></tr><tr><td>Behavior Net.</td><td>4</td><td>4.25</td><td>3.37</td><td>3.25</td><td>3.25</td><td>3.87</td><td>3.62</td><td>3.62</td><td>4</td><td>3.12</td></tr></tbody></table>

companion with internal mood state. The internal mood state of the agent could also come from another smartphone user. The AI model was developed based by fusion of a novel DBNs and Behavior network model. The system was tested by users, and the fitness of the generated behaviors were confirmed.

## 5 Acknowledgement

This research is supported by the Singapore National Research Foundation under its International Research Center Keio-NUS CUTE Center @ Singapore Funding Initiative and administered by the IDM Program Office.

## References

1. Alfonsi, B.: Sassy Chatbot Wins with Wit. IEEE Intelligent Systems pp. 6–7 (2006)

2. Duffy, B.R.: Anthropomorphism and the social robot. Robotics and autonomous systems 42(3), 177–190 (2003)

3. Han, S.J., Cho, S.B.: A hybrid personal assistant based on bayesian networks and a rule-based system inside a smartphone. International Journal of Hybrid Intelligent Systems 2(3), 221–234 (2005)

4. Han, S.J., Cho, S.B.: Synthetic character with bayesian network and behavior network for intelligent smartphone. In: Knowledge-Based Intelligent Information and Engineering Systems. pp. 737–743. Springer (2005)

5. Herrero, P., de Antonio, A.: Modelling intelligent virtual agent skills with human-like senses (2004)

6. LiKamWa, R., Liu, Y., Lane, N.D., Zhong, L.: Moodscope: building a mood sensor from smartphone usage patterns. In: Proceeding of the 11th annual international conference on Mobile systems, applications, and services. pp. 389–402. ACM (2013)

7. Malm, E.J., Jani, M., Kela, J., et al.: Managing context information in mobile devices. IEEE pervasive computing 2(3), 42–51 (2003)

8. Ouwerkerk, M.: Unobtrusive emotions sensing in daily life. In: Sensing Emotions, pp. 21–39. Springer (2011)

9. Saadatian, E., Samani, H., Toudeshki, A., Nakatsu, R.: Technologically mediated intimate communication: An overview and future directions. In: Entertainment Computing–ICEC 2013, pp. 93–104. Springer (2013)

10. Samani, H.A., Saadatian, E.: A multidisciplinary artificial intelligence model of an affective robot. International Journal of Advanced Robotic Systems 9 (2012)

11. Welsh, S., Pisan, Y.: Enhancing information acquisition in game agents (2005)

12. Yoon, J.W., Cho, S.B.: An intelligent synthetic character for smartphone with bayesian networks and behavior selection networks. Expert Systems with Applications 39(12), 11284–11292 (2012)