Journal of Robotics, Networking and Artificial Life, Vol. 2, No. 1 (June 2015), 34-39

Developing a Dividual Model Using a Modular Neural Networkfor Human-Robot Interaction
Toshiyuki Tanaka

Aichi Prefectural University, 1522-3 Ibaragabasama
Nagakute, Aichi 480-1198, Japan
Kunikazu Kobayashi

Aichi Prefectural University, 1522-3 Ibaragabasama
Nagakute, Aichi 480-1198, Japan
E-mail: kobayashi@ist.aichi-pu.ac.jp
http://www.ist.aichi-pu.ac.jp/~koba/

Abstract
Currently, in the field of human-robot interaction (HRI), robots have a problem that can only interact the same at all
times with humans. In this paper, therefore, we introduce the concept called a dividual and build a model of thedividual to grow through interactions with others. In addition, using a modular neural network and reinforcement
learning (actor-critic), we confirmed the process to choose an appropriate dividual out of plural dividuals.
Keywords: Model of dividual, Human-robot interaction, Robot, Modular neural network, Reinforcement leaning.

1. Introduction
In recent years, many types of robots have beendeveloped and successfully applied to a variety offields, such as medical care and disaster relief. Androbots are getting into life place without realizing1,2,3.
For example, a voice recognition agent using artificial
intelligence technique on a smart phone, and ahumanoid communication robot manufactured by amajor communication company. In Japan, Prime
Minister Abe added “robot revolution” to economic
growth strategy4.
Currently, in the field of human-robot interaction(HRI), robots have a problem that they can only interact
with humans in a stereotypical way. Humans canhowever change correspondence depending on a human
to be interacted, and realize a variety of interactions(communications). On conventional interaction between
humans and robots, robots receive unilateral orders byperforming prearranged movement and utterance fromhumans, and perform given tasks5,6,7,8,9. This cannot

however realize robots which live together and support
humans.
As one of researches on HRI, Kojima tried to have amind like human for building a social relationship andproposed a model of development to obtaincommunication skills through social interactions10.
Shibata made robots play a role in human society. Thenhe developed a seal-shaped robot and reported thatelderly people get pleasure and spiritual comfort
through physical interactions such as touching andpetting11. Naya et al. also presented a system dealing
with haptic interactions between humans and robots12.
In the present paper, in the field of HRI, we develop
a dividual model to grow through interactions withothers. Then, we introduce a concept of dividual that it
is formed into a self with respect to another humanthrough repetitive communications with others13,14,15.
Individual cannot divide anymore whereas dividual candivide into plural ones. We use two machine learningtechniques to construct the dividual model. Then, weconfirm the process to choose an appropriate dividual
out of plural dividuals when we appropriately prepare

Published by Atlantis Press
Copyright: the authors
34

T. Tanaka and K. Kobayashi

an input set for a dividual model defined by category
elements and action ones.
The outline of this paper is as follows. Section 2
introduces a concept of dividual. Section 3 proposes a
dividual model using a modular neural network andactor-critic reinforcement learning. Section 4 explains
the results of computer simulations to verify theperformance of the proposed model. Finally, Section 5summarizes the present paper and states future work.
2. Dividual

a dividual model to grow through interactions withothers.
3.1. Design of Dividual Model
We introduce a modular neural network as a generalframework of the dividual model16,17. This is because a
dividual is represented by not a centralized network but
a distributed network. Then we suppose that a human
may switch dividuals not consciously but automatically.
One module corresponds to one dividual as show inFig.2.

A concept of dividual is proposed by Japanese novelist
Keiichiro Hirano to interact properly with anotherhuman13. Individual cannot divide anymore whereas
dividual can divide into plural ones. The dividual isformed for every partner of communication. Based on
the concept of dividual, a human can change to ananother self according to environments and humanrelations. Dividual is formed into a self with respect toanother human through repetitive communications withothers. In addition, dividual is strictly divided into threetypes as shown in Fig.1.

Fig. 2: Configuration of a dividual model.
Firstly, a dividual model creates a dividual as amodule by giving inputs and learning. This corresponds
to interaction between humans and robots. Secondly, themodel chooses an appropriate dividual out of plural
ones according to inputs by an output value from each
module. This corresponds robots could changeinteraction according to humans.

Fig. 1: Conceptual diagram of dividual.
The first one is a social dividual. This is a standard
dividual to interact with a stranger or an unfamiliar
person. The second one is a group-oriented dividual.
This corresponds to a dividual for a specific group suchas a school class or a tennis club. The third one is anindividual-oriented dividual. This is a dividual for a
specific person such as family members or a closefriend. In the present paper, the last one, i.e. individualoriented dividual is treated.
3. Dividual Model
We imagine a new HRI system which learn and growthrough interactions with family members, and develop

3.2. Learning of Dividual Model
We introduce an actor-critic reinforcement learning
method to train each module in Fig.318. This is because
a dividual is formed under the influence of external
factors such as environment and interactions withhumans. Reinforcement learning is a learning
framework that acquires the desirable output and time
sequence with a trial and error by being maximized
value for result of interaction with environment, that isreword of expectation. One of the representativetechnique is actor-critic. The actor-critic method
consists of an actor to select an action and a critic toevaluate its action.
The parameters are repeatedly modified as follows.

Published by Atlantis Press
Copyright: the authors
35

Developing a Dividual ModelTD error: Critic observes the rewords r and the next
state s ' = (x1, x2, … ,xN) T, and calculate temporal
difference (TD) error as learning index of critic andactor. TD error at time is expressed by Eq.(1).

(5)
where 𝜌 is a discount rate, 0 ≤ 𝜌 ≤ 1.

Selection probability of modules: We introduce a softmax technique based on Gibbs distribution in Eq.(6)so as to adjust the selection probability of modules.

(6)

where X refers to an input set and q(s) is updated byEq.(7).

Fig. 3: Structure of a neural network based on
actor-critic.

(7)
(1)

where γ is a discount rate, 0 ≤ γ ≤ 1.

Learning of middle unit: The output function of middle
unit is assumed as Gaussian function with anaverage 𝒄𝑗 and a variance 𝜎𝑗2 in Eq.(2).
(2)

The average 𝒄𝑗 becomes close to input s as Eq.(3).
(3)

where 𝜁 is a learning coefficient, 0≤ 𝜁 ≤ 1. In thepresent paper, the variance 𝜎𝑗2 of Gaussian function is assumed to be fixed.

where 𝛽 is a discount rate, 0 ≤ 𝛽 ≤ 1.
4. Computer Simulation
Through computer simulations, we confirmed theprocess to form dividuals and to choose an appropriate
dividual out of plural dividuals.
4.1. Design of Input Information
We prepare an input set for the dividual model inadvance. It basically consists of two kinds ofinformation. The first one is category information toidentify a person, e.g. name, sex, age, nationality, hobby
and etc. The second one is action information to express
interactions, e.g. question, answer and etc. For
convenience of identifying a person, we should keep theinput dimensions of action information as small aspossible. Then we perform the following manipulations
sequentially for learning in a neural network.

Learning of critic: A weight 𝑣𝑗 is updated by Eq.(4) sothat TD error becomes zero.

Labeling elements: We label input elements withappropriate words and values.

(4)

Conversion 1: We convert labeled inputs into binary
digits. This manipulation digitizes personalinformation.

Learning of critic: A weight 𝜔𝑗 is updated so as to takea higher state value V(s) using Eq.(5).

Conversion 2: We convert ‘0’ into ‘-1’, and ‘-1’ into‘0’. The manipulation solves a problem that 0 has nowhere 𝜂 is a discount rate, 0 ≤ 𝜂 ≤ 1.

Published by Atlantis Press
Copyright: the authors
36

T. Tanaka and K. Kobayashi

information amount when updating weights in aneural network.
Thus, an input is given to the dividual model as avector consisting of ‘-1’, ‘0’ and ‘1’. In actuality, wedefine input information as Table 1.

Table 1: Input information
namesexage

nationality

hobby

AB
CD
E

man
man
woman
woman
man

21
40
30
10
19

Asia
America
Asia
Asia
Europesportsreadingcooking
gamemusic

Next, we label input elements for input sets using
Table 2. For example, if a person’s name is A, we labelas ‘1’. Then if it’s sex is man, we label as ‘1’. After that,
if his age is 21, we label as ‘2’. Similarly, we label other
input elements for input sets. ‘-1’ means no informationin action information.

Table 2: Labeling elements for input sets-1

category

actionname（3bit）
sex（2bit）
age（2bit）
nationality（2bit）
hobby（3bit）
question（2bit）
answer（3bit）

1

2

3

4

5

A
BCDEman woman
1～20 21～30
31～40
Asia EuropeAmerica
reading sports
cookingmusic gamenothing Where will you go for the vacation? etc.
nothing I’m going shopping. etc.
-Next, we perform two types of conversions as shownin Table 3. Conversion 1 converts labeled information
into binary digits. Accordingly, we can express “001 01
10 01 010 -1-1 -1-1”. Conversion 2 converts 0 into -1
and -1 into 0. Thus, we can express “-1-11 -11 1-1 -11 11-1 00 000”. In Table 3, ‘*’ can take any values from{-1, 0, 1}.

Table 3: Conversion of input information
person labelingABCDE

conversion 1conversion 211212 ** 001 01 10 01 010 ***** -1-11 -11 1-1 -11 -11-1 *****
21331 ** 010 01 10 10 001 ***** -11-1 -11 1-1 1-1 -1-11 *****
32313 ** 011 10 11 10 011 ***** -111 1-1 11 1-1 -111 *****
42115 ** 010 01 11 10 101 ***** -11-1 -11 11 1-1 1-11 *****
51124 ** 010 01 01 10 100 ***** -11-1 -11 -11 1-1 1-1-1 *****

4.2. Simulation Settings
We imagine interaction between humans and robots asfollows. On one day, a robot gets information about ahuman, and performs appropriate behavior towards
him/her. On next day, the robot thinks about him/her,and performs both different and appropriate behavior
towards him/her. Thus we think that a robot can perform
communication with him/her. So we prepare four
patterns for each person, that is, category elements areidentical but action elements are not. Then we defineboth positive and negative rewards. The positive
rewards denote input sets having good action for eachmodule. On the other hand, the negative rewards
express other inputs.
We define simulation settings as shown in Table 4.
So we use parameters of updating equations in an actorcritic neural network as shown in Table 5.

Table 4: Simulation settingsparameter
Learning times
Number of middle units
Number of Learning person
Number of input pattern
Positive reward
Negative reward

value
3000
534
20.0 or 1.0-1.0

Table 5: Parameter settings for update equations

parameter

value

Discount rate of TD error 𝛾
Learning coefficient of middle units 𝜁0.09
0.001
1.0
0.01
0.01
0.4

variance 𝜎 2

Learning coefficient of critic 𝜂
Learning coefficient of actor 𝜌
Learning coefficient of soft-max 𝛽

4.3. Simulation Results
When the proposed model newly gets an input for an
unfamiliar person, it creates a novel dividual module forhim/her. We designed that each dividual module gets a
positive reward for an input containing good action andPublished by Atlantis Press
Copyright: the authors
37

Developing a Dividual Modela negative reward for the other inputs so as to form anappropriate dividual in a module. The accumulated
rewards for three dividual modules are illustrated inFig.4. This figure shows that each module learns
correctly because the accumulated rewards are globallymonotonically increasing.

Fig. 4: Accumulated reward for each module.

Figure 5 illustrates the transition of V(s). As shownin this figure, the output of each module takes higher
value than other two modules after 3,000 learning steps.
We can therefore identify the person by observing V(s).
Thus, we can confirm the process to choose anappropriate dividual out of plural dividuals for inputinformation.

5. Conclusion
In the present paper, in the field of HRI, we introduced
a concept of dividual. In addition, we developed adividual model to grow through interactions with others
using a modular neural network and reinforcement
learning (actor-critic). Then we confirmed the process toform dividuals and to choose an appropriate dividual outof plural dividuals.
As future work, we should implement social
dividual and group-oriented dividual. And we should
extend the dividual model considering actuallyinteraction. Finally, we should verify the validity of thedividual model.
In the near future, when a home robot is introduced
to our house, we hope that it should be developed tolearn and grow through interactions with family
members, and can separately correspond to each family
member. In order to realize such a home robot, we must
develop a system to enable smooth communication
between humans and robots. Once a dividual based
human-robot interaction system is developed, robots cantake a role in a family member and a doctor and thenrelationship between humans and robots will bedramatically changed.

Acknowledgements
This work was partly supported by President’s Special
Research Fund from Aichi Prefectural University,
Japan.

References

Fig. 5: State value V(s) from each module.

1. H. Inoue, T. Kanade, Y. Anzai, and H. Sena, “Robotics
Creation,” Iwanami Shoten, (2005) (in Japanese).
2. I. Shimoyama, T. Shibata, K. Ikuta, N. Hata, J. Tani, T.
Mathui, and H. Hirukawa, “Robots Frontier,” Iwanami
Shoten, (2005) (in Japanese).
3. M. Inaba, A. Kagami and K. Nishiwaki, “RobotsAnatomy,” Iwanami Shoten, (2005) (in Japanese).
4. Y. Honda, “Robots Revolution : Why do Google andAmazon invest it?,” Shodensha (2014) (in Japanese).
5. T. Fong, I. Nourbakhsh, and K. Dautenhahn, “A Survey ofSocially Interactive Robots,” Robotics and AutonomousSystems, 42, 143–166 (2003).
6. M. A. Goodrich and A. C. Schultz, “Human-RobotInteraction: A Survey,” Human-Computer Interaction,
1(3), 203– 275 (2007).

Published by Atlantis Press
Copyright: the authors
38

T. Tanaka and K. Kobayashi

7. T. Kanda, “Research Trends towards Social Robots inHRI,” Journal of the Robotics Society of Japan, 29(1), 2–5 (2011) (in Japanese).
8. T. Ono, M. Imai, T. Etani, and R. Nakatsu, “Construction ofRelationship between Humans and Robots,” Journal of Information Processing, 41(1) 158–166 (2000) (in Japanese).

9. K. Kobayashi and S. Yamada, “Human-RobotCooperation with Commands Embedded in Actions,”
Transactions of the Japanese Society forIntelligence, 21(1), 63–72 (2006) (in Japanese).

Artificial

10. H. Kozima, “Epigenesis of social intelligence and“theory of mind”,” PSJ SIG Notes. ICS, ICS122-3, 13–18 (2000) (in Japanese).
11. T. Shibata, “Research on Interaction between Human andSeal Robot, PARO,” Journal of the Robotics Society ofJapan, 29(1), 31–34 (2011) (in Japanese).
12. F. Naya, K. Shinozawa, and K. Kogure, “Human-Robot
Tac-tile Interaction,” IPSJ Magazine, 44(12), 1227–1232(2003) (in Japanese).
13. K. Hirano, “Who am I? : From Individual to Dividual,”
Kodansha Shinsho (2012) (in Japanese).
14. K. Suzuki, “Smooth Society and the Enemy,” Keisoshobo(2013) (in Japanese).
15. M. Kawamura and K. Kobayashi, “An Action Selection
Method Using Degree of Cooperation in a Multi-agent
Reinforcement Learning System,” Journal of Robotics,
Net-working and Artificial Life, 1(3), 231–236 (2014).
16. T. Tanaka and K. Kobayashi, “Development of aDividual Model Using a Modular Neural Network forHuman-Robot Interaction,” Proceedings of International
Conference on Artificial Life and Robotics(ICAROB2015), pp.107–110 (2015).
17. R. Anand, K. Mehrotra, C. K. Mohan, and S. Ranka,“Efficient Classification for Multiclass Problems Using
Modular Neural Network,” IEEE Trans. NeuralNetworks, 6(1), 117– 124 (1995).
18. R. S. Sutton and A. G. Barto, “Reinforcement Learning:
An introduction,” MIT press (1998).

Published by Atlantis Press
Copyright: the authors
39