Journal of Robotics, Networking and Artificial Life
Vol. 7(2); September (2020), pp. 81–85
DOI: https://doi.org/10.2991/jrnal.k.200528.002; ISSN 2405-9021; eISSN 2352-6386
https://www.atlantis-press.com/journals/jrnal

Research Article

Behavior Selection System for Human–RobotCooperation using Tensor SOM
Moeko Tominaga1,*, Yasunori Takemura2, Kazuo Ishii1
Graduate School of Life and Science Systems Engineering, Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu-ku, Kitakyushu,Fukuoka 808-0196, Japan2
Department of Engineering, Nishinippon Institute of Technology Technology, 1-11 Aratsu, Kanda-town, Miyako-gun, Fukuoka 800-0397, Japan
1ARTICLE INFOABSTRACT

Article History

With the progress of technology, the realization of a symbiotic society with human beings and robots sharing the sameenvironment has become an important subject. An example of this kind of systems is soccer game. Soccer is a multi-agent gamethat requires strategies by taking into account each member’s position and actions. In this paper, we discuss the results of thedevelopment of a learning system that uses self-organizing map to select behaviors depending on the situation. A set of possible
actions in soccer game is decided in advance and the algorithm is able to select the best option, given some specific conditions.

Received 29 November 2019
Accepted 01 April 2020
Keywords
Strategy
self-organizing map
team behavior
Tensor SOM
multi-agent system
human–robot cooperation© 2020 The Authors. Published by Atlantis Press SARL.
This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION

2. COOPERATIVE BEHAVIOR

Recently, the implementation of robots in society has become apossible solution to many problems, such as ensuring the safety ofa sustainable society, responding to a rapid population aging andpopulation decrease. Moreover, robots will represent the foundation of the future industry.

Cooperative behavior becomes a crucial aspect when different
autonomous agents interact while performing a common task.
Often a single agent is not much effective in accomplishing a task,
and in the last years many researchers have been studying MultiAgent Systems (MAS) to solve difficult problems.

To properly implement a robotized society, it is necessary to conduct and to present research outcomes in a manner that is easy tounderstand, avoiding differences between social expectations andthe direction of research and development.

The agents interact to each other and with the environment by taking
real time decisions based on the data acquired from the sensors [2,3].

Therefore, it is essential to discuss how to achieve coexistence withrobots and what a symbiotic society should look like. In such asociety, humans and robots interact with each other and they arecapable of mutual understanding. Not only they need to be awareof their own actions, but also of all the other agents’ actions, whereagent means each of the active subject involved.
The aim of this work is to develop a suitable algorithm to create
intelligent robots able to share the environments with humans.
Since soccer involves strategies, cooperation, unpredictable movements and common targets, it represents a good test bed for developing such algorithm.
Tensor Self-Organizing Map (Tensor SOM) [1] is used for thisscope.
Corresponding author. Email: tominaga.moeko382@mail.kyutech.jp*As a test bed of MAS, RoboCup [4], a project aimed to win theSoccer World Cup against Humans, encourages the cooperation ofmulti-agents using learning methods, such as reinforcement learning and neural networks. According to Sandholm and Crites [5],reinforcement learning can be used successfully for the iterated
prisoner’s dilemma, if sufficient measurements data and actions areavailable. In addition, Arai et al. [6] compared the Q-learning andProfit Sharing methods about the Pursuit Problem in a MAS, whenthe environment is modelled as a grid, and showed that cooperative behaviors emerge clearly among Profit Sharing. However, thesestudies have not yet considered applications for robots that operate
in a real environment.

3. ROBOT LEARNING
Among the learning algorithms, unsupervised learning is apromising method for MAS systems.

82

M. Tominaga et al. / Journal of Robotics, Networking and Artificial Life 7(2) 81–85

The advantage in unsupervised learning is that the robot is notrequired to have any previous information concerning the environment and prerequisite knowledge of the robot itself. However, sincemany tests and experiences are needed for learning, it is important for the designer to define in advance and clearly what are theimportant parameters and variables to consider during the game,so as to define the state space and the action space, and to reducethe cost and the time.
The design of the state space depends on what the robot is able to
do in the action space. The two spaces are mutually interconnected.
To design the state space in an efficient way, Takahashi and Asada[7] proposed a method in which the state space is initially divided
into two main states, and then divided recursively into many layerswith an increasing number of states. Nonetheless, they found thatsome bias can still affect the behavior of the robot and cause somewrong action and habit.

3.1. Human–Robot Cooperation
As mentioned above, a humans–robots cooperative society is a
topic of interest for many researchers and several works have been
carried out.
In such a symbiotic system, it is mandatory for the robots tounderstand and interpret humans’ behavior and act accordingly.
However, the target in research is often about the robots’ behavior only, or a behavior based on an interaction defined preciselybetween a specific number of humans and robots. Also, in a MAS
the communication between humans and robots is usually carriedout by means of some kind of interface, for example command
voice or gestures. But this might not be enough, especially when
the system is very complex.
Accordingly, the robot should be able to understand some situations
and adapt its behavior in a predictive manner, just as humans do.
The main goal of this research is to analyze the cooperative behaviors among soccer players and apply them to robots (see Figure 1).
A neural network based on Tensor SOM is designed to achieve
this target.
The input vector elements for a neural network is chosen so as
to achieve the lowest possible gap between humans and robots’actions. To obtain this, it is important at first to understand humans’
behavior and develop robots able to understand such behaviors andimitate them.

This paper focuses on the study of robots’ behavior in a futsal game,because this represents a good test bed, with a dynamic environment, several constrains and it requires a real-time planning. Theseare the characteristics of a common symbiotic system in which therobots may operate in the future.

4. SOM AND TENSOR SOM
Self-organizing map [8] is an unsupervised learning techniqueand is known as competitive learning, similar to information processing via neural circuits. Competitive learning is an importantconcept in hierarchical neural networks. Each input neuron in the
input layer (input space) is connected to all output layer neurons.
The strength of the relation between two neurons in consecutivelayers is decided by a specific weight assigned to each connection.
The input vectors “compete” with each other so as to find the best
output vector. With best output, it is meant that the output vector
whose elements are as similar as possible to the input vector elements.
To establish this similarity, the Euclidean distance is used. By
doing this, clusters into the output map are obtained. The cooperative hierarchy deals with the output vectors. While performing
the algorithm, the elements value of each vector is adjusted. If ahigh change of an element occurs in a vector, the vectors aroundit change accordingly, but in such a way that the strength of thischange decreases with the distance in the map from the first vector.
In particular, the amount of change of each element is decided by
the Gaussian function [9].
Since each vector modifies itself depending on the surrounding
vectors, this behavior is said to be cooperative. Input vectors
can have high dimension but the output results are shown in a
two-dimensional map.
With this structure, multi-dimensional input vectors can have
reduced dimensions in the output layer, and the features are clustered, so as to have similar features in the same area of the map.
Since the output results are shown in a two-dimensional form,
SOM is suitable for the visualization of complex scenarios.
Furthermore, the cooperative hierarchy and the smoothing process
performed by the Gaussian function allow to synthesize elements
and contents that were not clearly specified in the input vectors.
SOM has a high interpolation performance and it can generalize
concepts starting from a limited amount of input data. Because of
these characteristics, it is well suited for unsupervised learning andhighly valid outputs are expected.
A limit of SOM is that input vectors with variables of different nature are difficult to normalize. Also, the relations between
these inputs to produce outputs is relatively limited. An alternative
approach is to use Tensor SOM. With Tensor SOM it is possible tocreate a higher order input variables and a wider combinations ofinputs and, accordingly, output (Figure 2).

4.1. Algorithm
The Tensor SOM algorithm proceeds according to the following steps.
Figure 1 | A human–robot symbiotic system.

When an input vector is introduced in the algorithm, its Euclidean
distance from all the weights vectors is evaluated. The unit (neuron)M. Tominaga et al. / Journal of Robotics, Networking and Artificial Life 7(2) 81–85

832ùé 1a ki = exp ê - 2 z (1*) - z k(1) ú ki
êë 2s
úû

(3)

2ù
é 1
bl j = exp ê - 2 z (*2) - z l(2) ú lj
êë 2s
úû

(4)

where z (1*) - z k(1), z (*2) - z l(2) is the distance between the BMU andkilj

the surrounding units.
With the weights adjustments found above, the secondary and primary models values are adjusted as in Equations (5)–(9).
Update the secondary model:
Figure 2 | Structure of input/output tensor in Tensor SOM.

ykld =with the most similar weights is selected as the Best Matching
Unit (BMU).
Figure 2 shows the structure used to update the weights and training of the neural network. This method is detailed in Iwasaki and
Furukawa [1].
The input tensor X consists of all the input data. These inputs aredivided into some categories, three in Figure 2, and the indices are:d, i and j.
The secondary model represents the weights output tensor. At first
this is initialized with some random values.
In order to update the output tensor values, two primary models
are used. In the first of the primary model (above the input tensor
in Figure 2) the combinations of all the d and i conditions are evaluated. This forms a stack of l layers. Same happens for the other primary model (at the right of the input tensor in Figure 2), with thecombinations of all d and j conditions, producing a stack of k layers.
The steps required for the algorithm are detailed below.

The selection of the BMU in each layer, ki* , li* is carried out foreach input vector uild, vkjd with the following Equations (1) and (2),in which the Euclidean distance is calculated.

kl *j = arg minlLDåå(u - y ) ildkld2IJååa b x i =1 j =1

ki l j ijd

Igk =åa (6)kii =1Jgl =åb (7)ljj =1

Update the primary model:
uild =vkjd =1glJ

åb x 

1gk

l j ijd(8)

åa x (9)

j =1

Ii =1

ki ijd(1)

The target is to verify whether the Tensor SOM can be used as abehavior selection algorithm in a futsal game. In this case the player
we want to take some decision is that one who has the ball during
the game action analyzed.
As a first step, the tensor inputs need to be specified. We decided touse three categories of inputs: agents, environments and behaviors(Figure 3).

l =1 d =1KD

åå(v - y ) kjdkld2(5)5. EXPERIMENTS

4.1.1. Choose best matching unit

ki* = arg min

1gk gl(2)k =1 d =1

where ki* is the BMU in the ki layer, l *j is the BMU in the lj layer,and ykld represents the outputs in the secondary model.

4.1.2. Weight adjustments
Based on the distance from the BMU in the layer and on the radiuss, the weight adjustments are defined as in Equations (3) and (4).

Figure 3 | Image of input data.

M. Tominaga et al. / Journal of Robotics, Networking and Artificial Life 7(2) 81–85

84

Figure 4 | Structure of input data.

Figure 5 | Result of Tensor SOM expressed by the Component Plane
Matrix after learning.

Table 1 | Training parameters using Tensor SOM
Number of iterations
Map size
Max neighboring radius
Minimum neighboring radius
Number of learningn10smaxsmin
Epoch

100 × 100
2.0
0.2
200

The agent here is the target player, namely the player with the ball,and his current action, in movement (named run) or standing(named stop).

5.1. Experimental Results and Discussion
The results are analyzed by means of the Component Plane Matrix.
In Figure 5, it is shown the agent position and current action on theleft, the four environments conditions on the right side and finallythe matrix with the behaviors in the center.
We can see as the target player chose its behavior according to thedecision criteria summarized in Figure 4.

Environments consist of four situations:

In fact, in this situation he is running in his own field half, there areboth teammates and opponents in his surroundings and his choice
is to pass the ball (high value in the Component Plane Matrix).

1. Press & Helper: around the target player there are both teammates and opponents.

6. CONCLUSION

2. Nobody: the target player has no teammates nor opponents in
his surroundings.
3. Helper: around the target player there are only teammates.
4. Press: around the target player there are only opponents.
Finally, the behaviors consist of four possible actions performed by
the target player:
1. Pass: the ball is passed to a teammate.
2. Dribble: the player decides to continue running with the ball.
3. Shoot: the player tries to score a goal.
4. Kick out: the player decides to send the ball out of the field.
To each of this possible behavior is assigned a score from 1 to 4,
based on the situation in the game (environments inputs), where 4means best choice and 1 means worst choice.
In Figure 4, the inputs with the score for each behavior is shown,
given a position of the player in the field and his current action, forexample run. In this case, if nobody is in the surroundings of thetarget player, the best choice is to keep running with the ball, while
kick out the ball would be the worst decision.
With the defined conditions we trained the algorithm. In Table 1
are shown the training parameters used.

In this paper, we created a behavior selection system for soccer
robot players in a futsal game.
At first we developed an algorithm using Tensor SOM. This algorithm was selected because of its ability to deal with complex
systems where several inputs are present. Also, it is suited for unsupervised learning.
These inputs consist of three categories: behaviors, agents andenvironments. Then we created a set of training data to train thenetwork.
The results provided in the Component Plane Matrix are in agreement with the expectations and shows as this algorithm is able toselect the predicted behavior. Given a position and current actionof the target player and some specific situation in the game, he wasable to choose the best action. Tensor SOM can be a powerful toolin the development of MAS, even though further tests and experiments in real conditions are required to evaluate its potential.
As a future work, this algorithm will be implemented on the realsoccer robots used for RoboCup Middle Size League and tested inreal conditions.

CONFLICTS OF INTEREST
The authors declare they have no conflicts of interest.

M. Tominaga et al. / Journal of Robotics, Networking and Artificial Life 7(2) 81–85

REFERENCES[1] T. Iwasaki, T. Furukawa, Tensor SOM and tensor GTM: nonlinear tensor analysis by topographic mappings, Neural Netw. 77(2016), 107–125.
[2] P. Stone, M. Veloso, Multiagent systems: a survey from a machinelearning perspective, Autonom. Rob. 8 (2000), 345–383.
[3] K. Ueda, Emergence and multi agent system, Bifuran (2007)(Article in Japanese).
[4] S. Behnke, R. Sheh, S. Sariel, D.D. Lee (Eds.), RoboCup 2016: robot
World Cup XX, Springer-Verlag, Berlin, Heidelberg, 2017, p. 645.
[5] T.W. Sandholm, R.H. Crites, On multiagent Q-learning in asemi-competitive domain, In: G. Weiß, S. Sen (Eds.), Adaption
and Learning in Multi-Agent Systems. IJCAI 1995. Lecture Notes

85in Computer Science (Lecture Notes in Artificial Intelligence),
Springer, Berlin, Heidelberg, 1995, pp. 191-205.
[6] S. Arai, K. Miyazaki, S. Kobayashi, Methodology in multi-agent reinforcement learning: approaches by Q-learning and profit sharing, J.
Japanese Soc. Artif. Intell. 13 (1998), 609–618 (Article in Japanese).
[7] Y. Takahashi, M. Asada, Behavior acquisition by multi-layered
reinforcement learning, J. Robot. Soc. Japan 18 (2000), 1040–1046.
[8] A. Ultsch, H.P. Siemon, Kohonen’s self organizing feature maps
for exploratory data analysis, Proceedings of the International
Neural Network Conference (INNC), Kluwer Academic Press,
Dordrecht, Netherlands, 1990, pp. 305–308.
[9] L.A. Tu, Sustainability in urban planning and design, Improving
feature map quality of SOM based on adjusting the neighborhood
function, IntechOpen, 2019, pp. 1–17.

AUTHORS INTRODUCTION
Ms. Moeko Tominaga

Prof. Kazuo Ishii

She received her M.S. degree fromDepartment of Human Intelligence Systems,
at Kyushu Institute of Technology, Japan, in2018. She is pursuing the PhD at Kyushu
Institute of Technology, Department ofLife Science and Systems Engineering,under the supervision of Prof. Kazuo Ishii.
Her research interests include symbiosis ofhumans and robots.

He received PhD degree from theDepartment of Naval Architecture andOcean Engineering, University of Tokyo,Tokyo, Japan, in 1996. He is currently aProfessor with the Department of HumanIntelligences Systems, and the Director of
the Center for Socio-Robotic Synthesis,
Kyushu Institute of Technology, Kitakyushu,
Japan. His research interests include underwater robots, agricultural robots, RoboCup soccer robots and intelligent systems.

Dr. Yasunori Takemura
He received PhD degree from Kyushu
Institute of Technology, Japan in 2010 andnow he is working as an Associate Professor
at Nishinippon Institute of Technology, inJapan. His research area is about machinelearning, data mining and robotics.