# On adaptive self-organization in artificial robot organisms

Serge Kernbach, Heiko Hamann, Juergen Stradtner, Ronald Thenius, Thomas Schmickl, Karl Crailsheim, Anne van Rossum, Michèle Sebag, Nicolas Bredeche, Yao Yao, et al.

<figure><img src="image_1.png" /></figure>

## To cite this version:

Serge Kernbach, Heiko Hamann, Juergen Stradtner, Ronald Thenius, Thomas Schmickl, et al. On adaptive self-organization in artificial robot organisms. ADAPTIVE 2009 - The First International Conference on Adaptive and Self-adaptive Systems and Applications, Nov 2009, Athens, Greece. 10.1109/ComputationWorld.2009.9. inria-00413281

HAL Id: inria-00413281

https://inria.hal.science/inria-00413281

Submitted on 26 Mar 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# On adaptive self-organization in artificial robot organisms

Serge Kernbach*, Heiko Hamann†, Jürgen Stradner†, Ronald Thenius†, Thomas Schmickl†, Karl Crailsheim†, A.C. van Rossum‡, Michele Sebag§, Nicolas Bredeche§, Yao Yao¶, Guy Baele¶, Yves Van de Peer¶, Jon Timmis||, Maizura Mohktar||, Andy Tyrrell||, A.E. Eiben**, S.P. McKibbin††, Wenguo Liu‡‡, Alan F.T. Winfield‡‡

*Institute of Parallel and Distributed Systems, University of Stuttgart, Germany, Serge.Kernbach@ipvs.uni-stuttgart.de

†Artificial Life Lab, Karl-Franzens-University Graz, Universitätsplatz 2, A-8010 Graz, Austria,
{heiko.hamann,juergen.stradner,ronald.thenius,thomas.schmickl,karl.crailsheim}@uni-graz.at

‡Almende B.V., 3016 DJ Rotterdam, Netherlands, anne@almende.com

§TAO, LRI, Univ. Paris-Sud, CNRS, INRIA Saclay, France, {Michele.Sebag, Nicolas.Bredeche}@lri.fr

¶VIB Department Plant Systems Biology, Ghent University, Belgium,
{Yao.Yao, Guy.Baele, Yves.VandePeer}@psb.vib-ugent.be

||University of York, York, United Kingdom, {mm520, jt517, amt}@ohm.york.ac.uk

**Free University Amsterdam, gusz@cs.vu.nl

††Materials and Engineering Research Institute (MERI), Sheffield Hallam University, s.mckibbin@shu.ac.uk

‡‡Bristol Robotics Laboratory (BRL), UWE Bristol, {wenguo.liu, alan.winfield}@uwe.ac.uk

**Abstract**—In Nature, self-organization demonstrates very reliable and scalable collective behavior in a distributed fash-ion. In collective robotic systems, self-organization makes it possible to address both the problem of adaptation to quickly changing environment and compliance with user-defined target objectives. This paper describes on-going work on artificial self-organization within artificial robot organisms, performed in the framework of the Symbrion and Replicator European projects.

**Keywords**—adaptive system, self-adaptation, adaptive self-organization, collective robotics, artificial organisms

# I. INTRODUCTION

Adaptivity is a much-desired property of real-world sys-tems, where the system itself can adjust its own functionality or behavior to uncertainties and variations of the environ-ment. The issue of adaptivity has been considered in the theory of adaptive control (e.g. [1]), knowledge-based and deliberative systems (e.g. [2]), situated [3] and embodied [4] systems. Many different approaches are devoted to achieving adaptivity: different learning techniques [5], behavior-based [6], bio-inspired [7], evolutionary approaches [8] and many other.

Considering multi-robot systems, such as collective, swarm [9], reconfigurable and cellular robotics [10], we should note that these systems utilize another principle of control and organization: instead of one or several cen-tral controllers, collective systems undergo different self-organizing (SO) processes [11]. In particular, this work addresses a new type of collective systems [12]: many single swarm robots can aggregate into one multi-robot organism, see Fig. 1. This system is an object of research in the SYMBRION and REPLICATOR projects. Terminologically we say the disaggregated robots are in swarm-mode, whereas aggregated robots are in organism-mode. Here we observe

a new challenge. Robots in swarm-mode utilize SO phe-nomena as the main means of regulating functionality at the collective level. Aggregation, decision making, energetic homeostasis and other collective activities are created by artificial SO through bio-inspired [6] or derived [13] local rules. As shown by these and other works, the SO in the swarm-mode provides efficient, scalable and very reliable behavior. However, when measuring collective reactivity in terms of how fast a collective system is able to process information [6], we should remark, that SO remains a relatively slow organizational process.

<figure><img src="image_3.png" /><figcaption>Figure 1. Examples of swarm- and organism-modes. (a)-(b) Demo of concept, 2007: Real large-scale swarm of Jasmine robots and topological model of an organism; (c)-(d) Prototypes, 2009: a few robots in a swarm-mode and in a simple organism (images ©SYMBRION, REPLICATOR projects).</figcaption></figure>
Considering organism-mode, we face two contradictory requirements: we need decentralization, scalability and reliability provided by artificial SO, however we need much faster and much more adaptive regulative functionality. Making SO more self-adaptive and faster, while keeping scalability and decentralism represents one of the main challenges here. This paper focuses on different SO processes in the context of swarm- and organism-modes as well as in the transition between them. SO is viewed as a means toward an end — the ability of the self-organized robots to come up with a competent and robust response to an open environment under limited resources. In Sec.II this work gives a short overview about different SO processes running onboard and online in artificial organisms and introduces these approaches in the following sections. In Sec. XII we discuss several open problems and finally conclude this work.

## II. COMMON PICTURE OF DIFFERENT SO PROCESSES IN ARTIFICIAL ORGANISMS

To represent different SO processes in artificial organisms, let us consider a topological model of a planar (aggregated organism on 2D plane) dog-like structure, shown in Fig. 2. First, an organism represents an aggregation of indepen-

<figure><img src="image_4.png" /><figcaption>Figure 2. Topological model of a planar dog-like organism.</figcaption></figure>

dent modules, each of which possesses sensors, actuators (shown as active joints), internal controllers and an artificial “genome”, which reflects configurations of a module. From this viewpoint organism-mode is very similar to swarm-mode with the difference that all robots are spatially fixed on a grid, but with additional degrees of freedom through common buses and common actuation. Thus, it is expected that SO phenomena, although a characteristic of swarm-mode, can also be widely used in organism-mode. Considering Fig. 2, we can distinguish five self-organization processes running in the organism:

* **Swarm-mode.** As mentioned, swarm-mode is a classical application field of different SO phenomena. The Sec. III gives an overview of using learning and evolving in swarms. Since the system will be in swarm-mode only 15%-20% of the whole time, the relevance of swarm-mode for an organism is relatively low.

* **Developmental level.** Developmental processes describe how the structure and functionality are “growing” from non-aggregated and not-differentiated modules to organisms with complex monofunctional activity. More generally, during developmental phase, which can be performed on-line and on-board but also off-line and off-board, different adaptive mechanisms can be evolved through evolutionary/learning processes, as described in Sec. 3. Three examples of such approaches are described in Secs. V, VI and VII.

* **Homeostatic level.** All modules have different levels of energy, different genome, different goals. Being aggregated in the organism, from all modules should emerge an internal homeostatic system, which maintains endogenous steady state and protects the whole organism. Different SO processes are utilized on the homeostatic level, as described in Sec. VIII.

* **Cognitive level.** All modules possess independent sensors and through aggregation there appear several effects like spatial distribution of sensors, overlapping of functionality and increasing of redundancy. Sec. X describes several cognitive processes, which use SO effects.

* **Level of macroscopic locomotion.** Macroscopic locomotion represents one of the largest tasks for SO processes in the organisms. For example, an organism, shown in Fig. 2, has 4 legs, which have around 12 motors in active joints. All of them should be synchronized in order to obtain a uniform macroscopic locomotion. For a macroscopic SO-based control, such approaches as synchronization of coupled oscillators or adaptive hormone system, as described in Sec.XI can be used.

Despite the fact that all of these processes use SO phenomena on different levels, they all contribute to making the whole system act as one entity, being thereby fully decentralized with scalable functionality and behavior. In Sec.XII we give a common view on different SO processes and mention open problems in the organism-mode.

## III. SWARM-MODE: SELF-ORGANIZATION THROUGH LEARNING AND EVOLUTION

While biological entities are implicitly required to survive with a competitive advantage, self-organization in swarm of robots is also assessed from the collective behaviors of all individuals regarding the designer's objective. Hence, *both* environmental conditions *and* internal motivations define an implicit objective function. In other words, swarms of robots should be able to converge towards an efficient
behavioral strategy at both the individual and the population levels that maximize the intended objective and comply with the environment specific properties. This implies that each individual within the swarm must be endowed with internal variables, rewards and rules enforcing the desired behavior in an environment- and self-driven fashion. On the one hand, biological entities may rely on their instincts (curiosity, fear, ...) to provide a set of basic behaviors to explore the environment that are at least partially correlated with the optimal survival strategy. On the other hand, designing efficient SO artificial entities raises the question on how to provide capabilities to address survival, or in a broader sense: task optimization, in complex environments. This raises two main challenges:

* Defining self-driven rewards (curiosity, cognitive dissonance, ...) yielding a sufficient and safe exploration of the policy space;

* Defining a decentralized optimization process favoring desirable policies on the individual level and communication rules enforcing convergence towards an efficient behavioral strategy at the swarm level (i.e. emergence of a collective behavior).

An integrated approach to the above challenges explored within SYMBRION relies on the combination of Evolutionary and Machine Learning methods. Typically, criteria derived from Information Theory can be used to measure the new information gathered by an entity. A built-in instinct (maximize the gathered information) provides the entity with a “curiosity-like” bias, which should at least be partially correlated with the optimal strategy (e.g. curiosity is a first step towards finding energy sources in order to maximize both autonomy and exploration). An evolutionary framework driven by curiosity should provide a variety of behaviors, bootstrapping the optimization process towards an efficient swarm self-organization. Moreover, this kind of criteria can be reformulated so as to cope with designer preferences, standing for the Darwinian milieu. Setting the human back into the loop makes it possible to shape some specific desired behaviors related to the desired task to be solved (sparse interactive optimization, supported by preference learning).

Such locally defined criteria make it possible to address the behavior bootstrapping problem in complex environment. Then, the swarm self-organization process can be reformulated as a decentralized optimization problem where the swarm behavior may either be homogeneous or heterogeneous, and where the self-organization process is based on local diffusion so as to reach an equilibrium in the DS. This implies both the definition of specific optimization operator, which can be defined as stochastic, such as evolutionary operators, or deterministic, such as babbling algorithms, for spreading behaviors and enforcing convergence. Moreover, the particular setup imply that local explicit bootstrap (e.g. curiosity fitness) or human-driven (e.g. shaped curiosity

fitness) criteria should be taken into account as well as a global, not straight-forward, and implicit objective (e.g. ensuring energy autonomy) should be addressed. While explicit and implicit criteria may be correlated to some extent, they might diverge in the final state (e.g. maximizing curiosity may not be the best compromise to guarantee autonomy and survival in an extreme environment).

## IV. DEVELOPMENTAL LEVEL: ACHIEVING ADAPTIVITY THROUGH EVOLUTION

The role of evolutionary adaptive mechanisms is essential in all forms and at all levels of SYMBRION & REPLICA-TOR projects, including individual robots in swarm mode as well as in aggregated organisms. The main vision behind these projects includes that the controllers, or behavioral policies, undergo pervasive adaptation. Evolution and learning are two pivotal components of the “adaptation engine” facilitating this. To make such a system work one basically needs good reward systems (to support selection) and good evolutionary and learning operators (to support variation). Inherently to the SYMBRION philosophy, these mechanisms should work without any central control, and even with some degree of self-adaptivity to regulate their own parameters. In the system we obtain this way, one can distinguish three levels of evolution, see 3. First, genetic

<figure><img src="image_5.png" /><figcaption>Figure 3. Achieving adaptivity through evolution: three levels of evolution.</figcaption></figure>

evolution, concerning the artificial genomes in the individual robots. At this level, the individual robots form the units of selection and variation operators (mutation and crossover) work on the genomes. Second, cultural or social evolution, concerning the controllers directly, i.e., not the underlying genomes encoding them. At this level, the individual robots form the units of selection and variation operators work on the controllers. It makes sense to see this mechanism as a cultural or social process, where individual learning plays the role of mutation (change of controller within one robot) and individually acquired features are spread over the group of robots through communication (cross-fertilization or crossover of good ideas). At the third level we find organism evolution, where artificial organisms form the unit of selection and variation operators work on the joint genome or controller of the organism.

While in most evolutionary applications, including evolutionary robotics, selection and variation is managed by a central authority (the main EA loop or sometimes the
user), the SYMBRION system is inherently decentralized. In particular, we will not have a master or oracle determining which robots or organisms can mutate or transmit their genes/controllers to others. Instead, the units of selection will be autonomous, running all evolutionary and learning operators onboard, in an online fashion. Hence we will obtain a highly sophisticated version of embodied evolution [14]. The resulting system will give rise to the emergence of spatial, temporal and functional structures over time, thus being inherently self-organizing.

One of the greatest challenges to this end is represented by the reward functions. In general, we can use two types of rewards here. On the one hand, self-driven rewards that can be biased towards explorative behavior (curiosity), social behavior (e.g., sex-drive or some "basic instinct" for information sharing). On the other hand, we can use rewards that are based on measurable task-performance, for example, related to self-maintenance (energy level, system integrity), the size of the organism the robot is part of, or even some user defined task, like the number of red rocks collected or the time needed to identify the exit of a given room.

## V. DEVELOPMENTAL LEVEL: SHAPING ROBOT ORGANISMS AND CONTROLLERS BY VIRTUAL EMBRYOGENESIS

Another approach to shape a robotic organism is to simulate the self-organising processes, observable during the process of biological embryogenesis [15]. In this approach the robotic modules represent the single cells of an embryo. The behaviour of a single module (e.g., allow docking, switch to a predefined controller) is controlled by virtual morphogenes, that diffuse throughout the whole robotic organism. External influences (e.g. sensor inputs) or internal influences (e.g., defined morphogene concentrations) lead to the emission of other morphogenes. The conditions, under which a morphogene emission or a robotic behaviour is triggered, is coded in an artificial genome. By using artificial evolution on this genome, it is possible to optimise the body shape.

From the feedback-system, consisting of morphogene gradients, body-shape and robotic behaviour, a self-organised, evolvable, bio-inspired process arrieses, that allows the development of different robotic body-shapes in a evolutionary manner.

The process of virtual embryogenesis can also be used for shaping the controller of single robotic modules [16]. In this approach the growth of a virtual embryo is simulated to shape the network topology of a neural network. The usage of this approach seems to be advantageous for the shaping of heterogeneous neural networks (Fig. 4) consisting of cells with different functionalities, for example controlling cells or teaching cells, as described in [17].

The self-organised process of virtual embryogenesis [16] enables the adaptation of a (multi-) robotic systems as in an

<figure><img src="image_6.png" /><figcaption>Figure 4. Development of an artificial neural network in a virtual embryo [16]. In Fig. 4(a) the embryo is depicted. white dots indicate single cells. The shape of the embryo is influencing the specialisation of cells (Fig. 4(b)) within the embryo, that later develop into nodes of the neural network (Fig. 4(c))</figcaption></figure>

evolutionary manner on both, the level of the single robotic unit, and the level of the robotic organism.

## VI. DEVELOPMENTAL LEVEL: SO THROUGH REAL ROBOT AUTONOMOUS MORPHOGENESIS

A key requirement in the SYMBRION project is the transition from swarm to organism: autonomous morphogenesis. This is the process by which firstly, one or more robots (in swarm-mode) ‘decide’ that they need to self-assemble into an organism (e.g. in response to a barrier which a single robot cannot climb over) then, secondly, the robots self-assemble into the correct planar arrangement (as seen in Fig. 2) and, thirdly, the robots in the 2D planar organism assume the correct functionality (i.e. differentiate) according to their position in the organism. These three steps we can label as initiation, assembly and differentiation, respectively; the key stages in autonomous morphogenesis. After these three stages are complete the organism can lift itself from 2D planar configuration to 3D configuration and, with respect to locomotion, will function as a macroscopic whole, as outlined in section XI. Consider these three stages.
**1. Initiation.** Prior to initiation the robots are operating in swarm-mode, as outlined in section III. Initiation requires that only one robot in the swarm makes the decision to start the transition from swarm to organism – that robot then forms the ‘seed’ robot for the new organism. If we take the example of a barrier that needs to be crossed then a single robot might determine this condition through, firstly, its collision with the fixed barrier; secondly by running along the barrier and colliding with another robot and, thirdly, by then running along the barrier in the opposite direction and colliding with a third robot. Following this sequence of cues the robot needs to retreat from the barrier (in order to give the organism room to self-assemble); stop; select the genetic instructions for ‘barrier crossing organism’; then transition to assembling, as outlined below. Note that this SO process of initiation could allow several robots to make the same decision to initiate at the same time – however, this problem might be simply resolved by arranging that the signal to dock (see below) also suppresses the initiation behaviour in those robots close enough to see the signal.

**2. Assembly.** In this stage one or more of the robots in a partially assembled organism will express its DNA by signalling for other robots to dock with it, on certain faces, in order to build the planar organism. Several relevant approaches have been described in the literature; in one approach aimed at self-assembling static structures (i.e. 3D ‘houses’ built from intelligent autonomous robot bricks) each robot has a stigmergic rule set describing the whole structure and decides what to be (i.e. wall, corner) according to where it finds itself after randomly attaching [18]. While this methodology is provably correct it has the drawback of requiring a high level of random motion (i.e. energy) in the swarm so that robots will, eventually, find themselves in the right position for the growing structure. Another approach aimed at self-assembling planar mobile robot groups proposes a script (SWARMORPH) that describes how the structure self-assembles [19] the script is transferred into each newly-docked robot which then interprets the script to determine whether (and on which face) it should signal for another robot to dock, or not if it’s at the end of the chain. Since this approach actively signals for another robot it (probably) requires less energy in the swarm to complete the assembly process.

**3. Differentiation.** This stage is entered when the planar structure is physically complete but each robot is in effect an undifferentiated ‘cell’; it is in the right place in the body of the organism but has no specialised function. In the differentiating state each robot assumes a specialised function: for instance: *FootBot* if it is at the end of a leg structure and will be placed on the ground during walking; *LegBot* if it is one of several robots joining a foot to a knee or other joint or *JointBot* if it is required to bend at the junction between leg or arm structures. It is possible that differentiation will take place during assembly, in which case

this will not be a separate state; it is however important to recognise that assembly and differentiation are key functions required by each robot’s controller in organism-mode. The simplest approach to differentiation is one in which the function of each robot in the organism is also encoded in the genetic instructions for building the organism.

## VII. DEVELOPMENTAL LEVEL:

### ADAPTIVE SO THROUGH GENE EXPRESSION

This section discusses a new approach to deal with artificial organisms and robotic co-evolution using an agent-based framework/model, a powerful simulation modeling technique (see e.g. [20]).

**Problem definition.** As is commonly known, the genome and the concept of gene expression play critical roles in cell specification and morphogenesis. The genome encodes a bottom-up developmental approach which drives the single stem cell developing to complicated organisms with highly specific functionality (see [21] for an example). In addition, this extremely complicated process is performed without any centralised control while interaction with the environment allows for adaptation. These features of real-life genomes allow biological organisms to possess a robust self-organizing ability as well as flexible adaptation by providing various expression patterns. These features are also part of what we plan to achieve in the development of an artificial organism.

We propose to build an agent-based framework which is used to mimic a real-life genome and gene expression operations. This way, our framework can use a similar approach as in biology to solve the adaptive self-organization problem in our artificial organisms. This section discusses the general idea of how our mechanism works in self-organization. The robotic system (either in swarm or aggregated mode) can hence gain good functional or cooperative patterns through this self-organization process and the influences of the external environment.

**Methodology.** The general idea is that of a society where each member receives a particular identification according to its performance in cooperation. This identification will be used to guide the future cooperative operations. When some basic cooperation/ sub-organism occurs in a group, the partners in the cooperation/ sub-organism will compete with each other for leadership and the winning partner will activate the high level information (next stage development knowledge; for example the templates of shape, the group’s internal communication rules, ...) in its gene and build an agent for processing high level computation. Lower level cooperation can combine to higher level cooperation in a similar way but only the winning partner will be involved in this higher level combination because it can represent its whole group as one entity in a higher level cooperation.

**Implementation.** An agent-based framework will be loaded into each robot, along with an artificial genome. Important knowledge concerning cooperation constraints,
possible templates, etcetera will be decomposed into a
robot's genetic code. The encoding and decomposing rules
will be modeled according to Piaget stages of development
theories (see e.g. [22]). The goal is not to simply distribute
the knowledge into genes as the knowledge will be classified
corresponding to different stages in the whole development
process. Different stage developmental information will be
encoded in different levels of abstraction in the genome. In
the initial stage, for example, the system should focus on
one-to-one docking and communication operations, so there
will have to be a layer present in the genome which is used
to store this kind of knowledge. In a later stage, the system
needs to take care of more complex aspects like shape,
functionalities and physical constraints, the knowledge of
which will be stored in a higher layer in the genome.

In each robot, there is a controller which will be able to
develop various agents to read the different genes from the
robot's genome (see Fig. 5). The controller's functionalities
will be determined by the interaction of these self-developed
agents and the functional patterns of the controller will
emerge from the self-organizing process in swarm agents.
A controller can change its knowledge and functions by
developing new agents to read new genes in its genome.
Each robot shares similar genes in this design but according
to its specific environment, each robot will be able to activate
different genetic options and develop different functions dur-
ing its lifetime. Initially, each robot has one basic controller
which is built using the initial stage information stored in
its genes. Further, each robot also has a certain cooperation
credit value from the moment the controller is built. This
credit value shows the usefulness of this robot to the system.
When a robot wants to cooperate with other robots, it
needs to pay some credit value to its potential partner.
When another partner accepts this value, the cooperation
will be able to proceed. If a potential partner refuses to
receive this value (for example the partner doesn't need to
cooperate with others or the value is not enough to meet
the partner's expectation) the cooperation will be canceled.
This way, the system will have an emerged identification
pattern which come from the self-organizing process of
robot's cooperation.

**Example.** For example, the robot will be able to adjust its cooperation price (credit value) according to its fitness status when it is involved in cooperation. If the leading controller in a cooperation can not afford the new prices of other robots, the robots may decide to find a new leading controller in the group. This ensures that only good cooperating groups will be selected. This way, a system can develop through several stages to achieve better shape and functionality until such complex cooperation is no longer required in the given environment (for instance, the task is done). In the development process, each robot controller can specify its functionality by creating new agents and remove or replace old agents. The way the system is doing this will depend on

<figure><img src="image_7.png" /><figcaption>Figure 5. Structure diagram of the agent-based framework for robotic cooperation.</figcaption></figure>

the information in the genome and the environmental situa-
tion. Except for a few predefined constraints (for example,
for safety and avoiding simple mistakes), all functionalities
and strategies will emerge from the interaction of robots in
development processes. The formation process depends on
the environment. The integration conditions will be classified
into different levels of abstraction and stored in genes
independently.

VIII. HOMEOSTATIC LEVEL:

SO WITHIN ROBOTIC ARTIFICIAL IMMUNE SYSTEMS

Each SYMBRION robot will contain an individual artificial immune system (AIS) that is capable of identifying and predicting when the robot will fall out of normal operational conditions. This may be a fault that has occurred within the robot caused by some mechanical problem, or an environmental impact on the system. However, the AIS is not limited to a single robot, but an artificial immune network will be created between robots to allow for the sharing of immunological information between units, and the communication between units to identify potential problematic units and prevent them from joining the organism.

In our work, we make use of various immune system
metaphors for the creation of our artificial system, ranging
from the provision of an *innate* or pre-programmed type of
response that is typically static during the lifetime of the unit,
and an *adaptive* response that improves during the operation of the unit. Each AIS has the potential to be unique on every robot, as what each robot senses and performs during it's lifetime may well be different : and this will drive the evolution of the adaptive AIS.

Of most interest to work in this paper is the adaptive aspect of the AIS. We envisage a system that can maintain it's own unique immune system through a combined process of clonal selection (a way in which new artificial detectors can be generated that takes into account heuristic information) which allows us to develop a self-organising memory structure for the AIS, based on stimulation and death of error detectors in the AIS population. For inter-robot immunity, we employ ideas from immune network theory [23]. Immune network theory is now quite dated within the immunological community, but has many useful lessons for us to create self-organising memory structures. A network occurs due to the ability of paratopes (molecular portions of an antibody) located on B cells, to match against idiotopes (other molecular portions of an antibody) on other B cells. The binding between idiotopes and paratopes has the effect of stimulating the B cells. This is because the paratopes on B cells react to the idiotopes on similar B cells, as it would an antigen. However, to counter the reaction there is a certain amount of suppression between B cells which acts as a regulatory mechanism. This interaction of B cells due to the network, was said to contribute to a stable memory structure, and account for the retainment of memory cells, even in the absence of antigen.

This idea can be exploited in an artificial context through the creation of an artificial immune network, typically of robots that can be used to determine if other units within a network may be heading towards some form of failure. A simple *binding* system (or interaction) between units is required to compare how similar states of the unit are: one simple metric is Euclidean distance. This bind goes towards stimulating a B-cell (or in this case a robotic unit), with a strong bind indicating a strong similarity between the units. This *stimulation level* of the unit is used to help regulate the survival (or otherwise) of the robot, and help decide if the unit should be allowed to join an organism: as if the unit is potentially faulty, then joining the organism may well be problematic. What emerges is a network of self-similar robots supporting each other over time.

## IX. COGNITIVE LEVEL: SENSORY-MOTOR FUSION IN A MODULAR ROBOTICS SYSTEM

A modular robotic system is a domain that crosses several/all robotics domains in that the system operates on the basis of single autonomous robots, swarms of loosely coupled multiple robots and tightly coupled multiple robot collections in the form of complex robot-organisms. As a result of this generality of the system, the robots employed must be able to operate under all these circumstances

showing great adaptability to both environmental and system changes. In REPLICATOR, the single robot cell has a degree of autonomy that is not found in other modular robotic systems[24][25][10]. Each robot is capable of locomotion on a plane and is thus able to move around freely on its own under many conditions. One of the robot designs is capable of holonomic motion on a plane and the other is capable of overcoming small obstacles and, if it finds itself flipped onto its back, it can continue to move around upside-down. This level of autonomy places a higher degree of importance on the single robot cell within the system as a whole.

**Sensory-Motor Coordination.** From the perspective of sensory-motor coordination, it is a difficult problem to implement an architecture that is capable of operating under these different modes of operation. Sensory-motor coordination is an essential quality for any embodied agent to possess. It allows the efficient transfer of information from motor actions to sensory sensations and to higher levels that are more decoupled from the sensory-motor flow that can integrate dynamics over time. Also, efficient implementation of sensory-motor coordination can allow the robot to seek out and utilise sensory information that is most relevant to it under the current environmental conditions. Moreover, by developing a proprioceptive type sensory system, the robot agents can begin to ground sensory inputs to sensations that link them with their environment. The relationship between the robot and the environment in which it senses and acts can be learned and exploited.

**Sensory-Motor Fusion.** Our approach to the sensory-motor fusion for REPLICATOR modular robotic system consists in decentralising the system at the level of the individual robot cell. The single robot cell is the only unit that will not physically change during its operation, only its interface with other units is a dynamic configuration. The single robot is a natural level of decomposition as each cell is capable of acting on and sensing the environment immediately in its vicinity. Therefore any actions, sensations, processes running on an individual robot will be inherently relevant to that particular cell. Interfacing with other robot cells in the system can arise as part of a self-organizing process that can withstand perturbing fluctuations and yet still undergo useful transformations.

Since it is impossible to know in which situation a robot will find itself at any given moment in time, the system that implements the sensory-motor coordination should be adaptable to change and should work for a single robot, a robot swarm and a robot organism. What is clear from this is that the robot will operate either on its own or in conjunction with others. Normally, sensory-motor coordination can be achieved on a single robot by tightly coupling the sensory and motor systems in order to exploit specific sensor flows however, in a multi-robot multi-configuration system, this is not enough. The single robot should be able to sense if it is operating alone or in cooperation with others, i.e. if it is
physically connected to another robot cell or within communication range of another cell. By identifying these situations a robot cell can adapt its sensory-motor coordination to suit a particular context.

We propose a Recurrent Neural Network (RNN) that takes inspiration from similar models used for language processing, recognising sensory-motor flows and for imitation [26][27][28][29]. What is common to these models is that they are able to articulate sensory-motor flows over extended periods of time and that they do so in a dynamic and self-organising fashion. The benefit of using an RNN is that it is capable of learning temporal sequences by automatically fusing information from many sensors of possibly different modalities and across varying timescales. For mobile robots that operate in changing environments it is important that they can extract and exploit the dynamics of the environment and RNNs have been shown to be a suitable control representation for evolving this capability [30][31]. The drawback of using neural networks can be that they do not lend themselves well to the extraction of the rules on which they operate. This drawback can be outweighed by their ability to both generate new, but similar behaviours, on unseen input data or to recognise and generalise unseen sensor patterns. The idea of a Parametric Bias (PB) in the RNN model is to allow an outside interaction with the model that can change the way that it behaves according to the learned/evolved behaviours or sensory-motor patterns. It is through the use of these PB nodes that the RNN is capbale of both recognising and generating sensory-motor flows. This enables both a bottom-up and top-down approach to building intelligent systems. Another advantage of this approach is that multiple models residing on separate robot cells can be dynamically hooked together to produce a more complex system from similar individual units. The dynamics of the system should self-organise according to a distal fitness function or learning signal that promotes stability within the system.

Allowing individual robots to encapsulate and process the sensory-motor flow within a single robot cell enables the distributed processing of sensory information that can be shared through specific channels of docked robots cells or through communication channels of more loosely cooperating swarming robots.

# X. COGNITIVE LEVEL: SELF-ORGANIZATION IN COGNITIVE SENSOR FUSION

This section describes a self-organizing approach for sensor fusion on modular robots. It follows the course of problem definition, illustrative example, current work, methodology, implementation and discussion.

**Problem definition.** The type of robots that are addressed in this article are so-called modular robots. They can undergo drastic metamorphosis from say, a snake to a spider form. This poses stringent requirements on the way sensor

data is processed and the nature of the sensor fusion architecture. The scientific challenge is coined meso-morphosis: internal change alongside (body) metamorphosis. Or in other words: how to survive from caterpillar to butterfly.

**Leading example.** Consider a robot snake with cameras turned on at the head and the tail. First visual processing might occur at the modules near the head and the tail. Higher-level processing at subsequent modules. The sensor fusion architecture is aligned alongside the body of the snake robot. Then, the snake morphs into a spider robot. The body may now contain the cameras and the legs the visual data processing units.

**Related work.** There are so-called cognitive frameworks that allow for awareness of not only the environment, but also the body itself. Vernon et al. [32] have an excellent overview on cognitive frameworks. They distinguish the cognitivist approach from the emergent systems approach. The former being defined as operations on symbolic representations. The latter as an umbrella term for dynamical, connectionist and enactive systems. Their twelve distinctions will not be reiterated over here, but an example will be given in the context of sensor fusion. The cognitivist approach would ask for primitives like “is there a hole in the ground between me and robot X”. The designer defines a symbol “hole” and the visual architecture to classify entities in the field as a hole. An emergent approach would define a goal like “go to robot” or “aggregate”. It needs an environment with holes and it uses a reinforcement strategy to let the robot acquire sufficient knowledge about holes to reach the goal.

**Methodology.** In this section the emergent approach is endorsed. Vernon et al. explain: “adaptation ... in emergent systems ... entails a structural alteration or reorganization to effect a new set of dynamics”. In essence, sensor fusion combines data from multiple sensors across several modalities into representations that can be used in subsequent stages. For that reason, data processing units, or filters, are used in topological configurations. Filter output is fed into the input of next “layer” filters in a hetero-hierarchical set-up. For emergent cognitive sensor fusion three components are needed in this methodology for self-organized sensor fusion:

1. A network of primitive filters that perform sparse coding, auto-associative coupling, saliency detection, etcetera;

2. A reconfigurable topology of those filters and meta-information that guides reconfiguration;

3. A search process that finds those topologies that correspond to cognitive notions like attention, anticipation as in existing cognitive models.

**Implementation.** All three points of the methodology are addressed in this section:

1. The type of filters that have been implemented are 2D feature maps as described by Itti and Koch [33]. An orientation filter decomposes an image in patches that only
respond to lines with a specific orientation. This can be
done for color, intensity and even other modalities. The
feature filters contribute to an overall saliency map. Now,
the horizontal line filter might need to be downscaled in
importance with respect to color and intensity. The desired
weights of the feature maps (also a weight of zero) is almost
impossible to tell in advance and might differ per task
and robot morphology. Hence, a self-organized system that
comes up with a proper filter topology is what is required
over here.

2.) For a reconfigurable topology a developmental engine as described in section V is used. It is an implementation of the gene regulatory network by Bongard [34]. This type of engine takes a genome as input and has a graph as output. The graph is in Bongard’s work interpreted as the body of an artificial organism or as a neural network. For sensor fusion the graph will be interpreted as a topology between filters and connections between them.

3.) Comparison of the developed sensor fusion topology with cognitive models is necessary to describe its level of cognition. The basal ganglia analogue in Shanahan's robot [35] intercepts the recommendations of the saliency based system and modifies them using an internal simulation mechanism. In a world of power outlets and other robots, detecting the first might become biased (attention) when the robot is running low on batteries.

**Discussion.** The current implementation is at the second step, however, it is already worthwhile to look forward to step 4 and beyond. There are two additional steps necessary to obtain full-fledged meso-morphosis:

4.) Post-development use of the gene regulatory network. The network level is then able to make changes from spider to snake form without the need for a new genome: online self-organization;

5.) Online self-organization that preserves cognitive capabilities. To put it simply: the snake should be able to remember things from its life as spider.

XI. LEVEL OF MACROSCOPIC LOCOMOTION:
ARTIFICIAL HOMEOSTATIC HORMONE SYSTEMS

An option to generate SO processes, that are leveraged
to control complex multi-modular robot organisms, is the
mimicry of hormone systems. Our approach is based on
differential equations, that model the production, flow, and
reduction of hormones, and on structures of compartments
containing hormone concentrations that create the embodi-
ment of the hormone system, see [36], [37]. The compart-
ments establish a certain locality either within the robot or
within the robotic organism. Hormone productions, that are
triggered by sensor input, will establish a decreasing gradient
within the robot/organism from the position of the sensor
to the opposite side. Homeostatic processes are intrinsic to
such systems and will be formed automatically. Thus, the
hormone system can be interpreted as a SO dynamic system.

<figure><img src="image_8.png" /><figcaption>Figure 6. Schematic representation of a hormone concentration; in a first phase, an artificial organism of two robotic modules moves governed by an oscillating hormone concentration; in a second phase, an environmental change (obstacle) disturbers this limit cycle behavior; in a third phase, a third module connects to the organism and another limit cycle is established.</figcaption></figure>

Given a static sensor input or unchanged periodic sensory
stimuli for longer periods the system will always converge
to an attractor/equilibrium (e.g., fixed point, limit cycle, see
Fig. 6). A changing environment or the robots' actions them-
selves change the sensory input, thus, disturbing the current
equilibrium. As a reaction, the hormone system rearranges
the hormone concentrations and will reach (possibly another)
equilibrium in a self-organized process (see Fig. 6).

Furthermore, there exists a second process besides the dy-
namic environment which is a disturbing factor to the current
equilibrium of the hormone values. A reconfigurable robot
organism consists of autonomous robot modules. Further
modules dock to or connected modules release themselves
from the organism. This change of the shape of the robot
organism has, in turn, an effect on all the other modules of
the organism (see Fig. 6).

In an organism forming a simple line (e.g., snake-like
shape, see Fig. 6), environmental influence triggers, through
sensory input, the production of hormones that result in
a gradient of hormone concentrations within the organism.
In a process of symmetry breaking a differentiation into a
head module and a tail module is generated. Furthermore,
a threshold of a “head-” and a “tail-hormone” determines
the positions for legs in the middle of the snake-organism.
In this way different body shapes are established by self-
organized reconfiguration processes. We prefer SO as the
main design paradigm instead of standard approaches (e.g.,
predefined hand-coded shapes) because the latter would be
constricted to situations for which they were designed. How-
ever, our applications will have dynamic environments with
unforeseen properties. The approach of self-organization in
connection with evolutionary methods will help to overcome
the challenge of designing adaptive behavior in dynamic environments. The possibility of self-reconfiguration gives the organism the needed plasticity and adaptability.

## XII. CONCLUSION: COMMON VIEW

In this paper we represented an overview over diverse adaptive processes in artificial organisms, most of which use different self-organizing approaches as main control mechanisms. Since these approaches and mechanisms are work-in-progress, we are not intended to give any detailed results, in contrast we are going to create a common picture of all of them.

Generalizing, we observe SO-related processes on three following levels:

* **The whole artificial organism is a SO-system.** It consists of a group of autonomous robots, whose behavior is regulated by their individual controllers. These entities interact with each other and the environment, thus producing group behavior that is not a simple linear aggregation of the individual behaviors. We face the canonical challenge of engineering SO systems here: the point of impact of the designer/experimenter is at local level (specifying the controller for each individual), while the target behavior is specified on the level of the group.

* **There exist particular types of controllers that are SO systems themselves.** This means that the controller cannot be described as a usual input-output machine (like a rule-set or neural net, or decision tree). Instead, the controller itself represent a set of interacting agents/rules and the response of this controllers to a particular input is determined by the interaction rules.

* **Developmental phase.** There are methods for making/finding/developing controllers that fall in the category of SO systems. The difference with the option 2 is that in option 2 it is the result of the making/finding/developing (i.e., the controller) that is an SO system, here it is the method leading to this result.

There are several open problems of creating artificial SO: benchmarks, performance measurement and optimizing SO-control - currently it is unclear how to do this; increasing predictability of self-organizing control - it is intended to use different derivation and evolving approaches in achieving desired emergence by SO; scalability and reliability - since not all SO-mechanisms provide scalability, we need to develop approaches guaranteeing scalability in a defined range.

Finalizing this work, we would like to point out one important issue: artificial organisms can be viewed as extremely simplified analogues of living organisms. Both living and artificial organisms face similar problems – getting energy, surviving in environment, different forms of self-protection and self-awareness, organization of long-term and short-term developmental processes and others. On the basis of artificial

organisms we can gain deeper insights into such issues as long-term evolution and its controllability, phenomena of individual and collective intelligence, mechanisms of multi-cellular regulation and others issues which are highly relevant in our understanding of the complexity of life.

## ACKNOWLEDGMENT

The “SYMBRION” project is funded by the European Commission within the work programme “Future and Emergent Technologies Proactive” under the grant agreement no. 216342. The “REPLICATOR” project is funded within the work programme “Cognitive Systems, Interaction, Robotics” under the grant agreement no. 216240. Additionally, we want to thank all members of projects for fruitful discussions.

## REFERENCES

[1] K. Astrom and B. Wittenmark, *Adaptive Control*. Addison Wesley, 1989.

[2] G. Weiss, *Multiagent systems. A modern approach to distributed artificial intelligence*. MIT Press, 1999.

[3] M. Mataric, 02, "Situated robotics," *Encyclopedia of Cognitive Science, Nature Publishers Group, Macmillian Reference Ltd.*, 2002.

[4] R. Pfeifer, F. Iida, and G. Gomez, "Morphological computation for adaptive behavior and cognition," *International Congress Series*, no. 1292, pp. 22–, 2006.

[5] L. Bull, M. Studley, A. Bagnall, and I. Whitley, "Learning classifier system ensembles with rule-sharing," *IEEE Transactions on Evolutionary Computation*, vol. 11, no. 4, pp. 496–502, August 2007.

[6] S. Kernbach, R. Thenius, O. Kernbach, and T. Schmickl, "Remembodiment of honeybee aggregation behavior in artificial micro-robotic system," *Adaptive Behavior*, vol. 17, no. 3, pp. 237–259, 2009.

[7] D. Cliff, "Biologically-inspired computing approaches to cognitive systems: a partial tour of the literature," *Hewlett-Packard Company*, 2003.

[8] A. Eiben and J. Smith, *Introduction to Evolutionary Computation*, ser. Natural Computing Series. Springer, 2003.

[9] S. Kornienko, O. Kornienko, and P. Levi, "Ir-based communication and perception in microrobotic swarms," in *Proc. of the IROS 2005, Edmonton, Canada*, 2005.

[10] S. Murata and H. Kurokawa, "Self-reconfigurable robot: Shape-changing cellular robots can exceed conventional robot flexibility," *IEEE Robotics & Automation Magazine*, 2007.

[11] H. Haken, *Synergetics: Introduction and Advanced Topics*. Springer, 1983.

[12] S. Kornienko, O. Kornienko, A. Nagarathinam, and P. Levi, "From real robot swarm to evolutionary multi-robot organism," in *Proc. of the CEC2007, Singapore*, 2007, pp. 1483–1490.
[13] S. Kornienko, O. Kornienko, and P. Levi, "Generation of desired emergent behavior in swarm of micro-robots," in *Proc. of the 16th European Conf. on AI (ECAI 2004)*, Valencia, Spain, 2004.

[14] R. A. Watson, S. G. Ficici, and J. B. Pollack, "Embodied evolution: Distributing an evolutionary algorithm in a population of robots," *Robotics and Autonomous Systems*, vol. 39, no. 1, pp. 1-18, April 2002. [Online]. Available: http://eprints.ecs.soton.ac.uk/10620/

[15] G. B. Müller, "Evo-devo: extending the evolutionary synthesis," *Nature Reviews Genetics*, vol. 8, pp. 943-949, 2007.

[16] R. Thenius, T. Schmickl, and K. Crailsheim, "Novel concept of modelling embryology for structuring an artificial neural network," in *MATHMOD 2009 - 6th Vienna International Conference on Mathematical Modelling*, 2009.

[17] S. Nolfi and D. Parisi, "Auto-teaching: networks that develop their own teaching input," in *Proc. Second European Conference on Artificial Life*, J. Deneubourg, H. Bersini, S. Goss, G. Nicolis, and R. Dagonnier, Eds., 1993.

[18] A. Grushin and J. A. Reggia, "Automated design of distributed control rules for the self-assembly of prespecified artificial structures," *Robot. Auton. Syst.*, vol. 56, no. 4, pp. 334-359, 2008.

[19] A. Christensen, R. O'grady, and M. Dorigo, "Swarmscript: a language for arbitrary morphology generation in self-assembling robots," *Swarm Intelligence*, vol. 2, no. 2, pp. 143-165, December 2008.

[20] E. Bonabeau, "Agent-based modeling: Methods and techniques for simulating human systems," *Proceedings of the National Academy of Sciences USA*, vol. 99, pp. 7280-7287, 2002.

[21] T. S. Tanaka, T. Kunath, W. L. Kimber, S. A. Jaradat, C. A. Stagg, M. Usuda, T. Yokota, H. Niwa, J. Rossant, and M. S. Ko, "Gene expression profiling of embryo-derived stem cells reveals candidate genes associated with pluripotency and lineage specificity," *Genome Research*, vol. 12, no. 12, pp. 1921-1928, 2002.

[22] R. N. Aslin and J. Fiser, "Methodological challenges for understanding cognitive development in infants," *Trends in Cognitive Sciences*, vol. 9, no. 3, pp. 92-98, 2005.

[23] J. D. Farmer, N. H. Packard, and A. S. Perelson, "The immune system, adaptation, and machine learning," *Physica D*, vol. 22, pp. 187-204, 1986.

[24] W.-M. Shen, M. Krivokon, H. Chiu, J. Everist, M. Rubenstein, and J. Venkatesh, "Multimode locomotion for reconfigurable robots," *Autonomous Robots*, vol. 20, no. 2, pp. 165-177, 2006.

[25] Y. Zhang, M. Yim, C. Eldershaw, D. Duff, and K. Rufas, "Scalable and reconfigurable configurations and locomotion gaits for chain-type modular reconfigurable robots," in *International Symposium on Computational Intelligence in Robotics and Automation*, 2004, pp. 893-899.

[26] Y. Sugita and J. Tani, *Mirror Neurons and the Evolution of Brain and Language*. John Benjamins Publishing, 2002, ch. A connectionist model which unifies the behavioral and the linguistic processes: Results from robot learning experiments, pp. 363-376.

[27] E. Tuci, V. Trianni, and M. Dorigo, "'feeling' the flow of time through sensorimotor co-ordination," *Connection Science*, vol. 16:4, pp. 301-324, 2004.

[28] J. Tani and S. Nolfi, "Learning to perceive the world as articulated: an approach for hierarchical learning in sensory-motor systems," *Neural Networks*, vol. 12, no. 7-8, pp. 1131-1141, 1999.

[29] R. Yokoya, T. Ogata, J. Tani, K. Komatani, and H. Okuno, "Experience-based imitation using rnnpb," *Advanced Robotics*, vol. 21, no. 12, pp. 1351-1367, 2007.

[30] S. P. McKibbin, B. Amavasai, A. N. Selvan, F. Caparrelli, and W. A. Othman, "Recurrent neural robot controllers: feedback mechanisms for identifying environmental motion dynamics," *Artif. Intell. Rev.*, vol. 27, no. 2-3, pp. 113-130, 2007.

[31] S. P. McKibbin, B. Amavasai, A. N. Selvan, F. Caparrelli, and W. Othman, "The role of sensory-motor coordination: Identifying environmental motion dynamics with dynamic neural networks," in *Proceedings of the International Conference on Informatics in Control, Automation & Robotics (ICINCO 2008)*, 2008.

[32] D. Vernon, G. Metta, and G. Sandini, "A survey of artificial cognitive systems: Implications for the autonomous development of mental capabilities in computational agents," *IEEE Transactions on Evolutionary Computation*, vol. 11, no. 2, pp. 151-180, 2007.

[33] L. Itti and C. Koch, "Computational modelling of visual attention," *Nature Reviews Neuroscience*, vol. 2, no. 3, pp. 194-203, 2001.

[34] J. Bongard, "Evolving modular genetic regulatory networks," in *Proceedings of The IEEE 2002 Congress on Evolutionary Computation (CEC2002)*, 2002, pp. 1872-1877.

[35] M. Shanahan, "A cognitive architecture that combines internal simulation with a global workspace," *Consciousness and Cognition*, vol. 15, no. 2, pp. 433-449, 2006.

[36] T. Schmickl and K. Crailsheim, "Modelling a hormone-based robot controller," in *MATHMOD 2009 - 6th Vienna International Conference on Mathematical Modelling*, 2009.

[37] J. Stradner, H. Hamann, T. Schmickl, and K. Crailsheim, "Analysis and implementation of an artificial homeostatic hormone system: A first case study in robotic hardware," in *IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'09)*. IEEE Press, 2009.