Guided Self-Organization in a Dynamic Embodied System Based on Attractor Selection Mechanism

Guided self-organization can be regarded as a paradigm proposed to understand how to guide a self-organizing system towards desirable behaviors, while maintaining its non-deterministic dynamics with emergent features. It is, however, not a trivial problem to guide the self-organizing behavior of physically embodied systems like robots, as the behavioral dynamics are results of interactions among their controller, mechanical dynamics of the body, and the environment. This paper presents a guided self-organization approach for dynamic robots based on a coupling between the system mechanical dynamics with an internal control structure known as the attractor selection mechanism. The mechanism enables the robot to gracefully shift between random and deterministic behaviors, represented by a number of attractors, depending on internally generated stochastic perturbation and sensory input. The robot used in this paper is a simulated curved beam hopping robot: a system with a variety of mechanical dynamics which depends on its actuation frequencies. Despite the simplicity of the approach, it will be shown how the approach regulates the probability of the robot to reach a goal through the interplay among the sensory input, the level of inherent stochastic perturbation, i.e., noise, and the mechanical dynamics.

Related papers:

title: Variants of guided self-organization for robot control
abstract: Autonomous robots can generate exploratory behavior by self-organization of the sensorimotor loop. We show that the behavioral manifold that is covered in this way can be modified in a goal-dependent way without reducing the self-induced activity of the robot. We present three strategies for guided self-organization, namely by using external rewards, a problem-specific error function, or assumptions about the symmetries of the desired behavior. The strategies are analyzed for two different robots in a physically realistic simulation.

title: Evolving Spatiotemporal Coordination in a Modular Robotic System
abstract: In this paper we present a novel information-theoretic measure of spatiotemporal coordination in a modular robotic system, and use it as a fitness function in evolving the system This approach exemplifies a new methodology formalizing co-evolution in multi-agent adaptive systems: information-driven evolutionary design The methodology attempts to link together different aspects of information transfer involved in adaptive systems, and suggests to approximate direct task-specific fitness functions with intrinsic selection pressures In particular, the information-theoretic measure of coordination employed in this work estimates the generalized correlation entropy K2 and the generalized excess entropy E2 computed over a multivariate time series of actuators' states The simulated modular robotic system evolved according to the new measure exhibits regular locomotion and performs well in challenging terrains.

title: A Sensor-Based Learning Algorithm for the Self-Organization of Robot Behavior
abstract: Author to whom correspondence should be addressed. 30 November 2008 / Accepted: 26 February 2009 / Published: 4 March 2009Abstract: Ideally, sensory information forms the only source of information to a robot. Weconsider an algorithm for the self-organization of a controller. At short time scales the con-troller is merely reactive but the parameter dynamics and the acquisition of knowledge by aninternal model lead to seemingly purposeful behavior on longer time scales. As a paradigmaticexample, we study the simulation of an underactuated snake-like robot. By interacting withthe real physical system formed by the robotic hardware and the environment, the controllerachieves a sensitive and body-speciﬁc actuation of the robot. Self-Organization; Autonomous Robot Control; Neural Networks; Homeokine-sis.

title: What Is Stochastic Resonance? Definitions, Misconceptions, Debates, and Its Relevance to Biology
abstract: Stochastic resonance is said to be observed when increases in levels of unpredictable fluctuations—e.g., random noise—cause an increase in a metric of the quality of signal transmission or detection performance, rather than a decrease. This counterintuitive effect relies on system nonlinearities and on some parameter ranges being “suboptimal”. Stochastic resonance has been observed, quantified, and described in a plethora of physical and biological systems, including neurons. Being a topic of widespread multidisciplinary interest, the definition of stochastic resonance has evolved significantly over the last decade or so, leading to a number of debates, misunderstandings, and controversies. Perhaps the most important debate is whether the brain has evolved to utilize random noise in vivo, as part of the “neural code”. Surprisingly, this debate has been for the most part ignored by neuroscientists, despite much indirect evidence of a positive role for noise in the brain. We explore some of the reasons for this and argue why it would be more surprising if the brain did not exploit randomness provided by noise—via stochastic resonance or otherwise—than if it did. We also challenge neuroscientists and biologists, both computational and experimental, to embrace a very broad definition of stochastic resonance in terms of signal-processing “noise benefits”, and to devise experiments aimed at verifying that random variability can play a functional role in the brain, nervous system, or other areas of biology.

title: Stochastic Resonance:. from Climate to Biology
abstract: In this paper I will review some basic aspects of the mechanism of stochastic resonance. Stochastic resonance was first introduced as a possible mechanism to explain long term climatic variation. Since then, there have been many applications of stochastic resonance in physical and biological systems. I will show that in complex system, stochastic resonance can substantially change as a function of the ``system complexity''. Also, I will briefly mention how to apply stochastic resonance for the case of Brownian motors.

title: Stochastic resonance in biology. How noise can enhance detection of weak signals and help improve biological information processing.
abstract: Noise is usually thought of as the enemy of order rather than as a constructive influence. In nonlinear systems that possess some sort of threshold, random noise plays a beneficial role in enhancing the detection of weak information-carrying signals. This phenomenon, termed stochastic resonance, does find useful applications in physical, biological, and biomedical contexts. Certain biological systems may even use this effect for optimizing function and behavior.

title: Brownian motion, fluctuation and life
abstract: The measurements of dynamic behaviors of biomolecules in relation to their functions have been allowed using single molecule measurements. Thermal Brownian motion causes random step motion of motor proteins and structural fluctuation of protein molecules between multiple states. In hierarchic structure of life, the fluctuation is modulated. Random fluctuation is biased to directional motion and reactions as a result of interaction of proteins. The fluctuation of kinetic state of signaling proteins results in polarization and localization of cells. A recognition process in brain is also explained by the equation analogous to biochemical reaction at the molecular level. Thus dynamic processes originated from thermal motion may play an important role in activation processes in life.

title: Order in Spontaneous Behavior
abstract: Brains are usually described as input/output systems: they transform sensory input into motor output. However, the motor output of brains (behavior) is notoriously variable, even under identical sensory conditions. The question of whether this behavioral variability merely reflects residual deviations due to extrinsic random noise in such otherwise deterministic systems or an intrinsic, adaptive indeterminacy trait is central for the basic understanding of brain function. Instead of random noise, we find a fractal order (resembling Lévy flights) in the temporal structure of spontaneous flight maneuvers in tethered Drosophila fruit flies. Lévy-like probabilistic behavior patterns are evolutionarily conserved, suggesting a general neural mechanism underlying spontaneous behavior. Drosophila can produce these patterns endogenously, without any external cues. The fly's behavior is controlled by brain circuits which operate as a nonlinear system with unstable dynamics far from equilibrium. These findings suggest that both general models of brain function and autonomous agents ought to include biologically relevant nonlinear, endogenous behavior-initiating mechanisms if they strive to realistically simulate biological brains or out-compete other agents.

title: Stochastic resonance and the evolution of Daphnia foraging strategy.
abstract: Search strategies are currently of great interest, with reports on foraging ranging from albatrosses and spider monkeys to microzooplankton. Here, we investigate the role of noise in optimizing search strategies. We focus on the zooplankton Daphnia, which move in successive sequences consisting of a hop, a pause and a turn through an angle. Recent experiments have shown that their turning angle distributions (TADs) and underlying noise intensities are similar across species and age groups, suggesting an evolutionary origin of this internal noise. We explore this hypothesis further with a digital simulation (EVO) based solely on the three central Darwinian themes: inheritability, variability and survivability. Separate simulations utilizing stochastic resonance (SR) indicate that foraging success, and hence fitness, is maximized at an optimum TAD noise intensity, which is represented by the distribution's characteristic width, sigma. In both the EVO and SR simulations, foraging success is the criterion, and the results are the predicted characteristic widths of the TADs that maximize success. Our results are twofold: (1) the evolving characteristic widths achieve stasis after many generations; (2) as a hop length parameter is changed, variations in the evolved widths generated by EVO parallel those predicted by SR. These findings provide support for the hypotheses that (1) sigma is an evolved quantity and that (2) SR plays a role in evolution.

title: Lévy Walks Evolve Through Interaction Between Movement and Environmental Complexity
abstract: Animals’ movements may not only respond to the environment, but may also shape it, and thus affect fitness. Ecological theory predicts that animal movement is shaped by its efficiency of resource acquisition. Focusing solely on efficiency, however, ignores the fact that animal activity can affect resource availability and distribution. Here, we show that feedback between individual behavior and environmental complexity can explain movement strategies in mussels. Specifically, experiments show that mussels use a Lévy walk during the formation of spatially patterned beds, and models reveal that this Lévy movement accelerates pattern formation. The emergent patterning in mussel beds, in turn, improves individual fitness. These results suggest that Lévy walks evolved as a result of the selective advantage conferred by autonomously generated, emergent spatial patterns in mussel beds. Our results emphasize that an interaction between individual selection and habitat complexity shapes animal movement in natural systems.

title: From Lévy to Brownian: A Computational Model Based on Biological Fluctuation
abstract: Theoretical studies predict that Lévy walks maximizes the chance of encountering randomly distributed targets with a low density, but Brownian walks is favorable inside a patch of targets with high density. Recently, experimental data reports that some animals indeed show a Lévy and Brownian walk movement patterns when forage for foods in areas with low and high density. This paper presents a simple, Gaussian-noise utilizing computational model that can realize such behavior.  We extend Lévy walks model of one of the simplest creature, Escherichia coli, based on biological fluctuation framework. We build a simulation of a simple, generic animal to observe whether Lévy or Brownian walks will be performed properly depends on the target density, and investigate the emergent behavior in a commonly faced patchy environment where the density alternates.  Based on the model, animal behavior of choosing Lévy or Brownian walk movement patterns based on the target density is able to be generated, without changing the essence of the stochastic property in Escherichia coli physiological mechanism as explained by related researches. The emergent behavior and its benefits in a patchy environment are also discussed. The model provides a framework for further investigation on the role of internal noise in realizing adaptive and efficient foraging behavior.

title: Bacteria-inspired underactuated mobile robot based on a biological fluctuation
abstract: Interpretation of the living organism’s ability to utilize noise from engineering point of view may be beneficial for realizing a simple, yet adaptive, robotic system. This paper presents a bacteria-inspired mobile robot, with a 1-DOF motor and a single sensor. Even with such limitations, the underactuated mobile robot is able to navigate toward a gradient-inducing goal in a two-dimensional space by properly utilizing environmental noise that directly affects its motion. The way the robot utilizes this external noise to control its navigation behavior is interpreted based on biological fluctuation: a simple perspective commonly used to describe how living beings utilize internally generated biological noises.

title: 'Yuragi'-Based Adaptive Mobile Robot Search With and Without Gradient Sensing: From Bacterial Chemotaxis to a Levy Walk
abstract: Many biologically inspired approaches have been investigated in relation to research on mobile robot(s) that can effectively locate targets that induce gradient information. Here, we concentrate on realizing an adaptive searching behavior in a mobile robot that is simple, yet effective with and without sensing the gradient information. We are interested in two searching behaviors found in biological creatures: bacterial chemotaxis (probably the simplest yet effective gradient sources searching behavior found in living creatures) and Levy walk (a specialized random walk with fractal movement trajectories that optimize random search for sparsely and randomly distributed target(s)). Our approach is to propose a unifying framework based on 'yuragi' (biological fluctuation) to implement and combine the two searching behaviors.

title: Self-Organization, Embodiment, and Biologically Inspired Robotics
abstract: Robotics researchers increasingly agree that ideas from biology and self-organization can strongly benefit the design of autonomous robots. Biological organisms have evolved to perform and survive in a world characterized by rapid changes, high uncertainty, indefinite richness, and limited availability of information. Industrial robots, in contrast, operate in highly controlled environments with no or very little uncertainty. Although many challenges remain, concepts from biologically inspired (bio-inspired) robotics will eventually enable researchers to engineer machines for the real world that possess at least some of the desirable properties of biological organisms, such as adaptivity, robustness, versatility, and agility.

title: The implications of embodiment for behavior and cognition: animal and robotic case studies
abstract: In this paper, we will argue that if we want to understand the function of the brain (or the control in the case of robots), we must understand how the brain is embedded into the physical system, and how the organism interacts with the real world. While embodiment has often been used in its trivial meaning, i.e. 'intelligence requires a body', the concept has deeper and more important implications, concerned with the relation between physical and information (neural, control) processes. A number of case studies are presented to illustrate the concept. These involve animals and robots and are concentrated around locomotion, grasping, and visual perception. A theoretical scheme that can be used to embed the diverse case studies will be presented. Finally, we will establish a link between the low-level sensory-motor processes and cognition. We will present an embodied view on categorization, and propose the concepts of 'body schema' and 'forward models' as a natural extension of the embodied approach toward first representations.

title: Morphological Computation and Morphological Control: Steps Toward a Formal Theory and Applications
abstract: Morphological computation can be loosely defined as the exploitation of the shape, material properties, and physical dynamics of a physical system to improve the efficiency of a computation. Morphological control is the application of morphological computing to a control task. In its theoretical part, this article sharpens and extends these definitions by suggesting new formalized definitions and identifying areas in which the definitions we propose are still inadequate. We go on to describe three ongoing studies, in which we are applying morphological control to problems in medicine and in chemistry. The first involves an inflatable support system for patients with impaired movement, and is based on macroscopic physics and concepts already tested in robotics. The two other case studies (self-assembly of chemical microreactors; models of induced cell repair in radio-oncology) describe processes and devices on the micrometer scale, in which the emergent dynamics of the underlying physical system (e.g., phase transitions) are dominated by stochastic processes such as diffusion.

title: Sensing through body dynamics
abstract: It has been shown that sensory morphology and sensory–motor coordination enhance the capabilities of sensing in robotic systems. The tasks of categorization and category learning, for example, can be significantly simplified by exploiting the morphological constraints, sensory–motor couplings and the interaction with the environment. This paper argues that, in the context of sensory–motor control, it is essential to consider body dynamics derived from morphological properties and the interaction with the environment in order to gain additional insight into the underlying mechanisms of sensory–motor coordination, and more generally the nature of perception. A locomotion model of a four-legged robot is used for the case studies in both simulation and real world. The locomotion model demonstrates how attractor states derived from body dynamics influence the sensory information, which can then be used for the recognition of stable behavioral patterns and of physical properties in the environment. A comprehensive analysis of behavior and sensory information leads to a deeper understanding of the underlying mechanisms by which body dynamics can be exploited for category learning of autonomous robotic systems.

title: An Energy-Efficient Hopping Robot Based on Free Vibration of a Curved Beam
abstract: This paper explores a design strategy of hopping robots, which makes use of free vibration of an elastic curved beam. In this strategy, the leg structure consists of a specifically shaped elastic curved beam and a small rotating mass that induces free vibration of the entire robot body. Although we expect to improve energy efficiency of locomotion by exploiting the mechanical dynamics, it is not trivial to take advantage of the coupled dynamics between actuation and mechanical structures for the purpose of locomotion. From this perspective, this paper explains the basic design principles through modeling, simulation, and experiments of a minimalistic hopping robot platform. More specifically, we show how to design elastic curved beams for stable hopping locomotion and the control method by using unconventional actuation. In addition, we also analyze the proposed design strategy in terms of energy efficiency and discuss how it can be applied to the other forms of legged robot locomotion.

title: Minimalistic Models of an Energy-Efficient Vertical-Hopping Robot
abstract: The use of free vibration in elastic structure can lead to energy-efficient robot locomotion, since it significantly reduces the energy expenditure if properly designed and controlled. However, it is not well understood how to harness the dynamics of free vibration for the robot locomotion, because of the complex dynamics originated in discrete events and energy dissipation during locomotion. From this perspective, the goals of this paper are to propose a design strategy of hopping robot based on elastic curved beams and actuated rotating masses and to identify the minimalistic model that can characterize the basic principle of robot locomotion. Since the robot mainly exhibits vertical hopping, three 1-D models are examined that contain different configurations of simple spring-damper-mass components. The real-world and simulation experiments show that one of the models best characterizes the robot hopping, through analyzing the basic kinematics and negative works in actuation. Based on this model, the self-stability of hopping motion under disturbances is investigated, and design and control parameters are analyzed for the energy-efficient hopping. In addition, further analyses show that this robot can achieve the energy-efficient hopping with the variation in payload, and the source of energy dissipation of the robot hopping is investigated.

title: Estimating contact dynamics
abstract: Motion and interaction with the environment are fundamentally intertwined. Few people-tracking algorithms exploit such interactions, and those that do assume that surface geometry and dynamics are given. This paper concerns the converse problem, i.e., the inference of contact and environment properties from motion. For 3D human motion, with a 12-segment articulated body model, we show how one can estimate the forces acting on the body in terms of internal forces (joint torques), gravity, and the parameters of a contact model (e.g., the geometry and dynamics of a spring-based model). This is tested on motion capture data and video-based tracking data, with walking, jogging, cartwheels, and jumping.

title: Learning from Observation and from Practice Using Behavioral Primitives
abstract: We describe a memory-based approach to learning how to select and provide sub-goals for behavioral primitives, given an existing library of primitives. We demonstrate both learning from observation and learning from practice on a marble maze task, Labyrinth.

title: Deriving action and behavior primitives from human motion data
abstract: We address the problem of creating basis behaviors for modularizing humanoid robot control and representing human activity. These behaviors, called perceptual-motor primitives, serve as a substrate for linking a system's perception of human activities and the ability to perform those activities. We present a data-driven method for deriving perceptual-motor action and behavior primitives from human motion capture data. In order to find these primitives, we employ a spatio-temporal non-linear dimension reduction technique on a set of motion segments. From this transformation, motions representing the same action can be clustered and generalized. Further dimension reduction iterations are applied to derive extended-duration behaviors.

title: Active sampling and decision making in Drosophila chemotaxis
abstract: The ability to respond to chemical stimuli is fundamental to the survival of motile organisms, but the strategies underlying odour tracking remain poorly understood. Here we show that chemotaxis in Drosophila melanogaster larvae is an active sampling process analogous to sniffing in vertebrates. Combining computer-vision algorithms with reconstructed olfactory environments, we establish that larvae orient in odour gradients through a sequential organization of stereotypical behaviours, including runs, stops, lateral head casts and directed turns. Negative gradients, integrated during runs, control the timing of turns. Positive gradients detected through high-amplitude head casts determine the direction of individual turns. By genetically manipulating the peripheral olfactory circuit, we examine how orientation adapts to losses and gains of function in olfactory input. Our findings suggest that larval chemotaxis represents an intermediate navigation strategy between the biased random walks of Escherichia Coli and the stereo-olfaction observed in rats and humans.

title: Motile Behavior of Bacteria
abstract: Escherichia coli is a single‐celled organism that lives in your gut. It is equipped with a set of rotary motors only 45 nm in diameter. Each motor drives a long, thin, helical filament that extends several cell body lengths out into the external medium. The assemblage of motor and filament is called a flagellum. The concerted motion of several flagella enables a cell to swim. A cell can move toward regions that it deems more favorable by measuring changes in the concentrations of certain chemicals in its environment (mostly nutrients), deciding whether life is getting better or worse, and then modulating the direction of rotation of its flagella. Thus, in addition to rotary engines and propellers, E. coli's standard accessories include particle counters, rate meters, and gear boxes. This microorganism is a nanotechnologist's dream. I will discuss the features that make it so, from the perspectives of several scientific disciplines: anatomy, genetics, chemistry, and physics.

title: The Fundamental Role of Pirouettes in Caenorhabditis elegans Chemotaxis
abstract: To investigate the behavioral mechanism of chemotaxis inCaenorhabditis elegans, we recorded the instantaneous position, speed, and turning rate of single worms as a function of time during chemotaxis in gradients of the attractants ammonium chloride or biotin. Analysis of turning rate showed that each worm track could be divided into periods of smooth swimming (runs) and periods of frequent turning (pirouettes). The initiation of pirouettes was correlated with the rate of change of concentration (dC/dt) but not with absolute concentration. Pirouettes were most likely to occur when a worm was heading down the gradient (dC/dt < 0) and least likely to occur when a worm was heading up the gradient (dC/dt > 0). Further analysis revealed that the average direction of movement after a pirouette was up the gradient. These observations suggest that chemotaxis is produced by a series of pirouettes that reorient the animal to the gradient. We tested this idea by imposing the correlation between pirouettes and dC/dt on a stochastic point model of worm motion. The model exhibited chemotaxis behavior in a radial gradient and also in a novel planar gradient. Thus, the pirouette model ofC. elegans chemotaxis is sufficient and general.

title: Behaviour of dogs during olfactory tracking.
abstract: The ability to detect the direction of a track is of vital importance to animals of prey and is retained in many modern breeds of dogs. To study this ability, four trained German shepherd tracking dogs, equipped with head microphones to transmit sniffing activity, were video-monitored after being brought at right angles to a track where the position of each footprint was known. Three phases could be recognized in the dogs' behaviour: (1) an initial searching phase, during which the dog tried to find the track, (2) a deciding phase, during which it tried to determine the direction of the track and (3) a tracking phase, in which it followed the track. During ten tests on 20-min-old tracks on grass, and ten tests on 3-min-old tracks on concrete, the dogs always followed the track in the correct direction (i.e. in the direction the track was leading). During the deciding phase the dogs moved at half the speed and their periods of sniffing lasted three times as long as during the other two phases. The deciding phase lasted 3-5 s, while the dogs sniffed at 2-5 footprints. The dogs' ability to determine track direction in this time must rely on accurate methods of sampling air and a remarkable sensitivity for certain substances.

title: Neural bases of goal-directed locomotion in vertebrates—An overview
abstract: The different neural control systems involved in goal-directed vertebrate locomotion are reviewed. They include not only the central pattern generator networks in the spinal cord that generate the basic locomotor synergy and the brainstem command systems for locomotion but also the control systems for steering and control of body orientation (posture) and finally the neural structures responsible for determining which motor programs should be turned on in a given instant. The role of the basal ganglia is considered in this context. The review summarizes the available information from a general vertebrate perspective, but specific examples are often derived from the lamprey, which provides the most detailed information when considering cellular and network perspectives.

title: Toward open-ended evolutionary robotics: evolving elementary robotic units able to self-assemble and self-reproduce
abstract: In this paper, we discuss the limitations of current evolutionary robotics models and we propose a new framework that might solve some of these problems and lead to an open-ended evolutionary process in hardware. More specifically, the paper describes a novel approach where the usual concepts of population, generations and fitness are made implicit in the system. Individuals co-evolve embedded in their environment. Exploiting the self-assembling capabilities of the (simulated) robots, the genotype of a successful individual can spread in the population. In this way, interesting behaviours emerge spontaneously, resulting in chasing and evading other individuals, collective obstacle avoidance and co-ordinated motion of self-assembled structures.

title: Reinforcement Learning in Continuous Time and Space
abstract: This article presents a reinforcement learning framework for continuous-time dynamical systems without a priori discretization of time, state, and action. Basedonthe Hamilton-Jacobi-Bellman (HJB) equation for infinite-horizon, discounted reward problems, we derive algorithms for estimating value functions and improving policies with the use of function approximators. The process of value function estimation is formulated as the minimization of a continuous-time form of the temporal difference (TD) error. Update methods based on backward Euler approximation and exponential eligibility traces are derived, and their correspondences with the conventional residual gradient, TD (0), and TD () algorithms are shown. For policy improvement, two methodsa continuous actor-critic method and a value-gradient-based greedy policyare formulated. As a special case of the latter, a nonlinear feedback control law using the value gradient and the model of the input gain is derived. The advantage updating, a model-free algorithm derived previously, is also formulated in the HJB-based framework. The performance of the proposed algorithms is first tested in a nonlinear control task of swinging a pendulum up with limited torque. It is shown in the simulations that (1) the task is accomplished by the continuous actor-critic method in a number of trials several times fewer than by the conventional discrete actor-critic method; (2) among the continuous policy update methods, the value-gradient-based policy with a known or learned dynamic model performs several times better than the actor-critic method; and (3) a value function update using exponential eligibility traces is more efficient and stable than that based on Euler approximation. The algorithms are then tested in a higher-dimensional task: cart-pole swing-up. This task is accomplished in several hundred trials using the value-gradient-based policy with a learned dynamic model.

title: Developmental robotics: a survey
abstract: Developmental robotics is an emerging field located at the intersection of robotics, cognitive science and developmental sciences. This paper elucidates the main reasons and key motivations behind the convergence of fields with seemingly disparate interests, and shows why developmental robotics might prove to be beneficial for all fields involved. The methodology advocated is synthetic and two-pronged: on the one hand, it employs robots to instantiate models originating from developmental sciences; on the other hand, it aims to develop better robotic systems by exploiting insights gained from studies on ontogenetic development. This paper gives a survey of the relevant research issues and points to some future research directions.