Distributed Online Evolution for Swarm Robotics

Engineering multi-agent systems is difficult due to numerous constraints, such as noise, limited range of interaction with other agents, delayed feedback, and the distributed autonomy of the agents. One potential solution is to automate the design of multi-agent systems in simulation, using evolutionary algorithms (EAs) [1, 4]. In this paradigm, the EA evolves the behaviors of the agents (and their local interactions), such that the global task behavior emerges. A global observer monitors the collective, and provides a measure of performance to the individual agents. Agent behaviors that lead to desirable global behavior are hence rewarded, and the collective system is gradually evolved to provide optimal global performance [5]. There are several difficulties with this approach. First, a global observer may not exist. Second, some (but not all) agents may experience some form of reward for achieving task behavior, while others do not. Third, this reward may be delayed, or may be noisy. Fourth, the above paradigm works well in simulation (offline), but is not feasible for realworld online applications where unexpected events occur. Finally, the above paradigm may have difficulty evolving different individual behaviors for different agents (heterogeneity vs homogeneity). We propose a novel framework, called “Distributed Agent Evolution with Dynamic Adaptation to Local Unexpected Scenarios” (DAEDALUS), for engineering multi-agent systems that can be used either offline or online. We will explore how DAEDALUS can be used to achieve global aggregate behavior, by examining a case study, obstacle avoidance in robotic swarms. With the DAEDALUS paradigm, we assume that agents (whether software or hardware) move throughout some environment. As they move, they interact with other agents. These agents may be of the same species or of some other species [3]. Agents of different species have different roles in the environment. The goal is to evolve agent behaviors and interactions between agents, in a distributed fashion, Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Copyright 200X ACM X-XXXXX-XX-X/XX/XX . $5.00. such that the desired global behavior occurs. Let us further assume that each agent has some procedure to control its own actions, in response to environmental conditions and interactions with other agents.The precise implementation of these procedures is not relevant, thus they may be programs, rule sets, finite state machines, real-valued vectors, force laws, or any other procedural representation. Agents have a sense of self-worth, or “fitness”. Each robot of the swarm is an individual in a population that interacts with its neighbors. Each robot contains a slightly mutated copy of the optimized control procedure found with offline learning with an offline EA. This ensures that our robots are not completely homogeneous. We allowed this slight heterogeneity because when the environment changes, some mutations perform better than others. The robots that perform well in the environment will have higher fitness than the robots that perform poorly. When low fitness robots encounter high fitness robots, the low fitness robots ask for the high fitness robot’s rules. Hence, better performing robots share their knowledge with their poorer performing neighbors. When we apply DAEDALUS to obstacle avoidance, we focus on two aspects of our swarm: reducing obstacle-robot collisions and maintaining the cohesion of the swarm. Robots are penalized if they collide with obstacles and/or if they leave their neighbors behind. The second scenario arises when the robots are left behind in cul-de-sacs. This causes the cohesion of the formation to be reduced. In our results, we demonstrate how robots overcome these scenarios using DAEDALUS. In our offline environment, 40 robots were trained with 90 randomly distributed smaller obstacles and one goal. The offline EA with a population size of 100 with 100 generations takes at least a week in real time to produce an optimal control procedure. To measure the performance of the DAEDALUS approach, an experiment was carried out with 60 robots, 5 goals in the long corridor, and 90 obstacles that are three times larger than the obstacles used in offline environment (i.e. higher obstacle density) randomly distributed between each goal. The experiment was averaged over 50 runs of different robot, goal, and obstacle placements. Each robot is given equal initial fitness and “seeded” with a mutated copy of the optimized control procedure learned in offline mode. If a robot collides with an obstacle, it’s fitness is reduced. Whenever a robot encounters another robot with higher fitness, it takes the relevant parameters pertaining to the obstaclerobot interaction of the better performing robot. Our online DAEDALUS approach was able to learn obstacle avoidance in less than 30 minutes in real time. This is an impressive improvement over offline learning. 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0 1 2 3 4 5 R ob ot s C ol lid ed /R ob ot s S ur vi ve d Goal Number Percentage of Robots that Collide with Obstacles Figure 1: The ratio of colliding robots versus the number of surviving robots, for 60 robots moving through 5 goals with 90 obstacles in between each goal. All results are averaged over 50 independent runs. Figure 1 shows the ratio of the number of robots that collided with obstacles versus the number of robots that survived to reach the goals. The graph indicates that after only 2 goals, the percentage of robots that collide with obstacles has dropped from about 36% to well under 5%. Inspection of the obstacle-robot control procedures indicates that the repulsive component increased through the online process of mutation and the copying of superior control procedures (this was confirmed via inspection of the mutated control procedures). This first experiment did not attempt to alleviate the situation where robots are left behind; in fact, only roughly 43% of the original 60 robots reach the final goal (see Figure 2, lower line). This is caused by the large number of cul-desacs produced by the large obstacle density. Our second experiment attempts to alleviate this problem by focusing on the robot-robot interactions. Our assumption was that the control procedure needs to provide stronger cohesion, so that robots aren’t left behind. If robots are stuck behind in cul-de-sacs (i.e. they make no progress towards the goal) and they sense neighbors, they slightly mutate the robot-robot interaction parameters of their control procedures. In a situation in which they do not sense the presence of neighbors and do not progress towards the goal, they rapidly mutate their robot-goal interaction causing a “panic behavior”. These relatively large perturbations of the control procedure allow the robots to escape their motionless state. Figure 2 shows the results of this second experiment. In comparison with the first experiment (with survival rates of 43%), the survival rates have increased to 68%. As a control experiment, we ran our offline EA on this more difficult task. After five goals, the survival rate is about 78%. As can be seen, the DAEDALUS approach provides results only somewhat inferior to the offline approach, in real time, while the robots are in the environment. Although not shown in the graph, it is important to point 0 10 20 30 40 50 60 0 1 2 3 4 5 R ob ot s S ur vi ve d Goal Number Number of Robots that Survived at Each Stage Survival is Important (online) Collisions are Important (online) Survival in offline Figure 2: A comparison of (a) the number of robots that survive when rules are learned using offline learning, (b) the number of robots that survive when using online learning (where the focus is on reducing collisions), and (c) the number of robots that survive when using online learning (and the focus in on survivability). out that the collision rates were not affected in the second experiment. Hence, we believe that it is quite feasible to combine both aspects in the future. Collision avoidance can be improved via mutation of the obstacle-robot interaction, while survival can be improved via mutation of the robotrobot interaction and robot-goal interaction. Future work of this research will focus more on the issue of credit assignment. Current work in classifier systems uses mechanisms such as “bucket-brigade” or “profit sharing” to allocate rewards to individual “agents” appropriately [2]. However, these techniques rely on global blackboards and assume that all agents can potentially act with all others, through a bidding process. We intend to modify these approaches so that they are fully distributed, and appropriate for online systems.