Following Ancestral Footsteps: Co-Designing Agent Morphology and Behaviour with Self-Imitation Learning

Authors: sergio  hernández-gutiérrez
ville  kyrki
Venue: roboticsfoundation_org_RSS_2024_Workshop_EARL
Published: 2024-05-31 20:35:47

Abstract: In this paper we consider the problem of co-adapting the body and behaviour of agents, a long-standing research problem in the community of evolutionary robotics. 
Previous work has largely focused on the development of methods exploiting massive parallelization of agent evaluations with large population sizes, a paradigm which is not applicable to the real world. 
More recent data-efficient approaches utilizing reinforcement learning can suffer from distributional shifts to transition dynamics as well as to states and action spaces when experiencing new body morphologies. 
In this work, we propose a new co-adaptation method combining reinforcement learning and State-Aligned Self-Imitation Learning. 
We show that the integration of a self-imitation signal improves data-efficiency, behavioural recovery for unseen designs and performance convergence.

Main paper: 
 i. introduction: 
 Finding an optimal combination of body and morphology of agents has been a long-standing research problem, finding its roots in the community of evolutionary robotics [17, 7, 8] . Originally, research in this area largely focused on the use and development of evolutionary or genetic algorithms adapting body and control parameters at the same time [17, 27, 2, 5, 15] . More recent research [13, 19] has studied the benefits of considering the different time-scales on which co-adaptation of body and behaviour occurs in the real world: adaptation of the body is costly and time-consuming, as it involves growing appendices, organs and tissue in nature; likewise in robotics, where even fast manufacturing methods like 3D-printing require a considerable amount of work-hours and material.Recent years have brought forward several works considering the use of reinforcement learning (RL) methods for the problem of co-adapting robots [6, 21, 25, 19] , usually with fast behavioural learning and slower morphology adaptation. This allowed to develop methods capable of being deployed on real-world robotics due to their data-efficiency. However, dataefficient co-adaptation processes can suffer considerably from the problem of distributional shift inherent to the co-adaptation problem setting. Every new agent morphology the algorithms experiences brings with it changes to the transition distribution, as well as to the semantics of state and action spaces.We propose a novel co-adaptation methodology tackling the aforementioned problems by combining reward-driven RL and self-imitation learning (SIL) utilizing Wasserstein distances for data-efficient adaptation of body and behaviour. Our approach not only forces the RL algorithm to adapt body and behaviour for maximizing an objective function, but also to encourages the imitation of the agent's own previous behaviours to increase learning stability and accelerate the learning progress.In this paper, we present the following contributions: (C1) An extension of State-Alignment Imitation Learning (SAIL) [18] for mismatching morphologies to State-Aligned Self-Imitation Learning for the problem of co-adapting the morphology and behaviour of agents. (C2) A novel co-adaptation method, Co-Adaptation with Self-Imitation Learning (CoSIL), utilizing State-Aligned Self-Imitation Learning to optimize an agent's morphology and behaviour data-efficiently on fewer design iterations. (C3) We empirically demonstrate the benefits and limitations of a SIL signal by evaluating CoSIL versus a non-self-imitating baseline in a range of locomotion tasks.a) Multi-Body Reinforcement Learning: We consider an extension to the classic Markov Decision Process (MDP) suitable for modelling the fact that both behaviour and morphological parameters are adapted. The Multi-Body MDP (MB-MDP) consists of (S, A, Ξ, r, p(s t+1 |s t , a t , ξ), p(s 0 )) with state space S ∈ R s and action space A ∈ Rwa. Notably, in a MB-MDP the set Ξ models the morphological parameter space, containing individual instances of agent morphologies ξ ∈ Ξ. Throughout this paper, we will without a loss of generality consider Ξ ∈ R d for d continuous design parameters. As changes to the agent morphology impact its dynamics, the transition function p(s t+1 |s t , a t , ξ) depends on the current morphology parameter ξ. The reward function r(s t , a t , ξ) may also implicitly depend on ξ via the transition function, or explicitly if the manufacturing costs are taken into account, for example. The objective is to find a policy π θ (s t , ξ) = a t which maximizes the finite-horizon expected discounted rewardEQUATIONgiven an embodiment ξ, the policy π, and discount factor γ ∈ (0, 1). b) Co-Adaptation of Agent Body and Behaviour: The previous formalism allows us to formulate the joint optimization of behaviour and morphology of agents asEQUATIONin other words, we are interested in finding both the optimal morphology ξ * and optimal policy π * given a reward function r(s t , a t , ξ). If we consider the semantics of the parameters and the optimization time-scales (i.e., policy learning can be done faster than morphology adaptation), this problem can be considered a bi-level optimization problem. Given the current morphology of the agent in the inner optimization problem, we can solve the RL problem using Eq. ( 1). In the outer optimization problem, given performances R(ξ, π) of past morphology-policy pairs (ξ i , π i ), we can again utilize optimization methods or reinforcement learning to find new candidate morphologies ξ to evaluate.In this section, we will first introduce the individual components of Co-Adaptation with Self-Imitation Learning (CoSIL) using State-Aligned Imitation Learning (SAIL) [18] . We will end the section with a description of the main algorithm.Assume a MB-MDP (S, A, Ξ, r, p(s t+1 |s t , a t , ξ), p(s 0 )), as given in Section II-0a. Naturally, a co-adaptation process will produce a sequence of morphology-policy tuples {(ξ 0 , π 0 ), (ξ 1 , π 2 ), (ξ 3 , π 3 ), • • • }. Given two morphologypolicy pairs (ξ i , π i ) and (ξ j , π j ), we can formulate the trajectory distributionsq(τ i ) = p(s 0 |ξ i ) T -1 t=0 p(s t+1 |s t , a t , ξ i )π i (a t |s t , ξ i ) (3) and p(τ j |π j ) = p(s 0 |ξ j ) T -1 t=0 p(s t+1 |s t , a t , ξ j )π j (a t |s t , ξ j ). (4)We will now assume that the pair (ξ i , π i ) represents our expert. If we are now currently training on morphology ξ j , where j > i, then we can force the policy π j to imitate the previous agent by optimizingEQUATIONfor a divergence measure D expressing the distance between these two probability distributions. Importantly, we consider here that ξ j is fixed and not optimized, otherwise (ξ i , π i ) is a trivial solution to this problem. While different choices exist for this divergence measure, we will follow state alignmentbased imitation learning and use state-distribution matching via generative adversarial learning.B. Feature-State-Distribution Self-Imitation LearningAs previously described, a core problem for imitation learning (IL) between agents with different morphologies is that the semantic of state and action spaces can shift considerably. Hence, using the original state and action spaces is not suitable to use in the IL setting. Therefore, we assume in the following a function ϕ : S → S F which maps the state of the agent to a shared feature space S F modelled with motion capture markers placed on the robot's body.In our proposed SIL approach for co-adaptation, we are matching the state distributions between previous expert behaviour and the current agent, a technique used successfully in prior work [9, 22] . Similarly, we use the marginal featurespace state distributions for the expert trajectories from past morphologiesEQUATION) and for the current agent morphologyEQUATIONwith 1 being a Kronecker delta function, returning the value 1 iff ϕ(s t ) = ϕ(s) holds true and 0 otherwise. Using these state distributions we can now reformulate Eq. ( 5) withEQUATIONwhere we can use divergences such as Kullback-Leibler's, the Wasserstein distance, or the Jensen-Shannon divergence. Eq. ( 8) will be our main objective for enabling SIL across morphologies.CoSIL makes use of two reward functions: r RL for the environment reward we aim to maximize as the main objective and r IL for the SIL reward, given a demonstration dataset τ E . We implement r IL was State-Aligned Imitation Learning (SAIL) using the Wasserstein distance [18] with reward functionEQUATION)where ρ is a learned discriminator function (i.e., a neural network) modelling the Kantorovich's potential, assigning higher values to states similar to those seen in the expert dataset τ E .CoSIL makes use of Soft Actor Critic (SAC) [12] as the reinforcement learning backbone of the method with a slight adaptation to the learning rule for policy updates. As we have two reward functions, we propose to adapt SAC to learn two Q-functions withEQUATIONEQUATION)To avoid imbalances during training, we normalize both rewards using z-score normalization. This leads to the following loss function for the policy π with two Q-networks:EQUATIONin which we optimize the policy for both the objective-driven Q-function Q RL and the SIL Q-function Q IL , weighted by ω.Both critics use the double-Q trick proposed by [14] .Similar to the behaviour learning process, we extend the morphology optimization objective to incorporate SIL. Accordingly, we supplement the objective introduced in [19] by adding the Q-functionEQUATIONwhere ω opt is used to weigh the importance of the SIL reward versus the environment reward function. While in principle any optimization method can be used, we found the gradientfree Particle Swarm Optimization (PSO) optimizer [16] to be the most efficient. Since the distribution p(s 0 |ξ) is generally unknown, we replace it in practice with s 0 ∼ R 0 , where R 0 is a replay buffer containing only starting states.In order to understand the impact of a SIL signal in the coadaptation setting, we seek to empirically answer the following research questions: (RQ1) Does SIL help to better co-adapt the behaviour and morphology of agents? (RQ2) Can SIL help to counter distributional shifts and recover faster after a morphology change? (RQ3) What are the limitations of SIL? Is it always beneficial?In our experiments, we used variants of the OpenAI Gym library [4] environments Humanoid and HalfCheetah adapted to the co-adaptation setting, as previously proposed [22] , and also evaluate on a new adaptable variant of the Walker environment. These environments are implemented using the MuJoCo engine [26] . Experiments are conducted on a computing computer with Nvidia RTX A4500. We employed 32GB of RAM and were constrained by 72 hours of real time usage per experiment. The results are averaged across four distinct seeds. In all experiments, we compare CoSIL to a baseline which does not include a SIL signal neither in the critic's update function nor in the design evolution objective function. In this way, we can study the effects of a SIL signal in isolation from other performance factors (RQ1). It can be seen that the proposed method (CoSIL) shows better performance than using solely the environmental reward (Co-Adapt), with comparable perfomance when trained for 300 and 1000 episodes. The goal of the agent in the morphology-adaptable Humanoid task is to learn a policy allowing stable but fast forward locomotion, as proposed in [22] . To achieve this, the following objective function r RL is used:EQUATIONwhere c t is the position of the center of mass of the robot and standard deviation of the environmental reward, averaged over four seeds, and with morphologies sorted by their performance (worse performing morphologies left, better performing morphologies right). While using self-imitation for co-adaptation (CoSIL) shows considerably better performance in Walker (b), we found the performance gain in HalfCheetah (a) to be negligible. We hypothesise this is due to the simplicity of the HalfChetah task. at timestep t, ctrl t are the actuator activations at timestep t and cfrc ext t are the external forces acting on the body of the robot at timestep t. In this experiment, we compare CoSIL (red) using both r RL and r IL versus the standard Co-Adaptation baseline using only SAC and the objective reward r RL (blue). We adapt morphology and behaviour over 10,000 episodes, or 10 selected morphologies. For CoSIL, a weight-parameter of ω = ω opt = 0.2 was used, which was determined via hyperparameter optimization through grid-search. From the results in Fig. 1b , it can be seen that CoSIL outperforms standard Co-Adaptation without SIL after a handful of designs. Fig. 1a shows a comparison between CoSIL and Co-Adaptation when training on each morphology for only 300 episodes, which highlights the ability of CoSIL to find better performing morphology-policy combinations in a sample-efficient manner.Next, we investigate possible limitations of CoSIL over reward-driven co-adaptation (RQ3). For this, we evaluate CoSIL on the environments of varying difficulty, specifically the relatively simple HalfCheetah task and the harder Walker task. We selected two variations of the HalfCheetah locomotion task in MuJoco with the reward functionEQUATIONwhere a t,i is the i-th action taken at timestep t and c t is the position of the center-of-mass at time step t, and ∆t the time difference. For Walker we use the reward functionEQUATION)where θ is the Euler angle of Walker's orientation in radians.In both tasks, we optimize the leg-segment lengths of the agent, thus resulting in 6 morphology parameters for HalfCheetah and 4 for Walker. We use again ω = ω opt = 0.2 for both HalfCheetah and Walker. As we can see in Fig. 2b/d , CoSIL outperforms standard Co-Adaptation using only r RL in the Walker task, as also observed in the Humanoid task. However, we note that CoSIL and Co-Adaptation show a very similar performance during the first designs. The performance gap between both approaches closes further if we compare them on the simpler HalfCheetah task. Fig. 2 (a) clearly show that using a SIL reward r IL is not helpful in this case. Additionally, this could indicate that the distributional shift of state and action spaces in low-complexity tasks is lower, as one would expect.From this, we can conclude that, especially on simpler and less complex co-adaptation tasks, the additional use of SIL should not be preferred, especially given that CoSIL requires in total more compute due to the deep imitation learning methods used.Evolutionary Robotics: Designing robot hardware with evolutionary principles has been a long-standing research effort. Seminal work by [17] explored using genetic algorithms to co-adapt a simple controller. Earlier works by [24] used competition as a reward in a genetic algorithm to adapt the bodies of two robots in a fighting task. Approaches for evolutionary robotics have been successfully applied in simulation [3] , although recent works have identified that applying them in real world scenarios remains an open challenge [8] . Recent work has focused primarily on the fast changeability of robotic platforms as means to allow real world evolution of robots [20, 13, 1] , although this constrains the range of possible robot designs considerably.Co-Adaptation with Reinforcement Learning: Recent works on co-adaptation have sought to improve data-efficiency and feasibility by using a RL method as its main component. Seminal work by [11] employs REINFORCE to jointly coadapt the body and behaviour of agents [28] . [23] extended this approach by proposing a deep RL co-adaptation algorithm. Increased data-efficiency was achieved by [19] with the introduction of an off-policy deep RL method using the Qvalue function for design evaluations. Another recent work [10] employed deep RL with mass-parallelization of agent populations in simulation.We presented a new co-adaptation method named Co-Adaptation with Self-Imitation Learning (CoSIL) which introduces the idea of using a SIL reward within a reward-driven co-adaptation framework using deep reinforcement learning. To achieve this, we used State-Aligned Imitation Learning (SAIL) [18] , introduced a method to select and match expert data from previously seen morphology-policy combinations, and employed separate Q-value functions for the objective and imitation rewards to increase data-efficiency when optimizing the morphology parameters. In experiments on morphologyadaptable agents in simulation, we showed that by imitating previously seen behaviour we can combat the distributional shift in dynamics, action and state spaces, as well as recover faster when switching to a newly selected agent morphology. However, CoSIL requires a larger amount of computational effort due to additional deep neural network training, which makes it not preferable for simple co-adaptation problems. Nevertheless, with the methodology proposed in this paper we make a further step towards the useful integration of imitation learning techniques into co-adaptation techniques. Several interesting avenues for future work are opened up by our work, such as the use of quality-diversity approaches for selection of self-demonstrations, or further investigations of using a SIL reward during design optimization. 
 
 Appendix:  



Reviews: review: title: title: Interesting method for co-adapting the morphology and policy of agents
main_review: main_review: Summary: This paper proposes and evaluates a new method, Co-SIL, for co-adapting the morphology and policy of agents by combining reinforcement learning and state-aligned self-imitation learning.

Strengths: 
-	The paper is aligned with the workshop topic very well
-	The proposed Co-SIL method is novel
-	The experiments are conducted with clear research questions
-	Co-SIL achieves higher performance than the baseline which utilizes only environmental reward
Weaknesses:
-	The contributions C1 and C2 appear to be the same. The authors need to distinguish them better 
-	The authors mention that Co-SIL is computationally more expensive, some numerical estimates would be helpful
-	In Fig. 1 and 2, the morphologies on x-axis are sorted by their performance, but this misses the information about the sequence in which they were selected
-	The authors should explicitly state which experiment answers the RQ2, just like they state it for RQ1 and RQ3.
-	In the Multi-body MDP, why aren’t the state and action spaces functions of the morphology?