# An Algorithm of Reinforcement Learning for Maneuvering Parameter Self-Tuning Applying in Satellite Cluster

Authors: Xiao Wang; Peng Shi; Changxuan Wen; Yushan Zhao
Journal: Mathematical Problems in Engineering
 (2020)

---

## Abstract

Satellite cluster is a type of artificial cluster, which is attracting wide attention at present. Although the traditional empirical parameter method (TEPM) has the potential to deal with the mission of satellite flocking, it is difficult to select the proper parameters. In order to improve the flight effect in the problem of satellite cluster, as well as to make the selection of flight parameters more reasonable, the traditional sensing zones are improved. A 3σ position error ellipsoid and an induction ellipsoid are applied for substituting the traditional repulsing zone and attracting zone, respectively. Besides, we propose an algorithm of reinforcement learning for parameter self-tuning (RLPST), which is based on the actor-critic framework, to automatically learn the suitable flight parameters. To obtain the parameters in the repulsing zone, orientating zone, and attracting zone of each member in the cluster, a three-channel learning framework is designed. The learning process makes the framework finally find the suitable parameters. Numerical experimental results have shown the superiorities compared to the traditional method, which include trajectory deviation and sensing rate or terminal matching rate, as well as the improvement of the flight paths under the learning framework.


---

## Body

## 1. Introduction

Satellite cluster is a new space architecture emerging after satellite constellation and satellite formation in recent years [1–3]. Different from the satellite formation, which is generally required to design geometric configuration and control desired formation, satellite cluster emphasizes more on the coordination and cooperation among the members in a system. Through the specific technology of satellite cluster, multiple spacecraft with the same or different functions can be connected into an organic whole by a self-organizing network. Thus, the system is expected to have the flexibility to realize one or more tasks [4, 5].At present, the research of satellite cluster can be generally divided into two types: one is a long-distance loose cluster, which mainly considers the drift and periodic configuration design under long-term condition, and the other is a short-distance cluster, which involves the techniques about cooperative control and collision avoidance. For the long-distance cluster, the boundedness of the system is mainly considered, which means that the influence of various perturbations on the boundary of the cluster will be analysed [6]. From the perspective of fuel optimization, the maneuver sequences will be solved for maintaining the loose flying of the cluster for a long time. Mazal and Gurfil developed a cluster flight control algorithm based on fuel-efficiency for distance-keeping of spacecraft cluster [7]. Based on the relative elements, Wang and Nakasuka applied nonlinear programming for solving the orbit design of fractionated spacecraft [8]. For a long-distance cluster, Dang et al. found the analytic distance bounds for the coplanar relative motion [9]. On the contrary, the techniques of multiagent control, such as graph theory and consistency algorithms, are being continuously studied for the problems of a short-distance cluster. In the field of formation control modelled by second-order dynamics, Ren et al. studied the consensus-based formation control in the absence of centralised leadership [10, 11]. Considering the fixed and switching topologies, Olfati-Saber and Murray solved the consensus problems for networks of dynamic agents [12]. For the leader-follower consensus problem, Song et al. proposed a pinning control algorithm based on graph theory to handle the condition without a strongly connected interaction graph [13]. Aiming at the multivehicle system of double-integrator dynamic, Qin et al. investigated the consensus strategies to deal with the time-varying reference velocity [14]. However, the complex information link and massive computing are always confusing the algorithms based on graph or consistency theory.Inspired by biological clusters, humans have constructed a variety of artificial clusters, such as robot clusters and unmanned aerial vehicle clusters. The traditional empirical parameter method (TEPM) has been proved to be effective in many multiagent fields. For describing the motion of flocking particles, Reynolds created a distributed behavioural model [15]. Vicsek et al. suggested a model where the particles were driven with constant velocity and the system was biologically motivated [16]. To reveal the relationship between the individual and the group based on behavioural transitions, Couzin et al. presented a self-organizing model of group formation [17]. Based on these classic models, the algorithms for realizing the specific missions of the cluster were proposed [18], and some attraction/repulsion functions, which are used for achieving swarm aggregation, were designed [19]. With the deep discussion about the interactions between the particles in a swarm, the rule-based control or behaviour-based control was gradually concerned to act in dynamic multiagent systems [20, 21]. Specifically, in the field of aeronautic and aerospace, the behaviour-based path-planning technique for configuring the cluster structures [22], avoiding collision [23], and dealing with needs of aviation swarm convoy [24] were studied, respectively. Due to the wide application prospect of behaviour control methods in biological clusters, more and more space agencies are expecting to introduce the concept of biological clusters into space systems. In this way, it will be possible to make the satellite cluster similar to a biological cluster for completing complex space tasks with simple and cheap spacecraft. Nevertheless, the selection of the behaviour parameters, which is generally decided through the experiences from the scholars, has not been very deeply discussed yet. In order to train the behaviour parameters, one can apply supervised learning if the prior experiences can be obtained. However, such experiences are usually hard to obtain. Therefore, it is a promising direction to find a way to optimize the parameters without the experience data.In recent years, reinforcement learning has been paid more and more attention in the field of intelligent clusters. Through interacting with the environment, agents in the cluster can optimize their maneuvering strategies under the model-free condition [25,26]. Therefore, the traditional maneuvering strategies, which are based on man-made rules, can be improved. Instead of using the fixed rules, Morihiro et al. proposed a self-organized framework based on reinforcement learning for the flocking agents to conduct group missions [27]. In cooperative multirobot systems, Gu and Yang applied fuzzy policies with policy gradient approach to solve leader-follower problems [28]. Under self-organizing principles derived from natural interactions, Chen et al. solved a swarm pursuit game through a multiagent reinforcement learning framework [29], and Hung and Givigi presented a Q-learning algorithm, which was applicable to a stochastic environment, for the flocking fixed-wing unmanned aerial vehicles [30]. Therefore, reinforcement learning is a promising method for dealing with the cluster problem of multiple agents. As the most energetic branch of present reinforcement learning, the actor-critic method is suitable for motion problem of continuous agent systems [31].In this paper, we propose an innovative self-organizing algorithm of reinforcement learning for parameter self-tuning (RLPST), which is based on the actor-critic framework for the path planning of short-distance satellite cluster. The proposed learning algorithm is composed of three channels which can automatically adjust the flight parameters of an agent in the zone of repulsing (ZOR), zone of orientation (ZOO), and zone of attraction (ZOA), respectively. Through iterative learning, the maneuvering strategies of the cluster can be optimized. In this way, the disadvantages under traditional control strategies based on man-made experience parameters for the cluster are broken, and it is expectable to apply the proposed algorithm in a variety of clustering tasks.The main contributions of this paper are as follows: (1) it is the first time to apply the actor-critic framework into the path planning of satellite cluster. Aiming at two kinds of classical space cluster scenarios, we introduce the reinforcement learning to deal with the relative distance between the members of short-distance satellite cluster, which fills the blank of self-parameter tuning based on the heuristic method in the field of a short-distance satellite cluster. Under the same three flocking principles of Reynolds, we have compared the results under the proposed RLPST and the ones under TEPM and proved the superiority of the proposed algorithm. (2) It is the first time to apply the actor-critic framework to optimize the flight parameters of ZOR, ZOO, and ZOA in a cluster through three channels, respectively, instead of directly optimizing the maneuver of the agent. In this way, it makes full use of the known model information. Besides, the learning difficulty when applying the reinforcement learning to the satellite cluster problem, which has large-scale continuous state and action space, can be reduced.The structure of this paper is as follows: Section2 presents the model of satellite cluster and related sensing areas; Section 3 discusses reinforcement learning algorithms for continuous systems; Section 4 applies reinforcement learning to the motion of satellite cluster; Section 5 simulates the proposed algorithm under two classic scenarios, respectively; and Section 6 discusses the simulation results. Finally, Section 7 draws the conclusions.


## 2. Problem Statement

It is supposed that the subject of the research is a satellite cluster withN members, with a virtual host point, O, which is near to the center of the cluster in space. Figure 1 draws the flocking satellites and the virtual host point. In Figure 2, it shows that each member in the cluster is an agent, which has the ability of induction, interacting with the environment. By inputting the current states and the maneuvering strategy, the member is able to obtain the reward for preparing the correction of the strategy.Figure 1
Flocking satellites and the virtual host point.Figure 2
The interaction with the environment of the flocking satellite member.The symbolθ is denoted as the true anomaly, a as the semimajor axis, and e as the eccentricity of the orbit of the virtual host satellite, and then the following equations are obtained [32]:(1)θ˙=μa31+ecosθ21−e23/2,θ¨=−μa32esinθ1+ecosθ31−e23.To facilitate the description of the problem, the following coordinate systems are established: (a) Earth centered inertialOxiyizi; (b) orbital coordinate system of the member satellite Oxoyozo; and (c) orbital coordinate system of the virtual host satellite Oxryrzr. In Oxryrzr, the position vector xi=xi,yi,ziT is denoted. Therefore, according to the two-body motion rule of spacecraft, ignoring the second-order small quantities, the dynamic equation of the ith member in the cluster can be expressed as follows [32]:(2)x¨i−2θ˙y˙i−θ¨yi−θ˙2xi=μxrf32+3xrf+fix,y¨i+2θ˙x˙i+θ¨xi−θ˙i2y=−μyrf3−1+3xrf+fiy,z¨i=−μzrf3−1+3xrf+fiz,where rf represents the distance between the mass point of the satellite to the origin and fijj=x,y,z represents the force in the corresponding channel.


### 2.1. Position Error Model of a Cluster Member

The position of a satellite is objective; however, it cannot be accurately known. In different types of missions, there will always be measurement, navigation, control, and other deviations and the influences of perturbation. These factors will cause the real orbit diverged from the nominal orbit of the spacecraft, resulting in trajectory deviation. Taking the Gaussian distribution as an example, the covariance matrix of the spacecraft state distribution is denoted asP, and then, it is obtained that the real state vector x is in a hyperellipsoid, which is centered on the nominal state vector x¯ [33, 34]. The sphere of the hyperellipsoid can be expressed as follows:(3)x−x¯TP−1x−x¯=l2.Meanwhile, the probability density function of the relative state error distribution is(4)pl=12πn∫0lexp−12r2frdr,where n represents the dimension of the state space and l represents the Markov distance constant. Specifically, when l=3, equation (3) represents the 3σ error ellipsoid.The above equation shows a six-dimensional ellipsoid. The matrix,A, is denoted as(5)A=P−1l2=−l2=ArrArvAvrAvv,where the matrix A is a real symmetric positive definite matrix. The symbol, R, is denoted as the position component of x; therefore, the position error ellipsoid of the spacecraft can be expressed as follows:(6)RTArrR=1.


### 2.2. Sensing Area Division of a Cluster Member

Traditionally, for dealing with problems of cluster, the sensing areas, which are generally known as zone of repulsion (ZOR), zone of orientation (ZOO), and zone of attraction (ZOA), are defined from the inside to the outside as the spherical regions [17]. Figure 3(a) shows these traditional ZOR, ZOO, and ZOA. Under such uniform sensing areas, it will be certainly convenient for describing the problem and designing the cluster control strategy. However, considering the location deviations of spacecraft cluster members and the capability of the attaching sensors, it is necessary to improve the way for dividing sensing areas. Here, we redefine the sensing areas of a satellite member in the cluster, which is illustrated in Figure 3(b).Figure 3
Sensing area division of member satellite in the cluster: (a) traditional sensing areas; (b) proposed sensing areas.
(a)
(b)According to Figures3(a) and 3(b), we see that, for each member satellite in the cluster, there exists three sensing areas. Compared with the traditional sensing areas, the redefined ones have replaced the ZOR and ZOA part with the 3σ error ellipsoid and the induction ellipsoid, respectively. The details of the redefined sensing areas are expressed below.


#### 2.2.1. ZOR

The3σ error ellipsoid area of each member is defined as the zone of repulsion. It is assumed that the position deviation obeys the Gaussian distribution. As a result, each member in the cluster has its own position error ellipsoid. If the ellipsoid of a specific member interacts with the one of other individuals, the collision between the two members may occur. Therefore, such an ellipsoid is the ZOR for making repulsive force to avoid the probable collision. The members in the ZOR will make repulsive force on the center member. In this way, it will avoid the individuals in the cluster getting too close from each other.


#### 2.2.2. ZOO

The orientation area is defined as a standard sphere with a specific radius. For a specific member, its ZOO is an ideal zone that neighbours, which are located in such an area, keep suitable distance with this specific member. This member will receive the orientation force from the neighbours in its ZOO, which makes the member tends to align its speed with its neighbours gradually. In this way, the flight process will be smooth.


#### 2.2.3. ZOA

The attracting area is defined as the induction ellipsoid. Traditionally, the attracting area is uniform. However, in the case of spacecraft cluster problem, due to the capability of the sensing elements, the sensing ability may be strong or weak in different directions. Therefore, we use an ellipsoid model to nearly describe the induction area of the member in the cluster.


### 2.3. Location Criterion of Sensing Zones

For the members located in the sensing areas of theith satellite, it is important to determine which region these members belong to. In traditional ways, the belonging sensing area is usually determined by the location of the mass center of the member. Different from the traditional method, this paper applies the idea of the Box method [35], which takes the location relation of error ellipsoids as the criterion to judge whether two members in the cluster are repulsive or not. If the error ellipsoids intersect with each other, the repulsion force will be generated between the two members.In order to detect the position relation between the two error ellipsoids, the algebraic criterion is needed. During this process, it needs to carry out the affine transformation on the two ellipsoids. The process of affine transformation is shown in Figure4.Figure 4
Affine transformation process on the ellipsoids of theith and the jth member.Suppose that theS1 frame, which is centered at the nominal mass point of ith member, is parallel to Oxryrzr. Then, the position error ellipsoid of the ith member is expressed as follows:(7)RTArriR=1.Denote the symbolX=x,y,z,1T; therefore, the error ellipsoids of the ith member and the jth member can be transformed as(8)XTBiS1X=XTArri03×101×3−1X=0,XTBjS1X=XTT1−1TArrj03×101×3−1T1−1X=0,where BiS1 and BjS1 are ellipsoidal quadratic matrices and T1 is a translation matrix for the jth member.The details of the affine transformation process are shown in Appendix A. Here, we have the final expressions:(9)XTBiS4X=x−d12+y−d22+z−d22−1=0,(10)XTBjS4X=x2a2+y2b2+z2c2−1=0,where the frame comes to S4 and the condition a≤b≤c is satisfied.Therefore, we can obtain the standard discriminants as equations (9) and (10). The characteristic polynomial is defined as follows:(11)fλ=detλBjS4+BiS4.The relevant characteristic equation of the above polynomial is(12)fλ=0.According to the location judging algebraic criterion [36], the position relation between the ellipsoid of the ith member and the one of the jth members can be detected.If the characteristic equation has two different real roots, the two ellipsoids are separated. Thus, it will be easy to judge if thejth member locates in the ZOO or ZOA of the ith member or not. Otherwise, the two ellipsoids are not separated, which means that the jth member locates in the ZOR of the i member, and the repulsive force is generated.


### 2.4. Analysis of the Force Acting in Sensing Areas

For theith member, when the sensing sets Xr, Xo, and Xa, which relate to the ZOR, ZOO, and ZOA, respectively, are obtained, we can calculate the force directions of the ith member. It is noted that, for each member in a cluster, the member will have its own ZOR, ZOO, and ZOA. Therefore, we only talk about the condition of an arbitrary member in a cluster here. When this arbitrary member is mentioned, it is called “center member” for distinguishing it from the neighbours in its three sensing areas.


#### 2.4.1. Force Direction in ZOR

From the left part in Figure5, it shows that the individual m is closer to the center member than the individual n under the S1 frame. According to the traditional repulsive rule, the center member will be repulsed by the individual m more than the individual n. However, for the ellipsoidal ZOR of the center member, the intensity of the repulsive force should be related to the close degree from the individual m or n to the boundary of ZOR. Because the traditional repulsing area is defined as a standard sphere, the boundary is uniform in all directions, which is convenient to calculate the intensity of the repulsive force. Therefore, a special treatment is needed to deal with the nonuniform boundary of the repulsive area, which is shown in the right part in Figure 5. After the affine transformation process, the distance from the center member to the individual m and the one to the individual n is approximately equal. The reason of this situation is because that both of the individual m and n are originally near to the boundary of the repulsing area.Figure 5
Affine transformation process in repulsing area.Therefore, through the matrix,Ttr, which represents the transformation matrix from the S1 frame to the S2 frame, the direction of repulsive force of the ith member in the cluster is expressed as(13)pri=Ttr−1∑j∈Xr1r˜ij−1drr˜ij/∑j∈Xr1r˜ij−1drr˜ij,where r˜ij is the relative position vector in S2 frame and dr is the boundary of the repulsing area, which is expressed as follows:(14)dr=1.2maxj∈Xrr˜ij.It is noticed that the constant coefficient 1.2 is used to avoid ambiguity caused by zero denominator.


#### 2.4.2. Force Direction in ZOO

Due to the specific definition of ZOO, which is a standard sphere with radiusdm, the direction of orientation force can be expressed as(15)poi=vi+∑j∈Xovjvi+∑j∈Xmvj,where vi and vj represent the velocity of the ith member and the jth member in Oxryrzr, respectively.


#### 2.4.3. Force Direction in ZOA

Compared with the traditional ZOA, the proposed ZOA is set as the induction area of the center member, which means that the sensing ability is not uniform for the center member. In addition, we need to guarantee that the intensity of attractive force should be zero at the boundary of ZOO. As a result, the corresponding boundary of attracting area of each member in the cluster needs to be calculated.The direction of the attractive force is expressed as(16)pai=∑j∈Xa1/da−rij−1/da−dmrij1/da−rij−1/da−dmrij,where da is the boundary of the attracting area, which can be defined as follows:(17)da=1.2maxj∈Xarij.It is noted that, in Figure6, we see that the individual m and the individual n are located at the boundary of ZOA. The center member is expected to judge which is the farthest neighbour in its ZOA. Then, the relative position vector from the center to that neighbour will be used to generate da.Figure 6
Affine transformation process in attracting area.When the virtual host satellite is in a circle orbit or near-circle orbit, the conditions,θ˙=n and θ¨=0, are satisfied. To denote the symbol as X=xyzx˙y˙z˙T, the dynamic model, which is expressed in equation (2), can be rewritten as follows:(18)X˙=0001000000100000013n20002n0000−2n0000−n2000xyzx˙y˙z˙+000000000100010001fxfyfz=BX+Ca.To denote the symbols,cri∈−crimin,crimaxa, coi∈−coimin,coimax, and cai∈−caimin,caimax as the flight parameters related to ZOR, ZOO, and ZOA respectively, the motion controller of the ith member in the spacecraft cluster is designed as(19)ai=−BX−ν+cripri+coipoi+caipai,where ν=x˙y˙z˙T and i=1,2,…,N. It is noticed that the term −BX−ν is added to make the agent move stably during the control gap.Based on equations (13) to (16), the force directions of the ith member, pri, poi, and pai, can be calculated. Therefore, for the controller shown in equation (19), the key is to find the corresponding parameters cri, coi, and cai. The effect of cluster flight will be largely determined by these parameters.In traditional ways, the parameters are selected according to the experimental results or the expert experiences, which is known as TEPM. Nevertheless, considering the intelligent development of spacecraft and the raising labour cost, the satellite cluster needs to have a certain automatic capability to adjust the parameters in the future. To achieve this goal, an innovative algorithm of RLPST is proposed, which applies the reinforcement learning framework and is expected to make the flight parameters self-tuned along with multiple learning times.


## 2.1. Position Error Model of a Cluster Member

The position of a satellite is objective; however, it cannot be accurately known. In different types of missions, there will always be measurement, navigation, control, and other deviations and the influences of perturbation. These factors will cause the real orbit diverged from the nominal orbit of the spacecraft, resulting in trajectory deviation. Taking the Gaussian distribution as an example, the covariance matrix of the spacecraft state distribution is denoted asP, and then, it is obtained that the real state vector x is in a hyperellipsoid, which is centered on the nominal state vector x¯ [33, 34]. The sphere of the hyperellipsoid can be expressed as follows:(3)x−x¯TP−1x−x¯=l2.Meanwhile, the probability density function of the relative state error distribution is(4)pl=12πn∫0lexp−12r2frdr,where n represents the dimension of the state space and l represents the Markov distance constant. Specifically, when l=3, equation (3) represents the 3σ error ellipsoid.The above equation shows a six-dimensional ellipsoid. The matrix,A, is denoted as(5)A=P−1l2=−l2=ArrArvAvrAvv,where the matrix A is a real symmetric positive definite matrix. The symbol, R, is denoted as the position component of x; therefore, the position error ellipsoid of the spacecraft can be expressed as follows:(6)RTArrR=1.


## 2.2. Sensing Area Division of a Cluster Member

Traditionally, for dealing with problems of cluster, the sensing areas, which are generally known as zone of repulsion (ZOR), zone of orientation (ZOO), and zone of attraction (ZOA), are defined from the inside to the outside as the spherical regions [17]. Figure 3(a) shows these traditional ZOR, ZOO, and ZOA. Under such uniform sensing areas, it will be certainly convenient for describing the problem and designing the cluster control strategy. However, considering the location deviations of spacecraft cluster members and the capability of the attaching sensors, it is necessary to improve the way for dividing sensing areas. Here, we redefine the sensing areas of a satellite member in the cluster, which is illustrated in Figure 3(b).Figure 3
Sensing area division of member satellite in the cluster: (a) traditional sensing areas; (b) proposed sensing areas.
(a)
(b)According to Figures3(a) and 3(b), we see that, for each member satellite in the cluster, there exists three sensing areas. Compared with the traditional sensing areas, the redefined ones have replaced the ZOR and ZOA part with the 3σ error ellipsoid and the induction ellipsoid, respectively. The details of the redefined sensing areas are expressed below.


### 2.2.1. ZOR

The3σ error ellipsoid area of each member is defined as the zone of repulsion. It is assumed that the position deviation obeys the Gaussian distribution. As a result, each member in the cluster has its own position error ellipsoid. If the ellipsoid of a specific member interacts with the one of other individuals, the collision between the two members may occur. Therefore, such an ellipsoid is the ZOR for making repulsive force to avoid the probable collision. The members in the ZOR will make repulsive force on the center member. In this way, it will avoid the individuals in the cluster getting too close from each other.


### 2.2.2. ZOO

The orientation area is defined as a standard sphere with a specific radius. For a specific member, its ZOO is an ideal zone that neighbours, which are located in such an area, keep suitable distance with this specific member. This member will receive the orientation force from the neighbours in its ZOO, which makes the member tends to align its speed with its neighbours gradually. In this way, the flight process will be smooth.


### 2.2.3. ZOA

The attracting area is defined as the induction ellipsoid. Traditionally, the attracting area is uniform. However, in the case of spacecraft cluster problem, due to the capability of the sensing elements, the sensing ability may be strong or weak in different directions. Therefore, we use an ellipsoid model to nearly describe the induction area of the member in the cluster.


## 2.2.1. ZOR

The3σ error ellipsoid area of each member is defined as the zone of repulsion. It is assumed that the position deviation obeys the Gaussian distribution. As a result, each member in the cluster has its own position error ellipsoid. If the ellipsoid of a specific member interacts with the one of other individuals, the collision between the two members may occur. Therefore, such an ellipsoid is the ZOR for making repulsive force to avoid the probable collision. The members in the ZOR will make repulsive force on the center member. In this way, it will avoid the individuals in the cluster getting too close from each other.


## 2.2.2. ZOO

The orientation area is defined as a standard sphere with a specific radius. For a specific member, its ZOO is an ideal zone that neighbours, which are located in such an area, keep suitable distance with this specific member. This member will receive the orientation force from the neighbours in its ZOO, which makes the member tends to align its speed with its neighbours gradually. In this way, the flight process will be smooth.


## 2.2.3. ZOA

The attracting area is defined as the induction ellipsoid. Traditionally, the attracting area is uniform. However, in the case of spacecraft cluster problem, due to the capability of the sensing elements, the sensing ability may be strong or weak in different directions. Therefore, we use an ellipsoid model to nearly describe the induction area of the member in the cluster.


## 2.3. Location Criterion of Sensing Zones

For the members located in the sensing areas of theith satellite, it is important to determine which region these members belong to. In traditional ways, the belonging sensing area is usually determined by the location of the mass center of the member. Different from the traditional method, this paper applies the idea of the Box method [35], which takes the location relation of error ellipsoids as the criterion to judge whether two members in the cluster are repulsive or not. If the error ellipsoids intersect with each other, the repulsion force will be generated between the two members.In order to detect the position relation between the two error ellipsoids, the algebraic criterion is needed. During this process, it needs to carry out the affine transformation on the two ellipsoids. The process of affine transformation is shown in Figure4.Figure 4
Affine transformation process on the ellipsoids of theith and the jth member.Suppose that theS1 frame, which is centered at the nominal mass point of ith member, is parallel to Oxryrzr. Then, the position error ellipsoid of the ith member is expressed as follows:(7)RTArriR=1.Denote the symbolX=x,y,z,1T; therefore, the error ellipsoids of the ith member and the jth member can be transformed as(8)XTBiS1X=XTArri03×101×3−1X=0,XTBjS1X=XTT1−1TArrj03×101×3−1T1−1X=0,where BiS1 and BjS1 are ellipsoidal quadratic matrices and T1 is a translation matrix for the jth member.The details of the affine transformation process are shown in Appendix A. Here, we have the final expressions:(9)XTBiS4X=x−d12+y−d22+z−d22−1=0,(10)XTBjS4X=x2a2+y2b2+z2c2−1=0,where the frame comes to S4 and the condition a≤b≤c is satisfied.Therefore, we can obtain the standard discriminants as equations (9) and (10). The characteristic polynomial is defined as follows:(11)fλ=detλBjS4+BiS4.The relevant characteristic equation of the above polynomial is(12)fλ=0.According to the location judging algebraic criterion [36], the position relation between the ellipsoid of the ith member and the one of the jth members can be detected.If the characteristic equation has two different real roots, the two ellipsoids are separated. Thus, it will be easy to judge if thejth member locates in the ZOO or ZOA of the ith member or not. Otherwise, the two ellipsoids are not separated, which means that the jth member locates in the ZOR of the i member, and the repulsive force is generated.


## 2.4. Analysis of the Force Acting in Sensing Areas

For theith member, when the sensing sets Xr, Xo, and Xa, which relate to the ZOR, ZOO, and ZOA, respectively, are obtained, we can calculate the force directions of the ith member. It is noted that, for each member in a cluster, the member will have its own ZOR, ZOO, and ZOA. Therefore, we only talk about the condition of an arbitrary member in a cluster here. When this arbitrary member is mentioned, it is called “center member” for distinguishing it from the neighbours in its three sensing areas.


### 2.4.1. Force Direction in ZOR

From the left part in Figure5, it shows that the individual m is closer to the center member than the individual n under the S1 frame. According to the traditional repulsive rule, the center member will be repulsed by the individual m more than the individual n. However, for the ellipsoidal ZOR of the center member, the intensity of the repulsive force should be related to the close degree from the individual m or n to the boundary of ZOR. Because the traditional repulsing area is defined as a standard sphere, the boundary is uniform in all directions, which is convenient to calculate the intensity of the repulsive force. Therefore, a special treatment is needed to deal with the nonuniform boundary of the repulsive area, which is shown in the right part in Figure 5. After the affine transformation process, the distance from the center member to the individual m and the one to the individual n is approximately equal. The reason of this situation is because that both of the individual m and n are originally near to the boundary of the repulsing area.Figure 5
Affine transformation process in repulsing area.Therefore, through the matrix,Ttr, which represents the transformation matrix from the S1 frame to the S2 frame, the direction of repulsive force of the ith member in the cluster is expressed as(13)pri=Ttr−1∑j∈Xr1r˜ij−1drr˜ij/∑j∈Xr1r˜ij−1drr˜ij,where r˜ij is the relative position vector in S2 frame and dr is the boundary of the repulsing area, which is expressed as follows:(14)dr=1.2maxj∈Xrr˜ij.It is noticed that the constant coefficient 1.2 is used to avoid ambiguity caused by zero denominator.


### 2.4.2. Force Direction in ZOO

Due to the specific definition of ZOO, which is a standard sphere with radiusdm, the direction of orientation force can be expressed as(15)poi=vi+∑j∈Xovjvi+∑j∈Xmvj,where vi and vj represent the velocity of the ith member and the jth member in Oxryrzr, respectively.


### 2.4.3. Force Direction in ZOA

Compared with the traditional ZOA, the proposed ZOA is set as the induction area of the center member, which means that the sensing ability is not uniform for the center member. In addition, we need to guarantee that the intensity of attractive force should be zero at the boundary of ZOO. As a result, the corresponding boundary of attracting area of each member in the cluster needs to be calculated.The direction of the attractive force is expressed as(16)pai=∑j∈Xa1/da−rij−1/da−dmrij1/da−rij−1/da−dmrij,where da is the boundary of the attracting area, which can be defined as follows:(17)da=1.2maxj∈Xarij.It is noted that, in Figure6, we see that the individual m and the individual n are located at the boundary of ZOA. The center member is expected to judge which is the farthest neighbour in its ZOA. Then, the relative position vector from the center to that neighbour will be used to generate da.Figure 6
Affine transformation process in attracting area.When the virtual host satellite is in a circle orbit or near-circle orbit, the conditions,θ˙=n and θ¨=0, are satisfied. To denote the symbol as X=xyzx˙y˙z˙T, the dynamic model, which is expressed in equation (2), can be rewritten as follows:(18)X˙=0001000000100000013n20002n0000−2n0000−n2000xyzx˙y˙z˙+000000000100010001fxfyfz=BX+Ca.To denote the symbols,cri∈−crimin,crimaxa, coi∈−coimin,coimax, and cai∈−caimin,caimax as the flight parameters related to ZOR, ZOO, and ZOA respectively, the motion controller of the ith member in the spacecraft cluster is designed as(19)ai=−BX−ν+cripri+coipoi+caipai,where ν=x˙y˙z˙T and i=1,2,…,N. It is noticed that the term −BX−ν is added to make the agent move stably during the control gap.Based on equations (13) to (16), the force directions of the ith member, pri, poi, and pai, can be calculated. Therefore, for the controller shown in equation (19), the key is to find the corresponding parameters cri, coi, and cai. The effect of cluster flight will be largely determined by these parameters.In traditional ways, the parameters are selected according to the experimental results or the expert experiences, which is known as TEPM. Nevertheless, considering the intelligent development of spacecraft and the raising labour cost, the satellite cluster needs to have a certain automatic capability to adjust the parameters in the future. To achieve this goal, an innovative algorithm of RLPST is proposed, which applies the reinforcement learning framework and is expected to make the flight parameters self-tuned along with multiple learning times.


## 2.4.1. Force Direction in ZOR

From the left part in Figure5, it shows that the individual m is closer to the center member than the individual n under the S1 frame. According to the traditional repulsive rule, the center member will be repulsed by the individual m more than the individual n. However, for the ellipsoidal ZOR of the center member, the intensity of the repulsive force should be related to the close degree from the individual m or n to the boundary of ZOR. Because the traditional repulsing area is defined as a standard sphere, the boundary is uniform in all directions, which is convenient to calculate the intensity of the repulsive force. Therefore, a special treatment is needed to deal with the nonuniform boundary of the repulsive area, which is shown in the right part in Figure 5. After the affine transformation process, the distance from the center member to the individual m and the one to the individual n is approximately equal. The reason of this situation is because that both of the individual m and n are originally near to the boundary of the repulsing area.Figure 5
Affine transformation process in repulsing area.Therefore, through the matrix,Ttr, which represents the transformation matrix from the S1 frame to the S2 frame, the direction of repulsive force of the ith member in the cluster is expressed as(13)pri=Ttr−1∑j∈Xr1r˜ij−1drr˜ij/∑j∈Xr1r˜ij−1drr˜ij,where r˜ij is the relative position vector in S2 frame and dr is the boundary of the repulsing area, which is expressed as follows:(14)dr=1.2maxj∈Xrr˜ij.It is noticed that the constant coefficient 1.2 is used to avoid ambiguity caused by zero denominator.


## 2.4.2. Force Direction in ZOO

Due to the specific definition of ZOO, which is a standard sphere with radiusdm, the direction of orientation force can be expressed as(15)poi=vi+∑j∈Xovjvi+∑j∈Xmvj,where vi and vj represent the velocity of the ith member and the jth member in Oxryrzr, respectively.


## 2.4.3. Force Direction in ZOA

Compared with the traditional ZOA, the proposed ZOA is set as the induction area of the center member, which means that the sensing ability is not uniform for the center member. In addition, we need to guarantee that the intensity of attractive force should be zero at the boundary of ZOO. As a result, the corresponding boundary of attracting area of each member in the cluster needs to be calculated.The direction of the attractive force is expressed as(16)pai=∑j∈Xa1/da−rij−1/da−dmrij1/da−rij−1/da−dmrij,where da is the boundary of the attracting area, which can be defined as follows:(17)da=1.2maxj∈Xarij.It is noted that, in Figure6, we see that the individual m and the individual n are located at the boundary of ZOA. The center member is expected to judge which is the farthest neighbour in its ZOA. Then, the relative position vector from the center to that neighbour will be used to generate da.Figure 6
Affine transformation process in attracting area.When the virtual host satellite is in a circle orbit or near-circle orbit, the conditions,θ˙=n and θ¨=0, are satisfied. To denote the symbol as X=xyzx˙y˙z˙T, the dynamic model, which is expressed in equation (2), can be rewritten as follows:(18)X˙=0001000000100000013n20002n0000−2n0000−n2000xyzx˙y˙z˙+000000000100010001fxfyfz=BX+Ca.To denote the symbols,cri∈−crimin,crimaxa, coi∈−coimin,coimax, and cai∈−caimin,caimax as the flight parameters related to ZOR, ZOO, and ZOA respectively, the motion controller of the ith member in the spacecraft cluster is designed as(19)ai=−BX−ν+cripri+coipoi+caipai,where ν=x˙y˙z˙T and i=1,2,…,N. It is noticed that the term −BX−ν is added to make the agent move stably during the control gap.Based on equations (13) to (16), the force directions of the ith member, pri, poi, and pai, can be calculated. Therefore, for the controller shown in equation (19), the key is to find the corresponding parameters cri, coi, and cai. The effect of cluster flight will be largely determined by these parameters.In traditional ways, the parameters are selected according to the experimental results or the expert experiences, which is known as TEPM. Nevertheless, considering the intelligent development of spacecraft and the raising labour cost, the satellite cluster needs to have a certain automatic capability to adjust the parameters in the future. To achieve this goal, an innovative algorithm of RLPST is proposed, which applies the reinforcement learning framework and is expected to make the flight parameters self-tuned along with multiple learning times.


## 3. Continuous Reinforcement Learning

### 3.1. The Fuzzy Inference System

In order to apply the reinforcement learning into the space cluster, which has continuous dynamic systems, it is reasonable to find a way to not only avoid the curse of dimensionality but also have the clear physical meaning. Therefore, a zero-order Takagi–Sugeno (T-S) fuzzy system is employed as the approximator. It is assumed that the fuzzy system hasL rules and n input variables. The fuzzy inference rule is(20)Rulel:IFs1isF1l,…,andsnisFnlthenzl=ϕl,where sii=1,…,n represents the ith input of the fuzzy system, Fil represents the fuzzy set of the ith input variable, zl represents the output of the lth rule, and ϕl represents the output parameter.With theh membership functions of each si, the output of the fuzzy system is expressed as(21)Zs=∑l=1L∏i=1nμFilsiϕl∑l=1L∏i=1nμFilsi=∑l=1LΨlsϕl,where s=s1,…,snT is the state vector and μFil is the membership function of si under the lth rule. In addition, the expression of Ψls is as follows:(22)Ψls=∏i=1nμFilsi∑l=1L∏i=1nμFilsi=ωls∑l=1Lωls.


### 3.2. The Actor-Critic Learning Algorithm

Reinforcement learning is a type of algorithm that interacts with the environment. The agent optimizes its behaviour through the rewards obtained from the environment for maximizing the total benefits. In the Markov process, the value function of reinforcement learning can be expressed as(23)Vtst=E∑i=t∞γi−tRi,where γ∈0,1 is the discount factor and Ri is the immediate reward which is obtained from the environment.In order to solve Markov decision problem in continuous action space, a type of reinforcement learning algorithm called adaptive heuristic critic (AHC) has been widely studied and applied. In the AHC algorithm, the value function and the policy function are approximated, respectively. In this way, the learning structure is called the actor-critic framework. In such a learning algorithm, the critic part is used to estimate the value function, while the actor part is used to generate the action. To generalize the state space and the action space, the critic part and the actor part are both composed of T-S systems. To apply the temporal difference (TD) learning method, we need two critic parts for estimating the current value functionVtst and the next value function Vtst+1. The temporal difference can be expressed as follows:(24)Δt=Rt+γVtst+1−Vtst.E is denoted as the variance of the difference signal, which is shown as(25)E=12Δt2,and the adaptive update rule of the parameters in the critic is expressed as(26)ϕCt+1=ϕCt−α∂E∂ϕC,where α is the learning rate of the critic.Furthermore, according to the gradient descent method, it is shown that(27)∂E∂ϕC=Δt−∂Vtst∂ϕC.To sum up, we have(28)ϕCt+1=ϕCt−αRt+γVtst+1−Vtst−∂Vtst∂ϕC,(29)∂Vtst∂ϕC=Ψ1st,Ψ2st,…,ΨLst.Combining with equation (22), equation (28) can be solved.The adaptive update rule for the critic part is shown as above. As for the actor part, the adaptive update rule of the output parameter,ϕA, is expressed as(30)ϕAt+1=ϕAt+βΔt∂ut∂ϕA,where β is the learning rate of the actor. The partial derivative of ut is expressed as follows:(31)∂ut∂ϕA=Ψ1st,Ψ2st,…,ΨLst.


## 3.1. The Fuzzy Inference System

In order to apply the reinforcement learning into the space cluster, which has continuous dynamic systems, it is reasonable to find a way to not only avoid the curse of dimensionality but also have the clear physical meaning. Therefore, a zero-order Takagi–Sugeno (T-S) fuzzy system is employed as the approximator. It is assumed that the fuzzy system hasL rules and n input variables. The fuzzy inference rule is(20)Rulel:IFs1isF1l,…,andsnisFnlthenzl=ϕl,where sii=1,…,n represents the ith input of the fuzzy system, Fil represents the fuzzy set of the ith input variable, zl represents the output of the lth rule, and ϕl represents the output parameter.With theh membership functions of each si, the output of the fuzzy system is expressed as(21)Zs=∑l=1L∏i=1nμFilsiϕl∑l=1L∏i=1nμFilsi=∑l=1LΨlsϕl,where s=s1,…,snT is the state vector and μFil is the membership function of si under the lth rule. In addition, the expression of Ψls is as follows:(22)Ψls=∏i=1nμFilsi∑l=1L∏i=1nμFilsi=ωls∑l=1Lωls.


## 3.2. The Actor-Critic Learning Algorithm

Reinforcement learning is a type of algorithm that interacts with the environment. The agent optimizes its behaviour through the rewards obtained from the environment for maximizing the total benefits. In the Markov process, the value function of reinforcement learning can be expressed as(23)Vtst=E∑i=t∞γi−tRi,where γ∈0,1 is the discount factor and Ri is the immediate reward which is obtained from the environment.In order to solve Markov decision problem in continuous action space, a type of reinforcement learning algorithm called adaptive heuristic critic (AHC) has been widely studied and applied. In the AHC algorithm, the value function and the policy function are approximated, respectively. In this way, the learning structure is called the actor-critic framework. In such a learning algorithm, the critic part is used to estimate the value function, while the actor part is used to generate the action. To generalize the state space and the action space, the critic part and the actor part are both composed of T-S systems. To apply the temporal difference (TD) learning method, we need two critic parts for estimating the current value functionVtst and the next value function Vtst+1. The temporal difference can be expressed as follows:(24)Δt=Rt+γVtst+1−Vtst.E is denoted as the variance of the difference signal, which is shown as(25)E=12Δt2,and the adaptive update rule of the parameters in the critic is expressed as(26)ϕCt+1=ϕCt−α∂E∂ϕC,where α is the learning rate of the critic.Furthermore, according to the gradient descent method, it is shown that(27)∂E∂ϕC=Δt−∂Vtst∂ϕC.To sum up, we have(28)ϕCt+1=ϕCt−αRt+γVtst+1−Vtst−∂Vtst∂ϕC,(29)∂Vtst∂ϕC=Ψ1st,Ψ2st,…,ΨLst.Combining with equation (22), equation (28) can be solved.The adaptive update rule for the critic part is shown as above. As for the actor part, the adaptive update rule of the output parameter,ϕA, is expressed as(30)ϕAt+1=ϕAt+βΔt∂ut∂ϕA,where β is the learning rate of the actor. The partial derivative of ut is expressed as follows:(31)∂ut∂ϕA=Ψ1st,Ψ2st,…,ΨLst.


## 4. Algorithm of Reinforcement Learning for Parameter Self-Tuning in Satellite Cluster

The proposed learning framework in this paper is single-looped, which can be divided into three channels of repulsing arear, orientating area o, and attracting area a, respectively. The input of the fuzzy system is single, which is defined as the proportion of the total number of the sensing members in every sensing area. The ith member is taken as an example, and its inputs for fuzzy systems are expressed as(32)sa=naN,so=noN,sr=nrN,where na, no, and nr represent the number of sensing neighbours in the repulsing area, orientating area, and attracting area, respectively. Besides, the symbol N represents the total number of all sensing neighbours of the ith agent. Therefore, sa, so, and sr represent the proportions of sensing members in the corresponding areas.The critic part and the actor part are composed of fuzzy systems. The inference rule is shown as(33)Rlq:IFsqisAqlTHENZlq=ϕlCq,where ⋅q represents the variable in the q channel q=a,o,r. As mentioned in Section 3.1, it is supposed that the input has h membership functions and the fuzzy system has L rules in total. It is noticed that, because the input is single, it meets that L=h. Therefore, according to the membership degree of the input, the output can be calculated:(34)Ψlq=μAql∑l=1LμAql,(35)Vq=∑l=1LΨlq⋅ϕlCq,q=r,o,a.The fuzzy inference process of the actor part is similar to that of the critic part and the difference lies in the consequent parameter to each membership degree:(36)cq=∑l=1hΨlq⋅ϕlAq,q=r,o,a.The three sensing areas have different proportions of sensing members; therefore, the designed reward function,Rtq, is expressed as (37)Rt=Rtr=10,Rto=0,Rta=0,sr>0,Rta=10,Rto=0,Rtr=0,sr=0,sa>ε,Rto=10,Rtr=0,Rta=0,others.From the structure of the reward function, it is shown that if the proportion of the sensing member,sr, is positive, the system will receive a positive reward, which will stimulate the system to enhance the coefficient of force in the repulsing sensing area. Except that the condition where sr is positive, the rewards of other sensing areas will be decided according to the states of sa. When sa is larger than ε, which is a positive separator, the reward about ZOA is positive; otherwise, the reward about ZOO is positive. It is mentioned that the rewards of the three areas cannot be calculated simultaneously. Otherwise, it may make the parameters enhanced simultaneously, which may make the learning invalidate. Therefore, the calculation process of the reward needs to be prioritized according to specific tasks. The whole diagram of learning logic is illustrated in Figure 7.Figure 7
The diagram of learning logic.In Figure7, there exists two critic parts and one actor part in each channel. The two critic parts are applied to estimate the value of current time, Vt, and the value of next time, Vt+1. According to sat, sot, and srt, the parameters, cai, coi, and cri are calculated. Bring these parameters into the motion controller, which is designed in equation (19), srt+1, sot+1, and sat+1 are obtained. Besides, the immediate reward, Rt, is also acquired from the environment. According to Rt, Vt, and Vt+1, the time difference, Δt, is calculated. The output parameters of the critic part and the actor part can be adjusted according to Δt.To sum up, the learning algorithm of RLPST is shown as Algorithm1.Algorithm 1:RLPST.
(1)
for all cluster members do
(2)
for all channel do
(3)
Initialize the membership functions
(4)
InitializeVq=0, ϕlCq=0, ϕlAq=0, for l=1,…,L;
(5)
end for
(6)
end for
(7)
for each episode do
(8)
for all cluster members do
(9)
Initialize states of the cluster member
(10)
for all Time step do
(11)
Calculate the 3σ position error ellipsoid according to equation (7)
(12)
Maintain all sensing neighbours of the cluster member
(13)
Obtain the sensing sets,Xr, Xo, and Xa, respectively, based on the results of equation (11)-equation (12)
(14)
Calculate the force directpir, pio, and pia according to equation (13), equation (15), and equation (16), respectively
(15)
for all channel do
(16)
Calculate the output of the actorcq through equation (36)
(17)
Calculate the output of the criticVqt from equation (35)
(18)
Interact with the environment
(19)
Obtain the rewardRt, and the output of the critic Vqt+1
(20)
Calculate the time differenceΔt from equation (24)
(21)
UpdateϕlCq and ϕlAq=0 according to equation (26) and equation (30), respectively
(22)
end for
(23)
end for
(24)
end for
(25)
end for


## 5. Simulation

A cluster with four satellite members, which are numbered from No. 1 to No. 4, is selected as the numerical experimental object. It is supposed that the reference orbit is a circular orbit with the radius of104 km. The symbols, x10, x20, x30 and x40, are denoted as the initial states of the satellites in the cluster from No. 1 to No. 4. The first three items of these vectors represent the relative position in m, while the last three items represent the relative velocity in m/s. In this numerical experiment, for each cluster member, the quadratic matrix of the position error ellipsoid is set as A and that of the induction ellipsoid is set as M. Based on the reference in [33, 34, 37], the values of A and M are set as follows:(38)A=215.41−84.4356.29−84.43312.91−97.5056.29−97.50231.66,M=3.11e−72.19e−8−4.65e−92.18e−82.78e−73.63e−8−4.65e−93.63e−82.92e−7.Two classic scenarios, which include the scenario of adding members into the cluster and the scenario of members following a flight path, are considered, respectively. It is noticed that when we talk about adding members, the combination of No. 1 and No. 2 is set as the original cluster, while No. 3 and No. 4 are the ones who want to add into. Besides, when the members are following a flight path, No. 1 is set as the leader satellite, which holds a desired path, and others are the ones who need to follow the path of No. 1. With the limits ofcri∈−20,20, coi∈−20,20, and cai∈−20,20, the experimental results are shown below.


### 5.1. Scenario of Adding Members into the Cluster

In this scenario, satellites No. 1 and No. 2, which are considered as the members of original cluster, are flying together with a proper distance. Satellites No. 3 and No. 4 aim to merge into the cluster from the different directions, respectively. The goal of the mission is to make the four members get into a new cluster with a proper distance from each other at the terminal time. A proper distance means that there is no neighbour located in the ZOR of each member in the cluster, and it makes the neighbours located in the ZOO as much as possible. The initial states of the cluster are listed in Table1.Table 1
Initial states of the members in the cluster.
State
Value
x
10
0
;
10
0
;
0
;
2.678
×
10
−
2
;
−
4.715
×
10
−
5
;
0
T
x
20
0
;
−
10
0
;
0
;
−
2.678
×
10
−
2
;
−
5.608
×
10
−
3
;
0
T
x
30
0
;
1000
;
0
;
2.680
×
10
−
2
;
−
4.715
×
10
−
5
;
0
T
x
40
−
10
00
;
−
10
00
;
0
;
−
2.678
×
10
−
2
;
−
5.608
×
10
−
3
;
0
TFor representing the smoothness of the flight paths, the signalσ is defined to express the deviation degree from the whole flight path to the center baseline:(39)σ=∑i=3N∫t=0Tρit−ρreft2,where ρi and ρref represent the position vector of the ith member and the corresponding position vector on the center baseline, respectively.For the mission of adding members into an original cluster, it is appropriate to judge the terminal matching degree of new adding members. Therefore, the signalηm, which is called the terminal matching rate, is defined to represent the degree of terminal status in ZOO:(40)ηm=∑i=3NNumim∑i=3NN−2,where Numim represents the number of neighbours in the ZOO of the ith member.In order to express the effectiveness of improving the flight paths under the proposed RLPST, the signalCost is defined to represent the quality of distances among new adding members, which is shown as(41)Cost=∑i=3,j∈XrN∫t=0Tmr1rntdt+∑i=3,j∈XaN∫t=0Tmarntdt,where mr and ma represent the corresponding coefficients of sets Xr and Xa, respectively.k is denoted as the empirical parameter to substitute the value of cr, co, and ca in TEPM; then, the experimental results under the simulation time T with 1000s are shown below.From Figures8(a) and 8(b), the trajectories of TEPM with k=3 and k=6.5 are illustrated, respectively. In Figure 8(a), it is seen that the terminal positions of the four members are relatively far away from each other, which does not satisfy the requirement of the mission. This is because the empirical parameter is selected too small. On the contrary, from Figure 8(b), due to the large empirical parameter, it shows the nonsmooth trajectories of satellites No. 3 and No. 4. Although the requirement of terminal positions of the four satellites is guaranteed, the flying process will waste unnecessary fuels for the nonsmooth flight paths. Therefore, it is seen that we will easily get confused for the selection of empirical parameter under the TEPM. Whether the parameter is selected too large or too small, the flying effect cannot meet the goal of mission.Figure 8
Trajectories of TEPM with different empirical parameters in the scenario of adding members: (a)k = 3; (b) k = 6.
(a)
(b)To compare with the results of TEPM, we set the discount factorγ as 0.8, the reward separator ε as 0.5, the learning rate of the critic α as 10−7, the learning rate of the actor β as 10−8, the coefficients in equation (41) mr as 1000, and ma as 10−3. Thus, the results under the proposed RLPST are shown in Figures 9 and 10, where the results of members adding with different learning times are illustrated, respectively. From Figure 9(a), it is seen that the original cluster (which includes satellite No. 1 and satellite No. 2) keeps normally flying, and satellites No. 3 and No. 4 are far from the original cluster at the beginning. In addition, satellite No. 4 is moving nearly in the same direction as the original cluster, while satellite No. 3 is the opposite. Attracted by the original cluster, satellites No. 3 and No. 4 begin to move, where satellite No. 3 appears the tendency to change the moving direction, and satellite No. 4 keeps moving forward. In Figure 9(b), it can be seen that, with the increase in learning times, the direction changing is fixed, and the satellites No. 3 and No. 4 have basically determined the same flight direction as the original cluster. However, they have not integrated with each other yet and they are still attracted by the original cluster continuously. Figure 10 shows the finished training results after 55 times of learning. As the flight progresses, satellites No. 3 and No. 4 finally have merged into the original cluster to form a new cluster, and the mission of adding members is completed.Figure 9
Results of the training different times in the scenario of members adding: (a) 2 times; (b) 30 times.
(a)
(b)Figure 10
Results of the final training in the scenario of members adding.Figure11 shows the trajectory deviations under the TEPM and the proposed RLPST, respectively. It is seen that the deviation under the TEPM is relatively large when the empirical parameter is set too large or too small. The result is reasonable because when the empirical parameter is too small, the whole flying condition cannot meet the terminal requirement of the mission, and when the empirical parameter is too large, the flight paths are nonsmooth, which may cause large trajectory deviation as well. When the empirical parameter is set to be an acceptable value, the trajectory deviation will meet the low point in the figure. However, compared with the proposed RLPST, the deviation under the RLPST is obviously lower than that under the TEPM, which means that the proposed RLPST has more smooth flight path which is a benefit for saving fuels and avoiding complex maneuvering strategies.Figure 11
Curves of trajectory deviations under TEPM and RLPST in the scenario of adding members.The terminal matching rate represents the final states of the cluster, and the ideal value is equal to one, which means that the adding member keeps a moderate distance with not only the original cluster members but also other adding members. From Figure12, it is seen that the rate is different with different empirical parameters, which means that the matching rate cannot be guaranteed optimal under the TEPM. On the contrary, the solid line represents the rate under the RLPST, and it is clear that the rate is equal to one when the learning process is finished.Figure 12
Curves of terminal matching rate under TEPM and RLPST in the scenario of adding members.From Figure13, the variation in the cost line along with learning times is illustrated. It is clearly seen that the cost is generally decreased with the increasing learning times. The figure indicates that the learning process has reduced the cost effectively, which means that the total flying condition is improved gradually during the process.Figure 13
Curve of cost line along with learning times in the scenario of adding members.


### 5.2. Scenario of Members following a Flight Path

In this scenario, satellite No. 1 is a leader, which has the desired flight path. Satellites No. 2 to No. 4 are expected to follow the path of the leader. The task requires that satellites No. 2, No. 3, and No. 4 can trace the leader effectively. The initial states of cluster members are shown in Table2.Table 2
Initial states of the members in the cluster.
State
Value
x
10
0
;
0
;
0
;
0
;
0
;
0
T
x
20
500
;
500
;
0
;
−
2.678
×
10
−
2
;
−
5.608
×
10
−
3
;
0
T
x
30
−
500
;
500
;
0
;
2.680
×
10
−
2
;
−
4.715
×
10
−
5
;
0
T
x
40
−
500
;
−
500
;
0
;
−
2.678
×
10
−
2
;
−
5.608
×
10
−
3
;
0
TSimilar to Section5.1, to express the deviation degree from the whole flight path to the center baseline, the following definition is executed:(42)σ=∑i=2N∫t=0Tρit−ρreft2,where ρi and ρref represent the position vector of the ith member and the corresponding position vector on the center baseline, respectively.Besides, the signalCost is also defined to represent the quality of distances among new adding members, which is shown as follows:(43)Cost=∑i=2,j∈XrN∫t=0Tmr1rntdt+∑i=2,j∈XaN∫t=0Tmarntdt.In addition, in the scenario of members following a leader, it will be reasonable to care about how many neighbours can each member sense. The more neighbours that a member can sense, the more information can the member obtain, which will be benefit for planning the flight paths. Therefore, the symbolηs is defined as the sensing rate for representing the degree of sensing ability of the members in the cluster:(44)ηs=∑i=2NNumis∑i=2NN−1,where Numis represents the number of neighbours that the ith member can sense.We set the discount factorγ as 0.8, the reward separator ε as 0.2, the learning rate of the critic α as 5×10−4, the learning rate of the actor β as 10−4, the coefficients mr as 1000, and ma as 10−3. Thus, the experimental results under the simulation time T with 1000s are shown below.From Figures14(a) and 14(b), the trajectories under TEPM with k=6 and k=22 are illustrated, respectively. In Figure 14(a), it is seen that satellites No. 2 to No. 4 fly aside from satellite No. 1 in the latter half of flight. The reason why their flight paths deviate from the leader is because that the empirical parameter is set too small that the members cannot sense the leader. It indicates that, in TEPM, if the empirical parameter is too small, some unexpected situations will occur which may result in the failure of the mission. In Figure 14(b), it is seen that the flight paths of the members are nonsmooth, which is not a good flight condition for following the leader. Certainly, this is because of the large value of the selected empirical parameter. Figures 14(a) and 14(b) have further explained the difficulties for selecting empirical parameter. The unsuitable selections may cause bad flying results or even lead to the failure of the mission. Besides, compared with the results from the Figures 14(a) and 14(b), the final training effect of members flight following under RLPST is illustrated in Figure 15. It is seen that satellite No. 1 is the leader in the cluster and satellites No. 2, No. 3 and No. 4 are scattered from each other at initial condition. When the training is finished, satellites No. 2 to No. 4 have the ability to follow the leader successfully and keep a smooth flight path.Figure 14
Trajectories of TEPM with different empirical parameters in the scenario of members following a flight path: (a)k = 6; (b) k = 22.
(a)
(b)Figure 15
Results of final training in the scenario of members following a flight path.From Figure16, it is seen that when the empirical parameter is set from 12 to 16, the sensing rate is equal to one. In such a condition, all members in the cluster can sense other members during the whole flight. However, when the empirical parameter is too small or too large, the sensing rate will not be guaranteed to be one, which means that flight effect may be badly influenced. On the contrary, the solid line represents the sensing rate under the proposed RLPST, which is guaranteed to be one when the learning process is finished. The figure shows the superiority of the proposed RLPST because of the assurance of the optimal sensing rate.Figure 16
Curves of sensing rate under TEPM and RLPST in the scenario of members following a flight path.Figure17 shows the trajectory deviations under the TEPM and the proposed RLPST, respectively. Similar to the condition shown in Figure 11, when the empirical parameter is too small or too large, the deviation is obviously high because of the badly flight results. In the figure, if the parameter is chosen as about 12, it has the lowest value of deviation. However, compared with the value under RLPST, it indicates that the value under RLPST is still smaller than the lowest value under TEPM. Therefore, the RLPST method has the ability to meet the more lower deviation within the safety flight range, which makes the flight path more smooth.Figure 17
Curves of trajectory deviations under TEPM and RLPST in the scenario of members following a flight path.From Figure18, the variation in cost along with the learning times is illustrated. Similar to the curves drawn in Figure 13, the cost is generally decreased with the increase in learning times. When the learning process is finished, the near lowest cost for the mission is found. The figure indicates that the flight condition is gradually improved during the learning process, and the cost of the flight can be effectively reduced through the proposed RLPST.Figure 18
Curve of cost line along with learning times in the scenario of members following a flight path.


## 5.1. Scenario of Adding Members into the Cluster

In this scenario, satellites No. 1 and No. 2, which are considered as the members of original cluster, are flying together with a proper distance. Satellites No. 3 and No. 4 aim to merge into the cluster from the different directions, respectively. The goal of the mission is to make the four members get into a new cluster with a proper distance from each other at the terminal time. A proper distance means that there is no neighbour located in the ZOR of each member in the cluster, and it makes the neighbours located in the ZOO as much as possible. The initial states of the cluster are listed in Table1.Table 1
Initial states of the members in the cluster.
State
Value
x
10
0
;
10
0
;
0
;
2.678
×
10
−
2
;
−
4.715
×
10
−
5
;
0
T
x
20
0
;
−
10
0
;
0
;
−
2.678
×
10
−
2
;
−
5.608
×
10
−
3
;
0
T
x
30
0
;
1000
;
0
;
2.680
×
10
−
2
;
−
4.715
×
10
−
5
;
0
T
x
40
−
10
00
;
−
10
00
;
0
;
−
2.678
×
10
−
2
;
−
5.608
×
10
−
3
;
0
TFor representing the smoothness of the flight paths, the signalσ is defined to express the deviation degree from the whole flight path to the center baseline:(39)σ=∑i=3N∫t=0Tρit−ρreft2,where ρi and ρref represent the position vector of the ith member and the corresponding position vector on the center baseline, respectively.For the mission of adding members into an original cluster, it is appropriate to judge the terminal matching degree of new adding members. Therefore, the signalηm, which is called the terminal matching rate, is defined to represent the degree of terminal status in ZOO:(40)ηm=∑i=3NNumim∑i=3NN−2,where Numim represents the number of neighbours in the ZOO of the ith member.In order to express the effectiveness of improving the flight paths under the proposed RLPST, the signalCost is defined to represent the quality of distances among new adding members, which is shown as(41)Cost=∑i=3,j∈XrN∫t=0Tmr1rntdt+∑i=3,j∈XaN∫t=0Tmarntdt,where mr and ma represent the corresponding coefficients of sets Xr and Xa, respectively.k is denoted as the empirical parameter to substitute the value of cr, co, and ca in TEPM; then, the experimental results under the simulation time T with 1000s are shown below.From Figures8(a) and 8(b), the trajectories of TEPM with k=3 and k=6.5 are illustrated, respectively. In Figure 8(a), it is seen that the terminal positions of the four members are relatively far away from each other, which does not satisfy the requirement of the mission. This is because the empirical parameter is selected too small. On the contrary, from Figure 8(b), due to the large empirical parameter, it shows the nonsmooth trajectories of satellites No. 3 and No. 4. Although the requirement of terminal positions of the four satellites is guaranteed, the flying process will waste unnecessary fuels for the nonsmooth flight paths. Therefore, it is seen that we will easily get confused for the selection of empirical parameter under the TEPM. Whether the parameter is selected too large or too small, the flying effect cannot meet the goal of mission.Figure 8
Trajectories of TEPM with different empirical parameters in the scenario of adding members: (a)k = 3; (b) k = 6.
(a)
(b)To compare with the results of TEPM, we set the discount factorγ as 0.8, the reward separator ε as 0.5, the learning rate of the critic α as 10−7, the learning rate of the actor β as 10−8, the coefficients in equation (41) mr as 1000, and ma as 10−3. Thus, the results under the proposed RLPST are shown in Figures 9 and 10, where the results of members adding with different learning times are illustrated, respectively. From Figure 9(a), it is seen that the original cluster (which includes satellite No. 1 and satellite No. 2) keeps normally flying, and satellites No. 3 and No. 4 are far from the original cluster at the beginning. In addition, satellite No. 4 is moving nearly in the same direction as the original cluster, while satellite No. 3 is the opposite. Attracted by the original cluster, satellites No. 3 and No. 4 begin to move, where satellite No. 3 appears the tendency to change the moving direction, and satellite No. 4 keeps moving forward. In Figure 9(b), it can be seen that, with the increase in learning times, the direction changing is fixed, and the satellites No. 3 and No. 4 have basically determined the same flight direction as the original cluster. However, they have not integrated with each other yet and they are still attracted by the original cluster continuously. Figure 10 shows the finished training results after 55 times of learning. As the flight progresses, satellites No. 3 and No. 4 finally have merged into the original cluster to form a new cluster, and the mission of adding members is completed.Figure 9
Results of the training different times in the scenario of members adding: (a) 2 times; (b) 30 times.
(a)
(b)Figure 10
Results of the final training in the scenario of members adding.Figure11 shows the trajectory deviations under the TEPM and the proposed RLPST, respectively. It is seen that the deviation under the TEPM is relatively large when the empirical parameter is set too large or too small. The result is reasonable because when the empirical parameter is too small, the whole flying condition cannot meet the terminal requirement of the mission, and when the empirical parameter is too large, the flight paths are nonsmooth, which may cause large trajectory deviation as well. When the empirical parameter is set to be an acceptable value, the trajectory deviation will meet the low point in the figure. However, compared with the proposed RLPST, the deviation under the RLPST is obviously lower than that under the TEPM, which means that the proposed RLPST has more smooth flight path which is a benefit for saving fuels and avoiding complex maneuvering strategies.Figure 11
Curves of trajectory deviations under TEPM and RLPST in the scenario of adding members.The terminal matching rate represents the final states of the cluster, and the ideal value is equal to one, which means that the adding member keeps a moderate distance with not only the original cluster members but also other adding members. From Figure12, it is seen that the rate is different with different empirical parameters, which means that the matching rate cannot be guaranteed optimal under the TEPM. On the contrary, the solid line represents the rate under the RLPST, and it is clear that the rate is equal to one when the learning process is finished.Figure 12
Curves of terminal matching rate under TEPM and RLPST in the scenario of adding members.From Figure13, the variation in the cost line along with learning times is illustrated. It is clearly seen that the cost is generally decreased with the increasing learning times. The figure indicates that the learning process has reduced the cost effectively, which means that the total flying condition is improved gradually during the process.Figure 13
Curve of cost line along with learning times in the scenario of adding members.


## 5.2. Scenario of Members following a Flight Path

In this scenario, satellite No. 1 is a leader, which has the desired flight path. Satellites No. 2 to No. 4 are expected to follow the path of the leader. The task requires that satellites No. 2, No. 3, and No. 4 can trace the leader effectively. The initial states of cluster members are shown in Table2.Table 2
Initial states of the members in the cluster.
State
Value
x
10
0
;
0
;
0
;
0
;
0
;
0
T
x
20
500
;
500
;
0
;
−
2.678
×
10
−
2
;
−
5.608
×
10
−
3
;
0
T
x
30
−
500
;
500
;
0
;
2.680
×
10
−
2
;
−
4.715
×
10
−
5
;
0
T
x
40
−
500
;
−
500
;
0
;
−
2.678
×
10
−
2
;
−
5.608
×
10
−
3
;
0
TSimilar to Section5.1, to express the deviation degree from the whole flight path to the center baseline, the following definition is executed:(42)σ=∑i=2N∫t=0Tρit−ρreft2,where ρi and ρref represent the position vector of the ith member and the corresponding position vector on the center baseline, respectively.Besides, the signalCost is also defined to represent the quality of distances among new adding members, which is shown as follows:(43)Cost=∑i=2,j∈XrN∫t=0Tmr1rntdt+∑i=2,j∈XaN∫t=0Tmarntdt.In addition, in the scenario of members following a leader, it will be reasonable to care about how many neighbours can each member sense. The more neighbours that a member can sense, the more information can the member obtain, which will be benefit for planning the flight paths. Therefore, the symbolηs is defined as the sensing rate for representing the degree of sensing ability of the members in the cluster:(44)ηs=∑i=2NNumis∑i=2NN−1,where Numis represents the number of neighbours that the ith member can sense.We set the discount factorγ as 0.8, the reward separator ε as 0.2, the learning rate of the critic α as 5×10−4, the learning rate of the actor β as 10−4, the coefficients mr as 1000, and ma as 10−3. Thus, the experimental results under the simulation time T with 1000s are shown below.From Figures14(a) and 14(b), the trajectories under TEPM with k=6 and k=22 are illustrated, respectively. In Figure 14(a), it is seen that satellites No. 2 to No. 4 fly aside from satellite No. 1 in the latter half of flight. The reason why their flight paths deviate from the leader is because that the empirical parameter is set too small that the members cannot sense the leader. It indicates that, in TEPM, if the empirical parameter is too small, some unexpected situations will occur which may result in the failure of the mission. In Figure 14(b), it is seen that the flight paths of the members are nonsmooth, which is not a good flight condition for following the leader. Certainly, this is because of the large value of the selected empirical parameter. Figures 14(a) and 14(b) have further explained the difficulties for selecting empirical parameter. The unsuitable selections may cause bad flying results or even lead to the failure of the mission. Besides, compared with the results from the Figures 14(a) and 14(b), the final training effect of members flight following under RLPST is illustrated in Figure 15. It is seen that satellite No. 1 is the leader in the cluster and satellites No. 2, No. 3 and No. 4 are scattered from each other at initial condition. When the training is finished, satellites No. 2 to No. 4 have the ability to follow the leader successfully and keep a smooth flight path.Figure 14
Trajectories of TEPM with different empirical parameters in the scenario of members following a flight path: (a)k = 6; (b) k = 22.
(a)
(b)Figure 15
Results of final training in the scenario of members following a flight path.From Figure16, it is seen that when the empirical parameter is set from 12 to 16, the sensing rate is equal to one. In such a condition, all members in the cluster can sense other members during the whole flight. However, when the empirical parameter is too small or too large, the sensing rate will not be guaranteed to be one, which means that flight effect may be badly influenced. On the contrary, the solid line represents the sensing rate under the proposed RLPST, which is guaranteed to be one when the learning process is finished. The figure shows the superiority of the proposed RLPST because of the assurance of the optimal sensing rate.Figure 16
Curves of sensing rate under TEPM and RLPST in the scenario of members following a flight path.Figure17 shows the trajectory deviations under the TEPM and the proposed RLPST, respectively. Similar to the condition shown in Figure 11, when the empirical parameter is too small or too large, the deviation is obviously high because of the badly flight results. In the figure, if the parameter is chosen as about 12, it has the lowest value of deviation. However, compared with the value under RLPST, it indicates that the value under RLPST is still smaller than the lowest value under TEPM. Therefore, the RLPST method has the ability to meet the more lower deviation within the safety flight range, which makes the flight path more smooth.Figure 17
Curves of trajectory deviations under TEPM and RLPST in the scenario of members following a flight path.From Figure18, the variation in cost along with the learning times is illustrated. Similar to the curves drawn in Figure 13, the cost is generally decreased with the increase in learning times. When the learning process is finished, the near lowest cost for the mission is found. The figure indicates that the flight condition is gradually improved during the learning process, and the cost of the flight can be effectively reduced through the proposed RLPST.Figure 18
Curve of cost line along with learning times in the scenario of members following a flight path.


## 6. Discussion

In Section5.1, we simulate the scenario of members adding under the TEPM and the proposed RLPST, respectively. The results show that it is difficult to select proper empirical parameters under TEPM. In addition, the trajectory deviations and terminal matching rates under TEPM and RLPST are compared. The trajectory deviation under RLPST is lower than that under TEPM. On the contrary, the value of the terminal matching rate under RLPST is guaranteed to equal to one, while that under TEPM cannot be. Therefore, the superiorities of the proposed RLPST are obviously proved. The variation in the cost along with the learning times shows the flight paths can be gradually improved through the learning framework.In Section5.2, the scenario of members flight path following under the TEPM and the proposed RLPST are simulated, respectively. Apart from the superiorities of RLPST in the aspects of low trajectory deviation and decreasing cost, the sensing rate under RLPST shows the advantage compared with that under the TEPM, which means that RLPST method makes the cluster member be able to obtain more information from neighbours for completing the mission. A steep downward trend in Figure 18 is due to the selection of learning rates. At the final part of learning process, the performance of the cluster becomes sensitive to the learning rates, which is still a challenging problem.The time cost and iteration times for the two simulated scenarios are listed in Table3.Table 3
Time cost and iteration times for the simulated scenarios.
Scenario
Time cost (unit:s)
Iteration times
Adding members into a satellite cluster
131.96
55
Members following a flight path
69.44
68By comparing with the similar studies of [29, 30], it is seen that the time costs and iteration times of the two simulated scenarios are acceptable, which means the proposed RLPST can improve the flight path within a reasonable payment.


## 7. Conclusion

Due to the difficulties of parameter selection under TEPM for satellite cluster flying, a type of parameter-self-tuning method based on the actor-critic algorithm is proposed for handling the problem. Considering the specific condition of satellite cluster, the three sensing zones are redefined and the method for determining the belonging zones of sensing members of each cluster member is presented. To tune the flight parameter in each sensing zone, the fuzzy inference systems are employed to compose the actor and critic parts. With the proper design of reward function, a three-channel learning framework of parameter self-tuning for satellite cluster is designed. Compared with the TEPM, the proposed RLPST algorithm shows the superiorities. The results of simulation experiments indicate that the proposed RLPST has the lower trajectory deviation and guarantees the better terminal matching rate for scenario of members adding as well as the better sensing rate for scenario of members flight path following than the TEPM. Besides, the numerical experimental results also have shown the decrease in the cost along with the learning times in the two scenarios, which proves that the proposed RLPST has the ability to gradually improve the flight paths of the satellite cluster under the learning framework.



---

*Source: 1836159-2020-04-27.xml*