# A Novel Model-Based Reinforcement Learning Attitude Control Method for Virtual Reality Satellite

Authors: Jian Zhang; Fengge Wu
Journal: Wireless Communications and Mobile Computing
 (2021)

---

## Abstract

Observing the universe with virtual reality satellite is an amazing experience. An intelligent method of attitude control is the core object of research to achieve this goal. Attitude control is essentially one of the goal-state reaching tasks under constraints. Using reinforcement learning methods in real-world systems faces many challenges, such as insufficient samples, exploration safety issues, unknown actuator delays, and noise in the raw sensor data. In this work, a mixed model with different input sizes was proposed to represent the environmental dynamics model. The predication accuracy of the environmental dynamics model and the performance of the policy trained in this paper were gradually improved. Our method reduces the impact of noisy data on the model’s accuracy and improves the sampling efficiency. The experiments showed that the agent trained with our method completed a goal-state reaching task in a real-world system under wireless circumstances whose actuators were reaction wheels, whereas the soft actor-critic method failed in the same training process. The method’s effectiveness is ensured theoretically under given conditions.


---

## Body

## 1. Introduction

SpaceVR’s Overview 1 virtual reality satellites enable us to experience space firsthand using any mobile, desktop, or virtual reality device. Observing the universe with virtual reality is an exciting application of satellites. As one of the core subsystems in satellite, the attitude control system needs be developed to meet the requirement for space virtual reality applications. An intelligent method of attitude control plays a role part in the subsystem. Reinforcement learning gives a remarkable achievement to prove potentiality to realize intelligent control.In recent years, a significant progress has been made in the research of reinforcement learning (RL) [1]. Researchers have proposed many RL methods, such as the deep deterministic policy gradient (DDPG), trust region policy optimization (TRPO), proximal policy optimization (PPO), soft actor-critic (SAC), and twin-delayed deep deterministic policy gradient (TD3) methods [2–6]. These methods have been applied for playing Atari games [7] completing control tasks in OpenAI Gym [8] and defeating human players in Go [9]. However, compared with simulation environments, successful cases in real-world systems are relatively rare. Application of RL methods directly in real-world systems faces a series of problems, such as sample limitations, exploration of the state space leading to system damage, unspecified reward functions, unknown delays in the system actuators, and unstable measurements from sensors [10].In a real-world system, goal-state reaching is a task in which the control actuators adjust the system to reach a specified goal state. For example, attitude control of a satellite, including attitude maneuvering and attitude stabilization, can be defined as a task to reach a goal attitude state. The control tasks, such as balancing a car or an inverted pendulum, are also goal-state reaching problems in which the balancing point is specified as the goal state.Approaches to tackle the challenges of real-world systems without expert knowledge can be roughly divided into two categories: “simulation to real” and “real experience data to model to real.” “Simulation to real” methods build a simulation environment based on the real-world system and use RL methods for training. After training, the policies obtained from the simulation environment are applied in the real-world system. A number of previous studies have examined this type of method to learn robotic skills in real-world systems, including locomotion [11], robotic control [12], and vision-based autonomous flight [13].“Real experience data to model to real” methods learn the model of the environment dynamics based on experience data from real-world systems and interact with a learned dynamics model using RL methods, and then, they apply the learned policy in a real-world system. Nagabandi et al. [14] built an environment dynamics model with neural networks, used a model predictive controller to simulate the transition trajectory in the learned environment model, and selected the best actions for the real-world system, thus realizing four-legged robot motion along a specified path. Lambert et al. [15] built an environment model with real-world system data, simulated the transition trajectory in the environment model running on multiple computing nodes, obtained the best action, and realized the attitude stability of a four-wing aircraft with low-level control, where the input was sensor data and the output was a pulse-width modulation signal.The “simulation to real” methods depend on knowledge of the real-world system, the accuracy of the simulator, and the effectiveness of shifting the experience distribution between the simulator and the real-world system. The existing “real experience data to model to real” methods face two major challenges. First, sufficient experience data must be collected initially to model the environment dynamics. Second, the state space visited by the untrained policy without expert knowledge is insufficient for modelling the environment dynamics. In this paper, we present a method that overcomes the shortcomings of both classes of methods. Our method iterates in the loop of “collecting data using the latest policy-learning environment dynamics model-learning policy.” By using the latest policy to collect data from the real environment, it can gradually improve the model’s accuracy regarding the dynamics and policy’s performance without building a simulator. Although SAC was used for learning the latest policy, the difference from the original SAC method is that when exploring the unknown state space, we use the verified policy in the learned dynamics model, thus making the exploration more effective and safer.Compared with the “simulation to real” methods, our method does not require knowledge of the real-world system. Existing “real experience data to model to real” method collects training samples at one time. The policy used to collect data should cover all the state space to support modelling the environment dynamics precise. But it is impossible. The difference between our method and existing “real experience data to model to real” is that the data used to train the model in our method is collected gradually, which reduces the dependence on the initial random data and thus reduces the probability of falling into a state space that will lead to disasters of the system. At the same time, it can cover a wider state space, which makes the dynamics model more precise.The proposed method was tested using Cubli, a reaction-wheel-based inverted pendulum system. Cubli [16] is a cube whose height, width, and depth are 15 cm. It is equipped with three reaction wheels in three different planes. It is a typical nonlinear, unstable, and multidegree of freedom control system. The three reaction wheels are driven by motor rotation to generate torque to achieve attitude adjustment. Research on Cubli’s control is mainly based on classical methods of control, such as proportional–integral–derivative (PID), linear–quadratic regulator (LQR), and nonlinear control theory [16–19]. These controllers operate based on analytical models for Cubli’s dynamics, manual parameter tuning, and long design times. In our paper, we address the challenge of generating controllers without system knowledge in an end-to-end learning method.The main work of this paper includes the following: (1) A learning system for Cubli is constructed in Section2.1. The policy learned with our method achieves motor control from raw sensor data on board to drive the Cubli reaction wheels. The required torque produced by changing the spinning speeds of the reaction wheels leads the system to reach the goal state. (2) To solve the problems of an unknown delay of the actuator and inaccurate state information from the sensor data, a mixed environment dynamics model whose input window size varies is proposed (Section 2.3). (3) A method that can be used to learn the environment dynamics and policy based on the SAC method is presented. The convergence of the method was ensured theoretically (Section 2.5), and the effectiveness of the method was verified by experiments (Section 3).


## 2. Methodology

### 2.1. Learning System Construction

The state is generally composed of Euler angle and angular velocity around axis in a real-world system. Especially, in a goal-reaching system, the state is given by a measurement unit on board. The policy to learn outputs the control signal to the execution unit which is regarded as an action in a given state.This section mainly describes how the learning system was built. As shown in Figure1, the learning system was composed of four parts. (1) A real-world system called Cubli was used to collect state-action pair trajectory data. (2) The execution node executed the latest policy and interacted with Cubli via Bluetooth. (3) Real trajectory data was used to train an environmental dynamics model represented by a neural network. (4) In the environment modelled by the trained neural network, SAC method was used to train agents to obtain the latest policy. The latest policy was verified at the execution node.Figure 1
The overall structure of the learning system.At any time stept, Cubli obtained the state from an onboard sensor and submitted the state st to the execution node. The execution node used the latest policy neural network for inference and obtained the current action at. After Cubli executed the action, the execution node collected the next state data st+1, judged whether the current trajectory had ended through the feedback state data, and saved the trajectory into the trajectory buffer. The trajectory buffer was used to train the dynamics model Pst+1∣st,at.The training process is described in detail in Section2.4. The training of the agent was carried out through the interaction with the environment represented by the trained model. At time step t, st and at are submitted to generate st+1 by the model of the environment dynamics, and the reward signal is given based on st+1. In this study, the agent was trained by the SAC method, and the gradient of the value network and the policy network parameters estimated with batch samples were used to update the parameters. The execution node, the training of the agent, and the model of the environment dynamics were run on the upper machine. The entire learning process could operate automatically and independently, except that manual intervention was required in cases of Cubli power exhaustion or abnormalities. Although the new trajectory stored in the buffer was paused, the training of the agent did not stop. The tasks of training can be distributed between multiple machines. At the same time, the learning system can be connected with multiple systems to provide more trajectories and allow the dynamics model, the agent’s value network, and policy network’s parameters to converge faster.


### 2.2. Definition of Goal-State Reaching under Reinforcement Learning Paradigm

Reinforcement learning is a machine learning paradigm that maximizes expected cumulative return. The problem to be solved is defined as a Markov process. A Markov process is defined by the five-tuple<S,A,r,P,γ>, where S is the state space, A is the action space, and r is the reward function. In the problem of goal-state reaching, the reward function is often related to the current state stand the goal state sgoal. It can be written as rst,sgoal. P is the state transition probability function. P:S×A⟶S is a function that maps the state-action pair space to the state space, which we also call it in this paper the environmental dynamics model as Pst+1∣st,at. γ is the discount factor used to calculate the discount cumulative return. The policy function is represented byπ:S⟶A, which is a function that maps the state space to the action space. The agent uses the policy function to obtain the current action according to the current state at=πst and interacts with the environment to get the next state st+1. This process continuously loops to form a state-action trajectoryτ=<s1,a1,⋯,sT>. When using a neural network to represent the policy function, π is determined by the neural network architecture and the parameterθ, so it can be written as πθ, and the policy network training is to adjust the parameter θ. The process can be formalized as
(1)argmaxθEτ~πθ∑tγt·rst,sgoalwhererst,sgoal=−st−sgoal.The SAC is an RL method based on maximum entropy, which emphasizes the policy’s randomness when exploring an unknown state space. While maximizing the cumulative discount return, SAC prevents the parameters of the policy network from converging to the local optimum. The following equations describe the loss functions that need to be optimized:(2)JQϕi=Es,a,r,s′~DQϕis,a−r+γminj=1,2Qtarget,js′,a′−αlogπθa′∣s′,(3)Jπθ=Es~D,a~παlogπθa∣s−mini=1,2Qϕis,a,(4)Jα=Es~D,a~π−αlogπθa∣s−αH,whereH=−Dimensionsofactionspace.SAC concurrently learns a policyπθ and two Q-functionsQϕ1,Qϕ2. Q-function Qπs,a describes the cumulative reward following policy π after executing action a from current state s. Equation (2) is the loss function of the Q-function, which is also known as the mean squared bellman error (MSBE), Equation (3) is the loss function of the policy function that can be optimized for improving the policy’s Q value, and Equation (4) is the loss function of the temperature parameter α, which defines the explore-exploit tradeoff, with a higher α corresponding to more exploration, and a lower α corresponding to more exploitation. The SAC takes the minimum value of two sets of independent Q-function parameters to solve the problem of overestimating the Q-function, and it uses the target Q value network parameter to solve the training instability problem caused by using the same set of Q value network parameters to bootstrap. These techniques were first proposed by van Hasselt et al. [20]. The optimization of the temperature parameter α is used to control the trade-off between the exploration and the utilization and to ensure the stability of the policy in the later stage of training.


### 2.3. Learning the Environmental Dynamics Model

Because of the noise in the onboard sensor data, only using the state-action pair at one timestep as the input of the model will affect the accuracy of the environmental dynamics model. We propose a model that predicts the next state by combining state-action pairs of different sizes as the input. As shown in Figure2, for a state-action pair trajectory, the state at a certain moment is assumed to be St+3, which is marked by blue boxes with solid lines. The environmental dynamics model needs to predict the state at this moment with the historical state-action pairs. The red boxes with dashed lines in the figure represent the input, and we define the number of state-action pairs as the size of the input window. The sizes of the input windows in the first, second, and third parts in Figure 2 are 1, 2, and 3, respectively. A single model is represented by a fully connected neural network. Each model uses data with different sizes of the input windows as inputs to predict the state change value △st−1 for the next state, and the output of the K models are averaged as the actual predicted state change value. The parameters of the ith model are defined as φi, and a single-environment dynamic model is expressed as
(5)Pφist∣st−i,at−i,⋯,st−1,at−1+st−1.Figure 2
Schematic diagram of models with different input window sizes.When the size of the input window isK, a state-action pair trajectory of length N can construct N−K+1 “prediction-truth” pairs represented by <St′,St> where St′=1/K∑i=1KPφi·∣st−i,at−i,⋯,st−1,at−1+St−1. The training of the model is minimizing the mean square error between the prediction and the truth. The process is formalized as
(6)argminφiE<st−i,at−i,⋯,st−1,at−1,st>~D1K∑i=1KPφi·∣st−i,at−i,⋯,st−1,at−1+st−1−st2.


### 2.4. Training Environment Construction

In this section, we describe the construction of the learning environment. The state consisted of Euler angles (roll, yaw, and pitch) and the three-axis angular velocities. The state was directly measured by the MPU-6050 inertial measurement unit. Action outputs the pulse-width modulation to control the rotation of the motor.A statest=ϕ,θ,φ,ẋ,ẏ,żT is a six-dimensional vector, where ϕ,θ,andφ represent roll, yaw, and pitch, and ẋ,ẏ,andż represent the angular velocity of the rotations around the x-, y-, and z-axes, respectively. An action at=a1,a2,a3T is a three-dimensional vector, where a1,a2,anda3 represent the values of the pulse-width modulation (PWM) to control the rotational speeds of the motors.To standardize the state value, we replaced the Euler angles with their cosine and sine values. For the axis angular velocity, the measured data was obtained from the IMU and divided by the maximum value of each measurement, and each state component was limited to the value range of−1,1. Therefore, the actual state is expressed as follows:
(7)st=cosφ,sinφ,cosθ,sinθ,cosϕ,sinϕ,xMAX_Ẋ,yMAX_Ẏ,zMAX_ŻT.The control task of balancing Cubli on an edge is defined as follows. When the Cubli flips, the error from the balance point isϵ, and only the IMU measurement data and PWM signal of the control motor are used to maintain balance on the edge. Based on the position of the momentum wheel, three edges that can achieve the balance task are determined. Euler angles (roll, yaw, and pitch) can be used to describe the distance from the balance point. As described above, the reward function of the balance task is given as follows:
(8)rst,sgoal=−ϕ2+0.1∗xMAX_Ẋ21/2balancearoundXaxis,−θ2+0.1∗yMAX_Ẏ21/2balancearoundYaxis,−φ2+0.1∗zMAX_Ż21/2balancearoundZaxis.


### 2.5. Theoretical Analysis

Theory 1: when the reward function satisfiesrs,sgoal=−s−sgoal, the value function Vπk+1,P∗ of the policy πk+1acts on the real environment model P∗ will be greater than or equal to the value function Vπk,P∗ while the policy πk+1is obtained from πk+1,Pk+1=argmaxπ,PVπ,P−DTV,πkP,P∗Condition1 in the k+1th iteration, where
(9)Vπk+1,P∗≥Vπk,P∗.P∗ is the dynamics model in real environment, Pk is the dynamics model learned in the kth iteration, and DTV,πkP,P∗ is the absolute value of the maximum state difference between the experience trajectory collected in the real environment P∗ and the learned environment dynamics model P following the policy πk. From the definition, DTV,πkP∗,P∗ is zero. Vπk+1,Pk+1 is the value function of πk+1acting on the dynamics model Pk+1 learned in the k+1 iteration, and Vπk+1,P∗ is the value function of πk+1 acting on the real environment P∗. ρtπ,P is the probability density function of the state distribution at time t when the policy π is used to interact with the dynamics model P. st^ is the state at step t when acting on the learned dynamics, and s∗ is the goal state.Proof.
(10)Vπk+1,P∗−Vπk+1,Pk+1=Est~ρtπk+1,P∗,st^~ρtπk+1,Pk+1−∑t=0∞γtst−s∗−st^−s∗≥Est~ρtπk+1,P∗,st^~ρtπk+1,Pk+1−∑t=0∞γtst−st^≥−Est~ρtπk+1,P∗,st^~ρtπk+1,Pk+1∑t=0∞γtmaxtst−st^=−DTV,πk+1P∗,Pk+1≥−DTV,πkP∗,Pk+1inequality1.
From inequality 1,(11)Vπk+1,P∗≥Vπk+1,Pk+1−DTV,πkP∗,Pk+1≥Vπk,P∗−DTV,πkP∗,P∗Accordingtocondition1≥Vπk,P∗becauseDTV,πkP∗,P∗=0.☐Therefore, according to the steps described in Theory 1, the policy sequenceπ0,π1,⋯,πk with a monotonically increasing value function will be generated such that Vπk,P∗≥⋯≥Vπ1,P∗≥Vπ0,P∗.Algorithm 1: Learning algorithm with hybrid window’s length dynamic model.
Input: initialize three input length environment dynamic model parametersφ1,φ2,φ3.Initialize policy parametersθ, Q-function parameters: ∅1,∅2Empty replay bufferD, empty trajectory buffer T.Set targetQ-function parameters: ∅target,1⟵∅1,∅target,2⟵∅21: repeat2: Collect state-action trajectoryτ=s0,a0,s1,a1,⋯,sT, store trajectory into replay buffer T.3: Randomly sample a batchB1 of trajectory from replay buffer T, construct B1∗N−2 (prediction, truth) pair training samples.4: Update dynamic model by gradient decent using∇φi1/B1∗N−2∑st−3,at−3,⋯,st−1,st∈B11/3∑i=13Pφi∙st−i,at−i,⋯,st−i,at−1+st−1−st25:until convergence6:repeat7:fori in range(three dynamic models) do8:  Based on current statest−1 and sample action at−1~πθ∙s9:  calculate next states′ with executing action in dynamic models Pφi with previous state-action pair10:end for11: Record latest actiona, observe latest two state s,s′, reward r, and done signal d12: Stores,a,s′,d into replay buffer D13: if done signald is True then14:  Reset environment state.15:end if16:forj in range(update times) do17:  Randomly sample a batchB2 transitions from replay buffer D18:  UpdateQ-functions by gradient decent using∇∅i1/B2∑s,a,r,s′∈DQ∅is,a−r+λminj=1,2Qtarget,js′,a′−αlogπθa′s′19:  Update policy network by gradient decent using∇θ1/B2∑s∈Bαlogπθas−mini=1,2Q∅is,a20:  Update targetQ-function networks withQtarget,j⟵ρ∅target,j+1−ρ∅j,forj=1,221:end for22: if it is time to test in real systemthen23:  Run latest policy in real system, collect state-action trajectory, collect state-action trajectoryτ=s0,a0,s1,a1,⋯,sT, store trajectory into replay buffer T.24:  Observe the result running in real system25:  if task is completedthen26:completed⟵true27:  else28:   Update dynamic model as step 4.29:end if30:end if31: until completedThe procedure is presented in Algorithm1. The environment dynamics model represented by mixed models with different sizes of the input window is trained to convergence using trajectory data from the real environment. The agent is trained by the SAC. The update rule of the parameters in the policy function’s network is given by Equation (3), and the value function is given by Equation (2). During the application of the algorithm, Condition 1, which considers all the policy and environment dynamics models, is not satisfied. In our algorithm, we assume that the model represented by the neural network architecture includes the real environment dynamics model and that the error between the learned model trained with more real data and the real environment model is decreasing.


## 2.1. Learning System Construction

The state is generally composed of Euler angle and angular velocity around axis in a real-world system. Especially, in a goal-reaching system, the state is given by a measurement unit on board. The policy to learn outputs the control signal to the execution unit which is regarded as an action in a given state.This section mainly describes how the learning system was built. As shown in Figure1, the learning system was composed of four parts. (1) A real-world system called Cubli was used to collect state-action pair trajectory data. (2) The execution node executed the latest policy and interacted with Cubli via Bluetooth. (3) Real trajectory data was used to train an environmental dynamics model represented by a neural network. (4) In the environment modelled by the trained neural network, SAC method was used to train agents to obtain the latest policy. The latest policy was verified at the execution node.Figure 1
The overall structure of the learning system.At any time stept, Cubli obtained the state from an onboard sensor and submitted the state st to the execution node. The execution node used the latest policy neural network for inference and obtained the current action at. After Cubli executed the action, the execution node collected the next state data st+1, judged whether the current trajectory had ended through the feedback state data, and saved the trajectory into the trajectory buffer. The trajectory buffer was used to train the dynamics model Pst+1∣st,at.The training process is described in detail in Section2.4. The training of the agent was carried out through the interaction with the environment represented by the trained model. At time step t, st and at are submitted to generate st+1 by the model of the environment dynamics, and the reward signal is given based on st+1. In this study, the agent was trained by the SAC method, and the gradient of the value network and the policy network parameters estimated with batch samples were used to update the parameters. The execution node, the training of the agent, and the model of the environment dynamics were run on the upper machine. The entire learning process could operate automatically and independently, except that manual intervention was required in cases of Cubli power exhaustion or abnormalities. Although the new trajectory stored in the buffer was paused, the training of the agent did not stop. The tasks of training can be distributed between multiple machines. At the same time, the learning system can be connected with multiple systems to provide more trajectories and allow the dynamics model, the agent’s value network, and policy network’s parameters to converge faster.


## 2.2. Definition of Goal-State Reaching under Reinforcement Learning Paradigm

Reinforcement learning is a machine learning paradigm that maximizes expected cumulative return. The problem to be solved is defined as a Markov process. A Markov process is defined by the five-tuple<S,A,r,P,γ>, where S is the state space, A is the action space, and r is the reward function. In the problem of goal-state reaching, the reward function is often related to the current state stand the goal state sgoal. It can be written as rst,sgoal. P is the state transition probability function. P:S×A⟶S is a function that maps the state-action pair space to the state space, which we also call it in this paper the environmental dynamics model as Pst+1∣st,at. γ is the discount factor used to calculate the discount cumulative return. The policy function is represented byπ:S⟶A, which is a function that maps the state space to the action space. The agent uses the policy function to obtain the current action according to the current state at=πst and interacts with the environment to get the next state st+1. This process continuously loops to form a state-action trajectoryτ=<s1,a1,⋯,sT>. When using a neural network to represent the policy function, π is determined by the neural network architecture and the parameterθ, so it can be written as πθ, and the policy network training is to adjust the parameter θ. The process can be formalized as
(1)argmaxθEτ~πθ∑tγt·rst,sgoalwhererst,sgoal=−st−sgoal.The SAC is an RL method based on maximum entropy, which emphasizes the policy’s randomness when exploring an unknown state space. While maximizing the cumulative discount return, SAC prevents the parameters of the policy network from converging to the local optimum. The following equations describe the loss functions that need to be optimized:(2)JQϕi=Es,a,r,s′~DQϕis,a−r+γminj=1,2Qtarget,js′,a′−αlogπθa′∣s′,(3)Jπθ=Es~D,a~παlogπθa∣s−mini=1,2Qϕis,a,(4)Jα=Es~D,a~π−αlogπθa∣s−αH,whereH=−Dimensionsofactionspace.SAC concurrently learns a policyπθ and two Q-functionsQϕ1,Qϕ2. Q-function Qπs,a describes the cumulative reward following policy π after executing action a from current state s. Equation (2) is the loss function of the Q-function, which is also known as the mean squared bellman error (MSBE), Equation (3) is the loss function of the policy function that can be optimized for improving the policy’s Q value, and Equation (4) is the loss function of the temperature parameter α, which defines the explore-exploit tradeoff, with a higher α corresponding to more exploration, and a lower α corresponding to more exploitation. The SAC takes the minimum value of two sets of independent Q-function parameters to solve the problem of overestimating the Q-function, and it uses the target Q value network parameter to solve the training instability problem caused by using the same set of Q value network parameters to bootstrap. These techniques were first proposed by van Hasselt et al. [20]. The optimization of the temperature parameter α is used to control the trade-off between the exploration and the utilization and to ensure the stability of the policy in the later stage of training.


## 2.3. Learning the Environmental Dynamics Model

Because of the noise in the onboard sensor data, only using the state-action pair at one timestep as the input of the model will affect the accuracy of the environmental dynamics model. We propose a model that predicts the next state by combining state-action pairs of different sizes as the input. As shown in Figure2, for a state-action pair trajectory, the state at a certain moment is assumed to be St+3, which is marked by blue boxes with solid lines. The environmental dynamics model needs to predict the state at this moment with the historical state-action pairs. The red boxes with dashed lines in the figure represent the input, and we define the number of state-action pairs as the size of the input window. The sizes of the input windows in the first, second, and third parts in Figure 2 are 1, 2, and 3, respectively. A single model is represented by a fully connected neural network. Each model uses data with different sizes of the input windows as inputs to predict the state change value △st−1 for the next state, and the output of the K models are averaged as the actual predicted state change value. The parameters of the ith model are defined as φi, and a single-environment dynamic model is expressed as
(5)Pφist∣st−i,at−i,⋯,st−1,at−1+st−1.Figure 2
Schematic diagram of models with different input window sizes.When the size of the input window isK, a state-action pair trajectory of length N can construct N−K+1 “prediction-truth” pairs represented by <St′,St> where St′=1/K∑i=1KPφi·∣st−i,at−i,⋯,st−1,at−1+St−1. The training of the model is minimizing the mean square error between the prediction and the truth. The process is formalized as
(6)argminφiE<st−i,at−i,⋯,st−1,at−1,st>~D1K∑i=1KPφi·∣st−i,at−i,⋯,st−1,at−1+st−1−st2.


## 2.4. Training Environment Construction

In this section, we describe the construction of the learning environment. The state consisted of Euler angles (roll, yaw, and pitch) and the three-axis angular velocities. The state was directly measured by the MPU-6050 inertial measurement unit. Action outputs the pulse-width modulation to control the rotation of the motor.A statest=ϕ,θ,φ,ẋ,ẏ,żT is a six-dimensional vector, where ϕ,θ,andφ represent roll, yaw, and pitch, and ẋ,ẏ,andż represent the angular velocity of the rotations around the x-, y-, and z-axes, respectively. An action at=a1,a2,a3T is a three-dimensional vector, where a1,a2,anda3 represent the values of the pulse-width modulation (PWM) to control the rotational speeds of the motors.To standardize the state value, we replaced the Euler angles with their cosine and sine values. For the axis angular velocity, the measured data was obtained from the IMU and divided by the maximum value of each measurement, and each state component was limited to the value range of−1,1. Therefore, the actual state is expressed as follows:
(7)st=cosφ,sinφ,cosθ,sinθ,cosϕ,sinϕ,xMAX_Ẋ,yMAX_Ẏ,zMAX_ŻT.The control task of balancing Cubli on an edge is defined as follows. When the Cubli flips, the error from the balance point isϵ, and only the IMU measurement data and PWM signal of the control motor are used to maintain balance on the edge. Based on the position of the momentum wheel, three edges that can achieve the balance task are determined. Euler angles (roll, yaw, and pitch) can be used to describe the distance from the balance point. As described above, the reward function of the balance task is given as follows:
(8)rst,sgoal=−ϕ2+0.1∗xMAX_Ẋ21/2balancearoundXaxis,−θ2+0.1∗yMAX_Ẏ21/2balancearoundYaxis,−φ2+0.1∗zMAX_Ż21/2balancearoundZaxis.


## 2.5. Theoretical Analysis

Theory 1: when the reward function satisfiesrs,sgoal=−s−sgoal, the value function Vπk+1,P∗ of the policy πk+1acts on the real environment model P∗ will be greater than or equal to the value function Vπk,P∗ while the policy πk+1is obtained from πk+1,Pk+1=argmaxπ,PVπ,P−DTV,πkP,P∗Condition1 in the k+1th iteration, where
(9)Vπk+1,P∗≥Vπk,P∗.P∗ is the dynamics model in real environment, Pk is the dynamics model learned in the kth iteration, and DTV,πkP,P∗ is the absolute value of the maximum state difference between the experience trajectory collected in the real environment P∗ and the learned environment dynamics model P following the policy πk. From the definition, DTV,πkP∗,P∗ is zero. Vπk+1,Pk+1 is the value function of πk+1acting on the dynamics model Pk+1 learned in the k+1 iteration, and Vπk+1,P∗ is the value function of πk+1 acting on the real environment P∗. ρtπ,P is the probability density function of the state distribution at time t when the policy π is used to interact with the dynamics model P. st^ is the state at step t when acting on the learned dynamics, and s∗ is the goal state.Proof.
(10)Vπk+1,P∗−Vπk+1,Pk+1=Est~ρtπk+1,P∗,st^~ρtπk+1,Pk+1−∑t=0∞γtst−s∗−st^−s∗≥Est~ρtπk+1,P∗,st^~ρtπk+1,Pk+1−∑t=0∞γtst−st^≥−Est~ρtπk+1,P∗,st^~ρtπk+1,Pk+1∑t=0∞γtmaxtst−st^=−DTV,πk+1P∗,Pk+1≥−DTV,πkP∗,Pk+1inequality1.
From inequality 1,(11)Vπk+1,P∗≥Vπk+1,Pk+1−DTV,πkP∗,Pk+1≥Vπk,P∗−DTV,πkP∗,P∗Accordingtocondition1≥Vπk,P∗becauseDTV,πkP∗,P∗=0.☐Therefore, according to the steps described in Theory 1, the policy sequenceπ0,π1,⋯,πk with a monotonically increasing value function will be generated such that Vπk,P∗≥⋯≥Vπ1,P∗≥Vπ0,P∗.Algorithm 1: Learning algorithm with hybrid window’s length dynamic model.
Input: initialize three input length environment dynamic model parametersφ1,φ2,φ3.Initialize policy parametersθ, Q-function parameters: ∅1,∅2Empty replay bufferD, empty trajectory buffer T.Set targetQ-function parameters: ∅target,1⟵∅1,∅target,2⟵∅21: repeat2: Collect state-action trajectoryτ=s0,a0,s1,a1,⋯,sT, store trajectory into replay buffer T.3: Randomly sample a batchB1 of trajectory from replay buffer T, construct B1∗N−2 (prediction, truth) pair training samples.4: Update dynamic model by gradient decent using∇φi1/B1∗N−2∑st−3,at−3,⋯,st−1,st∈B11/3∑i=13Pφi∙st−i,at−i,⋯,st−i,at−1+st−1−st25:until convergence6:repeat7:fori in range(three dynamic models) do8:  Based on current statest−1 and sample action at−1~πθ∙s9:  calculate next states′ with executing action in dynamic models Pφi with previous state-action pair10:end for11: Record latest actiona, observe latest two state s,s′, reward r, and done signal d12: Stores,a,s′,d into replay buffer D13: if done signald is True then14:  Reset environment state.15:end if16:forj in range(update times) do17:  Randomly sample a batchB2 transitions from replay buffer D18:  UpdateQ-functions by gradient decent using∇∅i1/B2∑s,a,r,s′∈DQ∅is,a−r+λminj=1,2Qtarget,js′,a′−αlogπθa′s′19:  Update policy network by gradient decent using∇θ1/B2∑s∈Bαlogπθas−mini=1,2Q∅is,a20:  Update targetQ-function networks withQtarget,j⟵ρ∅target,j+1−ρ∅j,forj=1,221:end for22: if it is time to test in real systemthen23:  Run latest policy in real system, collect state-action trajectory, collect state-action trajectoryτ=s0,a0,s1,a1,⋯,sT, store trajectory into replay buffer T.24:  Observe the result running in real system25:  if task is completedthen26:completed⟵true27:  else28:   Update dynamic model as step 4.29:end if30:end if31: until completedThe procedure is presented in Algorithm1. The environment dynamics model represented by mixed models with different sizes of the input window is trained to convergence using trajectory data from the real environment. The agent is trained by the SAC. The update rule of the parameters in the policy function’s network is given by Equation (3), and the value function is given by Equation (2). During the application of the algorithm, Condition 1, which considers all the policy and environment dynamics models, is not satisfied. In our algorithm, we assume that the model represented by the neural network architecture includes the real environment dynamics model and that the error between the learned model trained with more real data and the real environment model is decreasing.


## 3. Results and Discussion

A Cubli is shown as Figure3. The yellow area marks the brake components, the blue area marks the control unit and the onboard measurement unit, and the green area marks 3 sets of motors. It is mainly composed of three brushless motors with encoders, three sets of brake devices, an MPU6050 digital motion processor, and a main control chip STM32F103RCT6. The communication with the host computer depends on the Bluetooth module. The communication protocol between Cubli and the upper computer consisted of seven floating-point numerical data (roll; pitch; yaw; angular velocities of rotation around the x-, y-, and z-axes; current motor speed code value; and current motor PWM value). The data were read from the DMP register in the MPU6050. The upper computer sent the target PWM value of the motor to the STM32F103RCT6 on Cubli, and the instruction was executed by the motor after it was written into the register. The frequency of setting the PWM value was 62.5 Hz, and the frequency of obtaining the onboard data was also 62.5 Hz, i.e., the period of the PWM value and obtaining onboard data were both 16 ms. The control loop is shown as Figure 4.Figure 3
Cubli and its structure.Figure 4
The control loop of Cubli.A single model of the environmental dynamics was represented by a fully connected neural network. The neural network had two hidden layers, each with 256 neurons, and it used random values to initialize each parameter. In the experiment, we used input windows with different sizes to test the actual effect. The neural network architecture, initialization parameters, learning method, batch size, and learning rate were fixed, and it was ensured that the experimental training conditions were as constant as possible. Table1 shows the specific conditions. At the beginning of the experiment, a random policy was used to collect 200 state-action pair trajectories at the execution node by interacting with the real environment, corresponding to about 40,000 state-action-state transition sample data. These data were used to train the three models. As shown in Figure 5, the collection of these training data began from the initialization state.Table 1
Parameters used in training the environmental dynamics model.
The environmental dynamics model with different sizes of the input windowSize of input window: 1Size of input window: 2Size of input window: 3Size of input layer unit1212+2412+24+36Size of output layer unit999Initialization of parametersRandomly generated from truncated normal distributionRandomly generated from truncated normal distributionRandomly generated from truncated normal distributionSize of hidden unit256256256Size of hidden layers2<2,2><2,2,2>Learning method/learning rateAdam/0.003Adam/0.003Adam/0.003Batch size256256256Figure 5
Balance task on edge.As shown in Figure6, using these 40,000 samples to train the environment dynamics model, we randomly initialized the parameters of each model and trained each model multiple times to evaluate the loss. After 50 times training with batch size of 256, all three models converged.Figure 6
Training of different models (the shaded regions represent min-max loss at the current step).Theoretically, the larger the input window length, the more mixed models, and the higher the prediction accuracy of the final environmental dynamics model. However, as the size of the input window and the number of mixed models increase, the number of final model parameters will increase, and the training and inference time will be extended.There was a trade-off between the size of the input window and the training time, and we found through experiments that the accuracy of the model and the requirements of the task could be satisfied when the size of input window was 3.Algorithm1 was used to train the agent, where the SAC was used as the controller training method to interact with the model of the environment dynamics. The agent’s policy and value functions were represented by a fully connected neural network, and it had two hidden layers, each with 256 neurons. To compare the model of the environment dynamics and verify the effectiveness of the method proposed in this paper, for each model, the same method was used to train the agent with the same setting for 100,000 steps, and the trained controller was used for verification in the real-world system. Each controller interacted with the environment for 100 epochs, and the results are shown in Figure 7. The controller trained using the model when the size of the input window was 1 was unable to complete the task in the real environment. The controller trained with the environment dynamics model with a window length of 2 could complete the task in some initial states of the real environment, but it still encountered initial states for which it could not complete the task. According to the method proposed in this paper, the controller trained using the model of the environment dynamics with an input window length of 3 could complete the task in the random initial state and quickly stabilize at the target point.Figure 7
The verification effect of the controller in the real environment (training: 100,000 steps in the model of the environment dynamics with different sizes of input window. The shaded regions represent the min-max reward at the current step).In addition, the method described in this paper was compared with the SAC. After 140,000 training steps, the controllers trained with our method and the SAC were used in a real environment. The PWM value output by the two controllers is shown in Figure8. The controller trained by our method completed the task and kept the PWM value around 600 in 15 steps, but the controller trained by the SAC failed, and the motor remained in a saturated state with a PWM value of 7200.Figure 8
(a) Pulse-width modulation (PWM) value output by trained controllers; (b) roll value when running trained controllers; and (c) gyro value for roll when running trained controllers (note: SAC: soft actor-critic).
(a)(b)(c)


## 4. Conclusion

As a promising method, reinforcement learning could be used to realize intelligent attitude control for virtual reality satellites in which the method of attitude control needs to have sufficient adaptability for meeting the various creative virtual reality satellite applications. To cope with the challenges of a high sample data acquisition cost and noise of the raw sensor data when using RL methods in real-world systems, a new method to learn the model of the environment dynamics and train the controller was developed in this study. Compared with the two main current methods, our method does not require knowledge of the system and has high coverage of the state space. Our method was verified to learn to reach the goal-state in a real-world system, Cubli, without prior knowledge. In our work, we found that the mixed model describing the environment model with different sizes of the input was effective for reducing the impact of noisy data on the model’s prediction accuracy. In our experiments, the models representing the environment dynamics with different architectures were trained to converge with the same samples, and the controllers were trained in environments with different models. Based on the results, the effectiveness of the model was verified. We found that our method of training based on the environmental dynamics and the policy gradually achieved better performance than the method of collecting data at a single time to train with the same number of training samples.The Cubli, whose actuators are reaction wheels, is a simple prototype of an attitude controller subsystem in a satellite. To the best of our knowledge, the model-based RL method proposed in this paper was applied to balance Cubli on its edge for the first time. Without prior knowledge, we learned the control policy of the PWM signal whose input was raw sensor data. Our method is an end-to-end learning method.Nevertheless, our work has limitations, which we will address in future work. (1) For the task of balancing on an edge, the effective action for Cubli is one-dimensional. In future work, we plan to expand the action space to three dimensions representing the PWMs of the three motors that control the spinning speed of three reaction wheels simultaneously. (2) The task of reaching the goal state in this work was to balance on an edge based on the mechanical abilities of Cubli. As such, in future work, we will enhance the mechanical ability to balance Cubli on a single corner and verify our methods on more complex tasks.



---

*Source: 7331894-2021-07-01.xml*