# Fuzzy Neural Network Q-Learning Method for Model Disturbance Change: A Deployable Antenna Panel Application

Authors: Zhiyong Liu; Hong Bao; Song Xue; Jingli Du
Journal: International Journal of Aerospace Engineering
 (2019)

---

## Abstract

This paper addresses the disturbance change control problem with an active deformation adjustment mechanism on a 5-meter deployable antenna panel. A fuzzy neural network Q-learning control (FNNQL) strategy is proposed in this paper for the disturbance change to improve the accuracy of the antenna panel. In the proposed method, the error of the model disturbance is reduced by introducing the fuzzy radial basis function (RBF) neural network into Q-learning, and the parameters of the fuzzy RBF neural network were optimized and adjusted by a Q-learning method. This allows the FNNQL controller to have a strong adaptability to deal with the disturbance change. Finally, the proposed method has been adopted in the middle plate of a 5-meter deployable antenna panel, and it was found that the method could successfully adapt the model disturbance change in the antenna panel. Results of the simulation also show that the whole control system meets the required accuracy requirements.


---

## Body

## 1. Introduction

To achieve long-term continuous meteorological observations, a 5-meter deployable antenna has been arranged in the geosynchronous orbit [1, 2]. The 5-meter deployable antenna panel can be divided into five units, which are the upper left plate, the upper right plate, the lower left plate, the lower right plate, and the retainer plate, as shown in Figure 1. An active adjustment system was adopted to meet the observation requirements of the 5-meter deployable antenna, as shown in Figure 2.
(1)
Each of the reflecting panels was supported by more than one active actuators, and the control model of the active adjustment system is a typical multiple-input multiple-output (MIMO) system
(2)
The strong coupling effects are caused because the active actuator interacts with each other during the adjustment processFigure 1
Deployable antenna panel.Figure 2
The active adjustment system of a deployable antenna.The control model of the active adjustment system is changed because of the disturbances which came from working conditions, model simplifications, and external interferences of the 5-meter deployable antenna. Added to that is that the disturbance can significantly affect the observation accuracy of the deployable antenna, particularly the disturbance change. To improve the observation accuracy of the deployable antenna, various control methods are worth our using for reference in the control process of the active adjustment system.In the past few years, a great deal of progress had been made in relation to the control of MIMO systems and numerous control strategies were introduced [3–5]. The intelligent control methods for MIMO systems have been put forward by researchers combining fuzzy control theory with artificial intelligence and adaptive control [6–8]. Hamdy et al. proposed a novel inverted fuzzy decoupling scheme [9]; moreover, the Smith predictor (SP) based on a fuzzy decoupling scheme was proposed for a MIMO chemical process with multiple time delays. The schemes were achieved using fuzzy logic, thereby avoiding the reliance on model-based analyses. Furthermore, Hamdy and Ramadan proposed another fuzzy decoupling-based PI controller [10]. Fuzzy decoupling schemes are simple to design and easy to implement, demonstrating good decoupling capabilities. However, such methods are only applicable to cases wherein the output is a state quantity without the disturbances.The discrete linear quadratic regulator (LQR) controller for the active adjustment system model ensured the observation accuracy requirement of the deployable antenna [11]. However, the LQR controller was dependent on the accurate control model, and it was presented without considering the disturbances.In [12], an adaptive reduced dimension fuzzy decoupling control strategy based on fusion functions was proposed to eliminate the effect uncertainties due to coupled subsystems by designing and incorporating a fuzzy decoupling disturbance observer. But the method would no longer be applicable, for the disturbances were greater than the range of the set threshold.According to [13], the Q-learning was considered to have a good capability for dealing with uncertainties. Thus, The Q-learning algorithm was applied for solving the mobile robot navigation in the dynamic environment problem by limiting the number of states based on a new definition for the state space [14]. The proposed method could reduce the dimension of the Q value table and improved the efficiency of the system. A novel Q-learning method was described for unknown discrete-time (DT) linear quadratic regulation problems [15]. In the method, Q-learning was proved to be an effective scheme for unknown dynamical systems without requiring a deep understanding of the system dynamics. The proposed two methods could deal with the uncertainties from the dynamic environment, but they were unable to cope with the disturbances from the model simplifications and external interferences.A backward Q-learning strategy was presented based on the combination with the Sarsa algorithm and the Q-learning [16]. The proposed method was found to enable converging more quickly to true Q values and improve final performance. A heuristic accelerated Q-learning method was developed by the heuristic function to affect action selection [17], and results of the simulation showed that a simple heuristic function could enhance the performance of the Q-learning method. The proposed two strategies could enhance the performance of the control system, but the Q value table of the Q-learning was only for an individual control model, respectively; it was not used for the other control model.Therefore, the key to solve the disturbance change is to make the parameters of Q-learning adapt to the control model change. But in the traditional Q-learning algorithm, the parameters of Q-learning are difficult to adapt to the control model change because the environment where the agent is located is spatially continuous. However, discretization of continuous state-action spaces could have huge impact on the dimension of theQ value table. One solution to this problem is introducing the fuzzy inference system (FIS) into Q-learning, which could be applied to discrete-time state mapping to continuous state-actions without affecting the dimension of Q value. During the last decade, a great deal of progress has been made on the area of combination the Q-learning method with the FIS.A model-free method was proposed by using a Q-learning scheme to tune a supervisory controller for a low-energy building system online [18]. The training time and computational demands were reduced by basing the supervisor on sets of fuzzy rules. A fuzzy Q-learning system by combining a fuzzy inference system (FIS) with Q-learning was presented [19]. The advantages of the system were that with limited expert experience, the fuzzy system could enable Sarsa (λ) to learn a certain amount of prior knowledge to improve the learning speed online. In combination with the above proposed strategies, the parameters of the fuzzy system were enhanced greatly by introducing the FIS into the Q-learning system. Nevertheless, the capability of the Q value applied to the model disturbance change was not mentioned.An extended fuzzy rule was proposed to incorporate Q-learning by Kim and Lee [20]. In the method, the interpolation technique was adopted to represent the appropriate Q value for the current state and action pair in each extended fuzzy rule, and then, the resultant structure based on the fuzzy inference system had the capability of solving the continuous state and action. In 2012, Song et al. focused on the neural network to initialization of Q value [21] so that the robot could get more optimized policy than the traditional Q-learning algorithm. A method was presented to get the navigation strategy for agricultural robot by autonomous learning [22]. This method was proved to be useful in constructing the discretization environment in Q-learning based on the information of the direction and distance of the continuously discrete obstacles in the fuzzy theory. The all above-proposed strategies were to obtain the appropriate Q value by introducing the FIS into Q-learning. However, the applicability of the Q value is not verified to the disturbance from the control model.The numerous existing strategies were widely used to robot navigation, but few mature methods were learned for the active adjustment system of a deployable antenna panel. In this paper, a high precision active control method named fuzzy neural network Q-learning control (FNNQL) strategy is proposed to overcome the model disturbance change of the active adjustment system of the deployable antenna panel. The main idea of the FNNQL controller is that the FIS is introduced into Q-learning, and the input of Q-learning is fuzzified. Therefore, the parameters of Q-learning are learned online by the fuzzy RBF neural network, which makes theQ value table suitable for the disturbance change from the control model. At the same time, the robustness of the control method is improved by ensuring the precision of the deployable antenna.The FNNQL controller has the following advantages:(1)
The error of the model disturbance is reduced because the input of Q-learning is fuzzified, and the accuracy of state transition is improved
(2)
A fuzzy rule base of the coupled MIMO system can construct using trial-and-error by Q-learning so that the accurate relationship between the premise and consequent parameters of the fuzzy system can be gotten
(3)
The outputs of the fuzzy system areQ value and action selection of Q-learning, respectively. And the value function and action can be modified by the RBF neural network with the change of state environment


## 2. Problem Statement

The active adjustment mechanism of the deployable antenna mathematical model can be described as follows [11]:
(1)Zck+1=Zck+B∗Uk,Yck=Zck,where Zck stands for the displacement before the kth adjustment, Uk is the external force of the kth adjustment, and B represents the transfer matrix of the external force and actual displacement.To study the effect of disturbance changing for the antenna precision, the model disturbance can be described by the uncertainty matrixΔB, which was randomly generated in the range of τ% based on the 2-norm of control matrix B, defined as
(2)ΔB=random−τ%B,τ%B,where τ is the positive constant belonging to [1, 100]. Based on Equations (1) and (2), the discrete multivariable control model with disturbance can be described as
(3)Zpk+1=Zpk+ΔB+B∗Uk,Ypk=Zpk.Since the active adjustment mechanism of a deployable antenna reflector is made of several active actuators, they affected each other during the antenna reflect panel adjustment process. Therefore, it is necessary to convert the complex control system into a more simplified control system. For any MIMO control system with a full rank transfer matrix, a compensator can be designed to decouple it [23].Assuming thatWs is the transfer function matrix of Equation (1), then
(4)Ws=CSI−A−1B,where C and A are unit diagonal matrices, respectively, and Ws is the full rank matrix. Thus, a compensator can be designed to decouple the MIMO system as shown in Equation (1).Based on a unit diagonal matrix decoupling principle, Equation (4) can be written as
(5)WpsWs=I,where Wps is the transfer function matrix of the feed-forward compensator.According to Equation (5), we can derive
(6)Wps=Ws−1=B−1SI−AC−1.From Equation (6), suppose the output matrix of the feed-forward compensator is B−1, the output equation of the feed-forward compensator can be written as
(7)Xk=B−1Zk.Rearranging Equation (7), we obtain
(8)Zk=B−1Xk.Substituting Equation (8) into Equations (1) and (3), the following equation can be obtained:
(9)BXpk+1=BXpk+ΔB+B∗Uk,Ypk=Xpk.Multiplying both sides of Equation (9) by B−1, Equation (9) can be further expressed as
(10)Xpk+1=Xpk+Uk+B−1ΔB∗Uk,Ypk=Xpk,where Xk is the state vector, Xk=x1k,x2k,⋯,xNk, N is the number of the actuators, and B−1ΔB is an unknown vector representing the disturbance matrix. From Equation (10), the discrete multivariable decoupled system can be decomposed into many independent single variable state equations. In this section, the middle plate of the deployable antenna panel is taken as an example to test the validity of Q-learning, as shown in Figure 3. In addition, the 25 actuators are distributed on the middle plate of the deployable antenna as shown in it.Figure 3
Middle plate of the deployable antenna panel.In this section, a simulation analysis is performed to test the validity of the Q-learning controller. During the simulation, control matrixB was obtained using finite element software ANSYS (version 14.5), and the raw data from tests is shown in Table 1. The disturbance matrixes ΔB can be obtained in the case of τ=15, τ=30, and τ=90, according to Equation (2), which can be seen in Tables 2–4, respectively. The initial value Xk is given as
(11)Xk=−1.1412.6712.66−3.994.03−3.18−12.14−11.930.300.050.47−9.30−14.892.122.010.1911.080.631.191.141.09−9.193.842.267.25.Table 1
Transfer matrixB of the middle plate.
Xij
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
1
90.78
0.91
4.14
-1.65
0.45
-0.10
-3.77
5.97
-0.08
-1.96
0.23
6.03
0.04
-0.12
-0.73
1.64
0.45
1.56
-0.01
0.16
-0.01
-0.13
-0.57
63.15
13.15
2
0.91
90.74
0.61
0.03
3.06
-3.45
-3.67
-0.09
5.99
-1.92
0.04
-0.13
0.23
6.04
-0.72
0.45
1.62
-0.01
1.56
0.18
-0.02
-0.11
-0.55
0.59
0.13
3
4.14
0.61
6.31
0.00
0.19
-0.03
0.03
0.60
0.06
-0.01
0.11
0.61
0.03
0.07
-0.11
1.66
0.19
0.17
0.02
0.11
0.08
0.11
-0.02
2.58
1.17
4
-1.64
0.03
0.01
3.65
-0.03
-0.14
5.44
-2.54
-0.19
3.12
0.03
-2.97
-0.01
-0.02
0.03
0.04
0.00
-0.49
0.02
0.01
0.01
0.00
0.93
-2.47
-5.37
5
0.44
3.06
0.18
-0.04
51.69
-0.12
0.05
0.01
0.55
0.01
0.02
0.02
0.10
0.55
0.14
0.14
1.07
0.01
0.17
0.07
0.05
0.09
-0.02
0.44
0.12
6
-0.10
-3.45
-0.03
-0.14
-0.11
133.91
11.64
0.15
-14.45
7.11
0.00
0.32
-0.01
-9.25
0.04
-0.02
-0.05
0.07
-1.13
-0.01
-0.01
-0.01
2.18
-0.04
0.36
7
-3.77
-3.67
0.03
5.43
0.05
11.63
174.39
10.08
9.75
96.53
0.07
7.11
0.06
6.94
0.23
0.14
0.13
0.59
0.57
0.03
0.05
-0.02
33.00
-5.03
-3.16
8
5.96
-0.09
0.60
-2.55
0.01
0.15
10.08
110.11
-1.15
11.66
0.16
68.16
-0.01
-1.34
0.04
0.38
0.11
7.17
-0.31
0.06
0.13
0.23
7.21
-2.03
63.70
9
-0.08
4.41
0.06
-0.20
0.56
-14.43
9.76
-1.14
110.11
11.58
-0.01
-1.33
0.17
68.17
0.04
0.12
0.38
-0.31
7.18
0.08
0.11
0.17
7.18
0.71
-1.29
10
-2.00
-1.96
-0.05
3.07
0.00
7.10
96.49
11.62
11.54
138.08
-0.01
11.60
-0.01
11.59
0.08
0.02
0.02
1.27
1.27
-0.03
-0.06
-0.23
60.07
-2.22
-0.54
11
0.23
0.04
0.11
0.03
0.02
0.00
0.07
0.17
-0.01
0.03
0.77
0.18
0.00
0.00
0.03
0.12
0.03
0.08
0.00
0.02
0.02
0.04
0.00
0.36
0.28
12
6.03
-0.12
0.62
-2.97
0.03
0.32
7.12
68.17
-1.33
11.65
0.19
115.94
0.00
-1.92
0.05
0.41
0.14
15.12
-0.57
0.08
0.18
0.37
9.68
-2.63
98.64
13
0.04
0.23
0.03
-0.01
0.11
-0.01
0.06
-0.01
0.17
0.03
0.00
0.00
0.77
0.19
0.03
0.03
0.12
0.00
0.08
0.02
0.02
0.04
0.00
0.04
0.01
14
-0.12
6.05
0.07
-0.01
0.56
-9.22
6.95
-1.33
68.17
11.64
0.00
-1.92
0.19
15.93
0.05
0.14
0.41
-0.57
15.11
0.09
0.15
0.28
9.68
0.62
-1.60
15
-0.73
-0.72
-0.11
0.03
0.15
0.04
0.23
0.04
0.04
0.12
0.03
0.05
0.03
0.05
29.91
0.70
0.68
0.03
0.03
0.15
0.03
0.13
0.03
0.06
0.02
16
1.64
0.46
1.67
0.04
0.15
-0.02
0.15
0.39
0.12
0.07
0.13
0.41
0.03
0.14
0.71
25.80
0.00
0.06
0.02
0.17
0.33
0.16
0.02
-0.12
0.67
17
0.44
1.62
0.19
-0.01
1.08
-0.05
0.13
0.11
0.38
0.06
0.03
0.13
0.12
0.41
0.67
-0.01
25.68
0.01
0.06
0.18
0.31
0.17
0.01
0.45
0.23
18
1.56
-0.01
0.17
-0.49
0.01
0.07
0.59
7.17
-0.31
1.31
0.08
15.11
0.00
-0.57
0.02
0.05
0.01
21.25
-0.39
0.02
0.21
0.90
1.27
2.25
14.29
19
-0.01
1.56
0.02
0.02
0.18
-1.12
0.58
-0.31
7.18
1.32
0.01
-0.57
0.09
15.11
0.03
0.02
0.06
-0.38
21.26
0.03
0.20
0.77
1.28
0.13
-0.43
20
0.16
0.18
0.11
0.00
0.08
-0.01
0.03
0.06
0.08
0.01
0.02
0.08
0.02
0.09
0.14
0.17
0.18
0.02
0.02
1.11
0.10
0.12
0.00
0.12
0.12
21
-0.01
-0.02
0.08
0.01
0.05
0.00
0.05
0.13
0.11
-0.01
0.02
0.18
0.02
0.15
0.03
0.33
0.31
0.21
0.19
0.10
18.52
17.27
-0.12
-0.99
0.34
22
-0.13
-0.11
0.11
-0.01
0.10
-0.01
-0.02
0.23
0.17
-0.19
0.04
0.36
0.04
0.27
0.13
0.16
0.17
0.90
0.77
0.12
17.27
54.71
-0.52
-2.33
0.87
23
-0.57
-0.55
-0.02
0.93
-0.01
2.15
33.01
7.22
7.18
60.11
0.01
9.68
0.01
9.68
0.03
0.02
0.01
1.28
1.28
0.00
-0.11
-0.51
76.49
-0.50
0.16
24
63.19
0.59
2.59
-2.51
0.45
-0.04
-5.01
-2.01
0.72
-2.21
0.36
-2.61
0.04
0.62
0.06
-0.12
0.45
2.26
0.13
0.12
-0.98
-2.31
-0.50
440.78
-8.24
25
13.19
0.13
1.17
-5.38
0.12
0.36
-3.16
63.69
-1.29
-0.50
0.28
98.64
0.01
-1.60
0.02
0.67
0.23
14.29
-0.43
0.12
0.34
0.87
0.16
-8.24
286.42Table 2
Uncertainty matrixΔB (the range of 15%).
Xij
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
1
0.09
0.08
-0.07
-0.05
-0.10
0.11
-0.02
-0.10
0.04
0.14
0.14
0.12
-0.13
-0.03
-0.06
-0.11
-0.02
-0.07
0.12
0.02
0.02
0.08
0.07
0.10
0.11
2
0.12
0.07
0.05
-0.09
0.09
-0.12
-0.14
0.04
-0.04
-0.02
0.01
0.12
0.05
0.01
0.06
0.03
-0.12
-0.02
-0.15
0.06
0.01
0.13
-0.03
0.08
0.02
3
-0.11
-0.03
0.05
-0.07
-0.06
-0.03
0.12
0.07
0.09
-0.12
0.01
-0.05
-0.14
-0.02
0.05
-0.08
0.03
0.01
-0.01
0.01
0.11
0.14
0.06
-0.10
0.13
4
0.12
0.05
-0.10
0.03
0.01
-0.07
0.13
0.04
0.01
-0.07
-0.08
0.06
-0.13
0.05
0.01
-0.03
-0.01
-0.01
-0.02
-0.02
-0.07
-0.09
0.06
0.11
0.06
5
0.04
-0.10
-0.11
-0.01
-0.10
0.09
0.00
-0.01
-0.04
-0.03
0.00
-0.09
0.01
0.04
0.06
0.02
0.06
0.11
-0.01
0.04
-0.05
-0.11
-0.02
0.15
0.02
6
-0.12
0.06
0.00
-0.04
0.03
-0.02
0.00
0.01
0.13
0.03
0.04
-0.14
-0.12
-0.06
0.05
-0.07
0.06
0.01
0.08
0.04
-0.11
0.06
-0.14
0.00
0.09
7
-0.07
-0.14
0.14
0.10
-0.07
0.12
-0.05
-0.06
0.11
-0.07
0.05
0.07
0.10
-0.02
-0.10
-0.06
0.04
0.13
-0.05
0.05
0.13
-0.12
-0.05
0.12
0.11
8
0.01
-0.07
-0.05
0.03
0.05
-0.10
0.12
0.07
0.02
0.03
-0.03
0.00
0.10
-0.15
-0.11
0.04
-0.14
0.04
0.09
0.04
0.04
0.01
-0.02
0.03
0.15
9
0.14
-0.14
0.03
0.01
0.06
-0.07
-0.04
-0.09
0.04
0.06
-0.04
-0.01
0.07
0.15
0.15
-0.07
-0.13
0.14
-0.01
0.13
-0.01
0.01
-0.07
-0.10
-0.15
10
0.14
-0.12
-0.08
0.13
0.07
-0.11
-0.12
0.06
0.03
-0.08
0.15
0.12
-0.11
-0.10
-0.10
0.10
-0.05
-0.08
-0.14
-0.09
0.04
0.11
-0.09
-0.09
0.11
11
-0.10
0.10
0.08
-0.06
-0.01
-0.11
0.08
-0.09
-0.09
-0.11
-0.14
0.03
0.05
-0.12
-0.14
0.14
0.01
0.05
-0.10
0.06
0.01
0.00
0.10
-0.03
0.03
12
0.14
0.06
-0.07
0.08
-0.12
0.11
-0.03
-0.04
-0.06
-0.06
0.12
0.04
0.01
-0.04
0.02
0.07
0.05
-0.06
0.07
-0.08
0.04
-0.03
-0.02
0.07
0.15
13
0.14
-0.05
0.00
0.08
-0.08
0.02
-0.08
0.04
-0.01
-0.05
0.12
0.11
0.14
-0.09
0.11
-0.05
-0.03
0.05
-0.01
-0.11
0.01
0.05
0.12
0.10
0.01
14
0.00
0.14
0.06
-0.04
0.12
0.01
-0.03
0.08
-0.08
-0.02
0.09
0.09
0.04
0.00
0.05
0.03
0.10
0.06
-0.10
0.03
0.07
0.07
-0.03
0.09
-0.01
15
0.09
-0.14
0.12
0.02
-0.10
-0.11
-0.12
-0.13
0.10
0.00
-0.12
0.02
0.09
-0.05
-0.09
-0.12
0.07
-0.13
-0.05
-0.01
0.01
0.01
0.08
-0.05
0.09
16
-0.11
-0.02
0.14
-0.13
0.10
0.11
-0.11
0.13
-0.09
-0.12
-0.07
-0.10
-0.01
0.14
-0.04
0.12
0.14
-0.07
0.03
-0.01
0.15
-0.05
-0.03
0.01
-0.08
17
-0.02
-0.04
0.01
-0.13
0.01
0.04
0.13
0.08
-0.08
-0.07
-0.05
-0.08
-0.02
0.13
-0.01
0.11
0.01
-0.08
-0.09
0.05
-0.08
-0.11
0.09
-0.12
0.00
18
0.12
0.08
-0.11
0.01
0.15
-0.04
0.14
0.00
-0.10
0.09
0.05
0.12
0.10
-0.13
0.14
0.10
-0.05
0.05
0.07
0.08
-0.12
0.03
0.08
-0.12
0.12
19
0.09
0.09
-0.11
0.08
-0.13
0.00
0.02
-0.02
-0.08
-0.14
-0.11
-0.14
-0.12
0.07
-0.10
-0.07
-0.12
0.10
-0.08
-0.04
-0.12
-0.07
-0.04
-0.11
0.02
20
0.14
-0.09
-0.07
0.13
-0.02
-0.03
-0.13
-0.02
-0.02
0.13
0.07
0.00
-0.11
-0.07
0.11
0.03
0.03
-0.05
0.13
0.05
-0.13
-0.14
-0.09
0.05
0.10
21
0.05
0.00
0.10
-0.11
-0.12
-0.13
-0.08
-0.06
-0.06
0.07
-0.12
-0.10
-0.10
-0.02
0.04
-0.14
0.08
0.08
-0.07
-0.03
-0.03
0.08
0.09
0.00
0.07
22
-0.14
-0.02
-0.07
0.02
0.14
-0.08
-0.04
0.00
0.13
0.00
0.05
0.14
-0.03
0.01
-0.04
-0.02
-0.02
0.05
0.08
0.10
-0.02
-0.08
0.13
-0.09
0.03
23
0.10
0.04
0.09
-0.01
-0.15
-0.11
0.10
0.00
-0.02
0.02
0.00
0.06
0.10
0.13
-0.09
-0.06
-0.12
-0.15
-0.09
0.10
-0.04
-0.02
-0.05
0.00
-0.08
24
0.13
0.06
-0.08
-0.15
0.08
-0.09
-0.15
0.10
-0.09
-0.08
0.08
0.00
0.09
-0.02
-0.02
-0.10
-0.07
0.03
-0.06
-0.07
0.08
0.06
0.05
-0.11
0.05
25
0.05
0.08
0.13
-0.05
0.10
-0.08
-0.14
0.09
0.12
-0.01
0.06
-0.01
-0.13
0.14
-0.01
-0.10
-0.10
-0.03
-0.12
0.03
0.04
-0.04
-0.02
-0.13
-0.12Table 3
Uncertainty matrixΔB (the range of 30%).
Xij
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
1
0.05
-0.14
0.05
0.02
-0.13
-0.02
0.01
0.07
-0.12
-0.16
0.01
0.20
-0.06
-0.13
0.17
0.05
-0.18
0.20
0.06
0.11
0.16
0.09
0.10
0.00
-0.14
2
0.06
-0.12
0.03
0.14
-0.12
-0.16
0.09
-0.15
-0.04
0.17
-0.16
-0.13
0.02
0.07
0.12
-0.06
0.09
-0.17
-0.10
0.08
-0.16
0.05
0.18
0.00
-0.06
3
0.09
-0.18
-0.18
-0.06
0.16
0.20
0.08
0.09
0.02
-0.13
0.05
-0.10
0.10
0.16
0.03
0.20
-0.06
0.18
-0.08
-0.20
-0.18
-0.06
-0.19
0.15
-0.15
4
0.16
0.05
0.17
-0.02
0.07
-0.07
0.11
-0.16
-0.11
-0.09
-0.15
-0.04
-0.03
0.01
-0.02
-0.11
0.06
-0.19
0.07
0.14
0.02
0.17
-0.06
-0.06
0.15
5
0.19
-0.09
0.09
-0.18
-0.01
-0.08
-0.08
-0.15
0.06
0.12
-0.15
-0.17
-0.03
0.08
-0.10
0.06
-0.05
0.07
0.01
0.17
0.11
-0.15
0.07
-0.02
-0.16
6
0.11
0.02
0.10
-0.13
0.16
-0.18
0.08
0.06
-0.01
0.00
-0.16
0.07
-0.15
-0.14
0.10
0.04
0.05
0.11
-0.04
0.11
-0.08
0.09
-0.09
0.19
0.17
7
0.03
0.08
-0.17
0.07
-0.16
-0.08
0.02
-0.07
-0.14
0.11
-0.14
-0.04
-0.19
0.18
-0.11
-0.05
-0.19
0.01
0.04
-0.18
-0.13
0.06
-0.11
-0.18
-0.04
8
0.17
0.00
0.14
-0.07
0.10
-0.18
-0.04
0.06
0.11
-0.04
-0.13
0.19
-0.08
0.02
-0.17
-0.14
0.16
0.15
0.10
-0.05
-0.06
0.13
0.08
0.19
-0.18
9
0.03
0.01
0.17
0.16
0.09
0.00
-0.18
0.10
-0.16
-0.09
-0.12
-0.04
-0.07
0.07
0.11
-0.19
0.12
0.16
0.03
0.08
-0.12
-0.04
0.05
-0.12
-0.06
10
-0.19
-0.02
0.19
-0.15
0.02
0.10
0.11
0.03
-0.08
-0.19
-0.07
0.05
0.06
-0.19
0.07
-0.03
0.10
0.05
0.02
0.09
0.00
0.10
0.04
0.07
0.09
11
-0.15
-0.15
0.14
0.20
-0.13
0.05
-0.06
0.10
-0.11
0.07
-0.07
-0.14
0.18
0.12
0.09
-0.13
0.13
-0.14
0.03
-0.11
0.16
0.13
0.06
0.03
0.12
12
0.15
0.00
0.11
0.02
0.04
-0.16
0.04
-0.11
0.01
-0.03
-0.11
-0.05
0.17
0.10
0.06
0.09
-0.05
-0.11
0.00
-0.09
0.05
-0.07
-0.18
0.07
0.02
13
-0.01
0.14
0.01
0.08
-0.08
-0.17
0.10
0.09
-0.16
-0.02
-0.10
-0.14
-0.02
-0.15
-0.03
-0.05
0.05
-0.13
-0.17
0.07
-0.16
0.02
-0.06
-0.06
0.07
14
0.14
0.15
-0.13
0.20
-0.15
0.11
-0.16
0.19
-0.04
0.04
0.16
0.10
-0.10
0.01
-0.04
0.14
0.03
-0.18
0.09
-0.01
-0.04
0.19
-0.02
0.05
0.16
15
-0.12
-0.09
-0.04
-0.08
-0.11
0.16
-0.15
0.15
-0.16
-0.18
0.08
0.15
0.11
-0.07
0.13
0.09
0.01
-0.16
0.20
0.05
-0.18
0.02
-0.10
0.12
-0.18
16
0.02
-0.12
-0.15
-0.03
0.16
0.01
0.02
-0.17
-0.16
-0.07
0.02
-0.06
0.10
0.02
-0.07
0.03
-0.09
0.05
-0.06
-0.11
0.00
-0.07
0.09
-0.19
-0.08
17
0.05
0.03
-0.19
-0.01
-0.17
-0.16
-0.01
-0.05
0.11
0.11
-0.13
0.07
0.10
-0.04
0.13
-0.13
-0.10
0.18
0.19
-0.13
-0.03
0.05
0.14
-0.17
-0.18
18
-0.19
0.06
0.18
0.11
-0.10
0.13
0.16
-0.05
-0.08
0.08
-0.12
-0.08
0.10
-0.03
0.12
0.18
-0.02
-0.06
-0.06
0.13
0.20
-0.06
-0.09
0.19
-0.12
19
0.05
-0.03
-0.08
0.13
-0.18
-0.06
0.12
0.07
0.04
-0.15
-0.17
0.01
-0.16
-0.13
0.14
-0.09
-0.11
-0.04
0.15
0.11
0.12
0.10
0.09
0.06
0.09
20
-0.06
-0.12
-0.08
-0.16
-0.02
-0.08
0.09
0.04
0.19
-0.15
0.17
0.13
0.07
-0.10
0.00
0.17
0.12
0.19
-0.02
0.17
-0.01
-0.03
-0.14
-0.11
0.09
21
-0.18
0.18
-0.07
-0.13
-0.19
0.10
-0.18
0.12
-0.03
-0.16
0.08
0.04
-0.01
-0.19
0.05
-0.11
0.19
0.18
-0.03
-0.16
0.16
0.00
0.13
-0.04
0.15
22
0.00
-0.17
-0.01
-0.06
0.16
-0.20
-0.17
-0.05
0.08
-0.20
0.02
-0.07
-0.12
0.17
0.18
-0.05
-0.19
0.07
-0.11
-0.13
-0.14
0.08
-0.14
-0.15
0.03
23
-0.12
-0.16
0.06
-0.18
-0.12
-0.18
-0.16
-0.12
0.10
-0.03
-0.07
-0.08
-0.16
0.06
-0.02
-0.16
0.01
0.20
-0.15
-0.16
-0.04
0.19
0.04
-0.09
-0.17
24
-0.15
-0.14
-0.19
0.01
-0.16
0.07
0.12
-0.17
-0.03
0.06
-0.13
-0.02
0.13
0.17
-0.18
0.06
-0.17
0.11
-0.08
0.00
0.17
-0.07
-0.05
-0.10
0.17
25
-0.12
-0.13
0.14
-0.07
-0.08
0.04
0.18
0.11
0.06
0.09
0.05
-0.03
-0.13
-0.13
0.15
-0.13
0.12
-0.07
0.09
-0.12
0.17
0.14
0.12
-0.07
0.12Table 4
Uncertainty matrixΔB (the range of 90%).
Xij
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
1
-0.39
-0.38
-0.17
-0.39
-0.69
-0.64
0.01
-0.13
-0.86
0.03
-0.65
-0.50
-0.33
0.78
0.31
0.86
0.26
-0.32
0.61
-0.09
0.56
0.11
-0.54
0.08
-0.47
2
0.08
0.54
0.30
0.31
0.57
0.24
0.20
0.35
-0.12
-0.59
-0.43
-0.03
0.20
0.57
-0.12
-0.39
-0.50
0.02
0.15
-0.21
-0.26
0.46
-0.55
0.07
0.61
3
0.87
0.71
0.78
0.30
-0.32
0.65
0.57
0.80
0.60
0.79
-0.74
-0.22
0.74
0.41
0.35
0.17
0.61
-0.79
0.81
0.52
-0.77
0.89
-0.31
-0.34
0.64
4
0.39
0.18
0.56
-0.68
-0.46
0.85
0.06
0.51
0.21
0.16
-0.13
0.04
0.74
-0.58
-0.44
0.83
0.85
0.41
-0.79
-0.24
0.16
0.83
0.68
-0.77
0.83
5
0.61
0.69
-0.03
-0.17
-0.28
0.13
-0.54
0.37
0.04
-0.11
-0.44
-0.42
0.16
-0.25
-0.88
-0.57
0.62
0.10
0.15
0.06
0.74
0.06
-0.05
-0.57
-0.02
6
-0.12
0.80
0.46
-0.40
-0.22
0.89
-0.08
-0.70
0.65
0.80
-0.36
-0.78
-0.30
-0.56
0.06
-0.55
0.01
0.05
-0.39
0.38
-0.55
0.83
-0.17
-0.73
-0.50
7
-0.05
0.09
-0.15
0.39
0.08
0.10
-0.13
-0.20
-0.72
0.28
-0.14
-0.11
0.64
-0.90
-0.40
-0.29
-0.40
0.59
0.59
0.67
-0.12
-0.69
-0.58
-0.07
-0.49
8
0.11
0.41
0.85
-0.39
0.11
0.03
0.84
0.16
0.73
-0.09
-0.69
-0.59
-0.10
-0.33
0.80
0.78
0.44
0.65
-0.56
-0.31
0.45
-0.81
0.84
-0.88
0.07
9
-0.42
0.14
0.88
0.71
-0.19
-0.30
0.22
-0.07
-0.71
0.61
-0.01
-0.85
0.73
0.36
0.73
-0.20
-0.47
0.52
-0.10
0.27
-0.83
-0.35
-0.17
0.75
0.47
10
0.45
-0.85
0.66
0.59
-0.18
-0.13
0.35
-0.81
0.03
0.06
0.37
0.82
-0.84
0.23
-0.19
-0.41
0.82
-0.33
-0.19
0.85
0.80
0.14
0.62
0.26
-0.27
11
0.01
-0.10
-0.20
-0.20
0.03
-0.01
0.40
-0.49
-0.64
0.10
-0.46
-0.12
0.06
0.08
-0.86
-0.63
0.22
-0.09
0.59
-0.76
0.47
0.06
0.21
-0.90
-0.07
12
0.26
0.26
-0.08
0.00
0.28
-0.77
-0.28
0.60
0.11
0.32
0.51
0.83
0.39
-0.11
0.31
-0.19
0.18
0.45
0.32
0.16
0.11
0.72
-0.22
-0.85
0.25
13
-0.35
0.04
-0.46
0.35
0.81
0.70
0.03
-0.87
-0.89
-0.24
-0.77
0.47
-0.58
-0.38
0.61
-0.23
-0.59
-0.70
-0.53
-0.16
-0.57
0.07
0.68
-0.52
0.75
14
-0.65
-0.23
0.51
0.60
0.40
-0.78
0.10
0.65
0.48
-0.47
-0.19
-0.89
-0.29
0.00
0.85
-0.66
-0.74
-0.70
-0.33
-0.34
0.00
-0.12
0.51
-0.08
-0.61
15
-0.04
0.79
0.69
0.20
-0.18
-0.11
-0.62
-0.76
0.63
0.14
-0.89
0.32
-0.56
0.47
-0.80
-0.12
-0.44
-0.41
-0.66
-0.43
0.03
0.08
-0.06
-0.67
0.39
16
-0.25
0.59
0.74
0.13
0.60
0.59
0.11
0.30
0.75
0.66
-0.50
0.37
-0.32
0.47
-0.09
-0.74
0.65
0.04
0.31
0.47
0.89
0.38
0.57
-0.88
0.14
17
0.52
0.63
0.10
-0.31
-0.66
-0.19
0.35
0.00
0.88
-0.17
-0.90
0.26
-0.17
0.14
0.15
0.21
0.74
0.85
0.13
0.89
0.64
-0.87
0.72
0.41
-0.12
18
0.50
-0.23
0.18
-0.08
-0.79
0.20
-0.13
-0.51
0.01
-0.70
-0.56
0.09
0.09
0.45
0.34
-0.88
0.36
0.38
-0.59
-0.56
0.83
0.54
-0.13
-0.26
0.69
19
0.30
0.17
-0.63
0.38
-0.75
0.57
0.61
0.13
-0.41
-0.10
-0.64
-0.51
-0.81
0.26
0.39
0.13
0.41
-0.34
-0.63
0.51
0.32
-0.64
-0.30
0.50
-0.19
20
-0.66
0.67
0.72
0.69
-0.60
0.70
0.42
-0.68
-0.72
-0.36
-0.42
0.49
0.09
-0.68
0.27
0.52
-0.49
-0.38
-0.04
-0.55
-0.17
-0.04
0.17
-0.11
-0.58
21
-0.86
0.78
-0.09
0.40
-0.32
0.78
-0.25
0.31
0.01
-0.18
-0.59
-0.49
-0.41
0.01
0.41
-0.48
0.14
0.63
0.73
0.89
0.78
-0.44
0.72
-0.11
0.24
22
0.11
0.30
-0.53
-0.87
-0.36
-0.56
-0.08
0.18
0.15
0.60
-0.65
-0.23
-0.47
-0.27
-0.23
-0.09
0.56
0.74
0.09
0.54
-0.04
-0.24
0.36
-0.81
0.22
23
-0.36
-0.53
0.72
0.31
-0.88
-0.43
-0.20
-0.80
0.47
-0.17
0.18
0.70
-0.46
-0.73
0.15
0.12
-0.17
0.25
-0.84
-0.14
-0.48
0.29
-0.22
-0.81
-0.31
24
0.79
0.28
0.47
-0.11
0.07
0.72
0.50
-0.80
-0.75
-0.20
0.72
0.64
-0.62
-0.63
-0.69
-0.79
0.88
-0.44
-0.80
0.41
-0.19
-0.59
0.42
-0.74
0.55
25
0.87
-0.77
0.69
-0.11
-0.73
0.17
0.42
-0.63
0.29
-0.25
0.79
-0.18
0.82
-0.54
-0.80
-0.01
-0.74
-0.74
0.55
0.00
0.37
-0.40
0.82
0.17
0.90The units are mm.In the simulation, the value in Table2 was introduced as disturbances into Equation (10) to assess the ability of the Q-learning controller. The displacement was adjusted as shown in Figure 4. In the figure, the curves were seen converged at the 25th step after large fluctuations, which indicated the system reached stability. Figure 5 shows the root mean square (RMS) error curve for the 25 actuators, and it is found that the displacements of 25 actuators are below 10-3 mm at the 8th step. The RMS is in the micron scale, which indicates a tendency to narrow. It is proved that the Q value table and the action selection obtained by continuous trial-and-error can adapt to the disturbance of 15%.Figure 4
Q-learning adjustment curve with disturbance of 15%.Figure 5
Root mean square (RMS) based on 25 points.In the next simulation, theQ value table which was obtained with disturbance of 15% was used to deal with the disturbance form in the range of 30% to 90%, respectively. The results can be shown in Figures 6 and 7. As shown in the figures, since the disturbances for the control system were changed, the excessive overshooting was caused.Figure 6
Q-learning adjustment curve with disturbance of 30%.Figure 7
Q-learning adjustment curve with a disturbance of 90%.That shows that theQ value table obtained under the condition of the certain disturbance could not adapt to other disturbance changes. If the system is to be stable under the other disturbances, it is necessary to get the Q value corresponding to the disturbance through continuous trial-and-error, which increases the workload.To keep the system’s stability with the consideration of disturbance change, the FNNQL control strategy was proposed in the following section. In the proposed controller, the two parts were included. In Part I: Fuzzy Parameter Learning, the fuzzy neural network was introduced into Q-learning, and the input of Q-learning is fuzzified. Meanwhile, the trial-and-error method was used in the Q-learning system to gain the fuzzy rule base. In Part II: Fuzzy Parameter Adjustment, with the decreasing of temporal-difference (TD) error, the parameters of the membership function were updated through the fuzzy RBF neural network [24]. The Q value and the state-selection of Q-learning could be obtained to adjust the control model with disturbance change while the control system reached stability.


## 3. Fuzzy Neural Network Q-Learning Controller Design

### 3.1. Part I: Fuzzy Parameter Learning

In this part, since the FIS was introduced into the Q-learning, the optimal parameters of the FNNQL controller would be obtained. The structure of the FNNQL controller can be shown in Figure8, which can be divided into five layers. The 1st layer is the system input, which is followed by the 2nd layer, where the fuzzy is illustrated. And then, the 3rd layer is the activation operation, and the 4th layer is the action selection. Finally, the output results can be obtained in the 5th layer. The detailed description on each layer will be introduced in the following paragraphs.Figure 8
Structure of fuzzy neural network Q-learning based on RBF.


#### 3.1.1. Input Layer

In this layer, to reflect the characteristics of disturbance change in real time, the error of the control state and the rate of the error change should be taken as the inputs of the FNNQL controller. According to Equations (1) and (10), the error of the control state can be described as
(12)ek=Yck−Ypk=Zck−Xpk.Based on Equation (12), the rate of the error change can be described as
(13)eck=ek+1−ek.According to Figure8, the inputs in this layer are also the previous parameter of the neural network FIS (NNFIS).


#### 3.1.2. Fuzzy Layer

The inputs of the fuzzy layer are the outputs of the1st layer. They would be fuzzified in this layer. The quantization factors ke and kec are introduced into the fuzzified equations to map the inputs to the fuzzy domain accurately effectively. The fuzzified equations for ek and eck are defined, respectively, as
(14)Feek=keek,Fececk=kececk.In the process of fuzzification, the three fuzzy subsets are defined forek and eck, which can be defined as ek=B,N,S and eck=B,N,S. The membership function is Gauss.SupposeFe can be divided into p regions and Fec can be divided into q regions, which are defined, respectively, as
(15)S=s1,s2,⋯,sp,(16)SC=sc1,sc2,⋯,scq.Based on Equations (15) and (16), the number of the fuzzy rules can be defined as
(17)c=p×q.In this paper, the fuzzy membership function can be defined as(18)fx,σ,c=exp−x−m22σ2,where σ is the width of the membership function and m is the centre of the membership function. Then, the membership functions of the two fuzzy inputs ek and eck can be defined as
(19)μi=μeik,σik,mik=exp−eik−mik22σi2k,(20)μci=μcecik,σcik,mcik=exp−ecik−mcik22σci2k,where i represents the rule number, i=1,2,⋯,n; mik and σik are the centre and the width of μi, respectively; and mcik and σcik are the centre and the width of μci, respectively. The detailed inner structure of the ith node in the fuzzy layer can be seen in Figure 9.Figure 9
The inner structure of theith node.In this layer, the inputs of the control system are fuzzified. In the next step, the rate of the fuzzy rule activated should be calculated to get the accurate output of the FNNQL control system.


#### 3.1.3. Activation Calculation Layer

First, theith fuzzy rule can be supposed as
(21)ifeikisnikandecikisncik,thenactionselectionisai,where ai is the optimal action selection of the FNNQL control.For measuring the usage of fuzzy rules in NNFIS, according to Equations (19) and (20), the rate of the fuzzy rule activated is defined as
(22)φi=∑i=1cμi×μci,where i represents the rule number and φi is the rate of the fuzzy rule activated. According to φi, the optimal action selection will be described with the ith fuzzy rule as an example in the following layer.


#### 3.1.4. Action Selection Layer

In this layer, the accurate output of the FNNQL control system can be obtained based on the premise parameters, and each fuzzy input of the control system should obtain its optimal action. According to theε-greedy policy, the equation of the action selection can be defined as
(23)ai=Rqi,ε<φi,Bqi,ε≥φi,where Rqi∈A, A=a1,a2,⋯,ac, is the set of the random action; Bqi is the optimal action selection; ε is the action selection probability; and i is the number of the fuzzy rule. To assess the action selection, an additional penalty function will be used in this layer as shown in
(24)ri=φi+λ,eik≤ξ,φi−λ,eik>ξ,where ξ⟶0, λ∈0,1. If the stated error is within the accuracy range, a positive reward is obtained and the action selection is enhanced correspondingly. When it is not satisfied, a negative or zero reward is obtained, and the action selection is weakened instead. With using multiple trial-and-errors and ε-greedy, the relation of the mapping between the fuzzy inputs and the optimal action selection can be obtained. Therefore, the fuzzy rule base can be constructed. Since the state error is caused by model disturbance, the fuzzy rules obtained with trial-and-error are consistent with disturbance change. The structure of this layer can be shown in Figure 10.Figure 10
Structure of the action selection layer.


#### 3.1.5. Output Layer

In this layer, theQ value table corresponding to the optimal continuous action selection can be obtained as the consequent parameters of NNFIS based on the fuzzy rule base constructed in Layer IV. According to the approximate Q-learning principle [25] and Equation (23), the value of the optimal continuous action selection can be defined as
(25)a=∑i=0nφi×Bqi.Similarly, theQ value function corresponding to the optimal continuous action selection can also be represented as
(26)Qj=∑j=12∑i=1nφji×Bqji,where Qj is the total of all Q value function and j is the number of the control states. Based on Equations (10), (21), and (25), the equation of the control system can be rewritten as
(27)Xpk+1=Xpk+B−1ΔB+aUk,where Uk=ek,eckT is the input vector of the FNNQL controller.In Part I: Fuzzy Parameter Learning, theQ value function Qj was obtained, which can be used to adjust the cluster centre and cluster width of the membership function in Part II: Fuzzy Parameter Adjustment.


### 3.2. Part II: Fuzzy Parameter Adjustment

According to Equation (27), the new displacement XPk+1 can be obtained. And then, the new error ek+1 and the rate of the error eck+1 can be derived based on Equations (12) and (13). The reward value vector r can be obtained based on Equation (24). And then, according to the Bellman optimality theory [26], the time difference error of TD (0) can be described as
(28)δk=r+γQk+1−Qk,where γ is the discount rate.The mean square error (RMS) function, which is used to measure control system deformation accuracy, can be defined:(29)RMSk=12δ2k.In the FNNQL controller, since the membership function is Gauss RBF, the centre of an ideal RBF should be in line with the location of the fuzzy input that occurs most frequently, and all widths are exactly the boundary values of the highest frequency region of fuzzy input. Therefore, it is necessary to get the most suitable centre and width by adjusting the parameters so as to get accurately the disturbance distribution.According to Equation (29), the centre and width can be updated by the gradient descent [27, 28]. The gradient descent linear function of ek can be expressed as
(30)mijk∝−∂RMSk∂mijk,(31)mijk+1=mijk−βmijk∂RMSk∂mijk,where ∂RMSk/∂mijk is the gradient descent, βmijk is the step-size of the gradient descent, and mijk is the centre of the jth input variables of the ith fuzzy rule. The gradient descent can be further expressed as
(32)∂RMSk∂mijk=∂RMSk∂δk·∂δk∂mijk=∂RMSk∂δk·∂δk∂Qk.∂Qk∂φik.∂φik∂mijk.Based on Equation (29),
(33)∂RMSk∂δk=∂δ2k/2∂δk=δk.Based on Equation (28),
(34)∂δk∂Qk=∂r+γQk+1−Qk∂Qk=−1.Based on Equation (26),
(35)∂Qk∂φk=∂∑i=1mφkBqij∂φk=Bqij.Based on Equations (20), (21), and (22),
(36)∂φik∂mijk=φi−nik−mikσi2k.Substituting Equations (33), (34), (35), and (36) into Equation (32), we obtain
(37)∂RMSk∂mijk=∂RMSk∂δk·∂δk∂mijk=δkBqijφinik−mikσi2k.Substituting Equation (37) into Equations (30) and (31), the gradient descent linear function of ek can be further expressed as
(38)mijk∝−δkBqijφinik−mikσi2k,mijk+1=mijk−βmijkδkBqijφinik−mikσi2.The gradient descent linear function ofeck can be expressed as
(39)mcijk+1=mcijk−βmcijk∂RMSk∂mcijk,where
(40)∂RMSk∂mcijk=δkBqijφincik−mcikσi2k.βmcijk is the step-size of the gradient descent and mcijk is the cluster centre of the jth input variables of the ith fuzzy rule.Similarly, the gradient descent linear function ofσijk can be expressed as
(41)σijk∝−∂RMSk∂σijk,(42)σijk+1=σijk−βσijk∂RMSk∂σijk,where ∂RMSk/∂σijk is the gradient descent and βσijk is the step-size of the gradient descent. The gradient descent can be further expressed as
(43)∂RMSk∂σijk=∂RMSk∂δk·∂δk∂σijk=∂RMSk∂δk·∂δk∂Qk.∂Qk∂φik.∂φik∂σijk.Based on Equations (20), (21), and (22),
(44)∂φik∂σijk=φinik−mik2σij3k.Substituting Equations (33), (34), (35), and (44) into Equation (43), Equation (45) can be obtained as the following:
(45)∂RMSk∂σijk=−δkBqijφinik−mik2σij3k.Substituting Equation (45) into Equations (41) and (42), the gradient descent linear function of σijk can be expressed as
(46)σijk∝δkBqijφinik−mik2σij3k,σijk+1=σijk+βσijkδkBqijφinik−mik2σij3k.The width of the rate of the error changeσcijk can be expressed with the linear relation of gradient descent as
(47)σcijk+1=σcijk−βσcijk∂RMSk∂σcijk,where
(48)∂RMSk∂σcijk=−δkBqijφincik−mcik2σcij3k.By using the above equations, the most suitable centre and width of the membership functions were obtained to get accurately the disturbance distribution. Thus, all parameters of the FNNQL controller were optimized and adjusted, which makes the FNNQL controller have a strong fault tolerance and adaptability. Meanwhile, the FNNQL controller has a strong adaptability to deal with the disturbance change.


### 3.3. The FNNQL Strategy Steps

The flow chart of the FNNQL strategy is given in Figure11.Figure 11
FNNQL strategy steps.


## 3.1. Part I: Fuzzy Parameter Learning

In this part, since the FIS was introduced into the Q-learning, the optimal parameters of the FNNQL controller would be obtained. The structure of the FNNQL controller can be shown in Figure8, which can be divided into five layers. The 1st layer is the system input, which is followed by the 2nd layer, where the fuzzy is illustrated. And then, the 3rd layer is the activation operation, and the 4th layer is the action selection. Finally, the output results can be obtained in the 5th layer. The detailed description on each layer will be introduced in the following paragraphs.Figure 8
Structure of fuzzy neural network Q-learning based on RBF.


### 3.1.1. Input Layer

In this layer, to reflect the characteristics of disturbance change in real time, the error of the control state and the rate of the error change should be taken as the inputs of the FNNQL controller. According to Equations (1) and (10), the error of the control state can be described as
(12)ek=Yck−Ypk=Zck−Xpk.Based on Equation (12), the rate of the error change can be described as
(13)eck=ek+1−ek.According to Figure8, the inputs in this layer are also the previous parameter of the neural network FIS (NNFIS).


### 3.1.2. Fuzzy Layer

The inputs of the fuzzy layer are the outputs of the1st layer. They would be fuzzified in this layer. The quantization factors ke and kec are introduced into the fuzzified equations to map the inputs to the fuzzy domain accurately effectively. The fuzzified equations for ek and eck are defined, respectively, as
(14)Feek=keek,Fececk=kececk.In the process of fuzzification, the three fuzzy subsets are defined forek and eck, which can be defined as ek=B,N,S and eck=B,N,S. The membership function is Gauss.SupposeFe can be divided into p regions and Fec can be divided into q regions, which are defined, respectively, as
(15)S=s1,s2,⋯,sp,(16)SC=sc1,sc2,⋯,scq.Based on Equations (15) and (16), the number of the fuzzy rules can be defined as
(17)c=p×q.In this paper, the fuzzy membership function can be defined as(18)fx,σ,c=exp−x−m22σ2,where σ is the width of the membership function and m is the centre of the membership function. Then, the membership functions of the two fuzzy inputs ek and eck can be defined as
(19)μi=μeik,σik,mik=exp−eik−mik22σi2k,(20)μci=μcecik,σcik,mcik=exp−ecik−mcik22σci2k,where i represents the rule number, i=1,2,⋯,n; mik and σik are the centre and the width of μi, respectively; and mcik and σcik are the centre and the width of μci, respectively. The detailed inner structure of the ith node in the fuzzy layer can be seen in Figure 9.Figure 9
The inner structure of theith node.In this layer, the inputs of the control system are fuzzified. In the next step, the rate of the fuzzy rule activated should be calculated to get the accurate output of the FNNQL control system.


### 3.1.3. Activation Calculation Layer

First, theith fuzzy rule can be supposed as
(21)ifeikisnikandecikisncik,thenactionselectionisai,where ai is the optimal action selection of the FNNQL control.For measuring the usage of fuzzy rules in NNFIS, according to Equations (19) and (20), the rate of the fuzzy rule activated is defined as
(22)φi=∑i=1cμi×μci,where i represents the rule number and φi is the rate of the fuzzy rule activated. According to φi, the optimal action selection will be described with the ith fuzzy rule as an example in the following layer.


### 3.1.4. Action Selection Layer

In this layer, the accurate output of the FNNQL control system can be obtained based on the premise parameters, and each fuzzy input of the control system should obtain its optimal action. According to theε-greedy policy, the equation of the action selection can be defined as
(23)ai=Rqi,ε<φi,Bqi,ε≥φi,where Rqi∈A, A=a1,a2,⋯,ac, is the set of the random action; Bqi is the optimal action selection; ε is the action selection probability; and i is the number of the fuzzy rule. To assess the action selection, an additional penalty function will be used in this layer as shown in
(24)ri=φi+λ,eik≤ξ,φi−λ,eik>ξ,where ξ⟶0, λ∈0,1. If the stated error is within the accuracy range, a positive reward is obtained and the action selection is enhanced correspondingly. When it is not satisfied, a negative or zero reward is obtained, and the action selection is weakened instead. With using multiple trial-and-errors and ε-greedy, the relation of the mapping between the fuzzy inputs and the optimal action selection can be obtained. Therefore, the fuzzy rule base can be constructed. Since the state error is caused by model disturbance, the fuzzy rules obtained with trial-and-error are consistent with disturbance change. The structure of this layer can be shown in Figure 10.Figure 10
Structure of the action selection layer.


### 3.1.5. Output Layer

In this layer, theQ value table corresponding to the optimal continuous action selection can be obtained as the consequent parameters of NNFIS based on the fuzzy rule base constructed in Layer IV. According to the approximate Q-learning principle [25] and Equation (23), the value of the optimal continuous action selection can be defined as
(25)a=∑i=0nφi×Bqi.Similarly, theQ value function corresponding to the optimal continuous action selection can also be represented as
(26)Qj=∑j=12∑i=1nφji×Bqji,where Qj is the total of all Q value function and j is the number of the control states. Based on Equations (10), (21), and (25), the equation of the control system can be rewritten as
(27)Xpk+1=Xpk+B−1ΔB+aUk,where Uk=ek,eckT is the input vector of the FNNQL controller.In Part I: Fuzzy Parameter Learning, theQ value function Qj was obtained, which can be used to adjust the cluster centre and cluster width of the membership function in Part II: Fuzzy Parameter Adjustment.


## 3.1.1. Input Layer

In this layer, to reflect the characteristics of disturbance change in real time, the error of the control state and the rate of the error change should be taken as the inputs of the FNNQL controller. According to Equations (1) and (10), the error of the control state can be described as
(12)ek=Yck−Ypk=Zck−Xpk.Based on Equation (12), the rate of the error change can be described as
(13)eck=ek+1−ek.According to Figure8, the inputs in this layer are also the previous parameter of the neural network FIS (NNFIS).


## 3.1.2. Fuzzy Layer

The inputs of the fuzzy layer are the outputs of the1st layer. They would be fuzzified in this layer. The quantization factors ke and kec are introduced into the fuzzified equations to map the inputs to the fuzzy domain accurately effectively. The fuzzified equations for ek and eck are defined, respectively, as
(14)Feek=keek,Fececk=kececk.In the process of fuzzification, the three fuzzy subsets are defined forek and eck, which can be defined as ek=B,N,S and eck=B,N,S. The membership function is Gauss.SupposeFe can be divided into p regions and Fec can be divided into q regions, which are defined, respectively, as
(15)S=s1,s2,⋯,sp,(16)SC=sc1,sc2,⋯,scq.Based on Equations (15) and (16), the number of the fuzzy rules can be defined as
(17)c=p×q.In this paper, the fuzzy membership function can be defined as(18)fx,σ,c=exp−x−m22σ2,where σ is the width of the membership function and m is the centre of the membership function. Then, the membership functions of the two fuzzy inputs ek and eck can be defined as
(19)μi=μeik,σik,mik=exp−eik−mik22σi2k,(20)μci=μcecik,σcik,mcik=exp−ecik−mcik22σci2k,where i represents the rule number, i=1,2,⋯,n; mik and σik are the centre and the width of μi, respectively; and mcik and σcik are the centre and the width of μci, respectively. The detailed inner structure of the ith node in the fuzzy layer can be seen in Figure 9.Figure 9
The inner structure of theith node.In this layer, the inputs of the control system are fuzzified. In the next step, the rate of the fuzzy rule activated should be calculated to get the accurate output of the FNNQL control system.


## 3.1.3. Activation Calculation Layer

First, theith fuzzy rule can be supposed as
(21)ifeikisnikandecikisncik,thenactionselectionisai,where ai is the optimal action selection of the FNNQL control.For measuring the usage of fuzzy rules in NNFIS, according to Equations (19) and (20), the rate of the fuzzy rule activated is defined as
(22)φi=∑i=1cμi×μci,where i represents the rule number and φi is the rate of the fuzzy rule activated. According to φi, the optimal action selection will be described with the ith fuzzy rule as an example in the following layer.


## 3.1.4. Action Selection Layer

In this layer, the accurate output of the FNNQL control system can be obtained based on the premise parameters, and each fuzzy input of the control system should obtain its optimal action. According to theε-greedy policy, the equation of the action selection can be defined as
(23)ai=Rqi,ε<φi,Bqi,ε≥φi,where Rqi∈A, A=a1,a2,⋯,ac, is the set of the random action; Bqi is the optimal action selection; ε is the action selection probability; and i is the number of the fuzzy rule. To assess the action selection, an additional penalty function will be used in this layer as shown in
(24)ri=φi+λ,eik≤ξ,φi−λ,eik>ξ,where ξ⟶0, λ∈0,1. If the stated error is within the accuracy range, a positive reward is obtained and the action selection is enhanced correspondingly. When it is not satisfied, a negative or zero reward is obtained, and the action selection is weakened instead. With using multiple trial-and-errors and ε-greedy, the relation of the mapping between the fuzzy inputs and the optimal action selection can be obtained. Therefore, the fuzzy rule base can be constructed. Since the state error is caused by model disturbance, the fuzzy rules obtained with trial-and-error are consistent with disturbance change. The structure of this layer can be shown in Figure 10.Figure 10
Structure of the action selection layer.


## 3.1.5. Output Layer

In this layer, theQ value table corresponding to the optimal continuous action selection can be obtained as the consequent parameters of NNFIS based on the fuzzy rule base constructed in Layer IV. According to the approximate Q-learning principle [25] and Equation (23), the value of the optimal continuous action selection can be defined as
(25)a=∑i=0nφi×Bqi.Similarly, theQ value function corresponding to the optimal continuous action selection can also be represented as
(26)Qj=∑j=12∑i=1nφji×Bqji,where Qj is the total of all Q value function and j is the number of the control states. Based on Equations (10), (21), and (25), the equation of the control system can be rewritten as
(27)Xpk+1=Xpk+B−1ΔB+aUk,where Uk=ek,eckT is the input vector of the FNNQL controller.In Part I: Fuzzy Parameter Learning, theQ value function Qj was obtained, which can be used to adjust the cluster centre and cluster width of the membership function in Part II: Fuzzy Parameter Adjustment.


## 3.2. Part II: Fuzzy Parameter Adjustment

According to Equation (27), the new displacement XPk+1 can be obtained. And then, the new error ek+1 and the rate of the error eck+1 can be derived based on Equations (12) and (13). The reward value vector r can be obtained based on Equation (24). And then, according to the Bellman optimality theory [26], the time difference error of TD (0) can be described as
(28)δk=r+γQk+1−Qk,where γ is the discount rate.The mean square error (RMS) function, which is used to measure control system deformation accuracy, can be defined:(29)RMSk=12δ2k.In the FNNQL controller, since the membership function is Gauss RBF, the centre of an ideal RBF should be in line with the location of the fuzzy input that occurs most frequently, and all widths are exactly the boundary values of the highest frequency region of fuzzy input. Therefore, it is necessary to get the most suitable centre and width by adjusting the parameters so as to get accurately the disturbance distribution.According to Equation (29), the centre and width can be updated by the gradient descent [27, 28]. The gradient descent linear function of ek can be expressed as
(30)mijk∝−∂RMSk∂mijk,(31)mijk+1=mijk−βmijk∂RMSk∂mijk,where ∂RMSk/∂mijk is the gradient descent, βmijk is the step-size of the gradient descent, and mijk is the centre of the jth input variables of the ith fuzzy rule. The gradient descent can be further expressed as
(32)∂RMSk∂mijk=∂RMSk∂δk·∂δk∂mijk=∂RMSk∂δk·∂δk∂Qk.∂Qk∂φik.∂φik∂mijk.Based on Equation (29),
(33)∂RMSk∂δk=∂δ2k/2∂δk=δk.Based on Equation (28),
(34)∂δk∂Qk=∂r+γQk+1−Qk∂Qk=−1.Based on Equation (26),
(35)∂Qk∂φk=∂∑i=1mφkBqij∂φk=Bqij.Based on Equations (20), (21), and (22),
(36)∂φik∂mijk=φi−nik−mikσi2k.Substituting Equations (33), (34), (35), and (36) into Equation (32), we obtain
(37)∂RMSk∂mijk=∂RMSk∂δk·∂δk∂mijk=δkBqijφinik−mikσi2k.Substituting Equation (37) into Equations (30) and (31), the gradient descent linear function of ek can be further expressed as
(38)mijk∝−δkBqijφinik−mikσi2k,mijk+1=mijk−βmijkδkBqijφinik−mikσi2.The gradient descent linear function ofeck can be expressed as
(39)mcijk+1=mcijk−βmcijk∂RMSk∂mcijk,where
(40)∂RMSk∂mcijk=δkBqijφincik−mcikσi2k.βmcijk is the step-size of the gradient descent and mcijk is the cluster centre of the jth input variables of the ith fuzzy rule.Similarly, the gradient descent linear function ofσijk can be expressed as
(41)σijk∝−∂RMSk∂σijk,(42)σijk+1=σijk−βσijk∂RMSk∂σijk,where ∂RMSk/∂σijk is the gradient descent and βσijk is the step-size of the gradient descent. The gradient descent can be further expressed as
(43)∂RMSk∂σijk=∂RMSk∂δk·∂δk∂σijk=∂RMSk∂δk·∂δk∂Qk.∂Qk∂φik.∂φik∂σijk.Based on Equations (20), (21), and (22),
(44)∂φik∂σijk=φinik−mik2σij3k.Substituting Equations (33), (34), (35), and (44) into Equation (43), Equation (45) can be obtained as the following:
(45)∂RMSk∂σijk=−δkBqijφinik−mik2σij3k.Substituting Equation (45) into Equations (41) and (42), the gradient descent linear function of σijk can be expressed as
(46)σijk∝δkBqijφinik−mik2σij3k,σijk+1=σijk+βσijkδkBqijφinik−mik2σij3k.The width of the rate of the error changeσcijk can be expressed with the linear relation of gradient descent as
(47)σcijk+1=σcijk−βσcijk∂RMSk∂σcijk,where
(48)∂RMSk∂σcijk=−δkBqijφincik−mcik2σcij3k.By using the above equations, the most suitable centre and width of the membership functions were obtained to get accurately the disturbance distribution. Thus, all parameters of the FNNQL controller were optimized and adjusted, which makes the FNNQL controller have a strong fault tolerance and adaptability. Meanwhile, the FNNQL controller has a strong adaptability to deal with the disturbance change.


## 3.3. The FNNQL Strategy Steps

The flow chart of the FNNQL strategy is given in Figure11.Figure 11
FNNQL strategy steps.


## 4. Simulation

In this section, the middle plate of the 5-meter deployable antenna panel was taken as an example to test FNNQL strategy. The initial value ofXk, the transfer matrix B, and the disturbances of the uncertainties are the same as in Section 2. The parameters of the FNNQL controller can be given as follows: the step-size of the gradient descent is β=0.4. The quantization factors ke=0.1 and kec=0.05. The discount rate γ=0.8. The initial values of the centre m, the error change rate of centre mc, the width σ, and the error change rate of width σc can be found in Tables 5–8.Table 5
Initial values of clustering centrem.
Actuator
m1
m2
m3
m4
m5
u
B
-6
-6
-6
-6
-6
N
0
0
0
0
0
S
6
6
6
6
6
Actuator
m6
m7
m8
m9
m10
u
B
-6
-6
-6
-6
-6
N
0
0
0
0
0
S
6
6
6
6
6
Actuator
m11
m12
m13
m14
m15
u
B
-6
-6
-6
-6
-6
N
0
0
0
0
0
S
6
6
6
6
6
Actuator
m16
m17
m18
m19
m20
u
B
-6
-6
-6
-6
-6
N
0
0
0
0
0
S
6
6
6
6
6
Actuator
m21
m22
m23
m24
m25
u
B
-6
-6
-6
-6
-6
N
0
0
0
0
0
S
6
6
6
6
6Table 6
Initial values of the error change ratemc.
Actuator
mc1
mc2
mc3
mc4
mc5
u
B
-3
-3
-3
-3
-3
N
0
0
0
0
0
S
3
3
3
3
3
Actuator
mc6
mc7
mc8
mc9
mc10
u
B
-3
-3
-3
-3
-3
N
0
0
0
0
0
S
3
3
3
3
3
Actuator
mc11
mc12
mc13
mc14
mc15
u
B
-3
-3
-3
-3
-3
N
0
0
0
0
0
S
3
3
3
3
3
Actuator
mc16
mc17
mc18
mc19
mc20
u
B
-3
-3
-3
-3
-3
N
0
0
0
0
0
S
3
3
3
3
3
Actuator
mc21
mc22
mc23
mc24
mc25
u
B
-3
-3
-3
-3
-3
N
0
0
0
0
0
S
3
3
3
3
3Table 7
Initial values of clustering widthσ.
Actuator
σ1
σ2
σ3
σ4
σ5
u
B
1
1
1
1
1
N
1
1
1
1
1
S
1
1
1
1
1
Actuator
σ6
σ7
σ8
σ9
σ10
u
B
1
1
1
1
1
N
1
1
1
1
1
S
1
1
1
1
1
Actuator
σ11
σ12
σ13
σ14
σ15
u
B
1
1
1
1
1
N
1
1
1
1
1
S
1
1
1
1
1
Actuator
σ16
σ17
σ18
σ19
σ20
u
B
1
1
1
1
1
N
1
1
1
1
1
S
1
1
1
1
1
Actuator
σ21
σ22
σ23
σ24
σ25
u
B
1
1
1
1
1
N
1
1
1
1
1
S
1
1
1
1
1Table 8
Initial values of the error change rateσc.
Actuator
σc1
σc2
σc3
σc4
σc5
u
B
1
1
1
1
1
N
1
1
1
1
1
S
1
1
1
1
1
Actuator
σc6
σc7
σc8
σc9
σc10
u
B
1
1
1
1
1
N
1
1
1
1
1
S
1
1
1
1
1
Actuator
σc11
σc12
σc13
σc14
σc15
u
B
1
1
1
1
1
N
1
1
1
1
1
S
1
1
1
1
1
Actuator
σc16
σc17
σc18
σc19
σc20
u
B
1
1
1
1
1
N
1
1
1
1
1
S
1
1
1
1
1
Actuator
σc21
σc22
σc23
σc24
σc25
u
B
1
1
1
1
1
N
1
1
1
1
1
S
1
1
1
1
1In the case ofμc=B,N,S,μ=B,N,S, we can get 9 fuzzy sets for fuzzy control through some combinations. The fuzzy sets can be expressed as
(49)μ,μc∣μ∈B,N,S,μc∈B,N,S.To assess the effectiveness of the FNNQL controller on dealing with the disturbance change, the value functionsQk are trained according to the initial value, which can be given in Equation (11); the training curve is shown as Figure 12.Figure 12
Training curve of value function.As seen in Figure12, after 5000 random trial-and-errors, the curves of 25 nodes were all seen a tendency to narrow and they were kept down to just ±0.03. That further proved that the Q-learning system had been familiar with the corresponding relationship between control correction and disturbance change. Moreover, the following parameters are obtained for the stable system: the value functions of the 25 actuators could also be obtained, and the value function Qk of the 21st actuator is taken as an example, which can be found in Table 9. The centres and the error change rates of the centres, the width, and the error change rates of width of 25 actuators can be obtained in Tables 10–13, respectively.Table 9
Value functionQk of actuator 21.
ΔKP21
a1
a2
a3
a4
a5
i
B,B
-567
-153
-880.9
-123
-520
B,N
-22.4
-478.8
-483.5
-516.1
-266.4
B,S
-39.6
-189
-79.6
-25.6
-21.9
N,B
-709.4
-362.8
-1241
-529.6
-253.7
N,N
-475.7
-260.6
-58.0
-295.7
4.3
N,S
-266.1
-329.4
-63.4
-518.5
35.5
S,B
-377.3
-578.8
-846.6
-1070
-467.3
S,N
-157
-241.4
-22.1
-567.7
-326.0
S,S
-1112
-151.9
-420.9
-5.3
78.9
ΔKP21
a6
a7
a8
a9
a10
i
B,B
-351.6
-189.1
-203.1
-235.9
-516.1
B,N
-940.5
-6.0
-897.0
-643.9
-160.1
B,S
111.6
169.9
-402
-176
-189.3
N,B
-439.6
-574.8
-918.1
-852.1
-1673
N,N
-637.1
-987.2
-471.8
-357.4
-198.9
N,S
-580.2
-369.9
-67.6
118.4
-121.3
S,B
-609.5
-1318
-669.1
-959.4
-1216
S,N
-635.6
-249.0
-474.3
-224.5
-364.1
S,S
-350.5
-517.3
-46.9
-151.9
-314.5Table 10
Clustering centrem of 25 actuators.
Actuator
m1
m2
m3
m4
m5
u
B
42.6
-43.6
-30.1
-8.2
-7.2
N
-9.2
79.4
21.8
34.5
12.3
S
5.02
46.3
43.2
6.5
9.5
Actuator
m6
m7
m8
m9
m10
u
B
6.2
-32.1
-6.5
-6.8
-7.8
N
21.4
5.59
8.9
15.1
32.2
S
6.46
8.94
6.14
15.6
25.3
Actuator
m11
m12
m13
m14
m15
u
B
35.2
-15.2
-9.2
-8.6
-7.6
N
25.3
6.15
6.32
2.3
7.8
S
34.2
54.6
62.3
39.5
9.6
Actuator
m16
m17
m18
m19
m20
u
B
24.2
-62.9
-9.6
-16
-8.6
N
9.5
11.4
9.5
-5.6
-9.2
S
8.6
51.6
15.3
21.6
9.6
Actuator
m21
m22
m23
m24
m25
u
B
32.5
-10.6
-21.6
-3.6
-12.6
N
12.5
7.8
9.6
18.6
21.4
S
32.1
19.8
21.5
20.1
19.3Table 11
Error change ratemc.
Actuator
mc1
mc2
mc3
mc4
mc5
u
B
6.09
-14.1
-5.30
-76.7
-9.17
N
2.10
12.3
-0.9
-65.7
-1.27
S
0.96
0.91
-3.3
3.77
1.44
Actuator
mc6
mc7
mc8
mc9
mc10
u
B
-8.93
5.13
-3.70
-9.3
-12
N
-44.5
-11.8
-1.95
-9.11
-5.08
S
-12.7
4.17
0.00
5.03
4.79
Actuator
mc11
mc12
mc13
mc14
mc15
u
B
-8.3
-13
8.90
-23
-9.6
N
-9.11
6.30
-2.1
0.8
-12
S
4.28
3.14
12.3
13.6
10.3
Actuator
mc16
mc17
mc18
mc19
mc20
u
B
-14.5
9.5
21.3
35.2
9.5
N
9.6
15.2
-5.6
14.3
-6.3
S
-12.3
7.9
12.1
5.6
7.6
Actuator
mc21
mc22
mc23
mc24
mc25
u
B
34.2
47.6
36.4
-8.9
-21.3
N
-15
-7.8
-25.2
-16.9
-8.6
S
12.2
9.7
12.3
8.12
0.78Table 12
Clustering widthσ of 25 actuators.
Actuator
σ1
σ2
σ3
σ4
σ5
u
B
-8.92
3.95
-3.58
0.97
0.95
N
1.10
15.33
-0.68
2.96
5.86
S
0.93
0.94
-3.33
-0.06
1.05
Actuator
σ6
σ7
σ8
σ9
σ10
u
B
1.10
1.24
1.14
-9.78
3.96
N
-0.99
-0.92
0.26
2.10
25.45
S
1.65
1.08
2.57
0.89
0.89
Actuator
σ11
σ12
σ13
σ14
σ15
u
B
-458
0.97
0.95
1.10
1.24
N
-0.69
2.96
5.86
-1.2
-0.64
S
-3.36
-0.12
1.21
1.65
1.08
Actuator
σ16
σ17
σ18
σ19
σ20
u
B
1.21
-8.99
6.95
2.58
0.99
N
0.35
1.15
5.33
-0.67
2.96
S
3.57
0.89
0.92
-3.35
-0.06
Actuator
σ21
σ22
σ23
σ24
σ25
u
B
0.95
1.10
1.24
1.14
2.1
N
5.86
-0.99
-0.92
0.26
6.21
S
1.05
1.65
1.08
2.57
12.3Table 13
Error change rateσc.
Actuator
σc1
σc2
σc3
σc4
σc5
u
B
-0.31
-2.31
0.75
9.01
0.12
N
12.21
0.04
8.12
1.97
-0.28
S
0.58
0.01
1.64
0.53
39.77
Actuator
σc6
σc7
σc8
σc9
σc10
u
B
1.65
0.12
0.61
-0.31
-3.32
N
1.69
0.85
-1.67
12.21
0.06
S
0.87
0.88
-0.61
0.58
0.02
Actuator
σc11
σc12
σc13
σc14
σc15
u
B
0.78
9.08
0.45
2.65
0.16
N
9.12
2.00
-0.31
2.69
0.89
S
1.69
0.63
30.79
0.97
1.20
Actuator
σc16
σc17
σc18
σc19
σc20
u
B
079
1.65
0.12
0.61
-0.31
N
-3.67
1.69
0.85
-1.67
12.21
S
-0.69
0.87
0.88
-0.61
0.58
Actuator
σc21
σc22
σc23
σc24
σc25
u
B
-3.32
0.78
9.08
0.45
2.65
N
0.06
9.12
2.00
-0.31
2.69
S
0.02
1.69
0.63
30.79
0.97Next, the parameters which are found in Tables9–13 were applied to the deployable antenna active adjustment system with different disturbances to verify the adaptability of the FNNQL controller. In the active adjustment system of a deployable antenna panel with disturbances, which could be found in Table 3, the deformation was adjusted by the FNNQL controller and the adjustment curve was shown in Figures 13 and 14. Contrast Figure 13 with Figure 6, after adjustment by the FNNQL controller, the displacements of all adjustment actuators were below 10-3 mm after the 40 adjustments, and the RMS was below 10-3 mm when the system was regarded as stable.Figure 13
FNNQL adjustment curve with a disturbance of 30%.Figure 14
Root mean square (RMS) based on 25 actuators.Similarly, the active adjustment system of a deployable antenna panel with different disturbances was found, which could be found in Table4; the deformation was adjusted by the FNNQL controller, and the adjustment curve was shown in Figures 15 and 16, respectively. Contrast Figure 15 with Figure 7, after adjustment by the FNNQL controller, the displacements of all adjustment actuators were also below 10-3 mm after the 50 adjustments.Figure 15
FNNQL adjustment curve with a disturbance of 90%.Figure 16
Root mean square (RMS) based on 25 actuators.As seen in Figures14–16, the actuators’ displacements were adjusted by the FNNQL controller with the consideration of different disturbances and the RMS of the active adjustment system was below 10-3 mm when the system was regarded as stable. The results of the simulation in this paper indicated that the proposed method could be used to optimize the model disturbance change.


## 5. Conclusions

In this paper, the FNNQL control method was studied for the active deformation adjustment mechanism of a deployable antenna panel. Firstly, the error of the model disturbance was reduced because the input of Q-learning is fuzzified; the accuracy of state transition is improved. A fuzzy rule base of the coupled MIMO system was constructed using trial-and-error by Q-learning so that the accurate relationship between the premise and consequent parameters of the fuzzy system was obtained. Secondly, the width and the centre of the membership function were modified by the RBF neural network with the change of state environment. Finally, the effectiveness and the superiority of the proposed approach were demonstrated using the middle plate of the 5 m deployable antenna panel as an example. The results of the simulation indicated that the proposed method was an effective tool for the active deformation adjustment with disturbance change of the 5 m deployable antenna panel.However, in the paper, the value function was obtained after 5000 random trial-and-errors; the computing time of the control method is long due to the fact that the number of the trial-and-errors is big. How to reduce the time complexity of the method is our future research.



---

*Source: 6745045-2019-12-14.xml*