## 4 Two-Time Scale Control
### Slow Subsystem and the Corresponding Controller  
The buffer compliance control based on reinforcement learning is shown in Figure 4, Where ASN is used to approximate the unknown nonlinear term of the system, ACN is used to construct reinforcement signals to optimize ASN. Define the trajectory tracking error as:  
\[\mathbf{e}=\mathbf{q}_{\theta d}-\mathbf{q}_{\theta} \tag{27}\]  
where \(\mathbf{q}_{\theta d}\in\mathbf{R}^{3\times 1}\) are desired trajectories of the hybrid system. At the same time, the error evaluation signal is defined as:  
\[\mathbf{z}=\dot{\mathbf{e}}+\mathbf{\Lambda}\mathbf{e} \tag{28}\]  
where \(\mathbf{\Lambda}\in\mathbf{R}^{3\times 3}\) is a positive definite diagonal matrix. Invoking Equations (27) and (28), the dynamic equation of the slow subsystem can be written as:  
\[\mathbf{D}_{x\theta}\dot{\mathbf{z}}=-\mathbf{C}_{x\theta}\mathbf{z}+\mathbf{d}-\mathbf{\tau}_{x\theta} \tag{29}\]  
where \(\mathbf{d}=\mathbf{D}_{x\theta}(\tilde{\mathbf{q}}_{\theta d}+\mathbf{\Lambda}\dot{\mathbf{e}})+\mathbf{ C}_{x\theta}(\tilde{\mathbf{q}}_{\theta d}+\mathbf{\Lambda}\mathbf{e})\) is the unknown nonlinear term of the system. Considering that it cannot be obtained directly, it can be approximated by the ASN:  
\[\mathbf{d}=\mathbf{W}_{a}^{\mathrm{T}}\mathbf{\Phi}(\mathbf{x})+\xi(\mathbf{x}) \tag{30}\]  
where \(\mathbf{W}_{a}\in\mathbb{R}^{n\times 3}\) is the ideal weight matrix of a radial basis function neural network (RBFNN), \(\xi(\mathbf{x})\) is the optimal approximation error. The radial basis kernel functions \(\mathbf{\Phi}(\mathbf{x})=[\Phi_{1},\Phi_{2},\cdots\Phi_{n}]^{\mathrm{T}}\) are represented by a Gaussian radial basis function (GRBF) as:  
\[\mathbf{\Phi}(\mathbf{x})=\exp(\frac{\left\|\mathbf{x}-\mathbf{c}\right\|^{2}}{2\sigma^{2}}) \tag{31}\]  
where \(\mathbf{x}=[\mathbf{q}_{\theta}^{\mathrm{T}},\tilde{\mathbf{q}}_{\theta}^{\mathrm{T}}, \tilde{\mathbf{q}}_{\theta d}^{\mathrm{T}},\tilde{\mathbf{q}}_{\theta d}^{\mathrm{T}}] ^{\mathrm{T}}\), \(\mathbf{c}\) and \(\sigma\) are the variance and the centre vector of the GRBF. On this basis, the slow rigid motion subsystem control law is given as:  
\[\mathbf{\tau}_{x\theta}=\mathbf{\hat{W}}_{a}^{\mathrm{T}}\mathbf{\Phi}(\mathbf{x})+\mathbf{K}_{z} \mathbf{z}+\mathbf{\tau}_{a} \tag{32}\]  
where \(\hat{W}_{a}\) is the estimate of the ideal weight \(\mathbf{W}_{a}\). Defining the estimation error \(\mathbf{\tilde{W}}_{a}=\mathbf{W}_{a}-\hat{W}_{a}\), it satisfies \(\dot{\mathbf{\tilde{W}}}_{a}=-\hat{W}_{a}\). \(\mathbf{K}_{z}\in\mathbb{R}^{3\times 3}\) is a positive definite diagonal matrix. \(\mathbf{\tau}_{a}\) is a robust control law, which is defined as:  
\[\mathbf{\tau}_{a}=\mathbf{K}_{a}\mathbf{z}/\left\|\mathbf{z}\right\| \tag{33}\]  
where \(\mathbf{K}_{a}\in\mathbb{R}^{3\times 3}\) is a positive definite diagonal matrix. Figure 4: The buffer compliance control block diagram based on Reinforcement Learning. Substituting Equations (32) and (33) into Equation (29), we have:  
\[\mathbf{D}_{x\theta}\dot{\mathbf{z}}=-(C_{x\theta}+K_{z})\mathbf{z}-K_{a}\mathbf{z}/\|\mathbf{z}\|+ \widetilde{W}_{a}\mathbf{\Phi}+\varsigma(\mathbf{x}) \tag{34}\]  
In order to optimize ASN, reinforcement learning signals are defined by CAN:  
\[\mathbf{r}=\mathbf{z}+\|\mathbf{z}\|\widetilde{\mathbf{W}}_{c}^{\mathrm{T}}\mathbf{\Phi}(\mathbf{x}) \tag{35}\]  
where \(\widetilde{W}_{c}\in\mathbb{R}^{m\times 3}\) is the estimate of the ideal weight \(W_{c}\). **Assumption 1**. : _The ideal weights \(\mathbf{W}_{a}\), \(\mathbf{W}_{c}\) are bounded and satisfy:_  
\[\mathbf{z}=\dot{\mathbf{e}}+\mathbf{\Lambda}\mathbf{e} \tag{28}\]0 \[\mathbf{z}=\dot{\mathbf{e}}+\mathbf{\Lambda}\mathbf{e} \tag{28}\]1  
_where \(W_{aM}\), \(W_{cM}\) is an unknown positive constant._  
**Assumption 2**.