# Data-Driven Robust Control Using Reinforcement Learning
## 2 Materials and Methods
### Robust Control Using Reinforcement Learning
#### 2.1.1 Estimation of the Value Function by the Critics  
In the RL context, the accumulation of cost over time, when starting in the state \(x(t)\) and following policy \(\pi\), is defined as the value function of policy \(\pi\), i.e.,  
\[V^{\pi}(x(t))=E_{\pi}\bigg{\{}\int_{t}^{\infty}\gamma^{\tau-t}r(\tau)d\tau \bigg{\}}, \tag{2}\]  
where \(\gamma\) is the discount factor. The cost \(r(t)\) is assumed to be a quadratic function of the states:  
\[r(t)=x^{T}(t)Qx(t), \tag{3}\]  
where the positive definite matrix \(Q\in\mathbb{R}^{n\times n}\) is symmetric, positive semidefinite (since the cost is assumed to be non-negative), and contains the weighting factors of the variables that are minimized.  
In order to facilitate the formulation of the stability condition in the form of linear matrix inequalities (LMI), the value function \(V(x(t))\) is approximated by a quadratic function of the states:  
\[V^{\pi}(x(t))\approx x^{T}(t)Px(t), \tag{4}\]  
where the kernel matrix \(P\in\mathbb{R}^{n\times n}\) is symmetric and positive semidefinite (since matrix \(Q\) in the cost function is symmetric and positive semidefinite).  
By using the Kronecker operation, the approximated value function can be expressed as a linear combination of the basis function \(\phi(x(t))=(x(t)\otimes x(t))\):  
\[V^{\pi}(x(t)) \approx x^{T}(t)Px(t)=vec(P)^{T}(x(t)\otimes x(t)) \tag{5}\] \[=w^{T}(x(t)\otimes x(t))=w^{T}\phi(x(t)),\]  
where \(w\) is the parameter vector, \(\phi(x(t))\) is the vector of basis functions, and \(\otimes\) is the Kronecker product. The transformation between \(w\) and \(P\) can be performed as follows:  
\[w=vec(P)=[P_{11},P_{21},\dots,P_{n1},P_{12},\dots,P_{1n},P_{nn}]^{T},\]where \(P_{i,j}\) is the element of matrix \(P\) in the \(i\)th row and \(j\)th column. With \(T\) as the interval time for data sampling, the integral RL Bellman equation can be used to update the value function [8]:  
\[V^{\pi}(x(t))=\int_{t}^{t+T}\gamma^{\tau-t}r(\tau)d\tau+V^{\pi}(x(t+T)). \tag{7}\]  
By using the quadratic cost function (Equation (3)) and the approximated value function (Equation (5)), the integral RL Bellman equation can be written as follows:  
\[x^{T}(t)Px(t)=\int_{t}^{t+T}x(\tau)^{T}Qx(\tau)d\tau+x^{T}(t+T)Px(t+T) \tag{8}\]  
or  
\[w^{T}\phi(x(t))=\int_{t}^{t+T}x(\tau)^{T}Qx(\tau)d\tau+w^{T}\phi(x(t+T)). \tag{9}\]  
At each iteration, \(n\) samples along the state trajectory are collected \((x^{(1)}(t),x^{(2)}(t),\)\(\ldots,x^{(n)}(t))\). The mean value of \(w\) can be obtained by using least-square technique:  
\[\hat{w}=(XX^{T})XY, \tag{10}\]  
where  
\[X=[\phi_{\Delta}^{1}\quad\phi_{\Delta}^{2}\quad\ldots\quad\phi_{\Delta}^{N}]^ {T}, \tag{11}\]  
\[\phi_{\Delta}^{i}=\phi(x^{i}(t))-\phi(x^{i}(t+T)), \tag{12}\]  
\[Y=[d(x^{1}(t))\quad d(x^{2}(t))\quad\ldots\quad d(x^{n}(t))]^{T} \tag{13}\]  
and  
\[d(x^{i}(t))=\int_{t}^{t+T}x^{i}(\tau)^{T}Qx^{i}(\tau)d\tau \tag{14}\]  
with \(i=1,2,\ldots,N\).  
The confidence interval for the coefficient \(w^{(j)}\) is given by  
\[w^{(j)}\in[\hat{w}^{(j)}-q_{1-\frac{\theta}{2}}\sqrt{\tau_{j}\phi^{2}},\hat{w }^{(j)}+q_{1-\frac{\theta}{2}}\sqrt{\tau_{j}\phi^{2}}], \tag{15}\]  
where \(1-\theta\) is the confidence level, \(q_{1-\frac{\theta}{2}}\) is the quantile function of standard normal distribution, \(\tau_{j}\) is the jth element on the diagonal of \((XX^{T})^{-1}\), and \(\phi^{2}=\frac{\delta^{T}\delta}{n-\rho}\), with \(\epsilon=Y-\hat{w}X\). From that, the uncertainty \(\Delta w\) is defined as the deviation interval around the nominal value:  
\[\Delta w=\Big{[}-q_{1-\frac{\theta}{2}}\sqrt{\tau_{j}\phi^{2}},\ -q_{1-\frac{ \theta}{2}}\sqrt{\tau_{j}\phi^{2}}\Big{]}. \tag{16}\]  
Matrices \(\hat{P}\) and \(\Delta P\) can be obtained by placing elements of \(\hat{w}\) and \(\Delta w\) into columns.