# Efficient Robot Skills Learning with Weighted Near-Optimal Experiences Policy Optimization
## 2 Preliminary  
In this paper, the reason why WNEPO is introduced in the background of a biped robot walking task is that the state and action dimensions of this task are high, and the system dynamics are nonlinear [21], which puts forward higher requirements for the learning algorithm. Scholars have proposed gait planning methods based on different principles, such as methods based on a mathematical model [22], imitating human walking characteristics [23] and a central pattern generator [24]. All the methods above need accurate modeling and have poor generalization. The RL-based method proposed in this paper can provide some ideas for solving these problems. Our goal is to have the robot learn to walk in a straight line using minimal control effort without any prior information. As a sequential decision-making problem, this walking task can be modeled as a Markov decision process, where at any moment \(t\), the agent in state \(s_{t}\) selects action \(a_{t}\) with probability \(\pi_{\theta}(a_{t}|s_{t})\), causing the environment to enter a new state \(s_{t+1}\) with state transition probability \(p(s_{t+1}|s_{t},a_{t})\), and then receives an instantaneous reward \(r_{t}\)[25]. In this paper, we use \(u\) and \(x\) instead of \(a\) and \(s\) as the control variable and system state, respectively, to be consistent with the conventions in optimal control. A trajectory \(\tau=\{x_{0},u_{0},\ldots,x_{t},u_{t},x_{t+1},\ldots\}\) is a sequence of state-action pairs \((x_{t},u_{t})\) along the timeline from \(x_{0}\) until the biped robot enters a terminal state. The quality of robot walking can be evaluated by calculating the reward \(R(\tau)\) of trajectory \(\tau\). The reward function used in this paper is inspired by reference [25]. The compound probability of a possible trajectory \(\tau\) can be expressed as \(p_{\theta}(\tau)=p(x_{0})\prod\tau_{\theta}(u_{t}|x_{t})p(x_{t+1}|x_{t},u_{t})\), where \(\pi_{\theta}(u_{t}|x_{t})\) is the policy (or probability) of choosing action \(u_{t}\) in state \(x_{t}\) with the parameter \(\theta\). A real trajectory of robot interaction with the environment \(\tau\) is an experience or a sampling trajectory which is stored in the replay buffer for policy learning. The goal of reinforcement learning is to optimize the parameter \(\theta\) in policy \(\pi_{\theta}(u_{t}|x_{t})\) that maximizes the expectation of the cumulative reward, which is an optimization problem of Equation (1) [26]:  
\[ \begin{split}\theta^{*}=\underset{\theta}{\text{argmax}}J^{\pi} (\theta)=\underset{\theta}{\text{argmax}}E_{\tau\sim p_{\theta}(\tau)}[R(\tau )]=\underset{\tau}{\text{argmax}}\sum_{\tau}R(\tau)p_{\theta}(\tau)\\ R(\tau)=\sum_{t}\gamma^{t}r_{t}\end{split}  \tag{1}\]  
where \(r_{t}\) is the reward at time \(t\), and the discount factor \(\gamma\) is a real value in \([0,1]\). The following of this section focuses on the policy gradient approach, which is widely used in model-free RL and is advantageous in solving high-dimensional robotic problems. The policy gradient method estimates the gradient of \(J^{\pi}(\theta)\) and then updates the parameters \(\theta\) using the mini-batch gradient descent. Because policy and environment dynamics are independent of each other, the formula for policy gradient is as follows. \[ \begin{split}&\
abla_{\theta}J^{\pi}=\int\
abla_{\theta}p_{ \theta}(\tau)R(\tau)d\tau=\int p_{\theta}(\tau)R(\tau)\
abla_{\theta}\log p_{ \theta}(\tau)d\tau\\ &=E_{\tau\sim p_{\theta}(\tau)}[R(\tau)\
abla_{\theta}\log p_{ \theta}(\tau)]\\ &=E_{\tau\sim p_{\theta}(\tau)}[\sum_{t}\
abla_{\theta}\log\pi_{ \theta}(u_{t}|x_{t})R(\tau)]\\ &\approx\sum_{n=1}^{N}\sum_{t=1}^{T_{n}}R(\tau^{n})\
abla_{\theta }\log\pi_{\theta}(u_{t}^{n}|x_{t}^{n})\end{split}  \tag{2}\]  
Then, we can update \(\theta\) iteratively by calculating \(\
abla_{\theta}J^{\pi}\) using \((u_{t}^{n},x_{t}^{n},R(\tau^{n}))\) tuples, which is called the minimum decision unit (MDU) in this paper.