# Comprehensive Review of Robotics Operating System-Based Reinforcement Learning in Robotics
## 2 Background
### Reinforcement Learning (RL) in Robotics
Bellman equations are described as follows: Equation (1) refers to the Value of state (V), and Equation (2) refers to the Quality of state (Q):  
\[V_{\pi}(s)=\sum_{a}\pi(a|s)\sum_{s^{\prime}}P(s^{\prime}|s,a)\left[R(s,a,s^{ \prime})+\gamma V_{\pi}(s^{\prime})\right] \tag{1}\]  
\[Q_{\pi}(s,a)=\sum_{s^{\prime}}P(s^{\prime}|s,a)\left[R(s,a,s^{\prime})+\gamma \sum_{a^{\prime}}\pi(a^{\prime}|s^{\prime})Q_{\pi}(s^{\prime},a^{\prime})\right] \tag{2}\]  
where \(s\) denotes the current state, \(s^{\prime}\) denotes the next state, \(a\) denotes the set of actions for the current state \(s\), \(P\) represents the probability of transitioning to the next state, \(R\) represents the reward function that specifies the immediate reward received after the taken action, \(\pi\) denotes the learning policy that maps states to actions in order to maximize cumulative rewards, and \(\gamma\) denotes the discount factor for the accumulation of future rewards. For a complete mathematical study, please see [30; 31]. Bellman equations are based on Markov Decision Processes (MDPs). In an MDP, only the current state is used for learning a policy that defines the agent's behavior. Bellman  
Figure 1: Overview of the reinforcement learning process. equations are used to compute the optimal value of a state by evaluating immediate and future rewards from potential actions. MDPs provide a systematic approach to decision making under uncertainty, integrating the principles of incentives, policies, and discount factors. This helps assess various tactics and identify the most effective course of action in dynamic contexts, ensuring the optimal outcome for the chosen actions [30; 31]. Various RL algorithms were developed as a result of this pioneering work, and we list some of the most popular ones:  
1. _The Temporal Difference (TD)_ algorithm enhances MDPs by incorporating the Monte Carlo (MC) algorithm and dynamic programming, facilitating agents' learning [31; 32; 33]. 2. _The State Action Reward State Action (SARSA)_ algorithm, based on the MDP, uses the agent's current policy to select actions and update their value estimates [34]. 3. _The Q-Learning_ algorithm improves Q-value estimates. This lets agents find the best policies to learn about action-state pairs without needing a model of the environment [35; 36]. 4. _The Deep Q-Networks (DQNs)_ algorithm enhances _Q-learning_ by utilizing _DL_ networks to manage high-dimensional state spaces, marking a significant advance in RL [36; 37]. 5. _Policy Gradient (PG)_ algorithms optimize policies by maximizing expected returns through gradient ascent, enhancing the likelihood of choosing actions with higher rewards and making them suitable for robotics applications [38]. 6. _The Actor-Critic (AC)_ algorithm uses a policy gradient approach, combining value-based and policy-based methods, to help actors achieve optimal policy choices [34]. 7. _The Soft Actor-Critic (SAC)_ algorithm integrates value-based and policy-based methodologies to maximize rewards and promote exploration in continuous action spaces, therefore enhancing learning efficiency [39]. 8. _The Trust Region Policy Optimization (TRPO)_ method is a policy gradient method that improves policies by limiting policy changes with each update. It does this using the Kullback-Leibler divergence (K-L Divergence) to stop big changes that could lead to poor performance [40]. K-L Divergence quantifies the divergence of one distribution of probabilities from a second reference distribution of possibilities [41]. 9. _The Deep Deterministic Policy Gradient (DDPG)_ algorithm integrates _PG_ methods with value-based strategies and is designed for continuous action spaces [42]. 10. _Proximal Policy Optimization (PPO)_ combines _DL_ and _PG_ methods together, which helps in finding the best policy in changing situations with clear incentives [43]. 11.