# Comprehensive Review of Robotics Operating System-Based Reinforcement Learning in Robotics
## 2 Background
### Reinforcement Learning (RL) in Robotics
_Direct Preference Optimization (DPO)_ is a novel RL method that uses human feedback to identify the optimal policy with minimal loss of classification, making it reliable, efficient, and computationally lightweight [44]. Determining the best RL approaches includes observing adaptive behaviors and evaluating action benefits in real-world applications by using statistical and dynamic programming techniques [27]. There are different models that help agents (robots) acquire the appropriate behaviors for complex tasks without direct programming. RL models are categorized into three distinct types: model-free, model-based, and hybrid models. Each of these models is suitable for different types of applications. They are described in what follows:  
1. _The model-free RL_ is a process of learning an optimal policy using perception data only without relying on a trained model. Applying variance reduction techniques and addressing average reward problems simplifies the understanding of long-term rewards [45]. 2. _The model-based RL_ entails a process of learning an optimal policy by creating a model of the environment. Because the environment model is available, behavior predictions improve the use of robotic systems and can provide intricate estimations of rewards and future states, including potentially risky evaluations [28]. 3. _The hybrid model_ entails a process that brings together the advantages of model-free and model-based RL to efficiently manage data and enhance the robustness of the model against errors encountered during robot operation [46; 47]. Comparison of Popular RL Algorithms  
Figure 2 provides a pictorial landscape of the popular RL algorithms. We first divide the class of RL algorithms into two broad categories of model-based and model-free algorithms. Here, the model refers to the model of the environment. If a model for the environment in which the agent operates in can be represented, then this results in better sample efficiency in the learning process. An example of such a model-based framework is AlphaZero [48], which was developed by DeepMind to learn the games of chess, shogi, and go via RL. The advantage in this model-based approach is that the agent can analyze several moves in the future and thus learn a better policy. Other popular model-based RL algorithms include I2A [49], MBMF [50], and MBVE [51]. For robotics applications, the model-based RL is not effective because a model of the environment in which the robot will operate is usually not available to the agent. Learning the model from experiencing with the environment may not produce a completely effective model of the environment, resulting in an agent which performs sub-optimally, as model learning is fundamentally difficult. In model-free approaches, several popular algorithms have been developed that can be broadly categorized into direct policy learning, indirectly learning the policy by first learning a Q function, or employing a value function for the states involved. Some of the early deep-learning-network-based RL algorithms were based on _DQN_. Here, the deep network receives the state and the action as an input and outputs the Q-value. The training of a _DQN_ is usually used in conjunction with an experience replay that stores the steps of the episode in memory that is referred to as offline learning. During the training process, the samples are drawn from the replay memory at random. The _DQN_ was further improved to use two networks. Further on, two variations of this emerged: one referred to as the double DDQN and the other as the _Dueling DQN (DDQN)_[52]. _The Double DQN_ helps in reducing the fluctuations of Q-values and, as a result, trains faster with more stable learning. _The Dueling DQN_ was designed to decompose Q(a,s) into the value and advantage components to further improve Q-learning. It decomposes the network in two paths and then merges the two as Q(a,s) = V(s) + A(s,a).