# Comprehensive Review of Robotics Operating System-Based Reinforcement Learning in Robotics
## 2 Background
### Robot Operating System (ROS)  
Robotic systems generally involve a multitude of hardware components, such as sensors, actuators, complex hardware, and software interfaces. The development of advanced middleware platforms such as ROS has changed the design, development, and research practices in robotics. In the past, roboticists had to learn multiple disciplines to manage the robot, such as share sensor data, programming actuators, and map data communication within the robotic platform. With ROS, roboticists are able to accelerate the process of designing, simulating, and deploying robotic systems across different application domains. The ROS provides standardized interface and communication protocols that enable the integration of various robot hardware components, wherein it is stated that the _\"ROS is not an operating system in the traditional sense of process management and scheduling; rather, it provides a structured communications layer above the host operating systems of a heterogenous compute cluster\"_[4]. ROS middleware is crucial for guiding robot actions effectively in various domains [55]. The ROS is a multipurpose middleware for developing robotic applications, and its modular system aims at improving productivity and promoting innovation in robotic systems. The modular architecture of the ROS promotes code reutilization and allows roboticists to switch hardware components with minor code alterations. The ROS enables the seamless integration of diverse sensors and actuators in several robotic systems. The modularity of the ROS middleware offers advantages by enabling developers to break down complex robotic systems into manageable components, promoting code exchange, reuse, and scalability in robotic software development. Additionally, the ROS addresses scalability concerns by supporting large systems and several robot configurations. The ROS provides libraries and tools that simplify complex behaviors on various robot platforms, encompassing hardware abstraction, device drivers, and package setup [56]. The ROS also benefits from a robust and active community that provides a wide range of tools and tutorials to help developers and researchers. ROS versions have been enhanced to address the shortcomings of earlier iterations, improving functionality by enhancing real-time performance, security protocols, multi-robot support, and cross-platform compatibility. ROS 1 has been upgraded to ROS 2 to address some of its shortcomings by using contemporary technologies for efficient communication [29; 56; 57], as shown in Table 3.  
Training robots in virtual scenarios using simulators has been essential prior to deploying them in real-world applications. Using simulation-in-loop prototyping tools, such as Gazebo, provides a cost-effective approach for testing and building robots for real-world \begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**Feature** & **ROS 1** & **ROS 2** \\ \hline Middleware, Communication, and & \begin{tabular}{l} Standard communication layer and relies \\ on TCP and UDP for communication \\ with basic security features \\ \end{tabular} & \begin{tabular}{l} Improved communication uses layer and \\ relies on DDS, its built-in authentication, \\ encryption, and access control \\ \end{tabular} \\ \hline Platform Support & Supports Ubuntu and Debian & \begin{tabular}{l} Supports Windows, macOS, and \\ Real-Time Operating Systems \\ \end{tabular} \\ \hline Node Management & \begin{tabular}{l} Centralized system with single \\ master node \\ \end{tabular} & \
\begin{tabular}{l} Decentralized system without \\ master node \\ \end{tabular} \\ \hline API and Libraries & Custom APIs and libraries & Standard libraries like C++, Python \\ \hline Backward Compatibility & Not Applicable & Engineered to co-exist with ROS1 \\ \hline Testing and Simulation &
\begin{tabular}{l} Supports simulation framework like \\ Gazebo and RViz \\ \end{tabular} & Supports newer simulation framework \\ \hline \hline \end{tabular}
\end{table}