# RL-Based Sim2 Real Enhancements for Autonomous Beach-Cleaning Agents
## 6 Experiments and Results
### First Experiment: Navigating an Obstacle-Free Environment  
In the primary experiment, the capabilities of the RL models, PPO, and DDPG, were evaluated to guide a mobile robot towards a set target within a clear space. This foundational test of positional control was designed to simulate the basic task of beach cleaning, challenging the agents to execute precise navigational maneuvers without the complexity of obstacles. As depicted in Figure 11, the simulation environment places the Khepera IV robot at an opposing start point to the target, set one meter apart, demanding precision in  
Figure 10: Training progression—mean episode reward graph. execution for successful task completion, analogous to directing a beach-cleaning robot to specific locations on the sand. Figure 12 illustrates the trajectory paths traversed by the DDPG and PPO agents in simulated and actual environments. The paths showcase the strategic movements from the starting point at the origin (0,0), marked by the robot, to the target at (1,0), highlighted by a red circle. The robot's initial position, oriented away from the target, required a calculated turn, as evidenced by the early curvature of the paths, before proceeding directly towards the endpoint. This maneuvering efficiency is crucial for a beach-cleaning robot that needs to navigate to various locations on a beach while avoiding static and dynamic obstacles. The trajectory graph affirms the success of the Sim2 Real transfer process, as both agents reliably achieved the core objective of reaching the target point. This successful transfer from simulation to real-world performance is promising for ongoing research and indicative of the potential applicability of the RL strategies to the domain of autonomous beach-cleaning robots. The ability to maintain consistent performance across both domains suggests that the refined control policies developed through simulation could effectively translate to a beach-cleaning robot navigating in a natural and less predictable environment. In this work, the evaluation of the algorithms is structured around four specific performance indices, which are analyzed in scenarios both with and without obstacles. These  
Figure 11: Simulated environment of the first experiment. Figure 12: Trajectory plot of the first experiment. indices include the Integral of Absolute Error (IAE), which measures the total magnitude of error; the Integral of Squared Error (ISE), assessing the cumulative squared error; the Integral of Time Multiplied by Absolute Error (ITAE), reflecting the error magnitude over time; and the Integral of Time Multiplied by Squared Error (ITSE), which combines the error's square with its duration. For these performance indices, the error used is the distance (\(d\)) from the robot to the TP. These metrics provide a comprehensive framework for quantifying the effectiveness of the algorithms under varying conditions [48]. \[IAE=\int_{0}^{\infty}\lvert e(t)\rvert dt \tag{5}\]  
\[ISE=\int_{0}^{\infty}e^{2}(t)dt \tag{6}\]  
\[ITAE=\int_{0}^{\infty}te^{2}(t)dt \tag{7}\]  
\[ITSE=\int_{0}^{\infty}t\lvert e(t)\rvert dt \tag{8}\]  
Table 1 presents a comparison of performance metrics. The purpose of this comparative table is to establish a metric for evaluating the agent's adaptability to different environments. Specifically, when the performance indices exhibit similarity or equality between the agent in the real and simulated environments, it indicates that the agent's behavior remains consistent across both settings. This alignment in performance metrics signifies a successful Sim2 Real transfer, suggesting that the agent is well-adjusted to the complexities of the real-world environment. The data comparison indicates that both DDPG and PPO agents are competent in directing a robot towards a target, which is fundamental for a beach-cleaning robot's task of reaching specific areas for waste collection.