# Facilitating Robot Learning in Virtual Environments: A Deep Reinforcement Learning Framework
## 3 Design Patterns of Reinforcement Learning in Virtual Environments
Additionally, it incorporates the Emitter and Receiver classes provided by the Webots system to simulate communication between robots using string messages over virtual radio waves. In the proposed WebotsRL system, the Emitter and Receiver are used for communication between the SupervisorRobot and SlaveRobot, with the Emitter broadcasting messages and the Receiver receiving them. Figure 2: (**Left**) Simple design pattern with integrated Supervisor for robot RL implementation. (**Right**) Design pattern with separated Supervisor using Emitter–Receiver interface. Instances of SupervisorRobot are typically not typical robots and usually lack mass or physical properties in the simulation. For example, they can represent a computational device, such as a PC, that transmits actions to robots without interacting with the scene. Similarly, the Emitter and Receiver components could be simulated wireless devices, like Wi-Fi or Bluetooth modules. In this design pattern, the reinforcement learning framework operates as follows: the virtual environment is first reset to its initial state. Then, one or more SlaveRobot instances collect initial observations from the environment and use their Emitter to send the data to the SupervisorRobot. The SupervisorRobot, through its Receiver, gathers these observations and passes them to the RLModel instance. Additionally, the SupervisorRobot can augment the observations with extra information from the Webots Supervisor class, such as distances, absolute positions, and object speeds, which are inaccessible to the SlaveRobot through its sensors. The RLModel generates an action based on these observations, which is transmitted back to the SlaveRobot using the Emitter. The SlaveRobot then executes the action through its actuators. This process is repeated iteratively until the SlaveRobot achieves its objective or a termination condition, such as reaching the maximum episode count or violating predefined constraints, is met. Following a significant drawback of the second RL design pattern is its reliance on the SupervisorRobot class, which inherits from the Webots Supervisor class. While this inheritance simplifies the implementation of methods such as reset()--intended to reset the environment to its initial state--it renders the design impractical for real-world applications. The functionality provided by the Supervisor class violates physical constraints inherent in real-world scenarios, making it infeasible to transfer these capabilities from simulation to reality. This limitation underscores the intermediate nature of virtual robot simulations, which ultimately serve as a stepping stone toward physical implementations of RL systems. To overcome this constraint, we propose a third RL design pattern that eliminates all references to the Webots Supervisor class within the simulation code. Instead of utilizing a SupervisorRobot class instance, this pattern introduces the concept of an external server, referred to as the RobotServer, which is accessed through internet-based communication, such as URL links. As shown in Figure 3 (UML class diagram), the RobotServer handles RL tasks for robots lacking the computational capacity to process complex tasks, such as object detection during visual processing. Figure 3: Proposed design pattern to facilitate the transfer of reinforcement learning models from virtual simulations to real-world applications. Additionally, this design pattern introduces two types of robots: (1) RLRobot instances, which possess the computational power to perform RL tasks locally within the simulation, and (2) Robot instances, which rely solely on their sensors and actuators. These Robot instances communicate with the RobotServer via the Communicator class over the internet or interact with other robots using the Emitter and Receiver classes.