# Facilitating Robot Learning in Virtual Environments: A Deep Reinforcement Learning Framework
## 5 Example Robots and Experiment Results  
To evaluate the RL patterns from Section 2 and the digital twin method from Section 3, we selected two Webots environments, as shown in Figure 5. Both environments involve the inverted pendulum problem [30]. The first (Figure 5, left) replicates the Deepbots reinforcement learning framework for Webots [28]. It consists of a cart with a one-meter pole attached via a free hinge, equipped with a sensor to measure the pole's angle. The task requires maintaining the pole in a vertical position by moving the cart forward or backward, using a discrete proximal policy optimization algorithm [31]. The observation space includes the cart's position and velocity, the pole's angle, and its angular velocity. The agent selects between two actions--moving forward or backward--and receives a reward of +1 per step, including the termination step. Episodes terminate after 1950 steps or when the pole falls or the cart moves beyond \(\pm\)0.4 m. A task is considered solved if the agent achieves an average score above 1950 over 100 consecutive episodes.  
We used this environment to validate the consistency of our framework against results from [28] and to assess the impact of Webots \"speed-up simulation\" mode on learning efficiency. The PPO agent, implemented with a two-layer neural network (10 ReLU neurons per layer), successfully solved the problem within approximately 3.5 h of simulated time, consistent with prior work. However, running the simulation in \"speed-up\" mode reduced execution time to less than 10 min without affecting performance. The learning curve (Figure 6, left) confirms alignment with previous results.  
The second environment utilizes a Webots model of the Pioneer 3-AT, a four-wheel, skid-steer robot. This platform is selected for its suitability in reinforcement learning experiments due to its well-defined action space (skid-steer drive with motor control), sensor availability (wheel encoders), and robust physical characteristics (12 kg weight, 0.7 m/s max speed, 35% max traversable grade). The robot's microcontroller and I/O capabilities, including digital and analog inputs, allow for diverse sensor integration and control. These features facilitate the development and testing of RL algorithms for tasks involving navigation and manipulation in varied terrains.  
Figure 5: Two Webots environments used to test the methods presented in this paper.  
The Pioneer 3-AT Webots model closely replicates its physical counterpart. This study evaluates whether an RL algorithm can achieve stable inverted pendulum control, similar to the toy cart example. Figure 6 (right) shows the learning curve over the first 1 million steps, where the model required approximately 30,000 training episodes but achieved only around 60 cumulative rewards per episode, indicating slower learning compared to the simpler system. To test long-term learning potential, we extended training to 32 million steps, randomizing the initial pendulum angle for each episode to enhance realism. We observed faster convergence when initial conditions matched the toy cart experiment, but by randomizing the initial pendulum angle in each episode, we noticed a significant drop in learning curve convergence. Figure 7 presents results using the trained model, where the average cumulative reward reached 500. However, occasional low rewards persisted, highlighting the challenge of RL in complex environments and the potential for errors in learned policies.